Porting iptables 1.4.10 to Android

(Update 25 Aprile 2011: This port is now in Google Code called iptables4n1).

Introduction 

The Android source comes with iptables 1.3.7. But that distribution is not compatible with Linux kernel 2.6.32 or newer versions. For example, even though iptables 1.3.7 can be built into the Android Emulator and Google Dev Phone 2 (a.k.a. HTC Magic) which are based on Linux kernel 2.6.29, it won't compile with kernel 2.6.32 used in Nexus One and many newer Android devices. The main difference between iptables 1.3.7 and 1.4.x is the emergence of "xtables" in the latter version. Fortunately, Google has done a commendable job in keeping Android in sync with Linux kernel releases. This makes porting a new version of iptables to Android largely an exercise of makefile changes. I will outline the steps here as a service to the open source community.

Steps

Below are steps for porting iptables 1.4.10 to Android/Linux kernel 2.6.32.

1. Check out the Android source.
2. Download iptables 1.4.10 source from the Netfilter project.
3. Go to the checked out Android source and change to $SRC/external/iptables. Delete the content underneath that directory (or make a backup if you want) and then copy the downloaded iptables 1.4.10 content to that directory.
4. Create a new Android.mk file under $SRC/external/iptables. This is the Android makefile. You can model yours after the one from the original Android source but you need to accommodate naming changes to iptable extensions that went from libiptXXX in 1.3.7 to libxtXXX in 1.4.10.
5. Change to $SRC/external/iptables/extensions and create a new create_initext4 file there. 
6. Change back to $SRC/external/iptables and run make. Fix header inclusion issues as needed.

Sample Android.mk

ifneq ($(TARGET_SIMULATOR),true)
  BUILD_IPTABLES := 1
endif
ifeq ($(BUILD_IPTABLES),1)
LOCAL_PATH:= $(call my-dir)
#
# Build libraries
#
# libxtables
include $(CLEAR_VARS)
LOCAL_C_INCLUDES:= \
    $(LOCAL_PATH)/include/ \
    $(KERNEL_HEADERS)
LOCAL_CFLAGS:=-DNO_SHARED_LIBS
LOCAL_CFLAGS+=-DXTABLES_INTERNAL
LOCAL_CFLAGS+=-DIPTABLES_VERSION=\"1.4.10\"
LOCAL_CFLAGS+=-DXTABLES_VERSION=\"1.4.10\" # -DIPT_LIB_DIR=\"$(IPT_LIBDIR)\"
LOCAL_CFLAGS+=-DXTABLES_LIBDIR
LOCAL_SRC_FILES:= \
    xtables.c
LOCAL_MODULE_TAGS:=
LOCAL_MODULE:=libxtables
include $(BUILD_STATIC_LIBRARY)
# libip4tc
include $(CLEAR_VARS)
LOCAL_C_INCLUDES:= \
    $(KERNEL_HEADERS) \
    $(LOCAL_PATH)/include/
LOCAL_CFLAGS:=-DNO_SHARED_LIBS
LOCAL_CFLAGS+=-DXTABLES_INTERNAL
LOCAL_SRC_FILES:= \
    libiptc/libip4tc.c
LOCAL_MODULE_TAGS:=
LOCAL_MODULE:=libip4tc
include $(BUILD_STATIC_LIBRARY)
# libext4
include $(CLEAR_VARS)
LOCAL_MODULE_TAGS:=
LOCAL_MODULE:=libext4
# LOCAL_MODULE_CLASS must be defined before calling $(local-intermediates-dir)
#
LOCAL_MODULE_CLASS := STATIC_LIBRARIES
intermediates := $(call local-intermediates-dir)
LOCAL_C_INCLUDES:= \
    $(LOCAL_PATH)/include/ \
    $(KERNEL_HEADERS) \
    $(intermediates)/extensions/
LOCAL_CFLAGS:=-DNO_SHARED_LIBS
LOCAL_CFLAGS+=-DXTABLES_INTERNAL
LOCAL_CFLAGS+=-D_INIT=$*_init
LOCAL_CFLAGS+=-DIPTABLES_VERSION=\"1.4.10\"
LOCAL_CFLAGS+=-DXTABLES_VERSION=\"1.4.10\"
PF_EXT_SLIB:=ah addrtype ecn 
PF_EXT_SLIB+=icmp #2mark
PF_EXT_SLIB+=realm
PF_EXT_SLIB+=ttl unclean DNAT LOG #DSCP ECN
PF_EXT_SLIB+=MASQUERADE MIRROR NETMAP REDIRECT REJECT #MARK
PF_EXT_SLIB+=SAME SNAT ULOG # TOS TCPMSS TTL
PF_EXT_SLIB+=TAG
EXT_FUNC+=$(foreach T,$(PF_EXT_SLIB),ipt_$(T))
# xtable stuff
NEW_PF_EXT_SLIB:=comment conntrack connmark dscp tcpmss esp
NEW_PF_EXT_SLIB+=hashlimit helper iprange length limit mac multiport
NEW_PF_EXT_SLIB+=owner physdev pkttype policy sctp standard state tcp
NEW_PF_EXT_SLIB+=tos udp CLASSIFY CONNMARK
NEW_PF_EXT_SLIB+=NFQUEUE NOTRACK
EXT_FUNC+=$(foreach N,$(NEW_PF_EXT_SLIB),xt_$(N))
# generated headers
GEN_INITEXT:= $(intermediates)/extensions/gen_initext4.c
$(GEN_INITEXT): PRIVATE_PATH := $(LOCAL_PATH)
$(GEN_INITEXT): PRIVATE_CUSTOM_TOOL = $(PRIVATE_PATH)/extensions/create_initext4 "$(EXT_FUNC)" > $@
$(GEN_INITEXT): PRIVATE_MODULE := $(LOCAL_MODULE)
$(GEN_INITEXT):
    $(transform-generated-source)
$(intermediates)/extensions/initext4.o : $(GEN_INITEXT)
LOCAL_GENERATED_SOURCES:= $(GEN_INITEXT)
LOCAL_SRC_FILES:= \
    $(foreach T,$(PF_EXT_SLIB),extensions/libipt_$(T).c) \
    $(foreach N,$(NEW_PF_EXT_SLIB),extensions/libxt_$(N).c) \
    extensions/initext4.c
LOCAL_STATIC_LIBRARIES := \
    libc
include $(BUILD_STATIC_LIBRARY)
#
# Build iptables
#
include $(CLEAR_VARS)
LOCAL_C_INCLUDES:= \
    $(LOCAL_PATH)/include/ \
    $(KERNEL_HEADERS)
LOCAL_CFLAGS:=-DNO_SHARED_LIBS
LOCAL_CFLAGS+=-DXTABLES_INTERNAL
LOCAL_CFLAGS+=-DIPTABLES_VERSION=\"1.4.10\"
LOCAL_CFLAGS+=-DXTABLES_VERSION=\"1.4.10\" # -DIPT_LIB_DIR=\"$(IPT_LIBDIR)\"
#LOCAL_CFLAGS+=-DIPT_LIB_DIR=\"$(IPT_LIBDIR)\"
LOCAL_SRC_FILES:= \
    iptables.c \
    iptables-standalone.c \
        xshared.c
LOCAL_MODULE_TAGS:=
LOCAL_MODULE:=iptables
LOCAL_STATIC_LIBRARIES := \
    libip4tc \
    libext4  \
        libxtables
include $(BUILD_EXECUTABLE)
endif

Sample create_initext4

#!/bin/sh
echo ""
for i in $1; do
    echo "extern void lib${i}_init(void);";
done;
echo "void init_extensions(void);"
echo "void init_extensions(void) {"
for i in $1; do
    echo "    lib${i}_init();";
done
echo "}"


That is all. Happy hacking!

Socket and O_NONBLOCK

It is often stated that a socket is created in blocking mode by default. However, not all implementations of the socket calls adhere to that behavior. Specialized operating systems can be particularly tricky when it comes to blocking and non-blocking mode . One should always test the O_NONBLOCK flag on a socket to be sure if it is in the blocking or nonblocking mode operation before proceeding to send/recv. Here are some example c code to demonstrate how to check and manipulate the O_NONBLOCK flag on a streaming socket.

/* !!! Error checking is omited for brevity !!! */
/* Create a streaming socket */
int sock = socket(AF_INET, SOCK_STREAM, 0);

/* Get the file descriptor flag */
int flags = fcntl(sock, F_GETFL, 0);

/* Set O_NONBLOCK if it is set
 * or unset it if it is not
 */
if ((flags & O_NONBLOCK ) == O_NONBLOCK) {
   fcntl(sock, F_SETFL, flags | O_NONBLOCK);  /* Set */
} else {
   fcntl(sock, F_SETFL, flags & (~O_NONBLOCK));  /* Unset */
}

How to Check the Version of a Linux Release

The tool uname is usually used to print the OS version on a Unix or Unix-like machine. But it does not always reveal the actual Linux distribution. For example:

myhost:/home/user$ uname -a
myhost:/home/user $ Linux myhost 2.6.32-24-generic-pae #43-Ubuntu SMP Thu Sep 16 15:30:27 UTC 2010 i686 GNU/Linux

If this Linux distribution is LSB compliant, we can use lsb_release to get what we need:

myhost: /home/user$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 10.04.1 LTS
Release:        10.04
Codename:       lucid

Tips in Using Gcore Dump with Java Tools

When analyzing gcore dump from a running JVM process, it is very important to know which Java binary used by gcore to generate the core dump. This is because gcore may not use the Java binary on user's path. If they are different, the user will not be able to attach to the core file to the Java tool in the path. Knowing this will save you hours of frantic head scratching. More information can be found on this discussion about using gcore with jmap.

Another issue is the core file size. A 32-bit Java tool will have problem attaching to a core dump greater than 2GB in size. A workaround is to generate the gcore on a 64-bit JVM and then use the 64-bit Java tool to load the core file.

Socket and DataInputStream

While Java provides great I/O abstraction facilities like InputStream and OutputStream, the non-blocking nature of Java NIO often leads to subtle behavior differences in high-level network programming constructs.  The case in point is DataInputStream, which enables an application to parse Java primitive types right out of an input stream.  However, the stream reading methods in the DataInputStream class exhibit two distinct behaviors when the underline stream is a socket.  The following methods are non-blocking:

read(byte[] b)
read(byte[] b, int off, int len)

A non-blocking read means that the first read(byte[] b, int off, int len)may not return the expected len number of bytes.  Therefore, the reader program is usually put into a loop to read repeatedly until the len of bytes are read or the EOF is reached.

The primitive type reading methods in DataInputStream are blocking.  Take readInt() as an example.  That read will block if there are not enough bytes, 4 in this case, in the stream.

Consistency is a good software engineering principle to practice.  When using DataInputStream to parse a socket input stream, it is best to always use the primitive type reading methods to avoid confusion of blocking vs non-blocking socket I/O.

Effect of Data Traffic Explosion on Mobile LBS

These days it is hard to go to any mobile conference without hearing someone talking about the pending explosion of data traffic volume brought on by the dawn of the Internet video age.  If you haven't tuned into the latest yet, GigaOM has a nice recap on the latest Cisco traffic study.  Is there a correlation between this onslaught of data volume and mobile LBS?  You bet.  Read on.

A key issue in mobile LBS is location accuracy and positioning duration.  Today, a mobile location is typically estimated from either GPS signals or cell towers.  GPS is well suited for high-accuracy LBS applications such as car navigation.  But the consumer applications of GPS have long been hindered by technology limitations such as long signal acquisition time and limited signal coverage.  The recent boom of mobile LBS has largely been fueled by cell-based positioning technologies.  Unlike GPS coverage, cell locations are not only much quicker to obtain but also 100% available both outdoors and indoors.  The accuracy of a cell location depends on the cell size.  The smaller the cell-size, the more accurate the cell-derived location.  This where the looming data explosion will benefit LBS.

A two-part series, titled "The Network Paradox: Meeting the Mobile Data Demand", quoted industry sources as saying that the only way to increase mobile data capacity by more than 9X is to deploy 4 or more times of cell sites.  These new cell sites will lead to at least 2 times reduction of the overall cell size according to cellular traffic engineering rules.

So, mobile LBS will see its better days ahead.

A Gotcha Moment with the View Attribute android:id

When editing layout xml files using Google's Eclipse plugin, the plugin is usually smart enough to flag typos when they cause code generation problems. However, errors that slip through the plugin undetected can cause nasty bugs at run-time. One such error involves the use of the attribute android:id. Usually, an id should be specified like this:

<!-- Correct Syntax -->
<Button android:id="@+id/abutton" />

But the following XML will compile fine without any error:

<!-- Incorrect Syntax -->
<Button id="@+id/xbutton" />

Android will even draw the button on the screen. The gotcha moment comes when you try to lookup this button in the view on a newer (such as Android 2.1) emulator/device:

Button aButton = (Button) thisView.findViewbyId(R.id.xbutton);

The aButton variable will be null after this call! What makes this a particularly hideous bug is that it was actually valid pre-Android 1.6! So apparently, the plug-in and the compiler are made backward compatible to allow the old syntax but not the run-time. Welcome to the bleeding edge...