ec2-bundle-vol Error "cert-ec2.pem: No such file or directory"

cert-ec2.pem is Amazon's public X.509 certification. ec2-bundle-vol needs it to bundle up an image to S3. But the EC2 API tools shipped with Amazon Linux AMI seem to exclude this crucial file when bundling up from an instance-store backed instance. This can cause problems down the road when you want to further customize your AMI. For example, Amazon has a 32-bit instance-store backed AMI in the us-east region with ID "ami-4b814f22". We launch an EC2 instance with this AMI, customize it, bundle it up using ec2-bundle-vol, and finally register the bundle. So we are now at, say, ami-12345678. We launch a new EC2 instance with ami-12345678, customize it again and then bundle up the new customization. But the ec2-bundle-vol command will fail this time with an error like this:

error reading certificate file /opt/aws/amitools/ec2/etc/ec2/amitools/cert-ec2.pem: No such file or directory - /opt/aws/amitools/ec2/etc/ec2/amitools/cert-ec2.pem
This looks like a bug in EC2 tools shipped by Amazon. An easy but tedious workaround is to launch an instance off the original Amazon AMI, i.e ami-4b814f22 and then copy over the cert-ec2.pem before running ec2-bundle-vol .

This problem has been reported here. Hope Amazon will devise a fix soon to save users from this misery.

Fixing libssl and libcrypto Errors in Datastax OpsCenter Startup

Update 2011-12-19: For 64-bit Amazon Linux AMI, install openssl0.9.8 by the command "sudo yum install openssl098e-0.9.8e-17.7.amzn1.x86_64". Thanks to thobbs from the datatax forum for this tip.

The AWS Linux AMI I use has openssl 1.0.0 but DataStax OpsCenter 1.3.1 requires version 0.9.8 of libssl and libcrypto. Why didn't they say so in the docs?? The worst customer experience you can give to your user base is to let your software blow up at startup like this:

Failed to load application: libcrypto.so.0.9.8: cannot open shared object file: No such file or directory

This problem was apparently reported a month ago:
http://www.datastax.com/support-forums/topic/issue-starting-opscenterd-service

But no action has been taken to correct it...sigh...

Here is how we can fix it temporarily on our own before the Cassandra devs get their acts together:

1) Install openssl 0.9.8
sudo yum install openssl098e-0.9.8e-17.7.amzn1.i686

2) Change to /usr/lib and manually create following two symbolic links:
sudo ln -s libssl.so.0.9.8e libssl.so.0.9.8
sudo ln -s libcrypto.so.0.9.8e libcrypto.so.0.9.8

Now OpsCenter will start without the dreaded ssl error.

Cassandra Range Query Using CompositeType

CompositeType is a powerful technique to create indices using regular column families instead of super families. But there is a dearth of information on how to use CompositeType in Cassandra. Introduced in 0.8.1 in May 2011 , it is a relatively new comer to Cassandra. It doesn't help that it is not even in the "official" datatype documentation on Casandra 1.0 and 0.8! This article pieces together various tidbits to bring you a complete how-to guide on programming CompositeType. The code examples will use Hector.

Let's say we want to define a column family as the following:
row key: string
column key: composite of an integer and a string
column value: string

We can define the following schema on the cli:

create column family MyCF
    with comparator = 'CompositeType(IntegerType,UTF8Type)'
    and key_validation_class = 'UTF8Type'
    and default_validation_class = 'UTF8Type';

We can also define the same schema programmatically in Hector:

// Step 1: Create a cluster
CassandraHostConfigurator chc 
      = new CassandraHostConfigurator("localhost");
Cluster cluster = HFactory.getOrCreateCluster(
                        "Test Cluster", chc);

// Step 2: Create the schema
ColumnFamilyDefinition myCfd 
      = HFactory.createColumnFamilyDefinition(
            "MyKS", "MyCF", ComparatorType.COMPOSITETYPE);
// Thanks to Shane Perry for this tip.
// http://groups.google.com/group/hector-users/
//       browse_thread/thread/ffd0895a17c7b43e)
myCfd.setComparatorTypeAlias("(IntegerType, UTF8Type)");
myCfd.setKeyValidationClass(UTF8Type.class.getName());
myCfd.setDefaultValidationClass(UTF8Type.class.getName());
KeyspaceDefinition myKs = HFactory.createKeyspaceDefinition(
      "MyKS", ThriftKsDef.DEF_STRATEGY_CLASS, 1, 
      Arrays.asList(myCfd));

// Step 3: Add schema to the cluster
cluster.addKeyspace(myKs, true);
KeySpace ks = HFactory.createKeyspace(myKs, cluster);

Now let's insert a single row with 2 columns:

String rowKey = "row1";

// First column key
Composite colKey1 = new Composite();
colKey1.addComponent(1, IntegerSerializer.get());
colKey1.addComponent("c1", StringSerializer.get());

// Second column key
Composite colKey2 = new Composite();
colKey2.addComponent(2, IntegerSerializer.get());
colKey2.addComponent("c2", StringSerializer.get());

// Insert both columns into row1 at once
Mutator<String> m 
      = HFactory.createMutator(ks, LongSerializer.get());
m.addInsertion(rowKey, "MyCF", 
      HFactory.createColumn(colKey1, "foo", 
                            new CompositeSerializer(), 
                            StringSerializer.get()));
m.addInsertion(rowKey, "MyCF", 
      HFactory.createColumn(colKey2, "bar", 
                            new CompositeSerializer(), 
                            StringSerializer.get()));
m.execute();

After the insertion, the column family should look like this table:

row1	{1, c1}	{2, c2}
	foo	bar

Now let's retrieve the first column using a slice query on only the first integer component of composite column key. Since Cassandra orders composite keys by components in each composite, we can construct a search range from {0, "a"} to {1, "\uFFFF} which will include {1, "c1"} but not {2, "c2"}.

SliceQuery<String, Composite, String> sq 
      =  HFactory.createSliceQuery(ks, StringSerializer(), 
                                   new CompositeSerializer(), 
                                   StringSerializer());
sq.setColumnFamily("MyCF");
sq.setKey("row1");

// Create a composite search range
Composite start = new Composite();
start.addComponent(0, IntegerSerializer.get());
start.addComponent("a", StringSerliazer.get());
Composite finish = new Composite();
finish.addComponent(1, IntegerSerializer.get());
finish.addComponent(Character.toString(Character.MAX_VALUE), 
                    StringSerliazer.get());
sq.setRange(start, finish, false, 100);

// Now search.
sq.execute();
// TODO: Parse the result to get the first column

It is unfortunate that a JavaDoc typo in the Cassandra source code prevents tools like Eclipse from displaying documentation about CompositeType. But you can always view the source online to get the precision definition and encoding scheme of CompositeType. Reading source code has been and is still the best way of learning new features in Cassandra.

GuiceFilter and Static Resources

GuiceContextListener and GuiceFilter can eliminate servlet mappings completely from the web.xml file. They also preserve the default servlet handling logic by re-routing a URL request back to the default servlet when none of Guice-configured servlet matches the requested URL. This technique can be used to serve static resources from an Guice-configured web app. For example, assuming this simple web.xml that routes everything through Guice:

<webapp>
  <listener>
    <listener-class>com.myapp.MyGuiceContextListener</listener-class>
  </listener>
  <filter>
    <filter-name>guiceFilter</filter-name>
    <filter-class>com.google.inject.servlet.GuiceFilter</filter-class>
  </filter>
</webapp>

If we have a file named myIndex.html at the same directory level with WEB-INF in the web app layout, we can then easily request this file by using an URL like this:

http://mydomain/myIndex.html

In this case, the GuiceFilter is smart enough to reroute the request to the default servlet for serving the myIndex.html file.

Rules for Configuring ThreadPoolExecutor Pool Size

I came across a blog Rules of ThreadPoolExecutor Pool Size. In that blog, the author explained reasonably well how TPE creates new threads in relation to thread pool size. But the author incorrectly stated that the TPE would always fill a task queue first before creating a new thread. Thread creation is determined by queuing strategy. Choosing a direct handoff strategy will achieve the author's "user anticipated way". But there is risk in that particular way. Here is a table listing the general pros and cons of each queuing strategy:

Queuing Strategy	Example	Design Trade-off	Worst Case	Usage
Direct Handoff	SynchronousQueue	Zero task queue but can create unlimited number of threads.	OutOfMemory	For tasks that may have interdependency. For example, task i changes a global state that affects the execution of task j.
Unbounded Queue	LinkedBlokingQueue	Limit the number of threads by the max pool size but allows submission of unlimited tasks.	OutOfMemory	For tasks that are completely independent of each other.
BoundedQueue	ArrayBlockingQueue	Limit both the number of threads and the queue size.	Low throughput from imbalanced pool and queue sizes.	For burning midnight oil.

The “initial_token” in Cassandra Means the “Very First Time”

Cassandra uses tokens to split key ranges across nodes. When a Cassandra node is started the very first time, it will check if an “initial token” is specified in cassandra.yaml; otherwise, the node will generate a token from the cluster it is joining. But how does a node know that it is being started the “very first time”? It is simple. The token is stored on the local disk and persists across process start/stop. Therefore, once a token is stored, changing the “initial_token” parameter in cassandra.yaml will have no effect. When multiple nodes have the same token, Cassandra will elect a new owner of the token, print out an warning and then continue on. The nodetool however will under-report the number of nodes in a ring because it only probes nodes that have unique tokens. It is such a common problem when making Cassandra VM images that it even gets its own FAQ on the Cassandra wiki. The only safe way to create a new token cleanly is to wipe out the data and commit logs and then restart the node.

Configuring Maven to Use a Local Library Folder

The official "Maven Way" of dependency management is to use Maven Central and local repository specified in the settings.xml file (which usually points to $HOME/.m2/repository. While it works great for projects that rely on a large number of open source libraries and satisfies 95% of dependency management needs in those projects, there is that 5% of the time when a jar is not sourced from a Maven project. One example is a jar using JNI so is only available for certain OS platforms. How do we integrate this jar into Maven dependency management? If you search the web for hints, you may be led to believe that you either have to bend to the Maven Way or to use the systemPath. But the Maven Way will force you to maintain a local repository for a trival library. The systemPath on the other hand does not work naturally with packaging. Developers will most likely ask "Can I check in this library to my (your_favorite_VCS) with my project and still have Maven use it in a way just like any other dependency?" The answer is YES. Just follow the steps below:

1. Create a directory under your project, say "lib".

2. Use Maven to install your jar to the lib directory.
mvn install:install-file -Dfile=path_to_mylib.jar -DgroupId=com.mylib -DartifactId=mylib -Dversion=1.0 -Dpackaging=jar -DlocalRepositoryPath=path_to_my_project/lib

3. Setup your POM like this.

  <repositories>
     <repository>
         <!-- DO NOT set id to "local" because it is reserved by Maven -->
         <id>lib</id>
         <url>file://${project.basedir}/lib</url>
     </repository>
  </repositories>
  <dependencies>
    <dependency>
        <groupId>com.mylib</groupId>
        <artifactId>mylib</artifactId>
        <version>1.0</version>
    </dependency>
  </dependencies>

Now you can check in/out mylib.jar just like any other file in your project and Maven will manage the dependency on mylib.jar just like any other dependency artifact. Perfect harmony. :-)