The “initial_token” in Cassandra Means the “Very First Time”

Cassandra uses tokens to split key ranges across nodes. When a Cassandra node is started the very first time, it will check if an “initial token” is specified in cassandra.yaml; otherwise, the node will generate a token from the cluster it is joining. But how does a node know that it is being started the “very first time”? It is simple. The token is stored on the local disk and persists across process start/stop. Therefore, once a token is stored, changing the “initial_token” parameter in cassandra.yaml will have no effect. When multiple nodes have the same token, Cassandra will elect a new owner of the token, print out an warning and then continue on. The nodetool however will under-report the number of nodes in a ring because it only probes nodes that have unique tokens. It is such a common problem when making Cassandra VM images that it even gets its own FAQ on the Cassandra wiki. The only safe way to create a new token cleanly is to wipe out the data and commit logs and then restart the node.

Configuring Maven to Use a Local Library Folder

The official "Maven Way" of dependency management is to use Maven Central and local repository specified in the settings.xml file (which usually points to $HOME/.m2/repository. While it works great for projects that rely on a large number of open source libraries and satisfies 95% of dependency management needs in those projects, there is that 5% of the time when a jar is not sourced from a Maven project. One example is a jar using JNI so is only available for certain OS platforms. How do we integrate this jar into Maven dependency management? If you search the web for hints, you may be led to believe that you either have to bend to the Maven Way or to use the systemPath. But the Maven Way will force you to maintain a local repository for a trival library. The systemPath on the other hand does not work naturally with packaging. Developers will most likely ask "Can I check in this library to my (your_favorite_VCS) with my project and still have Maven use it in a way just like any other dependency?" The answer is YES. Just follow the steps below:

1. Create a directory under your project, say "lib".

2. Use Maven to install your jar to the lib directory.
mvn install:install-file -Dfile=path_to_mylib.jar -DgroupId=com.mylib -DartifactId=mylib -Dversion=1.0 -Dpackaging=jar -DlocalRepositoryPath=path_to_my_project/lib

3. Setup your POM like this.

  <repositories>
     <repository>
         <!-- DO NOT set id to "local" because it is reserved by Maven -->
         <id>lib</id>
         <url>file://${project.basedir}/lib</url>
     </repository>
  </repositories>
  <dependencies>
    <dependency>
        <groupId>com.mylib</groupId>
        <artifactId>mylib</artifactId>
        <version>1.0</version>
    </dependency>
  </dependencies>

Now you can check in/out mylib.jar just like any other file in your project and Maven will manage the dependency on mylib.jar just like any other dependency artifact. Perfect harmony. :-)

Log4J Appender Additivity in Plain English

Let's start with the root logger in a Log4j.properties file:

log4j.rootLogger=INFO,stdout

This root logger is configured to have a logging level INFO with an appender named stdout. Now we want to turn debug on in our own package but keep the rest at the INFO level. So we add this to our Log4j.properties file:

log4j.category.com.mypackage.name=DEBUG
log4j.rootLogger=INFO,stdout

Everything looks good. But then we want to pipe our debug log to a different appender so we change the configuration to:

log4j.category.com.mypackage.name=DEBUG, myappender
log4j.rootLogger=INFO,stdout

When we start our app, we suddently notice that our debug logs still show up in stdout in addition to myappender! This is caused by appender additivity. To turn it off, change the additivity flag to false:

log4j.category.com.mypackage.name=DEBUG, myappender
log4j.additivity.com.mypackage.name=false
log4j.rootLogger=INFO,stdout

Counting All Rows in Cassandra

Update Oct. 25, 2011: Fixed missing key type in the code fragment.

The SQL language makes counting rows deceptively simple:

SELECT count(*) from MYTABLE;

The count function in the select clause iterates through all rows retrieved from mytable to arrive at a total count. But it is an anti-pattern to iterate through all rows in a column family in Cassandra because Cassandra is a distributed datastore. By its very nature of Big-Data, the total row count of a column family may not even fit in memory on a single 32-bit machine! But sometimes when you load a large static lookup table into a column family, you may want to verify that all rows are indeed stored in the cluster. However, before you start writing code to count rows, you should remember that:

• Counting by retrieving all rows is slow.
• The first scan may not return the total count due to delay in replication.

Now, we know why we shouldn't iterate through all rows in Cassandra in the first place, we can proceed to write a little function to do exactly that for those rare occasions. Below is an example using Hector and the iterative method. The key space in this example uses Random Partitioner. The example function uses the Range Slice Query technique to iterate through all rows in the order of MD5 hash value of keys. Note that Cassandra uses MD5 hash interally for Random Partitioner.

   public int totalRowCount() {
      String start = null;
      String lastEnd = null;
      int count = 0;
      while (true) {
         RangeSlicesQuery<String, String, String> rsq = 
            HFactory.createRangeSlicesQuery(ksp, StringSerializer.get(),
                  StringSerializer.get(), StringSerializer.get());
         rsq.setColumnFamily("MY_CF");
         rsq.setColumnNames("MY_CNAME");
         // Nulls are the same as get_range_slices with empty strs.
         rsq.setKeys(start, null); 
         rsq.setReturnKeysOnly(); // Return column names instead of values
         rsq.setRowCount(1000); // Arbiturary default
         OrderedRows<String, String, String> rows = rsq.execute().get();
         int rowCount = rows.getCount();
         if (rowCount == 0) {
            break;
         } else {
            start = rows.peekLast().getKey();
            if (lastEnd != null && start.compareTo(lastEnd) == 0) {
               break;
            }
            count += rowCount - 1; // Key range is inclusive
            lastEnd = start;
         }
      }
      if (count > 0) {
         count += 1;
      }
      return count;
   }

Recursion would be a more elegant solution but be aware of the stack limitation in Java.