How to Quickly Evaluate Cloud Fitness

Cloud computing is an effective platform to build highly scalable and fault tolerant software products. But cloud computing is often confused with clustering. The so-called private cloud often muddles the water even more. While virtualization gives programmers the illusion of disappearing physical server boundaries, in reality virtualization also makes CPUs and networks themselves disappear easily on a whim. Therefore, a software product that is designed to run on physical servers in a data center often fails to live up to its promise in an elastic cloud environment. This article offers four simple clues to reveal if a software product is fit for cloud. The “cloud” used here refers to Amazon Web Service (AWS).

Clue #1: Are the addresses of all cluster nodes defined in a configuration file?
This is the first sign of trouble. An elastic virtual machine doesn’t have an IP address until it boots up.

Clue #2: Is shared storage required between cluster nodes?
There is no shared storage in the cloud.

Clue #3: Does the cluster use multicast?
Multicast strikes fear to the hearts of network admins. Muticast packets are problematic in traversing subnets. A layer 2 network emulated on top of layer 3 only makes the problem worse. No wonder multicast is disabled in AWS.

Clue #4: Does the cluster rely on UDP to manage cluster membership?
Your “subnet” in the cloud actually runs on an overlay network. Ever heard of routers dropping UDP packets in time of congestion?

Armed with these three clues, one can quickly filter out a lot of vendor noise in the web sphere today.

Poor Man's Static IP for EC2 a.k.a. Elastic Network Interface

Amamzon's Elastic Network Interface (EIN) allows you to "reserve" an IP address. This is immensely useful in VPC because an EIN can function as a pseudo static IP for elastic instances. Granted, you have to use two IPs for a single instance. But EIN lets you assign a fixed private IP address to an elastic instance without having to go through the trouble of setting up dynamic DNS update. Unfortunately, Amazon's documentation is missing key information on configurting secondary IP with EIN. Even if you have attached an EIN to an elastic instance, you cannot access the instance using the private IP associated with the EIN. What gives? The missing piece is IP interface and routing configuration. Below is a step-to-step guide to configure the EIN interface. This guide assumes that you have followed the official AWS guide to the point where you have configured an EIN and have brought up an elastic instance that is attached with that EIN. Further, it assumes that the primary interface is assigned an IP 10.3.1.190 and the secondary interface, which is the EIN, is assigned an IP 10.3.1.191. At the end of the exercise, we will be able to ssh to the secondary IP address in addition to the primary one.

First, check IP address binding to each network interface.

$ sudo ip address

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:26:69:f0:87:46 brd ff:ff:ff:ff:ff:ff
inet 10.3.1.190/24 brd 10.3.1.255 scope global eth0
inet6 fe80::26:69ff:fef0:8746/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 02:26:69:dc:cc:62 brd ff:ff:ff:ff:ff:ff

We see from the output the current network interface assignment is of the following:

eth0: 10.3.1.190
eth1: none

Therefore,  the first order of business is to assign the EIN IP address to the interface eth1:

$ sudo ip address add 10.3.1.191/24 brd + dev eth1

Next, bring up the interface:

$ sudo ip link set dev eth1 up

Verify that eth1 is indeed up:

$ sudo ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:26:69:f0:87:46 brd ff:ff:ff:ff:ff:ff
inet 10.3.1.190/24 brd 10.3.1.255 scope global eth0
inet6 fe80::26:69ff:fef0:8746/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:26:69:dc:cc:62 brd ff:ff:ff:ff:ff:ff
inet 10.3.1.191/24 brd 10.3.1.255 scope global eth1
inet6 fe80::26:69ff:fedc:cc62/64 scope link
valid_lft forever preferred_lft forever

Next, find out the default gateway:

$ ip route show
default via 10.3.1.1 dev eth0
10.3.1.0/24 dev eth0 proto kernel scope link src 10.3.1.190
10.3.1.0/24 dev eth1 proto kernel scope link src 10.3.1.191

The default gateway is 10.3.1.1 in the output. It is bound to the virtual gateway associated with the VPC.  Since it is currently only bound to eth0, any traffic from eth1 that is destined to IP addresses outside the 10.3.1.0/24 IP block will be dropped! We need to reconfigure IP routing on the elastic instance to allow IP packets leaving eth1 to be routed through the default gateway. Here is how you do it.

First, add a new routing table called "awsein":

$ sudo echo 2 awsein >> /etc/iproute2/rt_tables

It will add a table called "awsein" to rt_tables as entry 2:
$ cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
2 awsein

Now adds a default route in the new table to use the same default gateway as the one used by eth0:

$ sudo ip route add default via 10.3.1.1 dev eth1 table awsein
$ sudo ip route flush cache

Confirm that the new route is indeed added:

$ ip route show table awsein
default via 10.3.1.1 dev eth1 metric 1000

Next, we need to create a new routing rule to trigger the default route on eth1 by its source IP. To do this, we first check existing routes:
$ ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default

Note the number 32766 for the rule "main". We will now add a new rule to "awsein" with a priority smaller than the one for "main".

$ sudo ip rule add from 10.3.1.191 lookup awsein prio 1000
Finally, verify the new rule configuration:

$ ip rule
0:      from all lookup local
1000:   from 10.3.1.191 lookup awsein
32766:  from all lookup main
32767:  from all lookup default

Now you can ssh into the instance using the EIN IP 10.3.1.191! Happy hacking.

Bundling Local File System with ec2-user

Remote root login is disabled on all Amazon Linux AMIs to prevent exploits. By default, you can only login as "ec2-user" to access instances launched from those AMIs. The ec2-user is in the sudo group so it can perform tasks that require root privilege. One example is ec2-bundle-vol that bundles up local file system for creating custom AMI. But unlike ec2-user, sudo is not configured with the default AWS environment variables like EC2_HOME and etc. An easy workaround is to use the "-E" option to let sudo inherit environment variables from ec2-user. From the sudo man page:

The -E (preserve environment) option will override the env_reset option in sudoers(5)). It is only available when either the matching command has the SETENV tag or the setenv option is set in sudoers(5).
Now, we can execute the ec2-bundle-vol command like this:

sudo -E /opt/aws/bin/ec2-bundle-vol -k /media/ephemeral0/private-key.pem -c /media/ephemeral0/cert.pem -u XXXXXXXXXXXX

I am surprised that this is not documented in the official doc.

Configuring and Customizing Elastic Bamboo

The Elastic Bamboo feature in JIRA Studio can utilize Amazon EC2 for builds. But integrating EC2 into Bamboo can be an overwhelming task for a beginner. To complement Atlassian's documentation, this article gives a few tips on how to setup a EC2 build account and configure EB. Even though this is not a Bamboo or a AWS tutorial, you can follow links in the article to learn more about the subjects applicable to the task at hand.

Setting up a Special Build User in AWS

Since EB needs AWS security credentials to launch and manage EC2 instances, you can use AWS Identity and Access Management (IAM) to setup a special build user with only EC2 permissions. Let's call this special user "eb-builder". Give this user its own access keys and X.509 certificates. Follow the IAM User Guide if you don't already know how to do this.

Configuring Elastic Bamboo to Use AWS

SSH Key Pair
These are the keys you will need later to log onto the EC2 instance launched by EB. When you configure EB for the first time, it will automatically generate a key pair for you called elasticbamboo. If you use hosted JIRA Studio, the private key is stored in the Atlassian host and can be downloaded later.

AWS Access Keys
EB needs access keys to call AWS APIs in order to perform AWS actions such as us launching EC2 instances. Instead of supplying EB the access keys from your master AWS account, use access keys you generated earlier for eb-builder.

AWS X.509 Certificate and Private Key
EB needs both the certificate and the private key to use EBS. Again, supplying EB with the certificate and the private key you generated earlier for eb-builder, not the ones you created for your master AWS account. This is a good security practice to keep in mind.

Customizing Elastic Bamboo Default Image

There are two reasons for customizing the default EB image:

(1) To persist build dependencies like 3rd party libraries. The default image uses instance-store which does not persist after the EC2 instance is gone. One easy way to persist build dependencies is to use an EBS volume.
(2) To add your own builder. For example, you may want Maven 3 instead of Maven 2 that comes with the default image.

Use EBS
EBS comes in handy when you have relatively static build dependencies like 3rd party libraries. It also allows you to install your own build tools instead of using those from the default image. You can follow the steps here to create an EBS volume to store your custom bits. But note Step 2 in Creating your first EBS snapshot on that link. That step is not applicable to the hosted version of JIRA Studio. In that case, you must file a support ticket and ask Atlassian to load your X.509 certificate and the private key for you. If you install a new Maven release like Maven 3 or a new JDK, do not put them in the bamboo:bamboo user path. Instead, configure their paths when you add a new custom builder as below.

Add Custom Builder
Let's say you have installed Maven 3 under /mnt/bamboo-ebs/maven3 and made a snapshot of the EBS volume. To configure EB to use the new Maven bits, you follow the Add Capabilities steps here. When adding a new builder for Maven 3, make sure you select the "Maven 2.x" capability type. Hopefully Atlassian will add an official Maven 3 capability type in the near future. Once you complete those steps, the new Maven 3 builder will show up as a builder candidate when you setup a build plan for your project.

That's the gist of configuring and customizing Elastic Bamboo. Happy building!