Highly-Available Load Balancing of Apache Tomcat using HAProxy, stunnel and keepalived

This article will describe the process required to set up a highly-available SSL-enabled load balancer using HAProxy, stunnel and keepalived to front a pair of Apache Tomcat instances.

The configuration will start off simple, and extend and grow as more functionality is added. First, a session-aware HAProxy load balancer is configured. Next, Tomcat clustering is configured with session replication and the responsibility for maintaining session state is moved to Tomcat. Then, keepalived is added – providing a failover IP between HAProxy instances running on two nodes. The solution is then fully HA. To complete the article, SSL support will be enabled by way of adding stunnel in front of the HAProxy instances, and a few cleanup tasks performed.

Faithful dolan and gooby will be used for this in my lab environment. Each of the servers is running CentOS 6.3 x86_64 and already has a Tomcat instance installed to /usr/local/tomcat7 (running as the tomcat user).

As you can see, I’ve also reserved an IP address for our floating IP (or Virtual IP – VIP).

The end goal is to have dolan and gooby both running Tomcat as before, also running their own HAProxy and stunnel instances but also running a failover VIP provided by keepalived.

Session-aware Load Balancing with HAProxy

First, download HAProxy and install to the default location (which will place the binary at /usr/local/sbin/haproxy, and expect its configuration to be at /etc/haproxy/haproxy.conf:

Install the software on both nodes. Only configure on dolan for now as HA will be added later. Create haproxy.conf as follows:

Most of the configuration above are sensible defaults taken from the HAProxy documentation. A few points to note: HAProxy is configured to bind on all interfaces on port 80 – this will enable us to test by hitting individual nodes, or to hit the floating IP address that we will configure later. Within the backend configuration, option http-server-close is very important – without this, only the first request will have its headers inspected for cookies and hence our JSESSIONID – so sticky sessions will not work. Newer versions of HAProxy (I’m using 1.4.22) support the appsession directive. This will, in our configuration, store the JSESSIONID in a hashtable (well – 52 bytes of it) for 3 hours, along with the backend node that serviced the request – thus implementing sticky sessions.

This can also be done with the following backend declaration:

This will cause HAProxy to check for a specific nodename value within the session ID.

I opted for appsession. Configure the Tomcat <Engine> directive to correctly support jvmRoute based load balancing (only important if you opt for the second configuration). This will set the JSESSIONID as SESSIONID.jvmRoute which will enable HAProxy to route the request appropriately (as we define cookie checking in the server directives in the second example). Even when using appsession (which maintains its own session state table) it will help for testing to see which backend node set the SESSIONID in the first place, so I’ll set it:

These values should be set to whichever nodename you’re checking for in your HAProxy configuration, if not using appsession (or even if you are – see above). Restart Tomcat (depending upon your setup this will be probably be different):

Now, start HAProxy referencing the configuration file created previously. Note: HAProxy does support running inside a chroot as well as dropping privileges to a different user – I’ve not bothered for this simple article – it is easy to do and is well documented):

Next, browse to (presuming you’ve left the Tomcat examples intact)

You will see your session ID returned. Refresh the page a few times, add a few session variables using the example, make sure the same session ID is returned each time. If you see new session IDs being generated, check your Tomcat access logs – you’ve misconfigured something and are probably round-robin load balancing across your Tomcat nodes, and your whole setup is session-unaware. Also check all your HAProxy directives are correct. Logging will be configured for HAProxy towards the end of the article once everything else is setup.

So – we now have a session-aware load balancer configured using HAProxy and appsession-based session tracking. The trouble with this configuration is that if we chose to make it highly-available and attempt a failover of HAProxy to a second node, the session state information would not be failed over with it – and all sessions would be broken.

Tomcat Clustering

A rethink is required. Presuming your well-behaved application deployed to Tomcat is distributable and serialised, session replication can be configured within Tomcat. Presuming you have no other clusters running on your network, you can just use the default cluster configuration supplied with Tomcat and implemented via the org.apache.catalina.ha.tcp.SimpleTcpCluster class. Remove the jvmRoute statements added to your <Engine> directives if you’ve opted for using them (depending on how you implemented the first stage of load balancing). Uncomment, or add, the following to server.xml:

You can modify the default configuration by following the documentation available on the Tomcat site. Restart your Tomcat instances.

If you check catalina.out, you’ll see messages such as the following when cluster membership changes. This example is taken from dolan as gooby joined the cluster:

Next, again presuming your Tomcat examples are still deployed to your instances, modify /path/to/tomcat7/webapps/examples/WEB-INF/web.xml, and add the following just above the <filter> directives:

You will need to restart Tomcat or redeploy the application. We will be able to test using one of the examples provided. It is worth noting from the Tomcat 7.0 documentation:

– All your session attributes must implement java.io.Serializable
– Uncomment the Cluster element in server.xml
– Make sure your web.xml has the distributable element

So, if you do not implement java.io.Serializable then it’s still not going to work. I’ll just opt for one of the examples that does – but (as an example) the shopping cart example provided with Tomcat 7.0 does not implement java.io.Serializable. You can modify the code and recompile the class if you’d like to test further, but the Hello World JSP test will work fine, along with a header inspector to check our session IDs.

Next, modify the HAProxy configuration as follows:

As you can see, there is no special handling of cookies or sessions with our HAProxy configuration – it will all be handled by session replication within Tomcat.

Restart HAProxy:

Browse to the following whilst using something like Live HTTP Headers with Firefox:

Note the session ID. Refresh your browser a few times; you should note the session ID doesn’t change, but the load is still being spread across your backend Tomcat servers (checking your Tomcat access logs for now – HAProxy will be configured for logging later).

Shutdown a tomcat instance, the session ID will be the same. Bring a Tomcat instance back up, and shut the other one down – you should note the ID is not changing. Session replication is working at the Tomcat layer. If the session ID is changing, check your Tomcat cluster configuration, and that you’ve made the application both distributable and implemented serialisation if required.

Making our Load Balancer Highly-Available: keepalived

We can now make our load balancer highly-available by installing keepalived. First, copy the HAProxy configuration across from the first node to the second node, as it will not change any further:

Test HAProxy on the second node:

Hit the Tomcat examples and verify that all is working as expected. All remaining steps should be performed on both nodes unless noted.

Install the required prerequisite packages:

Now, download the latest keepalived and install it:

Without specifying a --prefix during the compile, make install happily scatters files all over /usr/local. Ho hum. For now, copy the appropriate files into place:

Next, configure keepalived:

As per the comment – configure the master server (dolan) to have a higher priority than the slave (gooby).

Modify the daemon line in /etc/init.d/keepalived so it actually works:

Ensure that net.ipv4.ip_nonlocal_bind is set:

Before starting keepalived, ensure that HAProxy is running on both nodes. Start keepalived:

You should see the VIP as a subinterface of eth0 (or whatever you’ve defined in keepalived.conf) on the master (dolan):

Shutdown HAProxy on the master:

You should see the IP address failover to the slave:

Start HAProxy again on the master, and hit the Tomcat examples via the VIP address. Your cluster should be working as expected. Verify the cluster by failing the VIP between nodes by starting/stopping HAProxy, and ensure that your session-replicated Tomcat application is still working, and that your session ID doesn’t change.

Adding SSL Support

Our home-brewed load balancer is working, but it lacks one important thing for any decent web-facing application: SSL Support. Only the latest development releases of HAProxy support SSL natively, so I’ll use stunnel to provide SSL termination, listening on our VIP on port 443, before hand off to HAProxy listening on our VIP on port 80. Again, as with HAProxy, stunnel will need to be configured and running on both nodes, although only one node will be receiving requests at any one time.

Download stunnel from http://www.stunnel.org. Compile and install:

Create the configuration directory:

Create a self-signed certificate for testing:

Now for the important bit – stunnel expects the key and the certificate to be in the same file. Let’s do that:

As stunnel will have access to our SSL private key, I decided to chroot it and drop privileges down to user/group stunnel/stunnel. So, create the chroot and the appropriate user/group:

Configure stunnel appropriately:

As you can see, we accept traffic on VIP:443, terminate SSL, and handoff to HAProxy on VIP:80. pid must be set relative to the chroot.

Start stunnel manually on both nodes:

Hit the tomcat examples (https://VIP/examples) and verify that all is well. You’ll obviously need to accept any browser-based warnings about the self-signed certificate.

Modify /etc/keepalived/keepalived.conf to check that both HAProxy AND stunnel are running:

Now, failover will occur should either component fail.

Restart keepalived:

Test killing off either haproxy or stunnel on a node, and ensure failover occurs. Double check that SSL and the examples are still working.

We are almost there. Our load-balancer is now SSL-enabled, and the Tomcat backend is session-aware and happily clustered and replicating session data.

Time for a little cleanup.

Cleanup and Final Tasks

First, I’ll configure rsyslog (we’re using a RHEL 6.x derivative so rsyslog is being used).

On both servers, edit /etc/rsyslog.conf and add the following to the syslog rules:

Note – we’re using local0 as our facility – this must match whatever you’ve defined in haproxy.conf. Also, enable UDP reception by uncommenting the following lines:

Restart rsyslog:

Next, restart your HAProxy instances (watching keepalived failing the VIP around), and watch /var/log/haproxy.log. Make some requests to the load balancer and you should see entries such as:

As you can see, our requests are hitting the two backend nodes (defined in our haproxy.conf as node1 and node2) round-robin, which is what we expect.

Next, install the /etc/init.d script for HAProxy. The sed commands are specific to my installation and you may not need them:

From this point, haproxy should be managed via the service command. Install the /etc/init.d script for stunnel:

And (re)start it:

Again, stunnel should now also be managed via the service command.

The HA load balancer solution is complete!


This article has walked through building a load balancer that slowly grows in complexity, into a highly functional SSL-enabled HA load-balancer pair, sitting in front of a clustered Tomcat installation.

HAProxy, stunnel, keepalived and Tomcat all support much more functionality than I can cram into a single article, but I’ll be sure to write about any interesting nuggets over at tokiwinter.com.