Jails and Services

When I work on a computer, be it building up the infrastructure or creating a fairly complex web application, I work with layers. Each layer does its own thing and builds on the previous layers while leaving itself open enough that it can be used to build the next higher layer. This comes in part from my experience with FORTH, a stack-based language that organizes code into dictionaries of word definitions so that the programmer works entirely bottom-up. The result is often a set of tools for attacking a problem instead of a monolithic application.

I took this layered approach when engineering our server cluster supporting digital humanities, instructional technology, and miscellaneous college services. The result is illustrated in Figure 1 with a configuration management system named Gorgik that manages the actual servers Udrog and Nivu. These two servers in turn manage bundles of interrelated services that are exposed to the outside world through IP addresses assigned to the services. In the particular example shown in Figure 1, the digital humanities and instructional technology bundles are running on Udrog while the college bundle is running on Nivu.

Figure 1: Services arrangement on Udrog and Nivu

This ability to manage services at a granularity finer than an entire server allows us not only to have one server fail without losing services, but it allows us to balance services across servers. We can make use of all of our resources all of the time instead of having a system sitting idle waiting to take over when another system fails. This arrangement isn't always ideal since it means both systems are consuming power and heating the machine room. Whether or not you want servers to sit idle is a judgment call you will need to make based on your particular circumstances.

We assign services to a bundle based on what other services they access and which IP address clients expect to use when accessing those services. For example, all of the digital humanities services use dh-site.tamu.edu and dh-svc.tamu.edu as their public IP addresses with more mnemonic hostnames added as aliases (e.g., dh-shell.tamu.edu and dh.tamu.edu).

Every service we offer is contained within a FreeBSD jail. Jails are a way to isolate programs so that they don't see everything else on the system. We also don't have to commit resources that can't be used by other services, allowing a more flexible use of memory and disk than is usually allowed by virtualization software that runs on top of an operating system (e.g., Parallels). Instead of running a separate operating system kernel for each jail, they all share the same kernel, memory, and other resources with the kernel acting as the gatekeeper. We use the firewall of the underlying system to map the external IP addresses to the internal jails as illustrated in Table 1.

Jails also allow us to partition our data so only the information needed for a particular service is available to that service. In addition to the file systems shown in Figure 2, we maintain a single copy of the base operating system and installable packages, providing it on a read-only basis to each jailed service.

Figure 2: File system mappings for digital humanities services.

Next time, we'll take a look at the Puppet configuration on Gorgik and see how it manages Udrog/Nivu and the jails.