Using Honcho to Create a Multi-Process Docker Container
A common misconception is that Docker is only for creating single-process or single-service containers. While it’s true that the
docker run command options are designed for running a single process, that doesn’t mean that Docker itself doesn’t allow for a multi-process Docker container.
In fact, Docker’s documentation has a very useful tutorial on how to run multi-process containers using Supervisor to manage the processes within the container.
Why I Don’t Use Supervisor for Docker Containers
While Supervisor is a great tool (I’m a big fan), I don’t personally like using it for multi-process Docker containers. This is due to how Supervisor handles process failures.
Supervisor is an application designed to start processes and keep those processes running if they fail. This feature can be very useful in some cases if configured appropriately. If left to its default configuration however, you can easily find yourself with a container that is not providing the service it is supposed to provide, and you may not even know that service is down.
By default, Supervisor will auto-restart processes that fail. If the managed process fails three times in a row or exits in a way Supervisor is not configured to handle, the
supervisord process will stop restarting the process. The problem is that the
supervisord process itself does not exit; it simply logs the event.
This means that the container will still be running, but the important service within the container is dead.
Using Honcho to Create a Multi-Process Container
Honcho is a tool for managing Procfile-based applications. A
Procfile is a file that specifies which commands should be executed to start an application via tools like Honcho.
We can use Honcho to start and manage all of the processes defined within a
Procfile. The nice thing about Honcho is if any one of these processes exit abnormally, by default the whole Honcho process and sub-processes are stopped. This causes the Docker container to exit, which is the desired effect for this type of scenario.
In this article, we will use Honcho to create a custom multi-process Docker container. This container will be used to host a Redis service that supports TLS connections.
Redis-tls Docker container
Redis is a highly popular in-memory datastore. It does not, however, currently support SSL or TLS. To add TLS support to Redis, we will need to run another application called stunnel. The stunnel service is an SSL/TLS proxy that can be used as a reverse proxy to perform TLS offloading for services that do not natively support TLS.
While it is possible to deploy stunnel as a stand-alone container and have it linked to a Redis container, for this article we will be combining these two processes into a single container.
redis-tls Docker container we will be launching today will run two processes: the first being
stunnel and the second being
redis-server. As Redis traffic arrives to the containe, it will first pass through
stunnel, which will perform all of the TLS communication. From there,
stunnel will then forward the unencrypted traffic to the
This allows for communications outside of the container to be encrypted, with non-encrypted traffic being contained within the container.
redis-tls container image does not exist today, we will also be creating our own custom Docker image as part of this article. To start this process, we will first need to define a
Creating a custom Docker image from a Redis base
The first instruction in any
Dockerfile is usually
FROM. This instruction is used to define what base image the Docker container should be created from.
Since we will be creating a Redis container with TLS support, we can base our container from the standard
redis Docker image by specifying
redis in the
A benefit of basing our image on the official
redis image is that our image will inherit all of the latest features of the official
redis image when updates occur. This keeps the
redis-server installation up to date with the latest security and bug fixes.
Installing stunnel and pip
redis image as our base image, we don’t have to go through the effort of installing Redis. We will however need to install the
stunnel package. We’ll also need to install the
python-pip package to install
pip command will be used to install Honcho within the
To install these packages, we will add a
RUN instruction calling the Apt package manager.
FROM redis RUN apt-get update --fix-missing && \ apt-get install -y stunnel python-pip && \ rm -rf /var/lib/apt/lists/*
The above will run three instructions during the Docker build process:
- First instruction: the
apt-getcommand with the
updateparameter. This updates the Apt repository cache within the container.
- Second instruction: the
apt-getcommand, but this time with the
installparameter. This instruction will install the
- Third instruction:
rm -rf /var/lib/apt/lists/*, which clears Apt’s package cache. This is useful for keeping our Docker container small as the
apt-get updatecommand creates quite a bit of cached data.
After the installation of packages with Apt, the next build step will be to install Honcho. To do this, we’ll use the
RUN instruction again but this time to call the
FROM redis RUN apt-get update --fix-missing && \ apt-get install -y stunnel python-pip && \ rm -rf /var/lib/apt/lists/* RUN pip install honcho
After this build step, we will have completed all of the installation steps. Next, we’ll focus on configuration of both stunnel and Honcho.
With stunnel, some configuration is needed before it can provide the TLS reverse proxy functionality. Luckily, the stunnel configuration is fairly straight forward.
To configure stunnel, we need to specify that stunnel runs in the foreground, which port to accept connections on (
6379 the default Redis port), where to forward those connections to (
6380 a port unique for this container), and the certificates to use for the TLS encryption.
We will add all of these configurations into the
stunnel.conf file within our local build directory.
foreground = yes debug = 7 [redis] accept = 0.0.0.0:6379 connect = localhost:6380 cert = /certs/cert.pem key = /certs/key.pem
stunnel.conf file is created, we need tell Docker to add this file to the container during build. We can do this by using the
ADD instruction within the
FROM redis RUN apt-get update --fix-missing && \ apt-get install -y stunnel python-pip && \ rm -rf /var/lib/apt/lists/* RUN pip install honcho ADD stunnel.conf /stunnel.conf
During the Docker build process, Docker will place the
stunnel.conf file from the build directory into the
/ directory within the container. One important item to remember is that when we start the
stunnel process, we will need to specify the location of this configuration file. This configuration will go into the
Creating a Procfile
As described earlier in this article, Procfiles are used by tools like Honcho to start up applications. As with the
stunnel.conf file, we will need to first create a
Procfile within our local build directory.
stunnel: /usr/bin/stunnel4 /stunnel.conf redis: /usr/local/bin/redis-server /etc/redis/redis.conf
The above is the contents of our
Procfile; the format is quite simple. The first element in a line is the name of the process, and the second (after the
:) is the command to run. One of the useful things about
Procfile tools like Honcho is that you can control and launch processes by name. For example, if we wished to start just the Redis process, we could do so by executing
honcho start redis.
Procfile define, we once again need to add it to the container using the
ADD instruction within the
FROM redis RUN apt-get update --fix-missing && \ apt-get install -y stunnel python-pip && \ rm -rf /var/lib/apt/lists/* RUN pip install honcho ADD stunnel.conf /stunnel.conf ADD Procfile /Procfile
Procfile defined and added to the
Dockerfile build steps, we can now move on to the
Dockerfile instructions for starting our applications.
To execute commands within our
Dockerfile, we previously used the
RUN instruction. These instructions are only executed during the build process for Docker containers. To specify how to start our application, we will use the
FROM redis RUN apt-get update --fix-missing && \ apt-get install -y stunnel python-pip && \ rm -rf /var/lib/apt/lists/* RUN pip install honcho ADD stunnel.conf /stunnel.conf ADD Procfile /Procfile WORKDIR / CMD honcho start
The command to start our application is simply
honcho start. This tells Honcho to read through the
Procfile and start all processes defined within it. You may notice another
Dockerfile instruction shown above as well:
WORKDIR instruction is used to define the working directory that the command within
CMD is executed. Since the
Procfile is in
/, the working directory should also be
/. By default, Honcho checks the current working directory.
Building the Container
At this point, our
Dockerfile is defined, and configuration files have been created. Our next step is to build the container using the
docker build command.
$ docker build -t redis-tls . Successfully built a7bbe84cb52b
Once the image build is complete, we can go ahead and start the container.
Starting the container
To start the container, we will use
docker run just like any other Docker container. However, with this specific container, there are a few other options we need to pass.
$ docker run -d -p 6379:6379 -v /path/to/certs:/certs --name redis-tls redis-tls
When executing the
docker run command, we passed the
-d flag, which starts the container in “detached” mode, sending the container into the background.
We also passed the
-p flag with the options of
6379:6379. This option sets up port forwarding from the Docker host port
6379 to port
6379 within the container. This will be needed to connect to the stunnel and Redis services.
We also passed the
-v flag which sets up Docker volumes. When passing the argument of
/path/to/certs:/certs, we are mapping the host directory of
/certs within the container. This allows for the
key.pem to be created on the host and referenced from within the container.
Validating That Everything Is Running
To validate whether or not the processes have started appropriately, we can use the
docker logs command.
$ docker logs redis-tls 23:44:59 redis.1 | 13:M 30 Jun 23:44:59.544 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 23:44:59 redis.1 | 13:M 30 Jun 23:44:59.544 * The server is now ready to accept connections on port 6380 23:44:59 system | stunnel.1 started (pid=12) 23:44:59 stunnel.1 | 2016.06.30 23:44:59 LOG5: Configuration successful 23:44:59 stunnel.1 | 2016.06.30 23:44:59 LOG7: Listening file descriptor created (FD=6) 23:44:59 stunnel.1 | 2016.06.30 23:44:59 LOG7: Service [redis] (FD=6) bound to 0.0.0.0:6379
From the above, we can see that both the
stunnel processes are running, but what happens if one of the processes were to stop?
$ docker logs redis-tls 20:39:15 system | redis.1 stopped (rc=0) 20:39:15 system | sending SIGTERM to stunnel.1 (pid 12)
If one process stops, Honcho will send a signal to all other processes to stop them as well. This means that we can have a multi-process container stop in the same fashion as a single-process container during failures.
In today’s article, we created a custom multi-process Docker container with Honcho, we created a Redis container with TLS support, and we learned a little about sharing directories from a Docker host to a Docker container.
For those interested in using the
redis-tls container, you can do so by simply executing a
docker run command using the
$ docker run -d -p 6379:6379 -v /path/to/certs:/certs --name redis-tls madflojo/redis-tls
The contents of the build directory and
Dockerfile are also available on GitHub for those interested in contributing or modifying the
Have another tool for running multi-process containers? Throw a comment below and share your experiences.
|Reference:||Using Honcho to Create a Multi-Process Docker Container from our WCG partner Ben Cane at the Codeship Blog blog.|