Testing Mesos, Marathon, Chronos on Docker
Today I decided to test out Apache Mesos, a tool that provides a layer of abstration over a datacenter. Essentially, it allows you to manage a cluster of servers and distribute tasks.
I also worked with a couple frameworks that run on top of Mesos. Marathon is a framework for long-running applications, like services. Chronos is a framework for running scheduled tasks in a distributed manner.
I’m running this all locally using Docker, which simplifies and isolates installation, but the networking is slightly different so following tutorials to the letter will likely lead to connection problems.
ZooKeeper
Mesos, Marathon and Chronos all use Apache ZooKeeper for distributed coordination of services.
Bash commands
Below are the bash commands I used to create each container. I ran each in a separate console tab, but you could easily run them as daemon processes. There is one container for ZooKeeper, one for the Mesos master, one for the Mesos slave, one for Marathon, and one for Chronos.
The IP address 172.17.42.1 is what docker is using by default on my local machine. To find the docker IP address on your machine, execute the following command:
ip route | grep docker | awk '{ print $NF }'
# Start Zookeeper on ports 2181, 2888, and 3888
docker run -p 2181:2181 -p 2888:2888 -p 3888:3888 jplock/zookeeper
# Start Mesos master on port 5050
docker run -e MESOS_LOG_DIR=/var/log \
-e MESOS_WORK_DIR=/var/log \
-e MESOS_ZK=zk://172.17.42.1:2181/mesos \
-e MESOS_HOSTNAME=172.17.42.1 \
-e MESOS_QUORUM=1 \
-p 5050:5050 \
redjack/mesos-master
#Start Mesos slave on port 5051 (port 31800 used later for a service) ** SEE NOTES BELOW **
docker run \
-e MESOS_LOG_DIR=/var/log \
-e MESOS_MASTER=zk://172.17.42.1:2181/mesos \
-e MESOS_HOSTNAME=172.17.42.1 \
-p 5051:5051 \
-p 31800:31800 \
redjack/mesos-slave
# start marathon on port 8080
docker run -p 8080:8080 mesosphere/marathon --master zk://172.17.42.1:2181/mesos --zk_hosts 172.17.42.1:2181
# start chronos on port 4400
docker run -p 4400:4400 tomaskral/chronos --master zk://172.17.42.1:2181/mesos --zk_hosts zk://172.17.42.1:2181
That should get everything up and running and connected. Mesosphere has a tutorial for running an app using marathon and a tutorial for chronos as well.
A couple notes about the setup:
The redjack/mesos-slave image does not have the unzip package installed. Mesos slave uses unzip for, you guessed it, extracting zip files. So, if you are following a tutorial that uses a zip file for deployment, it will fail. I’ve created a pull request to add the package, but it may not be accepted. In any case, it is easy to create your own image with the required unzip package installed. Here is my updated Dockerfile that you can use to build a new image.
The redjack/mesos-slave image contains version Java JDK version 6. The mesosphere marathon tutorial uses a java app that requires version 7 of the JDK. If you are following the tutorial, you need to build your own image to include the openjdk-7-jre-headless package.
I ran this locally on one machine, so I might have been able to use internal docker links rather than exposing all the ports on the host machine. However, there is some bi-directional communication between the mesos master and slave that might not have been easily accomplished using docker links. See this post on StackOverflow for some possible solutions.
I opened port 31800 on the mesos slave to test running an application using Marathon. There are default port ranges used by the Mesos slave that are allowed. This tripped me up for a little while – if you try to configure an application with a port that is out of range it will randomly choose a different port in the range allowed. Hard coding the port doesn’t seem like a great solution. A better solution would be to use nginx to expose a known port and ZooKeeper to coordinate the internal port number between the mesos slave and nginx.
The Mesos images don’t respond to ctrl+C. Instead, issue a docker stop to get the docker container to quit gracefully.
Final thoughts
I am quite impressed by Mesos, Marathon, Chronos and ZooKeeper. They really seem to work well together and were relatively easy to get running after overcoming some minor networking issues caused by using Docker.
Docker seems to be making sweeping changes across this landscape. I think there is still a bit of flux and overlap with different solutions in this space. Do you run everything as a docker image? How about simple chron functionality? Do you use Mesos to drive Docker? Do you run Mesos in Docker?
Next Up: CoreOS
Next, on my DevOps tour I’d like to take a look at CoreOS and the functionality it provides.