Devgun: Creating Development Environments with Kubernetes

When we first approached the problem of creating local development environments, we reached for common tools like vagrant. But as with most vagrant-built environments, build times are long. This means vagrant images have long lives and tend to drift overtime.

Our services and infrastructure change constantly, so the vagrant build was almost always broken. We wanted to avoid the traditional “new developer right of passage,” which easily entails days of work to get vagrant to successfully build a new development VM.

gif of cat waiting and filing nails

Ideally, new developers can be productive within hours of hire, not days or weeks.

All of this was compounded by the fact that we have a lot of micro services that need to run in the development environment and integrate seamlessly.

The goals of the project were simple:

  • Setup and destroy a dev environment quickly and in a reproducible manner.
  • Single step development environment setup and teardown.
  • Start and stop services, including their dependencies, at will.

Evaluating Orchestration Platforms

Since our micro services were already running in containers, we knew we needed to choose an orchestration system.

Several devs on our team were interested in Mesos Marathon, so we tested that out first. Unfortunately, the memory and hardware requirements required to run marathon in a small local development environment were far too restrictive for our purposes. At the time we tested, minimesos was also unable to correctly network containers in Marathon to the local network – a major roadblock for our project. These issues made it clear that Mesos would be impractical for our needs.

Next, we evaluated Kubernetes and Minikube as we were attracted to the vibrant community around these projects. We found the memory footprint to be minimal, and networking worked perfectly. Most of the features that interested us worked out of the box, and getting our third-party applications running inside Minikube proved easier than expected.

So with Minikube as our chosen platform, we were able to move forward with the project quickly and spend most of our time developing what would become Devgun.

Building A CLI

To accomplish our project goals, we decided to build an all-in-one command-line interface written in Golang. We wanted a CLI that could be distributed as a single binary to our developers, so we created one called Devgun.

Devgun Mailgun Development Environment

Devgun makes no assumptions about the development environment. It automatically downloads all the tools it requires, including Minikube, kubectl, and Mailgun-specific tools, into the /.devgun/bin directory. All you need to do is add your path after initial setup has been completed. Installing everything under a single directory makes uninstall as simple as rm -rf ~/devgun.

We compiled all the kubernetes manifest files into the static binary using the go-bindata project, thus enabling a single file download and run to install a development environment.

Running Devgun looks like this:

$ devgun start
Pulling kubectl                              [done]  
Pulling minikube                             [done]  
Writing Minikube configuration               [done after 0.702698 seconds]  
Booting Minikube environment                 [done after 77.297039 seconds]  
Waiting for Kubernetes..                     [done]  
INFO[0148] Setting up minikube routes -- enter your sudo password if prompted  
Modifying routes                             [done after 0.012867 seconds]  
Adding kube-dns resolver entry               [done after 0.002070 seconds]  
Applying kube-dns-external configuration     [done after 0.212881 seconds]  
Enabling Heapster                            [done after 37.506335 seconds]  
========================================================
Minikube is now configured and running....

# Set up path for our bin directory with dev tools
export PATH="$PATH:/Users/thrawn/.devgun/bin"

Next you should choose a mailgun service to start using `devgun service start`

* Use `devgun service list` to see a list of services to start
* Use `devgun service start` to start a service and its dependencies
* Use `kubectl get pods` to list running pods
* Use `kubectl describe pod <pod-name>` to inspect a pod and see the pod log
* Use `kubectl log <pod-name>` to see the container logs
========================================================

In addition to creating a Kubernetes environment, Devgun also adds some network routes to the local box. For instance, on OS X, the following routes are added:

route -n add 172.17.0.0/16 <minikube ip>  
route -n add 10.0.0.0/24 <minikube ip>  

This allows our developers to run and test applications on their local boxes with full connectivity to containers in the k8 cluster as if they were running in the cluster themselves. We also enable DNS lookup by utilizing the resolver functionality built into OS X. We create /etc/resolver/local with the content:

search cluster.local svc.cluster.local default.svc.cluster.local  
nameserver 172.17.0.3  

Thus enabling DNS resolution of kubernetes services.

$ devgun service start etcd
Starting service: etcd                       [done after 7.084730 seconds]

$ ping etcd
PING etcd.default.svc.cluster.local (172.17.0.4): 56 data bytes  
64 bytes from 172.17.0.4: icmp_seq=0 ttl=63 time=0.220 ms  
^C
--- etcd.default.svc.cluster.local ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss  
round-trip min/avg/max/stddev = 0.220/0.220/0.220/0.000 ms  

Managing Custom and Third-Party Services with a Spec

Because Devgun has to manage a mix of custom and third-party services, we developed a custom yaml spec file we define for each service. The spec file looks something like this:

service:  
  name: scout
  resources:
    <<: *k8s_deploy_resources
  depends_on:
  - name: vulcand
  - name: cassandra
  - name: kafka-pixy
  - name: kafka
  - name: etcd
  - name: metrics
  hooks:
    pre:
      start:
        - exec: >
            cqlsh -e "CREATE KEYSPACE IF NOT EXISTS scout WITH REPLICATION={'class':'SimpleStrategy','replication_factor':1}" $POD_IP
          where: service:cassandra:cassandra

It defines a custom service called ‘scout’ which reads events from Kafka and records analytics about the events in Cassandra. The spec tells devgun which custom and third-party services it depends on and which hooks to run on different containers before starting the scout service.

The resources key references the kubernetes manifest files like config map, deployments, and service descriptions defined elsewhere in the spec file that devgun uses to start the containers in K8.

Where Do We Go From Here… Future Work

When we designed Devgun, we started from the bottom up without consideration of the bigger development and deployment stories. Now that we’ve had time to use Devgun, we find ourselves wondering if we got the abstractions correct.

We started asking questions: What is the experience we want our developers to have when they sit down to create and deploy a new service? Where do the responsibilities of the developer end and operations begin? How much of operations can we automate and how much requires human interaction? How do we facilitate the passing of information between developer and operations?

We’ve started to define stories and personas that will shape how Devgun evolves going forward. In my next post, I’ll share some of our work and thoughts on how the developer experience could evolve as we continue to work with Kubernetes and Devgun. Be sure to subscribe to our blog so you don’t miss out.

comments powered by Disqus

Mailgun Stay in Touch

Get new posts delivered straight to your inbox.