Maqpie blog

Build and deploy applications using Drone CI, Docker and Ansible

Andrew Orsich -

Can you imagine that you will never have to define dependencies and customize configurations manually on your continuous integration server? Do you believe that every step of your build can be really isolated and work exclusively in Docker containers? And finally, would you like to try the tool that is one of the top 20 of all open projects written in Golang, and has 9k+ stars on Github?

In this whitepaper we would like to talk about the excellent Drone CI which has already helped us to simplify and improve our continuous integration. We'll share the details of Drone CI installation and show the use through the example of a small project. If you do not like reading a lot and you want to try it right away, there are links to Github repositories that will help with a quick start at the end of the article.

tl;dr;

Before moving on to the main topic, I would like to thank our readers for the large number of kind reviews on the article about [docker-compose](GHOST_URL/2017/02/22/fully-automated-development- Environment-with-docker-compose /), Docker - for posting the article in the blog and in Twitter and, of course, Brad Rydzewski, the author of Drone CI, without whom this article would not exist. This is a great motivation for us to write further!

Some background

If you are familiar with the term "Continuous Integration" (CI) and would like to know more - start with a wonderful article by Martin Fowler. Continuous integration has long been a part of our life, which helps to identify bugs and provides complete automation of the application deployment process, saving a huge amount of time for the whole team.

I've been using CI servers for 7 years now and, like many of us, started with Jenkins, later I used TeamCity and then fell in love with Travis CI. Each of these products has done a lot to develop the practice of continuous integration. One of the reasons that makes me change tools from time to time is the ability of full automation of any process. Jenkins and TeamCity have a very advanced user interface that allows you to configure continuous integration for any project, but it is rather difficult to automate. Travis is a very good tool and it still remains the number 1 option for all my open-source initiatives, cause "Testing your open source project is 10000% free". Travis was the first tool that made it possible to configure most of the continuous integration steps with a single .travis.yml file.

Pipeline as a code

"Pipeline as a code" - is a relatively new approach that allows you to configure the deployment pipeline with code instead of manually configuring the running CI service. This concept is very popular today and I know at least five players in this segment: LambdaCD, Concourse, Drone, GoCD and Travis CI. This approach not only makes it easier to automate continuous integration and continuous deployment, but also allows you to test the infrastructure for deployment. This concept made the world look different, but the most important thing is that it really allows you to use CI more efficiently and elegantly.

How we continuously integrate

Today, our team has 6 members, and our approach to continuous integration and deployment is quite simple. We actively use Github, Pull Requests, Code Review. If you want to get better understanding of "Continuous Delivery" - pay attention to "Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation" by Jez Humble и David Farley.

The main steps of our continuous integration and continuous deployment:

  1. Each member of the team works in a separate branch and creates a Pull Request as soon as the task is done.
  2. Each commit in the Pull Request is checked by our CI server. The unit, integration, UI tests and linting are started. The launch status is displayed in Github:
  3. Each Pull Request is necessarily checked by two team members. One of them runs the code locally and checks all the changes in the user interface. If the launch of the tests was successful and the review is completed, the code will be merged into the master and the process of automated deployment to the pre-production environment will begin.
  4. Deployment consists of two steps. First of all we publish all our Docker containers on the DockerHub, and then we run a small Ansible script that deploys the application on our servers.
  5. After a short manual testing, all changes are manually merged into the production git branch, and automated deployment to the production environment starts.

The process has several drawbacks and will require some changes when our team grows bigger. But for today it completely suits our needs and, as it seems to us, is ideal for small teams of up to 10 people. The main shortcomings include:

  1. Despite a large number of tests, some changes that fall into the master contain defects. This blocks the deployment of other Pull Requests until the defects are fixed.
  2. When our master branch is not in the "production ready" state (which is disaster for the huge teams), we have to deviate from the process and fix the problems that our customers have directly in the production branch, which adds the need of reverse transfer of these changes to the master branch.
  3. We have not fully worked out the rollback process in case something went wrong. It involves rolling back of the "production" branch to the state before deployment. In case any changes in the database are required, they are perfomed manually at this stage.

We will deal with these shortcomings as our team grows. For today we are already able to deploy our project almost painlessly several times a day. We do not yet deploy the application like Github, but the first steps are already done :)

Drone CI

After comparing various tools our choice fell on Drone CI and for the last three months we have switched to it completely. Drone CI has 9003 stars on Github (March 15, 2017) and is in top 20 of applications written in Go on Github Channel in Gitter never sleeps - there you can get answers to any questions.

Installation

Drone CI is a single container with the size of 8 megabytes. This container has two services:

  1. Drone UI - a simple user interface and a server that coordinates the work of agents and displays build statuses.
  2. Drone Agent - another service where the build process for your projects is launched.

All data about past and current builds is stored in the database. Sqlite is used by default, but it is possible to use other relational databases, such as PostgreSQL and MySQL. Specifically for this whitepaper, we have prepared a Github repository that will help you to install Drone CI on your local environment or on your production environment just in a couple of minutes.

Beginning of work

To begin with, I would like to give you a short introduction into the way of how Drone CI works. After you have installed and logged into Drone CI with your Github account, Drone automatically displays all of your repositories. The first step is to enable repositories you want to configure continuous integration for:

At this stage the work with user interface is almost finished. Further you will need it only for checking the state of your builds. All the configuration of the deployment steps is carried out in one file: drone.yml. This file is usually located at the root of your repository and completely describes everything that happens on your CI server.

In fact, .drone.yml is a random set of steps, each runs in a separate, isolated Docker container. With each commit, before running the steps from .drone.yml, Drone automatically clones our repository and mounts it as a Docker volume.

To make it clear, let's look at a simple configuration that runs tests, builds a Docker image, publishes it to the Dockerhub, and sends a simple notification to the Slack channel when the build is finished.

pipeline:
  run-tests:
    image: node:6.3.0
    commands:
      - cd ./api && npm i --quiet
      - npm test

  publish-api-docker:
    image: plugins/docker:1.12
    username: ${DOCKER_USERNAME}
    password: ${DOCKER_PASSWORD}
    email: ${DOCKER_EMAIL}
    repo: anorsich/ds-api
    tags:
      - latest
    dockerfile: ./api/Dockerfile
    context: ./api/

  slack-notification:
    image: plugins/slack
    webhook: https://hooks.slack.com/services/...
    username: drone-ci
    channel: andrew
    icon_emoji: ":rocket:"

And that's it! Now each time you commit to the repository, you will have tests run, Docker images built and you will get a notification in Slack.

Third-party dependencies and isolation of build steps

Each step in Drone runs in a separate Docker container, which allows you not to worry about installing and updating dependencies on server agents. An important difference is that the dependencies for each step can be completely different. In one step, you can run tests for Node.JS, and in the next run the build of the application written in Go. Migration to new versions of platforms/frameworks is carried out just by changing the version of the Docker image. For example, we can easily add a new step that will run tests and build the project on the latest version of Node.JS:

  run-tests-on-latest-node:
    image: node:7.7
    commands:
      - cd ./api && npm i --quiet
      - npm test

It means that you need to configure your CI server just in one case - if there is a new version of the CI server itself. Everything else is configured directly in the repository in .drone.yml without a single click. After installation you can forget about it! :)

It is important to mention that each step is absolutely isolated from the others, since it is performed in a separate container. No more version conflicts!

Another possibility worth mentioning (though we are not using it yet) is the support of matrix builds, which allows immediate testing of your code on Different versions of platforms, frameworks, databases and so on.

Conditional builds

Sometimes there is a need to run a step in your build only under certain conditions. By default, all steps are run sequentially and if one of the steps fails, the subsequent steps are not started. The main coditional checks that we use are the following:

  1. The name of the branch (master, production)
  2. Status of build (success, failure)
  3. Github event (pull_request, push, tag, deployment)
    For example, we want to send a notification to Slack when the build was successful and when it broke. To do this, we use the when section and add status:
  slack-notification:
    image: plugins/slack
    ...
    when:
      status: [ success, failure ]
      event: [ push, tag, deployment, pull_request ]

More details about the conditional builds can be found here.

Another interesting feature worth mentioning is the ability not to run the build by adding to the commit messages: [ci skip].

Plugins

Plugins is the Drone CI approach for integration with third-party services, such as Amazon S3, Dockerhub, Slack. A complete list of all the plugins can be found here. Each plugin is a separate Docker container that performs a predetermined task. In our example above, we have used two plugins:

  1. Docker plugin (plugins/docker) - to build and publish the Docker image to the Dockerhub.
  2. Slack plugin (plugins/slack) - to send a notification to Slack.

Plugins solve most common problems, but not all. To run any task specific for your project, you just need to wrap this task in the Docker image - and you can start using it in Drone CI. Any programming language that you can run in Docker is available to you. Continuous integration for your special steps can be easily organized using Drone.

Console Utility

Drone Console Utility
allows you to communicate with a remote server and perform various administrative tasks. After installation, you need to connect to your remote server. To do this, you need to export two variables in the terminal:

  1. export DRONE_SERVER=http://MY_DRONE_URL - URL of your Drone server.
  2. export DRONE_TOKEN= - a personal token, which is created after you are authorized in Drone UI. You can find it on the next page: https://MY_DRONE_URL/account. Just click SHOW TOKEN and copy it.

Now the console utility is configured.

Secrets

Drone has very convenient tools for safe work with private information, such as passwords and ssh keys (in a word - secrets). In the example above, we have used the Docker plugin to publish the Docker image to the Dockerhub, which needs your password and username.

In Github Repository with an example of continuous integration for Node.JS applications using Drone, we also use Ansible. As a result, we run Docker containers on a remote server. To communicate with a remote server, you need a separate ssh key.

As you know, storing sensitive information in the Github repository is a bad practice, as it can be used by hackers to compromise your application. In Drone CI, this problem is solved.

First, let's figure out how to add our password and username for Dockerhub (used by the plugins/docker plugin):

drone global secret add DOCKER_USERNAME andrew
drone global secret add DOCKER_PASSWORD password
drone global secret add DOCKER_EMAIL email

You can add secrets only to a particular repository or to all repositories within your Drone CI using global. If you want to add a secret only for a specific repository, you need to specify its name and not use the global:

drone secret add maqpie/drone-starter DOCKER_USERNAME andrew

Once secrets added, you can use them using simple syntax ${DOCKER_USERNAME} right in the .drone.yml. Here is an example:

publish-api-docker:
  image: plugins/docker:1.12
  username: ${DOCKER_USERNAME}
  password: ${DOCKER_PASSWORD}
  email: ${DOCKER_EMAIL}

As we have mentioned before, we also need an ssh key to work with the remote server. The key can be added the following way:

drone global secret add SSH_KEY @/Users/andrew/.ssh/id_rsa-drone-demo

Secrets protection

To protect your secrets, Drone uses a simple signature mechanism for your .drone.yml. After each change in .drone.yml, you need to re-sign it. Drone checks the signature every time before running builds. If the signature does not match, the secrets will not be published and will not be available to the build steps.
For signing Drone uses your personal authorization key, which you added while configuring the console utility:

drone sign maqpie/drone-starter

It is necessary to specify a name of a repository in where .drone.yml is located. As a result of this command, the file .drone.yml.sig should appear. This file needs to be pushed to the repository to be available for Drone.

Important: Before you sign your .drone.yml for your repository, make sure that this repository is enabled in the Drone UI - otherwise the signature file will not be created.
Despite a good mechanism for protecting secrets in Drone, hackers still have several options to access them. For example, if you use the bash script in one of the steps, the hacker can use curl or another tool to access your password or ssh key. If you don't use custom scripts - you should be good, but still always apply the general safety principles.

Services

Services in Drone UI allow you to run any container during the execution of your build process. All services are in the same subnet with the process build containers. This capability can be very useful for various types of integration testing. Services are usually declared at the end of .drone.yml. An example of starting a MySQL database looks like this:

services:
  database:
    image: mysql
    environment:
      - MYSQL_DATABASE=test
      - MYSQL_ALLOW_EMPTY_PASSWORD=yes

At the current stage, we do not use services, instead we use an approach which is simpler for us using docker-compose.

Running tests with docker-compose

In our previous article we shared our love for docker-compose. I would like to share how we run our tests in Drone CI. The step in .drone.yml looks like this:

  run-tests-in-compose:
    image: michalpodeszwa/docker-compose:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    commands:
      - ./bin/drone-run-tests.sh api-tests
      - ./bin/drone-run-tests.sh web-tests
    when:
      event: [pull_request]

/bin/drone-run-tests.sh runs tests using a small script wrapper over the docker-compose. Before going deeper into our approach, I would like to talk about the compromises that we have had to make. First of all, we broke containers isolation in the following line:

volumes:
   - /var/run/docker.sock:/var/run/docker.sock

To put it short, this line actually enables this container to run any command that the docker service itself can run. Essentially, this allows running any command on the host machine. More details about all the consequences can be found in this article. This method is definitely not suitable for public repositories.

Why did we take such a risk? There are several reasons:

  1. We work on a private product.
  2. Our Drone CI is running on a separate server, and even if someone wants to hack it, taking advantage of the lack of isolation - we will not lose anything. To deploy the new Drone CI, we will need not more than 5 minutes including the DNS update.
  3. The most important reason is that we wanted to effectively use the Docker cache. We make about 20-100 commits into repository every day, and each of them runs our tests. If we did not use the Docker cache, at each commit the containers would be rebuilt again, which takes a lot of time.

Now that you know the downsides in our approach, let's look at the approach itself. In fact, to run tests, we simply use docker-compose up --file docker-compose.drone-tests.yml. In the same way, tests are run in development environments. The only problem we encountered when running tests through docker-compose is that the exit code for docker-compose was always 0. Drone, like any other CI, in this case thinks the build was successful and moves on to the next Step.

To change the situation for the better, we've written a small script that analyzes the exit codes of containers after the tests are started and, if at least one of the containers exited with a nonzero code, use this code to exit and let Drone know about the failure.

The contents of the script /bin/drone-run-tests.sh, you've seen above looks like this:

#!/bin/sh
# remove old containers
docker-compose --file docker-compose.drone-tests.yml rm -f 
# run tests
docker-compose --file docker-compose.drone-tests.yml up --build

echo "Inspecting exited containers:"
docker-compose --file docker-compose.drone-tests.yml ps
docker-compose --file docker-compose.drone-tests.yml ps -q | xargs docker inspect -f '{{ .State.ExitCode }}' | while read code; do
    if [ "$code" != "0" ]; then
       exit $code
    fi
done

Links and tips

As with any tool, in Drone CI there are some things that are in development and do not always work as we would like them to work. Fortunately, there were a few:

  1. Drone UI freezes if your build process displays too much information. In all our Docker files for Node.JS applications, we had to add --quiet when to the npm install to reduce amount of the useless output. It is weird why in npm this option is not used by default.
  2. There are two versions of Drone 0.4 and 0.5. The latter is now in active development, but fairly stable (we've had no problems for 2 months). There are very many differences with 0.4. When searching for various answers, we often found old answers that are no longer relevant.
  3. The documentation for version 0.5 is located at the following address: http://readme.drone.io/0.5/. Documentation, perhaps, is the weakest part of Drone at the moment. The first version of Webpack documentation (read the comments below) hasn't the best either, but it hasn't stopped it from becoming the standard for most Web projects.

If we've managed to convince you to try Drone, below you can find a list of useful links:

  1. Drone github
  2. Drone plugins github
  3. Drone 0.5 documentation
  4. Drone gitter
  5. Drone 0.4 documentation - some parts of this documentation have not yet been fully transferred to the Drone 0.5.

Conclusion

Drone CI - is a modern solution for the continuous integration. Using Drone, you will never have to configure your servers again, you will get a completely isolated environment for running your builds and will be able to scale your CI server as much as you need. We have been working with it for the last two months and have completely fallen in love with it.

To help you start using Drone CI, we have prepared two Github repositories:

  1. Deploy Drone - simplifies the process of installing Drone CI for development and production environments.
  2. Drone Starter - continuous integration with Drone CI, Ansible, Docker using the example of a simple three-service Node.js application.

Share your experience and ask questions!

We have not decided yet, but there is a good chance, that in our next article we will share our approaches for organizing the application monitoring system. In order not to miss - subscribe to our twitter. If you have a minute - share your opinion about the tool that we are developing. Thank you!

P.S. If you liked the article for at least 51%, or did not like it for only 49% - give us a star on Github so more people will be aware of Drone :) We will really appreciate it.

We hope that the article has been useful and it'll help to make your product better!