2014

Continuous Deployment with GitHub, CircleCI and Heroku

Back in the dark days of 2013 we were using a self-hosted Jenkins installation for continuous integration and deploying onto Docker containers hosted on our own server. (That’s right, we were using Docker before it was cool.) Administering our own server and dealing with Jenkins configuration issues was a major hassle, however, so we decided about a year ago to move everything into the cloud. We settled on an approach that uses three popular hosted services: Github, CircleCI and Heroku. We have a very robust continuous deployment process that we hope will be of interest to others.

➔ ➔

Project directory structure

Let’s dive straight into our code. First, a few words about our default project structure. A project can consist of multiple interdependent subapplications, such as a server, web app front-end, browser extension or mobile app. A web app with back-end and front-end components is the most common scenario. Usually we develop both of them in parallel, in the same language (JavaScript) and with the same release cycle and development team. We like to keep them in one GitHub repository, which simplifies project versioning and deployment.

The repository consists of a code directory, where all subapplications are stored separately in subdirectories. No subapplication should assume anything about the directory structure of other subapplications; it should be possible to extract it into a new repository and deploy it separately. There are also a few configuration files. Here is a sample structure for a typical project of this type:

  • code/
    • client/ – AngularJS front-end code
      • .gitignore
      • package.json, bower.json – client dependencies
      • Gruntfile.js or gulpfile.js – client makefile
      • …
    • server/ – NodeJS back-end code
      • .gitignore
      • package.json – server dependencies
      • Gruntfile.js or gulpfile.js - server tasks
      • …
  • circle.yml – CircleCI config
  • Procfile – Heroku config
  • package.json – global dependencies (such as grunt-cli, gulp, bower)

Workflow

To manage our development workflow we use a modified version of Gitflow as described in A successful Git branching model. Specifically, we use the following branch naming scheme:

  • feature/[story_id]/slug – feature branch. We use Pivotal Tracker for project management so the branch name contains the Pivotal Tracker story ID. These branches should not be commited to GitHub.
  • develop – latest trunk code
  • release/x.y.z – release with the given version number that been staged for quality assurance
  • client – code staged for acceptance by the customer
  • master – production code
  • try – short-lived branch where developers can push their code to try it in the deployment environment

To make it easier to work with all these branches, we forked the Gitflow Git plugin and added features like automatically linking a feature branch to the corresponding story in Pivotal Tracker when starting a feature and posting a review request to Review Board (which we use for code reviews) when finishing a story. We’ve open-sourced our modified version of this plugin in case anyone else is interested in using it.

CircleCI

CircleCI is a hosted continuous integration service that takes care of automated testing, building and deployment of our applications. It can be connected to any project that is hosted on GitHub. To start using CircleCI, log in via GitHub OAuth and “follow” the repository you want to use. CircleCI inserts an SSH key into your repository settings that is used for checking out the latest source code. It also inserts a service hook that will ping CircleCI every time you push new commits to GitHub. Whenever CircleCI detects new commits, it pulls the latest source code, starts a new VM for the project and runs the tasks defined in the circle.yml configuration file in the repo root. If any of those tasks fails, the build is marked red and cancelled, otherwise is marked green and deployed to the runtime environment.

circle.yml

This is a YAML file. It specifies which commands should be run on the CI server to test, build and deploy the application. Here’s an example:

machine:
  environment:
    PATH: ${PATH}:${HOME}/${CIRCLE_PROJECT_REPONAME}/node_modules/.bin
    NODE_ENV: test
    TZ: Europe/Prague
  node:
    version: v0.10.33
    
general:
  artifacts:
    - code/client/build/karma-coverage
    
dependencies:
  cache_directories:
    - node_modules
    - code/server/node_modules
    - code/client/node_modules
    - code/client/vendor
  override:
    - npm prune && npm install
    - npm prune && npm install:
        pwd: code/server
    - npm prune && npm install:
        pwd: code/client
    - bower prune && bower install:
        pwd: code/client

test:
  override:
    - grunt test:
        pwd: code/server
    - grunt test && grunt compile:
        pwd: code/client
  post:
    - sed -i -e 's/\/node_modules//' code/server/.gitignore
    - sed -i -e 's/\/bin//' code/client/.gitignore
    - git config user.name "circleci"
    - git config user.email "xxxxx@salsitasoft.com"
    - git add -A
    - git commit -m "build"
        
deployment:
  try:
    branch: /try.*/
    commands:
      - git push -f git@heroku.com:${CIRCLE_PROJECT_REPONAME}-try.git ${CIRCLE_BRANCH}:master
  dev:
    branch: develop
    commands:
      - git push -f git@heroku.com:${CIRCLE_PROJECT_REPONAME}-dev.git ${CIRCLE_BRANCH}:master
  qa:
    branch: /release.*/
    commands:
      - git push -f git@heroku.com:${CIRCLE_PROJECT_REPONAME}-qa.git ${CIRCLE_BRANCH}:master
  stage:
    branch: client
    commands:
      - git push -f git@heroku.com:${CIRCLE_PROJECT_REPONAME}-stage.git ${CIRCLE_BRANCH}:master
  prod:
    branch: master
    commands:
      - git push -f git@heroku.com:${CIRCLE_PROJECT_REPONAME}-prod.git ${CIRCLE_BRANCH}:master

As you can see, this configuration gives us control over:

  • environment variables
  • NodeJS version
  • build artifacts (files/directories to be saved for each successful build)
  • commands to install dependencies
  • directories to be cached between builds (it speeds up builds dramatically if dependencies don’t have to be reinstalled every time)
  • commands to run tests and builds (if you don’t have any commands to run in this section, you must provide a dummy command such as ls so that CircleCI doesn’t complain about missing tests)
  • commands to deploy the app to the runtime environment (keyed by the specific Git branch where the commit landed)

Note especially the test.post section. Using the default setup, an application is built twice, first on CircleCI when running tests, and then on Heroku to deploy it. We wanted to get rid of the second redundant build. Normally the build-related directories are not pushed to the repo since they are in .gitignore. So we use sed to remove them and then commit them to the repo. (Note that these files are not pushed to the GitHub repo, only to Heroku). Now the repo contains everything necessary to run the app in Heroku environment without building it again.

And there’s a lot more. See the Circle CI documentation for a complete list of configuration keys and values.

Heroku

Heroku is a PaaS (platform as a service) where you can run server applications in every imaginable language. Applications run in containers called "dynos". These are essentially virtual machines that contains a clean Linux system (currently Ubuntu 14.04 x64) with some useful tools such as interpreters for common languages (see installed packages). These dynos can be started and killed at will, in which case the contents of system memory and the file system are lost.

If you want to store any global application state, you must use an addon (e.g. database or logging) or static files storage (AWS S3). Once of the biggest strengths of Heroku is the wealth of addons available for popular database systems and other purposes.

The first Heroku dyno is free and is automatically stopped after one hour of inactivity (no incoming requests). As a result, the first request to a free dyno often takes about 10-20 seconds while the dyno "wakes up" and your application is restarted. In development environments this is generally fine. In production you should have either at least one paid dyno, which will prevent even free dynos from being shut down for inactivity. Or you can setup a service that will ping your application periodically.

Every dyno is equivalent to one virtual machine with one running application process. If you choose to pay for multiple dynos, there is a load balancer that proxies all requests to your processes. The best way to manage a running Heroku application is via their CLI tool.

Procfile

This file must be present in the root directory of the repository that is pushed to Heroku. Each line contains the command to be run for a given "process type" (usually web). Each process runs in its own dyno and can be scaled up or down independently. This is a sample file that we use for our Node.js projects:

web: sh -c 'cd code/server && npm start'

Consult the Heroku documentation for more information.

Runtime environments

We have five different runtime environments corresponding to the five branch types in our Git repositories:

  • [projectname]-dev for develop
  • [projectname]-qa for release
  • [projectname]-stage for client
  • [projectname] for master
  • [projectname]-try for try

We create Heroku applications with these names for every project. An application is then accessible under the [appname].herokuapp.com domain.

Addons

There is a wealth of Heroku addons available for interfacing with third-party services such as database servers. They can be added under the Resources tab in Heroku application configuration. Almost every addon has free plan, which is usable for development or low-load production. Some of our favorite addons are:

  • MongoLab for MongoDB database servers (use the process.env.MONGOLAB_URI connection string)
  • SendGrid for sending emails (use SMTP at smtp.sendgrid.com:567, auth process.env.SENDGRID_USERNAME, process.env.SENDGRID_PASSWORD)
  • Papertrail for logging (logs all Heroku events such as starting or stopping dynos)
    • logs all requests coming to your application (path, status code, processing time)
    • logs any output your application writes to stdout/stderr

Random server-side tricks

Environment variables can be set using both the Heroku web dashboard and the command-line toolbelt. Confidential data such as authorization tokens should be stored in there instead of directly in source code. This follows the The Twelve-Factor App manifesto published by Heroku.

If the SSL Endpoint addon is used to allow HTTPS connections, in ExpressJS applications req.secure will always return false, because encryption is terminated at the SSL Endpoint. You can tell Express to trust proxy headers (X-Forwarded-Proto) by enabling the trust proxy option.

Jan Žák

Jan Žák

Polyglot Developer in Salsita, focused on web services, scalability and security