Headless Chrome

Google Chrome version 59 will ship with the headless option. This means you can test your web applications using chrome without needing a running window manager or xvfb.

There is one problem: the latest chromedriver (version 2.29) does not support versions of Chrome higher than 58.

The solution is to build the latest chromedriver that does support the latest chrome/chromium. Unfortunately, Google does not make nightly builds of chromedriver public. You have to download the chromium source and build chromedriver yourself.

UPDATE: It turns out that chromedriver still relies on XOrg for keycode decoding when handling sendKeys commands from selenium. Thus, you still need xvfb for real testing to work.

How all the pieces work together

In my case, I am running the ruby version of cucumber. Cucumber uses the capybara gem to send commands to selenium-webdriver. Selenium-webdriver in turn starts up your local copy of chromedriver, which then starts up chrome and controls the browser through a special debug port.

This means our system has to have a copy of the latest chromium installed, along with our custom chromedriver build.

To get the latest chromium, I used a tool on github that downloads the latest snapshot compiled for linux.

To get the latest chromedriver, I followed the build instructions but ran it all in a docker container in order to maintain a clean building environment. I then copied the built chromedriver and its shared libraries out of the container.

Once I had the built chrome and chromedriver, I was ready to run selenium/capybara tests against it.

Configuring the tests

When configuring capybara, you need to tell selenium-webdriver the path to your custom chromium binary and send the --headless flag, along with other flags you’ll likely need in a CI build node environment.

For running in docker:

Capybara.register_driver :headless_chromium do |app|
  caps = Selenium::WebDriver::Remote::Capabilities.chrome(
    "chromeOptions" => {
      'binary' => "/chromium-latest-linux/466395/chrome-linux/chrome",
      'args' => %w{headless no-sandbox disable-gpu}
  driver = Capybara::Selenium::Driver.new(
    browser: :chrome,
    desired_capabilities: caps

Capybara.default_driver = :headless_chromium

Now, capybara will drive a headless chromium!

I’ve created a repo that lets you test this all out.


Accidental Continuous Delivery and Docker

When I started at Faraday in the Summer of 2014, the company had begun a transition into a Saas product company. There were several individual apps that had grown into web services and queue workers. One of my first goals was to have a Continuous Integration system to run all the tests on each individual project.

Because many of our services relied on system libraries for GIS work and most of the tests were tightly coupled to a single postgres database, a lot of hosted CI solutions lost their appeal. I considered the various services we had that were written in varying versions of ruby and node. I also recognized that as our team grew and added programmers from different specialties (i.e. data scientists using python or R), I realized self-hosting a CI server configured to run all these different languages and platforms would become a mess. Especially when configuring deployments of our CI system.

But, I thought, what if the CI server could just be a web server and a bunch of dumb agents, and the projects themselves could define the platform they ran on? What if each test could run in its own sandbox with any external services linked in as needed? This idea is what initially attracted me to docker.

First steps

In late 2014 I set up a Jenkins server to do this very thing. One by one, I converted each project to define a CI-specific Dockerfile. This mainly involved making each app 12-Factor compatible, configured by environment variables.

I initially waded through the various plugins for Jenkins that dealt with docker, but many of them seemed to be related to strange use cases I wasn’t considering yet. None of them involved simply building images and running them on a docker host. As I looked around at various CI servers, I ran into GoCD. GoCD’s pipeline workflows really attracted me, as I recognized that Jenkins was pretty ill equiped to handle the multiple stages needed for building and running test containers along with backing services. Go also relies less on plugins and more on defining per-project shell scripts to handle job setup. Since I would be doing a lot of manual service startup and docker commands anyway, I decided to switch to Go.

Continuous Delivery is a Thing

One thing I learned while using Go is that it is built quite specifically for supporting a pipelined Continuous Delivery workflow. It was a concept I was never aware of, but quickly grew to appreciate and adopt. My coworker, Eric, was a big help in bringing CD ideas to fruition and pointed me to the excellent book on Continuous Delivery.

Some of the biggest changes and challenges in adopting CD were: Making small changes directly to the mainline. It’s a major change in mindset to stop making huge branches on each service and instead make small, backwards-compatible changes to individual services, deprecating old functionality as older services are upgraded. This meant I no longer needed to worry about how to hammer GoCD around to try and get it to test PRs and branches. Testing master is enough.

Treating CD build errors as “andon” events. That is, when tests break, it’s everyone’s immediate job to get the CD pipelines fixed. It’s interesting that the CD pipeline effectively becomes your “car factory” in the Toyota Production System/Lean Software sense. If there’s a problem in the pipeline, it is a blocker that prevents your team from pushing new software.

The docker/CD combination is pretty instrumental in the idea of “immutable infrastructure.” The CD system contains your entire app’s cloud configuration and you can deploy any version of your system (including the specific combinations of service versions) at any time.

From Test Images to Production

While our work on GoCD started as a way to test services, my team decided to move each service to run in docker in production. Our initial solution was to use docker-machine to spin up instances and docker-compose to define and run our cluster. This has worked exceedingly well, although it is not a zero-downtime or rapidly scalable solution. Luckily, services like Amazon ECS are providing easier ways to deploy dockerized apps. We provide a docker-compose.yml configuration and they mostly do the rest.

Development Environments

When I joined the team, all of the services were managed by some shell scripts sitting in a repo called “cage.” These scripts checked out branches on each service and used foreman to start the whole app cluster locally. As we migrated to docker, this cage tool became an app all its own.

One of the most difficult things about docker, I think, is using it in development. There are several tradeoffs involving how to install gems as you develop and making the service available for local testing/browsing.

I wasn’t attracted to the idea of rebuilding docker images and restarting the cluster each time I made a change. Instead, I came up with a way to run each app where local code changes appear immediately and libraries are shared in order to shorten image build/startup time. Initially I wrote separate Dockerfiles for dev, test, CI, and production environments, but later found that a single Dockerfile for all environments is sufficient.

Here’s an example docker-compose.yml and Dockerfile for a Rails app:

# docker-compose.yml
    - myservice:/app
    - /var/lib/docker/gems:/usr/local/bundle
    RAILS_ENV: development
    - myservice:/app
    - /var/lib/docker/gems:/usr/local/bundle
    RAILS_ENV: test
FROM ruby:2.1.5


ADD Gemfile /app/Gemfile
ADD Gemfile.lock /app/Gemfile.lock

RUN bundle

ADD . /app


CMD rails s -p 5000

Lines 5–8 of the Dockerfile allow us to make changes to the app code without forcing a full bundle install. Only a change to Gemfile/Gemfile.lock will trigger a bundle.

When interactively writing tests, all that is necessary is to run them with: docker-compose run myservicetest rspec. Now, our internal cage app makes this more succinct, but this is essentially what happens under the hood. Running docker-compose up -d myservice will start up a running rails app with local code changes recognized.

While there have been and are some bumps in the road in running a fully dockerized development environment, I think this approach adds several benefits: We can specify every service to run using a CD-tested and approved version, meaning we don’t have to go into each service repo and make sure we’re on the right branch/git commit when running the cluster locally. We also have a tool that modifies the docker-compose.yml to use specific versions of each service (using docker image tags) derived from GoCD’s last good build. This lets a developer work on her/his specific service while resting assured that every other service is at “last known good.” It also lets us select specific release combinations to debug locally.

We don’t need to manage the software configuration for every developer’s virtual machine. This used to be the case, where we had to be careful that the correct GIS libs, postgres version, ruby and node versions, etc. were installed on our vagrant VMs. Now, we just spin up a VM with docker-machine and let the containers do their own configurations.

Backing services like postgres, rabbitmq, etc. are easy to spin up for development. No more init scripts or arcane configs to worry about! Integration/acceptance tests can be run in a dedicated cluster. Using docker links, we can run an entire test suite (or even multiple concurrent suites) completely separate from the development cluster, without involving manual cluster management. It’s just a docker-compose up command away.


I feel a bit like Bilbo going on an Unexpected Journey through all of this, but it has been very rewarding. We’ve gone from weekly/monthly releases, to at-will releases, usually once or twice a week (non-zero downtime deployment is our biggest blocker at the moment). I love how the Continuous Delivery philosophy/tooling forces us to break our changes into smaller, independent units. While it seems like an impediment, I think it has made us more able to quickly address critical issues when we don’t have to wait for huge new feature branches to be merged in.


Back to the Browser - a JavaScript Workflow for UNIX Nerds

When Apple announced Mac OS X Lion, their tagline was “Back to the Mac” as they were bringing some features from iOS into the desktop-oriented Mac OS. In the JavaScript world, a similar thing has happened: innovations in the Node.js space can be brought back to the browser. These innovations have made JavaScript development faster and cleaner with command-line tools and the npm packaging system.

As I began writing serious JavaScript libraries and apps, I wanted the same kind of workflow I enjoy when writing Ruby code. I wanted to write my code in vi, run tests in the command line, organize my code into classes and modules, and use versioned packages similar to Ruby gems. At the time, the standard way to write JavaScript was to manage separate files by hand and concatenate them into a single file. One of the only testing frameworks in town was Jasmine, which required you to run tests in the browser. Since then, there has been an explosion of command-line code packaging and testing frameworks in the Node.js community that have lent themselves well to client side development. What follows is the approach I find to be the most productive.

Here’s a list of the tools that correspond to their Ruby world counterparts:

Application Ruby Javascript
Testing RSpec vows, buster |
Package management rubygems, bundler npm, browserify |
Code organization require CommonJS |
Build tools jeweler, rubygems browserify |

By installing Node.js, you have access to a command-line JavaScript runtime, testing, package management, and application building. Running tests from the command-line allows you to more easily use tools like guard, run focused unit tests, and easily set up continuous integration.


Many JavaScripters run Jasmine in the browser for testing. While it does the job, its syntax is extremely verbose and it breaks the command-line-only workflow. There is a Node.js package for running Jasmine from the command line, but I have found it to be buggy and not as feature rich as a typical command line testing tool. Instead I prefer vows or buster.js. Each supports a simpler “hash” based syntax, as opposed to Jasmine’s verbose syntax:

    // Jasmine

describe('MyClass', function() {
  describe('#myMethod', function() {
    before(function() {
      this.instance = new MyClass('foo');
    it('returns true by default', function() {
    it('returns false sometimes', function() {
    // Vows

  '#myMethod': {
    topic: new MyClass('foo'),

    'returns true by default': function(instance) {
    'returns false sometimes': function(instance) {

Vows and buster can be used just like rspec to run tests from the command line:

> vows test/my-class-test.js
OK >> 22 honored

One advantage that buster has over vows is that it can run its tests both from the command line and from a browser in case you want to run some integration tests in a real browser environment.

For mocks and stubs, you can use the excellent sinon library, which is included by default with buster.js.

Integration testing

In addition to unit testing, it’s always good run a full integration test. Since every browser has its own quirks, it’s best to run integration tests in each browser. I write cucumber tests using capybara to automatically drive either a “headless” (in-memory) webkit browser with capybara-webkit and/or GUI browsers like Firefox and Chrome with selenium.

In features/support/env.rb you can define which type of browser is used to run the tests by defining custom drivers

    require 'selenium-webdriver'

    Capybara.register_driver :selenium_chrome do |app|
      Capybara::Selenium::Driver.new app, :browser => :chrome

    Capybara.register_driver :selenium_firefox do |app|
      Capybara::Selenium::Driver.new app, :browser => :firefox

    if ENV['BROWSER'] == 'chrome'
      Capybara.current_driver = :selenium_chrome
    elsif ENV['BROWSER'] == 'firefox'
      Capybara.current_driver = :selenium_firefox
      require 'capybara-webkit'
      Capybara.default_driver = :webkit

Now you can choose your browser with an environment variable: BROWSER=firefox cucumber features

If you are testing an app apart from a framework like Sinatra or Rails, you can use Rack to serve a static page that includes your built app in a <script> tag. For example, you could have an html directory with an index.html file in it:

    <title>Test App</title>
    <script type="text/javascript" src="application.js"></script>
  <body><div id="app"></div></body>

When you’re ready to run an integration test, compile your code into application.js using browserify:

> browserify -e lib/main.js -o html/application.js

Then tell cucumber to load your test file as the web app to test:

    # features/support/env.rb
    require 'rack'
    require 'rack/directory'

    Capybara.app = Rack::Builder.new do
      run Rack::Directory.new(File.expand_path('../../../html/', __FILE__))

Once cucumber is set up, you can start writing integration tests just as you would with Rails:

# features/logging_in.feature

Feature: Logging in

Scenario: Successful in-log
  Given I am on the home page
  When I log in as derek
  Then I should see a welcome message
    # features/step_definitions/log_in_steps.rb

    Given %r{I am on the home page} do
      visit '/index.html'
    When %r{I log in as derek} do
      click '#login'
      fill_in 'username', :with => 'derek'
      fill_in 'password', :with => 'secret'
      click 'input[type=submit]'
    Then %r{I should see a welcome message} do
      page.should =~ /Welcome, derek!/

Package management

One of the joys of Ruby is its package manager, rubygems. With a simple gem install you can add a library to your app. There has been an explosion of JavaScript package managers lately. Each one adds the basic ability to gather all of your libraries and application code, resolve the dependencies, and concatenate them into a single application file. I prefer browserify over all the others for two reasons. First, you can use any Node.js package, which opens you up to many more utilities and libraries than other managers. Second, it uses Node.js’ CommonJS module system, which is a very simple and elegant module system.

In your project’s root, place a package.json file that defines the project’s dependencies:

      "dependencies": {
        "JSONPath": "0.4.2",
        "underscore": "*",
        "jquery": "1.8.1"
      "devDependencies": {
        "browserify": "*",
        "vows": "*"

Run npm install and all of your project’s dependencies will be installed into the node_modules directory. In your project you can then make use of these packages:

    var _ = require('underscore'),
        jsonpath = require('JSONPath'),
        myJson = "...";

    _.each(jsonpath(myJson, '$.books'), function(book) {

If you’re looking for packages available for certain tasks, simply run npm search <whatever> to find pacakges related to your search terms. Some packages are tagged with “browser” if they are specifically meant for client side apps, so you can include “browser” as one of your search terms to limit your results accordingly. Many of the old standbys, like jquery, backbone, spine, and handlebars are there.

Code organization

As JavaScript applications get more complex, it becomes prudent to split your code into separate modules, usually placed in separate files. In the Ruby world, this was easily done by require-ing each file. Node.js introduced many people (including me) to the CommonJS module system. It’s a simple and elegant way to modularize your code and allows you to separate each module into its own file. Browserify allows you to write your code in the CommonJS style and it will roll all of your code up into a single file appropriate for the browser.

Ruby structure

For example, my Ruby project may look like:


Where lib/my_library.rb looks like:

    require 'my_library/book'

    class MyLibrary
      def initialize(foo)
        @book = Book.parse(foo)

And lib/my_library/book.rb looks like:

    require 'jsonpath'

class MyLibrary
  class Book
    def self.parse(foo)
      JSONPath.eval(foo, '$.store.book\[0\]')

And spec/my_library/book_spec.rb looks like:

    require 'json'
    require 'helper'
    require 'my_library/book'

    describe MyLibrary::Book do
      describe '.parse' do
        it 'parses a book object' do
          json = File.read('support/book.json')
          book = Book.parse(JSON.parse(json))
          book.title.should == "Breakfast at Tiffany's"

JavaScript structure

A javascript project would look similar:


Where lib/my-library.js looks like:

    var Book = require('./my-library/book');

var MyLibrary = function(foo) {
  this.book = new Book(foo);

module.exports = MyLibrary;

And lib/my-library/book.js looks like:

    var jsonpath = require('jsonpath');

var Book = {
  parse: function(foo) {
    return jsonpath(foo, '$.store.book\[0\]');

module.exports = Book;

And test/my-library/book-test.js looks like:

    var fs = require('fs');
var helper = require('../helper'),
    Book = require('../../lib/my_library/book');
    // NOTE: there are ways to set up your modules 
    // to be able to use relative require()s but
    // it is beyond the scope of this article

  '.parse': {
    'parses a book object': function() {
      var json = fs.readFileSync('support/book.json'),
          book = Book.parse(JSON.parse(json));
      assert.equal(book.title, "Breakfast at Tiffany's");

Build tools

Browserify will build concatenated JavaScript files when you’re ready to deploy your code on a website or as a general-purpose library. Its usage is simple:

> browserify -e <main_application_startup_code> -o <path_to_built_file>

Building a library

If we were building the library in the section above, we could run browserify -e lib/my-library.js -o build/my-library.js. Then, any user of your library can use your library with the require function:

    <script type="text/javascript" src="jquery.js"></script>
    <script type="text/javascript" src="my-library.js"></script>
    <script type="text/javascript">
      var myLibrary = require('my-library');
      $.ajax('/lib.json', function(data) {

You can also save the library user some time with a custom entry point for browsers:

    // in /browser.js
    window.MyLibrary = require('my-library');

Then run `browserify -e browser.js -o build/my-library.js

And the library user would use it thusly:

    <script type="text/javascript" src="jquery.js"></script>
    <script type="text/javascript" src="my-library.js"></script>
    <script type="text/javascript">
      $.ajax('/lib.json', function(data) {

Building a web app

A spine app might look something like:

    // in app/main.js
    var $ = require('jquery'),
        Spine = require('spine');

    Spine.$ = $;

    var MainController = require('./controllers/main-controller');

    var ApplicationController = Spine.Controller.sub({
      init: function() {
        var main = new MainController();
          '/': function() { main.active(); }

    Spine.Route.setup({ history: true });

It would be built with browserify -e app/main.js -o build/application.js and the application.js added to your website with a <script> tag.

You can extend browserify with plugins like templatify, which precompiles HTML/Handlebar templates into your app.

Together, npm packages, command-line testing and build tools, and modular code organization help you quickly build non-trivial JavaScript libraries and applications just as easily as it was in Ruby land. I’ve developed several in-production projects using this workflow, such as our CM1 JavaScript client library, our flight search browser plugin, and hootroot.com.