+rw

Modifying the web

Back to the Browser - a JavaScript Workflow for UNIX Nerds

When Apple announced Mac OS X Lion, their tagline was “Back to the Mac” as they were bringing some features from iOS into the desktop-oriented Mac OS. In the JavaScript world, a similar thing has happened: innovations in the Node.js space can be brought back to the browser. These innovations have made JavaScript development faster and cleaner with command-line tools and the npm packaging system.

As I began writing serious JavaScript libraries and apps, I wanted the same kind of workflow I enjoy when writing Ruby code. I wanted to write my code in vi, run tests in the command line, organize my code into classes and modules, and use versioned packages similar to Ruby gems. At the time, the standard way to write JavaScript was to manage separate files by hand and concatenate them into a single file. One of the only testing frameworks in town was Jasmine, which required you to run tests in the browser. Since then, there has been an explosion of command-line code packaging and testing frameworks in the Node.js community that have lent themselves well to client side development. What follows is the approach I find to be the most productive.

Here’s a list of the tools that correspond to their Ruby world counterparts:

Application Ruby Javascript
Testing RSpec vows, buster |
Package management rubygems, bundler npm, browserify |
Code organization require CommonJS |
Build tools jeweler, rubygems browserify |

By installing Node.js, you have access to a command-line JavaScript runtime, testing, package management, and application building. Running tests from the command-line allows you to more easily use tools like guard, run focused unit tests, and easily set up continuous integration.

Testing

Many JavaScripters run Jasmine in the browser for testing. While it does the job, its syntax is extremely verbose and it breaks the command-line-only workflow. There is a Node.js package for running Jasmine from the command line, but I have found it to be buggy and not as feature rich as a typical command line testing tool. Instead I prefer vows or buster.js. Each supports a simpler “hash” based syntax, as opposed to Jasmine’s verbose syntax:

    // Jasmine

describe('MyClass', function() {
  describe('#myMethod', function() {
    before(function() {
      this.instance = new MyClass('foo');
    });
  
    it('returns true by default', function() {
      expect(this.instance.myMethod()).toBeTruthy();
    });
    it('returns false sometimes', function() {
      expect(this.instance.myMethod(1)).toBeFalsy();
    });
  });
});
  
    // Vows

vows.describe('MyClass').addBatch({
  '#myMethod': {
    topic: new MyClass('foo'),

    'returns true by default': function(instance) {
      assert(instance.myMethod());
    },
    'returns false sometimes': function(instance) {
      refute(instance.myMethod(1));
    }
  }
}).export(module);
  

Vows and buster can be used just like rspec to run tests from the command line:

> vows test/my-class-test.js
................
OK >> 22 honored

One advantage that buster has over vows is that it can run its tests both from the command line and from a browser in case you want to run some integration tests in a real browser environment.

For mocks and stubs, you can use the excellent sinon library, which is included by default with buster.js.

Integration testing

In addition to unit testing, it’s always good run a full integration test. Since every browser has its own quirks, it’s best to run integration tests in each browser. I write cucumber tests using capybara to automatically drive either a “headless” (in-memory) webkit browser with capybara-webkit and/or GUI browsers like Firefox and Chrome with selenium.

In features/support/env.rb you can define which type of browser is used to run the tests by defining custom drivers

    require 'selenium-webdriver'

    Capybara.register_driver :selenium_chrome do |app|
      Capybara::Selenium::Driver.new app, :browser => :chrome
    end

    Capybara.register_driver :selenium_firefox do |app|
      Capybara::Selenium::Driver.new app, :browser => :firefox
    end

    if ENV['BROWSER'] == 'chrome'
      Capybara.current_driver = :selenium_chrome
    elsif ENV['BROWSER'] == 'firefox'
      Capybara.current_driver = :selenium_firefox
    else
      require 'capybara-webkit'
      Capybara.default_driver = :webkit
    end
  

Now you can choose your browser with an environment variable: BROWSER=firefox cucumber features

If you are testing an app apart from a framework like Sinatra or Rails, you can use Rack to serve a static page that includes your built app in a <script> tag. For example, you could have an html directory with an index.html file in it:

    <html>
  <head>
    <title>Test App</title>
    <script type="text/javascript" src="application.js"></script>
  </head>
  <body><div id="app"></div></body>
</html>
  

When you’re ready to run an integration test, compile your code into application.js using browserify:

> browserify -e lib/main.js -o html/application.js

Then tell cucumber to load your test file as the web app to test:

    # features/support/env.rb
    
    require 'rack'
    require 'rack/directory'

    Capybara.app = Rack::Builder.new do
      run Rack::Directory.new(File.expand_path('../../../html/', __FILE__))
    end
  

Once cucumber is set up, you can start writing integration tests just as you would with Rails:

# features/logging_in.feature

Feature: Logging in

Scenario: Successful in-log
  Given I am on the home page
  When I log in as derek
  Then I should see a welcome message
    # features/step_definitions/log_in_steps.rb

    Given %r{I am on the home page} do
      visit '/index.html'
    end
    
    When %r{I log in as derek} do
      click '#login'
      fill_in 'username', :with => 'derek'
      fill_in 'password', :with => 'secret'
      click 'input[type=submit]'
    end
    
    Then %r{I should see a welcome message} do
      page.should =~ /Welcome, derek!/
    end
  

Package management

One of the joys of Ruby is its package manager, rubygems. With a simple gem install you can add a library to your app. There has been an explosion of JavaScript package managers lately. Each one adds the basic ability to gather all of your libraries and application code, resolve the dependencies, and concatenate them into a single application file. I prefer browserify over all the others for two reasons. First, you can use any Node.js package, which opens you up to many more utilities and libraries than other managers. Second, it uses Node.js’ CommonJS module system, which is a very simple and elegant module system.

In your project’s root, place a package.json file that defines the project’s dependencies:

    {
      "dependencies": {
        "JSONPath": "0.4.2",
        "underscore": "*",
        "jquery": "1.8.1"
      },
      "devDependencies": {
        "browserify": "*",
        "vows": "*"
      }
    }
  

Run npm install and all of your project’s dependencies will be installed into the node_modules directory. In your project you can then make use of these packages:

    var _ = require('underscore'),
        jsonpath = require('JSONPath'),
        myJson = "...";

    _.each(jsonpath(myJson, '$.books'), function(book) {
      console.log(book);
    });
  

If you’re looking for packages available for certain tasks, simply run npm search <whatever> to find pacakges related to your search terms. Some packages are tagged with “browser” if they are specifically meant for client side apps, so you can include “browser” as one of your search terms to limit your results accordingly. Many of the old standbys, like jquery, backbone, spine, and handlebars are there.

Code organization

As JavaScript applications get more complex, it becomes prudent to split your code into separate modules, usually placed in separate files. In the Ruby world, this was easily done by require-ing each file. Node.js introduced many people (including me) to the CommonJS module system. It’s a simple and elegant way to modularize your code and allows you to separate each module into its own file. Browserify allows you to write your code in the CommonJS style and it will roll all of your code up into a single file appropriate for the browser.

Ruby structure

For example, my Ruby project may look like:

~lib/
 -my_library.rb
 -my_library/
   -book.rb
-my_library.gemspec
-spec/
 -my_library/
   -book_spec.rb

Where lib/my_library.rb looks like:

    require 'my_library/book'

    class MyLibrary
      def initialize(foo)
        @book = Book.parse(foo)
      end
    end
  

And lib/my_library/book.rb looks like:

    require 'jsonpath'

class MyLibrary
  class Book
    def self.parse(foo)
      JSONPath.eval(foo, '$.store.book\[0\]')
    end
  end
end
  

And spec/my_library/book_spec.rb looks like:

    require 'json'
    require 'helper'
    require 'my_library/book'

    describe MyLibrary::Book do
      describe '.parse' do
        it 'parses a book object' do
          json = File.read('support/book.json')
          book = Book.parse(JSON.parse(json))
          book.title.should == "Breakfast at Tiffany's"
        end
      end
    end
  

JavaScript structure

A javascript project would look similar:

~lib/
 -my-library.js
 -my-library/
   -book.js
-package.json
-test/
 -my-library/
   -book-test.js

Where lib/my-library.js looks like:

    var Book = require('./my-library/book');

var MyLibrary = function(foo) {
  this.book = new Book(foo);
};

module.exports = MyLibrary;
  

And lib/my-library/book.js looks like:

    var jsonpath = require('jsonpath');

var Book = {
  parse: function(foo) {
    return jsonpath(foo, '$.store.book\[0\]');
  }
};

module.exports = Book;
  

And test/my-library/book-test.js looks like:

    var fs = require('fs');
var helper = require('../helper'),
    Book = require('../../lib/my_library/book');
    // NOTE: there are ways to set up your modules 
    // to be able to use relative require()s but
    // it is beyond the scope of this article

vows.describe('Book').addBatch({
  '.parse': {
    'parses a book object': function() {
      var json = fs.readFileSync('support/book.json'),
          book = Book.parse(JSON.parse(json));
      assert.equal(book.title, "Breakfast at Tiffany's");
    }
  }
}).export(module);
  

Build tools

Browserify will build concatenated JavaScript files when you’re ready to deploy your code on a website or as a general-purpose library. Its usage is simple:

> browserify -e <main_application_startup_code> -o <path_to_built_file>

Building a library

If we were building the library in the section above, we could run browserify -e lib/my-library.js -o build/my-library.js. Then, any user of your library can use your library with the require function:

    <script type="text/javascript" src="jquery.js"></script>
    <script type="text/javascript" src="my-library.js"></script>
    <script type="text/javascript">
      var myLibrary = require('my-library');
      $.ajax('/lib.json', function(data) {
        console.log(myLibrary(data));
      });
    </script>
  

You can also save the library user some time with a custom entry point for browsers:

    // in /browser.js
    window.MyLibrary = require('my-library');
  

Then run `browserify -e browser.js -o build/my-library.js

And the library user would use it thusly:

    <script type="text/javascript" src="jquery.js"></script>
    <script type="text/javascript" src="my-library.js"></script>
    <script type="text/javascript">
      $.ajax('/lib.json', function(data) {
        console.log(MyLibrary(data));
      });
    </script>
  

Building a web app

A spine app might look something like:

    // in app/main.js
    
    var $ = require('jquery'),
        Spine = require('spine');

    Spine.$ = $;

    var MainController = require('./controllers/main-controller');

    var ApplicationController = Spine.Controller.sub({
      init: function() {
        var main = new MainController();
        this.routes({
          '/': function() { main.active(); }
        });
      }
    });

    Spine.Route.setup({ history: true });
  

It would be built with browserify -e app/main.js -o build/application.js and the application.js added to your website with a <script> tag.

You can extend browserify with plugins like templatify, which precompiles HTML/Handlebar templates into your app.

Together, npm packages, command-line testing and build tools, and modular code organization help you quickly build non-trivial JavaScript libraries and applications just as easily as it was in Ruby land. I’ve developed several in-production projects using this workflow, such as our CM1 JavaScript client library, our flight search browser plugin, and hootroot.com.

Graphite and Statsd – Beyond the Basics

The Graphite and statsd systems have been popular choices lately for recording system statistics, but there isn’t much written beyond how to get the basic system set up. Here are a few tips that will make your life easier.

Graphite + statsd - the rundown

The graphite and statsd system consists of three main applications

  • carbon: a service that receives and stores statistics
  • statsd: a node server that provides an easier and more performant, UDP-based protocol for receiving stats which are passed off to carbon
  • graphite: a web app that creates graphs out of the statistics recorded by carbon

Use graphiti

Several alternative front-ends to graphite have been written. I chose to use graphiti because it had the most customizable graphs. Note that graphiti is just a facade on top of graphite - you still need the graphite web app running for it to work. Graphiti makes it easy to quickly create graphs. I’ll cover this later.

The flow looks like:

|App| ==[UDP]==> |statsd| ==> |carbon| ==> |.wsp file|

|.wsp file| ==> |graphite| ==> |graphiti| ==> |pretty graphs on your dashboard|

Use chef-solo to install it

If you’re familiar with chef, you can use the cookboos that the community has already developed for installing graphite and friends. If not, this would be a good opportunity to learn. You can use chef-solo to easily deploy graphite to a single server. I plan to write a “getting started with chef-solo” post soon, so stay tuned!

Chef saved me a ton of time setting up python, virtualenv, graphite, carbon, whisper, statsd, and many other tools since there are no OS-specific packages for some of these.

Use sensible storage schemas

The default chef setup of graphite stores all stats with the following storage schema rule:

[catchall]
priority = 0
pattern = ^.*
retentions = 60:100800,900:63000

The retentions setting is the most important. It’s a comma-delimited list of data resolutions and amounts. * The number before the colon is the size of the bucket that holds data in seconds. A value of 60 means that 60 seconds worth of data is grouped together in the bucket. A larger number means the data is less granular, but more space efficient. * The number after the colon is the number of data buckets to store at that granularity. 100800 will cover (100800 * 60) = 70 days of data. That’s (100800 * 12) = 1.2MiB of space for those 70 days. A bigger number means more disk space and longer seek times.

Alternatively, you can specify retentions using time format shortcuts. For example, 1m:7d means “store 7 days worth of 1-minute granular data.”

Use a good stats client

In the ruby world, there are two popular client libraries: fozzie and statsd-ruby. Both provide the standard operations like counting events, timing, and gauging values.

Fozzie differs in that it integrates with Rails or rack apps by adding a rack middleware that automatically tracks timing statistics for every path in your web app. This can save time, but it also has the downside of sending too much noise to your statsd server and can cause excessive disk space consumption unless you implement tight storage schema rules. It also adds a deep hierarchy of namespaces based on the client machine name, app name, and current environment. This can be an issue on heroku web apps where the machine name changes frequently.

If you want more control over your namespacing, statsd-ruby is the way to go. Otherwise, fozzie may be worth using for its added conveniences.

Make sure you don’t run out of disk space

Seriously, if you do run out of disk, the graphite (whisper) data files can become corrupted and force you to delete them and start over. I learned this the hard way :) Make sure your storage schemas are strict enough because each separate stat requires its own file that can be several megabytes in size.

Use graphiti for building graphs and dashboards

Graphiti has a great interface for building graphs. You can even fork it and deploy your own custom version that fits your company’s needs and/or style. It’s a small rack app that uses redis to store graph and dashboard settings. There’s even a chef cookbook for it!

When setting up graphiti, remember to set up a cron job to run rake graphiti:metrics periodically so that you can search for metric namespaces from graphiti.

Use graphite’s built-in functions for summarizing and calculating data

Graphite provides a wealth of functions that run aggregate operations on data before it is graphed.

For example, let’s say we’re tracking hit counts on our app’s home page. We’re using several web servers for load balancing and our stats data is namespaced by server under stats.my_app.server-a.production.home-page.hits and stats.my_app.server-b.production.home-page.hits. If we told graphite to graph results for stats.my_app.*.production.home-page.hits we would get two graph lines – one for server-a and one for server-b. To combine them into a single measurement, use the sumSeries function. You can then use the alias function to give it a friendlier display name like “Home page.”

Graphiti has a peculiar way of specifying which function to use. In a normal series list, you have the following structure:

"targets": [
  [
    "stats.my_app.*.production.home-page.hits",
    {}
  ]
]

The {} is an object used to specify the list of functions to apply, in order, on the series specified in the parent array. Each graphite function is specified as a key and its parameters as the value. A true value indicates the function needs no parameters and an array is provided if the function requires multiple parameters.

You’ll notice in the function documentation that each function usually takes two initial arguments, a context and a series name. In graphiti, you won’t need to specify those first two arguments.

Here’s an example of sumSeries and alias used together. Note that the order matters!

"targets": [
  [
    "stats.my_app.*.production.home-page.hits",
    {
      "sumSeries": true,
      "alias": "Homepage hits"
    }
  ]
]

Different graph areaMode for different applications

While not well documented, graphite has a few options for displaying graph lines. By default, the “stacked” area mode stacks each measurement on top of each other into an area chart that combines multiple measurements that are uniquely shaded. This can be good for seeing grand totals. The blank option plots each measurement as a line on a line chart. This is preferable for comparing measurements.

Different metrics for different events

Each stats recording method provided by statsd-ruby and fozzie has different behavior, which isn’t well documented anywhere.

  • Stats.count is the base method for sending a count of some event for a given instance. It’s rarely used alone.
  • Stats.increment and Stats.decrement will adjust a count of an event. It’s useful for counting things like number of hits on a page, number of times an activity occurs, etc. It will be graphed as “average number of events per second”. So if your web app runs Stats.increment 'hits' 8 times over a 1 second period, the graph will draw a value of 8 for that second. Sometimes you will see fractional numbers charted. This is because graphite may average the data over a time period based on your schema storage settings and charting resolution.
  • Stats.timing will take a block and store the amount of time the code in the block took to execute. It also keeps track of average, min, and max times, as well as standard deviation and total number of occurrences.
  • Stats.gauge tracks absolute values over time. This is useful for tracking measurements like CPU, memory, and disk usage.
  • Fozzie provides Stats.event 'party' to track when an event happens. This is useful for tracking things like deploys or restarts. Equivalent functionality can be obtained in statsd-ruby by running Stats.count 'party', Time.now.to_i.

Bonus tip: Graphs on your Mac dashboard

If you’re using a mac, you can add your favorite graphs to your dashboard. Create a graph in graphiti, then view it on the graphiti dashboard with Safari. Click File->Open in Dashboard… and select the graph image with the select box. Now, you can quickly see important graphs at the press of a button!

Overall, statsd is a great tool and can add great visibility into your applications.

Add node.js/CommonJS Style Require() to Client-side JavaScript With Browserify

While working on Careplane recently, I ran into a problem that was made easier by JSONPath. I was already enjoying the ability to test my JavaScript with Node.js, Jasmine, and other npm packages. It would be nice if I could get the same package system I have with npm and use it with my client-side JavaScript.

With browserify, now I can!

There are three major benefits to using browserify, which I’ll detail below.

Using npm Modules in Your Client-Side Scripts

In Careplane, I wanted to use the JSONPath npm package. With browserify, I can write the following into my client-side code:

1
2
3
4
5
6
/* clientside.js */
var jsonpath = require('JSONPath');
var $ = require('jquery-browserify');

var owned_data = jsonpath.eval(myData, '$.paths[*].to.data[?(@.owner == "Derek")]');
$('p.important').html(owned_data);

From the command-line, I can create a single JavaScript file (I’ll call it application.js) that contains my code and includes the JSONPath package. I can even bake jQuery into application.js!

1
2
3
4
> npm install browserify
> npm install JSONPath
> npm install jquery-browserify
> ./node_modules/.bin/browserify -r JSONPath -r jquery-browserify -e clientside.js -o application.js

Now that I have a nicely wrapped application.js, I can include it from a web page, or in this case, the browser plugin I’m developing.

Here’s an example of use from a web page:

1
<script type="text/javascript" src="application.js"></script>

N.B. Not all npm packages will work client-side, especially ones that depend on core Node.js modules like fs or http.

Organize Your Code

Careplane has a ton of JavaScript split into a separate file for each class. Previously, I was using simple concatenation and a hand-crafted file list to construct an application.js and to determine script load order. Now it can be managed automatically by browserify because each of my class files can require other class files.

For instance, the Kayak class in src/drivers/Kayak.js depends on the Driver class in src/Driver.js. I can now do this:

1
2
3
4
5
6
7
/* src/Driver.js */

var Driver = function() {};

/* ... */

module.exports = Driver;
1
2
3
4
5
6
7
8
9
/* src/Kayak.js */
var Driver = require('../Driver')

var Kayak = function(foo) { this.foo = foo; };
Kayak.prototype = new Driver();

/* ... */

module.exports = Kayak;
1
2
3
4
5
/* src/main.js */

var Kayak = require('./drivers/Kayak'); // paths are relative to src, main.js' cwd

Kayak.getThePartyStarted();
1
> ./node_modules/.bin/browserify -e src/main.js -o application.js

Super simple!

Uniformity With Testing Environment

I like to run my Jasmine JavaScript tests from the command-line and so does our Continuous Integration system. With jasmine-node, I had to maintain a list of files in my src directory that were loaded in a certain order in order to run my tests. Now, each spec file can use Node.js’ CommonJS require statement to require files from my src directory, and all dependencies are automatically managed.

1
2
3
4
5
6
/* spec/javascripts/drivers/KayakSpec.js */

describe('Kayak', function() {
  var Kayak = require('src/drivers/Kayak');
  /* ... */
});
1
2
3
> rake examples[KayakSpec]
Starting tests
............FFF............FF

When I run my specs from the command-line with Node.js (I have an examples rake task that runs node and jasmine), Node’s built-in CommonJS require is used and it recognizes the require statements I wrote into src/drivers/Kayak.js.

I can also run my Jasmine specs from the standard Jasmine web server. All I have to do is create a browserified script that is included before Jasmine runs its tests in my browser.

1
2
/* src/jasmine.js */
var Kayak = require('./drivers/Kayak');  // browserify needs a starting point to resolve all dependencies
1
2
3
# spec/javascripts/support/jasmine.yml
src_files:
  - spec/javascripts/support/application.js
1
2
3
4
5
6
7
> ./node_modules/.bin/browserify -e src/jasmine.js -o spec/javascripts/support/application.js
> rake jasmine:server
your tests are here:
  http://localhost:8888/
[2011-08-04 16:15:56] INFO  WEBrick 1.3.1
[2011-08-04 16:15:56] INFO  ruby 1.9.2 (2011-02-18) [x86_64-darwin10.4.0]
[2011-08-04 16:15:56] INFO  WEBrick::HTTPServer#start: pid=61833 port=8888

When I visit http://localhost:8888 in my browser, my Jasmine tests all run!

The only drawback is that my code has to be re-browserified whenever it changes. This only adds about a second of overhead while I wait for my watchr script to run the browserification.

More

If you’d like to take a look at the environment I set up for Careplane, you can check out the source.

I hope you enjoy browserify as much as I have!

A Pattern for JavaScript Events

While working on Hootroot and Careplane, I found myself getting frustrated with the way I was having handling events. Over time, however, I stopped fighting the language and learned a pattern that I believe is easiest to test and read.

I’ll work with a simple example to show you my thought process.

Initially, I started handling my events with standard closures:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
//Rocket.js

Rocket = function(location) {
  this.location = location;
};

Rocket.prototype.ignite = function() { /* ... */ };
Rocket.prototype.scrub = function() { /* ... */ };

Rocket.prototype.launch = function() {
  var rocket = this;
  $.ajax('/launch_code', {
    data: { location: this.location },
    success: function(data) {
      if(rocket.isReady()) {
        alert('Launching from ' + rocket.location);
        rocket.ignite(data.launchCode);
      } else {
        alert('Not ready to launch!');
      }
    },
    error: function() {
      alert('Failed to get launch code for ' + rocket.location);
      rocket.scrub();
    }
  });
};

//RocketSpec.js

describe('Rocket', function() {
  describe('#launch', function() {
    var rocket;
    beforeEach(function() {
      rocket = new Rocket('Cape Canaveral, FL');
      fakeAjax({urls: {'/launch_code': {successData: '{ "launchCode": 12345 }'}}})
      spyOn(rocket, 'ignite');
      spyOn(rocket, 'scrub');
    });

    it('ignites if ready', function() {
      rocket.isReady = function() { return true; };
      rocket.launch();
      expect(rocket.ignite).toHaveBeenCalled();
    });
    it('waits if not ready', function() {
      rocket.isReady = function() { return false; };
      rocket.launch();
      expect(rocket.ignite).not.toHaveBeenCalled();
    });
    it('scrubs if a bad launch code is given', function() {
      fakeAjax({urls: {'/launch_code': { errorMessage: '{ "error": "too bad" }'}}})
      rocket.launch();
      expect(rocket.scrub).toHaveBeenCalled();
    });
  });
});

Because of the way this works in JavaScript, I had to assign the this that referred to the current instance of Rocket to a temporary variable that is referenced in the event handlers. This seemed kludgy to me, and I soon discovered the $.proxy() method that jQuery (and other frameworks similarly) provide:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
//Rocket.js

Rocket = function(location) {
  this.location = location;
};

Rocket.prototype.ignite = function() { /* ... */ };
Rocket.prototype.scrub = function() { /* ... */ };

Rocket.prototype.launch = function() {
  $.ajax('/launch_code', {
    data: { location: this.location },
    success: $.proxy(function(data) {
      if(this.isReady()) {
        alert('Launching from ' + this.location);
        this.ignite(data.launchCode)
      } else {
        alert('Not ready to launch!');
      }
    }, this),
    error: $.proxy(function() {
      alert('Failed to get launch code for ' + this.location);
      this.scrub();
    }, this)
  });
};

//RocketSpec.js

describe('Rocket', function() {
  describe('#launch', function() {
    var rocket;
    beforeEach(function() {
      rocket = new Rocket('Cape Canaveral, FL');
      fakeAjax({urls: {'/launch_code': {successData: '{ "launchCode": 12345 }'}}})
      spyOn(rocket, 'ignite');
      spyOn(rocket, 'scrub');
    });

    it('ignites if ready', function() {
      rocket.isReady = function() { return true; };
      rocket.launch();
      expect(rocket.ignite).toHaveBeenCalled();
    });
    it('waits if not ready', function() {
      rocket.isReady = function() { return false; };
      rocket.launch();
      expect(rocket.ignite).not.toHaveBeenCalled();
    });
    it('scrubs if a bad launch code is given', function() {
      fakeAjax({urls: {'/launch_code': { errorMessage: '{ "error": "too bad" }'}}})
      rocket.launch();
      expect(rocket.scrub).toHaveBeenCalled();
    });
  });
});

The problem now is there are all sorts of functions hanging around within Rocket#launch() that are a bit difficult to test in a straightforward manner. Solution: create some functions on Rocket that act as event handlers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
// Rocket.js

Rocket = function() {
  this.location = 'Cape Canaveral, FL';
};

Rocket.prototype.igniteWhenReady = function(data) {
  if(this.isReady()) {
    alert('Launching from ' + this.location);
    this.ignite(data.launchCode);
  } else {
    alert('Not ready to launch!');
  }
};

Rocket.prototype.invalidLaunchCode = function() {
  alert('Failed to get launch code for ' + this.location);
  this.scrub();
};

Rocket.prototype.launch = function() {
  $.ajax('/launch_code', {
    data: { location: this.location },
    success: $.proxy(this.igniteWhenReady, this),
    error: $.proxy(this.invalidLaunchCode, this)
  });
};

//RocketSpec.js

describe('Rocket', function() {
  var rocket;
  beforeEach(function() {
    rocket = new Rocket('Cape Canaveral, FL');
    spyOn(rocket, 'ignite');
    spyOn(rocket, 'scrub');
  });

  describe('#launch', function() {
    // we don't need to test launch() because we'd really just be testing $.ajax
  });

  describe('#igniteWhenReady', function() {
    it('ignites if ready', function() {
      rocket.isReady = function() { return true; };
      rocket.igniteWhenReady({ launchCode: 12345 });
      expect(rocket.ignite).toHaveBeenCalled();
    });
    it('does not ignite if not ready', function() {
      rocket.isReady = function() { return false; };
      rocket.igniteWhenReady();
      expect(rocket.ignite).not.toHaveBeenCalled();
    });
  });
  describe('#invalidLaunchCode', function() {
    it('scrubs if a bad launch code is given', function() {
      rocket.invalidLaunchCode();
      expect(rocket.scrub).toHaveBeenCalled();
    });
  });
});

This is much cleaner and easier to test, but those lingering $.proxy() calls were bugging me. They also made debugging a bit more tedious when having to step through the calls to $.proxy.

My solution: stop fighting with this and create my own event proxy pattern. Testing is now much cleaner.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
//Rocket.js

Rocket = function(location) {
  this.location = location;
};

Rocket.prototype.ignite = function() { /* ... */ };
Rocket.prototype.scrub = function() { /* ... */ };

Rocket.events = {
  igniteWhenReady: function(rocket) {
    return function(data)
      if(rocket.isReady()) {
        alert('Launching from ' + rocket.location);
        rocket.ignite(data.launchCode);
      } else {
        alert('Not ready to launch!');
      }
    };
  },

  invalidLaunchCode: function(rocket) {
    return function() {
      alert('Failed to get launch code for ' + rocket.location);
      rocket.scrub();
    };
  }
};

Rocket.prototype.launch = function() {
  $.ajax('/launch_code', {
    data: { location: this.location },
    success: Rocket.events.igniteWhenReady(this),
    error: Rocket.events.invalidLaunchCode(this)
  });
};

//RocketSpec.js

describe('Rocket', function() {
  var rocket, igniteWhenReady, invalidLaunchCode;
  beforeEach(function() {
    rocket = new Rocket('Cape Canaveral, FL');
    spyOn(rocket, 'ignite');
    spyOn(rocket, 'scrub');
    igniteWhenReady = Rocket.events.igniteWhenReady(rocket);
    invalidLaunchCode = Rocket.events.invalidLaunchCode(rocket);
  });

  describe('.events', function() {
    describe('.igniteWhenReady', function() {
      it('ignites if ready', function() {
        rocket.isReady = function() { return true; };
        igniteWhenReady({ launchCode: 12345 });
        expect(rocket.ignite).toHaveBeenCalled();
      });
      it('does not ignite if not ready', function() {
        rocket.isReady = function() { return false; };
        igniteWhenReady();
        expect(rocket.ignite).not.toHaveBeenCalled();
      });
    });
    describe('.invalidLaunchCode', function() {
      it('scrubs if a bad launch code is given', function() {
        invalidLaunchCode();
        expect(rocket.scrub).toHaveBeenCalled();
      });
    });
  });
});

The result is much more readable code, easier debugging (when absolutely necessary), and simpler testing without all those nested closures and AJAX stubs. As an added bonus, you get to keep the this in your event handlers that refers to the event itself.

This experience has led me to believe that a lot of the problems CoffeeScript tries to solve (like function binding) can really just be solved using good, simple JavaScript coding practices. I’m happy to hear from anyone who has a better pattern or has had similar experiences.

Carbon.js

Lately, I’ve been working on a Google Maps mashup that calculates CO2 emissions for automobile, bus, and rail trips. Since Google Maps has such a great JavaScript API, I decided to write the application almost entirely in JavaScript. Thus, Carbon.js was born!

Carbon.js makes it simple to calculate emissions. Its design is similar to the Ruby carbon gem. For any class that represents an “emission activity”, e.g. an automobile trip, you can extend it with Carbon.js to give it the ability to perform calculations.

Let’s look at an example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
RentalCar = function() {};
Carbon.emitter(RentalCar, function(emitter) {
  emitter.emitAs('automobile');
  emitter.provide('make_model');
  emitter.provide('weekly_distance');
  emitter.provide('timeframe');
});

var focus = new RentalCar;
focus.make_model = 'Ford Focus';
focus.weekly_distance = 436;
focus.timeframe = '2010-03-08/2010-03-14';

focus.getEmissionEstimate(function(estimate) {
  alert("My Focus' emissions for the rental period of March 8th-14th are: " + estimate.value());
});

As you can see, the configuration defined in the Carbon.emitter call maps properties on an instance of RentalCar to characteristics defined in automobile’s characteristics API. In order to calculate emissions, a call to getEmissionEstimate with a callback function parameter is required. Within that callback, you can update HTML on a page with the result.

Future Improvements

  • Carbon only maps properties of an object to characteristics. It will be simple to allow an instance function to be mapped to a characteristic.
  • Carbon.js requires that you use jQuery. One improvement would be to use platform-agnostic XHR and proxy functions, or intelligently detect available libraries, like Prototype, MooTools, etc.
  • For DRYness sake, Carbon.js operates on classes. It may be helpful to be able to decorate any JavaScript object on the fly.

Github Pages, Rocco, and Rake File Tasks

Recently, we spiffed up some of our emitters with enhanced documentation using Rocco, a Ruby port of Docco. We combined this with github’s ability to set up static html pages for a repo. By setting up a branch called gh-pages, Github will serve any html files in that branch. We use this feature to display Rocco-generated documentation of our carbon models (check out the flight emitter for an example).

This was great, but we were missing a way to automate the process by which Rocco will generate its documentation from the code in the master branch and place them in the gh-pages branch. Ryan Tomayko came to the rescue and wrote a rake task that creates a docs directory within the project and initializes a new git repository within the directory, which points to the project’s gh-pages branch. When documentation is generated, it is copied to the docs folder and pushed to gh-pages.

What intrigued me about the rake task was its use of file tasks. It’s a feature of rake I had never noticed before, but it’s pretty slick. A file task says, “if the specified path does not exist, execute the following code.” Since many unix tools use files for configuration, this feature plays well with many utilities, such as git, your favorite editor, etc.

For example, you could define a rake task that will create a .rvmrc for your project using your current RVM-installed ruby:

1
2
3
4
5
6
7
# Rakefile

file '.rvmrc' do |f|
  File.open(f.name, 'w') do |rvmrc|
    rvmrc.puts "rvm #{ENV['rvm_ruby_string']}"
  end
end

When you run rake .rvmrc, your .rvmrc will be generated. Try it out!

There is all kinds of magic you can work with a file task. A novel way in which Ryan’s Rocco tasks use file tasks is when deciding whether to create a git remote based on whether there is a file referencing the remote in the .git configuration directory:

1
2
3
4
5
# Rakefile

file '.git/refs/heads/gh-pages' => 'docs/' do |f|
  `cd docs && git branch gh-pages --track origin/gh-pages`
end

Happy raking!

Back From JRubyConf

I just got back from JRubyConf in Columbus, Ohio. I had a great time meeting other Rubyists and learned a great deal about how Ruby is being integrated into enterprise environments via JRuby. It’s really exciting to see “the big guys” embracing a language and ecosystem that we at Brighter Planet enjoy using on a daily basis.

When I got back home, I decided this would be a good opportunity to try out our new meeting and lodging emissions models to come up with a carbon footprint for my trip to the conference.

I started out with the meeting space itself. I fired up Google Earth and measured the square footage of the conference center we were at - about 1,150 square meters. With the area of the space and the location, I came up with 3.08 tons of CO2e generated by the heating, cooling, and electricity usage of the space: methodology. Of course, if we considered my individual footprint, it would be 150th of that amount, given that 150 of us shared the space. This comes out to 0.02 tons.

I then looked at my hotel stay for two nights at the Hampton Inn and came up with 0.09 tons of CO2e: methodology. I looked up the lodging class from our data repository

To get to the conference, I carpooled with a couple other Rubyists from Ann Arbor, Michigan. I used 18 gallons of gas, which comes out to 68.1L. When we developed the automobile trip emitter I was surprised to learn from Seamus and Ian that the type of engine you have doesn’t make a significant difference in the amount of CO2 emitted, it really comes down to how much gasoline is burned. Therefore, our automobile trip emitter looks at the type of fuel and the quantity burned. Transportation to and from the conference came out to 0.18 tons of CO2e: methodology. Divided by the three of us, my personal transportation footprint was 0.06 tons.

Of course, no conference weekend is complete without a few good meals with fellow programmers. Over the course of the weekend, I mostly ate vegetarian, but I splurged on a trip to City Barbecue for a tasty beef brisket sandwich. I looked at all the food I ate, and calculated the share of each of the food groups I ate. My foodprint for the weekend was 0.02 tons of CO2e: methodology.

Overall, my total footprint was 0.19 tons of CO2e. Had I not carpooled, the biggest factor would have been the car trip to the venue. With carpooling, most of my emissions came from the hotel stay.

Over the past few years, there have been more and more regional Ruby conferences, and it’s great to see the community buzzing. I think it would be really cool if someone could whip up a web app that analyzes Ruby (or other) conference footprints, perhaps based on data gathered from attendees. We could make a competition out of it to see which conference has the best average per-attendee footprint!

Bundler to the Max

I have been spending the past few weeks creating and refactoring our carbon model gems, with the goal of making them easy to enhance, fix, and test by climate scientists and Ruby developers. I wanted to make contributing a simple process and bundler fit the bill quite well.

A not-so-widely-known feature of the Rubygems API is the ability to declare a gem’s development dependencies, along with its runtime dependencies. If one planned on making changes to one of the emitter gems and testing it, she could run gem install <emitter_gem> --development and have any needed testing gems installed for the emitter gem.

This is all fine and good, but I chose to use bundler to manage our dependencies, as it adds a few extras that have been a tremendous help to us. To contribute to any of our gems, a developer can follow a simple process:

1
2
3
4
5
$ git clone git://github.com/brighterplanet/<gem>.git
$ cd <gem>
$ gem install bundler --pre  # this is needed until bundler 1.0 is released
$ bundle install
$ rake

And Bob’s your uncle!

Bundler + Gemspecs

The first goodie that bundler provides is the ability to use the gem’s own gemspec to define the dependencies needed for development. For instance, our flight gem has a gemspec with dependencies:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Gem::Specification.new do |s|
  # ...
  s.add_development_dependency(%q<activerecord>, ["= 3.0.0.beta4"])
  s.add_development_dependency(%q<bundler>, [">= 1.0.0.beta.2"])
  s.add_development_dependency(%q<cucumber>, ["= 0.8.3"])
  s.add_development_dependency(%q<jeweler>, ["= 1.4.0"])
  s.add_development_dependency(%q<rake>, [">= 0"])
  s.add_development_dependency(%q<rdoc>, [">= 0"])
  s.add_development_dependency(%q<rspec>, ["= 2.0.0.beta.17"])
  s.add_development_dependency(%q<sniff>, ["= 0.0.10"])
  s.add_runtime_dependency(%q<characterizable>, ["= 0.0.12"])
  s.add_runtime_dependency(%q<data_miner>, ["= 0.5.2"])
  s.add_runtime_dependency(%q<earth>, ["= 0.0.7"])
  s.add_runtime_dependency(%q<falls_back_on>, ["= 0.0.2"])
  s.add_runtime_dependency(%q<fast_timestamp>, ["= 0.0.4"])
  s.add_runtime_dependency(%q<leap>, ["= 0.4.1"])
  s.add_runtime_dependency(%q<summary_judgement>, ["= 1.3.8"])
  s.add_runtime_dependency(%q<timeframe>, ["= 0.0.8"])
  s.add_runtime_dependency(%q<weighted_average>, ["= 0.0.4"])
  # ...
end

Instead of defining these dependencies in both flight.gemspec and in Gemfile, we can instead give the following directive in our Gemfile:

1
gemspec :path => '.'

Bundler + Paths

We have a chain of gem dependencies, where an emitter gem depends on the sniff gem for development, which in turn depends on the earth gem for data models. In the olden days (like, 4 months ago) if I made a change to sniff, I would have to rebuild the gem and reinstall it. With bundler, I can simply tell my emitter gem to use a path to my local sniff repo as the gem source:

1
gem 'sniff', :path => '../sniff'

Now, any changes I make to sniff instantly appear in the emitter gem!

I had to add some special logic (a hack, if you will) to my gemspec definition for this to work, because the above gem statement in my Gemfile would conflict with the dependency listed in my gemspec (remember, I’m using my gemspec to tell bundler what gems I need). To get around this, I added an if clause to my gemspec definition that checks for an environment variable. If this variable exists, the gemspec will not request the gem and bundler will instead use my custom gem requirement that uses a local path:

1
2
3
4
5
6
# Rakefile  (we use jeweler to generate our gemspecs)
Jeweler::Tasks.new do |gem|
  # ...
  gem.add_development_dependency 'sniff', '=0.0.10' unless ENV['LOCAL_SNIFF']
  # ...
end
1
2
# Gemfile
gem 'sniff', :path => ENV['LOCAL_SNIFF'] if ENV['LOCAL_SNIFF']

So now, if I want to make some changes to the sniff gem and test them out in my emitter, I do:

1
2
3
4
5
6
7
8
9
$ cd sniff
  # work work work
$ cd ../[emitter]
$ export LOCAL_SNIFF=~/sniff
$ rake gemspec
$ bundle update
  # ...
sniff (0.0.13) using path /Users/dkastner/sniff
  # ...

And then Bob is my uncle.

Bundler + Rakefile

This next idea has some drawbacks in terms of code cleanliness, but I think it offers a good way to point contributers in the right direction. One thing that frustrated me about Jeweler was that if I wanted to contribute to a gem, my typical work flow went like:

1
2
3
4
5
6
7
8
$ cd [project]
  # work work work
$ rake test
LoadError: No such file: 'jeweler'
$ gem install jeweler
$ rake test
LoadError: No such file: 'shoulda'
  # etc etc

I attempted to simplify this process, so a new developer who doesn’t read the README should be able to just do:

1
2
3
4
5
6
7
8
$ cd [emitter]
  # work work work
$ rake test
You need to `gem install bundler` and then run `bundle install` to run rake tasks
$ gem install bundler
$ bundle install
$ rake test
All tests pass!

I achieved this by adding the following code to the top of the Rakefile:

1
2
3
4
5
6
7
require 'rubygems'
begin
  require 'bundler'
  Bundler.setup
rescue LoadError
  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
end

This was convenient, but it created a chicken and egg problem: in order to generate a gemspec for the first time, bundler needed to know which dependencies it needed, which meant that it needed the gemspec, which is generated by the Rakefile, which requires bundler, which requires the gemspec, etc. etc. I overcame this problem by allowing an override:

1
2
3
4
5
6
7
8
9
require 'rubygems'
unless ENV['NOBUNDLE']
  begin
    require 'bundler'
    Bundler.setup
  rescue LoadError
    puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
  end
end

So, if you’re really desparate, you can run rake test NOBUNDLE=true

More on Local Gems

Now that I had a way to easily tell bundler to use an actual gem or a local repo holding the gem, I wanted a way to quickly “flip the switch.” I wrote up a quick function in my ~/.bash_profile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
function mod_devgem() {
  var="LOCAL_`echo $2 | tr 'a-z' 'A-Z'`"

  if [ "$1" == "disable" ]
  then
    echo "unset $var"
    unset $var
  else
    dir=${3:-"~/$2"}
    echo "export $var=$dir"
    export $var=$dir
  fi
}

function devgems () {
  # Usage: devgems [enable|disable] [gemname]
  cmd=${1:-"enable"}

  if [ "$1" == "list" ]
  then
    env | grep LOCAL
    return
  fi

  if [ -z $2 ]
  then
    mod_devgem $cmd characterizable
    mod_devgem $cmd cohort_scope
    mod_devgem $cmd falls_back_on
    mod_devgem $cmd leap
    mod_devgem $cmd loose_tight_dictionary
    mod_devgem $cmd sniff
    mod_devgem $cmd data_miner
    mod_devgem $cmd earth
  else
    mod_devgem $cmd $2
  fi
}

This gives me a few commands:

1
2
3
4
5
6
$ devgems enable sniff
  # sets LOCAL_SNIFF=~/sniff
$ devgems disable sniff
  # clears LOCAL_SNIFF
$ devgems list
  # lists each LOCAL_ environment variable

I now have a well-oiled gem development machine!

Overall, after a few frustrations with bundler, I’m now quite happy with it, especially the power and convenience it gives me in developing gems.

I’m really interested to hear any of your thoughts on this. Drop me a line at @dkastner.

Gemspecs

I don’t follow Yehuda Katz’ blog (though I should) but he posted an article on basic gem configuration and build steps. It irks me to no end when I clone a git repo and run through the following process:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
> gem build somegem.gemspec
> gem install ./
[write code using gem]
> run code
#ERROR# some gem code is missing a file in "require_once"
> ls /path/to/gem/file/that/is/missing
File not found:  /path/to/gem/file/that/is/missing
> cd /path/to/gem/repo
> ls  /path/to/gem/repo/path/to/gem/file/that/is/missing
missing.rb
> rake build
#ERROR# missing gem my_crappy_test_library
> gem install my_crappy_test_library
> rake build
#ERROR# missing gem my_crappy_mock_library
> rake build
OK
> gem install pkg/

When it could simply be:

1
2
3
4
> gem build somegem.gemspec
> gem install ./
[write code using gem]
> run code

The first scenario happens when someone uses a tool like jeweler to build their gemspec and forgets to commit the latest generated (or what should have been generated) gemspec. Jewler likes to list every single file packaged with the gem, so the gemspec must be refreshed whenever a file is added or removed. When writing a gemspec by hand one, can simply say “all the files in the lib directory are part of this gem.” You can use a handy tool like a .gitignore to prevent unwanted files from appearing in your gem.

Why is this such a big issue? Because, as Yehuda wrote, writing your own gemspec isn’t that hard. In fact, it’s kind of enjoyable, being that it’s a Ruby DSL. Ruby was designed to cause the lowest amount of pain for developers. Our Ruby code should reflect that ethos.

Text to Speech on Mac OS X

My wife is a graduate student who spends most of her time reading a ton of articles for her classes.  Once a week, she has to drive an hour and a half to Detroit for work.  One night, she was worried that she’d have enough time to work in Detroit and read all of her articles.  The proverbial light bulb went off in my head and I remembered a feature that Mac OS X has had since Jaguar (2002): Text to Speech.

Previously, I could just select a block of text, then go to the current application menu, click “Services,” then “Speak selected text.”  However, Apple seems to have removed this feature in Snow Leopard.  After a little digging, I found it again.  Here’s how to get it working:

(Note: click the images to get a full view)

Open System Preferences (either click the spotlight search magnifying glass in the top left and type “System Preferences” or find System Preferences in the Applications folder).

Click the “Keyboard” icon toward the middle/top.

Click the “Keyboard Shortcuts” tab at the top.

In the left pane, select the “Services” item.  In the right pane, scroll down and check the box next to “Add to iTunes as a Spoken Track.”

This will add an item to the “Services” submenu when you open the main application menu in any program.

For example, let’s open a page in Safari and convert the text to speech.

We’ll go to http://www.jabberwocky.com/carroll/walrus.html and convert the poem to a sound file.

In Safari, browse to http://www.jabberwocky.com/carroll/walrus.html

Highlight all of the text.  It’s OK if some images get in there, they should be ignored by the Text to Speech conversion.

In the top menu (next to the Apple) click “Safari”, then click “Services” and then click “Add to iTunes as a Spoken Track.”

You should see a spinning cog in your top menu bar as Mac OS creates an audio file for you and putting it in iTunes. Click the spinning cog if you want to cancel.

When the cog disappears, there should be a new playlist in iTunes called “Spoken Text.”  In it, you’ll see a track called “Text to Speech.”  You can rename this to “The Walrus and the Carpenter” and copy the track to your iPod.

And that’s it!  Enjoy your free audio books!

Bonus:

If you’d like to change the voice of the computer’s reader, there are six standard voices you can choose from.  Simply go to System Preferences, click “Speech”, then click the “Text to Speech” tab, then select the voice you want from the “System voice” drop-down list.  The voice you select here will be used the next time you use the “Add to iTunes as a Spoken Track” service menu item.