8th Light Blog

There’s a common pattern I’ve seen for developing DSLs (Domain Specific Language) in Ruby. It’s used in RSpec, the Statemachine Gem, and Unclebob’s Clean Code talk at RailsConf 2007. I haven’t seen a name for this pattern so I’ll call it the DSL Block Pattern.

RSpec

describe "Bowling Game" do
    it "should score 0 on a gutter game" do
        game = Game.new
        20.times { game.roll(0) }
        game.score.should eql(0)
    end
end

Statemachine

sm = Statemachine.build do
    trans :locked, :coin, :unlocked
    trans :locked, :pass, :locked
    trans :unlocked, :pass, :locked
    trans :unlocked, :coin, :unlocked
end

Parser

parser = Args.expect do
    boolean "l"
    number "p"
    string "d"
end

Here’s the problem. You’ve got to write code for specific domain such as writing specifications (RSpec), defining a Statemachine, or defining command line arguments (Unclebob’s Clean Code talk). These domains have a contained and well defined terminology set. Often the cleanest, most elegant way to express this code is to create a DSL.

Before diving into the example, let me say that I like coffee as much as the next guy. But I feel lost when ever I go to a Starbucks. As you know, Starbucks has a it’s own language, DSL if you will, for ordering coffee. What follows is a DSL Block for ordering Starbucks coffee.

The general grammar for ordering coffee is: Size, Adjective (optional), Type of Coffee. This is by no means comprehensive but it’s sufficient for the example. So if you wanted to order a large coffee, for example, you would say, Grande Coffee. A small espresso: Short Americano. An extra large mixture of regular and decaffeinated coffee with some half and half: Venti Breve Half Caff.

Given the task to code these coffee orders, I’d like to be able to code it like this:

 Starbucks.order do
    grande.coffee
    short.americano
    venti.breve.half_caff
end

Ok that looks good, but as you look closely, you’ll start to wonder about those methods, grande, short, and venti “Do they have to be defined on the Kernel?” you may ask. Defining them on the Kernel is a scary prospect. And that may convince you to clutter the syntax by passing an object into the block like this:

Starbucks.order do |order|
    order.grande.coffee
    order.short.americano
    order.venti.breve.half_caff
end

This would allow you to define the grande, short, and venti methods on the object passed into the block. Although you do need an object where grande, short, and venti will be defined, you don’t need to add an argument to the block. You’ll find code out there, such as Migrations, that uses this less optimal route. It’s not necessary. The trick to get rid of the argument is below:

module Starbucks

  def self.order(&block)
    order = Order.new
    order.instance_eval(&block)
    return order.drinks
  end

  class Order

    attr_reader :drinks

    def initialize
      @drinks = []
    end

    def short
      @size = "small"
      return self
    end

    def grande
      @size = "large"
      return self
    end

    def venti
      @size = "extra large"
      return self
    end

    def coffee
      @drink = "coffee"
      build_drink
    end

    def half_caff
      @drink = "regular and decaffeinated coffee mixed together"
      build_drink
    end

    def americano
      @drink = "espresso"
      build_drink
    end

    def breve
      @adjective = "with half and half"
      return self
    end

    private

    def build_drink
      drink = "#{@size} cup of #{@drink}"
      drink << " #{@adjective}" if @adjective
      @drinks << drink

      @size = @drink = @adjective = nil
    end
  end

end

You can see that the Order object is doing all the work. It’s got the responsibility of interpreting the DSL, so let’s call it the Interpreter Object. The Module::order method simply creates an instance of Order and calls istance_eval on it. This causes the block to execute using the binding of the Order instance. All of the methods on Order will be accessible to the block.

The Interpreter Object can do any number of things as it interprets the DSL. In this case it simply generates a translation for Starbucks newbies. But, the sky’s the limit really.

Show all the source code.

    module Starbucks

      def self.order(&block)
        order = Order.new
        order.instance_eval(&block)
        return order.drinks
      end

      class Order

        attr_reader :drinks

        def initialize
          @drinks = []
        end

        def short
          @size = "small"
          return self
        end

        def tall
          @size = "medium"
          return self
        end

        def grande
          @size = "large"
          return self
        end

        def venti
          @size = "extra large"
          return self
        end

        def coffee
          @drink = "coffee"
          build_drink
        end

        def decaf
          @drink = "decaffeinated coffee"
          build_drink
        end

        def half_caff
          @drink = "regular and decaffeinated coffee mixed together"
          build_drink
        end

        def americano
          @drink = "espresso"
          build_drink
        end

        def mocha
          @drink = "espresso mixed with chocolate, milk, and topped with whipped cream"
          build_drink
        end

        def cappuchino
          @drink = "espresso mixed with milk and topped with steamed milk"
          build_drink
        end

        def no_whip
          @adjective = "without the whipped cream"
          return self
        end

        def non_fat
          @adjective = "with non-fat milk"
          return self
        end

        def breve
          @adjective = "with half and half"
          return self
        end

        def extra_shot
          @adjective = "with an extra shot of espresso"
          return self
        end

        def soy
          @adjective = "with soy milk"
          return self
        end

        def extra_hot
          @adjective = "with super heated milk"
          return self
        end


        private

        def build_drink
          drink = "#{@size} cup of #{@drink}"
          drink << " #{@adjective}" if @adjective
          @drinks << drink

          @size = @drink = @adjective = nil
        end
      end

    end

    require File.expand_path(File.dirname(__FILE__) + "/starbucks")

    #### Specs ####

    describe "Starbucks" do

      it "should handle empty orders" do
        Starbucks.order {}.should eql([])
      end

      it "should handle regular coffee with all the sizes" do
        Starbucks.order do
          short.coffee
          tall.coffee
          grande.coffee
          venti.coffee
        end.should eql(["small cup of coffee",
                        "medium cup of coffee",
                        "large cup of coffee",
                        "extra large cup of coffee"])
      end


      it "should deal with types of drinks" do
        Starbucks.order do
          short.decaf
          short.half_caff
          short.americano
          short.mocha
          short.cappuchino
        end.should eql(["small cup of decaffeinated coffee",
                        "small cup of regular and decaffeinated coffee mixed together",
                        "small cup of espresso",
                        "small cup of espresso mixed with chocolate, milk, and topped with whipped cream",
                        "small cup of espresso mixed with milk and topped with steamed milk"])
      end

      it "should handle the adjectives" do
        Starbucks.order do
          short.no_whip.coffee
          short.non_fat.coffee
          short.breve.coffee
          short.extra_shot.coffee
          short.soy.coffee
          short.extra_hot.coffee
        end.should eql(["small cup of coffee without the whipped cream",
                        "small cup of coffee with non-fat milk",
                        "small cup of coffee with half and half",
                        "small cup of coffee with an extra shot of espresso",
                        "small cup of coffee with soy milk",
                        "small cup of coffee with super heated milk",])
      end

    end 

There’s a common pattern I’ve seen for developing DSLs (Domain Specific Language) in Ruby. It’s used in RSpec, the Statemachine Gem, and Unclebob’s Clean Code talk at RailsConf 2007. I haven’t seen a name for this pattern so I’ll call it the DSL Block Pattern.

RSpec

describe "Bowling Game" do
    it "should score 0 on a gutter game" do
        game = Game.new
        20.times { game.roll(0) }
        game.score.should eql(0)
    end
end

Statemachine

sm = Statemachine.build do
    trans :locked, :coin, :unlocked
    trans :locked, :pass, :locked
    trans :unlocked, :pass, :locked
    trans :unlocked, :coin, :unlocked
end

Parser

parser = Args.expect do
    boolean "l"
    number "p"
    string "d"
end

Here’s the problem. You’ve got to write code for specific domain such as writing specifications (RSpec), defining a Statemachine, or defining command line arguments (Unclebob’s Clean Code talk). These domains have a contained and well defined terminology set. Often the cleanest, most elegant way to express this code is to create a DSL.

Before diving into the example, let me say that I like coffee as much as the next guy. But I feel lost when ever I go to a Starbucks. As you know, Starbucks has a it’s own language, DSL if you will, for ordering coffee. What follows is a DSL Block for ordering Starbucks coffee.

The general grammar for ordering coffee is: Size, Adjective (optional), Type of Coffee. This is by no means comprehensive but it’s sufficient for the example. So if you wanted to order a large coffee, for example, you would say, Grande Coffee. A small espresso: Short Americano. An extra large mixture of regular and decaffeinated coffee with some half and half: Venti Breve Half Caff.

Given the task to code these coffee orders, I’d like to be able to code it like this:

 Starbucks.order do
    grande.coffee
    short.americano
    venti.breve.half_caff
end

Ok that looks good, but as you look closely, you’ll start to wonder about those methods, grande, short, and venti “Do they have to be defined on the Kernel?” you may ask. Defining them on the Kernel is a scary prospect. And that may convince you to clutter the syntax by passing an object into the block like this:

Starbucks.order do |order|
    order.grande.coffee
    order.short.americano
    order.venti.breve.half_caff
end

This would allow you to define the grande, short, and venti methods on the object passed into the block. Although you do need an object where grande, short, and venti will be defined, you don’t need to add an argument to the block. You’ll find code out there, such as Migrations, that uses this less optimal route. It’s not necessary. The trick to get rid of the argument is below:

module Starbucks

  def self.order(&block)
    order = Order.new
    order.instance_eval(&block)
    return order.drinks
  end

  class Order

    attr_reader :drinks

    def initialize
      @drinks = []
    end

    def short
      @size = "small"
      return self
    end

    def grande
      @size = "large"
      return self
    end

    def venti
      @size = "extra large"
      return self
    end

    def coffee
      @drink = "coffee"
      build_drink
    end

    def half_caff
      @drink = "regular and decaffeinated coffee mixed together"
      build_drink
    end

    def americano
      @drink = "espresso"
      build_drink
    end

    def breve
      @adjective = "with half and half"
      return self
    end

    private

    def build_drink
      drink = "#{@size} cup of #{@drink}"
      drink << " #{@adjective}" if @adjective
      @drinks << drink

      @size = @drink = @adjective = nil
    end
  end

end

You can see that the Order object is doing all the work. It’s got the responsibility of interpreting the DSL, so let’s call it the Interpreter Object. The Module::order method simply creates an instance of Order and calls istance_eval on it. This causes the block to execute using the binding of the Order instance. All of the methods on Order will be accessible to the block.

The Interpreter Object can do any number of things as it interprets the DSL. In this case it simply generates a translation for Starbucks newbies. But, the sky’s the limit really.

Show all the source code.

    module Starbucks

      def self.order(&block)
        order = Order.new
        order.instance_eval(&block)
        return order.drinks
      end

      class Order

        attr_reader :drinks

        def initialize
          @drinks = []
        end

        def short
          @size = "small"
          return self
        end

        def tall
          @size = "medium"
          return self
        end

        def grande
          @size = "large"
          return self
        end

        def venti
          @size = "extra large"
          return self
        end

        def coffee
          @drink = "coffee"
          build_drink
        end

        def decaf
          @drink = "decaffeinated coffee"
          build_drink
        end

        def half_caff
          @drink = "regular and decaffeinated coffee mixed together"
          build_drink
        end

        def americano
          @drink = "espresso"
          build_drink
        end

        def mocha
          @drink = "espresso mixed with chocolate, milk, and topped with whipped cream"
          build_drink
        end

        def cappuchino
          @drink = "espresso mixed with milk and topped with steamed milk"
          build_drink
        end

        def no_whip
          @adjective = "without the whipped cream"
          return self
        end

        def non_fat
          @adjective = "with non-fat milk"
          return self
        end

        def breve
          @adjective = "with half and half"
          return self
        end

        def extra_shot
          @adjective = "with an extra shot of espresso"
          return self
        end

        def soy
          @adjective = "with soy milk"
          return self
        end

        def extra_hot
          @adjective = "with super heated milk"
          return self
        end


        private

        def build_drink
          drink = "#{@size} cup of #{@drink}"
          drink << " #{@adjective}" if @adjective
          @drinks << drink

          @size = @drink = @adjective = nil
        end
      end

    end

    require File.expand_path(File.dirname(__FILE__) + "/starbucks")

    #### Specs ####

    describe "Starbucks" do

      it "should handle empty orders" do
        Starbucks.order {}.should eql([])
      end

      it "should handle regular coffee with all the sizes" do
        Starbucks.order do
          short.coffee
          tall.coffee
          grande.coffee
          venti.coffee
        end.should eql(["small cup of coffee",
                        "medium cup of coffee",
                        "large cup of coffee",
                        "extra large cup of coffee"])
      end


      it "should deal with types of drinks" do
        Starbucks.order do
          short.decaf
          short.half_caff
          short.americano
          short.mocha
          short.cappuchino
        end.should eql(["small cup of decaffeinated coffee",
                        "small cup of regular and decaffeinated coffee mixed together",
                        "small cup of espresso",
                        "small cup of espresso mixed with chocolate, milk, and topped with whipped cream",
                        "small cup of espresso mixed with milk and topped with steamed milk"])
      end

      it "should handle the adjectives" do
        Starbucks.order do
          short.no_whip.coffee
          short.non_fat.coffee
          short.breve.coffee
          short.extra_shot.coffee
          short.soy.coffee
          short.extra_hot.coffee
        end.should eql(["small cup of coffee without the whipped cream",
                        "small cup of coffee with non-fat milk",
                        "small cup of coffee with half and half",
                        "small cup of coffee with an extra shot of espresso",
                        "small cup of coffee with soy milk",
                        "small cup of coffee with super heated milk",])
      end

    end 

Recently I had the pleasure and frustration of working the net-sftp gem for Ruby. Pleasure because it’s a well written library, with an easy to use syntax that looks something like this:

Net::SFTP.start( 'localhost', :registry_options => { :logs => { :levels => { "sftp.*" => :debug } } }) do |sftp|
  sftp.put_file "test.data", "temp/blah.data"
  puts "getting remote file to local location..."
  sftp.get_file "temp/blah.data", "new.data"
end

The above is just a shortened version from one of the examples in the GEM itself. It’s simple to use and easy to read. Having written similar code in C++, for Windows no less, I can really appreciate how quickly this can get an FTP application off the ground. The frustration came when I went to test drive this guy. Net::SFTP.start is a module method, not a class member. I can’t stub it in the traditional way using the RSpec stub! command or use should_receive. On top of that it passes back a block, which needs to be tested to make sure it’s being called correctly. After a few shots at mocking it out Paul and I test drove it with an actual FTP server. In the short term that was necessary anyway, as a hazard of frequent mocking can be that you are only testing how well you read the API. You see that when the tests pass and the first shot at actually running the code blows up. In the long term the customer asked for a few new small features, changing directories and what not, and I really want to get this under long-term test to do that. So how do we do it?

Well we could stop using the .start command entirely. We could pass in a mock Net::SFTP object and test it, making sure to close it manually. Unfortunately that eliminates the clean code we see above, and if possible I’d like to keep it. The solution is to intercept the start method.

The first thing I do is monkey patch the code like so:

module Net ; module SFTP
  def start( *args, &block )
  end
  module_function :start
end ; end
context "My Context" do

I’ve put it before the context, to make sure it’s redefined before the object I’m testing is created. Next we need to expose our mock objects in the context to the monkey patch. This isn’t done with traditional writers and readers, because that would require finding the specific specification running for each time through the monkey patched start. Instead we make our mocks class members in the setup method, and create a class method in the context to retrieve the variables. The class method looks like this:

def Spec.get_mock_ftp_objects
  return @@mock_starter, @@mock_session
end

This reveals a bit of the underworkings of RSpec. Each block in the context block is turned into a class method using class_eval, as part of the Spec object. Making the method static allowed this new monkey patched method:

module Net
  module SFTP
    def start( *args, &block )
      @mock_starter, @mock_session = Spec.get_mock_ftp_objects
      @mock_starter.start args[0], args[1], args[2]
      yield @mock_session
    end
    module_function :start
  end
end

The code gets the two mock objects via our new method. Isn’t it grand how Ruby lets you return multiple objects? The call to start allows me to make sure that the arguments passed to the real start are correct. The real interesting call is the yield. By yielding the mock back to the object it will replace the sftp in the original code. Now I can test it! In fact I’ve already realized a bug in my code (in stopping) just by the process of doing this. I love it when a plan comes together. The final code example is here, I ended up extracting out a new class, so the names have changed. This one tests both the starting object and the block yielded:

require 'net/sftp'
require File.expand_path(File.dirname(__FILE__) + "/ftp_client")

module Net
  module SFTP
    def start( *args, &block )
      @mock_starter, @mock_session = Spec.get_mock_ftp_objects
      @mock_starter.start args[0], args[1], args[2]
      yield @mock_session
    end
    module_function :start
  end
end

context "FTP Client" do  
  setup do
    @client = FtpClient.new("test_server", "user", "password", "directory")
    @@mock_starter = mock('mock_starter')
    @@mock_session = mock('mock_session')
    @@mock_session.stub!(:opendir)
    @@mock_session.stub!(:close_handle)
  end

  specify "makes ftp connection, to proper place" do
    @@mock_starter.should_receive(:start).with("test_server", "user", "password")
    @client.read_from_server
  end

  specify "changes to ftp_directory, better close that handle" do
    @@mock_starter.should_receive(:start).with("test_server", "user", "password")
    @@mock_session.should_receive(:opendir).with("directory").and_return("fake handle")
    @@mock_session.should_receive(:close_handle).with("fake handle")
    @client.read_from_server
  end

  def Spec.get_mock_ftp_objects
    return @@mock_starter, @@mock_session
  end
end

Maybe this isn’t the best way to do this, but I like it. I’m looking forward to comments.