Posts Tagged ‘double-blind testing’

Double-Blind Test-Driven Development in Rails 3: Part 3

February 2, 2011

  1. Simple Tests
  2. Double-Blind Tests
  3. Making it Practical with RSpec Matchers

This is the last article in this series describing the concept of double-blind test-driven development. This style of testing can add time to development, but this can be cut significantly using RSpec matchers.

If you’re not familiar with matchers, they’re the helpers that give RSpec its english-like syntax, and they can be a powerful tool speeding up all of your test-driven development – whether you follow the double-blind method or not.

If you’re using RSpec, you’re already using their built-in matchers. Say we have a Site model, and its url method takes the host attribute and appends the ‘http://’ protocol. Here’s a likely test:

describe Site, 'url'
  it "should begin with http://" do
    site = Site.new :host => 'example.com'
    site.url.should equal('http://example.com')
  end
end

The equal() method in the code above is the matcher. You can pass it to any of RSpec’s should or should_not methods, and it will magically work.

But the magic isn’t that hard, and you can harness it yourself for custom matchers that conform to your application.

The Many Faces of Custom RSpec Matchers

While I don’t want this article to turn into a primer on custom RSpec matchers (it’s a little off-topic), I’ll give you the three styles of defining them, and explain my recommendations. There are simple matchers, the Matcher DSL, and full RSpec matcher classes.

Let’s start by writing a test we want to run:

it "should be at least 5" do
  6.should be_at_least(5)
end

This test should always pass, provided we’ve defined our matcher correctly. The first way to do this is the simple matcher:

def be_at_least(minimum)
  simple_matcher("at least #{minimum}"){|actual| actual >= minimum}
end

As you might guess, actual represents the object that “.should” whatever – in this case “.should be_at_least(5)”. This version makes a lot of assumptions, including the auto-creation of generic pass and fail messages.

If you want a little more control, you can step up to RSpec’s Matcher DSL. This is the middle-of-the-road option for creating custom matchers:

RSpec::Matchers.define :be_at_least do |minimum|
  match do |actual|
    actual >= minimum
  end

  failure_message_for_should do |actual|
    "expected #{actual} to be at least #{minimum}"
  end

  failure_message_for_should_not do |actual|
    "expected #{actual} to be less than #{minimum}"
  end

  description do
    "be at least #{minimum}"
  end
end

Now we’re rocking custom failure messages, and test names. This is pretty cool, and honestly how I started out doing matchers. It’s also how I started out doing the matchers for double-blind testing.

The problem is that by skipping the creation of actual matcher classes, we lose the ability to do things like inheritance. Not a big deal if our matchers stay simple, but they won’t. Not if we use them as often as we should! I found myself re-defining the same helper methods in each matcher I defined this way.

So let’s see just how daunting a full-fledged custom matcher class really is:

module CustomMatcher  
  class BeAtLeast
    def initialize(minimum)  
      @minimum = minimum
    end  
  
    def matches?(actual)  
      @actual = actual
      @actual >= @minimum
    end  
  
    def failure_message_for_should  
      "expected #{@actual} to be at least #{@minimum}"  
    end  
  
    def failure_message_for_should_not  
      "expected #{@actual} to be less than #{@minimum}"  
    end  
  end  
  
  def be_at_least(expected)  
    BeAtLeast.new(expected)  
  end  
end  

This isn’t so bad! We’re defining a new class, but you can see it doesn’t have to inherit from anything, or use any unholy Ruby voodoo to work.

We just have to define four methods: initialize, match? (which returns true or false), and the two failure message methods. Along the way, we set some instance variables so we can access the data when we need it. Finally, we define a method that creates a new instance of this class, and that’s what RSpec will rely on.

You can add as many other methods as these four will rely on. But you also get other benefits over the DSL. You can use inheritance, moving common methods up the chain so you only have to define them once, instead of in each matcher definition. You can also write setup/teardown code in your parent classes, make default arguments a breeze, and standardize any error handling. I do all of these in the matchers I created for the example app.

The bottom line is this: defining your own matcher classes directly really DRY’s up your matchers, and that always makes life simpler. I think it’s the only way to go for serious and heavy RSpec users. It allows the class for my validate_presence_of matcher to be this short and sweet:

module DoubleBlindMatchers
  class ValidatePresenceOf < ValidationMatcher
    def default_options
      {:message => "can't be blank", :with => 'x'}
    end

    def match
      set_to @options[:with]
      @object.valid?
      check !@object.errors[@attribute].include?(@options[:message]), shouldnt_exist
      
      set_to nil
      check !@object.valid?, valid_when('nil')
      check @object.errors[@attribute].include?(@options[:message])
      
      set_to ""
      check !@object.valid?, valid_when("blank")
      check @object.errors[@attribute].include?(@options[:message])
    end
  end
  
  def validate_presence_of expected, options = {}
    ValidatePresenceOf.new expected, options
  end
end

And the Teacher model, which grew considerably during our double-blind testing, now looks like this (in its entirety):

# spec/models/teacher_spec.rb

require 'spec_helper'

describe Teacher do
  it {should have_many :subjects}
  
  it {should validate_presence_of :name}
  it {should validate_length_of :name, :maximum => 50, :message => "must be 50 characters or less"}
  
  it {should validate_presence_of :salary}
  it {should validate_numericality_of :salary, :within => (20_000..100_000), :message => "must be between $20K and $100K"}
end

Summary

Now that you’ve seen my entire proposal for double-blind testing, let me know what you think. Be cruel if you must, it’s the only way I’ll learn. I’ll do the best to explain (not defend) my reasoning, and keep an open mind to changes.

I’ll also be publishing my double-blind matchers as a gem so you can add them to your project.

Advertisements

Double-Blind Test-Driven Development in Rails 3: Part 2

February 1, 2011

  1. Simple Tests
  2. Double-Blind Tests
  3. Making it Practical with RSpec Matchers

The last article in this series defined the concept of double-blind test-driven development, but didn’t get much into real-world examples. In this article, we’ll explore several such examples.

The Example Application

This article includes a sample app that you can download using the link above. Be sure to checkout tag “double_blind_tests” to see the code as it appears in this article. The next article will have a lot of refactoring. I limited my samples to the model layer, where 100% coverage is a very realistic goal, and this is likely to be the greatest benefit.

I chose a simple high school scheduling app with teachers, the subjects they teach, students, and courses. In this case, I’m defining a course as a student’s participation in a subject. Teachers teach (ie, have) many subjects. Students take (have) many subjects, via courses. The course record contains that student’s grade for the given subject.

The database constraints are intentionally strict, and most of the validations in the models ensure that these constraints are respected in the application layer. We don’t want the user seeing an error page because of bad data. Depending on the application, that can be worse than actually having bad data creep in.

Associations

Here’s an example of a has_many association:

# excerpt from spec/models/teacher_spec.rb

describe Teacher do
  it "has many subjects" do
    teacher = Factory.create :teacher
    teacher.subjects.should be_empty

    subject = teacher.subjects.create Factory.attributes_for(:subject)
    teacher.subjects.should include(subject)
  end
end

In order to factor out our own assumptions, we have to ask what they are. The assumption is that the subject we add to the teacher’s subject list works because of the has_many relationship. So we’ll first test that teacher.subjects is, in fact, empty when we assume it would be. Then we’re free to test that adding a subject works as we expect.

Here’s a belongs_to association:

# excerpt from spec/models/subject_spec.rb

describe Subject do
  it "belongs_to a teacher" do
    teacher = Factory.create :teacher

    subject = Subject.new
    subject.teacher.should be_nil
    
    subject.teacher = teacher
    subject.teacher.should == teacher
  end
end

Again, we’re challenging the assumption that the association is nil by default, by testing against it before verifying that we can add a teacher. This tests that this is a true belongs_to association, and not simply an instance method. This is the kind of thing that can and will change over the life of an application.

Validations

Let’s test validates_presence_of:

# excerpt from spec/models/teacher_spec.rb

describe Teacher do
  describe "name" do
    it "is present" do
      error_message = "can't be blank"
      
      teacher = Teacher.new :name => 'Joe Example'
      teacher.valid?
      teacher.errors[:name].should_not include(error_message)

      teacher.name = nil
      teacher.should_not be_valid
      teacher.errors[:name].should include(error_message)

      teacher.name = ''
      teacher.should_not be_valid
      teacher.errors[:name].should include(error_message)
    end
  end
end

This example was actually explained in detail in the last article. Validate that the error doesn’t already exist before trying to trigger it. Don’t just test the default value when you create a blank object, test the likely possibilities. Refactor the error message to DRY up the test and add readability. And finally, test by modifying the object you already created (as little as possible) rather than creating a new object from scratch for each part of the test.

A more complex version is needed to validate the presence of an association:

# excerpt from spec/models/subject_spec.rb

describe Subject do
  describe "teacher" do
    it "is present" do
      error_message = "can't be blank"

      teacher = Factory.create(:teacher)
      subject = Factory.create(:subject, :teacher => teacher)
      subject.valid?
      subject.errors[:teacher].should_not include(error_message)
    
      subject.teacher = nil
      subject.should_not be_valid
      subject.errors[:teacher].should include(error_message)
    end
  end
end

While the test is more complex, the code to satisfy it is not:

# excerpt from app/models/subject.rb

validates_presence_of :teacher

testing validates_length_of:

# excerpt from spec/models/teacher_spec.rb

describe Teacher do
  describe "name" do
    it "is at most 50 characters" do
      error_message = "must be 50 characters or less"
      
      teacher = Teacher.new :name => 'x' * 50
      teacher.valid?
      teacher.errors[:name].should_not include(error_message)
      
      teacher.name += 'x'
      teacher.should_not be_valid
      teacher.errors[:name].should include(error_message)
    end
  end
end

And here’s the model code that satisfies the test:

# excerpt from app/models/teacher.rb

validates_length_of :name, :maximum => 50, :message => "must be 50 characters or less"

While you can definitely start to see a pattern in validation testing, this introduces a new element. Instead of freshly setting the name attribute to be 51 characters long, we test the valid edge case first and then add *just* enough to make it invalid – one more character.

This does two things: it verifies that our edge case was as “edgy” as it could be, and it makes our test less brittle. If we wanted to change the test to allow up to 100 characters, we’d only have to modify the test name and the initial set value.

validating a number’s range using validates_numericality_of:

# excerpt from spec/models/teacher_spec.rb

describe Teacher do
  describe "salary" do
    it "is at or above $20K" do
      error_message = "must be between $20K and $100K"
      
      teacher = Teacher.new :salary => 20_000
      teacher.valid?
      teacher.errors[:salary].should_not include(error_message)

      teacher.salary -= 0.01
      teacher.should_not be_valid
      teacher.errors[:salary].should include(error_message)
    end

    it "is no more than $100K" do
      error_message = "must be between $20K and $100K"

      teacher = Teacher.new :salary => 100_000
      teacher.valid?
      teacher.errors[:salary].should_not include(error_message)
      
      teacher.salary += 0.01
      teacher.should_not be_valid
      teacher.errors[:salary].should include(error_message)
    end
  end
end

And here’s the code that satisfies the test:

# excerpt from app/models/teacher.rb

validates_numericality_of :salary, :message => "must be between $20K and $100K",
  :greater_than_or_equal_to => 20_000, :less_than_or_equal_to => 100_000

We’re doing the same here as in our testing of name’s length. We’re setting the edge value that’s *just* within the allowed range, then adding or subtracting a penny to make it invalid. I split up the top and bottom edge tests, because it’s better to test as atomically as possible – one limit per test.

Defaults

Another tricky database constraint to test for is a default value:

# excerpt from spec/models/course_spec.rb

describe Course do
  describe "grade_percentage" do
    it "defaults to 1.0" do
      course = Course.new :grade_percentage => nil
      course.grade_percentage.should be_nil
      
      course = Course.new :grade_percentage => ''
      course.grade_percentage.should be_blank
      
      course = Course.new :grade_percentage => 0.95
      course.grade_percentage.should == 0.95
      
      course = Course.new
      course.grade_percentage.should == 1.0
    end
  end
end

In this case, we can’t avoid having to recreate the model from scratch, because the nature of the implementation. There’s no actual code in the model that makes this happen, it’s purely in the database schema. Why should we test it, then? Because we test any behavior we’re going to rely on in the application. The fact that this model behavior is implemented at the database level (and therefore, not purely TDD) is a small inconvenience.

What’s the assumption our double-blind test is verifying in this case? That the value is only set in the absence of other values being explicitly assigned. Testing with nil and blank values verifies that the default doesn’t override them – it only works in the complete absence of any assignment. I also test an arbitrary (but valid) value as the anti-assumption test before finally verifying that the default is setting to the correct value.

Most default tests verify only that the correct default value is set – the double-blind version verifies that it’s acting only as a default value in all cases.

Summary

The point of double-blind testing is bullet-proof tests, that can’t be reasonably thwarted by antagonistic coding – whether that’s your anti-social pairing partner, or yourself several months down the road. The bottom line is this: test all assumptions.

That being said, this is very time consuming, and we can see a ton of repetition even in this small test suite. What we need is a way to get back to speedy testing before our boss/client notices it now takes an hour to implement one validation.*

*Even if you work for a government owned/regulated institution that actually digs that kind of non-agile perversion, you WILL eventually go insane. Even in this small sample app, the voices in my head had to talk me off a building ledge twice.

The answer lies in RSpec matchers, which are easy to implement, and can grow with your application. The benefit is not just speedier development – it’s also consistency across your application. We’ll explore that in the last article of this series.