Behavior-Driven Test Data

23rd of August, 2020
rails, ruby, testing

Don't discard the database state at the end of your test. Use it!
by Tom Rothe

Summary

When running behavior tests, we seed the database with a defined snapshot called fixpoint. We do run the behavior test and save the resulting database state as another fixpoint. This method allows testing complex business processes in legacy applications without having to implement fixtures/factories upfront. By building one fixpoint on top of another, we can ensure that the process chain works without any gaps. Comparing each resulting database state at the end of a test with a previously recorded state ensures that refactoring did not have unintended side effects.

The Problem

I recently worked on a large system which had grown since 2014. Getting to know the system took a while. Grasping the business domain and the implemented features was tedious and slow. There were no tests and no documentation. So I could choose choose between annoying my peers with tons of questions or playing detective for hours on end.

When I started a new feature, I would writing a few browser tests to get to know the relevant areas of the system. This strategy worked well for features early in the user flow/business process chain. So for example, writing tests for creating a user account was easy because it did not rely on prior actions. The latest ticket however involved invoicing. Invoicing required the service to be carried out and adding the respective records in the database. To carry out the service, there are million other processes to be carried out before. Long story short, I needed a lot of database setup before I could even begin to explore & write tests.

I diligently wrote factories and identified more constraints, more dependencies and more quirks of the data model – frustrating work. Anyhow, since I could not get the data quite right for the feature I wanted to test, started writing a test for a feature that came earlier in the user journey. I found myself backtracking further and further to understand the necessary data constraints for invoicing.

The Idea

After each test completed, I had gained enough knowledge to refine the database setup for a subsequent test later in the process chain. But, implementing test fixtures/factories seemed redundant. Idea! I wrote a test that produced a state in the database. This exact state could serve as the basis of the next test, but I was not using it. Instead, I resorted to mimicking the state as closely as necessary in my test setup. That seemed inefficient and superfluous. The idea for fixpoints was born. Instead of neglecting the data at the end of a test, we save it and use it for the subsequent tests.

The Advantages

After doing a quick spike, I had a working implementation that worked with Rails system tests and RSpec:

it 'registers a user' do
  visit new_user_path
  fill_in 'Name', with: 'Hans'
  click_on 'Save'

  store_fixpoint :registred_user
  # creates YAML files containing all records (/spec/fixpoints/[table_name].yml)
end

it 'posts an item' do
  restore_fixpoint :registered_user
  
  user = User.find_by(name: 'Hans')
  visit new_item_path(user)
  fill_in 'Item', with: '...'
  click_on 'Post'

  compare_fixpoint(:posted_item, ignore_columns: [:release_date], store_fixpoint_and_fail: true)
  # compares the database state with the previously saved fixpoint and
  # raises if there is a difference. when there is no previous fixpoint,
  # it writes it and fails the test (so it can be re-run)  
end

I was delighted because:

• I did not have to write any fixtures or factories
• I could read the YAML file for a table and discover which records were actually created/changed¹ by the test’s actions
• after I refactored something, my tests would raise differences in database state and point me to unintended side effects
• my peers could run the tests on their machines and compare the results to the fixpoints I had checked into git

Conclusion

Behavior-driven test data can be a helpful tool for large legacy applications without much test coverage². Instead of spending time on implementing data setup for a test, we can invest time in producing the setup via a behavior test (which has value in its own). The full comparison of database state at the end of a test alerts the developer about problems when changing complex applications. There are lots more advantages (and disadvantages) which are not covered in this post, so there is more to come.

I have not yet released the fixpoint implementation as a Ruby gem. Please drop a comment on hackernews if you are interested.