Like countless others, I learned Ruby on Rails (and a great deal more) from Michael Hartl’s Ruby on Rails Tutorial. Though I haven’t kept up with all the changes in the most recent revisions, I know that back in 2012 things were a bit different. In particular, I learned that it was The Done Thing to eschew the testing framework that came with Rails in favor of RSpec, and to toss aside the fixtures library in favor of something called Factory Girl (since renamed Factory Bot). Now I believe Hartl has since abandoned much of that in preference to using only the things that come with Rails by default. I’m not that surprised - I recall this being a lot to learn all at once, and it felt awkward to be fighting from the outset the framework I was trying to learn - whose very ethos is the following of conventions it sets forth - before I could even grasp the nuances of why I might be doing this.

As influential as Hartl’s tutorial may have been, this change in the tide it seems has failed to make much of a splash in the Ruby community at large. I’ve only ever once collaborated on a Rails project that followed all of the conventions. Almost every project I’ve worked on professionally has been of the Rails/RSpec/Factory Bot variety. In this post I want to expose some of the assumptions underpinning these choices, and to question if these assumptions are still valid. I’ll also suggest that the Rails defaults are not merely a better choice for beginners, as Hartl indicates, but may be better for everyone all round.

I don’t want to dwell too much on the relative merits of RSpec. I think it is a fine testing framework, I’ve contributed to it, and I especially love its extensive expectations library, which I’ve also written about before. I also love Minitest. Though I’ve used it less, I am now using Minitest for all my new personal projects. I won’t say that one is better than the other. Maintainer Sam Phippen describes RSpec as a tool that “excels at testing legacy code”¹ and this bears out at least in my experience. I’d prefer to use RSpec to retrofit a test suite to a legacy codebase because of the ease with which you can reach through objects that are tightly coupled using the full flexibility that Ruby provides. I’d prefer to use Minitest for greenfield development precisely because of the resistance it provides to creating tightly coupled code in the first place.

Factory Bot, on the other hand, I think invites some scrutiny. thoughtbot sold Factory Bot entirely on the basis of addressing the Mystery Guest problem ². It trades on allowing you to write clearer tests at the expense of having to create and rollback every test fixture you need every time, starting every test with a pristine, blank database. This can be a major cause of test slowness, and it is worth pointing out that test slowness in general has been a common complaint for as long as I have been using Rails. You would think then that the Mystery Guest problem had better be a significant one for this to be a good trade off.

For anyone unfamiliar, the Mystery Guest problem is defined as follows:

The test reader is not able to see the cause and effect between fixture and verification logic because part of it is done outside the Test Method. ³

In a vanilla Rails application, this problem commonly manifests in the use of Rails’ persistent fixtures:

RSpec.describe User do
  it "has a full name" do
    expect(users(:alice).full_name).to eq("Alice Anderson")
  end
end

To the test reader, it’s not clear where the user’s full name comes from, or from what data it is derived. To contrast, with Factory Bot:

RSpec.describe User do
  it "has a full name" do
    user = create(:user, first: "Alice", last: "Anderson")

    expect(user.full_name).to eq("Alice Anderson")
  end
end

In this example it is immediately clear to the reader the relationship between the data that was passed to the fixture and the derived outcome of the full name text.

The Mystery Guest is an XUnit antipattern, and the use of factories facilitates a better XUnit pattern, namely the Four-Phase Test. The idea behind it is that every test should tell a story - one that has a beginning, a middle and an end (and sometimes an epilogue):

RSpec.describe User do
  it "can change address" do
    # setup
    user = create(:user)

    # exercise
    user.first = "Alice"
    user.last = "Anderson"

    # verify
    expect(user.full_name).to eq("Alice Anderson")

    # teardown
    user.destroy!
  end
end

Not all tests require all four phases, and can frequently omit some of the steps:

RSpec.describe User do
  it "can change address" do
    # setup
    user = create(:user, first: "Alice", last: "Anderson")

    # exercise
    # nothing to do here....

    # verify
    expect(user.full_name).to eq("Alice Anderson")

    # teardown
    # nothing to do here - the test framework takes care of tearing down fixtures
  end
end

It is only natural then that thoughtbot feel strongly that you should also follow other XUnit test patterns when you write your specs.⁴ The big problem here is that RSpec is not an XUnit-style test framework, and it is my observation that most people do not follow XUnit best practices, preferring to write highly idiomatic RSpec code instead:

RSpec.describe User do
  let(:first) { "Alice" }
  let(:last) { "Anderson" }
  subject(:user) { create(:user, first: first, last: last) }

  # 100s
  # of
  # lines
  # of
  # test
  # code

  describe "#full_name" do
    its(:full_name) { should eq("Alice Anderson") }
  end
end

As you can see, we have now thwarted our attempt at addressing the Mystery Guest problem simply by writing idiomatic RSpec code. It’s a particularly egregious example that can be mitigated to some degree but it’s clear that the Four-Phase Test pattern does not map neatly onto RSpec’s DSL. RSpec is not an XUnit test framework and it is not its goal to facilitate use of all the XUnit patterns.

If the authors of Factory Bot recommend that we discard parts of the RSpec DSL in the pursuit of more readable, XUnit-style tests, it is surprising that they did not recommend foremost the use of an XUnit test framework, such as Minitest, instead:

class TestUser < Minitest::Test
  def test_full_name
    user = create(:user, first: "Alice", last: "Anderson")

    assert_equal("Alice Anderson", user.full_name)
  end
end

This is arguably the most natural way to write this test in Minitest. Though it’s possible to muddle its readability there are far fewer opportunities to do so.

If then we find our test suites still rife with Mystery Guests in spite of using Factory Bot, we have gained nothing and only made our tests slower with all the extra overhead.

For this reason I recommend that if you want to use Factory Bot, to use it with an XUnit-style framework such as Minitest. It is much harder in my view to enforce the Four-Phase Test style in your organization when this goes against the grain of the RSpec DSL. I also recommend only using Factory Bot’s most basic features, avoiding for instance traits, which take details of the fixture setup away from the test body and abstract them into concepts defined in the factory definition - undercutting the original aim to eliminate the Mystery Guest problem.

Conversely, if you are not tied to using Factory Bot, consider using Rails’ fixtures instead. If you are already writing idiomatic RSpec and therefore most likely tolerating a certain amount of Mystery Guest pain, there is a good chance that you won’t feel any new discomfort by doing this alone. And your test suite will almost certainly be faster.

But honestly, why not Minitest and Rails’ fixtures? This is precisely the setup that Rails provides you with right from the moment you type rails new. It’s also the one that Michael Hartl teaches in his tutorial. It’ll make both learning Rails and onboarding people onto your project more accessible by reducing the number of things they have to learn just to get started. And it’s much simpler to be able to say that you follow all of the Rails conventions as you point to its documentation, than to have to explain all the individual choices you made when departing from the defaults. You’ll have faster tests, with less setup, because you won’t have to recreate the world for every integration test that you run. I don’t necessarily love everything about this approach. But that’s also true of other parts of Rails. And I still choose Rails because of the experience of ease I feel when I just follow the conventions that it has established. And that goes for writing tests too.

My experience right from starting with the Ruby on Rails Tutorial, through everything I’ve worked on professionally, has been that every project has had some variation of this complicated setup I learned back in 2012. If this is your experience too, I invite you to experiment in your next Rails project with the default test framework. Keep an open mind, play around with it, notice the differences, allow yourself to feel any of the pain points that come up (and perhaps any pleasant surprises) for yourself before evaluating. If you find yourself still preferring your bespoke setup, you may walk away with a richer understanding of the problems those tools were trying to address, and maybe you’ll write better tests for it. Or, like me, you may decide that whatever received wisdom we had back in 2012 doesn’t suit us as well in 2021, and it’s time for a change.