fhwang.net

When to mock

Dan Fox on mocking:

... I think it’s a good idea to stub out a local instance of a software service like solr, since maintaining a test environment version of processes outside the rails environment can be time consuming, even if they are on the same machine. In this case it’s worth it to mock out such a separate software piece, even though it means possibly incomplete test coverage.

My experience has been that any time I’ve tried to mock away some other piece of the stack—SQL database, Redis, etc—I end up eventually tearing away that abstraction. There’s a definite cost in terms of setup, but what you really don’t want is to mock out the behavior of some component and then realize you didn’t understand how it works in certain scenarios.

Another way to ask the question is this: Are there components of your stack that are 1) so powerful they’re worth the trouble of including them in your production infrastructure, and yet 2) so simplistic in their behavior that you could replicate how they behave in a few lines of Ruby? I can’t think of much that satisfies that condition.

The other reason to resort to mocks is if your test suite takes too long to run. This should only be used with the most egregious test runtime offenders, however. It’s always best to run full-stack tests (including the database) whenever possible; fewer bugs will slip through the cracks.

I’d state the case more strongly: There are many ways you can optimize your test suite that do not run the risk of introducing errors into your tests the that way mocks do. Nick Gauthier gave a great talk at last year’s GoRuCo on this very subject. Definitely worth a watch if your test suite gets slow. And your project is non-trivial, and you’re testing it right, it’s gonna get slow.

It’s interesting to think about what I said more than two years ago about mocking. I’ve actually become a little looser about when to mock, and now in the Profitably code base I occasionally mock in between model classes. This is primarily because there are times that test setup can be extremely cumbersome because of the underlying data setup, and maybe I just want to test an edge case.

But the thing to keep in mind when mocking is that every time you do it, you are strengthening the contract of that method name, the arguments it takes, the class it belongs to, etc., etc. Some people might say, “yeah, what’s the big deal, all code is sort of a contract”, but I think that misses the point: Not all contracts are created equal. Not all method names are created equal. Some have been around for a while, and they’ve settled nicely, so sure, wrap that up in a bunch of mock declarations if that makes your testing easy. But some are brand-new, and you don’t totally understand the domain, or what new features will be added in the next month. Wrapping mocks around that is only going to slow you down when—not if, but when—you have to change it.

As one extreme way to illustrate the difference between contracts: In Rails and elsewhere, we simulate controller hits by, at some level, simulating HTTP. But it’s not actually low-level HTTP—there’s no mucking around with TCP/IP or whatnot. We just run through Rack to simulate some request that gets routed through Rails so we can test how various controllers will act. This is fine, because HTTP is solid and boring (boring is a virtue in engineering) and unlikely to change any time soon. So maybe your entire test suite doesn’t have a single code path that simulates HTTP on the level of network traffic, and that’s great.

But if you’ve just written a big ball of data loading code, and you’re dealing with a lot of crazy edge cases in the data, and there might be a lot more edge cases coming tomorrow, you don’t know what your code is, and you might not have figured out the best way to even structure the method call. Probably best to avoid mocking that.

blog comments powered by Disqus
Tagged: testing, ruby

« Previous post

Next post »