Tests for your tests: Why you should be unit testing your automation frameworks5 min read

Before I get hammered by the checking vs testing advocates, please let me caveat this post by saying that this entire post is about automated checking, but for readability purposes I may from time to time use the words test or testing when I actually mean check or checking. If you’re offended by this, please feel free to mentally substitute the words as you read. Anyway, back to my story…

Over the past year I’ve been building a number of standalone test utilities, designed to centralise some common features I encountered when reviewing different team’s automation frameworks. These utilities exist in their own independent Git repositories and are shared as NuGet packages. As an example, one framework deals with CRUD operations on test users, allowing consumers to quickly and easily create test user accounts with randomised test data, as well as being able to tear them down afterwards.

After distributing v1 of the first package I opened up the repository to everyone in the organisation and encouraged them to contribute new features and bug fixes. After a few weeks, teams from across the department had begun consuming the package and much to my amazement some had even contributed back to the code base. I was delighted, not only were lots of people benefiting from my work, but they were also helping to make the package better for themselves and everyone else – “amazing, cross-team collaboration” I thought to myself.

Fast forward a few months, another tester suggests a new feature for the package. Later that week I pull down the latest changes and get to work. Much to my dismay I find a web of poorly written code, hard-coded values and newly introduced bugs – “Argh, but this was perfect when I left it!”.

I shouldn’t have been surprised, but I was. I’d left the door open and the keys in the ignition yet I was still shocked when I found that someone had wrapped it round a tree!

Drastic times called for drastic measures.

Quickly I got to work adding acceptance tests and hooking them up to the CI build in TeamCity. Testing a test framework seemed overkill, but they provided a way of regression testing the main features of the package after myself or other made changes and would help prevent others from introducing breaking changes. This worked well and things started to improve. Adding acceptance tests turned out to be a good idea and because they were written in Gherkin syntax they also had the helpful side-effects of documenting the features of the package and providing sample implementation code for others to follow when consuming it. WIN!

However, I was still facing the challenge of people failing to follow the kind of good engineering practices I had tried to employ while designing the original framework – KISS, DRY, YAGNI, SOLID etc.

Clearly I was missing something… unit tests.

I began adding unit tests to the public methods and refactoring as I went. At first I thought this would be a long winded and painful exercise, but actually, thanks to the suite of acceptance tests I was able to perform some aggressive refactors relatively quickly and could run the ATs to check I hadn’t broken the overall behaviour.

Today this framework, and the others that have followed, are in regular use by lots of team, coding standards remain high and bugs are rare. I still keep a keen eye on the code to ensure people are adding tests as they go and I offer code reviews when more junior team members need a helping hand, but all in all things are looking good and new teams are regularly adopting the packages.

Conclusions

Having automated tests and CI for your common test utilities and frameworks can be a great thing. Not only do they help to keep the quality high and prevent new bugs from being introduced, I also found that they can have a number of other benefits:

• Code Quality – Unit testing in particular encourages good coding practices. Have you ever tried unit testing a class with lots of dependencies without using dependency injection? Trust me, you won’t get far.
• Maintainability – A great advantage of having unit tests is that they make it much faster and easier to maintain and refactor. After making your changes, kick off your build and run your tests and if everything remains green, you can be pretty confident you didn’t just screw up all the existing features.
• Documentation – People generally have better things to do than write detailed documentation for their code. Even if they do get time, keeping docs up to date on a fast-moving open-source project is a whole different ball game. However, acceptance tests written in business language can provide decent descriptions of the main features of your code, therefore reducing the need for lengthy documentation. What’s more, your ATs can also serve as example code for people looking to consume the package.
• Learning – Having a separate test framework that makes use of all the good engineering practices that you use in your production code base can be great for the quality and maintainability of the code, but can also gives less experienced developers and testers a safer environment to practice coding and increase their confidence.
• Experimentation – Ever wanted to try out a new library or feature but didn’t want to risk screwing up the production code? Having a like-production test code base can be the perfect playpen to try things out with less impact if things don’t go to plan. As an example we were recently discussing different branching strategies (we currently use GitFlow but found it to be somewhat over complicated for some projects) so we experimented with a different branching strategy on a test framework first to try it out.
• Collaboration – In an environment of microservices and “platform teams”, developers and testers often seem happy in their little worlds and tend to forget that the guys and girls on the next bank of desks may have already solved the problem you’ve been trying to crack for the last 2 days. Shared utilities and frameworks allows disparate teams to benefit from each other’s hard work, while also giving them fresh opportunities to collaborate.

So why not give it a try, add some unit tests to your common test utilities project and see what benefits they bring.