Unit, Integration and End-To-End Tests - Finding the Right Balance

Umer Mansoor Umer Mansoor Follow Jul 05, 2016 · 4 mins read

This is something I have regrettably noticed in many backend projects that I have worked on. Developers write “unit tests” that in reality are ‘end-to-end’ tests. They test the entire flow of the application from start to the end. There is no isolation of units and the notion of the unit is the whole system, along with all of its external dependencies like databases, queues, caches, and other services. For a web server project, these tests start the server, initialize a HTTP client, make a HTTP request and check the response to make sure it has all the expected information. If so, the test is declared a success. By treating the whole system as a unit and not testing independent units in isolation and their interplay, we loose many benefits that unit and integration tests offer.

Technically speaking, these developers aren’t violating the definition or principles of unit testing. Unit testing is ill-defined. I don’t claim to be an expert, but in my humble opinion:

  • Unit testing should focus on testing small units (typically a Class or a complex algorithm).
  • Units should be tested in isolation and independent of other units. This is typically achieved by mocking the dependencies.
  • Unit tests should be fast. Usually shouldn’t take more than a few seconds to provide feedback.

Most projects benefit from having a balanced mix of various automated tests to capture different types of errors. The exact composition of the mix varies depending on the nature of the project, as we’ll see later.

End-to-end tests are good at capturing certain kinds of bugs, but their biggest drawback is that they cannot pin-point the root cause of failure. Anything in the entire flow could have contributed to the error. In large and complex systems, it’s like finding a needle in the haystack: you’ll find the root cause, but it will take time. Because unit tests focus on small modules that are tested independently, they can identify the lines of code that caused the failure with laser-sharp accuracy, which can save a lot of time.

Another nice thing about unit tests is that they always work, and they work fast. Unlike end-to-end tests that rely on external components, unit tests are not flaky. If I can build a project on my machine, I should be able to run its unit tests. In contrast, end-to-end tests would fail if some external component, like a database or a messaging queue, is not available or cannot be reached. And they can take a lvery ong time to run.

Unit tests allow developers to refactor and add new features with confidence. When I’m refactoring a complex project that has well-written unit tests, I run them often, usually after every small change. In a matter of few seconds, I know whether I broke something or not. Even better, a failing test usually prints a nice message telling me what broke: whether some GuardAssertion failed or the expected response was off by one, helps me isolate the failure.

Between unit and end-to-end tests lie integration tests. They have one major advantage over unit tests: they ensure that modules which work well in isolation, also play well together. Integration tests typically focus on a small number of modules and test their interactions.

The key is to find the right balance between unit, integration and end-to-end tests. According to Google’s Testing Blog:

To find the right balance between all three test types, the best visual aid to use is the testing pyramid. Here is a simplified version of the testing pyramid […]:

test_pyramid

The bulk of your tests are unit tests at the bottom of the pyramid. As you move up the pyramid, your tests gets larger, but at the same time the number of tests (the width of your pyramid) gets smaller.

As a good first guess, Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests. The exact mix will be different for each team, but in general, it should retain that pyramid shape. Try to avoid these anti-patterns:

  • Inverted pyramid/ice cream cone. The team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests.

  • Hourglass. The team starts with a lot of unit tests, then uses end-to-end tests where integration tests should be used. The hourglass has many unit tests at the bottom and many end-to-end tests at the top, but few integration tests in the middle.

70/20/10 split between unit, integration and end-to-end tests is a good, general rule of thumb. If a project has large number of integrations or complex interfaces, it should have more integration and end-to-end tests. A project that is primarily focused on computation or data, should have more unit tests and fewer integration tests. The right mix depends on the nature of the project but the key is to retain the pyramid shape of the testing pyramid, that is, Unit > Integration > End-to-End Tests.

#java #testing

You May Also Enjoy


If you like this post, please share using the buttons above. It will help CodeAhoy grow and add new content. Thank you!


Comments (6)


Kamil

You can divide app to 3 layers: DB, Logic (backend with e.g. RESTful API), UI (frontend). And you can provide chip e2e tests for RESTful API wich allow you to make refactor without any test changes :)


Anthony Jackman

You are awesome. I can recite the definition of each level of test. But what I could never find is how that relates to programming. Thank you for putting a little perspective on things.


Michael Freidgeim

In the article “Write tests. Not too many. Mostly integration.” (https://kentcdodds.com/blog/write-tests) Kent C. Dodds suggests to use Trophy instead of triangle.


firstpostcommenter

Thanks for clarification


haughtycool

Hi Umer Mansoor.
The balance between Unit, Integration and End-To-End Tests always make me confused. I try to write unit test as much as I can so that it cover almost case. However, I don’t know how to write the integration test and end to end test without unit test duplication. Could you recommend me a book or an article?

Second, What is the order of those tests? Which should I write first if I write code before test and vice versa?


Ryan Gomez

Here’s my take on the 3 on an MVC pattern:
- Unit-test = mostly on models, focusing on business logic, or rather expected value of the property on a class, mocking any dependencies on the specific item you are testing
- Integration Test = Starts on the controller, testing the behaviour including dependencies(DB integration). You usually revert any changes made on the data after running the test
- End-to-end = this tests the whole application iself. Usually starts on the front-end, using tools like selenium for web, or jmeter for api testing. This is typically performed by QA engineers rather than the developers


Speak Your Mind