Test Pyramid In Practice

Nowadays, we want to deliver new software faster and more frequently, typically with processes like Continuous Integration or Delivery in place. However, while a faster development has its benefits, a common bottleneck of these approaches is the need for automated UI tests.
We have an enormous amount of these and, consequently, we end up waiting as much as a few hours for test results. Even then, we often find a lot of false negative results, so we spend even more time to analyse and confirm that it was the test’s fault, not the application’s.

In this article, I want to guide you and explain how we can apply this test pyramid approach on an example application

What is Test Pyramid?

The basic idea of a test pyramid approach is to have a proper balance of automated tests on different layers. Tests written on the UI level are most fragile and slow. Because of this, we should have only few of them.

The idea of test pyramid approach is to move as many tests as possible to lower layers. If we have some validations or calculations done on the backend (BE) side, we should move these tests to the service layer. Likewise, of course, for proper software health and maintenance, we should have a solid base of unit tests that are very fast and easy to write/maintain.

Why are testers often fail to implement the pyramid?

When it comes to testing teams and the wider industry, we’ve all heard the idea of test pyramids and agree with the concept, because it just seems right to us.

Yet, very often, we still have a problem implementing it.

How come?

Instead of a pyramid, we often end up with something that can be compared to test silos or an ice cream cone. We test everything and we test it everywhere; consequently, we have tonnes of unit, integration and UI tests.

Why? Because most testers started their experience with automated tests on the UI layer. We’ve seen how tests were executed and that it ‘clicks’ through the application as normal user does. We’ve seen that such tests are good in finding regression bugs and we trust them.

On the other hand, many of us didn’t have much experience with tests on lower layers and, even if we had, it wasn’t a good experience or the results of such tests were invisible for us testers. With the idea of a test pyramid, we should resign from what gives us a sense of security for tests on lower layers. And, in the beginning, it is hard!

In this article, I want to guide you and explain how we can apply this test pyramid approach on an example application. Let’s have a look at how it looks and works.

Test Pyramid Example

Before we begin, let’s explore the dummy application that will be the star of this example:

The application consists of 2 pages: a list of users and an “add new user” page.

From the backend point of view, we have only 2 endpoints: one for creating a new user and one for retrieving the list of users.

A typical approach for such application would be to write automated tests on the UI level, with tools like Selenium WebDriver to cover scenarios for creating new users and checking the list of users page. Such tests would cover all positive and negative scenarios. Developers would write some basic unit tests that would check UsersService, which is responsible for business logic.

Is such an approach wrong? Well, in the case of such a small application, it doesn’t make a big difference where we will automate it, as running all these tests will take us just 5 minutes. But when our application will grow, we will have much more tests in the future. This is real business concern that many have to address – and now is the perfect time to plan how we want to approach this upcoming issue.

So, let’s imagine a situation where we want to change something in our backend service and release it quickly. With approach mentioned earlier, we would need to run all our automated tests to be sure we didn’t break anything. The next question is “should we run them all?”

If we run only unit tests, would that be enough? We all know that the answer is no. Our change can be potentially backward incompatible and we may break the whole application by having our unit tests green at the same time. This is why we need to run our end-to-end (e2e) tests on the UI level. But these are fragile and slow, alongside being hard to write and maintain. Let’s think how we can plan our tests to be able release fast and often – for both the frontend (FE) and backend (BE) of the app.

So, how does that look in practice?

Testing Pyramid Foundation

A typical test pyramid model looks to move more tests on to the API level. Right – that’s fine for our BE, but what about our FE app? Can we also release it faster and have some other tests that will tell us about potential issues?

The answer to all of the above questions and concerns can be found in the following test pyramid, which is divided into 2 halves: one for BE and one for FE.

Testing Pyramid For Backend

First, let’s have a look at the right side of pyramid – this covers tests that need to be run to release our BE application.

Step 1. Unit tests

These should test our services, as well as the business logic inside them and, furthermore, these tests should be small and independent. Fortunately, we all agree that we need them. They are easy to write and maintain, in addition to being fast to perform.

Step 2. Component tests

The next layer of the pyramid for BE services is the component test level. Here, we would like to check whether our endpoints work and behave correctly. BE component tests, together with unit tests, will let us know if our BE service is working as expected, complete with all the information we need. At this point, it is tempting to say that, based on these tests, we can release our BE service with good conscience. However, what may still happen? We may still break our app when it goes live – and what could be the reason for this?

Either we introduced a backward incompatible change, or we changed the address of our service. But shouldn’t BE component tests catch backward incompatible change? Well, in theory – yes – but the team that worked on the respective change (that part that might be backward incompatible) might not have thought about it about. They ran the tests, some of which have failed, but since they introduced this change, they also updated the tests to reflect it. This, in short, is why having only unit & component tests is not sufficient. To help resolve such a problem, we can introduce next pyramid level – integration tests.

Step 3. Integration tests

Let’s go back to our problem – the team introduced an incompatible change and unit tests have passed, while component tests have failed, but the team decided to fix them (as the change was intentional) so all tests so far are green. What can help us here are consumer-driven contract tests. These are written by the consumer of our BE service – it can be our FE team, for example.

Such tests define how the consumer will use our BE and what their expectations are. Here, we do not test the business logic of the endpoints, as this is done on the component level, and instead we focus only on how our consumer will use these endpoints.

In our example application, we can define tests which expect that, if we call GET “api/users” endpoint, we will receive a list of users, with each user object having following information: id, name, and surname. We can define the type of expected value and, of course, the expected response status code. Pact (an example framework for consumer-driven contract tests) offers a pact broker; a server on which we can store all “pacts”, which are files that keep information about what our consumers expect from our BE service. So, when we want to run all the required tests, we can simply download and run them against our BE service. In cases where something will fail, the team cannot just simply fix the tests, as they were written by consumer app team. So, it forces us to make our change backward compatible or to communicate with the consumer team about planned changes and agree how to approach them.

At this point, we have ran unit tests, component tests, and integration tests. We know that our service is working as expected and our changes are backward compatible. Are we ready to release?

Some of us might say “yes!” and I agree we shouldn’t find any issue related with our BE on the e2e test level, but there’s always a chance that we might – especially when we are still learning how to use a test pyramid in our project.

We also might not be sure what to test on each level.

It might happen, for example, that we will not cover the application with proper tests on the lower levels and bugs will escape to higher levels. In such a situation, I recommend first reproducing the found bug with tests on the lower level (unit/component). This will help us to understand what bugs can be found by automated tests on each level. E2E tests are important and the aim of them is not to find bugs (as these should be found earlier), but to give us confidence that the application, as a whole, is working correctly. At this level, we should have as few tests as possible and those that we still have should focus on user flow.

In our example application, I see only one E2E test that can be written: user creates new user account, user is redirected to list of users and can see the created user on the users list.  You might be expecting tests for checking validation on the FE layer – they are here, but on the left side of pyramid.

So, let’s have a look now on the other side of pyramid responsible for visual side of our application.

Testing Pyramid for Frontend

Step 1. Unit tests

Once again, we start with unit tests. I will not focus much on these, as we all agree we need them.

Step 2. Component tests

Many modern FE applications are built upon Storybooks. We build our FE application from small components that are reusable; an example of such a component can be an input with validation. This is an approach that we use in our Design Systems, for example. So, component tests for the FE application will include all the tests that check these small components. Such tests should be run with each change in our Storybook.

Step 3. Integration tests

If we have the FE app separated from BE services, we can add the next level – integration tests. We can mock all our BE services and test our pages, forms, and templates easily, without worrying about a working BE. Such tests are faster to run and much more stable (as we do not rely on any test data). In our example application, we can test all possible negative scenarios on creating a user form, we can test various combination of users list (empty, many results, paging, sorting, etc.).

Again, we might think that, by having FE unit tests, component and integration tests, we are safe enough to release the FE. Are we? Well – no, not yet. As mentioned earlier, our FE might have wrong paths to the BE, so it might simply not work. It may happen that, as the FE team, we didn’t write proper contract tests and the team responsible for BE has broken the contract, as they were not aware of it. This is why, again, again need some e2e tests to be run in our pipeline.

The Top Of The Pyramid

Finally, we’ve reached the top of our pyramid, we have implemented each layer with automated tests, and we have also covered our FE & BE apps with a sufficient number of automated tests. The icing on a cake for our test pyramid are exploratory tests. This testing technique helps us to discover new ways of how our application might be used by simply testing manually our application without prepared test script or scenarios.

As a result of all this, we can catch bugs as early as possible and fix them on the fly!

Business Perspective

Broad testing is essential for any project to succeed, but an unrefined strategy will simply slow the time to market down. However, a properly implemented test pyramid, designed to support both the frontend and backend – including a helping of automation – will ensure faster release schedules without compromising on final quality.