Why Testing in Production is Critical to Your Development Process

Victoria Bezsmolna

September 23, 2019

In ideal world, all developers would want to believe that by the time their software solution reaches production, every bug is fixed and there was no need for further testing. However, every sane and experienced developer knows it’s not possible in the real world.

There is only one surefire way to reveal all the deficiencies in the production code, and that is testing in production. Of course, testing your product at the earlier development stages is critical, but it is not enough.

Testing in production allows you to perform an in-depth product review in a live production environment, and it is the only way to ensure that the system works as intended. Read on to learn more about testing in production and why it is critical to the development process.

What is Testing in Production

In a nutshell,

Testing in production (TiP) is carrying out continuous testing and monitoring of the production environment after the deployment.

Testing in production implies performing different tests in the live environment of the production state. It allows developers to detect issues that cannot be predicted before the product goes live. Only while testing in production, you can check to see that the app functions as expected after the deployment.

It’s essential to understand that testing in production doesn’t imply skipping testing at earlier stages. On the contrary, TiP is used to polish the product in live mode. It doesn’t encourage rolling out unverified features/updates and hoping they work when the end-user gets to them. Some DevOps engineers try to avoid testing in production, believing that it brings risks they can avoid. In reality, TiP is all about minimizing the risks and managing them effectively rather than creating new threats. The primary goal of TiP is to catch and eliminate any bugs in user experience as soon as possible.

Benefits of Testing in Production

TiP is great for monitoring the stability of the app in a live environment. Let’s have a closer look at its benefits.

Better Performance Testing

With TiP, developers can see how the app behaves in real-life when the testing scenarios are not defined in advance. This way, developers can quickly identify issues that are not so easy to detect in any other environment.

Beta Programs With Instant Feedback

Implementing testing in production in the product beta programs of your solution allows developers to receive early feedback on the newly-rolled-out features. Beta programs give users instant feedback on the user experience issues as well.

Fast Problem Detection

Some developers are hesitant to implement testing in production because they think it will interfere with the experience of real app users and threaten their data safety or stability. We argue that TiP allows developers to detect problems quickly before they reach the end-users. What’s more, TiP empowers developers to be one step ahead: find and fix bugs faster than users face and report them.

Live App Performance Monitoring

It’s a good idea to monitor app performance in real life when real users change data and interact with the app. This way, developers can analyze app performance while having full data at their disposal.

Running Unpredictable Scenarios

No matter how hard you try, you won’t be able to run every possible user interaction scenario before the product is released. Testing in production allows developers to monitor app performance in unpredictable circumstances. Production is the only place where developers can observe unexpected interactions in the app and fix the issues that emerge during such interactions.

Evaluating User Experience Effectiveness Objectively

You can predict how your users respond to a new feature or a change of design to some extent, based on your target audience analysis. Yet, you can never be a hundred percent sure that your prediction is correct until the product goes live and real users interact with it. Testing in production provides developers with objective information on the quality of user experience with every single update.

Continuous Testing Feedback

Live production traffic and live user data allow developers to catch bugs that cannot be spotted otherwise and fix them quickly. As a result, developers see how user experience changes and how users react to it. Testing in production provides developers with faster testing and code integration processes, allowing them to polish the app faster and more effectively.

Why Some DevOps Are Afraid of Testing in Production

TiP is something many managers and DevOps engineers try to avoid. Here are the main reasons why they fear implementing it:

It seems too risky for some engineers
Poor monitoring leads to poor results
TiP is sometimes considered unsafe for the end-users
It can possibly cause system overload
TiP can lead to messed-up traffic logs and analytics data

True, these concerns are not groundless and could take place in real life, messing up your product performance and reputation. The good news is that you can prevent them by implementing TiP correctly.

How to Do Testing When Your Project is in Production

Choosing the right testing tool is the first step to implementing a testing strategy. It’s fair to say that the means you pick can bring any testing strategy to failure or success. A well-established DevOps process, combined with the right testing tools, can ensure a smooth operation at every stage, from development to production. To incorporate TiP into your project correctly, make sure to explain to your team that production testing should be integrated daily. We recommend you to prepare daily checklists that would specify the main functionality of the app that requires testing.

Deployment Strategies and Methods

Let’s take a closer look at the strategies and methods that you can use to perform testing when your project is already in the production stage.

First, there are the blue-green method and the red-black method, which share the same concept. The blue-green method means that there is a production environment (blue) that gets all the traffic and an idle clone of it (green). When a new version of the app is ready, it is deployed to the idle green environment and is tested there. In case everything goes well in testing, the green environment becomes the new production. With a red-black method, the live production is the red version. Developers deploy the black version to the server(s) when it is well-tested and operates as intended. At any given time, only one environment is live, either red or black.

With the automated rollback strategy, you return the system to the previous stable version when a failure is detected during the release monitoring phase. When implemented correctly, a rollback will help you get back to a stable app state and avoid exposing errors to the users. However, when implemented poorly, rollback can cause data loss, so be careful.

With the controlled test flight, you do not test the updates in the system. In this method, you test whether users respond to changes as intended. You choose a group of real users and monitor their behavior. In case everything goes as expected, you can roll out the new feature.

With fault injection testing, you can create a problem in a live environment on purpose to see if the system can handle it. Fault injection testing is done to protect the system from specific errors. Recovery testing usually accompanies fault injection testing. In every app, some of the processes are almost never tested in live mode. Recovery testing handles these processes to make sure that the system can handle the problems it is meant to handle.

With canary testing, you release a new version to a part of your users and move to the full rollout after it is thoroughly tested. When you make sure that the new feature and the entire system work well under a portion of the live traffic, you can be more confident that it will also work under a lot of traffic. In a way, canary testing is similar to A/B testing.

In A/B testing, you distribute traffic among different versions of the app and see which version converts the most. Eventually, you roll out the version that converts better. With A/B testing, you can run several app versions in parallel and distribute traffic evenly to see which version shows better results. A/B testing is powerful and usually provides excellent insights, but it requires a lot of work, expertise and coordination.

Monitoring Results of TiP

It’s essential to monitor the results of testing in production effectively. Even if you incorporate the best deployment strategies and testing methods, some bugs can still sneak past your attention unnoticed. For this reason, we recommend you to have a system recovery plan at all times.

You cannot prevent all the failures, but you can make sure that your system recovers. Observe the performance of your app in regular real-time conditions, check how the system responds to peak and load hours and always keep an eye on server and database performance. Effective monitoring in production will give you a deep understanding of the system, find and eliminate issues and bottlenecks. This way, production monitoring will help you study your users perfectly and offer them more business value.

Final Thoughts

One of the best ways of making your app testing strategy more effective is implementing testing in production. The importance and benefits of TiP can be overlooked. Some developers avoid testing in production since they think it poses many threats to the live app. In real practice, when used skilfully, TiP helps to manage risks and eliminate issues before they get to the user.

TiP allows developers to detect problems quickly in scenarios that can’t be staged anywhere but the live app mode. It also gives developers a chance to get instant user feedback in beta programs, evaluate user experience effectiveness and monitor app performance in real-life conditions. In all seriousness, not testing your app in production could be one of the reasons your product isn’t scoring as high as you’d like, so it’s high time to embrace it.