From Flaky to Reliable: How QMetry Keeps Your Pipeline Clean
Not every failure is a bug and not every bug is what it seems. Sometimes, a test fails without warning. No code changes, no environment issues, just a red mark where there should be green. You rerun it, and it passes.
These are flaky tests. And they do more than create noise. They drain team time, stall releases, and make it harder to trust automation at all. Left unchecked, they quietly become one of the most expensive problems in testing.
Flakiness is a roadblock with a real and (very costly) price tag
Google found that flaky tests made up nearly 5% of all test failures and burned through over 2% of total coding time – roughly one full person-year lost for a 50-person team1. Microsoft put the cost even higher: $1.14 million per year in wasted developer time and lost productivity2.
QMetry Test Management’s Flaky Score and Success Rate features, powered by SmartBear HaloAI, help you catch instability before it becomes a blocker. By highlighting patterns in test behavior, they make it easier to spot flaky tests early, understand their impact, and take action before they slow down your team, compromise release quality or cut into your ROI.
See how it works in practice.
Detect Instability Before It Spreads
QMetry uses AI to monitor test executions and automatically assign a Flaky Score, revealing which test cases are showing signs of instability. The score is based on real execution history and highlights non-deterministic behavior you might otherwise miss.
To make the data easy to interpret and act on, each score is paired with a clear color indicator (red/yellow/blue) so teams can quickly spot high-risk tests and prioritize accordingly. With one glance, testers or developers can spot unstable tests that need attention: no need to maintain spreadsheets, no guesswork.
With QMetry, you can also set how many test executions to evaluate, filter by release, platform, or build, and configure what counts as a pass or fail. Once enabled, the score appears across your test case views: list, detail, execution, and links.
Click into any score to see a breakdown of recent runs, so you can quickly understand what’s driving the instability and act accordingly.
Understand Test Behavior Over Time with Success Rate
Identifying flakiness is a crucial first step, but it’s not the whole story. Some tests might not be flagged as flaky, yet still fail more often than they should. That’s where QMetry’s Success Rate metric adds an extra layer of clarity.
Success Rate tracks how consistently a test passes over time, helping you spot trends, catch regressions, and evaluate the true reliability of your test suite. Whether you’re trying to clean up unstable tests, build trust in automation, or improve your CI/CD efficiency, this metric gives you the insight you need to make better testing decisions, faster.
In QMetry, the Success Rate shows the percentage of times a test case has passed over a set number of runs and days. It’s a quick way to check if your test cases are reliable and whether recent fixes are actually working.
You can customize how the Success Rate is calculated:
- Choose how many executions to include
- Set a timeframe (e.g., last 90 days)
- Pick specific releases, builds, or test case versions
Once enabled, QMetry displays Success Rate across your test list, detail views, and execution screens so reliability is always visible. A high score (81–100%) signals a stable test, while lower scores point to tests that may need a closer look.
You can dive into the Success Rate to view the exact runs that contributed to the result, making it easier to troubleshoot issues, understand patterns, and track progress over time.
Watch the demo to see it in action.
Less Time Debugging. More Time Delivering.
Without visibility into test stability, teams waste hours chasing false failures. They rerun tests. Reassign bugs. Sometimes, they ignore alerts altogether which leaves the door open for real issues to slip through.
With QMetry, you:
- Spot flaky tests before they derail releases
- Focus your time on real failures, not noise
- Track progress with clear, reliable metrics
- Build confidence in your automation and your results
As test automation scales, this kind of insight isn’t a nice-to-have – it’s essential. Flaky Score and Success Rate help you move faster, catch risks earlier, and release with confidence.
Try QMetry Test Management for free.
Start Free TrialSources:
1 – https://www.stickyminds.com/article/hidden-costs-flaky-tests-deep-dive-test-reliability