Flaky Tests vs. Brittle Tests

A test fails on Monday. It passes on Tuesday without any code change. The team tags it as flaky and moves on. Two weeks later, after a UI refresh, it fails on every run.

That's two different problems wearing the same label. The Monday failure was a flaky test – non-deterministic, driven by something outside the application. The persistent failure after the UI update is a brittle test – coupled to an implementation detail that changed. Tagging both as flaky and applying the same fix solves neither. Retry logic won't make a brittle test less brittle. It just runs the wrong test more times.

Key takeaway: Flaky tests vs. brittle tests at a glance

Flaky tests and brittle tests both reduce trust in automated test results, but they fail for different reasons. A flaky test passes and fails inconsistently without a relevant change to the application – the result is non-deterministic, usually because of timing, environment, shared state, or concurrency. A brittle test fails consistently after an implementation change that doesn't affect the behavior being tested – the test is over-coupled to something like an unstable selector, hard-coded data, or an internal assertion. Here's how the two compare across the dimensions that matter most.

How to tell the difference between a flaky test and a brittle test

When a test fails unexpectedly, the first question is whether the application changed. If it didn't, start investigating flakiness – something outside the application is likely influencing the result. If it did change, re-run the test against the same build. A result that changes across runs points to timing, environment, or shared state issues. A failure that repeats consistently after a UI, data, or implementation change points to brittleness.

Before changing the test either way, confirm whether the failure exposes a real defect. A flaky-looking result can still reveal an application race condition, and a brittle-looking result can still reveal a behavior change the team needs to understand.

What is a flaky test?

A flaky test is a test that produces inconsistent results without any change to the application being tested. It passes one run and fails the next – same code, same build, different outcome. Most of the time, the failure isn't coming from the application. It's coming from something the test depends on that it doesn't fully control.

A test that checks whether a dashboard loads after login might fail not because anything is wrong with the login flow, but because it asserts on an element before an async call has finished. The application behavior was correct – the test ran faster than the data came back.

What causes flaky tests

Flaky tests typically trace back to one of four root causes.

• Timing and asynchronous behavior: Tests often become flaky when they assert too early – before a network request, animation, lazy-loaded element, or background process has finished. Fixed sleeps can make the suite slower without guaranteeing that the application is actually ready. A reliable wait should be tied to a specific application state or event.
• Environment dependencies: Tests become flaky when they depend on systems the test doesn't control. Common examples include third-party APIs, shared environments, shared test accounts, unstable network conditions, and test databases modified by other jobs.
• Test order dependencies: Tests that share state can pass or fail depending on what ran before them. A reliable test should pass whether it runs first, last, alone, or in parallel.
• Resource contention: In parallel execution environments, tests competing for the same database records, browser session, queue, or environment configuration can interfere with each other unpredictably.

How to fix flaky tests

Start by confirming that the failure is truly non-deterministic. Re-run the test against the same build and review whether the failure pattern changes across runs. Then isolate the variable the test doesn't control.

• Replace fixed sleeps with explicit waits tied to application readiness.
• Mock or control unstable external dependencies.
• Reset test data and environment state before each run.
• Give parallel tests independent users, records, sessions, or resources.
• Clean up test artifacts during teardown.
• Quarantine recurring flaky tests while the root cause is being investigated.
• Track flaky tests with an owner, failure pattern, suspected cause, and resolution status.
• Move eligible tests earlier in the SDLC, before deployment to shared test environments.

Retries can help confirm non-determinism or reduce short-term pipeline disruption, but the durable fix is removing the uncontrolled condition that made the result unreliable.

What is a brittle test?

A brittle test breaks when the application changes in ways that don't affect the behavior the test is meant to validate. The test is coupled to implementation details – specific selectors, structural decisions – rather than to what the application is supposed to do.

A test that clicks a submit button located by its CSS class .btn-primary-v2 will fail when a designer renames the class to .btn-submit during a UI refresh – even though the button still works exactly the same way. The test was checking implementation, not behavior.

What causes brittle tests

Brittle tests share one root pattern: they test how something is built rather than what it does.

• Unstable selectors: Tests become brittle when they rely on CSS classes, long XPath chains, DOM position, or generated element IDs – identifiers that reflect how the UI is built rather than what it does. Any of these can change during a routine refactor without affecting functionality, turning a valid test into a false failure.
• Hard-coded test data: Brittle tests often depend on exact values that aren't part of the actual requirement. A test that expects exactly 12 records will fail when valid test data changes, even if the application is working correctly.
• Internal implementation assertions: Tests become brittle when they verify internal state or structure instead of observable behavior. A refactor should not fail a test if the user-visible behavior remains correct.

How to fix brittle tests

Fix brittle tests by moving the test away from implementation details and closer to the behavior the user or system depends on.

• Replace CSS classes, long XPath chains, and DOM-position selectors with stable locators such as test IDs.
• Prefer selectors tied to intent – accessible names, visible text, ARIA roles, or test-specific attributes.
• Assert on user-visible outcomes instead of internal implementation.
• Remove hard-coded values unless the exact value is part of the requirement.
• Use page objects or shared abstractions so selector changes can be updated in one place.
• Review brittle failures after UI refactors to identify tests that are validating structure instead of behavior.

Brittle: Click the element with class .btn-primary-v2.

Better: Click the button with the accessible name Submit order.

Best: Submit the order and verify that the confirmation page displays an order number.

How testing tools approach these problems

The right tool depends on the failure mode. Some capabilities address flakiness directly – through smarter waits and environment controls. Others address brittleness – through more resilient object recognition or by moving away from the scripted model entirely.

Features that reduce flakiness and brittleness in automated testing

Self-healing capabilities detect when a selector has changed – a renamed class, a shifted element – and update the reference automatically rather than failing the test. Smart auto-waiting handles timing dependencies by waiting for the application to reach a ready state rather than relying on fixed sleep calls. Stable object recognition that looks beyond raw selectors reduces brittleness at the point of execution rather than requiring the test author to anticipate every possible UI change.

SmartBear's TestComplete and Reflect bring these capabilities to automated test suites. TestComplete serves teams automating complex desktop and web applications, including those in regulated or offline environments. Reflect is built for teams prioritizing codeless automation across web, mobile, and API. Both reduce the day-to-day maintenance cost of flaky and brittle failures – though neither eliminates the underlying coupling problem. The scripts still exist, and the application can still outpace them.

How autonomous testing changes the equation

Self-healing and stable object recognition make scripts more resilient, but the scripted model has a ceiling. When the application changes frequently enough, the cost of keeping tests current outpaces the value they provide. Autonomous testing approaches the problem at a different level.

Rather than executing a predefined sequence of steps against specific selectors, an autonomous testing tool understands application behavior. It observes how the application works and validates against that understanding – not against a script that specifies exactly which elements to interact with. When the UI changes, there's no brittle selector to break. The application has evolved, and the test evolves with it – not because a self-healing mechanism updated a reference, but because the test was never tied to the implementation in the first place.

SmartBear BearQ™ is built around this model. For teams where brittleness has become a structural problem – where the maintenance overhead of scripted tests has started to outweigh the coverage they provide – autonomous testing changes the economics of the problem rather than managing the symptoms.

A suite you can act on

Flaky and brittle tests undermine the same thing: application integrity . A team that can't trust its suite makes release decisions on intuition rather than evidence. Naming the problem correctly – flaky or brittle, test isolation or implementation coupling – is the starting point for fixing it. The approach that follows depends on the root cause, whether that means improving how existing tests are written or reconsidering the model that produces them.

Frequently asked questions about flaky and brittle tests

What is a flaky test?

A flaky test is a test that produces inconsistent results without any change to the application being tested. It may pass one run and fail the next on identical code. The failure typically comes from factors outside the application – timing issues, environment dependencies, or shared state between tests – rather than from a defect in the application itself.

What is a brittle test?

A brittle test is a test that breaks when the application changes in ways that don't affect the behavior the test is meant to validate. Brittle tests are coupled to implementation details – specific UI selectors, hard-coded data values, or internal structure – rather than to observable application behavior.

What is the difference between a flaky test and a brittle test?

Flaky tests fail unpredictably on unchanged code. Brittle tests fail predictably when code changes, even when those changes don't affect the behavior being tested. A flaky test signals a reliability problem in the test itself – an external dependency or isolation issue. A brittle test signals an over-coupling problem – the test is too closely tied to how the application is built rather than what it does.

Are flaky tests always caused by bad tests?

No. A flaky result can come from the test, the environment, or the application itself. An intermittent failure may reveal a real race condition in the application. Diagnose the root cause before changing the test or adding retries.

Are retries a good way to fix flaky tests?

Retries can help confirm that a failure is non-deterministic and can reduce short-term pipeline disruption. They should be paired with root-cause investigation so the team can fix the timing, state, environment, or dependency issue causing the unreliable result.

Can a test be both flaky and brittle?

Yes. A test can rely on an unstable selector and also have a timing issue. In that case, it may fail intermittently on unchanged code and fail consistently after a UI change. Diagnose each failure mode separately, so the fix addresses both problems.

What is an example of a brittle test?

A brittle UI test might click a button using a CSS class like .btn-primary-v2. If that class changes during a harmless UI update, the test fails even though the button still works. A more resilient test would locate the button by its role, accessible name, visible text, or a stable test-specific attribute.

How do you fix flaky tests?

Start by determining whether the failure points to a real application issue or a test problem. For failures that trace back to the test, quarantine it and address the root cause – add explicit waits, isolate test data, and remove shared environment state. Retry logic is useful for confirming non-determinism but isn't a fix on its own.

How do you fix brittle tests?

Write tests against observable behavior rather than implementation details. Use stable selectors tied to the application's intent – test data attributes, ARIA labels, visible text – and assert on what the application does rather than how it does it. Abstraction patterns like the page object model keep implementation-specific details in one place, reducing the surface area that breaks when the UI changes.