What 100,000+ Monthly Test Executions Reveal About Reliability, Flakiness, and Maintenance

Once automated testing becomes part of every release cycle, the harder question is whether you can trust the results it produces. Failures, flaky behavior, reruns, UI changes, and test maintenance all affect how much engineering work follows each regression run.

Over the past 12 months, we reviewed automation activity across Android, iOS, and cross-browser testing. That work spans 30+ enterprise customers, 200+ projects, 10,000+ automated test cases, and more than 100,000 test executions per month.

The reason we conducted this exercise was to understand where automation still demands attention after tests are running: what causes failures, how often flakiness appears, where reruns come from, and how frequently UI changes create maintenance.

The percentages below reflect recurring ranges our engineering team sees across these environments. Please read them as operational observations rather than universal benchmarks.

To see TestGrid in action, request a free trial.

TL;DR

• Roughly 5% of executions fail, with causes ranging from product behavior and test issues to environment conditions, timing, and application changes.
• Around 5% of failed executions involve UI, element, or locator changes, while fewer than 10% of regression cases require maintenance related to those changes.
• Flaky behavior appears in roughly 4% to 5% of testing activity, while reruns remain below 5%.
• Healing becomes relevant in approximately 5% to 10% of applicable scenarios, and successful recovery rarely requires additional human editing.
• In one engineering workflow, generating an end-to-end use case with TestGrid tools takes around 15 minutes, while manually writing and executing the same type of script can take more than an hour.

Roughly 5% of Executions End in Failure

That percentage covers several different problems.

The application may behave incorrectly. An element may fail to load. Test data may be unavailable. The environment may behave differently from a previous run. An application update may also leave the test referring to an earlier version of the UI.

Consider an automated checkout test that cannot interact with the payment button. You still need to determine whether the payment function failed, the element changed, the page rendered late, or an earlier action left the application in an unexpected state.

A pass-fail result tells you where execution stopped. Failure classification tells you where to investigate. When you separate product failures, test failures, environment problems, and application-change issues, your team has a clearer triage path.

With TestGrid, you can inspect execution logs, session details, and test reports after a failed run to trace what happened during execution and narrow the issue to the application, test flow, or environment.

Around 5% of Failed Executions Involve UI, Element, or Locator Changes

These cases often appear after an application update, when dynamic elements behave differently, or when a UI enhancement changes the information a test uses to identify an element.

The user journey can continue to work as expected even when the automation reference becomes outdated. That creates maintenance work because the test needs to catch up with the application.

Across the regression activity we reviewed, fewer than 10% of cases require maintenance related to UI or element changes.

The amount of work varies. A small UI update may require a simple element correction. A larger application change can affect several steps or require additional test logic.

We handle this through regular monitoring, along with biweekly or monthly maintenance depending on the customer environment.

If you track this category independently, you can see how often application changes generate automation work even when the underlying business flow remains valid.

TestGrid lets you compare baseline and current UI states across browsers, devices, operating systems, and screen sizes. Visual testing helps you identify layout changes, overlapping elements, rendering differences, and other UI regressions alongside functional checks.

Flaky Behavior Appears in Roughly 4% to 5% of Testing Activity

We treat a test as flaky when it fails and later passes without a meaningful change to the application or test.

An element-loading issue is one example. The element may be unavailable during one execution and appear normally during the next. Timing, asynchronous rendering, network conditions, application state, and test data can create similar results.

Flakiness weakens the signal your regression suite gives you. When the same test produces different outcomes under equivalent conditions, you first have to determine whether the application has a problem or the execution itself was unstable.

That uncertainty is why flakiness deserves its own measurement.

With TestGrid’s Real Device Cloud, you can reproduce unstable scenarios across Android and iOS devices while monitoring factors such as CPU, memory, network usage, battery consumption, and UI responsiveness that may affect execution.

Reruns Remain Below 5%

A rerun tells you that another execution happened after the initial result but it doesn’t explain why the first run failed.

The distinction becomes useful when a failed test passes on the second attempt. Your regression can continue, but the original instability still needs context.

If the same test scenarios repeatedly require another attempt, they consume device time, execution capacity, and engineering attention even when the final result is green.

That’s why it’s vital to track which tests need reruns, how often that happens, and what triggered the second execution. That gives you a clearer view of recurring instability.

TestGrid supports parallel execution across browsers, devices, and OS versions, giving you a practical way to rerun affected scenarios or expand regression coverage without processing every environment sequentially.

Healing Becomes Relevant in Roughly 5% to 10% of Applicable Scenarios

This frequency depends on the application and its technology. Applications with dynamic elements or frequent UI changes create more situations where the element information used by an existing test becomes outdated.

In our experience, additional human editing after a successful heal is uncommon. For your own environment, measure how often healing is triggered, whether execution continues successfully, and whether someone still needs to modify the test afterward.

Together, those measurements show how much manual maintenance recovery is removing from your regression process.

TestGrid’s AutoHeal can respond to changed element references by identifying an updated locator for the affected object. This helps valid test flows continue when a minor UI change would otherwise require a manual locator update.

The execution data shows where automation requires attention after tests begin running.

Engineering Example: Around 15 Minutes to Generate an End-To-End Use Case

We also looked at the effort required to create an executable test before it enters regression.

In some scenarios, manually writing a script and executing it can take more than an hour. Using TestGrid, generating a simple or complex end-to-end use case can take around 15 minutes. This is an engineering example rather than an average across all 10,000+ test cases.

The effort depends on what you are testing. A straightforward UI journey may involve navigation, data entry, actions, and visible validations. A more complex flow may depend on application state, custom business rules, external services, or logic that needs to be written explicitly.

We see UI-level scenarios as a strong fit for CoTester-assisted test creation. Complex flows can still require additional logical code.

For your team, authoring time is best calculated from the point you receive a test requirement to the point you have an executable test that has been reviewed and accepted. That captures generation, validation, corrections, and any additional logic required before the test enters regression.

With CoTester, you can describe test scenarios in plain English or provide existing requirements and test steps, then turn them into executable flows across real browsers and devices.

You can review and refine the generated flow and switch to code mode or add custom logic when a scenario requires more explicit control.

What to Measure After Execution Volume

Once your automation program reaches meaningful execution volume, the useful questions begin after the test runs.

You need to know why it failed, whether the result was repeatable, whether another execution was required, whether an application change created maintenance, and whether automated recovery still needed human input.

Keeping those categories separate gives you a clearer view of test reliability and the engineering effort required to maintain it.

Your percentages will differ from ours, but these categories give you a practical baseline for examining where your own automation program is creating work.

Find out what TestGrid reveals in your test suite. Request a free trial.