TestingBudget

Run

Flaky failures per run

The share of cases that fail for no product reason — timing, environment, brittle selectors — and still have to be investigated.

Also called: flaky tests · non-deterministic failures · false positives (the non-deterministic half) · false alarms

Unit
% of cases per run
Where you enter it
Run tab, Detailed mode, per vendor.

What it means

Flaky failures are failures that a re-run on the same build makes disappear. They come from timing, test data, environment instability and brittle element locators, and they cost the same triage time as a real failure until someone has proved they are not one. The rate is a property of the tool and how it is used.

It is one of the three components of failures per run. Entered as a percentage of cases; the archetype default stands in until a proof of concept gives a better number.

Why it matters to the cost

Google found roughly one run in seven hits a flaky failure and Microsoft measured about thirty minutes per investigation; teams report five to ten hours a week lost to false failures. A one-point difference in flake rate on a 300-case suite is three extra investigations per run, every run.

Typical values

AI-native tools around 1%; script-based tools around 2%; homegrown frameworks around 3%; none for manual testing.

Typical values describe a category of tool — AI-native, codeless, script-based, homegrown, manual — never a named product. Why we do not price named products.

Where it appears in the model

Run — every release

tests broken per run
Total test cases × Test breakage rate ÷ Runs per year — found here, repaired under Maintain
flaky per run
Total test cases × Flaky failures per run
Failures per run
Defects found per run + tests broken per run + flaky per run — derived, never asked
Run effort
Prepare and start a run + Failures per run × Triage one failure + Defects found per run × Write up one defect + Report and sign off — an automation tool
Run effort
Prepare and start a run + Total test cases × Execution time per case + Defects found per run × Write up one defect + Report and sign off — a manual approach — the tester is the run
Machine time per run
Total test cases × Execution time per case — automation only; duration, not labour
Run cost per year
Run effort per run × Runs per year × Effective hourly rate + Token cost

Every name is a link to its definition. See the whole model.

Try it with your numbers

Flaky failures per run is a row in the calculator. Change it and every total on the page moves with it — free, in your browser, nothing to install.

See how flaky failures per run impacts the calculation of test automation costs

Related terms

  • False positives per run Test failures that are not product defects — tests the product broke plus flaky failures. The model counts both, on separate rows, because they have different owners.
  • Failures per run Everything a person has to look at after a run: defects found plus tests the product broke plus flaky failures — derived, never typed.
  • Triage one failure The time to look at one failing test and decide what it is: a real defect, a broken test, an environment problem, or a flake.
  • Self-healed cases per year The share of broken cases an AI tool repairs on its own — which then need a person to confirm the repair was right.

Last reviewed 2026-09-15. All terms.