Defect validation

What it means

After a run, an automation tool has produced failures, not defects. Defect validation is the work of turning one into the other: reproduce the failure, rule out the false positives — a broken test, an environment problem, a flake — and, for the ones that survive, document the bug well enough that a developer can act on it. It is the largest recurring human cost of running an automated suite, and the one an execution-speed comparison never shows.

The model prices it as two activities rather than one number, because they scale with different things. Triage is paid once per failure, real or not, so it multiplies with the tool's false-positive rate. Write-up is paid once per confirmed defect, so it multiplies with how many bugs the product actually has and is the same for every tool. Splitting them is what lets the model show that a tool with better diagnostics saves triage time without pretending it finds fewer bugs.

It is not the same as validating a built case, which happens once in the Build phase and checks that a new test asserts the right thing.

Why it matters to the cost

On the default suite a tool sees about sixteen failures a run; at twenty minutes of triage each and twenty more to write up each of the five real defects, defect validation is roughly seven hours per run — eighty-plus hours a year at twelve runs, before a single test is repaired. It is where a tool's failure diagnostics (screenshots, traces, root-cause hints) earn their licence, and it is the number to measure in a proof of concept: run the suite against a known-bad build and time the validation.

Typical values

Thirty minutes of triage per failure with basic reporting, ten to fifteen with good diagnostics; fifteen to thirty minutes to write up a confirmed defect. For manual testing only the write-up applies: the tester validates as they execute.

Typical values describe a category of tool — AI-native, codeless, script-based, homegrown, manual — never a named product. Why we do not price named products.

The formulas it lives in

Every name is a link to its definition. See the whole model.