The mutation engine builds an oracle graph that maps test observe statements to the values and identifiers they cover, and validates oracle quality per test. Together, these tell you which parts of your code have strong oracles and which tests need better assertions.
For the conceptual foundation of oracles, see Oracles. For an overview of all advanced features, see Advanced overview.
Two systems work together:
Oracle graph: maps observe statements to the expressions and identifiers they reference, forming a two-hop coverage map emitted as oracle-graph.json.
Oracle validation: the OracleValidator parses test AST to classify each test's oracle quality and computes an oracle coverage metric.
The oracle graph contains three kinds of nodes:
| Node kind | What it represents |
|---|---|
observe | An observe statement in a test block |
value | The expression being observed |
ref | An identifier referenced by the observed expression |
Edges connect observe → value → ref, forming a two-hop coverage map.
Each observe node is tagged with a determinism class based on static analysis:
| Tag | Meaning |
|---|---|
pure | Literals, identifiers, arithmetic, comparisons |
io | Calls to print, read, file_read, etc. |
time | Calls to now, clock, sleep |
random | Calls to rand, random, uuid, shuffle |
unknown | Everything else |
When a mutant survives and the oracle graph shows that no observe covers the mutated expression, the --why output includes a hint:
Oracle: No observe depends on mutated expression at line 15
If the mutant is covered but the observe is nondeterministic:
Oracle: Oracle is nondeterministic (time); result may be flaky
The OracleValidator classifies each test:
| Quality | Meaning |
|---|---|
STRONG | Test has observe statements that reference function outputs or computed values |
CONSTANT | Test only observes constant expressions (observe True, observe 1 == 1) |
NONE | Test has no observe statements at all |
Dead oracles (e.g., if False { observe ... }) are detected and excluded.
The output shows oracle coverage and flags weak tests:
Oracle Validation
Oracle coverage: 75% of tests have strong oracles
2 test(s) with NO oracle:
- test_setup
- test_init
Oracle analysis runs automatically as part of vary mutate. The oracle graph is emitted to the report directory.
Use --why to see oracle hints for specific survivors:
vary mutate source.vary --tests test.vary --why m-0015
Configure oracle warnings in vary.toml:
[mutation]
allow_no_oracle_tests = false # Warn on tests without observe (default)
Set allow_no_oracle_tests = true to suppress warnings about oracle-less tests.
| You see... | What to do |
|---|---|
| "No observe depends on mutated expression" | Add an observe statement that checks the mutated value |
| "Oracle is nondeterministic" | The test covers the mutation site but uses a flaky oracle; stabilize or accept the risk |
| Test classified as CONSTANT | Every assertion is tautological (e.g., observe True); replace with value-checking assertions |
| Test classified as NONE | The test runs code but never checks results; add observe statements |
| Low oracle coverage percentage | Many tests lack meaningful assertions; prioritize tests for critical functions |
| Page | Topic |
|---|---|
| Lie detection | Automatic flagging of tests with constant or missing assertions |
| Observability | Runtime tracing for deeper survivor diagnosis |
| Effects | Understand determinism tags in the context of effect classification |