Alpha. Vary is under active development and not ready for production use. Syntax, APIs, performance, and behaviour may change between releases.
Oracle analysis
The mutation engine builds an oracle graph that maps test observe statements to the values and identifiers they cover, and validates oracle quality per test. Together, these tell you which parts of your code have strong oracles and which tests need better assertions.
For the conceptual foundation of oracles, see Oracles. For an overview of all advanced features, see Advanced overview.
What it does
Two systems work together:
Oracle graph: maps observe statements to the expressions and identifiers they reference, forming a two-hop coverage map emitted as oracle-graph.json.
Oracle validation: the OracleValidator parses test AST to classify each test's oracle quality and computes an oracle coverage metric.
What it outputs
Oracle graph
The oracle graph contains three kinds of nodes:
| Node kind | What it represents |
|---|---|
observe | An observe statement in a test block |
value | The expression being observed |
ref | An identifier referenced by the observed expression |
Edges connect observe → value → ref, forming a two-hop coverage map.
Determinism classification
Each observe node is tagged with a determinism class based on static analysis:
| Tag | Meaning |
|---|---|
pure | Literals, identifiers, arithmetic, comparisons |
io | Calls to print, read, file_read, etc. |
time | Calls to now, clock, sleep |
random | Calls to rand, random, uuid, shuffle |
unknown | Everything else |
Oracle hints
When a mutant survives and the oracle graph shows that no observe covers the mutated expression, the --why output includes a hint:
Oracle: No observe depends on mutated expression at line 15
If the mutant is covered but the observe is nondeterministic:
Oracle: Oracle is nondeterministic (time); result may be flaky
Oracle validation
The OracleValidator classifies each test:
| Quality | Meaning |
|---|---|
STRONG | Test has observe statements that reference function outputs or computed values |
CONSTANT | Test only observes constant expressions (observe True, observe 1 == 1) |
NONE | Test has no observe statements at all |
Dead oracles (e.g., if False { observe ... }) are detected and excluded.
The output shows oracle coverage and flags weak tests:
Oracle Validation
Oracle coverage: 75% of tests have strong oracles
2 test(s) with NO oracle:
- test_setup
- test_init
How to use it
Oracle analysis runs automatically as part of vary mutate. The oracle graph is emitted to the report directory.
Use --why to see oracle hints for specific survivors:
vary mutate source.vary --tests test.vary --why m-0015
Configure oracle warnings in vary.toml:
[mutation]
allow_no_oracle_tests = false # Warn on tests without observe (default)
Set allow_no_oracle_tests = true to suppress warnings about oracle-less tests.
How to interpret the results
| You see... | What to do |
|---|---|
| "No observe depends on mutated expression" | Add an observe statement that checks the mutated value |
| "Oracle is nondeterministic" | The test covers the mutation site but uses a flaky oracle; stabilize or accept the risk |
| Test classified as CONSTANT | Every assertion is tautological (e.g., observe True); replace with value-checking assertions |
| Test classified as NONE | The test runs code but never checks results; add observe statements |
| Low oracle coverage percentage | Many tests lack meaningful assertions; prioritize tests for critical functions |
What to do next
| Page | Topic |
|---|---|
| Lie detection | Automatic flagging of tests with constant or missing assertions |
| Observability | Runtime tracing for deeper survivor diagnosis |
| Effects | Understand determinism tags in the context of effect classification |