Alpha. Vary is under active development and not ready for production use. Syntax, APIs, performance, and behaviour may change between releases.
Effects & integrity
Every function is tagged with a set of effects based on the built-in functions it calls. The mutation engine uses this classification to skip flaky mutants, seal nondeterminism at test time, and compute an effect stability score that feeds into the integrity grade.
For an overview of all advanced features, see Advanced overview.
What it does
The engine uses fixpoint analysis to propagate effects through the call graph. Each function receives one or more effect tags:
| Effect | Triggered by | Example built-ins |
|---|---|---|
PURE | No side effects | abs, len, str, sorted, range |
IO | Console/output | print, println, input |
TIME | Clock access | time, sleep |
RANDOM | Nondeterminism | rand_int, rand_seed |
NETWORK | HTTP calls | _http_request |
FS | File system | read_text, write_text, list_dir, mkdir |
PROCESS | OS interaction | exit, getenv, argv, spawn |
Effects propagate transitively: a function that calls a RANDOM function is also tagged RANDOM.
What it outputs
Nondeterministic mutant filtering
Mutants inside functions tagged TIME, RANDOM, or NETWORK are skipped by default because they produce flaky results:
Skipped 3 nondeterministic mutants (use --unstable to include)
Runtime sealing
For RANDOM-effect functions, the engine sets a deterministic seed before running each mutant. This makes rand_int() repeatable and rand_seed() a no-op during mutation testing, so RANDOM mutants produce stable results even without --unstable.
Effect stability score
The effect stability metric measures the fraction of functions with stable (non-flaky) effects. It contributes 20% to the integrity score.
Integrity scoring
The integrity score is a weighted composite of four metrics:
| Component | Weight | Source |
|---|---|---|
| Mutation score | 40% | Fraction of mutants killed |
| Contract adequacy | 20% | Fraction of contract obligations defended |
| Oracle coverage | 20% | Fraction of tests with strong oracles |
| Effect stability | 20% | Fraction of functions with stable effects |
The score produces a letter grade:
| Grade | Score range |
|---|---|
| A | 90 to 100 |
| B | 75 to 89 |
| C | 60 to 74 |
| D | 40 to 59 |
| F | 0 to 39 |
The integrity score is included in mutation certificates and the manifest. Set a minimum:
[mutation]
min_integrity = 75.0 # Fail if integrity score < 75 (grade B)
When the threshold is set, vary mutate exits with code 1 if the integrity score falls below it. The --why output includes a component breakdown showing which area is weakest.
How to use it
Effect classification runs automatically. To include nondeterministic mutants:
vary mutate source.vary --tests test.vary --unstable
To see which functions have which effects, use --observe:
vary mutate source.vary --tests test.vary --observe
How to interpret the results
| You see... | What to do |
|---|---|
| Mutants skipped as nondeterministic | These functions use TIME/RANDOM/NETWORK; use --unstable to include them or accept the filtering |
| Low effect stability | Many functions use nondeterministic built-ins; consider isolating side effects behind interfaces |
RANDOM mutants still flaky with sealing | The function may use external randomness not captured by rand_seed; quarantine or isolate |
| Integrity grade below threshold | Check the component breakdown: the weakest component tells you where to focus |
What to do next
| Page | Topic |
|---|---|
| Signatures | How mutant identities survive refactors |
| Contracts | Contract adequacy contributes 20% to the integrity score |
| Oracle analysis | Oracle coverage contributes 20% to the integrity score |