Mutation Testing
Vary has built-in mutation testing. These pages cover the theory, the workflow, and the CLI.
Alpha. Vary is under active development and not ready for production use. Syntax, APIs, performance, and behaviour may change between releases.
Code coverage tells you which lines ran. It does not tell you whether your tests checked anything. A test that calls a function and ignores the return value gets full coverage and catches zero bugs.
Mutation testing asks a different question: if something in the code changed, would any test notice? Vary's compiler makes small changes to your compiled bytecode (replacing + with -, flipping conditions, returning default values) and runs your tests against each changed version. Each change is a mutant. If the tests still pass, that mutant survived, and your tests have a gap.
In Vary, vary mutate ships with the compiler. There is nothing to install, no plugin to configure, and no separate tool to learn. Mutation testing runs in milliseconds per mutant because it operates on bytecode, not source text, so there is no re-parsing or re-compiling per mutation.
|
|
| Overview | What mutation testing is and why it exists, explained without jargon |
| Introduction | What mutation testing is and why it matters |
| Smallest Example | The simplest possible mutation-testing example: one function, one test, one mutant |
| Golden path | Step-by-step guide from weak tests to strong ones |
| Lie detection | Find tests that pass but never actually check anything |
| Designing for mutation | How to write Vary code that mutation testing can measure well |
| Oracles | How tests decide whether program output is correct |
| Contracts | How contracts kill mutants automatically and how contract adequacy measures specification strength |
| Operators | All 33 mutation operators: 27 AST operators (17 classic + 10 semantic) and 6 bytecode operators |
| Lie detection: deep dive | Ledger demo showing how contracts and exact assertions interact with lie detection |
| Observability | Runtime tracing, differential detection, and assertion groups for understanding why mutants survive |
| Advanced overview | Six pillars that turn mutation scores into actionable quality signals |
| Oracle analysis | Oracle graph structure, determinism classification, and oracle quality validation |
| Effects & integrity | Effect classification, nondeterministic mutant filtering, runtime sealing, and integrity scoring |
| Signatures & manifest | Stable mutant identities, expression IDs, and the mutation manifest for CI and tooling |
| Infrastructure | Caching, quarantine, policy gates, certificates, result storage, and CLI reference |
| Equivalent mutants | Detecting and handling mutants that cannot be killed because they are semantically identical to the original |
| Strict mode | Evidence-based test selection, kill-first scheduling, warm workers, and the hot-swap backend |
| Benchmark | Cross-mode parity benchmark that validates strict-mode optimizations preserve correctness |
| Fast mode | Heuristic test-filter narrowing with a continuous validation pass that auto-falls back when miss rate is too high |
| Incremental inference | Per-mutant reuse of prior outcomes when source, tests, compiler, and operators are unchanged |
| Parity gate | Cross-backend parity validation: every performance backend must match the fresh-loader reference |
| Survivor tail | Per-file accounting that separates the main mutant loop from the flake-detection rerun pass |
Documentation