Mutation testing

VAST deliberately breaks generated programs in small ways (swapping + for -, flipping conditions) and checks that its comparison engine notices the difference. This is a self-test: it makes sure VAST itself is not missing bugs.

Mutation expansion in VAST

Mutation testing in VAST is not the same as vary mutate. In vary mutate, mutation probes whether your tests catch code changes. In VAST, mutation probes whether VAST itself catches compiler bugs.

The question is: if we inject a known-wrong change into a generated program, does VAST's multi-path comparison actually detect it? If it does not, VAST has a blind spot.

This is called mutation expansion because it expands the set of programs VAST tests by creating controlled variants of each generated program.

How it works

1. Generate program P
2. Execute P on all paths --> must agree
3. Apply a mutation operator --> mutant M
4. Execute M on all paths
5. Check: did any path disagree?

If the original program P already disagrees across paths, it is saved as a regular differential testing failure. Mutation expansion only runs on programs that pass.

For each passing program, VAST applies multiple mutation operators and generates several mutants. Each mutant is a program that should behave differently from the original. If all paths still produce the same result as the original, the mutation was not detected. This is tracked per operator to identify weak spots.

Mutation operators

VAST implements nine mutation operators that cover different fault classes:

Operator	What it changes	Example
Operator replace	Swaps a binary operator	x + y becomes x - y
Negate condition	Flips an if-condition	if x > 0 becomes if x <= 0
Swap branch	Swaps then/else blocks	if c { A } else { B } becomes if c { B } else { A }
Constant boundary	Changes a literal constant	0 becomes 1, 1 becomes 0
String literal	Modifies string content	"hello" becomes "hellox"
List literal	Removes a list element	[1, 2, 3] becomes [1, 3]
Enum variant	Changes enum variant index	Color.Red becomes Color.Blue
None swap	Swaps None and Some	None becomes Some(0)
Match case swap	Reorders match cases	Case ordering changed

Each operator targets a specific kind of expression or statement. The generator applies operators probabilistically, controlled by the --mutants-per-seed flag (default: 5 mutants per program).

What the results mean

For each mutant, VAST records whether the multi-path comparison detected the change:

Outcome	Meaning
Detected	At least one path produced a different result from the original (good)
Undetected	All paths produced the same result as the original, despite the mutation
Error	The mutant caused a compilation or runtime error (neutral, not a detection failure)

Undetected mutations are not necessarily bugs. Some mutations are semantically equivalent to the original. For example, swapping x * 1 to x * 0 is detected, but swapping dead code branches might not change the result.

VAST tracks detection rates per operator. A consistently low detection rate for an operator suggests that VAST's generation or comparison has a gap in that area.

Example

Original program:

def __vast_compute() -> Int {
    let x = 5
    let y = 3
    return x + y
}

Operator-replace mutant (+ to -):

def __vast_compute() -> Int {
    let x = 5
    let y = 3
    return x - y
}

The original returns 8. The mutant returns 2. All paths should report 2 for the mutant. Since 8 != 2, the mutation was detected. If any path still returned 8 for the mutant, that would be a VAST infrastructure bug.

Running mutation expansion

vary vast --mutate --count 100 --seed 42
vary vast --mutate --count 50 --seed 42 --mutants-per-seed 10

The output includes per-operator statistics showing how many mutants were generated, how many were detected, and the detection rate.

In CI deep mode, mutation expansion runs automatically as part of nightly verification.

How this differs from vary mutate

	VAST mutation expansion	vary mutate
Purpose	Validates VAST's own detection infrastructure	Validates your test suite
Target	Generated programs	Your source code
Mutations applied to	VAST-generated ASTs	Compiled bytecode (or AST)
What a survivor means	VAST has a blind spot	Your tests have a gap

For a full comparison, see VAST vs mutate.