Alpha. Vary is under active development and not ready for production use. Syntax, APIs, performance, and behaviour may change between releases.
CLI reference
Run
vary vastto generate random programs and test the compiler. Flags control program count, language features, seed, coverage reporting, and CI modes.
Basic usage
vary vast
With no flags, this runs 100 programs using the core profile and a clock-based seed.
Options
Core options
| Flag | Short | Default | Description |
|---|---|---|---|
--profile | -p | core | Generation profile (core, control, complete, etc.) |
--count | -n | 100 | Number of programs to generate and test |
--seed | -s | clock | Base random seed (deterministic when set) |
--verbose | -v | off | Print per-seed results |
--timeout | 5000 | Per-program JVM execution timeout in milliseconds | |
--stop-on-first-mismatch | off | Stop after the first mismatch | |
--size | none | Program size tier: small, medium, or large |
Mode and CI options
| Flag | Default | Description |
|---|---|---|
--mode | explore | CI mode: explore, fast, deep, continuous, or rc-blocking |
--rotate-seed | off | Combine seed with SHA-256(date + commit) for daily variation |
--commit-hash | auto-detect | Git commit hash for seed rotation |
--metrics-file | .vast-logs/vast-metrics.jsonl | Path for JSONL metrics output |
--regression-dir | none | Directory for regression artifacts (deep mode) |
--duration | 60 | Time budget in seconds (continuous mode only) |
Reduction and corpus options
| Flag | Default | Description |
|---|---|---|
--reduce | off | Shrink mismatches to minimal reproducers |
--max-shrink | 1000 | Max oracle calls during reduction |
--max-reduce-time | 30000 | Max reduction time in milliseconds |
--corpus-dir | none | Directory for reproducer corpus |
--replay-corpus | off | Replay saved corpus before exploration |
Testing mode options
| Flag | Default | Description |
|---|---|---|
--metamorphic | off | Run metamorphic testing with semantics-preserving transforms |
--mutate | off | Run mutation expansion testing |
--mutants-per-seed | 5 | Max mutants to test per seed |
--round-trip | off | Run parser round-trip validation |
--opt-check | off | Enable 4-path optimizer differential validation |
--stress | off | Enable stress testing with boundary values and edge cases |
--calibrate | off | Run path calibration before testing |
Specialized generator options
These flags enable specialized program generators that target specific compiler subsystems. They require --mode explore (not deep) because deep mode silently ignores specialized flags.
| Flag | Default | Description |
|---|---|---|
--stateful | off | Generate stateful programs with persistent mutable state |
--state-steps | 100 | Number of state transition steps per program |
--state-structs | 1 | Number of state struct fields to generate |
--aliasing | off | Generate heap aliasing verification programs |
--concurrency | off | Generate concurrency semantics test programs |
--threads | 4 | Number of simulated concurrent threads |
--exception-propagation | off | Generate exception propagation verification programs |
--symbolic | off | Symbolic input guidance for deeper state coverage |
--ir-check | off | Verify IR translation equivalence (AST to IR to JVM) |
--verify-all-passes | off | Verify all optimizer passes independently |
--verify-pass | none | Verify a specific optimizer pass |
--large-programs | off | Generate large programs (500-10000 AST nodes) |
--large-program-count | 10 | Number of large programs to generate |
Acceptance and closeout options
| Flag | Default | Description |
|---|---|---|
--check-acceptance | off | Check acceptance-set coverage against the corpus |
--write-acceptance-corpus | off | Write acceptance-set reproducers to the corpus directory |
--closeout-report | off | Generate the VAST closeout report with evidence tiers, trigger quality table, and verdict |
--show-lanes | off | Print the nightly lane registry and exit |
--verify-config | off | Verify CI mode configuration invariants (profile lists, seeds, counts, lane registry) |
Lane and metrics options
| Flag | Default | Description |
|---|---|---|
--lane | none | Lane name for nightly metrics tagging (e.g., vast-deep-differential) |
--write-summary | off | Write a nightly summary JSON to the verification-history ledger |
--run-class | nightly | Run class for summary: nightly, rc, or release |
--prune-history | off | Remove nightly summaries beyond retention limit |
--check-history | off | Check ledger for staleness |
Corpus management options
| Flag | Default | Description |
|---|---|---|
--validate-corpus | off | Validate corpus entries against schema |
--corpus-index | off | Generate INDEX.md listing all corpus entries |
--history-summary | off | Generate SUMMARY.md from verification history |
Coverage and reporting options
| Flag | Default | Description |
|---|---|---|
--show-coverage | off | Print feature coverage table |
--semantic-coverage | off | Print semantic coverage table (27 properties) |
--interaction-coverage | off | Print feature interaction coverage |
--confidence | off | Print confidence report (LOW/MODERATE/HIGH/VERY_HIGH) |
--path-health | off | Print path agreement matrix and reliability |
Profiles
core
Straight-line programs only. No loops, no helper functions.
Constructs: integer and boolean literals, variables (let/mut), arithmetic (+, -, *, //, %), comparisons (==, !=, <, >, <=, >=), boolean logic (and, or, not), unary negation, if/else statements, if-expressions, return.
Fast execution. Use this for high-volume CI runs.
vary vast --profile core --count 1000 --seed 1
control
Adds helper functions and bounded while loops to everything in core.
Slower due to JVM compilation of larger programs. Use this for deeper coverage.
vary vast --profile control --count 100 --seed 1
Feature profiles
| Profile | Adds |
|---|---|
text | String literals, concatenation, comparison |
enums | Enum declarations, variant construction, match |
records | Data declarations, field access, construction |
collections | List literals, indexing, len(), for..in |
optional | T?, None, null narrowing |
float | Float literals, arithmetic, int() conversion |
patterns | Pattern matching with destructuring |
errors | try/except, raise |
generics | Generic functions and data types |
Composite profiles
| Profile | Includes |
|---|---|
types | Core + all feature profiles (without patterns/errors/generics) |
complete | Everything: all features enabled |
CI modes
| Mode | Programs | Profiles | Use case |
|---|---|---|---|
explore | Configurable | 1 | Manual investigation, single profile. Required for specialized flags. |
fast | ~100 | 4 | Quick smoke test, under 2 minutes |
deep | ~9,000 | 12 | Nightly CI, broad coverage |
continuous | Time-bounded | 12 (adaptive) | Long-running exploration, overnight fuzzing |
rc-blocking | ~100 | 3 | Release candidate gate with fixed seed |
vary vast --mode fast # Quick smoke test
vary vast --mode deep --verbose # Nightly
vary vast --mode continuous --duration 300 # 5-minute exploration
See CI integration for how these are used in practice.
Examples
CI smoke test
vary vast --profile core --count 200 --seed 12345
Deterministic and fast. Runs the same programs every time. Exit 0 on success, exit 1 on any mismatch.
Optimizer validation
vary vast --profile types --count 100 --seed 42 --opt-check
Runs four paths (AST, IR, JVM-optimized, JVM-unoptimized) to catch optimizer bugs.
Stress testing
vary vast --profile core --count 100 --seed 42 --stress
Generates programs with boundary values (Long.MAX_VALUE, 0, -1), overflow chains, identity operations, and float precision stress patterns.
Continuous exploration
vary vast --mode continuous --duration 600 --stress
Runs for 10 minutes, adaptively selecting profiles based on coverage gaps. Produces a report with per-profile summary, confidence score, and gap analysis.
Full coverage report
vary vast --profile complete --count 50 --seed 42 \
--show-coverage --semantic-coverage --interaction-coverage --confidence
Replay a specific seed
vary vast --seed 842193 --count 1
Generates and tests exactly one program. The same seed always produces the same program and the same result.
Stop on first mismatch
vary vast --count 10000 --seed 1 --stop-on-first-mismatch
Stops as soon as a mismatch is found and prints the full mismatch report with replay command.
Exit codes
| Code | Meaning |
|---|---|
| 0 | All programs agreed (or were invalid) |
| 1 | At least one mismatch, path failure, or equivalence failure |
Output format
The summary line always appears:
VAST: 100/100 passed (0.3s)
If mismatches occur, each one is printed before the summary with the seed, verdict, path outcomes, generated source, and replay command.