Vary Course

Mutation Testing

Five runnable lessons covering vary mutate, scores, survivors, boundary mutations, and the inner-loop workflow.

First mutations

Run your first mutation pass

See that one happy-path test leaves most mutants alive.

pure def has_free_shipping(order_total: Int) -> Bool {
    return order_total >= 50
}

test "larger order qualifies" {
    observe has_free_shipping(75) == True
}

Run: vary mutate free_shipping.vary --tests free_shipping.vary --quick

Expected output:

(?m)Mutation score: \d{1,2}%

Pin down the survivors

Add a boundary and an opposite-branch test until the score hits 100%.

pure def has_free_shipping(order_total: Int) -> Bool {
    return order_total >= 50
}

test "larger order qualifies" {
    observe has_free_shipping(75) == True
}

test "smaller order does not qualify" {
    observe has_free_shipping(20) == False
}

test "exactly 50 qualifies" {
    observe has_free_shipping(50) == True
}

test "49 does not qualify" {
    observe has_free_shipping(49) == False
}

Run: vary mutate free_shipping.vary --tests free_shipping.vary --quick

Expected output:

(?m)Mutation score: 100%

Common survivors

Boundary mutants survive without edge tests

See how range predicates leak mutants when tests skip the edges.

pure def is_2xx(status: Int) -> Bool {
    return status >= 200 and status < 300
}

test "200 is 2xx" {
    observe is_2xx(200) == True
}

test "404 is not 2xx" {
    observe is_2xx(404) == False
}

Run: vary mutate is_2xx.vary --tests is_2xx.vary --quick

Expected output:

(?m)Mutation score: \d{1,2}%

Lock in every edge

Cover all four boundaries of a range predicate.

pure def is_2xx(status: Int) -> Bool {
    return status >= 200 and status < 300
}

test "200 is the lower edge" {
    observe is_2xx(200) == True
    observe is_2xx(199) == False
}

test "299 is the upper edge" {
    observe is_2xx(299) == True
    observe is_2xx(300) == False
}

test "404 is not 2xx" {
    observe is_2xx(404) == False
}

Run: vary mutate is_2xx.vary --tests is_2xx.vary --quick

Expected output:

(?m)Mutation score: 100%

Practical use

Mutation in the inner loop

Hit 100% on a multi-branch classifier with one edge pair per threshold.

pure def class_of(status: Int) -> Str {
    if status >= 500 {
        return "5xx"
    }
    if status >= 400 {
        return "4xx"
    }
    if status >= 200 {
        return "2xx"
    }
    return "other"
}

test "200 is the 2xx lower edge" {
    observe class_of(200) == "2xx"
    observe class_of(199) == "other"
}

test "400 is the 4xx lower edge" {
    observe class_of(400) == "4xx"
    observe class_of(399) == "2xx"
}

test "500 is the 5xx lower edge" {
    observe class_of(500) == "5xx"
    observe class_of(499) == "4xx"
}

test "deep 5xx still classifies as 5xx" {
    observe class_of(503) == "5xx"
}

Run: vary mutate class_of.vary --tests class_of.vary --quick

Expected output:

(?m)Mutation score: 100%