Mutation testing for Ruby, built on Prism. Open source, RSpec-integrated, Rails-first. Available on RubyGems.
active_mutator mutates your code one small change at a time (> becomes >=,
&& becomes ||, a statement gets deleted, a condition gets forced, and so
on). It runs exactly the examples that cover the mutated line, and reports
every mutant your suite fails to kill. A surviving mutant is a behavior
change no test notices: a precise, machine-verified test gap.
defdiscount(total)return0iftotal < 100total / 10endit{expect(calc.discount(50)).toeq(0)}it{expect(calc.discount(200)).toeq(20)}Both examples pass. Line coverage on discount is 100%. Run active_mutator
and one mutant survives anyway:
Surviving mutants:
Calculator#discount (lib/calculator.rb:11)
replace `<` with `<=`
- total < 100
+ total <= 100
Nothing in the test suite calls discount(100), the one input where <
and <= disagree. The tests pass, and coverage is green. But the boundary
is still unverified. That gap is invisible to coverage and obvious to
mutation testing. Add it { expect(calc.discount(100)).to eq(0) } and the
mutant is killed.
Coverage answers "did a test run this line?" Mutation testing answers "would a test notice if this line were wrong?" That is a different, and usually more useful, question.
active_mutator applies one small, syntactically valid change to your code (a "mutant") and re-runs only the examples that cover it. If a test fails, the mutant is killed: your tests correctly reject that wrong behavior. If every covering test still passes, the mutant survived: something changed and nothing noticed. A survivor is not a hypothetical. It is the exact line, the exact before and after diff, and proof that no assertion depends on the difference.
Mutation score is (killed + timeout) / (killed + timeout + survived + error).
100% is usually not the right target. Some mutants are behaviorally
equivalent to the original and can never be killed by any test. That is
why active_mutator has a committed acceptance ledger. It lets you close
survivors out with a stated reason instead of chasing an unreachable score.
Full primer, including the origin of the technique and further reading:
docs/guides/what-is-mutation-testing.md.
# Gemfilegroup:development,:testdogem"active_mutator"endRequires Ruby 3.2 or later, RSpec, and a green suite. Linux/macOS (MRI fork).
bundle install
bundle exec active_mutator app/models/calculator.rbThe first run performs an instrumented baseline of your suite to build the
coverage map. The map is cached in .active_mutator/ and refreshed
incrementally after that (see
docs/guides/how-it-works.md).
Then each mutant runs in its own fork against only its covering examples.
$ bundle exec active_mutator app/models/calculator.rb
.....S..T...U..A....
killed: 14
survived: 1
timeout: 1
error: 0
uncovered: 1
accepted: 1
invalid (discarded): 2
Mutation score: 93.8%
Surviving mutants:
Calculator#discount (app/models/calculator.rb:9)
replace `<` with `<=`
- total < 100
+ total <= 100
Each character on the progress line is one mutant, printed as it finishes:
| Char | Status | Meaning |
|---|---|---|
. | killed | a covering test failed. Good, the mutant is dead |
S | survived | every covering test passed. This is a test gap |
T | timeout | ran past its time budget. Counts as detected (the mutant likely made a loop never end), and the summary lists each one with elapsed vs budget so a tight budget is visible |
E | error | the worker crashed, or the mutated code raised outside a test assertion. Not detected: counts against the score and fails the run |
U | uncovered | no test executes the mutated line at all. This is coverage debt, worse than a survivor |
A | accepted | matches a known-equivalent entry in the acceptance ledger. Excluded from the score |
invalid mutants (edits that don't even re-parse as valid Ruby) are
discarded before scheduling and reported as a count only. Exit code is 1
if unaccepted survivors or errors exist (or, with --fail-at, if the score
is below the threshold), 0 otherwise, including when there are only
uncovered or accepted results. The JSON report's exit_reason field
(unaccepted_survivors, worker_errors, clean) is independent of the
--fail-at gate. A --since
or --subject run that plans zero mutants prints no score; it warns with the
cause and exits 1 unless --allow-empty is given.
When survivors exist, the summary also prints a per-operator table showing how often each operator's mutants survive, to help spot likely-equivalent mutant patterns.
- Subject discovery: a Prism visitor finds every method (
def) in your target files. - Source-span edits: each operator emits byte-range text edits
against the original file, not a rewritten AST. Every mutant is
re-parsed with Prism and discarded (
invalid) if the edit produced something that doesn't parse. No unparser is ever built or maintained. - Coverage-mapped test selection: one instrumented baseline run maps every source line to the examples that cover it. Incremental runs refresh only what changed instead of re-running the whole suite.
- Fork-per-mutant kill runs: the parent preloads your app and spec helper once. Each mutant is inserted and exercised in its own fork against just its covering examples, so results can't bleed state between mutants.
Full architecture, including the coverage-cache format, the fork pipeline,
the serial lane for browser specs, timeout budgets, and every status, is in
docs/guides/how-it-works.md.
active_mutator # mutate app/ and lib/, full run
active_mutator app/models # scope by path (directory)
active_mutator app/models/document.rb # scope to a single file
active_mutator --changed # uncommitted work only (dev loop)
active_mutator --since origin/main # PR scope (CI)
active_mutator --subject 'Foo::Bar#baz'# one method
active_mutator --exclude 'lib/generated'# skip a subtree (repeatable)--subject also takes broader expressions: Foo::Bar (all methods of
that constant), Foo::Bar* (raw name prefix — matches Foo::Bar::Qux
and also Foo::Barn), Foo::Bar#* (instance
methods only), Foo::Bar.* (singleton methods only).
--exclude PAT is a glob relative to the project root, applied during
subject discovery, and gitignore-like: lib/generated, lib/generated/,
and lib/generated/** all exclude the whole subtree. File globs like
**/legacy/* work too.
Skip a single method by putting # active_mutator:skip on the line above
its def:
# active_mutator:skipdeflegacy_delegatortarget.callendStatuses: killed (test failed, this is good), survived (test gap),
timeout (counts as detected), uncovered (no covering example, this is
coverage debt), accepted (known-equivalent, see ledger), error,
invalid (discarded).
Exit code is 1 if unaccepted survivors exist (or, with --fail-at, if the
score is below the threshold). Mistyped positional paths (a file that
doesn't exist, or a non-.rb file) are an error (exit 2) instead of a
vacuous green run.
Score = (killed + timeout) / (killed + timeout + survived + error).
TDD until green, then verify the tests constrain the behavior:
bundle exec active_mutator --changed --format jsonKill survivors by writing the missing tests. For genuine equivalent mutants:
bundle exec active_mutator --changed --accept-survivors # records to ledger
git add .active_mutator_accepted.json # committed stateAcceptance takes effect on the next run. The accepting run still exits 1.
Scoped accepting runs (--changed, --subject, path args) are safe: the
ledger only prunes entries in files fully scanned by non-narrowed runs, so
out-of-scope acceptances are never dropped.
Agent workflow: see docs/skills/mutation-check.md.
--format stryker-json writes .active_mutator/mutation-report.json in the
Stryker mutation-testing-report-schema
v2 format. Open it in the
Stryker report viewer
for per-file mutant maps with inline diffs, filterable by status.
--format github prints one ::warning annotation per surviving mutant, so
survivors show inline on the PR diff. Pairs with the CI recipe:
bundle exec active_mutator --since origin/main --format github
- Per-PR:
active_mutator --since origin/main --format github(minutes; survivors annotate the PR diff) - Nightly:
active_mutator --force-baseline(full run; also recovers the residual blind spot — constant-reference detection handles the common newly-covering-example case since 0.2)
| Flag | Default | Meaning |
|---|---|---|
--jobs N | half the cores | fork-pool width |
--changed | none | mutate uncommitted + untracked work |
--since REF | none | mutate methods changed since REF |
--subject EXPR | none | subject expression, e.g. Foo#bar, Foo::Bar, Foo::Bar*, Foo#*, Foo.* |
--exclude PAT | none | skip files matching glob during subject discovery (repeatable, gitignore-like) |
--max-mutants N | none | deterministic sample of the first N mutants (quick smoke run on huge scopes; accepted/uncovered mutants count against N) |
--debug-plan | off | print planned mutants as JSON and exit without running |
--allow-empty | off | exit 0 when --since/--subject plan no mutants (default: warn and exit 1) |
--format terminal|json|stryker-json|github | terminal | report format |
--accept-survivors | off | record survivors to the acceptance ledger |
--force-baseline | off | ignore cached coverage map |
--preload-helper FILE / --no-preload-helper | auto-detect | parent spec-helper preload |
--serial-pattern PAT | spec/system/, spec/features/ | covering-path prefixes forced serial |
--spec-path DIR | spec/ | where spec files live, relative to the project root (repeatable; the first use replaces the default spec/), e.g. --spec-path engines/billing/spec --spec-path spec |
--browser-boot-seconds S | 15 | serial-lane timeout bump |
--timeout-factor F / --timeout-floor S | 8 / 10 | mutation timeout budget |
--[no-]adaptive-timeout | on | scale timeout budgets from observed worker wall times (median utilization, grow-only, clamped 1x–4x; --timeout-factor/--timeout-floor set the starting budget) |
--require FILE | none | preload files (repeatable) |
--operator FILE | none | load a custom operator file before analysis (repeatable) |
--[no-]class-level | on | mutate class-level code (macros, constants, DSL/scope lambdas) via class-body subjects |
--fail-at SCORE | none (strict) | exit 0 if score >= SCORE even with survivors (opt-in relaxation for gradual adoption; 0 = report-only) |
--spec-path tells active_mutator where spec files live (coverage
classification, digests, escalation); RSpec's own discovery is still the
project's job — a project with specs under test/ also needs
--default-path test in its .rspec. The serial-lane defaults stay
spec/system/ and spec/features/ regardless of --spec-path; a
custom spec root with browser specs should set --serial-pattern
(e.g. --serial-pattern test/system/) itself.
--debug-plan prints the planned mutant list as one JSON document
({"planned": [...], "pre_resolved": {...}}) and exits without running
anything. A coverage baseline is still built or loaded, since timeouts
and covering examples come from it.
Every active_mutator process sets ENV["ACTIVE_MUTATOR"] = "1". Use it to
guard SimpleCov or other tooling in your spec helper:
SimpleCov.start"rails"unlessENV["ACTIVE_MUTATOR"]Put team-wide settings in .active_mutator.yml at the project root; CLI
flags override file values (--require and --exclude add to the file's
lists; the first --serial-pattern replaces them). Recognized keys:
jobs, format, timeout_factor, timeout_floor,
browser_boot_seconds, fail_at, exclude, serial_patterns,
spec_paths (where spec files live, relative to the project root;
replaces the default spec),
requires, operators (custom operator files, loaded before analysis; see
Custom operators),
preload_helper (a path, or false to skip preload),
adaptive_timeout (true/false),
class_level (true/false, default true — mutate class-level code),
class_level_closure_cap (integer, default 10 — max constants a
class-body mutant may reload before it is skipped).
Unknown keys and wrong types are errors, not silent no-ops.
# .active_mutator.ymljobs: 4exclude:
- lib/generatedserial_patterns:
- spec/system/spec_paths:
- engines/billing/spec
- specfail_at: 90# legacy suite: gate on score instead of zero-survivorsClass-level code — macros (validates, scope, has_many), constants,
and DSL/scope lambdas — IS mutated. Each Zeitwerk-shaped file gets a
… (class body) subject alongside its method subjects, and the same
operator set runs over its class-level statements. Because re-running a
macro accumulates rather than replaces (calling validates twice adds a
second validator), a class-body mutant can't be inserted with class_eval
the way a def mutant is. Instead active_mutator removes the target
constant and re-evaluates the whole mutated file, reloading anything
attached to it (includers, subclasses, extenders) in dependency order. See
docs/guides/how-it-works.md for the full
closure-reload pipeline.
Disable it with --no-class-level (or class_level: false in the config
file). A class-body mutant whose closure can't be reloaded faithfully — the
closure exceeds class_level_closure_cap (default 10), the constant was
reopened elsewhere, or an attacher is anonymous/native — is reported
skipped (progress char -): listed but not counted in the score,
because a mutant we can't insert faithfully must not be called survived or
killed.
Method bodies and Zeitwerk-shaped class bodies are mutated; the remaining limits are:
- Class-body mutation requires a Zeitwerk-shaped file — exactly one top-level class/module per file. Multi-constant files and core-class monkey-patches/reopens are not class-body-mutated (issue #32). Their method bodies still are.
- Most code inside blocks is not mutated.
ActiveSupport::ConcernDSL blocks (included/prepended/class_methods do … end) ARE mutated — their bodies re-run as class-level code in the includer (issue #31). Every other block (has_many :x do … endand anydo … end/{ … }body) is pruned to avoid false survivors from mutating code whose run-time context is unknown. - Constants captured by value go stale. A reference that holds the
target by value rather than by ancestry — an alias (
ALIAS = SomeClass), a registry the class was pushed into, a memoized instance, a class variable captured at load — keeps pointing at the pre-reload object after the closure reload. Such stale references can produce false survivors. - Whole-file re-eval re-runs class-body side effects. The reload re-evaluates the target and every attacher's class body, so non-idempotent load-time side effects (global self-registration, descendant tracking) run twice — which can double or mask a count a spec asserts on.
refine-based modules are not discovered or reloaded. Refinements are anonymous and don't appear in normalancestors.- RSpec only. Test selection, worker setup, and the world-group filter are all RSpec-API-shaped.
- Method-body scope details: plain heredoc bodies ARE mutated (emptied);
interpolated heredocs are skipped.
class << selfbodies are mutated as singleton subjects (class << objand top-levelclass << selfare skipped). Nested defs mutate as part of the enclosing method's body — they get no subject of their own (a directly-inserted mutant would be reverted whenever the outer method re-runs thedef). - The incremental baseline's residual blind spot: constant-reference
detection handles the common case since 0.2; a few residual cases (pure
indirection, partially-covering files, leaf-only or wrapper-only
references,
class ::Foo,Data.define/Struct.newvalue objects) are caught by nightly--force-baseline.
- What is mutation testing?: the concepts. Kill/survive, score, equivalent mutants, further reading.
- How it works: architecture. Subject discovery, source-span edits, the coverage map, the fork pipeline, and honest limits.
- Operator reference: every mutation active_mutator can generate, with before/after examples and what a survivor of each one means.
- Custom operators: write and load your
own mutation operators with
--operator/ theoperators:config key. - Mutation-check skill: the agent-facing workflow. Run, read survivors, strengthen tests, or accept with a reason.
Issues and pull requests welcome. Run bundle exec rspec before sending a
change. Also run bundle exec active_mutator --changed on your own diff
before sending a change that touches lib/. This is a good idea for the
same reason you'd want it run on any other codebase.
If a run dies with baseline suite failed and a LoadError mentioning
bundler-2.x/lib/gems/bundler-2.x/exe/bundle, your Ruby manager (seen with
mise) breaks nested bundle exec: the baseline shells out to bundle exec rspec, and bundler's exported RUBYLIB makes the inner binstub resolve the
wrong path. Skip the outer bundler instead:
ruby -Ilib exe/active_mutator lib --since origin/main # same as the CI mutation jobThe :e2e specs nest bundle exec on their own inside the fixture project,
so on such a machine they fail either way; rely on the CI e2e job for those.
MIT.