Skip to content

Fix agent going inactive on --increment speed benchmarks - #71

Open
urbanadventurer wants to merge 2 commits into
hashtopolis:masterfrom
urbanadventurer:fix/speed-benchmark-increment
Open

Fix agent going inactive on --increment speed benchmarks#71
urbanadventurer wants to merge 2 commits into
hashtopolis:masterfrom
urbanadventurer:fix/speed-benchmark-increment

Conversation

@urbanadventurer

Copy link
Copy Markdown

Problem

An agent assigned a mask/brute-force task that uses --increment becomes
Inactive and never recovers. The server log shows the benchmark being retried
and failing over and over:

Agent 2 sent error: Speed benchmark failed!
Agent 2 sent error: Speed benchmark failed!
...

Example task (from a supertask):

-a 3 #HL# ?1 --increment --increment-min=1 --increment-max=6 -1 ?l?u?d?s

Root cause

The speed benchmark runs hashcat with --progress-only, and hashcat refuses to
combine --progress-only with --increment:

$ hashcat --progress-only -a 3 hash '?1' --increment --increment-min=1 --increment-max=6 -1 '?l?u?d?s'
Increment is not allowed in combination with --progress-only.
$ echo $?
255

So the benchmark exits non-zero for every increment task. The agent turns
that into a clientError. On the server, APIClientError.php deactivates an
agent on its first error whenever ignoreErrors == 0 (the default), so the
agent flips to isActive = 0 and can only be revived by hand. The agent loop
keeps re-requesting the task every ~10 s, reproducing the repeated log lines.

A second, smaller bug makes this hard to diagnose: in run_speed_benchmark,
output is reset to b'' immediately before the try, so on
CalledProcessError the code logs output.decode() (always empty) and never
looks at e.output, which holds hashcat's real message. The local error log is
always blank on any speed-benchmark failure.

Fix

Commit 1 — the increment benchmark failure:

  1. Strip the increment flags before running --progress-only. The measured
    cracking speed is independent of the mask length / increment range, so the
    benchmark stays valid. A new strip_increment() helper in helpers.py is
    applied to the speed-benchmark command only (not to --keyspace or the
    --runtime run benchmark, which accept --increment).
  2. Capture hashcat's error output — read e.output instead of the
    always-empty output, so the error log line for a genuine benchmark failure
    is no longer blank. The error sent to the server is unchanged
    ("Speed benchmark failed!"), to avoid bloating server-side AgentError
    records with multi-line hashcat output.

strip_increment() intentionally iterates the tokens directly rather than
routing them through the existing clean_list() helper (see commit 2 for why).

Commit 2 — related robustness bugs found while fixing the above (kept as a
separate commit so it can be reviewed or split off on its own):

  1. clean_list() deleted from a list while iterating it (del element_list[index]
    inside for part in element_list), which skips elements and leaves real
    tokens — and empty strings — behind. On the runs of spaces in the assembled
    benchmark command this dropped the attack mask itself (e.g.
    -a 3 #HL# ?d?d?d?d?d → bare -a 3), after which hashcat falls back to its
    built-in default mask, which references ?1, and fails with
    Custom-charset 1 is undefined. Rewritten as [p for p in element_list if p].
    This also stops get_wordlist() from raising IndexError on the empty
    strings clean_list() used to leave (its part[0] check assumes none remain).
    clean_list()'s only two callers (get_wordlist, get_rules_and_hl) use the
    return value, so returning a fresh list instead of mutating in place changes
    nothing for them.
  2. measure_keyspace() had the same swallowed-output bug as (2) — fixed the
    same way (e.output).
  3. The PRINCE branch of run_speed_benchmark() called
    get_rules_and_hl(update_files(task['attackcmd'])) with one argument; the
    signature is (command, alias), so it raised TypeError. Pass the hashlist
    alias, matching the correct call in the preprocessor path. (PRINCE is
    deprecated; included only because it's a guaranteed crash if that path runs.)

Alternatives considered

  • Skip the speed benchmark for increment tasks rather than rewrite the
    command — but the agent still needs a speed figure for chunk sizing, so
    stripping --increment and measuring the base mask is the least-disruptive fix
    (cracking speed for a mask attack is independent of mask length / the increment
    range, so the number stays valid).
  • Fix it server-side by not requesting a "speed" benchmark for increment
    tasks — larger blast radius, and the agent would still mishandle a
    --progress-only+--increment command it was handed. Keeping the fix in the
    agent that builds the command is the more contained change.

Testing

  • tests/test_helpers.py: 8 unit tests, all passing — strip_increment (=
    form, space-separated form, short -i, no-op cases, and a regression test
    that the mask survives runs of spaces) and clean_list (consecutive empties,
    full list untouched).
  • Verified against hashcat v6.2.5 that the raw increment command errors with
    "Increment is not allowed in combination with --progress-only." (exit 255)
    and that strip_increment clears it.
  • Verified end-to-end on a live 6.2.6 deployment: after the fix the agent
    benchmarks both a real increment task and a plain-digit mask task cleanly,
    cracks, and stays active — zero benchmark errors.

Notes / follow-ups (not in this PR)

  • Hashcat masks (?d, ?l, …) are interpolated into the command string that is
    run with shell=True, unquoted; ? and * are shell globs, so a mask can be
    glob-expanded against files in the cracker/working directory. It rarely
    matches real filenames, but quoting masks would remove the hazard.
  • --keyspace accepts --increment but reports the keyspace of only the base
    mask, so chunking can under-count for increment tasks.
  • The server deactivating an agent on a single benchmark-measurement error is
    arguably too aggressive; treating measurement errors as task-level rather than
    agent-level would be a more robust server-side change, but that's out of scope
    here.

urbanadventurerand others added 2 commits August 17, 2026 14:54
hashcat refuses to run --progress-only together with --increment
("Increment is not allowed in combination with --progress-only."),
so the speed benchmark exited non-zero for every increment task. The
agent sent a clientError, and the server deactivates an agent on its
first error when ignoreErrors=0 (the default), leaving the agent stuck
Inactive until re-enabled by hand.
Strip the increment flags before running --progress-only; the measured
cracking speed is independent of the increment range, so the benchmark
stays valid. Add strip_increment() with unit tests.
strip_increment iterates the tokens directly and must NOT route them
through clean_list(): the assembled benchmark command contains runs of
spaces, and clean_list() deletes real tokens when it hits consecutive
empty entries (it mutates the list while iterating it). That bug dropped
the attack mask entirely (e.g. -a 3 ?d?d?d?d?d became bare -a 3), after
which hashcat fell back to its default ?1 mask and failed with
"Custom-charset 1 is undefined".
Also capture hashcat's real output on failure: 'output' was reset to
b'' before the try block, so CalledProcessError.output was discarded and
the error log line was always empty. Read e.output instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found while fixing the speed-benchmark increment issue:
* clean_list() deleted entries from a list while iterating it, which
skips elements and leaves real tokens (and empty strings) behind.
Rewrite it as a filtered comprehension. This also stops get_wordlist()
from raising IndexError on the empty strings clean_list used to leave.
* measure_keyspace() reset 'output' to b'' before the try, so on failure
it logged an always-empty Output and discarded hashcat's real message
in CalledProcessError.output. Read e.output, matching run_speed_benchmark.
* run_speed_benchmark()'s PRINCE branch called get_rules_and_hl() with
only one argument; the function requires (command, alias), so it raised
TypeError. Pass the hashlist alias, as the preprocessor path already does.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@urbanadventurer