Skip to content

fix(runner): runtime errors are detected by string-matching the test's own output #992

Description

@Chemaclass

Summary

bashunit::runner::_classify_runtime_error (src/runner/diagnostics.sh) decides whether a failing test also suffered a shell error by scanning its captured output for a list of literal phrases:

case"$runtime_output"in*"command not found"* | *"unbound variable"* | *"permission denied"* | \
*"no such file or directory"* | *"syntax error"* | *"bad substitution"* | \
… )

The output being scanned is the test's whole captured stream, which includes text bashunit itself wrote. So any failure message that happens to quote one of these phrases is misread as a shell error, and the test is reported twice — once as Failed, once as Error — for one cause.

How it surfaced

While implementing #982 I made assert_true report command not found: <arg> instead of a bare exit code: 127. That is the natural wording, and it immediately produced:

✗ Failed: True missing
✗ Error: True missing

I worked around it by choosing wording that avoids the phrase (unknown command:), which is why #982 does not carry this bug. But the workaround is a landmine for the next person: the most descriptive phrasing for a not-found command is the one that must not be used, and nothing in the code says so except a comment I added at the call site.

The general case

The same collision is reachable without any framework change. A test whose subject is error handling will naturally have these strings in its output — asserting that a script prints a useful message, capturing stderr from a command that failed, snapshotting an error path:

functiontest_reports_a_helpful_error() {
local out; out=$(./my-installer --bogus 2>&1)
assert_contains "SOMETHING""$out"# $out contains "command not found"
}

When that assertion fails, the failure output carries the phrase and the test is additionally flagged as a runtime error. Passing tests are unaffected — the classifier only runs on the failure path — so this is invisible until a test starts failing, which is exactly when clear reporting matters most.

Why this is worth fixing rather than documenting

The double report is not just noise. Failed and Error mean different things in this framework — one is "your assertion did not hold", the other is "your test could not run properly" — and conflating them sends the reader looking for a broken test when the test is fine.

Proposal

The root problem is that one string carries two things: what the command under test emitted, and what bashunit rendered about it. Options, roughly in order of preference:

  1. Classify from the command's output only, before bashunit's own failure rendering is appended. If the two are separable at capture time, this removes the ambiguity entirely rather than narrowing it.
  2. Anchor the patterns. Real shell diagnostics arrive as bash: line N: foo: command not found — prefixed with a source and line. Matching that shape instead of a bare substring would ignore prose that merely mentions the phrase. Narrower, still heuristic.
  3. Use the exit code where one is available. 127/126 are unambiguous and need no string matching. This would not cover every entry in the list, but it would cover the most common ones.

Whatever is chosen, the comment at the assert_true call site in src/assert/core.sh explaining why the phrase is avoided should be removed as part of it.

Constraints

  • Bash 3.0+; case and parameter expansion only, no new syntax surface.
  • This runs on the per-test failure path — no new fork. See .claude/rules/perf-fork-budget.md.
  • Failure output is compared verbatim across the acceptance suite, so any change to what is classified will move snapshots; regenerate deliberately and read the diff.
  • The classifier's existing behaviour on genuine shell errors must not regress — those are the reason it exists.

Acceptance criteria

  • A failing assertion whose message or captured output contains command not found is reported as Failed only, not also Error
  • A test that genuinely hits a shell error is still reported as Error — regression covered by a test
  • Reproduced first: a test that fails while its output legitimately quotes one of the listed phrases
  • The assert_true wording constraint in src/assert/core.sh is lifted, or the comment explaining it is updated to say it is no longer required
  • make sa · make lint · ./bashunit --parallel --simple --strict tests/ · bash build.sh bin -v

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
fix(runner): runtime errors are detected by string-matching the test's own output · Issue #992 · TypedDevs/bashunit · GitHub
Skip to content

fix(runner): runtime errors are detected by string-matching the test's own output #992

Description

@Chemaclass

Summary

bashunit::runner::_classify_runtime_error (src/runner/diagnostics.sh) decides whether a failing test also suffered a shell error by scanning its captured output for a list of literal phrases:

case"$runtime_output"in*"command not found"* | *"unbound variable"* | *"permission denied"* | \
*"no such file or directory"* | *"syntax error"* | *"bad substitution"* | \
… )

The output being scanned is the test's whole captured stream, which includes text bashunit itself wrote. So any failure message that happens to quote one of these phrases is misread as a shell error, and the test is reported twice — once as Failed, once as Error — for one cause.

How it surfaced

While implementing #982 I made assert_true report command not found: <arg> instead of a bare exit code: 127. That is the natural wording, and it immediately produced:

✗ Failed: True missing
✗ Error: True missing

I worked around it by choosing wording that avoids the phrase (unknown command:), which is why #982 does not carry this bug. But the workaround is a landmine for the next person: the most descriptive phrasing for a not-found command is the one that must not be used, and nothing in the code says so except a comment I added at the call site.

The general case

The same collision is reachable without any framework change. A test whose subject is error handling will naturally have these strings in its output — asserting that a script prints a useful message, capturing stderr from a command that failed, snapshotting an error path:

functiontest_reports_a_helpful_error() {
local out; out=$(./my-installer --bogus 2>&1)
assert_contains "SOMETHING""$out"# $out contains "command not found"
}

When that assertion fails, the failure output carries the phrase and the test is additionally flagged as a runtime error. Passing tests are unaffected — the classifier only runs on the failure path — so this is invisible until a test starts failing, which is exactly when clear reporting matters most.

Why this is worth fixing rather than documenting

The double report is not just noise. Failed and Error mean different things in this framework — one is "your assertion did not hold", the other is "your test could not run properly" — and conflating them sends the reader looking for a broken test when the test is fine.

Proposal

The root problem is that one string carries two things: what the command under test emitted, and what bashunit rendered about it. Options, roughly in order of preference:

  1. Classify from the command's output only, before bashunit's own failure rendering is appended. If the two are separable at capture time, this removes the ambiguity entirely rather than narrowing it.
  2. Anchor the patterns. Real shell diagnostics arrive as bash: line N: foo: command not found — prefixed with a source and line. Matching that shape instead of a bare substring would ignore prose that merely mentions the phrase. Narrower, still heuristic.
  3. Use the exit code where one is available. 127/126 are unambiguous and need no string matching. This would not cover every entry in the list, but it would cover the most common ones.

Whatever is chosen, the comment at the assert_true call site in src/assert/core.sh explaining why the phrase is avoided should be removed as part of it.

Constraints

  • Bash 3.0+; case and parameter expansion only, no new syntax surface.
  • This runs on the per-test failure path — no new fork. See .claude/rules/perf-fork-budget.md.
  • Failure output is compared verbatim across the acceptance suite, so any change to what is classified will move snapshots; regenerate deliberately and read the diff.
  • The classifier's existing behaviour on genuine shell errors must not regress — those are the reason it exists.

Acceptance criteria

  • A failing assertion whose message or captured output contains command not found is reported as Failed only, not also Error
  • A test that genuinely hits a shell error is still reported as Error — regression covered by a test
  • Reproduced first: a test that fails while its output legitimately quotes one of the listed phrases
  • The assert_true wording constraint in src/assert/core.sh is lifted, or the comment explaining it is updated to say it is no longer required
  • make sa · make lint · ./bashunit --parallel --simple --strict tests/ · bash build.sh bin -v

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix(runner): runtime errors are detected by string-matching the test's own output · Issue #992 · TypedDevs/bashunit · GitHub
Skip to content

fix(runner): runtime errors are detected by string-matching the test's own output #992

Description

@Chemaclass

Summary

bashunit::runner::_classify_runtime_error (src/runner/diagnostics.sh) decides whether a failing test also suffered a shell error by scanning its captured output for a list of literal phrases:

case"$runtime_output"in*"command not found"* | *"unbound variable"* | *"permission denied"* | \
*"no such file or directory"* | *"syntax error"* | *"bad substitution"* | \
… )

The output being scanned is the test's whole captured stream, which includes text bashunit itself wrote. So any failure message that happens to quote one of these phrases is misread as a shell error, and the test is reported twice — once as Failed, once as Error — for one cause.

How it surfaced

While implementing #982 I made assert_true report command not found: <arg> instead of a bare exit code: 127. That is the natural wording, and it immediately produced:

✗ Failed: True missing
✗ Error: True missing

I worked around it by choosing wording that avoids the phrase (unknown command:), which is why #982 does not carry this bug. But the workaround is a landmine for the next person: the most descriptive phrasing for a not-found command is the one that must not be used, and nothing in the code says so except a comment I added at the call site.

The general case

The same collision is reachable without any framework change. A test whose subject is error handling will naturally have these strings in its output — asserting that a script prints a useful message, capturing stderr from a command that failed, snapshotting an error path:

functiontest_reports_a_helpful_error() {
local out; out=$(./my-installer --bogus 2>&1)
assert_contains "SOMETHING""$out"# $out contains "command not found"
}

When that assertion fails, the failure output carries the phrase and the test is additionally flagged as a runtime error. Passing tests are unaffected — the classifier only runs on the failure path — so this is invisible until a test starts failing, which is exactly when clear reporting matters most.

Why this is worth fixing rather than documenting

The double report is not just noise. Failed and Error mean different things in this framework — one is "your assertion did not hold", the other is "your test could not run properly" — and conflating them sends the reader looking for a broken test when the test is fine.

Proposal

The root problem is that one string carries two things: what the command under test emitted, and what bashunit rendered about it. Options, roughly in order of preference:

  1. Classify from the command's output only, before bashunit's own failure rendering is appended. If the two are separable at capture time, this removes the ambiguity entirely rather than narrowing it.
  2. Anchor the patterns. Real shell diagnostics arrive as bash: line N: foo: command not found — prefixed with a source and line. Matching that shape instead of a bare substring would ignore prose that merely mentions the phrase. Narrower, still heuristic.
  3. Use the exit code where one is available. 127/126 are unambiguous and need no string matching. This would not cover every entry in the list, but it would cover the most common ones.

Whatever is chosen, the comment at the assert_true call site in src/assert/core.sh explaining why the phrase is avoided should be removed as part of it.

Constraints

  • Bash 3.0+; case and parameter expansion only, no new syntax surface.
  • This runs on the per-test failure path — no new fork. See .claude/rules/perf-fork-budget.md.
  • Failure output is compared verbatim across the acceptance suite, so any change to what is classified will move snapshots; regenerate deliberately and read the diff.
  • The classifier's existing behaviour on genuine shell errors must not regress — those are the reason it exists.

Acceptance criteria

  • A failing assertion whose message or captured output contains command not found is reported as Failed only, not also Error
  • A test that genuinely hits a shell error is still reported as Error — regression covered by a test
  • Reproduced first: a test that fails while its output legitimately quotes one of the listed phrases
  • The assert_true wording constraint in src/assert/core.sh is lifted, or the comment explaining it is updated to say it is no longer required
  • make sa · make lint · ./bashunit --parallel --simple --strict tests/ · bash build.sh bin -v

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix(runner): runtime errors are detected by string-matching the test's own output · Issue #992 · TypedDevs/bashunit · GitHub
Skip to content

fix(runner): runtime errors are detected by string-matching the test's own output #992

Description

@Chemaclass

Summary

bashunit::runner::_classify_runtime_error (src/runner/diagnostics.sh) decides whether a failing test also suffered a shell error by scanning its captured output for a list of literal phrases:

case"$runtime_output"in*"command not found"* | *"unbound variable"* | *"permission denied"* | \
*"no such file or directory"* | *"syntax error"* | *"bad substitution"* | \
… )

The output being scanned is the test's whole captured stream, which includes text bashunit itself wrote. So any failure message that happens to quote one of these phrases is misread as a shell error, and the test is reported twice — once as Failed, once as Error — for one cause.

How it surfaced

While implementing #982 I made assert_true report command not found: <arg> instead of a bare exit code: 127. That is the natural wording, and it immediately produced:

✗ Failed: True missing
✗ Error: True missing

I worked around it by choosing wording that avoids the phrase (unknown command:), which is why #982 does not carry this bug. But the workaround is a landmine for the next person: the most descriptive phrasing for a not-found command is the one that must not be used, and nothing in the code says so except a comment I added at the call site.

The general case

The same collision is reachable without any framework change. A test whose subject is error handling will naturally have these strings in its output — asserting that a script prints a useful message, capturing stderr from a command that failed, snapshotting an error path:

functiontest_reports_a_helpful_error() {
local out; out=$(./my-installer --bogus 2>&1)
assert_contains "SOMETHING""$out"# $out contains "command not found"
}

When that assertion fails, the failure output carries the phrase and the test is additionally flagged as a runtime error. Passing tests are unaffected — the classifier only runs on the failure path — so this is invisible until a test starts failing, which is exactly when clear reporting matters most.

Why this is worth fixing rather than documenting

The double report is not just noise. Failed and Error mean different things in this framework — one is "your assertion did not hold", the other is "your test could not run properly" — and conflating them sends the reader looking for a broken test when the test is fine.

Proposal

The root problem is that one string carries two things: what the command under test emitted, and what bashunit rendered about it. Options, roughly in order of preference:

  1. Classify from the command's output only, before bashunit's own failure rendering is appended. If the two are separable at capture time, this removes the ambiguity entirely rather than narrowing it.
  2. Anchor the patterns. Real shell diagnostics arrive as bash: line N: foo: command not found — prefixed with a source and line. Matching that shape instead of a bare substring would ignore prose that merely mentions the phrase. Narrower, still heuristic.
  3. Use the exit code where one is available. 127/126 are unambiguous and need no string matching. This would not cover every entry in the list, but it would cover the most common ones.

Whatever is chosen, the comment at the assert_true call site in src/assert/core.sh explaining why the phrase is avoided should be removed as part of it.

Constraints

  • Bash 3.0+; case and parameter expansion only, no new syntax surface.
  • This runs on the per-test failure path — no new fork. See .claude/rules/perf-fork-budget.md.
  • Failure output is compared verbatim across the acceptance suite, so any change to what is classified will move snapshots; regenerate deliberately and read the diff.
  • The classifier's existing behaviour on genuine shell errors must not regress — those are the reason it exists.

Acceptance criteria

  • A failing assertion whose message or captured output contains command not found is reported as Failed only, not also Error
  • A test that genuinely hits a shell error is still reported as Error — regression covered by a test
  • Reproduced first: a test that fails while its output legitimately quotes one of the listed phrases
  • The assert_true wording constraint in src/assert/core.sh is lifted, or the comment explaining it is updated to say it is no longer required
  • make sa · make lint · ./bashunit --parallel --simple --strict tests/ · bash build.sh bin -v

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' fix(runner): runtime errors are detected by string-matching the test's own output · Issue #992 · TypedDevs/bashunit · GitHub
Skip to content

fix(runner): runtime errors are detected by string-matching the test's own output #992

Description

@Chemaclass

Summary

bashunit::runner::_classify_runtime_error (src/runner/diagnostics.sh) decides whether a failing test also suffered a shell error by scanning its captured output for a list of literal phrases:

case"$runtime_output"in*"command not found"* | *"unbound variable"* | *"permission denied"* | \
*"no such file or directory"* | *"syntax error"* | *"bad substitution"* | \
… )

The output being scanned is the test's whole captured stream, which includes text bashunit itself wrote. So any failure message that happens to quote one of these phrases is misread as a shell error, and the test is reported twice — once as Failed, once as Error — for one cause.

How it surfaced

While implementing #982 I made assert_true report command not found: <arg> instead of a bare exit code: 127. That is the natural wording, and it immediately produced:

✗ Failed: True missing
✗ Error: True missing

I worked around it by choosing wording that avoids the phrase (unknown command:), which is why #982 does not carry this bug. But the workaround is a landmine for the next person: the most descriptive phrasing for a not-found command is the one that must not be used, and nothing in the code says so except a comment I added at the call site.

The general case

The same collision is reachable without any framework change. A test whose subject is error handling will naturally have these strings in its output — asserting that a script prints a useful message, capturing stderr from a command that failed, snapshotting an error path:

functiontest_reports_a_helpful_error() {
local out; out=$(./my-installer --bogus 2>&1)
assert_contains "SOMETHING""$out"# $out contains "command not found"
}

When that assertion fails, the failure output carries the phrase and the test is additionally flagged as a runtime error. Passing tests are unaffected — the classifier only runs on the failure path — so this is invisible until a test starts failing, which is exactly when clear reporting matters most.

Why this is worth fixing rather than documenting

The double report is not just noise. Failed and Error mean different things in this framework — one is "your assertion did not hold", the other is "your test could not run properly" — and conflating them sends the reader looking for a broken test when the test is fine.

Proposal

The root problem is that one string carries two things: what the command under test emitted, and what bashunit rendered about it. Options, roughly in order of preference:

  1. Classify from the command's output only, before bashunit's own failure rendering is appended. If the two are separable at capture time, this removes the ambiguity entirely rather than narrowing it.
  2. Anchor the patterns. Real shell diagnostics arrive as bash: line N: foo: command not found — prefixed with a source and line. Matching that shape instead of a bare substring would ignore prose that merely mentions the phrase. Narrower, still heuristic.
  3. Use the exit code where one is available. 127/126 are unambiguous and need no string matching. This would not cover every entry in the list, but it would cover the most common ones.

Whatever is chosen, the comment at the assert_true call site in src/assert/core.sh explaining why the phrase is avoided should be removed as part of it.

Constraints

  • Bash 3.0+; case and parameter expansion only, no new syntax surface.
  • This runs on the per-test failure path — no new fork. See .claude/rules/perf-fork-budget.md.
  • Failure output is compared verbatim across the acceptance suite, so any change to what is classified will move snapshots; regenerate deliberately and read the diff.
  • The classifier's existing behaviour on genuine shell errors must not regress — those are the reason it exists.

Acceptance criteria

  • A failing assertion whose message or captured output contains command not found is reported as Failed only, not also Error
  • A test that genuinely hits a shell error is still reported as Error — regression covered by a test
  • Reproduced first: a test that fails while its output legitimately quotes one of the listed phrases
  • The assert_true wording constraint in src/assert/core.sh is lifted, or the comment explaining it is updated to say it is no longer required
  • make sa · make lint · ./bashunit --parallel --simple --strict tests/ · bash build.sh bin -v

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix(runner): runtime errors are detected by string-matching the test's own output · Issue #992 · TypedDevs/bashunit · GitHub
Skip to content

fix(runner): runtime errors are detected by string-matching the test's own output #992

Description

@Chemaclass

Summary

bashunit::runner::_classify_runtime_error (src/runner/diagnostics.sh) decides whether a failing test also suffered a shell error by scanning its captured output for a list of literal phrases:

case"$runtime_output"in*"command not found"* | *"unbound variable"* | *"permission denied"* | \
*"no such file or directory"* | *"syntax error"* | *"bad substitution"* | \
… )

The output being scanned is the test's whole captured stream, which includes text bashunit itself wrote. So any failure message that happens to quote one of these phrases is misread as a shell error, and the test is reported twice — once as Failed, once as Error — for one cause.

How it surfaced

While implementing #982 I made assert_true report command not found: <arg> instead of a bare exit code: 127. That is the natural wording, and it immediately produced:

✗ Failed: True missing
✗ Error: True missing

I worked around it by choosing wording that avoids the phrase (unknown command:), which is why #982 does not carry this bug. But the workaround is a landmine for the next person: the most descriptive phrasing for a not-found command is the one that must not be used, and nothing in the code says so except a comment I added at the call site.

The general case

The same collision is reachable without any framework change. A test whose subject is error handling will naturally have these strings in its output — asserting that a script prints a useful message, capturing stderr from a command that failed, snapshotting an error path:

functiontest_reports_a_helpful_error() {
local out; out=$(./my-installer --bogus 2>&1)
assert_contains "SOMETHING""$out"# $out contains "command not found"
}

When that assertion fails, the failure output carries the phrase and the test is additionally flagged as a runtime error. Passing tests are unaffected — the classifier only runs on the failure path — so this is invisible until a test starts failing, which is exactly when clear reporting matters most.

Why this is worth fixing rather than documenting

The double report is not just noise. Failed and Error mean different things in this framework — one is "your assertion did not hold", the other is "your test could not run properly" — and conflating them sends the reader looking for a broken test when the test is fine.

Proposal

The root problem is that one string carries two things: what the command under test emitted, and what bashunit rendered about it. Options, roughly in order of preference:

  1. Classify from the command's output only, before bashunit's own failure rendering is appended. If the two are separable at capture time, this removes the ambiguity entirely rather than narrowing it.
  2. Anchor the patterns. Real shell diagnostics arrive as bash: line N: foo: command not found — prefixed with a source and line. Matching that shape instead of a bare substring would ignore prose that merely mentions the phrase. Narrower, still heuristic.
  3. Use the exit code where one is available. 127/126 are unambiguous and need no string matching. This would not cover every entry in the list, but it would cover the most common ones.

Whatever is chosen, the comment at the assert_true call site in src/assert/core.sh explaining why the phrase is avoided should be removed as part of it.

Constraints

  • Bash 3.0+; case and parameter expansion only, no new syntax surface.
  • This runs on the per-test failure path — no new fork. See .claude/rules/perf-fork-budget.md.
  • Failure output is compared verbatim across the acceptance suite, so any change to what is classified will move snapshots; regenerate deliberately and read the diff.
  • The classifier's existing behaviour on genuine shell errors must not regress — those are the reason it exists.

Acceptance criteria

  • A failing assertion whose message or captured output contains command not found is reported as Failed only, not also Error
  • A test that genuinely hits a shell error is still reported as Error — regression covered by a test
  • Reproduced first: a test that fails while its output legitimately quotes one of the listed phrases
  • The assert_true wording constraint in src/assert/core.sh is lifted, or the comment explaining it is updated to say it is no longer required
  • make sa · make lint · ./bashunit --parallel --simple --strict tests/ · bash build.sh bin -v

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix(runner): runtime errors are detected by string-matching the test's own output · Issue #992 · TypedDevs/bashunit · GitHub
Skip to content

fix(runner): runtime errors are detected by string-matching the test's own output #992

Description

@Chemaclass

Summary

bashunit::runner::_classify_runtime_error (src/runner/diagnostics.sh) decides whether a failing test also suffered a shell error by scanning its captured output for a list of literal phrases:

case"$runtime_output"in*"command not found"* | *"unbound variable"* | *"permission denied"* | \
*"no such file or directory"* | *"syntax error"* | *"bad substitution"* | \
… )

The output being scanned is the test's whole captured stream, which includes text bashunit itself wrote. So any failure message that happens to quote one of these phrases is misread as a shell error, and the test is reported twice — once as Failed, once as Error — for one cause.

How it surfaced

While implementing #982 I made assert_true report command not found: <arg> instead of a bare exit code: 127. That is the natural wording, and it immediately produced:

✗ Failed: True missing
✗ Error: True missing

I worked around it by choosing wording that avoids the phrase (unknown command:), which is why #982 does not carry this bug. But the workaround is a landmine for the next person: the most descriptive phrasing for a not-found command is the one that must not be used, and nothing in the code says so except a comment I added at the call site.

The general case

The same collision is reachable without any framework change. A test whose subject is error handling will naturally have these strings in its output — asserting that a script prints a useful message, capturing stderr from a command that failed, snapshotting an error path:

functiontest_reports_a_helpful_error() {
local out; out=$(./my-installer --bogus 2>&1)
assert_contains "SOMETHING""$out"# $out contains "command not found"
}

When that assertion fails, the failure output carries the phrase and the test is additionally flagged as a runtime error. Passing tests are unaffected — the classifier only runs on the failure path — so this is invisible until a test starts failing, which is exactly when clear reporting matters most.

Why this is worth fixing rather than documenting

The double report is not just noise. Failed and Error mean different things in this framework — one is "your assertion did not hold", the other is "your test could not run properly" — and conflating them sends the reader looking for a broken test when the test is fine.

Proposal

The root problem is that one string carries two things: what the command under test emitted, and what bashunit rendered about it. Options, roughly in order of preference:

  1. Classify from the command's output only, before bashunit's own failure rendering is appended. If the two are separable at capture time, this removes the ambiguity entirely rather than narrowing it.
  2. Anchor the patterns. Real shell diagnostics arrive as bash: line N: foo: command not found — prefixed with a source and line. Matching that shape instead of a bare substring would ignore prose that merely mentions the phrase. Narrower, still heuristic.
  3. Use the exit code where one is available. 127/126 are unambiguous and need no string matching. This would not cover every entry in the list, but it would cover the most common ones.

Whatever is chosen, the comment at the assert_true call site in src/assert/core.sh explaining why the phrase is avoided should be removed as part of it.

Constraints

  • Bash 3.0+; case and parameter expansion only, no new syntax surface.
  • This runs on the per-test failure path — no new fork. See .claude/rules/perf-fork-budget.md.
  • Failure output is compared verbatim across the acceptance suite, so any change to what is classified will move snapshots; regenerate deliberately and read the diff.
  • The classifier's existing behaviour on genuine shell errors must not regress — those are the reason it exists.

Acceptance criteria

  • A failing assertion whose message or captured output contains command not found is reported as Failed only, not also Error
  • A test that genuinely hits a shell error is still reported as Error — regression covered by a test
  • Reproduced first: a test that fails while its output legitimately quotes one of the listed phrases
  • The assert_true wording constraint in src/assert/core.sh is lifted, or the comment explaining it is updated to say it is no longer required
  • make sa · make lint · ./bashunit --parallel --simple --strict tests/ · bash build.sh bin -v

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); fix(runner): runtime errors are detected by string-matching the test's own output · Issue #992 · TypedDevs/bashunit · GitHub
Skip to content

fix(runner): runtime errors are detected by string-matching the test's own output #992

Description

@Chemaclass

Summary

bashunit::runner::_classify_runtime_error (src/runner/diagnostics.sh) decides whether a failing test also suffered a shell error by scanning its captured output for a list of literal phrases:

case"$runtime_output"in*"command not found"* | *"unbound variable"* | *"permission denied"* | \
*"no such file or directory"* | *"syntax error"* | *"bad substitution"* | \
… )

The output being scanned is the test's whole captured stream, which includes text bashunit itself wrote. So any failure message that happens to quote one of these phrases is misread as a shell error, and the test is reported twice — once as Failed, once as Error — for one cause.

How it surfaced

While implementing #982 I made assert_true report command not found: <arg> instead of a bare exit code: 127. That is the natural wording, and it immediately produced:

✗ Failed: True missing
✗ Error: True missing

I worked around it by choosing wording that avoids the phrase (unknown command:), which is why #982 does not carry this bug. But the workaround is a landmine for the next person: the most descriptive phrasing for a not-found command is the one that must not be used, and nothing in the code says so except a comment I added at the call site.

The general case

The same collision is reachable without any framework change. A test whose subject is error handling will naturally have these strings in its output — asserting that a script prints a useful message, capturing stderr from a command that failed, snapshotting an error path:

functiontest_reports_a_helpful_error() {
local out; out=$(./my-installer --bogus 2>&1)
assert_contains "SOMETHING""$out"# $out contains "command not found"
}

When that assertion fails, the failure output carries the phrase and the test is additionally flagged as a runtime error. Passing tests are unaffected — the classifier only runs on the failure path — so this is invisible until a test starts failing, which is exactly when clear reporting matters most.

Why this is worth fixing rather than documenting

The double report is not just noise. Failed and Error mean different things in this framework — one is "your assertion did not hold", the other is "your test could not run properly" — and conflating them sends the reader looking for a broken test when the test is fine.

Proposal

The root problem is that one string carries two things: what the command under test emitted, and what bashunit rendered about it. Options, roughly in order of preference:

  1. Classify from the command's output only, before bashunit's own failure rendering is appended. If the two are separable at capture time, this removes the ambiguity entirely rather than narrowing it.
  2. Anchor the patterns. Real shell diagnostics arrive as bash: line N: foo: command not found — prefixed with a source and line. Matching that shape instead of a bare substring would ignore prose that merely mentions the phrase. Narrower, still heuristic.
  3. Use the exit code where one is available. 127/126 are unambiguous and need no string matching. This would not cover every entry in the list, but it would cover the most common ones.

Whatever is chosen, the comment at the assert_true call site in src/assert/core.sh explaining why the phrase is avoided should be removed as part of it.

Constraints

  • Bash 3.0+; case and parameter expansion only, no new syntax surface.
  • This runs on the per-test failure path — no new fork. See .claude/rules/perf-fork-budget.md.
  • Failure output is compared verbatim across the acceptance suite, so any change to what is classified will move snapshots; regenerate deliberately and read the diff.
  • The classifier's existing behaviour on genuine shell errors must not regress — those are the reason it exists.

Acceptance criteria

  • A failing assertion whose message or captured output contains command not found is reported as Failed only, not also Error
  • A test that genuinely hits a shell error is still reported as Error — regression covered by a test
  • Reproduced first: a test that fails while its output legitimately quotes one of the listed phrases
  • The assert_true wording constraint in src/assert/core.sh is lifted, or the comment explaining it is updated to say it is no longer required
  • make sa · make lint · ./bashunit --parallel --simple --strict tests/ · bash build.sh bin -v

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions