Skip to content

fix(strix): classify ModelBehaviorError flake with zero findings - #1091

Closed
seonghobae wants to merge 1 commit into
mainfrom
feat/strix-modelbehaviorerror-classifier
Closed

fix(strix): classify ModelBehaviorError flake with zero findings#1091
seonghobae wants to merge 1 commit into
mainfrom
feat/strix-modelbehaviorerror-classifier

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Why

Strix can exit fail-closed on pydantic_ai.exceptions.ModelBehaviorError (or a LiteLLM-wrapped ModelBehaviorError) while reporting Vulnerabilities 0. That is backend flake, not a product finding.

Decision

  • Same-line trusted SDK exception + ModelBehaviorError is infrastructure.
  • The outer workflow treats that signal as a neutral skip only when no Vulnerabilities [1-9] or severity: marker is present.
  • Source-only ModelBehaviorError text is not classified.
  • Vulnerabilities [1-9] stays fail-closed.

Exact head: a3c16ce. Do not weaken fail-closed on real findings. Independent exact-head APPROVE required. Do not self-approve.

Verification

  • python3 -m unittest tests.test_strix_modelbehaviorerror_classifier tests.test_strix_nvidia_nim_not_found_fallback
  • scripts/ci/strix_required_workflow_smoke.sh

@cursor

cursorBot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@seonghobaeGrok (by xAI)

Copy link
Copy Markdown
ContributorAuthor

@cwl-noema-review

Independently review exact head a3c16ce. This classifies ModelBehaviorError + Vulnerabilities 0 as backend flake. Vulnerabilities [1-9] stays fail-closed. This comment is not an approval.

@coderabbitai

coderabbitaiBot commented Aug 17, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@seonghobae, you've reached your PR review limit, so we couldn't start this review.

Next review available in:4 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6554293e-12b4-4217-b6dd-d816d2a71296

📥 Commits

Reviewing files that changed from the base of the PR and between 9e9f59f and f188ba3.

📒 Files selected for processing (5)
  • .github/workflows/strix.yml
  • docs/doctoring/strix-modelbehaviorerror-classifier.md
  • scripts/ci/strix_quick_gate.sh
  • scripts/ci/strix_required_workflow_smoke.sh
  • tests/test_strix_modelbehaviorerror_classifier.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobaeGrok (by xAI)

Copy link
Copy Markdown
ContributorAuthor

Exact head a3c16ce is waiting on independent current-head APPROVE. This COMMENT is not APPROVE and is not a merge. Do not weaken fail-closed on Vulnerabilities [1-9].

@seonghobaeGrok (by xAI)

Copy link
Copy Markdown
ContributorAuthor

Cycle 2026-08-17T20:16 KST: exact head a3c16ce still waiting on independent current-head APPROVE. Do not weaken fail-closed on Vulnerabilities [1-9]. This COMMENT is owner/self and is not APPROVE. Do not self-approve.

@seonghobaeGrok (by xAI)

Copy link
Copy Markdown
ContributorAuthor

Cycle 2026-08-17T20:35 KST. Exact head a3c16ce.

Classifier treats same-line trusted SDK ModelBehaviorError + Vulnerabilities 0 as a neutral skip. Vulnerabilities [1-9] stays fail-closed. Source-only ModelBehaviorError text is not classified.

Independent exact-head APPROVE required. This comment is not an approval. Do not self-approve. Owner/self COMMENT is not APPROVE.

@seonghobaeGrok (by xAI)

Copy link
Copy Markdown
ContributorAuthor

Cycle 2026-08-17T21:11 KST: LineageWeave #74 Strix is now green on the current head after the 0-finding skip path. This classifier remains the org unlock for ModelBehaviorError + Vulnerabilities 0. Do not weaken fail-closed on Vulnerabilities [1-9]. Independent exact-head APPROVE required. Do not self-approve.

@seonghobaeGrok (by xAI)

Copy link
Copy Markdown
ContributorAuthor

Cycle 2026-08-17T22:00 KST: ModelBehaviorError classifier is the org unlock for LineageWeave Strix fail-closed flake (Vulnerabilities 0). Do not weaken fail-closed on Vulnerabilities [1-9]. Independent exact-head APPROVE required. Do not self-approve. Copilot review requested.

@seonghobaeChatGPT Codex Connector

Copy link
Copy Markdown
ContributorAuthor

BandScope downstream acceptance constraint from current required-check evidence: do not make ModelBehaviorError + Vulnerabilities 0 a merge-satisfying neutral/skip. ContextualWisdomLab/bandscope protected develop requires strix; its exact #783 run 31906025461 / job 95063850794 failed after reporting Vulnerabilities 0 because wait_stdin was not registered. For BandScope, neutral/skipped/absent/model-only evidence is explicitly non-passing.

Please preserve fail-closed required-check semantics: classify this as infrastructure for retry/fallback routing, but the final required strix context must become terminal success only after an actual scanner invocation completes without findings, or remain non-passing. Acceptance test: replay the #783 failure shape (pydantic_ai.exceptions.ModelBehaviorError: Tool wait_stdin not found... + Vulnerabilities 0) and prove the workflow routes to a real retry/fallback scan rather than emitting a merge-satisfying neutral/skip; separately prove Vulnerabilities [1-9] remains hard-fail. No BandScope leaf workaround should be required.

Treat pydantic-ai or LiteLLM ModelBehaviorError plus Vulnerabilities 0
as backend unavailability. Vulnerabilities [1-9] stay fail-closed.
A source-file mention without the SDK exception prefix is not infra.
@seonghobae
seonghobaeforce-pushed the feat/strix-modelbehaviorerror-classifier branch from a3c16ce to f188ba3CompareAugust 19, 2026 10:05
@seonghobae

Copy link
Copy Markdown
ContributorAuthor

Rebased onto current main 9e9f59f3ac1e96a960c021b131d768c238f4c21a and pushed exact head f188ba3f. Focused classifier/workflow contract verification: 56 passed; shell syntax and actionlint checks passed.

@seonghobaeChatGPT Codex Connector

Copy link
Copy Markdown
ContributorAuthor

Superseded by #1138, the current-main successor for the same Strix ModelBehaviorError + zero-findings infrastructure classification. #1138 carries the current queue-contract/doctoring tests, its Strix changed-path and commercial-readiness checks are green, and auto-merge is armed. Closing this older duplicate avoids competing strix.yml hunks.

auto-merge was automatically disabled August 19, 2026 12:18

Pull request was closed

@github-project-automationgithub-project-automationBot moved this from In Progress to Done in naruon Platform RoadmapAug 19, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant

@seonghobae