Skip to content

[P2] Guard verdict policy confidence coercion against non-finite values #562

Description

@stranske

Why

In scripts/langchain/verdict_policy.py lines 73-88, _coerce_confidence and _normalize_confidence do not validate math.isfinite(value). When a provider verdict reports a non-finite confidence string (such as "nan" or "inf"), float('nan') propagates into ProviderVerdict.confidence. In _split_pass_concerns lines 177-181, comparing max_confidence >= CONCERNS_NEEDS_HUMAN_THRESHOLD evaluates to False when confidence is NaN, silently bypassing the needs_human = True human-review trigger on split high-stakes provider verdicts.

Scope

  • Update _coerce_confidence and _normalize_confidence in scripts/langchain/verdict_policy.py to validate math.isfinite and clamp invalid/non-finite confidence to 0.0.
  • Add unit tests in tests/test_main.py verifying split verdicts with non-finite confidence values are safely handled.

Non-Goals

  • Changing CONCERNS_NEEDS_HUMAN_THRESHOLD constant value (0.85).
  • Modifying verdict classification mappings in VERDICT_SEVERITY.
  • Scaffold-only completion does NOT count: modifying evaluate_verdict_policy without sanitizing _coerce_confidence and _normalize_confidence is a failure of this issue.

Tasks

  • Add math.isfinite validation to _coerce_confidence in scripts/langchain/verdict_policy.py
  • Add math.isfinite validation to _normalize_confidence in scripts/langchain/verdict_policy.py
  • Add unit tests in tests/test_main.py covering non-finite confidence strings ("nan", "inf", "-inf")

Acceptance Criteria

  • pytest tests/test_main.py passes with all tests green.
  • python -c "from scripts.langchain.verdict_policy import _coerce_confidence, _normalize_confidence; assert _coerce_confidence('nan') == 0.0 and _normalize_confidence(float('nan')) == 0.0" succeeds.
  • Deliberate-break demonstration: removing math.isfinite validation causes pytest tests/test_main.py to fail on non-finite confidence test cases, and reverting restores pass.

Implementation Notes

References: Code/Audits/Ready/2026-09-07-00-repo-map.md.
Ensure _coerce_confidence returns 0.0 for any unparseable or non-finite float value.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:normalStandard priority within opener-cap discipline

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions