[Follow-up] Update docs/verification-concerns-1307.md: Ensure (PR #1316)

[stranske]

Address Remaining Verification Gaps from PR #1316

Why

PR #1316 addressed issue #1315, but verification returned a CONCERNS verdict due to remaining gaps: high-severity concerns are not fully traceable to concrete resolutions/decisions with repo-local evidence, LLM invocation paths still lack tests that assert config/metadata propagation (and logging on fallback), model classification logic can misclassify names, and some workflows still reference older actions/upload-artifact versions. This follow-up issue closes those gaps with explicit documentation, evidence, tests, and minimal safe refactors.

Source

• Original PR: Codex belt for #1315 #1316
• Parent issue: [Follow-up] Address verification concerns from PR #1307 #1315

Tasks

Documentation & Traceability

• Add resolution links and summaries to each high-severity concern in docs/verification-concerns-1307.md that was resolved by code changes
• Add explicit DECISIONS.md references to each high-severity concern in docs/verification-concerns-1307.md that was resolved by architectural decision
• Add per-concern evidence links pointing to repo-local verification artifacts for each high-severity concern in docs/verification-concerns-1307.md
• Add dated entry to DECISIONS.md for the "missing artifacts" concern including decision statement, rationale, date (ISO format), and evidence link

Re-verification Evidence Artifacts

• Create docs/reverification/README.md explaining how evidence files are produced and what they contain
• Create a dated re-verification evidence markdown file under docs/reverification/ with clearly labeled sections for each high-severity concern
• Update docs/verification-concerns-1307.md to link to the dated re-verification evidence file for each high-severity concern

LLM Invocation Testing

• Add unit tests in tests/test_pr_verifier.py that mock client.invoke and assert config/metadata propagation for all LLM invocation call sites
• Add unit tests in tests/test_followup_issue_generator.py that mock client.invoke and assert config/metadata propagation for all LLM invocation call sites
• Add test in tests/test_followup_issue_generator.py that forces the TypeError path and asserts warning/error log emission with exception info and fallback phrase

Model Classification Refactoring

• Refactor tools/langchain_client._is_reasoning_model to use a start-anchored regex pattern that matches only o followed by digits
• Add unit tests in tests/test_langchain_client.py covering both intended model names (o1, o3, o123) and edge cases (o-mini, gpt-4o, foo, O1) for _is_reasoning_model classification

Logging Improvements

• Update followup_issue_generator.py _invoke_llm to log warning/error when TypeError is raised, including exception text and explicit phrase indicating config/metadata fallback

Deferred Tasks (Requires Human)

• Manual: Update all .github/workflows/*.yml files to use actions/upload-artifact@v7 instead of older versions (v3/v4/v5/v6) and adjust any related inputs for v7 compatibility

Acceptance Criteria

Documentation Traceability

• In docs/verification-concerns-1307.md, every high-severity concern entry includes a Resolution: field with at least one GitHub PR URL (containing /pull/) OR commit URL (containing /commit/) plus a non-empty Summary: line
• In docs/verification-concerns-1307.md, every high-severity concern entry that was resolved by architectural decision includes a Decision: field that references a DECISIONS.md entry by anchor or heading text
• In docs/verification-concerns-1307.md, every high-severity concern entry contains an Evidence: field with a relative link to a repo-local file path under docs/reverification/
• DECISIONS.md contains a dated entry for the "missing artifacts" concern with all of:
  • A one-sentence Decision statement
  • A Rationale section with at least one bullet or paragraph
  • A Date: field in ISO format YYYY-MM-DD
  • An Evidence: link to a repo-local file under docs/reverification/

Re-verification Evidence

• A repo-local re-verification evidence file exists at docs/reverification/<YYYY-MM-DD>-*.md where the filename contains a date prefix in YYYY-MM-DD format
• docs/reverification/README.md exists and describes how the evidence file is produced or what it contains
• docs/verification-concerns-1307.md links to the dated docs/reverification evidence file and each high-severity concern's Evidence: link points to a specific section within that file (markdown anchor) or to a per-concern evidence file under docs/reverification/

LLM Invocation Tests - pr_verifier.py

• tests/test_pr_verifier.py contains at least one test that patches/mocks the underlying LLM client's invoke method used by pr_verifier.py
• The test asserts that the invocation receives the constructed config/metadata argument (not just that _build_llm_config returns it)
• The test covers every call site in pr_verifier.py that triggers an LLM invocation

LLM Invocation Tests - followup_issue_generator.py

• tests/test_followup_issue_generator.py contains at least one test that patches/mocks the underlying LLM client's invoke method used by followup_issue_generator.py
• The test asserts that _invoke_llm passes the constructed config/metadata argument through to invoke
• The test covers every call site in followup_issue_generator.py that triggers an LLM invocation
• tests/test_followup_issue_generator.py includes a test that forces the TypeError path by mocking invoke to raise TypeError when called with config/metadata
• The TypeError test asserts a warning/error log record is emitted containing:
  • The exception text or repr
  • An explicit phrase indicating config/metadata fallback occurred
• The TypeError test verifies that a second invocation without config/metadata occurs (fallback behavior preserved)

Model Classification

• tools/langchain_client._is_reasoning_model uses a regex anchored at the start that matches only model names beginning with o followed immediately by one or more digits (e.g., ^o\d+)
• tools/langchain_client._is_reasoning_model does not match names like o-mini
• tests/test_langchain_client.py includes explicit cases asserting _is_reasoning_model returns True for: o1, o3, o123
• tests/test_langchain_client.py includes explicit cases asserting _is_reasoning_model returns False for: o-mini, gpt-4o, foo, O1

Logging

• followup_issue_generator.py _invoke_llm logs a warning or error when a TypeError is raised due to passing config/metadata to a client that does not accept it
• The log message includes both:
  • The exception text (or repr)
  • An explicit phrase indicating config/metadata was not passed due to fallback

Test Suite

• All newly added tests in tests/test_pr_verifier.py, tests/test_followup_issue_generator.py, and tests/test_langchain_client.py pass individually when run with pytest
• No new test failures are introduced in existing test files when running the full test suite

Implementation Notes

Documentation traceability

• In docs/verification-concerns-1307.md, standardize each high-severity concern to include fields like Resolution: (PR/commit link) + Summary: or Decision: (link to DECISIONS.md entry), plus Evidence: pointing to a repo-local docs/ path.
• Prefer relative links (e.g., docs/reverification/2026-02-07-...md#concern-x) so they work in GitHub UI and remain stable.

Re-verification evidence artifact

• Add docs/reverification/README.md explaining what is captured (commands run, environment assumptions, how to regenerate).
• Add at least one dated evidence markdown file whose filename begins with YYYY-MM-DD and contains clearly labeled sections per high-severity concern to support anchored links.

LLM invocation tests

• Ensure tests validate real invocation paths: patch/mock the actual client.invoke used by each module, call the public/entry methods that trigger LLM calls, and assert the invoke args include the constructed config/metadata.
• Cover all LLM invocation call sites in pr_verifier.py and followup_issue_generator.py, not only helper builders.
• For the fallback behavior in followup_issue_generator.py, use caplog (or equivalent) to assert a warning/error log is emitted, and assert the invoke call happens twice (first with config → TypeError, second without config succeeds).

Model classification

• Update _is_reasoning_model to a start-anchored regex such as r"^o\d+" and add explicit tests including O1 to lock down case-sensitivity expectations.

Workflow changes (manual)

• Keep workflow edits scoped: only bump actions/upload-artifact to @v7 and adjust inputs solely as required for v7 compatibility.
• After manual updates, validate via ripgrep: rg "actions/upload-artifact@v[3-6]" .github/workflows/ should return no matches.

Background (previous attempt context)

• Lack of explicit re-verification evidence for high-severity concerns:

  • Earlier approach referenced workflow artifacts without ensuring repository documentation provided direct evidence.
  • Instead: incorporate embedded test results/artifact files in the repo, or dated repo-local evidence links in docs/verification-concerns-1307.md and DECISIONS.md.

• Insufficient tests for verifying metadata propagation in LLM invocation:

  • Prior iterations only tested the configuration builder (_build_llm_config) but not the use of this configuration in actual client.invoke calls.
  • Instead: mock/intercept client.invoke and assert constructed config/metadata is passed for each invocation path; include coverage for fallback behavior.

Critical Rules

1. Do NOT include "Remaining Unchecked Items" or "Iteration Details" sections unless they contain specific, useful failure context
2. Tasks should be concrete actions, not verification concerns restated
3. Acceptance criteria must be testable (not "all concerns addressed")
4. Keep the main body focused - hide background/history in the collapsible section
5. Do NOT include the entire analysis object - only include specific failure contexts from blockers_to_avoid

[stranske-keepalive]

🤖 Auto-pilot step $((STEP_COUNT + 1)): Analyzing issue for improvements...

Running optimizer inline (not via label trigger).

[stranske-keepalive]

Issue Optimization Suggestions

Review the suggestions below. If you want the agent to apply them, add the
agents:apply-suggestions label to this issue.

Task splitting

• Update docs/verification-concerns-1307.md so every high-severity concern includes: (a) a link to the resolving PR/commit and a short summary of the code/test change OR (b) an explicit reference to a DECISIONS.md entry; also add a per-concern evidence link (log/snippet/file) for re-verification results. (This task combines three distinct activities: adding resolution links, adding decision references, and adding evidence links. Each requires different research and documentation work.). Suggested split: Add resolution links and summaries to each high-severity concern in docs/verification-concerns-1307.md that was resolved by code changes, Add explicit DECISIONS.md references to each high-severity concern in docs/verification-concerns-1307.md that was resolved by architectural decision, Add per-concern evidence links pointing to repo-local verification artifacts for each high-severity concern in docs/verification-concerns-1307.md.
• Add unit tests that assert LLM metadata/config propagation during invocation: mock client.invoke and verify _invoke_llm passes the constructed config/metadata correctly for all call sites in pr_verifier.py and followup_issue_generator.py (not just _build_llm_config). (This task covers two separate modules with potentially different invocation patterns. Each module should be tested independently.). Suggested split: Add unit tests in tests/test_pr_verifier.py that mock client.invoke and assert config/metadata propagation for all LLM invocation call sites, Add unit tests in tests/test_followup_issue_generator.py that mock client.invoke and assert config/metadata propagation for all LLM invocation call sites.
• Refactor tools/langchain_client._is_reasoning_model to use a stricter regex (e.g., ^o\d+) and add unit tests covering intended and non-intended model names (including edge cases like o-mini) to prevent misclassification. (This combines implementation work (refactoring the regex) with test work (adding comprehensive test cases). These should be separate tasks.). Suggested split: Refactor tools/langchain_client._is_reasoning_model to use a start-anchored regex pattern that matches only o followed by digits, Add unit tests in tests/test_langchain_client.py covering both intended model names and edge cases for _is_reasoning_model classification.
• Add a repo-local re-verification evidence artifact: create a dated log/markdown output file under docs/ (or similar) and update docs/verification-concerns-1307.md to link to it; optionally add a lightweight script target in CI that regenerates the file from stored inputs. (This task combines creating evidence files, updating documentation links, and optionally adding CI automation. The optional CI component makes this unclear.). Suggested split: Create a dated re-verification evidence markdown file under docs/reverification with clearly labeled sections for each high-severity concern, Create docs/reverification/README.md explaining how evidence files are produced and what they contain, Update docs/verification-concerns-1307.md to link to the dated re-verification evidence file for each high-severity concern.

Blocked tasks

• Update GitHub workflow files that reference actions/upload-artifact to use actions/upload-artifact@v7 and adjust any related inputs if needed; keep changes limited to version bump and compatibility fixes. (Agent cannot modify .github/workflows/.yml files (protected per AGENT_LIMITATIONS) | Convert to manual task or create a separate issue for human review: 'Manual: Update all .github/workflows/.yml files to use actions/upload-artifact@v7 instead of older versions')
• All workflow YAML files under .github/workflows that reference actions/upload-artifact use actions/upload-artifact@v7 and no workflow references v3/v4/v5/v6 of actions/upload-artifact. (This acceptance criterion depends on modifying workflow files which the agent cannot do (protected per AGENT_LIMITATIONS) | Remove this criterion or mark it as requiring manual verification after human updates workflow files)

Objective acceptance criteria

• The full test suite passes, including the newly added/modified tests, with pytest exit code 0. (Agent cannot guarantee test suite passes due to potential environmental issues, existing failures, or integration problems outside the scope of changes | Rephrase to: 'All newly added tests in test_pr_verifier.py, test_followup_issue_generator.py, and test_langchain_client.py pass individually when run with pytest, and no new test failures are introduced in existing test files')

Missing sections

• None

Formatting issues

• The 'Tasks' section mixes implementation tasks with verification/documentation tasks without clear grouping, making it harder to track progress by category
• Some acceptance criteria are extremely long (100+ words) and could be broken into sub-bullets for readability
• The acceptance criterion about 'every high-severity concern entry includes either (A)... OR (B)...' is complex and could be split into two separate criteria for clarity

Notes

• This issue is well-structured overall with clear traceability to parent issues, comprehensive acceptance criteria, and detailed implementation notes. The main concerns are: (1) one task requires workflow modifications that agents cannot perform, (2) several tasks combine multiple distinct activities that should be split for better tracking, and (3) some acceptance criteria are overly complex. The issue correctly uses collapsible sections for background and avoids the anti-patterns mentioned in the critical rules. The acceptance criteria are mostly objective and testable, with only minor improvements needed. All required sections (Why, Source, Tasks, Acceptance Criteria, Implementation Notes) are present and well-populated.

[agents-workflows-bot]

🤖 Auto-pilot step $((STEP_COUNT + 1)): Applying optimization suggestions...

Running apply inline (not via label trigger).

[agents-workflows-bot]

🤖 Auto-pilot step 1: Issue prepared! Assigning to agent...

Adding agent:codex label. The capability check will run automatically.

⏳ Agent will create a PR shortly.

[stranske-automation-bot]

Codex belt dispatcher queued this issue and created branch codex/issue-1322.

[stranske-keepalive]

🤖 Auto-pilot step 1: Branch ready, waiting for commits

Branch codex/issue-1322 exists but has no commits yet.
✅ Dispatched the Codex belt worker to generate changes.

📊 Stall count: 1/5 (pauses after 5)

⏳ Auto-pilot will check again on next trigger.

[github-actions]

Branch codex/issue-1322 created from main.

Option 1 (Invite) is enforced on issue events. PR creation is disabled by design so you are the PR author. Codex only engages on human-authored PRs.

Please open the PR as the author so Codex can work on it:

• Compare link: https://github.com/stranske/Workflows/compare/main...codex/issue-1322?expand=1
• Suggested title: Codex bootstrap for #1322
• Suggested body (copy/paste):

<!-- meta:issue:1322 -->

@codex Your objective is to satisfy the **Acceptance Criteria** by completing each **Task** within the defined **Scope**.

## Keepalive Next Task

**This round you MUST:**
1. Implement actual code or test changes that advance at least one incomplete task toward acceptance.
2. Commit meaningful source code (.py, .yml, .js, etc.)—not just status/docs updates.
3. Mark a task checkbox complete ONLY after verifying the implementation works.
4. Focus on the FIRST unchecked task unless blocked, then move to the next.

**Guidelines:**
- Keep edits scoped to the current task rather than reshaping the entire PR.
- Use repository instructions, conventions, and tests to validate work.
- Prefer small, reviewable commits; leave clear notes when follow-up is required.
- Do NOT work on unrelated improvements until all PR tasks are complete.

**COVERAGE TASKS - SPECIAL RULES:**
If a task mentions "coverage" or a percentage target (e.g., "≥95%", "to 95%"), you MUST:
1. After adding tests, run TARGETED coverage verification to avoid timeouts:
   - For a specific script like `scripts/foo.py`, run:
     `pytest tests/scripts/test_foo.py --cov=scripts/foo --cov-report=term-missing -m "not slow"`
   - If no matching test file exists, run:
     `pytest tests/ --cov=scripts/foo --cov-report=term-missing -m "not slow" -x`
2. Find the specific script in the coverage output table
3. Verify the `Cover` column shows the target percentage or higher
4. Only mark the task complete if the actual coverage meets the target
5. If coverage is below target, add more tests until it meets the target

IMPORTANT: Always use `-m "not slow"` to skip slow integration tests that may timeout.
IMPORTANT: Use targeted `--cov=scripts/specific_module` instead of `--cov=scripts` for faster feedback.

A coverage task is NOT complete just because you added tests. It is complete ONLY when the coverage command output confirms the target is met.

**The Tasks and Acceptance Criteria are provided in the appendix below.** Work through them in order.

<!-- 
  DO NOT ADD TASK CONTENT BELOW THIS LINE
  
  Task content is dynamically injected from the PR body via the `appendix` input.
  This template file should only contain instructions, not PR-specific tasks.
  
  If you see task content in this file, it was incorrectly committed and should
  be removed. The guard script will flag this as an error.
-->

[stranske]

Opened PR #1323 via Codex belt worker.