[Follow-up] Modify .github/workflows/agents-auto-pilot.yml to (PR #1447)

[stranske]

Workflow LLM Dependency Pinning and Test Enforcement

Why

PR #1447 addressed issue #1437, but verification failed because the actual workflow files were not updated to use pinned LLM dependency installs and proper pip caching, and the enforcement tests were effectively skipped in normal CI. This follow-up closes the remaining gaps by updating the live GitHub Actions workflows, enabling non-skipped enforcement tests, and hardening check_prompt_injection() to avoid runtime errors on malformed detector outputs.

Source

• Original PR: Codex belt for #1437 #1447
• Parent issue: Issue: Pin LangChain deps and enable pip caching in high-volume workflows (auto-pilot + verifier) #1437

Tasks

Test Updates (Agent-Executable)

• Remove the unconditional skip logic from tests/workflows/test_workflow_llm_installs.py so tests run in normal CI
• Add assertion to tests/workflows/test_workflow_llm_installs.py to verify agents-auto-pilot workflow contains the pinned install step and lacks unpinned langchain commands
• Add assertions to tests/workflows/test_workflow_llm_installs.py to verify reusable-agents-verifier workflow contains pinned install steps in both evaluate and compare modes
• Add assertions to tests/workflows/test_workflow_llm_installs.py to verify both workflows include actions/cache@v4 with correct cache keys containing Python version and requirements hash
• Add unit tests to validate check_prompt_injection() handles malformed detector outputs without raising exceptions
• Add unit tests to validate check_prompt_injection() correctly handles known-good reason code formats

Code Hardening (Agent-Executable)

• Update check_prompt_injection() to validate detector output format before parsing (check prefix/delimiters and guard integer conversion)
• Implement safe fallback path in check_prompt_injection() for unexpected formats (no exception, defined behavior)

Workflow Update Snippets (Agent-Executable)

• Create docs/workflow-updates/agents-auto-pilot-changes.yml with complete YAML snippet for pip install step and cache configuration
• Create docs/workflow-updates/reusable-agents-verifier-changes.yml with complete YAML snippets for evaluate/compare mode pip install steps and cache configuration
• Add docs/workflow-updates/README.md with instructions for applying the YAML snippets to protected workflow files

Deferred Tasks (Requires Human)

Note: The following tasks require modifications to protected .github/workflows/ files that cannot be modified by the agent. YAML snippets will be provided in docs/workflow-updates/ for human review and application.

• Add a dedicated step to .github/workflows/agents-auto-pilot.yml that executes pip install -r tools/requirements-llm.txt
• Remove all unpinned pip install langchain* commands from .github/workflows/agents-auto-pilot.yml
• Add a step to .github/workflows/reusable-agents-verifier.yml evaluate mode that executes pip install -r .workflows-lib/tools/requirements-llm.txt
• Add a step to .github/workflows/reusable-agents-verifier.yml compare mode that executes pip install -r .workflows-lib/tools/requirements-llm.txt
• Remove all unpinned pip install langchain* commands from evaluate and compare modes in .github/workflows/reusable-agents-verifier.yml
• Add actions/cache@v4 pip cache step to .github/workflows/agents-auto-pilot.yml with cache key including Python version and hashFiles('tools/requirements-llm.txt')
• Add actions/cache@v4 pip cache step to .github/workflows/reusable-agents-verifier.yml with cache key including Python version and hashFiles('.workflows-lib/tools/requirements-llm.txt')

Acceptance Criteria

Workflow Files (Human-Applied)

• .github/workflows/agents-auto-pilot.yml contains a step that executes exactly pip install -r tools/requirements-llm.txt (as a run command in the workflow)
• .github/workflows/agents-auto-pilot.yml does NOT contain any pip install commands that install langchain via an unpinned specifier, including (but not limited to) pip install langchain, pip install langchain*, pip install langchain== (missing version), or pip install git+...langchain...
• .github/workflows/reusable-agents-verifier.yml includes a step that executes exactly pip install -r .workflows-lib/tools/requirements-llm.txt in the evaluate mode execution path
• .github/workflows/reusable-agents-verifier.yml includes a step that executes exactly pip install -r .workflows-lib/tools/requirements-llm.txt in the compare mode execution path
• .github/workflows/reusable-agents-verifier.yml does NOT contain any pip install commands that install langchain via an unpinned specifier in the evaluate or compare execution paths (e.g., pip install langchain*, pip install langchain, pip install langchain-community without a version pin)
• .github/workflows/agents-auto-pilot.yml contains an actions/cache@v4 step that caches pip (cache path includes pip cache directory) and uses a cache key that includes BOTH the Python version and ${{ hashFiles('tools/requirements-llm.txt') }}
• .github/workflows/reusable-agents-verifier.yml contains an actions/cache@v4 step that caches pip (cache path includes pip cache directory) and uses a cache key that includes BOTH the Python version and ${{ hashFiles('.workflows-lib/tools/requirements-llm.txt') }}

Test Enforcement

• tests/workflows/test_workflow_llm_installs.py is executed under normal CI (i.e., it is not unconditionally skipped) and will fail if the agents-auto-pilot workflow lacks the pinned requirements install step
• tests/workflows/test_workflow_llm_installs.py asserts that .github/workflows/agents-auto-pilot.yml contains pip install -r tools/requirements-llm.txt AND asserts it does not contain any floating pip install langchain*/unversioned langchain install commands
• tests/workflows/test_workflow_llm_installs.py asserts that .github/workflows/reusable-agents-verifier.yml contains pip install -r .workflows-lib/tools/requirements-llm.txt for BOTH evaluate and compare modes (separate assertions per mode/path, not a single generic check)
• tests/workflows/test_workflow_llm_installs.py asserts that BOTH workflows include an actions/cache@v4 step and that each cache key includes Python version + the correct hashFiles(...) call for the correct requirements path

Code Hardening

• check_prompt_injection() does not raise an exception when detect_prompt_injection returns an unexpected or malformed reason code (e.g., empty string, missing delimiter/prefix, non-integer suffix); instead it follows a defined fallback path (e.g., returns a safe default or logs and continues)
• check_prompt_injection() continues to correctly handle valid reason code formats by producing the same outcome as before for at least one known-good reason code value
• Unit tests pass that verify check_prompt_injection() behavior with both malformed and known-good detector outputs

Documentation Deliverables

• docs/workflow-updates/agents-auto-pilot-changes.yml exists and contains valid YAML snippet for pip install and cache steps
• docs/workflow-updates/reusable-agents-verifier-changes.yml exists and contains valid YAML snippets for evaluate/compare mode pip install and cache steps
• docs/workflow-updates/README.md exists and provides clear instructions for applying the YAML snippets to the protected workflow files

Implementation Notes

Agent-Executable Work

Tests:

• Update tests/workflows/test_workflow_llm_installs.py to remove unconditional skip logic (e.g., gating on AGENT_ENV == 'agent-high-privilege')
• Add explicit assertions for:
  • Presence of the exact pinned pip install -r ... commands
  • Absence of floating/unpinned pip install ... langchain... patterns
  • Presence and correctness of actions/cache@v4 and cache key contents (Python version + correct hashFiles(...))
• Prefer robust checks (YAML parsing where feasible) rather than only substring matching; if regex scanning is used, ensure it cannot be bypassed by formatting differences

check_prompt_injection() hardening:

• Validate the detector output format before parsing (e.g., check prefix/delimiters; guard integer conversion)
• On unexpected formats, follow a safe fallback path (no exception)
• Add/extend unit tests by stubbing/mocking detect_prompt_injection for both malformed and known-good return values

Workflow Update Snippets:
Create complete, ready-to-apply YAML snippets in docs/workflow-updates/ directory:

• agents-auto-pilot-changes.yml - Contains the pip install step and cache configuration for agents-auto-pilot workflow
• reusable-agents-verifier-changes.yml - Contains the pip install steps for evaluate/compare modes and cache configuration
• README.md - Instructions for human to apply these snippets to the protected workflow files

Human-Required Work

Workflow Changes:

• Make changes directly in the live workflow files:
  • .github/workflows/agents-auto-pilot.yml: add a dedicated step with run: pip install -r tools/requirements-llm.txt and remove any separate pip install langchain... commands
  • .github/workflows/reusable-agents-verifier.yml: ensure both evaluate and compare gated paths include run: pip install -r .workflows-lib/tools/requirements-llm.txt, and remove/avoid unpinned langchain installs in those paths
• Add actions/cache@v4 for pip caching in both workflows:
  • Use a pip cache path (commonly ~/.cache/pip)
  • Cache key must combine Python version + requirements hash:
    • agents-auto-pilot: include hashFiles('tools/requirements-llm.txt')
    • reusable-agents-verifier: include hashFiles('.workflows-lib/tools/requirements-llm.txt')

Background (previous attempt context)

• The agent only updated documentation snippets and tests without modifying the actual workflow files (.github/workflows/agents-auto-pilot.yml and .github/workflows/reusable-agents-verifier.yml). This failed because documentation changes did not propagate to the live workflow configurations.
• The test for workflow enforcement is skipped under standard environments. Skipping tests unless AGENT_ENV is agent-high-privilege makes it unreliable for catching configuration errors in typical CI/dev environments.
• The PR adds a 'needs-human' documentation comment (agents/codex-1447.md) acknowledging workflow updates are required, but documentation is not a substitute for implementing the acceptance criteria (post-merge verification is about actual code state).
• Test file tests/workflows/test_workflow_llm_installs.py skips all assertions when AGENT_ENV != 'agent-high-privilege', meaning the tests cannot verify the acceptance criteria are met.

[agents-workflows-bot]

🤖 Auto-pilot step $((STEP_COUNT + 1)): Analyzing issue for improvements...

Running optimizer inline (not via label trigger).

[agents-workflows-bot]

Issue Optimization Suggestions

Review the suggestions below. If you want the agent to apply them, add the
agents:apply-suggestions label to this issue.

Task splitting

• Update .github/workflows/agents-auto-pilot.yml to install LLM dependencies using the pinned requirements file by adding a step that runs pip install -r tools/requirements-llm.txt, and remove any runtime pip install langchain* (or other unpinned langchain) commands. (This task combines two distinct actions (adding a step and removing commands) that should be tracked separately for clarity.). Suggested split: Add a dedicated step to .github/workflows/agents-auto-pilot.yml that executes pip install -r tools/requirements-llm.txt, Remove all unpinned pip install langchain* commands from .github/workflows/agents-auto-pilot.yml.
• Update .github/workflows/reusable-agents-verifier.yml so that in evaluate and compare modes it installs LLM dependencies via pip install -r .workflows-lib/tools/requirements-llm.txt (and remove/avoid any unpinned pip install langchain* commands in those paths). (This task combines adding install steps for two different modes and removing unpinned commands, making it difficult to track progress granularly.). Suggested split: Add a step to .github/workflows/reusable-agents-verifier.yml evaluate mode that executes pip install -r .workflows-lib/tools/requirements-llm.txt, Add a step to .github/workflows/reusable-agents-verifier.yml compare mode that executes pip install -r .workflows-lib/tools/requirements-llm.txt, Remove all unpinned pip install langchain* commands from evaluate and compare modes in .github/workflows/reusable-agents-verifier.yml.
• Enable enforcement tests by updating tests/workflows/test_workflow_llm_installs.py to not skip under normal CI; assert (1) agents-auto-pilot contains the pinned install step and does not contain floating pip install langchain*, (2) reusable-agents-verifier contains the pinned install step in evaluate/compare modes, and (3) both workflows include the expected actions/cache@v4 configuration with keys based on Python version + requirements hash. (This task bundles enabling tests with adding multiple different assertions, making it too broad to complete atomically.). Suggested split: Remove the unconditional skip logic from tests/workflows/test_workflow_llm_installs.py so tests run in normal CI, Add assertion to verify agents-auto-pilot workflow contains the pinned install step and lacks unpinned langchain commands, Add assertions to verify reusable-agents-verifier workflow contains pinned install steps in both evaluate and compare modes, Add assertions to verify both workflows include actions/cache@v4 with correct cache keys containing Python version and requirements hash.

Blocked tasks

• Update .github/workflows/agents-auto-pilot.yml to install LLM dependencies using the pinned requirements file (Agent cannot modify files in .github/workflows/ directory (protected) | Mark this task as requiring human intervention or create a pull request template that a human must apply)
• Update .github/workflows/reusable-agents-verifier.yml so that in evaluate and compare modes it installs LLM dependencies (Agent cannot modify files in .github/workflows/ directory (protected) | Mark this task as requiring human intervention or create a pull request template that a human must apply)
• Add an actions/cache@v4 pip cache step to both workflows (Agent cannot modify files in .github/workflows/ directory (protected) | Provide the exact YAML snippets in a separate file that a human can copy into the workflow files)

Objective acceptance criteria

• None

Missing sections

• None

Formatting issues

• The 'Critical Rules' section at the end appears to be meta-instructions for issue creation rather than content for the agent, and should be removed from the issue body
• The AGENT_LIMITATIONS section is meta-content that should not be in the issue body itself

Notes

• This issue is well-structured with comprehensive acceptance criteria and implementation notes. However, there is a fundamental blocker: the agent cannot modify .github/workflows/*.yml files due to protection rules. The issue should either (1) be marked as requiring human intervention for workflow changes, or (2) restructure tasks so the agent creates the necessary YAML snippets in a separate location for human review and application. The test-related tasks are agent-compatible. The acceptance criteria are objective and testable. Consider removing the meta-instructions (Critical Rules, AGENT_LIMITATIONS) that appear to have been accidentally included in the issue body.

[agents-workflows-bot]

🤖 Auto-pilot step $((STEP_COUNT + 1)): Starting issue formatting...

Running formatter inline (not via label trigger).

[stranske-keepalive]

🤖 Auto-pilot step $((STEP_COUNT + 1)): Applying optimization suggestions...

Running apply inline (not via label trigger).

[stranske-keepalive]

🤖 Auto-pilot step 1: Issue prepared! Assigning to agent...

Adding agent:codex label. The capability check will run automatically.

⏳ Agent will create a PR shortly.

[stranske-automation-bot]

Codex belt dispatcher queued this issue and created branch codex/issue-1453.

[agents-workflows-bot]

🤖 Auto-pilot step 1: Branch ready, waiting for commits

Branch codex/issue-1453 exists but has no commits yet.
✅ Dispatched the Codex belt worker to generate changes.

📊 Stall count: 1/5 (pauses after 5)

⏳ Auto-pilot will check again on next trigger.