fix: apply pinned LLM installs, pip cache, and enforce tests (#1453)

[stranske]

Source: Issue #1453

Automated Status Summary

Scope

PR #1447 addressed issue #1437, but verification failed because the actual workflow files were not updated to use pinned LLM dependency installs and proper pip caching, and the enforcement tests were effectively skipped in normal CI. This follow-up closes the remaining gaps by updating the live GitHub Actions workflows, enabling non-skipped enforcement tests, and hardening check_prompt_injection() to avoid runtime errors on malformed detector outputs.

Context for Agent

Related Issues/PRs

• #1447
• #1437

Tasks

Test Updates (Agent-Executable)

• Remove the unconditional skip logic from tests/workflows/test_workflow_llm_installs.py so tests run in normal CI
• Add assertion to tests/workflows/test_workflow_llm_installs.py to verify agents-auto-pilot workflow contains the pinned install step and lacks unpinned langchain commands
• Add assertions to tests/workflows/test_workflow_llm_installs.py to verify reusable-agents-verifier workflow contains pinned install steps in both evaluate and compare modes
• Add assertions to tests/workflows/test_workflow_llm_installs.py to verify both workflows include actions/cache@v4 with correct cache keys containing Python version and requirements hash
• Add unit tests to validate check_prompt_injection() handles malformed detector outputs without raising exceptions
• Add unit tests to validate check_prompt_injection() correctly handles known-good reason code formats

Code Hardening (Agent-Executable)

• Update check_prompt_injection() to validate detector output format before parsing (check prefix/delimiters and guard integer conversion)
• Implement safe fallback path in check_prompt_injection() for unexpected formats (no exception, defined behavior)

Workflow Update Snippets (Agent-Executable)

• Create docs/workflow-updates/agents-auto-pilot-changes.yml with complete YAML snippet for pip install step and cache configuration
• Create docs/workflow-updates/reusable-agents-verifier-changes.yml with complete YAML snippets for evaluate/compare mode pip install steps and cache configuration
• Add docs/workflow-updates/README.md with instructions for applying the YAML snippets to protected workflow files

Acceptance criteria

Workflow Files (Human-Applied)

• .github/workflows/agents-auto-pilot.yml contains a step that executes exactly pip install -r tools/requirements-llm.txt (as a run command in the workflow)
• .github/workflows/agents-auto-pilot.yml does NOT contain any pip install commands that install langchain via an unpinned specifier, including (but not limited to) pip install langchain, pip install langchain*, pip install langchain== (missing version), or pip install git+...langchain...
• .github/workflows/reusable-agents-verifier.yml includes a step that executes exactly pip install -r .workflows-lib/tools/requirements-llm.txt in the evaluate mode execution path
• .github/workflows/reusable-agents-verifier.yml includes a step that executes exactly pip install -r .workflows-lib/tools/requirements-llm.txt in the compare mode execution path
• .github/workflows/reusable-agents-verifier.yml does NOT contain any pip install commands that install langchain via an unpinned specifier in the evaluate or compare execution paths (e.g., pip install langchain*, pip install langchain, pip install langchain-community without a version pin)
• .github/workflows/agents-auto-pilot.yml contains an actions/cache@v4 step that caches pip (cache path includes pip cache directory) and uses a cache key that includes BOTH the Python version and ${{ hashFiles('tools/requirements-llm.txt') }}
• .github/workflows/reusable-agents-verifier.yml contains an actions/cache@v4 step that caches pip (cache path includes pip cache directory) and uses a cache key that includes BOTH the Python version and ${{ hashFiles('.workflows-lib/tools/requirements-llm.txt') }}

Test Enforcement

• tests/workflows/test_workflow_llm_installs.py is executed under normal CI (i.e., it is not unconditionally skipped) and will fail if the agents-auto-pilot workflow lacks the pinned requirements install step
• tests/workflows/test_workflow_llm_installs.py asserts that .github/workflows/agents-auto-pilot.yml contains pip install -r tools/requirements-llm.txt AND asserts it does not contain any floating pip install langchain*/unversioned langchain install commands
• tests/workflows/test_workflow_llm_installs.py asserts that .github/workflows/reusable-agents-verifier.yml contains pip install -r .workflows-lib/tools/requirements-llm.txt for BOTH evaluate and compare modes (separate assertions per mode/path, not a single generic check)
• tests/workflows/test_workflow_llm_installs.py asserts that BOTH workflows include an actions/cache@v4 step and that each cache key includes Python version + the correct hashFiles(...) call for the correct requirements path

Code Hardening

• check_prompt_injection() does not raise an exception when detect_prompt_injection returns an unexpected or malformed reason code (e.g., empty string, missing delimiter/prefix, non-integer suffix); instead it follows a defined fallback path (e.g., returns a safe default or logs and continues)
• check_prompt_injection() continues to correctly handle valid reason code formats by producing the same outcome as before for at least one known-good reason code value
• Unit tests pass that verify check_prompt_injection() behavior with both malformed and known-good detector outputs

Documentation Deliverables

• docs/workflow-updates/agents-auto-pilot-changes.yml exists and contains valid YAML snippet for pip install and cache steps
• docs/workflow-updates/reusable-agents-verifier-changes.yml exists and contains valid YAML snippets for evaluate/compare mode pip install and cache steps
• docs/workflow-updates/README.md exists and provides clear instructions for applying the YAML snippets to the protected workflow files

Head SHA: 538b8b6
Latest Runs: ✅ success — Gate
Required: gate: ✅ success

Workflow / Job	Result	Logs
Agents PR meta manager	❔ in progress	View run
CI Autofix Loop	✅ success	View run
Gate	✅ success	View run
Health 40 Sweep	✅ success	View run
Health 44 Gate Branch Protection	✅ success	View run
Health 45 Agents Guard	✅ success	View run
Health 50 Security Scan	✅ success	View run
Health 73 Template Completeness	✅ success	View run
Health 74 Template Drift	✅ success	View run
Maint 52 Validate Workflows	✅ success	View run
PR 11 - Minimal invariant CI	✅ success	View run
Selftest CI	✅ success	View run
Validate Sync Manifest	✅ success	View run

[stranske-keepalive]

Automated Status Summary

Head SHA: 15e3efb
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / guard
Required: core tests (3.11): ⏳ pending, core tests (3.12): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job	Result	Logs
(no jobs reported)	⏳ pending	—

Coverage Overview

• Coverage history entries: 1

Coverage Trend

Metric	Value
Current	93.12%
Baseline	85.00%
Delta	+8.12%
Minimum	70.00%
Status	✅ Pass

Top Coverage Hotspots (lowest coverage)

File	Coverage	Missing
src/cli_parser.py	81.8%	4
src/percentile_calculator.py	95.0%	1
src/aggregator.py	95.0%	2
src/__init__.py	100.0%	0
src/ndjson_parser.py	100.0%	0

Updated automatically; will refresh on subsequent CI/Docker completions.

---

Keepalive checklist

Scope

PR #1447 addressed issue #1437, but verification failed because the actual workflow files were not updated to use pinned LLM dependency installs and proper pip caching, and the enforcement tests were effectively skipped in normal CI. This follow-up closes the remaining gaps by updating the live GitHub Actions workflows, enabling non-skipped enforcement tests, and hardening check_prompt_injection() to avoid runtime errors on malformed detector outputs.

Context for Agent

Related Issues/PRs

• #1447
• #1437

Tasks

Test Updates (Agent-Executable)

• Remove the unconditional skip logic from tests/workflows/test_workflow_llm_installs.py so tests run in normal CI
• Add assertion to tests/workflows/test_workflow_llm_installs.py to verify agents-auto-pilot workflow contains the pinned install step and lacks unpinned langchain commands
• Add assertions to tests/workflows/test_workflow_llm_installs.py to verify reusable-agents-verifier workflow contains pinned install steps in both evaluate and compare modes
• Add assertions to tests/workflows/test_workflow_llm_installs.py to verify both workflows include actions/cache@v4 with correct cache keys containing Python version and requirements hash
• Add unit tests to validate check_prompt_injection() handles malformed detector outputs without raising exceptions
• Add unit tests to validate check_prompt_injection() correctly handles known-good reason code formats

Code Hardening (Agent-Executable)

• Update check_prompt_injection() to validate detector output format before parsing (check prefix/delimiters and guard integer conversion)
• Implement safe fallback path in check_prompt_injection() for unexpected formats (no exception, defined behavior)

Workflow Update Snippets (Agent-Executable)

• Create docs/workflow-updates/agents-auto-pilot-changes.yml with complete YAML snippet for pip install step and cache configuration
• Create docs/workflow-updates/reusable-agents-verifier-changes.yml with complete YAML snippets for evaluate/compare mode pip install steps and cache configuration
• Add docs/workflow-updates/README.md with instructions for applying the YAML snippets to protected workflow files

Acceptance criteria

Workflow Files (Human-Applied)

• .github/workflows/agents-auto-pilot.yml contains a step that executes exactly pip install -r tools/requirements-llm.txt (as a run command in the workflow)
• .github/workflows/agents-auto-pilot.yml does NOT contain any pip install commands that install langchain via an unpinned specifier, including (but not limited to) pip install langchain, pip install langchain*, pip install langchain== (missing version), or pip install git+...langchain...
• .github/workflows/reusable-agents-verifier.yml includes a step that executes exactly pip install -r .workflows-lib/tools/requirements-llm.txt in the evaluate mode execution path
• .github/workflows/reusable-agents-verifier.yml includes a step that executes exactly pip install -r .workflows-lib/tools/requirements-llm.txt in the compare mode execution path
• .github/workflows/reusable-agents-verifier.yml does NOT contain any pip install commands that install langchain via an unpinned specifier in the evaluate or compare execution paths (e.g., pip install langchain*, pip install langchain, pip install langchain-community without a version pin)
• .github/workflows/agents-auto-pilot.yml contains an actions/cache@v4 step that caches pip (cache path includes pip cache directory) and uses a cache key that includes BOTH the Python version and ${{ hashFiles('tools/requirements-llm.txt') }}
• .github/workflows/reusable-agents-verifier.yml contains an actions/cache@v4 step that caches pip (cache path includes pip cache directory) and uses a cache key that includes BOTH the Python version and ${{ hashFiles('.workflows-lib/tools/requirements-llm.txt') }}

Test Enforcement

• tests/workflows/test_workflow_llm_installs.py is executed under normal CI (i.e., it is not unconditionally skipped) and will fail if the agents-auto-pilot workflow lacks the pinned requirements install step
• tests/workflows/test_workflow_llm_installs.py asserts that .github/workflows/agents-auto-pilot.yml contains pip install -r tools/requirements-llm.txt AND asserts it does not contain any floating pip install langchain*/unversioned langchain install commands
• tests/workflows/test_workflow_llm_installs.py asserts that .github/workflows/reusable-agents-verifier.yml contains pip install -r .workflows-lib/tools/requirements-llm.txt for BOTH evaluate and compare modes (separate assertions per mode/path, not a single generic check)
• tests/workflows/test_workflow_llm_installs.py asserts that BOTH workflows include an actions/cache@v4 step and that each cache key includes Python version + the correct hashFiles(...) call for the correct requirements path

Code Hardening

• check_prompt_injection() does not raise an exception when detect_prompt_injection returns an unexpected or malformed reason code (e.g., empty string, missing delimiter/prefix, non-integer suffix); instead it follows a defined fallback path (e.g., returns a safe default or logs and continues)
• check_prompt_injection() continues to correctly handle valid reason code formats by producing the same outcome as before for at least one known-good reason code value
• Unit tests pass that verify check_prompt_injection() behavior with both malformed and known-good detector outputs

Documentation Deliverables

• docs/workflow-updates/agents-auto-pilot-changes.yml exists and contains valid YAML snippet for pip install and cache steps
• docs/workflow-updates/reusable-agents-verifier-changes.yml exists and contains valid YAML snippets for evaluate/compare mode pip install and cache steps
• docs/workflow-updates/README.md exists and provides clear instructions for applying the YAML snippets to the protected workflow files

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bbbaa9410d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[agents-workflows-bot]

🤖 Keepalive Loop Status

PR #1456 | Agent: Codex | Iteration 0/5

Current State

Metric	Value
Iteration progress	[----------] 0/5
Action	wait (missing-agent-label)
Disposition	skipped (transient)
Gate	success
Tasks	0/28 complete
Timeout	45 min (default)
Timeout usage	3m elapsed (7%, 42m remaining)
Keepalive	❌ disabled
Autofix	❌ disabled

🔍 Failure Classification

| Error type | infrastructure |
| Error category | resource |
| Suggested recovery | Confirm the referenced resource exists (repo, PR, branch, workflow, or file). |

[Copilot]

Pull request overview

Updates the live GitHub Actions workflows and associated tests to enforce pinned LLM dependency installs (via requirements files) and add pip caching, while strengthening prompt-injection guard test coverage.

Changes:

• Switch LLM dependency installs in workflows from floating pip install langchain*... to pip install -r .../requirements-llm.txt, and add actions/cache@v4 pip caching keyed by Python version + requirements hash.
• Remove the conditional CI skip so workflow-enforcement tests run in normal CI, and fix the langchain floating-install detection regex.
• Add additional unit tests ensuring check_prompt_injection() handles non-string/malformed reason values and recognizes all known reason codes.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
tests/workflows/test_workflow_llm_installs.py	Removes high-privilege skip and fixes regex to properly detect floating langchain installs; continues enforcing pinned install + cache presence.
tests/scripts/test_injection_guard.py	Adds parametrized tests for malformed/non-string detector reasons and for all known reason codes.
.github/workflows/reusable-agents-verifier.yml	Adds setup-python step IDs, pip cache steps, and switches evaluate/compare installs to requirements files.
.github/workflows/agents-auto-pilot.yml	Adds setup-python step ID, pip cache step, and replaces floating langchain installs with requirements-based install.