Skip to content

fix: reclaim root-owned sandbox firewall dirs to prevent EACCES on reused runners - #44276

Merged
pelikhan merged 13 commits into
mainfrom
copilot/fix-sandbox-cleanup-ownership
Jul 9, 2026
Merged

fix: reclaim root-owned sandbox firewall dirs to prevent EACCES on reused runners#44276
pelikhan merged 13 commits into
mainfrom
copilot/fix-sandbox-cleanup-ownership

Conversation

CopilotAI commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

On reused runners, a prior AWF run leaves /tmp/gh-aw/sandbox/firewall owned by root (Squid/Docker containers run as root). The existing post-run chmod -R a+rX adds read/execute only — uid=1001 still can't delete those dirs, so the next run's writeConfigs fails with EACCES.

Changes

Post-run ownership reclaim (pkg/workflow/engine_firewall_support.go)

Added sudo chown -R "$(id -u):$(id -g)" before the existing chmod in the generated "Print firewall logs" step. This transfers ownership back to the runner user after every run, so the next run's rm -rf /tmp/gh-aw never needs sudo at all — breaking the root cause.

# Best-effort: reclaim ownership and fix permissions (AWF cleanup may not have run).# Chown transfers ownership back to the runner user so the next run can rm -rf /tmp/gh-aw# without requiring sudo, preventing EACCES failures on reused runners.
sudo -n chown -R "$(id -u):$(id -g)" /tmp/gh-aw/sandbox/firewall 2>/dev/null ||true
sudo -n chmod -R a+rX /tmp/gh-aw/sandbox/firewall 2>/dev/null || chmod -R a+rX ...

Pre-flight cleanup + pre-creation (actions/setup/sh/create_gh_aw_tmp_dir.sh)

Defense-in-depth for runs where the post-run chown didn't execute (killed/aborted run):

  • If /tmp/gh-aw/sandbox/firewall is not writable, sudo -n rm -rf reclaims it before AWF starts
  • Pre-creates firewall/logs and firewall/audit as uid=1001 so AWF never needs to create them

Recompile + test updates

  • All 258 workflow lock files regenerated with the new chown step
  • WASM golden files updated
  • Fixed a pre-existing stale test assertion in compiled_lock_files_test.go: smoke-workflow-call.md no longer declares aw_context as a workflow_call input, but the lock file and test had not been updated to reflect this

Generated by 👨‍🍳 PR Sous Chef · 9.91 AIC · ⌖ 10.6 AIC · ⊞ 7.1K ·
Comment /souschef to run again


Generated by 👨‍🍳 PR Sous Chef · 6.93 AIC · ⌖ 5.06 AIC · ⊞ 7.1K ·
Comment /souschef to run again


Generated by 👨‍🍳 PR Sous Chef · 6.55 AIC · ⌖ 5.29 AIC · ⊞ 7.1K ·
Comment /souschef to run again

CopilotAIand others added 2 commits July 8, 2026 11:44
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
- Add `sudo chown` in the post-run "Print firewall logs" step so AWF
leaves the sandbox firewall dir owned by the runner user (uid=1001).
This ensures the next run's `rm -rf /tmp/gh-aw` succeeds without
needing sudo at all, breaking the EACCES cycle on reused runners.
- Add pre-flight cleanup + pre-creation of `/tmp/gh-aw/sandbox/firewall`
dirs in `create_gh_aw_tmp_dir.sh` as defense-in-depth: if the post-run
chown didn't execute (e.g. previous run was killed), sudo rm reclaims
the stale root-owned directory before AWF starts; then mkdir creates the
firewall dirs as the runner user so AWF never needs to create them.
- Update tests and WASM golden files for new chown step output.
- Fix stale test assertion for smoke-call-workflow aw_context forwarding
(smoke-workflow-call.md no longer declares aw_context as a workflow_call
input; recompile corrected the lock file; test updated to match).
- Recompile all 258 workflow lock files to pick up the chown step.
Closes#44242
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
CopilotAI changed the title [WIP] Fix stale-sandbox cleanup for Smoke CIfix: reclaim root-owned sandbox firewall dirs to prevent EACCES on reused runnersJul 8, 2026
CopilotAI requested a review from pelikhanJuly 8, 2026 12:07
@pelikhanpelikhan added the awf label Jul 8, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 PR Triage

FieldValue
Categorybug
Risk🟡 medium
Score45 / 100
Recommended action🕐 defer

Score breakdown: Impact 22 + Urgency 12 + Quality 11

Assessment: Fixes EACCES on reused runners caused by root-owned sandbox firewall dirs. Valid fix via sudo chown -R in post-run step. Draft — infra-touching, lower urgency than active bugs.

Next step: Author to undraft; defer until higher-priority bugs (#44254, #44282, #44283) are cleared.

Generated by 🔧 PR Triage Agent · 64.6 AIC · ⌖ 20.4 AIC · ⊞ 5.4K ·

@pelikhan
pelikhan marked this pull request as ready for review July 8, 2026 12:59
CopilotAI review requested due to automatic review settings July 8, 2026 12:59

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a runner-reuse failure mode where prior AWF runs leave /tmp/gh-aw/sandbox/firewall root-owned, causing EACCES on subsequent runs when AWF tries to write configs. The changes add both post-run “ownership reclaim” and pre-flight cleanup/pre-creation to ensure the runner user can delete and recreate the firewall sandbox directory reliably.

Changes:

  • Add chown (best-effort) before chmod in the generated firewall log parsing step so reused runners don’t hit EACCES on the next run.
  • Add targeted pre-flight cleanup for a stale, non-writable /tmp/gh-aw/sandbox/firewall plus pre-create firewall/{logs,audit} as the runner user before AWF starts.
  • Regenerate compiled workflow lock files and update WASM golden + lockfile-related tests.
Show a summary per file
FileDescription
pkg/workflow/engine_firewall_support.goGenerate firewall-log step with best-effort chown+chmod to avoid next-run cleanup failures on reused runners.
actions/setup/sh/create_gh_aw_tmp_dir.shPre-flight cleanup of stale root-owned firewall dir and pre-create firewall subdirs before AWF runs.
pkg/workflow/engine_firewall_support_test.goUpdate tests to assert the new chown behavior is present in generated steps.
pkg/workflow/compiled_lock_files_test.goUpdate expectations for forwarded workflow_call inputs in the compiled lock file.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/with-imports.goldenGolden update reflecting added chown in generated firewall-log step.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/smoke-copilot.goldenGolden update reflecting added chown in generated firewall-log step.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/playwright-cli-mode.goldenGolden update reflecting added chown in generated firewall-log step.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/basic-copilot.goldenGolden update reflecting added chown in generated firewall-log step.
pkg/workflow/testdata/TestWasmGolden_AllEngines/pi.goldenGolden update reflecting added chown in generated firewall-log step.
pkg/workflow/testdata/TestWasmGolden_AllEngines/gemini.goldenGolden update reflecting added chown in generated firewall-log step.
pkg/workflow/testdata/TestWasmGolden_AllEngines/copilot.goldenGolden update reflecting added chown in generated firewall-log step.
pkg/workflow/testdata/TestWasmGolden_AllEngines/codex.goldenGolden update reflecting added chown in generated firewall-log step.
pkg/workflow/testdata/TestWasmGolden_AllEngines/claude.goldenGolden update reflecting added chown in generated firewall-log step.
.github/workflows/*.lock.ymlRegenerated workflow lock files to include the updated firewall-log step (and other regenerated output).

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 271/271 changed files
  • Comments generated: 10
  • Review effort level: Low

Comment on lines 1 to +3
#!/usr/bin/env bash
set +o histexpand

Comment on lines 146 to 150
@@ -148,15 +148,25 @@ func generateFirewallLogParsingStep(workflowName string, workflowData *WorkflowD
// AWF's own post-run cleanup (fixArtifactPermissionsForRootless) normally handles
// this, but if AWF is killed by timeout or OOM, the cleanup never runs.
// Use a non-sudo chmod as a best-effort fallback for artifact upload accessibility.
t.Error("Expected firewall log parsing step to contain best-effort chmod for artifact upload even in network-isolation mode")
}

// Should contain best-effort non-sudo chown to reclaim ownership for next run
Comment on lines 437 to 450
// the caller job forwards both from the safe_outputs payload.
assert.Contains(t, lockContent, "payload: ${{ needs.safe_outputs.outputs.call_workflow_payload }}", "call-workflow job should forward payload from safe_outputs")
assert.Contains(t, lockContent, "${{ fromJSON(needs.safe_outputs.outputs.call_workflow_payload)['task-description'] }}", "call-workflow job should forward task-description from the handler payload")
})
Comment on lines 542 to 548
- name: Install GitHub Copilot CLI
run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" 1.0.68
env:
GH_HOST: github.com
- name: Install AWF binary
run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.27 --rootless
- name: Determine automatic lockdown mode for GitHub MCP Server
id: determine-automatic-lockdown
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 (source v9)
env:
GH_AW_GITHUB_TOKEN: ${{ secrets.GH_AW_GITHUB_TOKEN }}
GH_AW_GITHUB_MCP_SERVER_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN }}
GH_AW_GITHUB_MIN_INTEGRITY: 'approved'
with:
script: |
const determineAutomaticLockdown = require('${{ runner.temp }}/gh-aw/actions/determine_automatic_lockdown.cjs');
await determineAutomaticLockdown(github, context, core);
- name: Parse integrity filter lists
Comment on lines 588 to 592
- name: Install AWF binary
run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.27 --rootless
- name: Determine automatic lockdown mode for GitHub MCP Server
id: determine-automatic-lockdown
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 (source v9)
env:
GH_AW_GITHUB_TOKEN: ${{ secrets.GH_AW_GITHUB_TOKEN }}
GH_AW_GITHUB_MCP_SERVER_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN }}
GH_AW_GITHUB_MIN_INTEGRITY: 'approved'
with:
script: |
const determineAutomaticLockdown = require('${{ runner.temp }}/gh-aw/actions/determine_automatic_lockdown.cjs');
await determineAutomaticLockdown(github, context, core);
- name: Parse integrity filter lists
id: parse-guard-vars
env:
Comment on lines 553 to 557
- name: Install AWF binary
run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.27 --rootless
- name: Determine automatic lockdown mode for GitHub MCP Server
id: determine-automatic-lockdown
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 (source v9)
env:
GH_AW_GITHUB_TOKEN: ${{ secrets.GH_AW_GITHUB_TOKEN }}
GH_AW_GITHUB_MCP_SERVER_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN }}
GH_AW_GITHUB_MIN_INTEGRITY: 'approved'
with:
script: |
const determineAutomaticLockdown = require('${{ runner.temp }}/gh-aw/actions/determine_automatic_lockdown.cjs');
await determineAutomaticLockdown(github, context, core);
- name: Parse integrity filter lists
id: parse-guard-vars
env:
Comment on lines 492 to 496
env:
GH_HOST: github.com
- name: Install AWF binary
run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.27 --rootless
- name: Determine automatic lockdown mode for GitHub MCP Server
id: determine-automatic-lockdown
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 (source v9)
env:
GH_AW_GITHUB_TOKEN: ${{ secrets.GH_AW_GITHUB_TOKEN }}
GH_AW_GITHUB_MCP_SERVER_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN }}
GH_AW_GITHUB_MIN_INTEGRITY: 'approved'
with:
script: |
const determineAutomaticLockdown = require('${{ runner.temp }}/gh-aw/actions/determine_automatic_lockdown.cjs');
await determineAutomaticLockdown(github, context, core);
- name: Parse integrity filter lists
Comment on lines 492 to 496
env:
GH_HOST: github.com
- name: Install AWF binary
run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.27 --rootless
- name: Determine automatic lockdown mode for GitHub MCP Server
id: determine-automatic-lockdown
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 (source v9)
env:
GH_AW_GITHUB_TOKEN: ${{ secrets.GH_AW_GITHUB_TOKEN }}
GH_AW_GITHUB_MCP_SERVER_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN }}
GH_AW_GITHUB_MIN_INTEGRITY: 'none'
with:
script: |
const determineAutomaticLockdown = require('${{ runner.temp }}/gh-aw/actions/determine_automatic_lockdown.cjs');
await determineAutomaticLockdown(github, context, core);
- name: Parse integrity filter lists
Comment on lines 602 to 606
- name: Install AWF binary
run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.27.27 --rootless
- name: Determine automatic lockdown mode for GitHub MCP Server
id: determine-automatic-lockdown
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 (source v9)
env:
GH_AW_GITHUB_TOKEN: ${{ secrets.GH_AW_GITHUB_TOKEN }}
GH_AW_GITHUB_MCP_SERVER_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN }}
GH_AW_GITHUB_MIN_INTEGRITY: 'none'
GH_AW_GITHUB_REPOS: 'all'
with:
script: |
const determineAutomaticLockdown = require('${{ runner.temp }}/gh-aw/actions/determine_automatic_lockdown.cjs');
await determineAutomaticLockdown(github, context, core);
- name: Parse integrity filter lists
id: parse-guard-vars
env:
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot run pr-finisher skill

CopilotAIand others added 2 commits July 8, 2026 13:22
…pt hardening
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

CopilotAI commented Jul 8, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot run pr-finisher skill

Addressed all actionable review feedback in commit fix: address review feedback - shell hardening, comments, test rename:

  • create_gh_aw_tmp_dir.sh: Added set -euo pipefail so mkdir failures are surfaced as step failures rather than silently continuing with exit 0.
  • engine_firewall_support.go (lines 146-150): Rewrote the comment to accurately describe both branches — non-network-isolation mode uses plain sudo chown/chmod; network-isolation mode uses sudo -n with a non-sudo chmod fallback.
  • engine_firewall_support_test.go (line 316): Changed "non-sudo chown" → "sudo -n chown (non-interactive sudo)" to avoid confusion with the actual generated command.
  • compiled_lock_files_test.go (lines 437–450): Renamed TestCompiledLockFiles_SmokeCallWorkflowForwardsAwContext...ForwardsPayload and subtest CallWorkflowJobForwardsGeneratedAwContextCallWorkflowJobForwardsPayloadAndTaskDescription.

The determine-automatic-lockdown concerns in the lock file reviews were false positives — the step is still present and referenced correctly in all flagged files.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 PR Triage (Updated)

FieldValue
Categorybug (infra/firewall)
Risk🟡 Medium
Priority Score58 / 100
Action🚀 fast_track(upgraded from defer)

Score breakdown: Impact 30 + Urgency 20 + Quality 8

Rationale: DRAFT. Root-owned sandbox firewall EACCES fix on reused runners. Upgraded from defer as similar infra-fix PRs in this batch are being fast-tracked. Needs undraft + CI before merge.

Run §28986426320

Generated by 🔧 PR Triage Agent · 74.6 AIC · ⌖ 8.8 AIC · ⊞ 5.4K ·

@pelikhan

Copy link
Copy Markdown
Collaborator

/matt

@github-actions

github-actionsBot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

github-actions[bot]
github-actionsBot previously requested changes Jul 9, 2026

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs, /tdd, and /codebase-design — requesting changes on test coverage gaps and a correctness risk in the pre-flight guard.

📋 Key Themes & Highlights

Key Themes

  • Missing test filecreate_gh_aw_tmp_dir.sh has the most impactful new logic in this PR (defense-in-depth pre-flight cleanup) but is the only modified script in this directory without a companion _test.sh. The three branches (writable, non-writable+sudo ok, non-writable+sudo fail) need coverage.
  • Test coverage gaps in print_firewall_logs_test.sh — the chown/chmod ownership-reclaim branches (the actual bug fix) have no test coverage. The existing tests only exercise argument-parsing and the awf not installed path.
  • Correctness risk in writability guard[ ! -w /tmp/gh-aw/sandbox/firewall ] won't fire if the directory is writable but subdirectories (e.g. logs/, audit/) are root-owned. This is precisely the scenario the PR intends to handle.
  • GITHUB_STEP_SUMMARY undocumented and unguarded — the variable drives the critical tee output in the extracted script but is absent from the # Environment: header; if unset, tee -a "" silently drops step summary output.
  • Implicit path contractprint_firewall_logs.sh derives FIREWALL_DIR as dirname(AWF_LOGS_DIR); the Go side controls what that env var is set to. A comment near the constant would make this coupling explicit for future maintainers.

Positive Highlights

  • ✅ Excellent extraction of 258 inline bash blocks into a single reusable print_firewall_logs.sh — major maintainability win
  • ✅ The --rootless flag split is clean and the Go-side logic is well-tested
  • ✅ Defense-in-depth layering (post-run chown + pre-run cleanup) is the right architectural approach
  • ✅ Good test names and structure in both the Go and bash test files

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · 140.2 AIC · ⌖ 6.48 AIC · ⊞ 6.6K
Comment /matt to run again

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot run pr-finisher skill

…leanup-ownership
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

CopilotAI commented Jul 9, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot run pr-finisher skill

Branch merged with main (502a121, 10 new commits). Local validation clean:

  • ✅ Reviews — All threads addressed in prior commits.
  • ✅ Checks (local) — make fmt, make test-unit, make recompile all pass (259/259 lock files, no drift).
  • ❓ Checks (CI) — Stale after agent push; needs human re-trigger (approval gate for bot-authored commits).
  • ✅ Mergeable — Branch up to date with main, no conflicts.

Hand-off: A maintainer needs to approve the pending CI runs before merge.

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot address #44276 (review)

…MMARY guard, path comment, and tests
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

CopilotAI commented Jul 9, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot address #44276 (review)

Addressed in commit fix: address Matt Pocock Skills review:

  1. create_gh_aw_tmp_dir_test.sh — new companion test file with 5 cases: syntax, clean-start directory creation, no-op when writable, guard fires when parent is non-writable, guard fires when subdirectory is non-writable.

  2. print_firewall_logs_test.sh — added 3 tests covering chown/chmod branches: rootless non-sudo chmod fallback (test 6), default mode || true catch (test 7), and GITHUB_STEP_SUMMARY unset guard (test 8, using a mock awf). 14 tests total, all passing.

  3. Correctness fix (create_gh_aw_tmp_dir.sh) — expanded the writability guard to also check logs/ and audit/ subdirectories:

    if [ -d /tmp/gh-aw/sandbox/firewall ] && \
    { [ !-w /tmp/gh-aw/sandbox/firewall ] || \
    { [ -d /tmp/gh-aw/sandbox/firewall/logs ] && [ !-w /tmp/gh-aw/sandbox/firewall/logs ]; } || \
    { [ -d /tmp/gh-aw/sandbox/firewall/audit ] && [ !-w /tmp/gh-aw/sandbox/firewall/audit ]; }; };then
  4. GITHUB_STEP_SUMMARY guard (print_firewall_logs.sh) — added to the # Environment: header; tee -a "${GITHUB_STEP_SUMMARY}"tee -a "${GITHUB_STEP_SUMMARY:-/dev/null}".

  5. Path contract comment (print_firewall_logs.sh) — added above FIREWALL_DIR="$(dirname "${AWF_LOGS_DIR}")" explaining that the Go compiler sets AWF_LOGS_DIR to ${RUNNER_TEMP}/gh-aw/sandbox/firewall/logs.

@pelikhan
pelikhan merged commit 00d2d53 into mainJul 9, 2026
29 checks passed
@pelikhan
pelikhan deleted the copilot/fix-sandbox-cleanup-ownership branch July 9, 2026 02:39
@lpcox

lpcox commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

@copilot using sudo will break some arc-dind deployments. make sure that sudo is not used when arc-dind is enabled.

@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.82.7

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aw-failures] [aw-fix] Smoke CI sandbox setup hard-fails — stale root-owned /tmp/gh-aw/sandbox/firewall blocks AWF (uid=1001)

5 participants

@pelikhan@gh-aw-bot@lpcox