Improve Copilot 403 auth guidance for copilot-requests mode by Copilot · Pull Request #40052 · github/gh-aw

Copilot · 2026-06-18T12:44:54Z

Copilot CLI failures in copilot-requests: write mode were surfacing BYOK provider-token guidance even when the real problem was org-billing authorization. This updates the 403 path to point users at Copilot requests permissions and centralized billing docs instead.

403 diagnostics for Copilot requests
- Detect gh-aw API proxy auth failures returning HTTP 403 during Copilot requests flow.
- Replace COPILOT_PROVIDER_API_KEY / COPILOT_PROVIDER_BEARER_TOKEN guidance with messaging tailored to:
  - permissions.copilot-requests: write
  - org-level Copilot billing enablement
  - the billing reference docs
Mode detection hardening
- Normalize S2STOKENS as a truthy env flag instead of requiring an exact "true" match.
- Keeps the diagnostic path stable across equivalent flag values.
Regression coverage
- Add focused harness coverage for:
  - proxy-backed 403 failures in Copilot requests mode
  - truthy S2STOKENS variants
  - preservation of existing 401 behavior

Example of the new guidance path:

Copilot requests authentication failed through the gh-aw API proxy (HTTP 403, model=claude-sonnet-4.5, stage=starting the Copilot CLI request). This workflow is using permissions.copilot-requests: write, so Copilot requests must be allowed through your organization's centralized Copilot billing configuration. Verify that copilot-requests: write is granted to the workflow or job and that Copilot org billing is enabled for your organization. See https://github.github.com/gh-aw/reference/billing/ for details.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-06-18T12:48:41Z

 const stage = detectCopilotAuthFailureStage(output);
+ if (authFailure.statusCode === "403" && envFlagEnabled(env.S2STOKENS)) {
+ return (
+ `Copilot requests authentication failed through the gh-aw API proxy (HTTP 403, model=${selectedModel}, stage=${stage}). ` +


@copilot refactor to template file in actions/setup/md use renderTemplate

Addressed in ef58ece by moving the 403 copilot-requests guidance into an actions/setup/md template and rendering it with renderTemplateFromFile.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

✅ Ready to approve

The changes are narrowly scoped to diagnostics, include targeted regression tests, and align with the stated 403/Copilot-requests requirements without introducing risky behavioral changes.

Note: this review does not count toward required approvals for merging.

Pull request overview

This PR updates gh-aw’s Copilot CLI harness diagnostics so HTTP 403 failures that occur specifically in permissions.copilot-requests: write (Copilot requests / centralized billing) mode no longer show BYOK provider-token guidance, and instead direct users to org billing enablement and the billing reference docs.

Changes:

Added a dedicated 403 diagnostic message for Copilot requests mode when the failure is coming from the local gh-aw API proxy.
Hardened Copilot requests mode detection by treating S2STOKENS as a truthy env flag (not only an exact "true" match).
Added regression tests covering proxy-backed 403 behavior in Copilot requests mode and S2STOKENS truthy variants, while preserving existing 401 behavior.

File summaries

File	Description
`actions/setup/md/copilot_requests_proxy_auth_403.md`	New templated diagnostic message for proxy-backed HTTP 403 in Copilot requests mode, linking to billing guidance.
`actions/setup/js/copilot_harness.cjs`	Adds truthy env-flag parsing and a 403-specific diagnostic path gated on `S2STOKENS` (Copilot requests mode).
`actions/setup/js/copilot_harness.test.cjs`	Adds focused unit coverage for the new 403 rewrite behavior and truthy `S2STOKENS` handling.
`.changeset/patch-copilot-403-org-billing-message.md`	Records the user-facing patch change in release notes.

Copilot's findings

Files reviewed: 4/4 changed files
Comments generated: 0

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-06-18T13:16:17Z

✅ Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR #40052 does not have the 'implementation' label and has 0 new lines of code in business logic directories (threshold 100).

github-actions · 2026-06-18T13:16:27Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

github-actions · 2026-06-18T13:16:28Z

✅ PR Code Quality Reviewer completed the code quality review.

github-actions · 2026-06-18T13:16:29Z

✅ Test Quality Sentinel completed test quality analysis.

github-actions · 2026-06-18T13:26:01Z

🧪 Test Quality Sentinel Report

✅ Test Quality Score: 100/100 — Excellent

Analyzed 2 test(s): 2 design, 0 implementation, 0 guideline violation(s).

📊 Metrics & Test Classification (2 tests analyzed)

Metric	Value
New/modified tests analyzed	2
✅ Design tests (behavioral contracts)	2 (100%)
⚠️ Implementation tests (low value)	0 (0%)
Tests with error/edge cases	2 (100%)
Duplicate test clusters	0
Test inflation detected	No (23 test lines / 25 prod lines = 0.92)
🚨 Coding-guideline violations	0

Test	File	Classification	Issues Detected
`rewrites local proxy 403 errors in copilot-requests mode to org-billing guidance`	`actions/setup/js/copilot_harness.test.cjs`	✅ Design	—
`treats truthy S2STOKENS values as copilot-requests mode for 403 guidance`	`actions/setup/js/copilot_harness.test.cjs`	✅ Design	Minimal (1 assertion) — see detail below

JavaScript: 2 (*.test.cjs). No Go or other language test files changed.

💡 Optional Improvements (non-blocking)

treats truthy S2STOKENS values as copilot-requests mode for 403 guidance — The single assertion (toContain("Copilot requests authentication failed")) confirms the correct branch is taken for S2STOKENS=" YES " but does not verify the complete output contract (billing URL, permissions guidance, suppression of COPILOT_PROVIDER_API_KEY). Consider extending this test with the same assertions as the first 403 test, or folding it into a table-driven pattern covering multiple truthy values ("true", "1", " YES ").

Verdict

✅ Check passed. 0% implementation tests (threshold: 30%). Both new tests verify observable user-facing output from buildCopilotProxyAuthFailureDiagnostic — the 403 copilot-requests diagnostic content and the truthy env-flag edge case. No mock libraries, no missing build tags, no test inflation.

🧪 Test quality analysis by Test Quality Sentinel · ◷

github-actions

✅ Test Quality Sentinel: 100/100. Test quality is excellent — 0% of new tests are implementation tests (threshold: 30%).

github-actions

Skills-Based Review 🧠

Applied /tdd and /diagnose — commenting with improvement suggestions, no blocking issues.

📋 Key Themes & Highlights

Key Themes

Test coverage gaps: The new branch discriminator (envFlagEnabled(S2STOKENS) guards the 403 path) has no negative-case test. BYOK 403s should return empty, but removing the guard wouldn't be caught by the current suite.
envFlagEnabled is untested at the unit level: Falsy variants ("false", "0", "no") are never exercised; only the truthy path is covered integration-style.
Second test underspecified: The truthy-normalization test checks a single property, while the first test checks five — worth aligning the coverage depth.

Positive Highlights

✅ Clean guard restructuring — splitting the compound statusCode !== "401" condition into per-branch returns is more readable and extensible.
✅ envFlagEnabled is a well-scoped pure function with a clear contract; the "yes" / "1" normalization pre-empts common CI flag variation.
✅ The first regression test is thorough: six expectations lock down the message content, billing URL, and absence of wrong guidance in one shot.
✅ Changeset entry accurately scopes the change as patch with a clear one-sentence description.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

github-actions

REQUEST_CHANGES — 3 test coverage gaps + 1 throw-safety issue

The diagnostic logic and envFlagEnabled implementation are correct. The issues below are about insufficient test coverage that would let regressions slip through and a silent contract change that turns a no-throw function into one that can crash the harness.

Blocking findings summary

#	File	Severity	Finding
1	`copilot_harness.test.cjs:987`	medium	No assertion verifies `{selected_model}`/`{stage}` placeholders are resolved — static-string checks pass even if substitution is broken
2	`copilot_harness.test.cjs:996`	medium	No test for falsy `S2STOKENS` returning `""` and no test for non-proxy 403 + `S2STOKENS=true` returning `""` — both gate conditions are unverified on their off paths
3	`copilot_harness.cjs:434`	medium	`buildCopilotProxyAuthFailureDiagnostic` gains a filesystem I/O call with no error handling; a missing template file now crashes the harness retry loop instead of returning a fallback string

🔎 Code quality review by PR Code Quality Reviewer

github-actions · 2026-06-18T13:28:07Z

+ });
+
+ expect(diagnostic).toContain("Copilot requests authentication failed");
+ });


Falsy S2STOKENS and non-proxy-URL paths are completely untested: the test suite covers only truthy values, leaving two correctness gaps that would hide regressions.
💡 Suggested additions
Gap 1 — falsy S2STOKENS returns empty string (S2STOKENS gate correctness):
it("returns empty string for proxy 403 when S2STOKENS is not set",()=>{constdiagnostic=buildCopilotProxyAuthFailureDiagnostic("Authentication failed with provider at (172.30.0.30/redacted) (HTTP 403).",{COPILOT_MODEL: "claude-sonnet-4.5"}// no S2STOKENS);expect(diagnostic).toBe("");});
Gap 2 — non-proxy URL with S2STOKENS=true returns empty string (proxy-URL gate correctness):
it("returns empty string for non-proxy 403 even when S2STOKENS is true",()=>{constdiagnostic=buildCopilotProxyAuthFailureDiagnostic("Authentication failed with provider at (api.anthropic.com/redacted) (HTTP 403).",{COPILOT_MODEL: "claude-sonnet-4.5",S2STOKENS: "true"});expect(diagnostic).toBe("");});
Without these, accidentally inverting the envFlagEnabled check or removing the isLikelyAWFAPIProxyURL guard would silently pass the entire test suite.

pelikhan · 2026-06-18T13:29:26Z

@copilot run pr-finisher skill

github-actions · 2026-06-18T13:41:57Z

``
@copilot review all comments and address the requested changes in the latest review, especially the missing test coverage and throw-safety issue.

Generated by 👨‍🍳 PR Sous Chef · ◷

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-06-18T13:49:03Z

@copilot run pr-finisher skill

Done. Addressed all open github-actions[bot] review threads:

Wrapped renderTemplateFromFile in a try/catch to preserve the no-throw contract (commits Strengthen copilot 403 test coverage... and Fix template literal inconsistency...)
Exported envFlagEnabled and added a full describe("envFlagEnabled") suite covering all truthy/falsy/undefined cases
Strengthened the proxy-403 test with model=claude-sonnet-4.5 and stage=starting... substitution assertions
Strengthened the truthy-S2STOKENS test with billing URL and not.toContain("COPILOT_PROVIDER_API_KEY") assertions
Added three new negative tests: 403 without S2STOKENS (BYOK mode), 403 with falsy S2STOKENS, and non-proxy 403 with S2STOKENS=true

Local make fmt, make lint, and the full vitest run are all green. CI must be re-triggered by a maintainer before merge.

Done

pelikhan · 2026-06-18T14:01:03Z

+ try {
+ return renderTemplateFromFile(COPILOT_REQUESTS_PROXY_AUTH_403_TEMPLATE_PATH, {
+ selected_model: selectedModel,
+ stage,
+ });
+ } catch {
+ return `Copilot requests authentication failed through the gh-aw API proxy (HTTP 403, model=${selectedModel}, stage=${stage}). Verify that copilot-requests: write is granted and that Copilot org billing is enabled.`;
+ }


@copilot remove try/catch. Assume this will not fail

Done — try/catch removed in the latest commit.

…FailureDiagnostic Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

CopilotAIand others added 2 commits June 18, 2026 12:40

Add copilot 403 org billing guidance
41add2d
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Harden copilot 403 org billing detection
e9c0747
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

CopilotAI assigned Copilot and pelikhan Jun 18, 2026

CopilotAI changed the title ~~Improve Copilot 403 org-billing auth guidance~~Improve Copilot 403 auth guidance for copilot-requests modeJun 18, 2026

Copilotcreated this pull request from a session on behalf of pelikhan June 18, 2026 12:45 View session

Copilotfinished work on behalf of pelikhan June 18, 2026 12:45

CopilotAI requested a review from pelikhan June 18, 2026 12:45

pelikhan reviewed Jun 18, 2026
View reviewed changes

Copilotstarted work on behalf of pelikhan June 18, 2026 12:52 View session

Refactor copilot 403 guidance template
ef58ece
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilotfinished work on behalf of pelikhan June 18, 2026 13:02

CopilotAI requested a review from pelikhan June 18, 2026 13:02

pelikhan marked this pull request as ready for review June 18, 2026 13:05

CopilotAI review requested due to automatic review settings June 18, 2026 13:05

Copilotstarted reviewing on behalf of pelikhan June 18, 2026 13:05 View session

CopilotAI approved these changes Jun 18, 2026
View reviewed changes

github-actionsBot mentioned this pull request Jun 18, 2026
[aw] No-Op Runs #39849
Closed

github-actionsBot approved these changes Jun 18, 2026
View reviewed changes

github-actionsBot reviewed Jun 18, 2026
View reviewed changes

Comment threadactions/setup/js/copilot_harness.test.cjs
Comment threadactions/setup/js/copilot_harness.test.cjs
Comment threadactions/setup/js/copilot_harness.cjs
Comment threadactions/setup/js/copilot_harness.cjs

github-actionsBot previously requested changes Jun 18, 2026
View reviewed changes

github-actionsBot mentioned this pull request Jun 18, 2026
[PR Triage Report] [PR Triage] Copilot Agent PR Queue — 2026-06-18 (Run #27762753975) #40066
Closed

Copilotstarted work on behalf of pelikhan June 18, 2026 13:36 View session

CopilotAIand others added 2 commits June 18, 2026 13:46

Strengthen copilot 403 test coverage and add envFlagEnabled unit tests
23bce86
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Fix template literal inconsistency in try/catch fallback
fdb7934
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilotfinished work on behalf of pelikhan June 18, 2026 13:49

pelikhan reviewed Jun 18, 2026
View reviewed changes

Copilotstarted work on behalf of pelikhan June 18, 2026 14:04 View session

Remove try/catch from renderTemplateFromFile in buildCopilotProxyAuth…
30d839e
…FailureDiagnostic Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilotfinished work on behalf of pelikhan June 18, 2026 14:07

CopilotAI requested a review from pelikhan June 18, 2026 14:07

pelikhan merged commit 7dd6030 into mainJun 18, 2026
14 checks passed

pelikhan deleted the copilot/update-copilot-403-error-message branch June 18, 2026 14:17

coderabbitaiBot mentioned this pull request Jul 28, 2026
docs(integrations): modernise the gh-aw consumer for token-based inference matthiasbigl/err2issue#4
Merged
7 tasks

qodo-code-reviewBot mentioned this pull request Aug 21, 2026
fix(agentic): restore bounded B3 runtime evidence timerloggedout-spec/termux-monorepo#301
Merged

Uh oh!

Conversation

CopilotAI commented Jun 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelikhanJun 18, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAIJun 18, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

✅ Ready to approve

Copilot's findings

Uh oh!

github-actionsBot commented Jun 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Jun 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Jun 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Jun 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Jun 18, 2026

🧪 Test Quality Sentinel Report

Verdict

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

REQUEST_CHANGES — 3 test coverage gaps + 1 throw-safety issue

Uh oh!

Uh oh!

github-actionsBotJun 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pelikhan commented Jun 18, 2026

Uh oh!

github-actionsBot commented Jun 18, 2026

Uh oh!

CopilotAI commented Jun 18, 2026

Uh oh!

pelikhanJun 18, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAIJun 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CopilotAI commented Jun 18, 2026•
edited
Loading

github-actionsBot commented Jun 18, 2026•
edited
Loading

github-actionsBot commented Jun 18, 2026•
edited
Loading

github-actionsBot commented Jun 18, 2026•
edited
Loading

github-actionsBot commented Jun 18, 2026•
edited
Loading