Skip to content

ci: adopt the inline pr-agent lane — a public repo cannot call a private reusable workflow - #29

Open
yakimoto wants to merge 6 commits into
mainfrom
ci/adopt-inline-pr-agent
Open

ci: adopt the inline pr-agent lane — a public repo cannot call a private reusable workflow#29
yakimoto wants to merge 6 commits into
mainfrom
ci/adopt-inline-pr-agent

Conversation

@yakimoto

@yakimotoyakimoto commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

User description

User description

User description

Part of the fan-out tracked in wave-pen#388, proven first on wave-certify#44 where pr_agent returned success.

The defect

pr-agent.yml here calls wave-av/wave-foundation/.github/workflows/reusable-pr-agent.yml@main, and wave-foundation is private. GitHub does not permit a public repository to call a reusable workflow from a private one, so the uses: never resolves: the run dies before any job is created — conclusion: failure, total_count: 0, no log, and no check run on the head sha at all.

That is worse than a normal failure. There is nothing to click through to. Every PR on this repo has been carrying a red check that reports nothing, and external contributors see it.

Measured on this repo today — the last 5 pr-agent runs:

failure, failure, failure, failure, failure

Measured across the org on 2026-08-22: 7 public repos / 176 runs / 100% failure; 9 private repos / zero failures. A clean 16/16 split on visibility alone. Three competing hypotheses were each tested and refuted — missing OPENAI_KEY (present in both populations), a dead pinned ref (150ffae2 resolves, file exists at it), and @main vs a pinned sha (wave-realtime-edge pins @main and fails, wave-pen pins @main and works).

The fix already existed and was never adopted

wave-foundation-public/.github/workflows/pr-agent.yml is an inline copy of the same lane with no reference to the private repo. Its own header says it was written for exactly this. This PR adopts it verbatim.

So this is an adoption gap, not a design gap.

Why now, and not when #388 was filed

#388 named two blockers, and both are cleared as of wave-foundation-public#71:

  1. The shared concurrency key. The template carried pr-agent-${{ github.event.pull_request.number || … }}, shared between pull_request and issue_comment, so any bot comment cancelled a live review ~10s in (wave-pen#386). It now keys on github.event_name.
  2. Missing step-level timeouts. The template now carries 6.

Fanning out before those landed would have traded a red-with-no-log lane for a cancelled-on-every-comment lane — a different failure, not a fix.

Verified before opening this

  • The template is genuinely self-contained. Its only two wave-foundation/ mentions are in comments, not in a uses:. Checked rather than assumed, since that is the whole property this depends on.
  • The job id stays pr_agent. A job's id is its check-run context and branch protection matches on (context, app_id), so nothing needs touching on the protection side.
  • The workflow parses, and the source was read from a fresh clone of wave-foundation-public's default branch — not from a local checkout that might be parked on another branch.

The receipt is this PR, not the diff

A red lane and a working lane are indistinguishable until one actually runs — that is the whole reason 176 failures went unexamined. So the proof is pr-agent going green on this PR. If it does, the remaining 27 repos get the same change with evidence behind it. If it does not, we learn that here, on one low-traffic repo, instead of across the org's entire public surface.

Proven before fanning out.wave-certify#44 took this exact change first and its pr_agent run returned success on the pull_request event — a job with a real log, where the broken form produced no job at all. The other repos were not changed on hope.

Refs wave-pen#388

View with [code]smithAutofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Note

Medium Risk
CI-only, but the job holds OPENAI_KEY and write access to PRs. Fail-closed fork gating and advisory (non-blocking) verdicts reduce blast radius; a workflow bug can still skip reviews or spend the LLM key.

Overview
Replaces the uses: of private wave-foundation reusable pr-agent with an inline Qodo Merge job. Public repos cannot call a private reusable workflow, so the old lane failed with zero jobs and no check log.

Concurrency now keys on event name plus pr-vs-issue, so comments no longer cancel in-flight push reviews and issue comments no longer collide with same-numbered PRs.

Adds a fail-closed fork gate on slash commands (the issue_comment payload has no fork flag), member-only / commands, a 6-minute per-attempt timeout with CONFIG__AI_TIMEOUT at 300s, one retry after 45s, and a verdict step that classifies timeout/cancel/429 and never blocks the PR. /review on fork PRs is now declined.

Reviewed by Cursor Bugbot for commit 6ef66e1. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by Sourcery

Adopt a self-contained, guarded PR-agent workflow so public-repository reviews run reliably without exposing credentials or blocking pull requests on advisory-agent failures.

Bug Fixes:

  • Replace the unusable private reusable workflow reference with a self-contained PR-agent lane for public repositories.
  • Prevent fork-originated reviews and untrusted slash commands from using the workflow's credentials.
  • Ensure agent timeouts, retries, and incomplete attempts are classified safely without blocking pull requests.

Enhancements:

  • Separate concurrency for pull requests and issue comments so unrelated events do not cancel active reviews.
  • Improve advisory PR-agent resilience through bounded retries, model timeouts, and neutral handling of reviewer failures.

Review in cubic


PR Type

Bug fix, Enhancement


Description

  • Replaced private workflow reference with inline public workflow

  • Fixed fork handling and added explicit fork validation

  • Added retry logic with timeout management

  • Improved error classification and logging


Diagram Walkthrough

flowchart LR
A[".github/workflows/pr-agent.yml"] --> B["Inline workflow adoption"]
A --> C["Fork validation improvements"]
A --> D["Retry logic with timeout management"]
A --> E["Enhanced error classification"]
Loading

File Walkthrough

Relevant files
Enhancement
pr-agent.yml
Inline workflow adoption with enhanced fork handling

.github/workflows/pr-agent.yml

  • Replaced private workflow reference with inline public workflow
  • Added explicit fork validation via pulls endpoint
  • Implemented retry logic with 45s backoff between attempts
  • Added detailed timeout management and error classification
+325/-6
Documentation
CHANGELOG.md
Update changelog with workflow improvements

CHANGELOG.md

  • Added entry for pr-agent workflow fixes
  • Documented fork handling improvements
  • Noted timeout configuration changes
  • Recorded retry logic implementation
+26/-0

@codeant-ai

codeant-aiBot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 CodeAnt AI — Review Status

StatusCommitStarted (UTC)Finished (UTC)
✅ Incremental review completedd80d077Aug 24, 2026 · 14:3714:37
✅ Incremental review completed21acf4bAug 24, 2026 · 13:3613:36
✅ Reviewed your PRa5a6a7cAug 23, 2026 · 22:3322:34

@sourcery-ai

sourcery-aiBot commented Aug 23, 2026

Copy link
Copy Markdown

Reviewer's Guide

Replaces the broken reusable pr-agent workflow reference with a fully inlined, updated workflow definition tailored for public repos, adding concurrency fixes, timeouts, retry + verdict handling, and updated PR-Agent configuration so the check reliably runs without blocking merges on transient failures.

Sequence diagram for the inline PR-Agent review with retry and verdict

sequenceDiagram
participant GitHub
participant Workflow as Inline pr_agent workflow
participant Agent as PR-Agent
participant Router as LLM router
GitHub->>Workflow: Trigger pull_request or trusted / command
Workflow->>Workflow: Evaluate event-specific concurrency group and job conditions
Workflow->>Agent: Run review attempt 1
Agent->>Router: Request review analysis
Router-->>Agent: Result or upstream error
alt Attempt 1 succeeds
Agent-->>Workflow: success
else Attempt 1 fails
Workflow->>Workflow: Sleep 45 seconds
Workflow->>Agent: Run review attempt 2
Agent->>Router: Request review analysis
Router-->>Agent: Result or error
Agent-->>Workflow: success or failure
end
Workflow->>Workflow: Classify outcomes in verdict step
alt Any attempt succeeds
Workflow-->>GitHub: Success
else Workflow cancellation or timeout
Workflow-->>GitHub: Neutral advisory result
else Reviewer error after retry
Workflow-->>GitHub: Neutral advisory result
end
Loading

Flow diagram for PR-Agent outcome classification

flowchart TD
Start[Run attempt 1] --> First{Attempt 1 succeeds?}
First -->|Yes| Success[Exit success]
First -->|No| Retry[Wait 45 seconds and run attempt 2]
Retry --> Second{Any attempt succeeds?}
Second -->|Yes| Success
Second -->|No| Cancelled{Outcome cancelled?}
Cancelled -->|Yes| Neutral[Exit 0; advisory neutral result]
Cancelled -->|No| Elapsed{Elapsed time reaches step budget?}
Elapsed -->|Yes| Neutral
Elapsed -->|No| Neutral
Loading

File-Level Changes

ChangeDetailsFiles
Inline and modernize the pr-agent GitHub Actions workflow so public repos no longer depend on a private reusable workflow and the lane behaves correctly under concurrency, timeouts, and transient LLM/router failures.
  • Replace the reusable workflow call with a local job definition that runs The-PR-Agent/pr-agent directly using OPENAI_KEY and GITHUB_TOKEN.
  • Update concurrency group to include the GitHub event name, separating pull_request and issue_comment runs to prevent comments from cancelling active reviews.
  • Add explicit job-level and step-level timeouts plus a stamped start time to distinguish hangs from fast failures and keep retries within the overall budget.
  • Introduce guarded job conditions for slash-command comments and PR events to avoid running on bots, drafts, or forks without secrets.
  • Configure PR-Agent with specific OpenAI-based model, routing, and suggestion settings to match the current reusable lane behavior.
  • Add a second PR-Agent retry step with backoff and identical env configuration to mitigate transient upstream failures such as 429s.
  • Add a final verdict step that classifies outcomes (success, never-ran, cancelled, timed-out, failed) and converts advisory-review failures into neutral/warning so the workflow never hard-blocks the PR on reviewer flakes.
.github/workflows/pr-agent.yml

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@cursor

cursorBot commented Aug 23, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_e0c16684-77ba-401f-bab1-b31cda1aa226)

@codeant-aicodeant-aiBot added the size:L This PR changes 100-499 lines, ignoring generated files label Aug 23, 2026
@coderabbitai

coderabbitaiBot commented Aug 23, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 26 minutes.

View limit details

Limit details: You’ve used the included review currently available. Your 91 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 11f29d0e-3ab1-45ae-95d2-48aebaadc9c2

📥 Commits

Reviewing files that changed from the base of the PR and between d90796c and 6ef66e1.

📒 Files selected for processing (2)
  • .github/workflows/pr-agent.yml
  • CHANGELOG.md

Comment @coderabbitai help to get the list of available commands.

@macroscopeapp

macroscopeappBot commented Aug 23, 2026

Copy link
Copy Markdown

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR replaces a nonfunctional workflow reference with a large inline workflow that executes a pinned external action using OPENAI_KEY and pull-request write permissions, while adding new concurrency, fork-gating, retry, and slash-command behavior. Although application runtime and schemas are unchanged and the author owns the files, the workflow’s complexity and security-sensitive side effects require human review.

Not approved because:

  • Credit balance exhausted. Approvability relies on correctness review in order to determine eligibility

Review your spending limits in Billing settings. You can add or adjust custom eligibility rules. Learn more.

Comment thread.github/workflows/pr-agent.yml Outdated
@gitar-bot

gitar-botBot commented Aug 23, 2026

Copy link
Copy Markdown

Note

Automatic reviews are paused because your team has used its included automatic processing for this billing period (headroom scales with your seat count). You can still comment "Gitar review" to run one anytime, and automatic reviews resume on their own by September 1. Add seats for more headroom.
Learn more

Code Review✅ Approved1 resolved / 1 findings

Adopts the inline self-contained PR-Agent workflow to fix private reusable workflow failures on public repositories. Consider refining the verdict step logic to prevent potential misclassification of failures as timeouts.

✅ 1 resolved
Bug: Verdict step can misclassify failures as timeouts (message only)

📄 .github/workflows/pr-agent.yml:177-182
In the verdict step (.github/workflows/pr-agent.yml:177-182), ELAPSED is wall-clock time since AGENT_START, which is stamped once before attempt 1 (line 76) and never reset before attempt 2. Because attempt 1 duration + 45s backoff + attempt 2 duration are all summed, a case where attempt 1 runs close to but under the 360s per-attempt budget, fails, backs off 45s, and attempt 2 also fails quickly will push cumulative ELAPSED past STEP_BUDGET_S even though neither attempt actually timed out, so the step logs 'TIMED OUT' instead of the correct 'failed after 2 attempts / rate-limit' message. This only affects the diagnostic message (exit code is 0 in both branches, so the PR is never blocked), but it will mislead debugging. Fix by re-stamping AGENT_START immediately before each attempt (or tracking each attempt's elapsed time independently) so ELAPSED reflects a single attempt rather than cumulative multi-attempt time.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose 

Was this helpful? React with 👍 / 👎 | Gitar

@github-actions

github-actionsBot commented Aug 23, 2026

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 0a2d119)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Possible Issue

The retry step's env block sets the API key under a different name than attempt 1
(attempt 1 uses OPENAI_KEY, the retry block's first entry is not the same key name).
If this is not just a redaction artifact of the diff, attempt 2 runs without an API key
and will always fail, defeating the retry that this PR adds specifically for upstream 429s.
The inline comment above that block explicitly warns that the two env blocks must stay
identical — worth verifying the actual file content.

env:
OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
Misclassified verdict

ELAPSED is measured from the stamp attempt start step to the verdict step, so it
covers attempt 1 + the 45s backoff + attempt 2, but it is compared against
STEP_BUDGET_S (360s), which is a per-attempt budget. Two attempts that each return
a fast-ish error (e.g. 200s each) sum to 445s and get reported as "pr-agent TIMED OUT —
445s against a 360s per-attempt budget", which is exactly the confidently-wrong cause the
comment above the step warns against. Track per-attempt elapsed (stamp before each
attempt) or compare against 2*budget + backoff instead.

ELAPSED=$(( $(date +%s) - ${AGENT_START:-$(date +%s)} ))if [ "$ELAPSED" -ge "$STEP_BUDGET_S" ]; thenecho "::warning::pr-agent TIMED OUT — ${ELAPSED}s against a ${STEP_BUDGET_S}s per-attempt budget, so an attempt was killed by its step timeout rather than returning an error. A hang, NOT a rate limit. Rendering NEUTRAL: an advisory reviewer must not block the PR (#3128)."exit 0fiecho "::warning::pr-agent failed after 2 attempts (45s backoff, ${ELAPSED}s total — well inside the ${STEP_BUDGET_S}s budget, so it returned an error rather than hanging) — most commonly an upstream 429/rate-limit from the LLM router. Rendering NEUTRAL: an advisory reviewer must not block the PR (#3128)."

@bito-code-review

Copy link
Copy Markdown

The suggestion to re-stamp AGENT_START before the retry attempt is correct and addresses the diagnostic misclassification issue. By updating AGENT_START immediately before the second attempt, the ELAPSED calculation in the verdict step will correctly reflect the duration of the most recent attempt rather than the cumulative time of both attempts and the backoff period.

To apply this fix, add the following step before the retry attempt in .github/workflows/pr-agent.yml:

- name: stamp retry startif: steps.agent.outcome == 'failure'run: echo "AGENT_START=$(date +%s)" >> "$GITHUB_ENV"

This ensures that if the second attempt fails, the ELAPSED variable accurately represents only the time spent on that specific attempt, preventing it from incorrectly triggering the 'TIMED OUT' warning when the cumulative time exceeds the budget.

.github/workflows/pr-agent.yml

- name: stamp retry start
if: steps.agent.outcome == 'failure'
run: echo "AGENT_START=$(date +%s)" >> "$GITHUB_ENV"

@sourcery-aisourcery-aiBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!

Sourcery assessment

Needs a human reviewer. This workflow changes the repository's trust boundary by giving a pinned third-party action an OPENAI key and pull-request write capability, while enabling automatic reviews and commit-able suggestions on qualifying pull requests. Reverting stops future runs, but any secret exposure, API spend, comments, or branch changes made before the revert would require separate remediation.


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

CI: inline pr-agent workflow so public repos don’t call private reusable lanes

🐞 Bug fix⚙️ Configuration changes🕐 20-40 Minutes

Grey Divider

AI Description

• Replace private reusable-workflow call with an inline pr-agent workflow.
• Prevent comment-triggered runs from cancelling active PR reviews via event-scoped concurrency.
• Add step budgets, retry/backoff, and a verdict step that avoids blocking PRs on flake.
Diagram

graph TD
PR["pull_request"] --> WF["Workflow: pr-agent"] --> JOB["Job: pr_agent"] --> A1["Agent attempt 1"] --> D1{"Success?"}
D1 -->|"yes"| OK["Exit 0"]
D1 -->|"no"| BO["Backoff 45s"] --> A2["Agent attempt 2"] --> D2{"Any success?"}
D2 -->|"yes"| OK
D2 -->|"no"| V["Verdict: warn/neutral"]
IC["issue_comment ('/' cmd)"] --> WF
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Make the reusable workflow public
  • ➕ Retains a single source of truth with no per-repo drift
  • ➕ Simplifies updates across many repositories
  • ➖ May be impossible if the workflow depends on private org context
  • ➖ Still couples public repos to an external repo’s availability/permissions
2. Ship a composite action for pr-agent setup
  • ➕ Reusable across repos without reusable-workflow visibility constraints
  • ➕ Encapsulates retry/verdict logic in versioned action releases
  • ➖ More maintenance overhead than copying one workflow file
  • ➖ Still requires updating repos to adopt new action versions
3. Centralize via a public template repo + periodic sync
  • ➕ Keeps public-friendly workflow canonical while reducing drift
  • ➕ Sync can be automated (bot PRs)
  • ➖ Adds tooling/process complexity
  • ➖ Sync failures can silently reintroduce drift

Recommendation: Inlining is the most reliable immediate fix because it removes the forbidden public→private reusable-workflow dependency and restores check-run visibility. If drift becomes painful across many repos, the next step would be a public canonical workflow (or composite action) plus automated syncing; for now, the inline lane is the lowest-risk path that matches GitHub’s visibility constraints.

Files changed (1) +166 / -6

Other (1) +166 / -6
pr-agent.ymlInline pr-agent lane with event-scoped concurrency, retry, and non-blocking verdict+166/-6

Inline pr-agent lane with event-scoped concurrency, retry, and non-blocking verdict

• Replaces the private reusable-workflow 'uses:' reference with an inline job so the workflow can run in a public repo. Updates concurrency grouping to include 'github.event_name', adds job/step timeouts plus a retry with backoff, and adds a verdict step that classifies failures and avoids blocking PRs for advisory flake.

.github/workflows/pr-agent.yml

@qodo-code-review

qodo-code-reviewBot commented Aug 23, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0)📘 Rule violations (0)📜 Skill insights (0)

Grey Divider


Action required

1. Fork slash-command not skipped✓ Resolved🐞 Bug⛨ Security
Description
The issue_comment slash-command path does not check whether the PR comes from a fork, so a trusted
member’s /<cmd> comment can run this job (with OPENAI_KEY) against fork PRs despite the workflow
comment claiming forks are skipped. This is a security/cost-control gap because the job condition
only checks author association and comment prefix, not the PR head repo.
Code

.github/workflows/pr-agent.yml[R58-65]

+ # Slash commands: PR-only + trusted members (cost-abuse guard). Forks skipped (no secrets there).+ if: >-+ ${{+ (github.event_name == 'issue_comment'+ && github.event.issue.pull_request+ && startsWith(github.event.comment.body, '/')+ && contains(fromJSON('["MEMBER","OWNER","COLLABORATOR"]'), github.event.comment.author_association))+ || (github.event_name == 'pull_request'
Evidence
The job-level if only blocks forks on the pull_request branch (head.repo.fork == false) and
has no equivalent check on the issue_comment branch, while the agent step exports OPENAI_KEY
from secrets.

.github/workflows/pr-agent.yml[58-69]
.github/workflows/pr-agent.yml[95-105]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
`jobs.pr_agent.if` blocks fork PRs only for the `pull_request` event path, but not for the `issue_comment` (slash command) path. Because `issue_comment` runs in the base repo context, this job can execute with secrets (e.g., `OPENAI_KEY`) on fork PRs when a MEMBER/OWNER/COLLABORATOR comments `/...`.
## Issue Context
The workflow text says “Forks skipped (no secrets there).” That statement is currently only enforced for `pull_request` triggers.
## Fix Focus Areas
- .github/workflows/pr-agent.yml[58-69]
- .github/workflows/pr-agent.yml[95-105]
## Suggested fix
Add an explicit fork check for the `issue_comment` path. Since `issue_comment` payload doesn’t include `pull_request.head.repo.fork`, fetch the PR via the GitHub API in an early step (only for `issue_comment`) and exit early (0) if `head.repo.fork == true`.
Example approach (high level):
1. Add a `github-script` step gated on `github.event_name == 'issue_comment'` that calls `GET /repos/{owner}/{repo}/pulls/{pull_number}`.
2. Set an output like `is_fork`.
3. Add `if: steps.pr_meta.outputs.is_fork != 'true'` to the agent steps (or fail-fast with a no-op step).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Timeout verdict misclassified✓ Resolved🐞 Bug◔ Observability
Description
The verdict step compares ELAPSED (measured from a single stamp before attempt 1) against the
per-attempt budget (STEP_BUDGET_S=360), so a slow-but-non-timeout failure across attempt 1 + 45s
backoff + attempt 2 can be incorrectly reported as “TIMED OUT.” This produces a confidently wrong
diagnosis in the check output and makes debugging real failures harder.
Code

.github/workflows/pr-agent.yml[R177-180]

+ ELAPSED=$(( $(date +%s) - ${AGENT_START:-$(date +%s)} ))+ if [ "$ELAPSED" -ge "$STEP_BUDGET_S" ]; then+ echo "::warning::pr-agent TIMED OUT — ${ELAPSED}s against a ${STEP_BUDGET_S}s per-attempt budget, so an attempt was killed by its step timeout rather than returning an error. A hang, NOT a rate limit. Rendering NEUTRAL: an advisory reviewer must not block the PR (#3128)."+ exit 0
Evidence
The workflow stamps only once (AGENT_START) before attempt 1, sleeps 45s before retry, sets
STEP_BUDGET_S to 360 (per-attempt), and then compares total elapsed time since the single stamp to
that per-attempt budget to decide whether to print “TIMED OUT.”

.github/workflows/pr-agent.yml[75-76]
.github/workflows/pr-agent.yml[117-125]
.github/workflows/pr-agent.yml[156-159]
.github/workflows/pr-agent.yml[177-180]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
The workflow stamps `AGENT_START` once before attempt 1 and then uses it to compute `ELAPSED` in the verdict step. That elapsed time includes time spent in attempt 1, the 45s sleep, and attempt 2. The verdict then compares that total elapsed time to `STEP_BUDGET_S` (360s), which is described as a per-attempt timeout.
This can misclassify failures as timeouts even when neither attempt hit its 6-minute step timeout.
## Issue Context
`STEP_BUDGET_S` is intended to represent the step timeout (6 minutes) for each attempt, but `ELAPSED` is computed from a single timestamp and therefore is not per-attempt.
## Fix Focus Areas
- .github/workflows/pr-agent.yml[75-76]
- .github/workflows/pr-agent.yml[117-125]
- .github/workflows/pr-agent.yml[156-159]
- .github/workflows/pr-agent.yml[177-180]
## Suggested fix
Track timing separately for attempt 1 and attempt 2 (and exclude backoff), then classify timeout based on the relevant attempt’s elapsed time.
One workable pattern in Actions YAML:
- Add a `stamp attempt 1 start` step (already exists).
- Add a `stamp attempt 2 start` step gated on `steps.agent.outcome == 'failure'` placed immediately before the retry `uses:` step.
- In `verdict`, compute elapsed for attempt 2 if that stamp exists, otherwise use attempt 1 elapsed.
Alternatively, change the timeout heuristic to compare against an upper bound that matches what `ELAPSED` actually measures (e.g., `STEP_BUDGET_S*2 + 45 + buffer`) and adjust the message text so it no longer claims “per-attempt” timeout detection.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ REVIEW.md
Review mode: ⚖️ Balanced: This changes a security- and behavior-sensitive GitHub Actions workflow with retries, concurrency, permissions, secret use, conditional execution, and verdict classification; it is substantial but localized rather than clearly dense enough for redundant extended review.

Grey Divider

Tip of the day
💡 Did you know, you can switch off images and animations for a plain-text comment

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread.github/workflows/pr-agent.yml Outdated
Comment thread.github/workflows/pr-agent.yml Outdated
@github-actions

Copy link
Copy Markdown

PR Code Suggestions ✨

No code suggestions found for the PR.

…ate reusable workflow
This repo's pr-agent.yml calls
`wave-av/wave-foundation/.github/workflows/reusable-pr-agent.yml`, and
wave-foundation is PRIVATE. GitHub does not permit a PUBLIC repository to call
a reusable workflow from a private one, so the `uses:` never resolves: the run
dies before any job is created — conclusion: failure, total_count: 0, no log,
and no check run on the head sha to read. Every PR here has carried a red check
that reports nothing, and external contributors see it.
Measured across the org 2026-08-22: 7 public repos / 176 runs / 100% failure;
9 private repos / zero failures — a clean 16/16 split on visibility alone.
Three competing hypotheses (missing OPENAI_KEY, dead pinned ref, @main vs a
pinned sha) were each tested and refuted.
THE FIX already existed and was never adopted:
wave-foundation-public/.github/workflows/pr-agent.yml is an INLINE copy of the
same lane with no reference to the private repo. This adopts it verbatim.
PROVEN BEFORE FANNING OUT. wave-certify#44 took this exact change first and its
pr_agent run returned SUCCESS on the pull_request event — a job with a real log,
where the broken form produced no job at all. 27 repos were not changed on hope.
Two prerequisites named in wave-pen#388 are cleared as of
wave-foundation-public#71: the shared concurrency key that let any bot comment
cancel a live review ~10s in (wave-pen#386) now keys on github.event_name, and
the lane carries step-level timeouts.
The job id stays `pr_agent`, so the check-run context is unchanged and no
branch protection rule needs touching.
Refs wave-pen#388
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown

Qodo Fixer

✅ Merged (0) · ☑ Fixed (0)

Process

  • ⏭ Skipped (2)

@yakimoto
yakimotoforce-pushed the ci/adopt-inline-pr-agent branch from a5a6a7c to 0a2d119CompareAugust 23, 2026 22:40
@cursor

cursorBot commented Aug 23, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_81991e92-d3aa-440e-95f1-680153d5142d)

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 0a2d119

…t classification
Picks up wave-foundation-public#72, which landed after this PR was opened.
The template this PR originally copied classified timeouts on TOTAL job time
(attempt 1 + 45s backoff + attempt 2) against STEP_BUDGET_S=360, a PER-ATTEMPT
budget. Two healthy-but-slow attempts (~180s each) were therefore reported as
"TIMED OUT ... A hang, NOT a rate limit", and the else-branch claimed the run
was "well inside the budget" from the same misused total. Found by qodo review
on wave-monitor#48 and confirmed against the file.
Now stamps each attempt separately and classifies on the LONGEST attempt, with
if: always() end stamps so an attempt killed BY its step timeout still records
one. Verified by dry-running both cases before the template landed.
Updated in place rather than as a follow-up PR because this has not merged yet
— cheaper, and it keeps the repo from ever carrying the defective version.
Refs wave-av/wave-pen#417, wave-av/wave-pen#388
@cursor

cursorBot commented Aug 24, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_b60e9be6-6e56-44ef-af2e-a47f07c4e756)

@codeant-aicodeant-aiBot added size:L This PR changes 100-499 lines, ignoring generated files and removed size:L This PR changes 100-499 lines, ignoring generated files labels Aug 24, 2026
…s step cap
Re-syncs this PR to the hardened template before it merges, so it cannot land
carrying the two defects it was opened with (wave-pen#418, wave-foundation-public#73).
1. Fork status is now RESOLVED, not assumed. The job-level `if:` refuses forks
on the `pull_request` arm; it structurally cannot on `issue_comment`, because
fork status is absent from that payload — measured, with a positive control:
`issues/<n>.pull_request` carries exactly [diff_url, html_url, merged_at,
patch_url, url], while `pulls/<n>.head.repo.fork` answers. A `fork gate` step
asks the pulls endpoint and FAILS CLOSED: only a literal `false` proceeds; a
404, a revoked token, a rate limit and `.head.repo = null` all skip.
Scope: this lane runs no `actions/checkout`, so fork code is never fetched or
executed and no exfiltration path existed. The durable defect was the comment
claiming "Forks skipped (no secrets there)" — true of one arm, false of the
other, and exactly what would mislead whoever adds a checkout step later.
2. CONFIG__AI_TIMEOUT 600 -> 300, in both env blocks. 600s inside a 360s step is
unreachable: the runner killed the step first, so pr-agent never reached its
own timeout and never fell back to CONFIG__FALLBACK_MODELS.
3. A latent classifier bug the gate exposed: `stamp attempt 2 end` runs under
`if: always()`, so when attempt 2 never ran the arithmetic subtracted from
zero and reported a 1787580408-second attempt as a confident TIMED OUT. Fixed
at the arithmetic; the verdict also gains an explicit `skipped` branch.
The job id stays `pr_agent`, so the check-run context is unchanged and no branch
protection rule needs touching.
Refs wave-pen#418, wave-pen#417, wave-pen#388
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cursor

cursorBot commented Aug 24, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_fa7f97df-c7a7-40d1-96ab-deeb4b127201)

…ce of a true
Review of this wave found the fail-closed gate had a fail-OPEN consumer. Two
reviewers flagged it independently, on two different repos, and they were right.
if: steps.gate.outputs.fork != 'true' # grants when the output is EMPTY
The gate could only fail closed if it always wrote an output. It did, on every
path — so this did not fail open today, and the implicit success() on the
consumer covers a gate that errors outright. But the safety rested on an
argument rather than on the structure, and it is the very argument this change
exists to delete: absence must not read as permission.
Two independent changes, so neither carries the invariant alone:
- the gate now assigns a shell variable that STARTS at `true` and writes ONCE
at the end, so no future edit adding an early exit can emit nothing;
- the consumer requires `== 'false'`, an explicit affirmative, so an empty or
missing output skips the agent.
Also braces both sides of the A2 subtraction in the verdict step. The bare
`ATTEMPT2_START` was CORRECT — POSIX arithmetic expansion evaluates a bare name
as a variable, verified identical (180 == 180) — but a reviewer read it as a
literal token and filed it High. An expression that reads wrong on 27 repos gets
re-filed on 27 repos, so it is normalised rather than defended.
RECEIPTS. actionlint clean; zizmor clean; shellcheck clean. The gate was driven
through all six branches plus the reviewers' no-output scenario: only a literal
`false` reaches AGENT RUNS. The verdict was re-run across all six states and is
unchanged on the five that already worked.
LIVE: wave-av/api-spec merged the previous revision and its pull_request run
executed `fork gate (issue_comment only) -> success` in production, then ran the
agent — so the gate does not wrongly refuse a legitimate same-repo PR.
Upstream: wave-av/wave-foundation-public#73. Refs wave-pen#418, wave-pen#417.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cursor

cursorBot commented Aug 24, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_3f159ce6-3a1d-4e15-bc5f-75155b7ee486)

…mber space
Review found a SECOND concurrency collision, on a different axis from the one
this template already documents (wave-pen#386).
`issue_comment` fires for ISSUES as well as PRs, and GitHub draws both from ONE
number sequence. So a comment on Issue #30 and a `/review` on PR #30 entered the
same concurrency group. Concurrency is evaluated at WORKFLOW level, BEFORE the
job-level `if:` runs — so the Issue comment cancelled the PR review already in
flight, and was then skipped itself, having done nothing.
That is the identical shape as the #386 defect the block above exists to fix,
one axis over: a run that will not review taking the lane from the run that
would have. #386 separated the two EVENTS; it did not separate the two number
spaces inside one event.
pull_request PR 433 -> pr-agent-pull_request-pr-433
issue_comment on PR 30 -> pr-agent-issue_comment-pr-30
issue_comment on ISSUE 30 -> pr-agent-issue_comment-issue-30
The last two used to be one group. actionlint and zizmor clean.
Upstream: wave-av/wave-foundation-public#73. Refs wave-pen#418, wave-pen#417.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cursor

cursorBot commented Aug 24, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_e8ea77f8-5869-40bf-8570-9948df9b7424)

@codeant-aicodeant-aiBot added size:L This PR changes 100-499 lines, ignoring generated files and removed size:L This PR changes 100-499 lines, ignoring generated files labels Aug 24, 2026
A reviewer flagged the missing entry on wave-modules#41. 25 of the 28 repos in
this wave keep the same Keep-a-Changelog convention, so the entry lands in all
of them rather than only the repo whose review happened to catch it — fixing the
reported instance and leaving the class is the pattern this wave keeps undoing.
The change IS user-visible, which is why it belongs here: a maintainer's
`/review` on a fork PR is now declined with a warning instead of silently
running, so contributors on forks see different behaviour.
Refs wave-pen#418, wave-av/wave-foundation-public#73
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cursor

cursorBot commented Aug 24, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_2b0e865a-1da7-4c68-aa10-8d6f617643db)

@yakimoto

Copy link
Copy Markdown
ContributorAuthor

Review disposition — every finding on this PR, actioned or refuted with evidence

This branch has been updated. Reviews across this wave raised nine distinct findings; four were real and are fixed, three are working-as-intended, two are refuted with measurements. Nothing was silently skipped.

Source of truth for the template: wave-av/wave-foundation-public#73. Tracked as wave-pen#418.

Fixed — the reviewers were right

findingdisposition
Fork gate fails openif: … != 'true' grants on an empty outputAccepted. The gate wrote an output on every path, and a gate step that errors is caught by the consumer's implicit success(), so it did not fail open in practice — but the safety rested on an argument rather than the structure. The gate now writes once, from a variable that starts at the refusing value, and the consumer requires == 'false'. Two independent changes, so neither carries the invariant alone.
issue_comment has no fork check / secrets on fork PRsAccepted, and the cause is structural. Fork status is absent from an issue_comment payload — issues/<n>.pull_request carries exactly [diff_url, html_url, merged_at, patch_url, url], while pulls/<n>.head.repo.fork answers. A fork gate step now asks the pulls endpoint.
AI timeout exceeds step timeoutAccepted.CONFIG__AI_TIMEOUT: "600" inside a 360s step is unreachable — the runner killed the step first, so pr-agent never fell back to CONFIG__FALLBACK_MODELS. Now 300, in both env blocks.
issue_comment concurrency collision (PR #N vs Issue #N)Accepted, and it is a genuinely separate axis from wave-pen#386. PRs and Issues share one number sequence, and concurrency is evaluated before the job if: — so a comment on Issue #30 cancelled a /review on PR #30 and then skipped itself. The key now carries a pr/issue discriminator.
Verdict ELAPSED is cumulative, not per-attemptAlready fixed in wave-foundation-public#72, before this wave. Fixing the fork gate then exposed a fourth defect in the same step, see below.
Missing CHANGELOG entryAccepted — and widened. Raised on one repo; 25 of the 28 in this wave share the Keep-a-Changelog convention, so the entry landed in all 25. Fixing the reported instance and leaving the class is the pattern this wave keeps undoing.

Refuted — with the measurement, not an opinion

The-PR-Agent/pr-agent is an unverified org / possible typosquat. The best-reasoned finding here, and it inverts on checking. All three names are one repository:

$ gh api repos/qodo-ai/pr-agent --jq .full_name -> The-PR-Agent/pr-agent
$ gh api repos/Codium-ai/pr-agent --jq .full_name -> The-PR-Agent/pr-agent
stars=12688 created=2023-07-05 fork=false parent=none
description: "PR Agent: The Original Open-Source PR Reviewer. This project is not the Qodo free tier."

GitHub is following an org rename (Codium-ai → qodo-ai → The-PR-Agent) transparently. fork: false with no parent rules out a fork; 12.7k stars and a 2023 creation date rule out a fresh typosquat. The pinned SHA resolves to the same object through either name.

The suggested remedy would make things worse: qodo-ai/pr-agent is a stale name that resolves only via redirect, and a released org name can be re-registered by anyone. Pinning to the current name plus a commit SHA is the stronger position. Keeping as is.

A2 arithmetic is broken — bare ATTEMPT2_START is a literal token. Not so; POSIX arithmetic expansion evaluates a bare name as a variable:

$ ATTEMPT2_END=1000 ATTEMPT2_START=820 bash -c 'echo $(( ${ATTEMPT2_END:-0} - ATTEMPT2_START ))'
180
$ ATTEMPT2_END=1000 ATTEMPT2_START=820 bash -c 'echo $(( ${ATTEMPT2_END:-0} - ${ATTEMPT2_START:-0} ))'
180

Normalised anyway. An expression that reads wrong on 27 repos gets re-filed on 27 repos, so consistency is worth more than being right about it.

Committable suggestions lack contents: write. Not reproduced. Committable suggestions are GitHub ```suggestion blocks posted through the pull-requests API — the human clicks "Commit suggestion" and GitHub commits under their identity; the workflow never pushes. pull-requests: write is granted. Checked against a live run rather than argued: api-spec run `32733642988` ran with `Contents: read` and `commitable_code_suggestions: true`, concluded success, posted 3 comments, and its log contains no permission error (the one `403` substring match is inside a docker layer digest).

Working as intended

  • No actions/checkout. Deliberate, and load-bearing: it is exactly why the fork finding is defence-in-depth rather than a live exfiltration path. Fork code is never fetched or executed.
  • A neutral message with a success exit. pr-agent is an advisory reviewer — it annotates, it never gates correctness — so a flaked reviewer must not block a PR. That is wave-foundation-public#3128's whole point.
  • A gh api rate limit treated the same as a real fork. That is the fail-closed design. "I could not tell" must not reach the same answer as "not a fork" on the arm holding OPENAI_KEY; the cost of erring this way is one skipped advisory review.

One defect no reviewer found, surfaced by fixing the first

stamp attempt 2 end runs under if: always(), so when attempt 2 never ran the verdict subtracted from zero:

::warning::pr-agent TIMED OUT — the longest attempt ran 1787580408s against a 360s per-attempt budget

A 56-year attempt, stated as a confident diagnosis. Latent since #72; the fork gate is simply the first path that reaches it. Fixed at the arithmetic, and the verdict gained an explicit skipped branch.

Receipts

actionlint clean · zizmor --persona=regular clean · shellcheck clean on both new run: blocks. The gate was driven through all six branches plus the no-output scenario — only a literal false reaches AGENT RUNS. The verdict was executed old-vs-new across six states; the five that already worked are byte-identical. Live:wave-av/api-spec merged an earlier revision and its pull_request run executed fork gate (issue_comment only) -> success in production and then ran the agent, so the gate does not wrongly refuse a legitimate same-repo PR.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Review effort 2/5size:LThis PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@yakimoto