Uh oh!
There was an error while loading. Please reload this page.
Hotfix: demote azure.archive.ubuntu.com mirror priority, add apt acquire timeouts - #14596
Open
v-AndriiKhyliuk with Copilot wants to merge 2 commits into
Open
Hotfix: demote azure.archive.ubuntu.com mirror priority, add apt acquire timeouts#14596v-AndriiKhyliuk with Copilot wants to merge 2 commits into
v-AndriiKhyliuk with Copilot wants to merge 2 commits into
Conversation
Co-authored-by: v-AndriiKhyliuk <249118420+v-AndriiKhyliuk@users.noreply.github.com>
CopilotAI
changed the title
[WIP] Fix priority of Azure mirror in apt sourcesHotfix: demote azure.archive.ubuntu.com mirror priority, add apt acquire timeoutsAug 19, 2026
v-AndriiKhyliuk
marked this pull request as ready for review
August 19, 2026 19:23
Contributor
There was a problem hiding this comment.
Pull request overview
This hotfix reduces the chance of Ubuntu image builds (and downstream user workflows) hanging indefinitely or selecting an unhealthy apt mirror by (1) demoting azure.archive.ubuntu.com to lowest mirror priority and (2) adding explicit apt acquire timeouts as a fail-fast safeguard.
Changes:
- Reordered
/etc/apt/apt-mirrors.txtpriorities soarchive.ubuntu.comandsecurity.ubuntu.comare preferred overazure.archive.ubuntu.com. - Added
/etc/apt/apt.conf.d/81-timeoutsto bound HTTP/HTTPS acquire operations to 30 seconds, preventing silent hangs on stalled connections.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| images/ubuntu/scripts/build/configure-apt.sh | Adds apt acquire timeouts via a new 81-timeouts config file. |
| images/ubuntu/scripts/build/configure-apt-sources.sh | Reorders mirror priorities to prefer upstream Ubuntu mirrors over Azure. |
| images/ubuntu-slim/scripts/build/configure-apt.sh | Adds the same apt acquire timeouts for slim images. |
| images/ubuntu-slim/scripts/build/configure-apt-sources.sh | Reorders mirror priorities similarly for slim images. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This was referenced Aug 19, 2026
5 tasks
gr2m added a commit
to vercel/ai
that referenced
this pull request
Aug 19, 2026
## Background The Test RSC e2e job failed three times on August 19 while Playwright installed Chromium system dependencies: - https://github.com/vercel/ai/actions/runs/32226175669/job/95987241660 - https://github.com/vercel/ai/actions/runs/32296621196/job/96210594834 - https://github.com/vercel/ai/actions/runs/32302334578/job/96228309554 In each run, apt-get update stalled against azure.archive.ubuntu.com until the 15-minute timeout. Playwright had spawned apt-get through a nested sudo process, so apt-get survived the timeout and retained /var/lib/apt/lists/lock. The retry guard only watched /var/lib/dpkg/lock-frontend, causing attempts two and three to fail immediately. This matches the active GitHub runner-images incident: actions/runner-images#14594 ## Summary - run Playwright and its timeout as root so apt-get remains in the timeout process group - add apt acquisition timeouts and a bounded retry so stalled mirrors fail over promptly - wait for the apt lists, archives, and dpkg locks before every attempt - report an explicit error if locks remain held for five minutes ## End-to-End Verification This is a CI-only change. The intermittent mirror timeout cannot be forced locally; this pull request CI exercises the normal Test RSC e2e path. The timeout path now keeps the Playwright process and its privileged apt descendants in one process group and checks the exact lock observed in all three failures before retrying. ## Validation - pnpm check - pnpm type-check:full - workflow YAML parsing - embedded Bash syntax check - git diff --check ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [ ] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [ ] A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root) - [x] I have reviewed this pull request (self-review) ## Related Issues - actions/runner-images#14594 - actions/runner-images#14596
bghgary added a commit
to bghgary/arcana.cpp
that referenced
this pull request
Aug 19, 2026
The runner image deliberately sets APT::Acquire::Retries 10; overriding it to 1 trades resilience against ordinary transient failures for a faster fail in one pathological case. actions/runner-images#14596 fixes the root cause image-side by demoting the azure mirror, which a workflow cannot do. timeout-minutes already bounds the hang, and it bounds every other hang too. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
| printf "http://azure.archive.ubuntu.com/ubuntu/\tpriority:1\n" | tee -a /etc/apt/apt-mirrors.txt | ||
| printf "https://archive.ubuntu.com/ubuntu/\tpriority:2\n" | tee -a /etc/apt/apt-mirrors.txt | ||
| printf "https://security.ubuntu.com/ubuntu/\tpriority:3\n" | tee -a /etc/apt/apt-mirrors.txt | ||
| printf "https://archive.ubuntu.com/ubuntu/\tpriority:1\n" | tee -a /etc/apt/apt-mirrors.txt |
Contributor
There was a problem hiding this comment.
Do we have to have https here? Intuitively yes but there are mixed opinions on this matter and previously out top pick used plain http. And https for Ubuntu archives was added not long ago.
| @@ -17,6 +17,10 @@ systemctl disable apt-daily-upgrade.service | |||
| # Enable retry logic for apt up to 10 times | |||
| echo "APT::Acquire::Retries \"10\";" > /etc/apt/apt.conf.d/80-retries | |||
Contributor
There was a problem hiding this comment.
APT::Acquire::Retries
Copilot claims that it's Acquire::Retries and by setting this parameter incorrectly we leave it with default 3 retries. Is there a way to confirm that?
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Incident #5183:
apt-get update/apt-get installhung for up to 6 hours on Ubuntu runner images becauseazure.archive.ubuntu.comwas configured as the highest-priority apt mirror and stalled mid-transfer (TCP connects, then hangs) rather than failing cleanly. The existing retry wrapper inconfigure-apt-mock.shonly fires on stderr pattern matches after process exit, so a silent hang never triggers a retry or fallback to a healthier mirror.This hotfix demotes azure to lowest priority (kept as fallback, not removed) and adds an explicit apt timeout as a second line of defense — minimal, low-risk change to unblock the fleet ahead of a deeper retry/timeout rework.
Mirror priority reorder
images/ubuntu/scripts/build/configure-apt-sources.shandimages/ubuntu-slim/scripts/build/configure-apt-sources.sh:archive.ubuntu.com→ priority:1,security.ubuntu.com→ priority:2,azure.archive.ubuntu.com→ priority:3 (was priority:1)Acquire timeout (defense-in-depth)
images/ubuntu/scripts/build/configure-apt.shandimages/ubuntu-slim/scripts/build/configure-apt.sh: new/etc/apt/apt.conf.d/81-timeouts, alongside existing80-retries, bounding a stalled mirror connection instead of letting it hang indefinitely:configure-apt-mock.sh's retry wrapper is untouched — that rework is tracked separately.Check list