Use merge-base diffing in link-check lints - #21762

Closed
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure
Closed

Use merge-base diffing in link-check lints#21762
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure

Conversation

CopilotAI commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

link-check / lint-urls was diffing base..head, so PR jobs could lint unrelated URLs introduced on main after the branch diverged. This narrows both URL and xref linting to lines introduced by the PR by diffing from the merge base instead.

  • Scope

    • Update scripts/lint_urls.sh to use git diff base...head
    • Update scripts/lint_xrefs.sh to use the same merge-base semantics
  • Behavior change

    • Before: lint jobs could inspect files changed only on the base branch
    • After: lint jobs inspect only files/lines reachable from the PR head relative to the merge base
  • Regression coverage

    • Add a focused test that creates diverged main and feature histories and verifies only the feature-side changes are linted for:
      • URL checks
      • cross-reference checks
# before
git diff "$BASE..$HEAD"# after
git diff "$BASE...$HEAD"

Test plan

Added regression coverage in .ci/scripts/tests/test_link_check_diff_selection.py for both lint_urls.sh and lint_xrefs.sh.

@pytorch-bot

pytorch-botBot commented Aug 11, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21762

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit ed0c8cd with merge base e60faa2 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 11, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI changed the title [WIP] Fix failing GitHub Actions job link-check / lint-urlsUse merge-base diffing in link-check lintsAug 11, 2026
CopilotAI requested a review from psiddhAugust 11, 2026 22:37
@psiddh
psiddh marked this pull request as ready for review August 11, 2026 22:40
CopilotAI lite review requested due to automatic review settings August 11, 2026 22:40

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the link-check lint scripts to diff against the merge base (base...head) instead of the base tip (base..head), so URL and xref linting only inspects lines introduced by the PR even when main has advanced since the feature branch diverged.

Changes:

  • Switch scripts/lint_urls.sh and scripts/lint_xrefs.sh from git diff base..head to git diff base...head.
  • Add a regression test that constructs diverged main and feature histories and asserts only feature-side changes are linted.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
scripts/lint_xrefs.shUse merge-base diff semantics for PR-only xref linting.
scripts/lint_urls.shUse merge-base diff semantics for PR-only URL linting.
.ci/scripts/tests/test_link_check_diff_selection.pyRegression test covering diverged-history selection for both URL and xref linting.
Suppressed comments (2)

scripts/lint_urls.sh:78

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss URLs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

scripts/lint_xrefs.sh:43

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss xrefs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadscripts/lint_urls.sh Outdated
Comment threadscripts/lint_xrefs.sh
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI review requested due to automatic review settings August 11, 2026 23:05
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment on lines 30 to 33
# A push creating a tag or branch reports an all zero SHA no remote can serve.
if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags --depth=1 origin "$BASE_REF"
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")
CopilotAI review requested due to automatic review settings August 11, 2026 23:09

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (5)

.github/workflows/_link_check.yml:33

  • This workflow still uses the default shallow checkout (actions/checkout without fetch-depth: 0). If lint_urls.sh / lint_xrefs.sh use merge-base diffing (git diff A...B), a shallow clone can be missing the merge base even after git fetch origin "$BASE_REF", causing merge-base-based diffs to fail.

Consider unshallowing the checkout (or setting fetch-depth: 0) so merge-base computations are reliable.

 if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")

.ci/scripts/tests/test_link_check_diff_selection.py:104

  • Similarly, this xref test won't distinguish BASE..HEAD from BASE...HEAD because main.md is a base-side addition (which becomes a deletion in the diff and is filtered out). Make main remove an xref that exists in the merge base so BASE..HEAD would incorrectly lint it as an added line.
 def test_lint_xrefs_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
textwrap.dedent(
"""\
[feature](docs/feature-target.md)
"""
),
"main.md",
textwrap.dedent(
"""\
[main](docs/main-target.md)
"""
),
)

.ci/scripts/tests/test_link_check_diff_selection.py:80

  • As written, this test won't fail if lint_urls.sh still uses git diff BASE..HEAD: the main.md URL is introduced only on the base side, so it shows up as a deletion (and the script filters to + lines). To make the test distinguish .. vs ..., have main remove a URL that exists in the merge base (so BASE..HEAD would treat it as a new + line in HEAD).
 def test_lint_urls_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
"https://example.com/feature\n",
"main.md",
"https://example.com/main\n",
)

scripts/lint_xrefs.sh:40

  • git diff A...B depends on being able to compute the merge base. In the link-check workflow this script is typically run in an actions/checkout shallow clone (default fetch-depth: 1), and _link_check.yml still fetches BASE_REF shallowly for some jobs, which can leave the merge base unreachable and make this loop fail with "no merge base" errors.

Consider computing the merge base once with git merge-base (so failures can be reported with a clear message) and diffing $merge_base..$2 instead of using ... directly.

 if [ $# -eq 2 ]; then
for filename in $(git diff --name-only --unified=0 "$1...$2"); do
git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \

.ci/scripts/tests/test_link_check_diff_selection.py:24

  • The test setup doesn't currently create any URL/xref in the merge base commit, so it can't detect the difference between git diff BASE..HEAD and git diff BASE...HEAD. To exercise the regression, the merge base should contain a URL/xref that main later removes (so BASE..HEAD would see it as an added line in HEAD, even though it predates the branch).

Without that, these tests can pass even if the scripts still use .. diffing.

This issue also appears in the following locations of the same file:

  • line 74
  • line 90
 self.write("README.md", "base\n")
self.write("docs/feature-target.md", "feature target\n")
self.write("docs/main-target.md", "main target\n")
self.run_cmd("git add README.md docs/feature-target.md docs/main-target.md")

@psiddh

Copy link
Copy Markdown
Contributor

Closing this PR in favir of #21765

@psiddhpsiddh closed this Aug 12, 2026
psiddh added a commit that referenced this pull request Aug 12, 2026
…21765)
### Summary
The URL, xref, and file-size linters diffed `base..head`, where `base`
is the **tip** of the base branch rather than the point the branch left
it. A branch cut before recent commits still carries the lines those
commits replaced, so against the newer tip its old copies read as
additions, and the branch gets blamed for links someone else already
repaired.
#21729 failed exactly this way. It touches 18 files and adds no URLs at
all, but the two-dot diff scoped the lint to 143 files and flagged nine
links that #21694 had already fixed or `@lint-ignore`d. #21707 is
starker: one Python file with no URLs in it, 199 files linted, the same
nine failures.
```
base..head 143 files <- what CI linted
base...head 18 files <- what the PR actually changes
```
The stray `jq: parse error` lines in #21729's log are the same symptom
from the other direction: the job runs the branch's own pre-#21694 copy
of `lint_urls.sh`.
### Fix
The workflow resolves the merge base and passes it down. That is the
half that matters for branches already open: on a `pull_request` the
reusable workflow resolves from the merge commit, so it carries this fix
even though `scripts/` still comes from the branch itself. The scripts
also switch to three-dot ranges so `./scripts/lint_urls.sh main HEAD` by
hand behaves the same. `lint_xrefs.sh` and `lint_file_size.sh` had the
identical bug and get the identical change.
**Cost, stated plainly.** A merge base needs real history, so this
restores `fetch-depth: 0` on the `pull_request` path — line for line
what #17682 removed in February for speed. Measured on this branch, that
is ~25s of added wall clock and ~79s of runner time across the three
concurrent jobs. #17682's 6min → 10s was mostly the runner and Docker
change rather than the fetch depth, though the commit changes both at
once and I can't fully separate them. Pushes and the nightly whole-tree
scan cannot use a merge base and stay shallow:
```yaml
fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
```
The quotes matter — bare `0` is falsy in GitHub expressions, so `&& 0 ||
1` always yields `1` and would silently disable the fix.
**No silent fallback.** When there is no merge base there is no usable
range, so the range is left unset and the linters scan the whole tree.
Substituting the base tip instead produces a range the scripts fail on
quietly: verified against unrelated histories, `lint_urls.sh` and
`lint_xrefs.sh` both exit **0** having checked nothing, while
`lint_file_size.sh` exits 128 and the wrapper reports "some files exceed
the 1 MB limit", which is not what happened. Unset args are loud instead
— verified rc=1 on a tree containing a dead link.
**`--no-color`.** Both `git diff` calls now pass it, matching the `git
grep --no-color` two lines below. With `color.ui = always` in a
developer's config, added lines arrive wrapped in escape sequences,
`grep -E '^\+'` matches nothing, and the check passes having found
nothing (measured: 2 matches → 0).
### Test plan
`.ci/scripts/tests/test_link_check_diff_selection.py` builds a diverged
history where `main` repairs bad links and shrinks an oversized file
while the feature branch simply predates all of it. Four fixture files
each pin a different part, and `curl` is stubbed so there is no network:
| Fixture | Pins |
|---|---|
| `both_sides.md` — edited on both branches | the per-file diff range |
| `big.bin` — oversized at the branch point, shrunk on main |
`lint_file_size.sh`'s range |
| `colorful.gitconfig` — `color.ui = always` | `--no-color` |
| `base_only.md` / `feature_only.md` | that main-only changes stay
invisible and the branch's own additions are still checked |
```
pytest .ci/scripts/tests/test_link_check_diff_selection.py # 4 passed
```
Mutating each changed line individually:
```
inner per-file range -> .. 3 failed caught
lint_file_size range -> .. 1 failed caught
drop --no-color 1 failed caught
file-selection range -> .. 4 passed equivalent mutant, see below
```
The file-selection range is not pinned because it cannot be: with the
per-file diff at three dots, the extra files it selects produce empty
diffs. Replayed against #21707's real 199-file range, both variants emit
byte-identical output. It is changed for consistency, not behaviour.
**Narrowing the scope must not blunt the check**, so each linter also
has a positive control where the branch itself adds the bad thing:
```
lint_urls adds a dead URL -> rc=1, reports example.invalid/dead
lint_xrefs adds a broken reference -> rc=1, reports sub/missing.md
lint_file_size adds a 1MB+ file -> rc=1, reports feature_big.bin
```
End to end on real content: a synthetic commit on top of `main` that
puts #21694's nine repaired links back, as if a PR had added them, gives
`rc=1` with 9 FAIL and 5 OK against the live network. Incidentally
`musl.cc` answered this time where CI saw `000`, and
`pybind/cmake_example` now 404s where CI saw `301` — which is the
retry-then-WARN path earning its keep.
Also replayed #21729's and #21707's exact CI refs through the fixed
scripts (exit 0 each), and ran all three linters plus `lintrunner`
against this PR's own diff.
### Known limitations
- The checkout is `head.sha`, so references still resolve against the
branch tree rather than the merge result — the mirror image of the bug
fixed here. Pre-existing and not worsened by this PR.
- The whole-tree branch swallows a `git grep` failure and exits 0, so on
a git built without PCRE the scan silently checks nothing. Reachable
locally, not on the runners, which is why the nightly scan works.
Pre-existing; worth a follow-up rather than widening this PR.
### Rollout
This does **not** repair a currently red run. A rerun keeps the original
`GITHUB_SHA`, and advancing the base alone does not fire `synchronize`,
so an already-open PR picks the fix up only on its next newly triggered
pull-request run — any push, or close/reopen. That is still cheaper than
a content rebase: no history rewrite, no conflicts, nothing to
re-review.
Supersedes #21762, which fixes the same bug but leaves the checkout
shallow and misses `lint_urls.sh`. Its reviewer's shallow-checkout point
is exactly right: a `--depth=1` fetch writes a shallow graft even into
an otherwise complete clone, after which `git merge-base` fails and
`A...B` is fatal, so the two halves of this change are a pair. The
regression test here is adapted from that PR.
Authored with Claude Code (Claude Opus 5).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@psiddh
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all \u003cpre\u003e\u003ccode\u003e blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks"); } } catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); } })(); (function(){ try { var __m = "github.com"; var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Use merge-base diffing in link-check lints - #21762

Closed
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure
Closed

Use merge-base diffing in link-check lints#21762
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure

Conversation

CopilotAI commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

link-check / lint-urls was diffing base..head, so PR jobs could lint unrelated URLs introduced on main after the branch diverged. This narrows both URL and xref linting to lines introduced by the PR by diffing from the merge base instead.

  • Scope

    • Update scripts/lint_urls.sh to use git diff base...head
    • Update scripts/lint_xrefs.sh to use the same merge-base semantics
  • Behavior change

    • Before: lint jobs could inspect files changed only on the base branch
    • After: lint jobs inspect only files/lines reachable from the PR head relative to the merge base
  • Regression coverage

    • Add a focused test that creates diverged main and feature histories and verifies only the feature-side changes are linted for:
      • URL checks
      • cross-reference checks
# before
git diff "$BASE..$HEAD"# after
git diff "$BASE...$HEAD"

Test plan

Added regression coverage in .ci/scripts/tests/test_link_check_diff_selection.py for both lint_urls.sh and lint_xrefs.sh.

@pytorch-bot

pytorch-botBot commented Aug 11, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21762

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit ed0c8cd with merge base e60faa2 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 11, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI changed the title [WIP] Fix failing GitHub Actions job link-check / lint-urlsUse merge-base diffing in link-check lintsAug 11, 2026
CopilotAI requested a review from psiddhAugust 11, 2026 22:37
@psiddh
psiddh marked this pull request as ready for review August 11, 2026 22:40
CopilotAI lite review requested due to automatic review settings August 11, 2026 22:40

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the link-check lint scripts to diff against the merge base (base...head) instead of the base tip (base..head), so URL and xref linting only inspects lines introduced by the PR even when main has advanced since the feature branch diverged.

Changes:

  • Switch scripts/lint_urls.sh and scripts/lint_xrefs.sh from git diff base..head to git diff base...head.
  • Add a regression test that constructs diverged main and feature histories and asserts only feature-side changes are linted.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
scripts/lint_xrefs.shUse merge-base diff semantics for PR-only xref linting.
scripts/lint_urls.shUse merge-base diff semantics for PR-only URL linting.
.ci/scripts/tests/test_link_check_diff_selection.pyRegression test covering diverged-history selection for both URL and xref linting.
Suppressed comments (2)

scripts/lint_urls.sh:78

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss URLs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

scripts/lint_xrefs.sh:43

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss xrefs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadscripts/lint_urls.sh Outdated
Comment threadscripts/lint_xrefs.sh
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI review requested due to automatic review settings August 11, 2026 23:05
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment on lines 30 to 33
# A push creating a tag or branch reports an all zero SHA no remote can serve.
if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags --depth=1 origin "$BASE_REF"
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")
CopilotAI review requested due to automatic review settings August 11, 2026 23:09

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (5)

.github/workflows/_link_check.yml:33

  • This workflow still uses the default shallow checkout (actions/checkout without fetch-depth: 0). If lint_urls.sh / lint_xrefs.sh use merge-base diffing (git diff A...B), a shallow clone can be missing the merge base even after git fetch origin "$BASE_REF", causing merge-base-based diffs to fail.

Consider unshallowing the checkout (or setting fetch-depth: 0) so merge-base computations are reliable.

 if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")

.ci/scripts/tests/test_link_check_diff_selection.py:104

  • Similarly, this xref test won't distinguish BASE..HEAD from BASE...HEAD because main.md is a base-side addition (which becomes a deletion in the diff and is filtered out). Make main remove an xref that exists in the merge base so BASE..HEAD would incorrectly lint it as an added line.
 def test_lint_xrefs_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
textwrap.dedent(
"""\
[feature](docs/feature-target.md)
"""
),
"main.md",
textwrap.dedent(
"""\
[main](docs/main-target.md)
"""
),
)

.ci/scripts/tests/test_link_check_diff_selection.py:80

  • As written, this test won't fail if lint_urls.sh still uses git diff BASE..HEAD: the main.md URL is introduced only on the base side, so it shows up as a deletion (and the script filters to + lines). To make the test distinguish .. vs ..., have main remove a URL that exists in the merge base (so BASE..HEAD would treat it as a new + line in HEAD).
 def test_lint_urls_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
"https://example.com/feature\n",
"main.md",
"https://example.com/main\n",
)

scripts/lint_xrefs.sh:40

  • git diff A...B depends on being able to compute the merge base. In the link-check workflow this script is typically run in an actions/checkout shallow clone (default fetch-depth: 1), and _link_check.yml still fetches BASE_REF shallowly for some jobs, which can leave the merge base unreachable and make this loop fail with "no merge base" errors.

Consider computing the merge base once with git merge-base (so failures can be reported with a clear message) and diffing $merge_base..$2 instead of using ... directly.

 if [ $# -eq 2 ]; then
for filename in $(git diff --name-only --unified=0 "$1...$2"); do
git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \

.ci/scripts/tests/test_link_check_diff_selection.py:24

  • The test setup doesn't currently create any URL/xref in the merge base commit, so it can't detect the difference between git diff BASE..HEAD and git diff BASE...HEAD. To exercise the regression, the merge base should contain a URL/xref that main later removes (so BASE..HEAD would see it as an added line in HEAD, even though it predates the branch).

Without that, these tests can pass even if the scripts still use .. diffing.

This issue also appears in the following locations of the same file:

  • line 74
  • line 90
 self.write("README.md", "base\n")
self.write("docs/feature-target.md", "feature target\n")
self.write("docs/main-target.md", "main target\n")
self.run_cmd("git add README.md docs/feature-target.md docs/main-target.md")

@psiddh

Copy link
Copy Markdown
Contributor

Closing this PR in favir of #21765

@psiddhpsiddh closed this Aug 12, 2026
psiddh added a commit that referenced this pull request Aug 12, 2026
…21765)
### Summary
The URL, xref, and file-size linters diffed `base..head`, where `base`
is the **tip** of the base branch rather than the point the branch left
it. A branch cut before recent commits still carries the lines those
commits replaced, so against the newer tip its old copies read as
additions, and the branch gets blamed for links someone else already
repaired.
#21729 failed exactly this way. It touches 18 files and adds no URLs at
all, but the two-dot diff scoped the lint to 143 files and flagged nine
links that #21694 had already fixed or `@lint-ignore`d. #21707 is
starker: one Python file with no URLs in it, 199 files linted, the same
nine failures.
```
base..head 143 files <- what CI linted
base...head 18 files <- what the PR actually changes
```
The stray `jq: parse error` lines in #21729's log are the same symptom
from the other direction: the job runs the branch's own pre-#21694 copy
of `lint_urls.sh`.
### Fix
The workflow resolves the merge base and passes it down. That is the
half that matters for branches already open: on a `pull_request` the
reusable workflow resolves from the merge commit, so it carries this fix
even though `scripts/` still comes from the branch itself. The scripts
also switch to three-dot ranges so `./scripts/lint_urls.sh main HEAD` by
hand behaves the same. `lint_xrefs.sh` and `lint_file_size.sh` had the
identical bug and get the identical change.
**Cost, stated plainly.** A merge base needs real history, so this
restores `fetch-depth: 0` on the `pull_request` path — line for line
what #17682 removed in February for speed. Measured on this branch, that
is ~25s of added wall clock and ~79s of runner time across the three
concurrent jobs. #17682's 6min → 10s was mostly the runner and Docker
change rather than the fetch depth, though the commit changes both at
once and I can't fully separate them. Pushes and the nightly whole-tree
scan cannot use a merge base and stay shallow:
```yaml
fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
```
The quotes matter — bare `0` is falsy in GitHub expressions, so `&& 0 ||
1` always yields `1` and would silently disable the fix.
**No silent fallback.** When there is no merge base there is no usable
range, so the range is left unset and the linters scan the whole tree.
Substituting the base tip instead produces a range the scripts fail on
quietly: verified against unrelated histories, `lint_urls.sh` and
`lint_xrefs.sh` both exit **0** having checked nothing, while
`lint_file_size.sh` exits 128 and the wrapper reports "some files exceed
the 1 MB limit", which is not what happened. Unset args are loud instead
— verified rc=1 on a tree containing a dead link.
**`--no-color`.** Both `git diff` calls now pass it, matching the `git
grep --no-color` two lines below. With `color.ui = always` in a
developer's config, added lines arrive wrapped in escape sequences,
`grep -E '^\+'` matches nothing, and the check passes having found
nothing (measured: 2 matches → 0).
### Test plan
`.ci/scripts/tests/test_link_check_diff_selection.py` builds a diverged
history where `main` repairs bad links and shrinks an oversized file
while the feature branch simply predates all of it. Four fixture files
each pin a different part, and `curl` is stubbed so there is no network:
| Fixture | Pins |
|---|---|
| `both_sides.md` — edited on both branches | the per-file diff range |
| `big.bin` — oversized at the branch point, shrunk on main |
`lint_file_size.sh`'s range |
| `colorful.gitconfig` — `color.ui = always` | `--no-color` |
| `base_only.md` / `feature_only.md` | that main-only changes stay
invisible and the branch's own additions are still checked |
```
pytest .ci/scripts/tests/test_link_check_diff_selection.py # 4 passed
```
Mutating each changed line individually:
```
inner per-file range -> .. 3 failed caught
lint_file_size range -> .. 1 failed caught
drop --no-color 1 failed caught
file-selection range -> .. 4 passed equivalent mutant, see below
```
The file-selection range is not pinned because it cannot be: with the
per-file diff at three dots, the extra files it selects produce empty
diffs. Replayed against #21707's real 199-file range, both variants emit
byte-identical output. It is changed for consistency, not behaviour.
**Narrowing the scope must not blunt the check**, so each linter also
has a positive control where the branch itself adds the bad thing:
```
lint_urls adds a dead URL -> rc=1, reports example.invalid/dead
lint_xrefs adds a broken reference -> rc=1, reports sub/missing.md
lint_file_size adds a 1MB+ file -> rc=1, reports feature_big.bin
```
End to end on real content: a synthetic commit on top of `main` that
puts #21694's nine repaired links back, as if a PR had added them, gives
`rc=1` with 9 FAIL and 5 OK against the live network. Incidentally
`musl.cc` answered this time where CI saw `000`, and
`pybind/cmake_example` now 404s where CI saw `301` — which is the
retry-then-WARN path earning its keep.
Also replayed #21729's and #21707's exact CI refs through the fixed
scripts (exit 0 each), and ran all three linters plus `lintrunner`
against this PR's own diff.
### Known limitations
- The checkout is `head.sha`, so references still resolve against the
branch tree rather than the merge result — the mirror image of the bug
fixed here. Pre-existing and not worsened by this PR.
- The whole-tree branch swallows a `git grep` failure and exits 0, so on
a git built without PCRE the scan silently checks nothing. Reachable
locally, not on the runners, which is why the nightly scan works.
Pre-existing; worth a follow-up rather than widening this PR.
### Rollout
This does **not** repair a currently red run. A rerun keeps the original
`GITHUB_SHA`, and advancing the base alone does not fire `synchronize`,
so an already-open PR picks the fix up only on its next newly triggered
pull-request run — any push, or close/reopen. That is still cheaper than
a content rebase: no history rewrite, no conflicts, nothing to
re-review.
Supersedes #21762, which fixes the same bug but leaves the checkout
shallow and misses `lint_urls.sh`. Its reviewer's shallow-checkout point
is exactly right: a `--depth=1` fetch writes a shallow graft even into
an otherwise complete clone, after which `git merge-base` fails and
`A...B` is fatal, so the two halves of this change are a pair. The
regression test here is adapted from that PR.
Authored with Claude Code (Claude Opus 5).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@psiddh
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Use merge-base diffing in link-check lints - #21762

Closed
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure
Closed

Use merge-base diffing in link-check lints#21762
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure

Conversation

CopilotAI commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

link-check / lint-urls was diffing base..head, so PR jobs could lint unrelated URLs introduced on main after the branch diverged. This narrows both URL and xref linting to lines introduced by the PR by diffing from the merge base instead.

  • Scope

    • Update scripts/lint_urls.sh to use git diff base...head
    • Update scripts/lint_xrefs.sh to use the same merge-base semantics
  • Behavior change

    • Before: lint jobs could inspect files changed only on the base branch
    • After: lint jobs inspect only files/lines reachable from the PR head relative to the merge base
  • Regression coverage

    • Add a focused test that creates diverged main and feature histories and verifies only the feature-side changes are linted for:
      • URL checks
      • cross-reference checks
# before
git diff "$BASE..$HEAD"# after
git diff "$BASE...$HEAD"

Test plan

Added regression coverage in .ci/scripts/tests/test_link_check_diff_selection.py for both lint_urls.sh and lint_xrefs.sh.

@pytorch-bot

pytorch-botBot commented Aug 11, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21762

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit ed0c8cd with merge base e60faa2 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 11, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI changed the title [WIP] Fix failing GitHub Actions job link-check / lint-urlsUse merge-base diffing in link-check lintsAug 11, 2026
CopilotAI requested a review from psiddhAugust 11, 2026 22:37
@psiddh
psiddh marked this pull request as ready for review August 11, 2026 22:40
CopilotAI lite review requested due to automatic review settings August 11, 2026 22:40

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the link-check lint scripts to diff against the merge base (base...head) instead of the base tip (base..head), so URL and xref linting only inspects lines introduced by the PR even when main has advanced since the feature branch diverged.

Changes:

  • Switch scripts/lint_urls.sh and scripts/lint_xrefs.sh from git diff base..head to git diff base...head.
  • Add a regression test that constructs diverged main and feature histories and asserts only feature-side changes are linted.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
scripts/lint_xrefs.shUse merge-base diff semantics for PR-only xref linting.
scripts/lint_urls.shUse merge-base diff semantics for PR-only URL linting.
.ci/scripts/tests/test_link_check_diff_selection.pyRegression test covering diverged-history selection for both URL and xref linting.
Suppressed comments (2)

scripts/lint_urls.sh:78

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss URLs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

scripts/lint_xrefs.sh:43

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss xrefs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadscripts/lint_urls.sh Outdated
Comment threadscripts/lint_xrefs.sh
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI review requested due to automatic review settings August 11, 2026 23:05
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment on lines 30 to 33
# A push creating a tag or branch reports an all zero SHA no remote can serve.
if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags --depth=1 origin "$BASE_REF"
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")
CopilotAI review requested due to automatic review settings August 11, 2026 23:09

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (5)

.github/workflows/_link_check.yml:33

  • This workflow still uses the default shallow checkout (actions/checkout without fetch-depth: 0). If lint_urls.sh / lint_xrefs.sh use merge-base diffing (git diff A...B), a shallow clone can be missing the merge base even after git fetch origin "$BASE_REF", causing merge-base-based diffs to fail.

Consider unshallowing the checkout (or setting fetch-depth: 0) so merge-base computations are reliable.

 if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")

.ci/scripts/tests/test_link_check_diff_selection.py:104

  • Similarly, this xref test won't distinguish BASE..HEAD from BASE...HEAD because main.md is a base-side addition (which becomes a deletion in the diff and is filtered out). Make main remove an xref that exists in the merge base so BASE..HEAD would incorrectly lint it as an added line.
 def test_lint_xrefs_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
textwrap.dedent(
"""\
[feature](docs/feature-target.md)
"""
),
"main.md",
textwrap.dedent(
"""\
[main](docs/main-target.md)
"""
),
)

.ci/scripts/tests/test_link_check_diff_selection.py:80

  • As written, this test won't fail if lint_urls.sh still uses git diff BASE..HEAD: the main.md URL is introduced only on the base side, so it shows up as a deletion (and the script filters to + lines). To make the test distinguish .. vs ..., have main remove a URL that exists in the merge base (so BASE..HEAD would treat it as a new + line in HEAD).
 def test_lint_urls_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
"https://example.com/feature\n",
"main.md",
"https://example.com/main\n",
)

scripts/lint_xrefs.sh:40

  • git diff A...B depends on being able to compute the merge base. In the link-check workflow this script is typically run in an actions/checkout shallow clone (default fetch-depth: 1), and _link_check.yml still fetches BASE_REF shallowly for some jobs, which can leave the merge base unreachable and make this loop fail with "no merge base" errors.

Consider computing the merge base once with git merge-base (so failures can be reported with a clear message) and diffing $merge_base..$2 instead of using ... directly.

 if [ $# -eq 2 ]; then
for filename in $(git diff --name-only --unified=0 "$1...$2"); do
git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \

.ci/scripts/tests/test_link_check_diff_selection.py:24

  • The test setup doesn't currently create any URL/xref in the merge base commit, so it can't detect the difference between git diff BASE..HEAD and git diff BASE...HEAD. To exercise the regression, the merge base should contain a URL/xref that main later removes (so BASE..HEAD would see it as an added line in HEAD, even though it predates the branch).

Without that, these tests can pass even if the scripts still use .. diffing.

This issue also appears in the following locations of the same file:

  • line 74
  • line 90
 self.write("README.md", "base\n")
self.write("docs/feature-target.md", "feature target\n")
self.write("docs/main-target.md", "main target\n")
self.run_cmd("git add README.md docs/feature-target.md docs/main-target.md")

@psiddh

Copy link
Copy Markdown
Contributor

Closing this PR in favir of #21765

@psiddhpsiddh closed this Aug 12, 2026
psiddh added a commit that referenced this pull request Aug 12, 2026
…21765)
### Summary
The URL, xref, and file-size linters diffed `base..head`, where `base`
is the **tip** of the base branch rather than the point the branch left
it. A branch cut before recent commits still carries the lines those
commits replaced, so against the newer tip its old copies read as
additions, and the branch gets blamed for links someone else already
repaired.
#21729 failed exactly this way. It touches 18 files and adds no URLs at
all, but the two-dot diff scoped the lint to 143 files and flagged nine
links that #21694 had already fixed or `@lint-ignore`d. #21707 is
starker: one Python file with no URLs in it, 199 files linted, the same
nine failures.
```
base..head 143 files <- what CI linted
base...head 18 files <- what the PR actually changes
```
The stray `jq: parse error` lines in #21729's log are the same symptom
from the other direction: the job runs the branch's own pre-#21694 copy
of `lint_urls.sh`.
### Fix
The workflow resolves the merge base and passes it down. That is the
half that matters for branches already open: on a `pull_request` the
reusable workflow resolves from the merge commit, so it carries this fix
even though `scripts/` still comes from the branch itself. The scripts
also switch to three-dot ranges so `./scripts/lint_urls.sh main HEAD` by
hand behaves the same. `lint_xrefs.sh` and `lint_file_size.sh` had the
identical bug and get the identical change.
**Cost, stated plainly.** A merge base needs real history, so this
restores `fetch-depth: 0` on the `pull_request` path — line for line
what #17682 removed in February for speed. Measured on this branch, that
is ~25s of added wall clock and ~79s of runner time across the three
concurrent jobs. #17682's 6min → 10s was mostly the runner and Docker
change rather than the fetch depth, though the commit changes both at
once and I can't fully separate them. Pushes and the nightly whole-tree
scan cannot use a merge base and stay shallow:
```yaml
fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
```
The quotes matter — bare `0` is falsy in GitHub expressions, so `&& 0 ||
1` always yields `1` and would silently disable the fix.
**No silent fallback.** When there is no merge base there is no usable
range, so the range is left unset and the linters scan the whole tree.
Substituting the base tip instead produces a range the scripts fail on
quietly: verified against unrelated histories, `lint_urls.sh` and
`lint_xrefs.sh` both exit **0** having checked nothing, while
`lint_file_size.sh` exits 128 and the wrapper reports "some files exceed
the 1 MB limit", which is not what happened. Unset args are loud instead
— verified rc=1 on a tree containing a dead link.
**`--no-color`.** Both `git diff` calls now pass it, matching the `git
grep --no-color` two lines below. With `color.ui = always` in a
developer's config, added lines arrive wrapped in escape sequences,
`grep -E '^\+'` matches nothing, and the check passes having found
nothing (measured: 2 matches → 0).
### Test plan
`.ci/scripts/tests/test_link_check_diff_selection.py` builds a diverged
history where `main` repairs bad links and shrinks an oversized file
while the feature branch simply predates all of it. Four fixture files
each pin a different part, and `curl` is stubbed so there is no network:
| Fixture | Pins |
|---|---|
| `both_sides.md` — edited on both branches | the per-file diff range |
| `big.bin` — oversized at the branch point, shrunk on main |
`lint_file_size.sh`'s range |
| `colorful.gitconfig` — `color.ui = always` | `--no-color` |
| `base_only.md` / `feature_only.md` | that main-only changes stay
invisible and the branch's own additions are still checked |
```
pytest .ci/scripts/tests/test_link_check_diff_selection.py # 4 passed
```
Mutating each changed line individually:
```
inner per-file range -> .. 3 failed caught
lint_file_size range -> .. 1 failed caught
drop --no-color 1 failed caught
file-selection range -> .. 4 passed equivalent mutant, see below
```
The file-selection range is not pinned because it cannot be: with the
per-file diff at three dots, the extra files it selects produce empty
diffs. Replayed against #21707's real 199-file range, both variants emit
byte-identical output. It is changed for consistency, not behaviour.
**Narrowing the scope must not blunt the check**, so each linter also
has a positive control where the branch itself adds the bad thing:
```
lint_urls adds a dead URL -> rc=1, reports example.invalid/dead
lint_xrefs adds a broken reference -> rc=1, reports sub/missing.md
lint_file_size adds a 1MB+ file -> rc=1, reports feature_big.bin
```
End to end on real content: a synthetic commit on top of `main` that
puts #21694's nine repaired links back, as if a PR had added them, gives
`rc=1` with 9 FAIL and 5 OK against the live network. Incidentally
`musl.cc` answered this time where CI saw `000`, and
`pybind/cmake_example` now 404s where CI saw `301` — which is the
retry-then-WARN path earning its keep.
Also replayed #21729's and #21707's exact CI refs through the fixed
scripts (exit 0 each), and ran all three linters plus `lintrunner`
against this PR's own diff.
### Known limitations
- The checkout is `head.sha`, so references still resolve against the
branch tree rather than the merge result — the mirror image of the bug
fixed here. Pre-existing and not worsened by this PR.
- The whole-tree branch swallows a `git grep` failure and exits 0, so on
a git built without PCRE the scan silently checks nothing. Reachable
locally, not on the runners, which is why the nightly scan works.
Pre-existing; worth a follow-up rather than widening this PR.
### Rollout
This does **not** repair a currently red run. A rerun keeps the original
`GITHUB_SHA`, and advancing the base alone does not fire `synchronize`,
so an already-open PR picks the fix up only on its next newly triggered
pull-request run — any push, or close/reopen. That is still cheaper than
a content rebase: no history rewrite, no conflicts, nothing to
re-review.
Supersedes #21762, which fixes the same bug but leaves the checkout
shallow and misses `lint_urls.sh`. Its reviewer's shallow-checkout point
is exactly right: a `--depth=1` fetch writes a shallow graft even into
an otherwise complete clone, after which `git merge-base` fails and
`A...B` is fatal, so the two halves of this change are a pair. The
regression test here is adapted from that PR.
Authored with Claude Code (Claude Opus 5).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@psiddh
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length \u003e 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Use merge-base diffing in link-check lints - #21762

Closed
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure
Closed

Use merge-base diffing in link-check lints#21762
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure

Conversation

CopilotAI commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

link-check / lint-urls was diffing base..head, so PR jobs could lint unrelated URLs introduced on main after the branch diverged. This narrows both URL and xref linting to lines introduced by the PR by diffing from the merge base instead.

  • Scope

    • Update scripts/lint_urls.sh to use git diff base...head
    • Update scripts/lint_xrefs.sh to use the same merge-base semantics
  • Behavior change

    • Before: lint jobs could inspect files changed only on the base branch
    • After: lint jobs inspect only files/lines reachable from the PR head relative to the merge base
  • Regression coverage

    • Add a focused test that creates diverged main and feature histories and verifies only the feature-side changes are linted for:
      • URL checks
      • cross-reference checks
# before
git diff "$BASE..$HEAD"# after
git diff "$BASE...$HEAD"

Test plan

Added regression coverage in .ci/scripts/tests/test_link_check_diff_selection.py for both lint_urls.sh and lint_xrefs.sh.

@pytorch-bot

pytorch-botBot commented Aug 11, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21762

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit ed0c8cd with merge base e60faa2 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 11, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI changed the title [WIP] Fix failing GitHub Actions job link-check / lint-urlsUse merge-base diffing in link-check lintsAug 11, 2026
CopilotAI requested a review from psiddhAugust 11, 2026 22:37
@psiddh
psiddh marked this pull request as ready for review August 11, 2026 22:40
CopilotAI lite review requested due to automatic review settings August 11, 2026 22:40

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the link-check lint scripts to diff against the merge base (base...head) instead of the base tip (base..head), so URL and xref linting only inspects lines introduced by the PR even when main has advanced since the feature branch diverged.

Changes:

  • Switch scripts/lint_urls.sh and scripts/lint_xrefs.sh from git diff base..head to git diff base...head.
  • Add a regression test that constructs diverged main and feature histories and asserts only feature-side changes are linted.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
scripts/lint_xrefs.shUse merge-base diff semantics for PR-only xref linting.
scripts/lint_urls.shUse merge-base diff semantics for PR-only URL linting.
.ci/scripts/tests/test_link_check_diff_selection.pyRegression test covering diverged-history selection for both URL and xref linting.
Suppressed comments (2)

scripts/lint_urls.sh:78

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss URLs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

scripts/lint_xrefs.sh:43

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss xrefs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadscripts/lint_urls.sh Outdated
Comment threadscripts/lint_xrefs.sh
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI review requested due to automatic review settings August 11, 2026 23:05
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment on lines 30 to 33
# A push creating a tag or branch reports an all zero SHA no remote can serve.
if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags --depth=1 origin "$BASE_REF"
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")
CopilotAI review requested due to automatic review settings August 11, 2026 23:09

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (5)

.github/workflows/_link_check.yml:33

  • This workflow still uses the default shallow checkout (actions/checkout without fetch-depth: 0). If lint_urls.sh / lint_xrefs.sh use merge-base diffing (git diff A...B), a shallow clone can be missing the merge base even after git fetch origin "$BASE_REF", causing merge-base-based diffs to fail.

Consider unshallowing the checkout (or setting fetch-depth: 0) so merge-base computations are reliable.

 if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")

.ci/scripts/tests/test_link_check_diff_selection.py:104

  • Similarly, this xref test won't distinguish BASE..HEAD from BASE...HEAD because main.md is a base-side addition (which becomes a deletion in the diff and is filtered out). Make main remove an xref that exists in the merge base so BASE..HEAD would incorrectly lint it as an added line.
 def test_lint_xrefs_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
textwrap.dedent(
"""\
[feature](docs/feature-target.md)
"""
),
"main.md",
textwrap.dedent(
"""\
[main](docs/main-target.md)
"""
),
)

.ci/scripts/tests/test_link_check_diff_selection.py:80

  • As written, this test won't fail if lint_urls.sh still uses git diff BASE..HEAD: the main.md URL is introduced only on the base side, so it shows up as a deletion (and the script filters to + lines). To make the test distinguish .. vs ..., have main remove a URL that exists in the merge base (so BASE..HEAD would treat it as a new + line in HEAD).
 def test_lint_urls_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
"https://example.com/feature\n",
"main.md",
"https://example.com/main\n",
)

scripts/lint_xrefs.sh:40

  • git diff A...B depends on being able to compute the merge base. In the link-check workflow this script is typically run in an actions/checkout shallow clone (default fetch-depth: 1), and _link_check.yml still fetches BASE_REF shallowly for some jobs, which can leave the merge base unreachable and make this loop fail with "no merge base" errors.

Consider computing the merge base once with git merge-base (so failures can be reported with a clear message) and diffing $merge_base..$2 instead of using ... directly.

 if [ $# -eq 2 ]; then
for filename in $(git diff --name-only --unified=0 "$1...$2"); do
git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \

.ci/scripts/tests/test_link_check_diff_selection.py:24

  • The test setup doesn't currently create any URL/xref in the merge base commit, so it can't detect the difference between git diff BASE..HEAD and git diff BASE...HEAD. To exercise the regression, the merge base should contain a URL/xref that main later removes (so BASE..HEAD would see it as an added line in HEAD, even though it predates the branch).

Without that, these tests can pass even if the scripts still use .. diffing.

This issue also appears in the following locations of the same file:

  • line 74
  • line 90
 self.write("README.md", "base\n")
self.write("docs/feature-target.md", "feature target\n")
self.write("docs/main-target.md", "main target\n")
self.run_cmd("git add README.md docs/feature-target.md docs/main-target.md")

@psiddh

Copy link
Copy Markdown
Contributor

Closing this PR in favir of #21765

@psiddhpsiddh closed this Aug 12, 2026
psiddh added a commit that referenced this pull request Aug 12, 2026
…21765)
### Summary
The URL, xref, and file-size linters diffed `base..head`, where `base`
is the **tip** of the base branch rather than the point the branch left
it. A branch cut before recent commits still carries the lines those
commits replaced, so against the newer tip its old copies read as
additions, and the branch gets blamed for links someone else already
repaired.
#21729 failed exactly this way. It touches 18 files and adds no URLs at
all, but the two-dot diff scoped the lint to 143 files and flagged nine
links that #21694 had already fixed or `@lint-ignore`d. #21707 is
starker: one Python file with no URLs in it, 199 files linted, the same
nine failures.
```
base..head 143 files <- what CI linted
base...head 18 files <- what the PR actually changes
```
The stray `jq: parse error` lines in #21729's log are the same symptom
from the other direction: the job runs the branch's own pre-#21694 copy
of `lint_urls.sh`.
### Fix
The workflow resolves the merge base and passes it down. That is the
half that matters for branches already open: on a `pull_request` the
reusable workflow resolves from the merge commit, so it carries this fix
even though `scripts/` still comes from the branch itself. The scripts
also switch to three-dot ranges so `./scripts/lint_urls.sh main HEAD` by
hand behaves the same. `lint_xrefs.sh` and `lint_file_size.sh` had the
identical bug and get the identical change.
**Cost, stated plainly.** A merge base needs real history, so this
restores `fetch-depth: 0` on the `pull_request` path — line for line
what #17682 removed in February for speed. Measured on this branch, that
is ~25s of added wall clock and ~79s of runner time across the three
concurrent jobs. #17682's 6min → 10s was mostly the runner and Docker
change rather than the fetch depth, though the commit changes both at
once and I can't fully separate them. Pushes and the nightly whole-tree
scan cannot use a merge base and stay shallow:
```yaml
fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
```
The quotes matter — bare `0` is falsy in GitHub expressions, so `&& 0 ||
1` always yields `1` and would silently disable the fix.
**No silent fallback.** When there is no merge base there is no usable
range, so the range is left unset and the linters scan the whole tree.
Substituting the base tip instead produces a range the scripts fail on
quietly: verified against unrelated histories, `lint_urls.sh` and
`lint_xrefs.sh` both exit **0** having checked nothing, while
`lint_file_size.sh` exits 128 and the wrapper reports "some files exceed
the 1 MB limit", which is not what happened. Unset args are loud instead
— verified rc=1 on a tree containing a dead link.
**`--no-color`.** Both `git diff` calls now pass it, matching the `git
grep --no-color` two lines below. With `color.ui = always` in a
developer's config, added lines arrive wrapped in escape sequences,
`grep -E '^\+'` matches nothing, and the check passes having found
nothing (measured: 2 matches → 0).
### Test plan
`.ci/scripts/tests/test_link_check_diff_selection.py` builds a diverged
history where `main` repairs bad links and shrinks an oversized file
while the feature branch simply predates all of it. Four fixture files
each pin a different part, and `curl` is stubbed so there is no network:
| Fixture | Pins |
|---|---|
| `both_sides.md` — edited on both branches | the per-file diff range |
| `big.bin` — oversized at the branch point, shrunk on main |
`lint_file_size.sh`'s range |
| `colorful.gitconfig` — `color.ui = always` | `--no-color` |
| `base_only.md` / `feature_only.md` | that main-only changes stay
invisible and the branch's own additions are still checked |
```
pytest .ci/scripts/tests/test_link_check_diff_selection.py # 4 passed
```
Mutating each changed line individually:
```
inner per-file range -> .. 3 failed caught
lint_file_size range -> .. 1 failed caught
drop --no-color 1 failed caught
file-selection range -> .. 4 passed equivalent mutant, see below
```
The file-selection range is not pinned because it cannot be: with the
per-file diff at three dots, the extra files it selects produce empty
diffs. Replayed against #21707's real 199-file range, both variants emit
byte-identical output. It is changed for consistency, not behaviour.
**Narrowing the scope must not blunt the check**, so each linter also
has a positive control where the branch itself adds the bad thing:
```
lint_urls adds a dead URL -> rc=1, reports example.invalid/dead
lint_xrefs adds a broken reference -> rc=1, reports sub/missing.md
lint_file_size adds a 1MB+ file -> rc=1, reports feature_big.bin
```
End to end on real content: a synthetic commit on top of `main` that
puts #21694's nine repaired links back, as if a PR had added them, gives
`rc=1` with 9 FAIL and 5 OK against the live network. Incidentally
`musl.cc` answered this time where CI saw `000`, and
`pybind/cmake_example` now 404s where CI saw `301` — which is the
retry-then-WARN path earning its keep.
Also replayed #21729's and #21707's exact CI refs through the fixed
scripts (exit 0 each), and ran all three linters plus `lintrunner`
against this PR's own diff.
### Known limitations
- The checkout is `head.sha`, so references still resolve against the
branch tree rather than the merge result — the mirror image of the bug
fixed here. Pre-existing and not worsened by this PR.
- The whole-tree branch swallows a `git grep` failure and exits 0, so on
a git built without PCRE the scan silently checks nothing. Reachable
locally, not on the runners, which is why the nightly scan works.
Pre-existing; worth a follow-up rather than widening this PR.
### Rollout
This does **not** repair a currently red run. A rerun keeps the original
`GITHUB_SHA`, and advancing the base alone does not fire `synchronize`,
so an already-open PR picks the fix up only on its next newly triggered
pull-request run — any push, or close/reopen. That is still cheaper than
a content rebase: no history rewrite, no conflicts, nothing to
re-review.
Supersedes #21762, which fixes the same bug but leaves the checkout
shallow and misses `lint_urls.sh`. Its reviewer's shallow-checkout point
is exactly right: a `--depth=1` fetch writes a shallow graft even into
an otherwise complete clone, after which `git merge-base` fails and
`A...B` is fatal, so the two halves of this change are a pair. The
regression test here is adapted from that PR.
Authored with Claude Code (Claude Opus 5).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@psiddh
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Use merge-base diffing in link-check lints - #21762

Closed
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure
Closed

Use merge-base diffing in link-check lints#21762
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure

Conversation

CopilotAI commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

link-check / lint-urls was diffing base..head, so PR jobs could lint unrelated URLs introduced on main after the branch diverged. This narrows both URL and xref linting to lines introduced by the PR by diffing from the merge base instead.

  • Scope

    • Update scripts/lint_urls.sh to use git diff base...head
    • Update scripts/lint_xrefs.sh to use the same merge-base semantics
  • Behavior change

    • Before: lint jobs could inspect files changed only on the base branch
    • After: lint jobs inspect only files/lines reachable from the PR head relative to the merge base
  • Regression coverage

    • Add a focused test that creates diverged main and feature histories and verifies only the feature-side changes are linted for:
      • URL checks
      • cross-reference checks
# before
git diff "$BASE..$HEAD"# after
git diff "$BASE...$HEAD"

Test plan

Added regression coverage in .ci/scripts/tests/test_link_check_diff_selection.py for both lint_urls.sh and lint_xrefs.sh.

@pytorch-bot

pytorch-botBot commented Aug 11, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21762

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit ed0c8cd with merge base e60faa2 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 11, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI changed the title [WIP] Fix failing GitHub Actions job link-check / lint-urlsUse merge-base diffing in link-check lintsAug 11, 2026
CopilotAI requested a review from psiddhAugust 11, 2026 22:37
@psiddh
psiddh marked this pull request as ready for review August 11, 2026 22:40
CopilotAI lite review requested due to automatic review settings August 11, 2026 22:40

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the link-check lint scripts to diff against the merge base (base...head) instead of the base tip (base..head), so URL and xref linting only inspects lines introduced by the PR even when main has advanced since the feature branch diverged.

Changes:

  • Switch scripts/lint_urls.sh and scripts/lint_xrefs.sh from git diff base..head to git diff base...head.
  • Add a regression test that constructs diverged main and feature histories and asserts only feature-side changes are linted.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
scripts/lint_xrefs.shUse merge-base diff semantics for PR-only xref linting.
scripts/lint_urls.shUse merge-base diff semantics for PR-only URL linting.
.ci/scripts/tests/test_link_check_diff_selection.pyRegression test covering diverged-history selection for both URL and xref linting.
Suppressed comments (2)

scripts/lint_urls.sh:78

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss URLs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

scripts/lint_xrefs.sh:43

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss xrefs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadscripts/lint_urls.sh Outdated
Comment threadscripts/lint_xrefs.sh
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI review requested due to automatic review settings August 11, 2026 23:05
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment on lines 30 to 33
# A push creating a tag or branch reports an all zero SHA no remote can serve.
if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags --depth=1 origin "$BASE_REF"
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")
CopilotAI review requested due to automatic review settings August 11, 2026 23:09

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (5)

.github/workflows/_link_check.yml:33

  • This workflow still uses the default shallow checkout (actions/checkout without fetch-depth: 0). If lint_urls.sh / lint_xrefs.sh use merge-base diffing (git diff A...B), a shallow clone can be missing the merge base even after git fetch origin "$BASE_REF", causing merge-base-based diffs to fail.

Consider unshallowing the checkout (or setting fetch-depth: 0) so merge-base computations are reliable.

 if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")

.ci/scripts/tests/test_link_check_diff_selection.py:104

  • Similarly, this xref test won't distinguish BASE..HEAD from BASE...HEAD because main.md is a base-side addition (which becomes a deletion in the diff and is filtered out). Make main remove an xref that exists in the merge base so BASE..HEAD would incorrectly lint it as an added line.
 def test_lint_xrefs_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
textwrap.dedent(
"""\
[feature](docs/feature-target.md)
"""
),
"main.md",
textwrap.dedent(
"""\
[main](docs/main-target.md)
"""
),
)

.ci/scripts/tests/test_link_check_diff_selection.py:80

  • As written, this test won't fail if lint_urls.sh still uses git diff BASE..HEAD: the main.md URL is introduced only on the base side, so it shows up as a deletion (and the script filters to + lines). To make the test distinguish .. vs ..., have main remove a URL that exists in the merge base (so BASE..HEAD would treat it as a new + line in HEAD).
 def test_lint_urls_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
"https://example.com/feature\n",
"main.md",
"https://example.com/main\n",
)

scripts/lint_xrefs.sh:40

  • git diff A...B depends on being able to compute the merge base. In the link-check workflow this script is typically run in an actions/checkout shallow clone (default fetch-depth: 1), and _link_check.yml still fetches BASE_REF shallowly for some jobs, which can leave the merge base unreachable and make this loop fail with "no merge base" errors.

Consider computing the merge base once with git merge-base (so failures can be reported with a clear message) and diffing $merge_base..$2 instead of using ... directly.

 if [ $# -eq 2 ]; then
for filename in $(git diff --name-only --unified=0 "$1...$2"); do
git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \

.ci/scripts/tests/test_link_check_diff_selection.py:24

  • The test setup doesn't currently create any URL/xref in the merge base commit, so it can't detect the difference between git diff BASE..HEAD and git diff BASE...HEAD. To exercise the regression, the merge base should contain a URL/xref that main later removes (so BASE..HEAD would see it as an added line in HEAD, even though it predates the branch).

Without that, these tests can pass even if the scripts still use .. diffing.

This issue also appears in the following locations of the same file:

  • line 74
  • line 90
 self.write("README.md", "base\n")
self.write("docs/feature-target.md", "feature target\n")
self.write("docs/main-target.md", "main target\n")
self.run_cmd("git add README.md docs/feature-target.md docs/main-target.md")

@psiddh

Copy link
Copy Markdown
Contributor

Closing this PR in favir of #21765

@psiddhpsiddh closed this Aug 12, 2026
psiddh added a commit that referenced this pull request Aug 12, 2026
…21765)
### Summary
The URL, xref, and file-size linters diffed `base..head`, where `base`
is the **tip** of the base branch rather than the point the branch left
it. A branch cut before recent commits still carries the lines those
commits replaced, so against the newer tip its old copies read as
additions, and the branch gets blamed for links someone else already
repaired.
#21729 failed exactly this way. It touches 18 files and adds no URLs at
all, but the two-dot diff scoped the lint to 143 files and flagged nine
links that #21694 had already fixed or `@lint-ignore`d. #21707 is
starker: one Python file with no URLs in it, 199 files linted, the same
nine failures.
```
base..head 143 files <- what CI linted
base...head 18 files <- what the PR actually changes
```
The stray `jq: parse error` lines in #21729's log are the same symptom
from the other direction: the job runs the branch's own pre-#21694 copy
of `lint_urls.sh`.
### Fix
The workflow resolves the merge base and passes it down. That is the
half that matters for branches already open: on a `pull_request` the
reusable workflow resolves from the merge commit, so it carries this fix
even though `scripts/` still comes from the branch itself. The scripts
also switch to three-dot ranges so `./scripts/lint_urls.sh main HEAD` by
hand behaves the same. `lint_xrefs.sh` and `lint_file_size.sh` had the
identical bug and get the identical change.
**Cost, stated plainly.** A merge base needs real history, so this
restores `fetch-depth: 0` on the `pull_request` path — line for line
what #17682 removed in February for speed. Measured on this branch, that
is ~25s of added wall clock and ~79s of runner time across the three
concurrent jobs. #17682's 6min → 10s was mostly the runner and Docker
change rather than the fetch depth, though the commit changes both at
once and I can't fully separate them. Pushes and the nightly whole-tree
scan cannot use a merge base and stay shallow:
```yaml
fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
```
The quotes matter — bare `0` is falsy in GitHub expressions, so `&& 0 ||
1` always yields `1` and would silently disable the fix.
**No silent fallback.** When there is no merge base there is no usable
range, so the range is left unset and the linters scan the whole tree.
Substituting the base tip instead produces a range the scripts fail on
quietly: verified against unrelated histories, `lint_urls.sh` and
`lint_xrefs.sh` both exit **0** having checked nothing, while
`lint_file_size.sh` exits 128 and the wrapper reports "some files exceed
the 1 MB limit", which is not what happened. Unset args are loud instead
— verified rc=1 on a tree containing a dead link.
**`--no-color`.** Both `git diff` calls now pass it, matching the `git
grep --no-color` two lines below. With `color.ui = always` in a
developer's config, added lines arrive wrapped in escape sequences,
`grep -E '^\+'` matches nothing, and the check passes having found
nothing (measured: 2 matches → 0).
### Test plan
`.ci/scripts/tests/test_link_check_diff_selection.py` builds a diverged
history where `main` repairs bad links and shrinks an oversized file
while the feature branch simply predates all of it. Four fixture files
each pin a different part, and `curl` is stubbed so there is no network:
| Fixture | Pins |
|---|---|
| `both_sides.md` — edited on both branches | the per-file diff range |
| `big.bin` — oversized at the branch point, shrunk on main |
`lint_file_size.sh`'s range |
| `colorful.gitconfig` — `color.ui = always` | `--no-color` |
| `base_only.md` / `feature_only.md` | that main-only changes stay
invisible and the branch's own additions are still checked |
```
pytest .ci/scripts/tests/test_link_check_diff_selection.py # 4 passed
```
Mutating each changed line individually:
```
inner per-file range -> .. 3 failed caught
lint_file_size range -> .. 1 failed caught
drop --no-color 1 failed caught
file-selection range -> .. 4 passed equivalent mutant, see below
```
The file-selection range is not pinned because it cannot be: with the
per-file diff at three dots, the extra files it selects produce empty
diffs. Replayed against #21707's real 199-file range, both variants emit
byte-identical output. It is changed for consistency, not behaviour.
**Narrowing the scope must not blunt the check**, so each linter also
has a positive control where the branch itself adds the bad thing:
```
lint_urls adds a dead URL -> rc=1, reports example.invalid/dead
lint_xrefs adds a broken reference -> rc=1, reports sub/missing.md
lint_file_size adds a 1MB+ file -> rc=1, reports feature_big.bin
```
End to end on real content: a synthetic commit on top of `main` that
puts #21694's nine repaired links back, as if a PR had added them, gives
`rc=1` with 9 FAIL and 5 OK against the live network. Incidentally
`musl.cc` answered this time where CI saw `000`, and
`pybind/cmake_example` now 404s where CI saw `301` — which is the
retry-then-WARN path earning its keep.
Also replayed #21729's and #21707's exact CI refs through the fixed
scripts (exit 0 each), and ran all three linters plus `lintrunner`
against this PR's own diff.
### Known limitations
- The checkout is `head.sha`, so references still resolve against the
branch tree rather than the merge result — the mirror image of the bug
fixed here. Pre-existing and not worsened by this PR.
- The whole-tree branch swallows a `git grep` failure and exits 0, so on
a git built without PCRE the scan silently checks nothing. Reachable
locally, not on the runners, which is why the nightly scan works.
Pre-existing; worth a follow-up rather than widening this PR.
### Rollout
This does **not** repair a currently red run. A rerun keeps the original
`GITHUB_SHA`, and advancing the base alone does not fire `synchronize`,
so an already-open PR picks the fix up only on its next newly triggered
pull-request run — any push, or close/reopen. That is still cheaper than
a content rebase: no history rewrite, no conflicts, nothing to
re-review.
Supersedes #21762, which fixes the same bug but leaves the checkout
shallow and misses `lint_urls.sh`. Its reviewer's shallow-checkout point
is exactly right: a `--depth=1` fetch writes a shallow graft even into
an otherwise complete clone, after which `git merge-base` fails and
`A...B` is fatal, so the two halves of this change are a pair. The
regression test here is adapted from that PR.
Authored with Claude Code (Claude Opus 5).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@psiddh
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Use merge-base diffing in link-check lints - #21762

Closed
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure
Closed

Use merge-base diffing in link-check lints#21762
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure

Conversation

CopilotAI commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

link-check / lint-urls was diffing base..head, so PR jobs could lint unrelated URLs introduced on main after the branch diverged. This narrows both URL and xref linting to lines introduced by the PR by diffing from the merge base instead.

  • Scope

    • Update scripts/lint_urls.sh to use git diff base...head
    • Update scripts/lint_xrefs.sh to use the same merge-base semantics
  • Behavior change

    • Before: lint jobs could inspect files changed only on the base branch
    • After: lint jobs inspect only files/lines reachable from the PR head relative to the merge base
  • Regression coverage

    • Add a focused test that creates diverged main and feature histories and verifies only the feature-side changes are linted for:
      • URL checks
      • cross-reference checks
# before
git diff "$BASE..$HEAD"# after
git diff "$BASE...$HEAD"

Test plan

Added regression coverage in .ci/scripts/tests/test_link_check_diff_selection.py for both lint_urls.sh and lint_xrefs.sh.

@pytorch-bot

pytorch-botBot commented Aug 11, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21762

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit ed0c8cd with merge base e60faa2 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 11, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI changed the title [WIP] Fix failing GitHub Actions job link-check / lint-urlsUse merge-base diffing in link-check lintsAug 11, 2026
CopilotAI requested a review from psiddhAugust 11, 2026 22:37
@psiddh
psiddh marked this pull request as ready for review August 11, 2026 22:40
CopilotAI lite review requested due to automatic review settings August 11, 2026 22:40

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the link-check lint scripts to diff against the merge base (base...head) instead of the base tip (base..head), so URL and xref linting only inspects lines introduced by the PR even when main has advanced since the feature branch diverged.

Changes:

  • Switch scripts/lint_urls.sh and scripts/lint_xrefs.sh from git diff base..head to git diff base...head.
  • Add a regression test that constructs diverged main and feature histories and asserts only feature-side changes are linted.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
scripts/lint_xrefs.shUse merge-base diff semantics for PR-only xref linting.
scripts/lint_urls.shUse merge-base diff semantics for PR-only URL linting.
.ci/scripts/tests/test_link_check_diff_selection.pyRegression test covering diverged-history selection for both URL and xref linting.
Suppressed comments (2)

scripts/lint_urls.sh:78

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss URLs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

scripts/lint_xrefs.sh:43

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss xrefs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadscripts/lint_urls.sh Outdated
Comment threadscripts/lint_xrefs.sh
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI review requested due to automatic review settings August 11, 2026 23:05
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment on lines 30 to 33
# A push creating a tag or branch reports an all zero SHA no remote can serve.
if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags --depth=1 origin "$BASE_REF"
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")
CopilotAI review requested due to automatic review settings August 11, 2026 23:09

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (5)

.github/workflows/_link_check.yml:33

  • This workflow still uses the default shallow checkout (actions/checkout without fetch-depth: 0). If lint_urls.sh / lint_xrefs.sh use merge-base diffing (git diff A...B), a shallow clone can be missing the merge base even after git fetch origin "$BASE_REF", causing merge-base-based diffs to fail.

Consider unshallowing the checkout (or setting fetch-depth: 0) so merge-base computations are reliable.

 if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")

.ci/scripts/tests/test_link_check_diff_selection.py:104

  • Similarly, this xref test won't distinguish BASE..HEAD from BASE...HEAD because main.md is a base-side addition (which becomes a deletion in the diff and is filtered out). Make main remove an xref that exists in the merge base so BASE..HEAD would incorrectly lint it as an added line.
 def test_lint_xrefs_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
textwrap.dedent(
"""\
[feature](docs/feature-target.md)
"""
),
"main.md",
textwrap.dedent(
"""\
[main](docs/main-target.md)
"""
),
)

.ci/scripts/tests/test_link_check_diff_selection.py:80

  • As written, this test won't fail if lint_urls.sh still uses git diff BASE..HEAD: the main.md URL is introduced only on the base side, so it shows up as a deletion (and the script filters to + lines). To make the test distinguish .. vs ..., have main remove a URL that exists in the merge base (so BASE..HEAD would treat it as a new + line in HEAD).
 def test_lint_urls_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
"https://example.com/feature\n",
"main.md",
"https://example.com/main\n",
)

scripts/lint_xrefs.sh:40

  • git diff A...B depends on being able to compute the merge base. In the link-check workflow this script is typically run in an actions/checkout shallow clone (default fetch-depth: 1), and _link_check.yml still fetches BASE_REF shallowly for some jobs, which can leave the merge base unreachable and make this loop fail with "no merge base" errors.

Consider computing the merge base once with git merge-base (so failures can be reported with a clear message) and diffing $merge_base..$2 instead of using ... directly.

 if [ $# -eq 2 ]; then
for filename in $(git diff --name-only --unified=0 "$1...$2"); do
git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \

.ci/scripts/tests/test_link_check_diff_selection.py:24

  • The test setup doesn't currently create any URL/xref in the merge base commit, so it can't detect the difference between git diff BASE..HEAD and git diff BASE...HEAD. To exercise the regression, the merge base should contain a URL/xref that main later removes (so BASE..HEAD would see it as an added line in HEAD, even though it predates the branch).

Without that, these tests can pass even if the scripts still use .. diffing.

This issue also appears in the following locations of the same file:

  • line 74
  • line 90
 self.write("README.md", "base\n")
self.write("docs/feature-target.md", "feature target\n")
self.write("docs/main-target.md", "main target\n")
self.run_cmd("git add README.md docs/feature-target.md docs/main-target.md")

@psiddh

Copy link
Copy Markdown
Contributor

Closing this PR in favir of #21765

@psiddhpsiddh closed this Aug 12, 2026
psiddh added a commit that referenced this pull request Aug 12, 2026
…21765)
### Summary
The URL, xref, and file-size linters diffed `base..head`, where `base`
is the **tip** of the base branch rather than the point the branch left
it. A branch cut before recent commits still carries the lines those
commits replaced, so against the newer tip its old copies read as
additions, and the branch gets blamed for links someone else already
repaired.
#21729 failed exactly this way. It touches 18 files and adds no URLs at
all, but the two-dot diff scoped the lint to 143 files and flagged nine
links that #21694 had already fixed or `@lint-ignore`d. #21707 is
starker: one Python file with no URLs in it, 199 files linted, the same
nine failures.
```
base..head 143 files <- what CI linted
base...head 18 files <- what the PR actually changes
```
The stray `jq: parse error` lines in #21729's log are the same symptom
from the other direction: the job runs the branch's own pre-#21694 copy
of `lint_urls.sh`.
### Fix
The workflow resolves the merge base and passes it down. That is the
half that matters for branches already open: on a `pull_request` the
reusable workflow resolves from the merge commit, so it carries this fix
even though `scripts/` still comes from the branch itself. The scripts
also switch to three-dot ranges so `./scripts/lint_urls.sh main HEAD` by
hand behaves the same. `lint_xrefs.sh` and `lint_file_size.sh` had the
identical bug and get the identical change.
**Cost, stated plainly.** A merge base needs real history, so this
restores `fetch-depth: 0` on the `pull_request` path — line for line
what #17682 removed in February for speed. Measured on this branch, that
is ~25s of added wall clock and ~79s of runner time across the three
concurrent jobs. #17682's 6min → 10s was mostly the runner and Docker
change rather than the fetch depth, though the commit changes both at
once and I can't fully separate them. Pushes and the nightly whole-tree
scan cannot use a merge base and stay shallow:
```yaml
fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
```
The quotes matter — bare `0` is falsy in GitHub expressions, so `&& 0 ||
1` always yields `1` and would silently disable the fix.
**No silent fallback.** When there is no merge base there is no usable
range, so the range is left unset and the linters scan the whole tree.
Substituting the base tip instead produces a range the scripts fail on
quietly: verified against unrelated histories, `lint_urls.sh` and
`lint_xrefs.sh` both exit **0** having checked nothing, while
`lint_file_size.sh` exits 128 and the wrapper reports "some files exceed
the 1 MB limit", which is not what happened. Unset args are loud instead
— verified rc=1 on a tree containing a dead link.
**`--no-color`.** Both `git diff` calls now pass it, matching the `git
grep --no-color` two lines below. With `color.ui = always` in a
developer's config, added lines arrive wrapped in escape sequences,
`grep -E '^\+'` matches nothing, and the check passes having found
nothing (measured: 2 matches → 0).
### Test plan
`.ci/scripts/tests/test_link_check_diff_selection.py` builds a diverged
history where `main` repairs bad links and shrinks an oversized file
while the feature branch simply predates all of it. Four fixture files
each pin a different part, and `curl` is stubbed so there is no network:
| Fixture | Pins |
|---|---|
| `both_sides.md` — edited on both branches | the per-file diff range |
| `big.bin` — oversized at the branch point, shrunk on main |
`lint_file_size.sh`'s range |
| `colorful.gitconfig` — `color.ui = always` | `--no-color` |
| `base_only.md` / `feature_only.md` | that main-only changes stay
invisible and the branch's own additions are still checked |
```
pytest .ci/scripts/tests/test_link_check_diff_selection.py # 4 passed
```
Mutating each changed line individually:
```
inner per-file range -> .. 3 failed caught
lint_file_size range -> .. 1 failed caught
drop --no-color 1 failed caught
file-selection range -> .. 4 passed equivalent mutant, see below
```
The file-selection range is not pinned because it cannot be: with the
per-file diff at three dots, the extra files it selects produce empty
diffs. Replayed against #21707's real 199-file range, both variants emit
byte-identical output. It is changed for consistency, not behaviour.
**Narrowing the scope must not blunt the check**, so each linter also
has a positive control where the branch itself adds the bad thing:
```
lint_urls adds a dead URL -> rc=1, reports example.invalid/dead
lint_xrefs adds a broken reference -> rc=1, reports sub/missing.md
lint_file_size adds a 1MB+ file -> rc=1, reports feature_big.bin
```
End to end on real content: a synthetic commit on top of `main` that
puts #21694's nine repaired links back, as if a PR had added them, gives
`rc=1` with 9 FAIL and 5 OK against the live network. Incidentally
`musl.cc` answered this time where CI saw `000`, and
`pybind/cmake_example` now 404s where CI saw `301` — which is the
retry-then-WARN path earning its keep.
Also replayed #21729's and #21707's exact CI refs through the fixed
scripts (exit 0 each), and ran all three linters plus `lintrunner`
against this PR's own diff.
### Known limitations
- The checkout is `head.sha`, so references still resolve against the
branch tree rather than the merge result — the mirror image of the bug
fixed here. Pre-existing and not worsened by this PR.
- The whole-tree branch swallows a `git grep` failure and exits 0, so on
a git built without PCRE the scan silently checks nothing. Reachable
locally, not on the runners, which is why the nightly scan works.
Pre-existing; worth a follow-up rather than widening this PR.
### Rollout
This does **not** repair a currently red run. A rerun keeps the original
`GITHUB_SHA`, and advancing the base alone does not fire `synchronize`,
so an already-open PR picks the fix up only on its next newly triggered
pull-request run — any push, or close/reopen. That is still cheaper than
a content rebase: no history rewrite, no conflicts, nothing to
re-review.
Supersedes #21762, which fixes the same bug but leaves the checkout
shallow and misses `lint_urls.sh`. Its reviewer's shallow-checkout point
is exactly right: a `--depth=1` fetch writes a shallow graft even into
an otherwise complete clone, after which `git merge-base` fails and
`A...B` is fatal, so the two halves of this change are a pair. The
regression test here is adapted from that PR.
Authored with Claude Code (Claude Opus 5).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@psiddh
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Use merge-base diffing in link-check lints - #21762

Closed
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure
Closed

Use merge-base diffing in link-check lints#21762
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure

Conversation

CopilotAI commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

link-check / lint-urls was diffing base..head, so PR jobs could lint unrelated URLs introduced on main after the branch diverged. This narrows both URL and xref linting to lines introduced by the PR by diffing from the merge base instead.

  • Scope

    • Update scripts/lint_urls.sh to use git diff base...head
    • Update scripts/lint_xrefs.sh to use the same merge-base semantics
  • Behavior change

    • Before: lint jobs could inspect files changed only on the base branch
    • After: lint jobs inspect only files/lines reachable from the PR head relative to the merge base
  • Regression coverage

    • Add a focused test that creates diverged main and feature histories and verifies only the feature-side changes are linted for:
      • URL checks
      • cross-reference checks
# before
git diff "$BASE..$HEAD"# after
git diff "$BASE...$HEAD"

Test plan

Added regression coverage in .ci/scripts/tests/test_link_check_diff_selection.py for both lint_urls.sh and lint_xrefs.sh.

@pytorch-bot

pytorch-botBot commented Aug 11, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21762

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit ed0c8cd with merge base e60faa2 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 11, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI changed the title [WIP] Fix failing GitHub Actions job link-check / lint-urlsUse merge-base diffing in link-check lintsAug 11, 2026
CopilotAI requested a review from psiddhAugust 11, 2026 22:37
@psiddh
psiddh marked this pull request as ready for review August 11, 2026 22:40
CopilotAI lite review requested due to automatic review settings August 11, 2026 22:40

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the link-check lint scripts to diff against the merge base (base...head) instead of the base tip (base..head), so URL and xref linting only inspects lines introduced by the PR even when main has advanced since the feature branch diverged.

Changes:

  • Switch scripts/lint_urls.sh and scripts/lint_xrefs.sh from git diff base..head to git diff base...head.
  • Add a regression test that constructs diverged main and feature histories and asserts only feature-side changes are linted.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
scripts/lint_xrefs.shUse merge-base diff semantics for PR-only xref linting.
scripts/lint_urls.shUse merge-base diff semantics for PR-only URL linting.
.ci/scripts/tests/test_link_check_diff_selection.pyRegression test covering diverged-history selection for both URL and xref linting.
Suppressed comments (2)

scripts/lint_urls.sh:78

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss URLs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

scripts/lint_xrefs.sh:43

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss xrefs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadscripts/lint_urls.sh Outdated
Comment threadscripts/lint_xrefs.sh
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI review requested due to automatic review settings August 11, 2026 23:05
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment on lines 30 to 33
# A push creating a tag or branch reports an all zero SHA no remote can serve.
if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags --depth=1 origin "$BASE_REF"
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")
CopilotAI review requested due to automatic review settings August 11, 2026 23:09

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (5)

.github/workflows/_link_check.yml:33

  • This workflow still uses the default shallow checkout (actions/checkout without fetch-depth: 0). If lint_urls.sh / lint_xrefs.sh use merge-base diffing (git diff A...B), a shallow clone can be missing the merge base even after git fetch origin "$BASE_REF", causing merge-base-based diffs to fail.

Consider unshallowing the checkout (or setting fetch-depth: 0) so merge-base computations are reliable.

 if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")

.ci/scripts/tests/test_link_check_diff_selection.py:104

  • Similarly, this xref test won't distinguish BASE..HEAD from BASE...HEAD because main.md is a base-side addition (which becomes a deletion in the diff and is filtered out). Make main remove an xref that exists in the merge base so BASE..HEAD would incorrectly lint it as an added line.
 def test_lint_xrefs_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
textwrap.dedent(
"""\
[feature](docs/feature-target.md)
"""
),
"main.md",
textwrap.dedent(
"""\
[main](docs/main-target.md)
"""
),
)

.ci/scripts/tests/test_link_check_diff_selection.py:80

  • As written, this test won't fail if lint_urls.sh still uses git diff BASE..HEAD: the main.md URL is introduced only on the base side, so it shows up as a deletion (and the script filters to + lines). To make the test distinguish .. vs ..., have main remove a URL that exists in the merge base (so BASE..HEAD would treat it as a new + line in HEAD).
 def test_lint_urls_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
"https://example.com/feature\n",
"main.md",
"https://example.com/main\n",
)

scripts/lint_xrefs.sh:40

  • git diff A...B depends on being able to compute the merge base. In the link-check workflow this script is typically run in an actions/checkout shallow clone (default fetch-depth: 1), and _link_check.yml still fetches BASE_REF shallowly for some jobs, which can leave the merge base unreachable and make this loop fail with "no merge base" errors.

Consider computing the merge base once with git merge-base (so failures can be reported with a clear message) and diffing $merge_base..$2 instead of using ... directly.

 if [ $# -eq 2 ]; then
for filename in $(git diff --name-only --unified=0 "$1...$2"); do
git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \

.ci/scripts/tests/test_link_check_diff_selection.py:24

  • The test setup doesn't currently create any URL/xref in the merge base commit, so it can't detect the difference between git diff BASE..HEAD and git diff BASE...HEAD. To exercise the regression, the merge base should contain a URL/xref that main later removes (so BASE..HEAD would see it as an added line in HEAD, even though it predates the branch).

Without that, these tests can pass even if the scripts still use .. diffing.

This issue also appears in the following locations of the same file:

  • line 74
  • line 90
 self.write("README.md", "base\n")
self.write("docs/feature-target.md", "feature target\n")
self.write("docs/main-target.md", "main target\n")
self.run_cmd("git add README.md docs/feature-target.md docs/main-target.md")

@psiddh

Copy link
Copy Markdown
Contributor

Closing this PR in favir of #21765

@psiddhpsiddh closed this Aug 12, 2026
psiddh added a commit that referenced this pull request Aug 12, 2026
…21765)
### Summary
The URL, xref, and file-size linters diffed `base..head`, where `base`
is the **tip** of the base branch rather than the point the branch left
it. A branch cut before recent commits still carries the lines those
commits replaced, so against the newer tip its old copies read as
additions, and the branch gets blamed for links someone else already
repaired.
#21729 failed exactly this way. It touches 18 files and adds no URLs at
all, but the two-dot diff scoped the lint to 143 files and flagged nine
links that #21694 had already fixed or `@lint-ignore`d. #21707 is
starker: one Python file with no URLs in it, 199 files linted, the same
nine failures.
```
base..head 143 files <- what CI linted
base...head 18 files <- what the PR actually changes
```
The stray `jq: parse error` lines in #21729's log are the same symptom
from the other direction: the job runs the branch's own pre-#21694 copy
of `lint_urls.sh`.
### Fix
The workflow resolves the merge base and passes it down. That is the
half that matters for branches already open: on a `pull_request` the
reusable workflow resolves from the merge commit, so it carries this fix
even though `scripts/` still comes from the branch itself. The scripts
also switch to three-dot ranges so `./scripts/lint_urls.sh main HEAD` by
hand behaves the same. `lint_xrefs.sh` and `lint_file_size.sh` had the
identical bug and get the identical change.
**Cost, stated plainly.** A merge base needs real history, so this
restores `fetch-depth: 0` on the `pull_request` path — line for line
what #17682 removed in February for speed. Measured on this branch, that
is ~25s of added wall clock and ~79s of runner time across the three
concurrent jobs. #17682's 6min → 10s was mostly the runner and Docker
change rather than the fetch depth, though the commit changes both at
once and I can't fully separate them. Pushes and the nightly whole-tree
scan cannot use a merge base and stay shallow:
```yaml
fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
```
The quotes matter — bare `0` is falsy in GitHub expressions, so `&& 0 ||
1` always yields `1` and would silently disable the fix.
**No silent fallback.** When there is no merge base there is no usable
range, so the range is left unset and the linters scan the whole tree.
Substituting the base tip instead produces a range the scripts fail on
quietly: verified against unrelated histories, `lint_urls.sh` and
`lint_xrefs.sh` both exit **0** having checked nothing, while
`lint_file_size.sh` exits 128 and the wrapper reports "some files exceed
the 1 MB limit", which is not what happened. Unset args are loud instead
— verified rc=1 on a tree containing a dead link.
**`--no-color`.** Both `git diff` calls now pass it, matching the `git
grep --no-color` two lines below. With `color.ui = always` in a
developer's config, added lines arrive wrapped in escape sequences,
`grep -E '^\+'` matches nothing, and the check passes having found
nothing (measured: 2 matches → 0).
### Test plan
`.ci/scripts/tests/test_link_check_diff_selection.py` builds a diverged
history where `main` repairs bad links and shrinks an oversized file
while the feature branch simply predates all of it. Four fixture files
each pin a different part, and `curl` is stubbed so there is no network:
| Fixture | Pins |
|---|---|
| `both_sides.md` — edited on both branches | the per-file diff range |
| `big.bin` — oversized at the branch point, shrunk on main |
`lint_file_size.sh`'s range |
| `colorful.gitconfig` — `color.ui = always` | `--no-color` |
| `base_only.md` / `feature_only.md` | that main-only changes stay
invisible and the branch's own additions are still checked |
```
pytest .ci/scripts/tests/test_link_check_diff_selection.py # 4 passed
```
Mutating each changed line individually:
```
inner per-file range -> .. 3 failed caught
lint_file_size range -> .. 1 failed caught
drop --no-color 1 failed caught
file-selection range -> .. 4 passed equivalent mutant, see below
```
The file-selection range is not pinned because it cannot be: with the
per-file diff at three dots, the extra files it selects produce empty
diffs. Replayed against #21707's real 199-file range, both variants emit
byte-identical output. It is changed for consistency, not behaviour.
**Narrowing the scope must not blunt the check**, so each linter also
has a positive control where the branch itself adds the bad thing:
```
lint_urls adds a dead URL -> rc=1, reports example.invalid/dead
lint_xrefs adds a broken reference -> rc=1, reports sub/missing.md
lint_file_size adds a 1MB+ file -> rc=1, reports feature_big.bin
```
End to end on real content: a synthetic commit on top of `main` that
puts #21694's nine repaired links back, as if a PR had added them, gives
`rc=1` with 9 FAIL and 5 OK against the live network. Incidentally
`musl.cc` answered this time where CI saw `000`, and
`pybind/cmake_example` now 404s where CI saw `301` — which is the
retry-then-WARN path earning its keep.
Also replayed #21729's and #21707's exact CI refs through the fixed
scripts (exit 0 each), and ran all three linters plus `lintrunner`
against this PR's own diff.
### Known limitations
- The checkout is `head.sha`, so references still resolve against the
branch tree rather than the merge result — the mirror image of the bug
fixed here. Pre-existing and not worsened by this PR.
- The whole-tree branch swallows a `git grep` failure and exits 0, so on
a git built without PCRE the scan silently checks nothing. Reachable
locally, not on the runners, which is why the nightly scan works.
Pre-existing; worth a follow-up rather than widening this PR.
### Rollout
This does **not** repair a currently red run. A rerun keeps the original
`GITHUB_SHA`, and advancing the base alone does not fire `synchronize`,
so an already-open PR picks the fix up only on its next newly triggered
pull-request run — any push, or close/reopen. That is still cheaper than
a content rebase: no history rewrite, no conflicts, nothing to
re-review.
Supersedes #21762, which fixes the same bug but leaves the checkout
shallow and misses `lint_urls.sh`. Its reviewer's shallow-checkout point
is exactly right: a `--depth=1` fetch writes a shallow graft even into
an otherwise complete clone, after which `git merge-base` fails and
`A...B` is fatal, so the two halves of this change are a pair. The
regression test here is adapted from that PR.
Authored with Claude Code (Claude Opus 5).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@psiddh
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Use merge-base diffing in link-check lints - #21762

Closed
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure
Closed

Use merge-base diffing in link-check lints#21762
psiddh with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-lint-urls-failure

Conversation

CopilotAI commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

link-check / lint-urls was diffing base..head, so PR jobs could lint unrelated URLs introduced on main after the branch diverged. This narrows both URL and xref linting to lines introduced by the PR by diffing from the merge base instead.

  • Scope

    • Update scripts/lint_urls.sh to use git diff base...head
    • Update scripts/lint_xrefs.sh to use the same merge-base semantics
  • Behavior change

    • Before: lint jobs could inspect files changed only on the base branch
    • After: lint jobs inspect only files/lines reachable from the PR head relative to the merge base
  • Regression coverage

    • Add a focused test that creates diverged main and feature histories and verifies only the feature-side changes are linted for:
      • URL checks
      • cross-reference checks
# before
git diff "$BASE..$HEAD"# after
git diff "$BASE...$HEAD"

Test plan

Added regression coverage in .ci/scripts/tests/test_link_check_diff_selection.py for both lint_urls.sh and lint_xrefs.sh.

@pytorch-bot

pytorch-botBot commented Aug 11, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21762

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit ed0c8cd with merge base e60faa2 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 11, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI changed the title [WIP] Fix failing GitHub Actions job link-check / lint-urlsUse merge-base diffing in link-check lintsAug 11, 2026
CopilotAI requested a review from psiddhAugust 11, 2026 22:37
@psiddh
psiddh marked this pull request as ready for review August 11, 2026 22:40
CopilotAI lite review requested due to automatic review settings August 11, 2026 22:40

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the link-check lint scripts to diff against the merge base (base...head) instead of the base tip (base..head), so URL and xref linting only inspects lines introduced by the PR even when main has advanced since the feature branch diverged.

Changes:

  • Switch scripts/lint_urls.sh and scripts/lint_xrefs.sh from git diff base..head to git diff base...head.
  • Add a regression test that constructs diverged main and feature histories and asserts only feature-side changes are linted.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
scripts/lint_xrefs.shUse merge-base diff semantics for PR-only xref linting.
scripts/lint_urls.shUse merge-base diff semantics for PR-only URL linting.
.ci/scripts/tests/test_link_check_diff_selection.pyRegression test covering diverged-history selection for both URL and xref linting.
Suppressed comments (2)

scripts/lint_urls.sh:78

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss URLs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

scripts/lint_xrefs.sh:43

  • With set -euo pipefail, this per-file pipeline can terminate the whole diff scan when a changed file has no “+” lines (e.g., deletion-only diffs). In that case grep -Ev '^\+\+\+' returns exit code 1, which aborts the process substitution early and can cause the script to miss xrefs in later files.
 git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \
| grep -Ev '^\+\+\+' \
| perl -nle 'print for m#'"$pattern"'#g' \
| sed 's|^|'"$filename"':|'

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadscripts/lint_urls.sh Outdated
Comment threadscripts/lint_xrefs.sh
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>
CopilotAI review requested due to automatic review settings August 11, 2026 23:05
Co-authored-by: psiddh <2467117+psiddh@users.noreply.github.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment on lines 30 to 33
# A push creating a tag or branch reports an all zero SHA no remote can serve.
if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags --depth=1 origin "$BASE_REF"
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")
CopilotAI review requested due to automatic review settings August 11, 2026 23:09

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (5)

.github/workflows/_link_check.yml:33

  • This workflow still uses the default shallow checkout (actions/checkout without fetch-depth: 0). If lint_urls.sh / lint_xrefs.sh use merge-base diffing (git diff A...B), a shallow clone can be missing the merge base even after git fetch origin "$BASE_REF", causing merge-base-based diffs to fail.

Consider unshallowing the checkout (or setting fetch-depth: 0) so merge-base computations are reliable.

 if [ -n "$BASE_REF" ] && [ "$BASE_REF" != "0000000000000000000000000000000000000000" ]; then
git fetch --no-tags origin "$BASE_REF"
args=("$BASE_REF" "$HEAD_REF")

.ci/scripts/tests/test_link_check_diff_selection.py:104

  • Similarly, this xref test won't distinguish BASE..HEAD from BASE...HEAD because main.md is a base-side addition (which becomes a deletion in the diff and is filtered out). Make main remove an xref that exists in the merge base so BASE..HEAD would incorrectly lint it as an added line.
 def test_lint_xrefs_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
textwrap.dedent(
"""\
[feature](docs/feature-target.md)
"""
),
"main.md",
textwrap.dedent(
"""\
[main](docs/main-target.md)
"""
),
)

.ci/scripts/tests/test_link_check_diff_selection.py:80

  • As written, this test won't fail if lint_urls.sh still uses git diff BASE..HEAD: the main.md URL is introduced only on the base side, so it shows up as a deletion (and the script filters to + lines). To make the test distinguish .. vs ..., have main remove a URL that exists in the merge base (so BASE..HEAD would treat it as a new + line in HEAD).
 def test_lint_urls_uses_merge_base_for_changed_lines(self) -> None:
base_sha, head_sha = self.create_diverged_history(
"feature.md",
"https://example.com/feature\n",
"main.md",
"https://example.com/main\n",
)

scripts/lint_xrefs.sh:40

  • git diff A...B depends on being able to compute the merge base. In the link-check workflow this script is typically run in an actions/checkout shallow clone (default fetch-depth: 1), and _link_check.yml still fetches BASE_REF shallowly for some jobs, which can leave the merge base unreachable and make this loop fail with "no merge base" errors.

Consider computing the merge base once with git merge-base (so failures can be reported with a clear message) and diffing $merge_base..$2 instead of using ... directly.

 if [ $# -eq 2 ]; then
for filename in $(git diff --name-only --unified=0 "$1...$2"); do
git diff --unified=0 "$1...$2" -- "$filename" "${excludes[@]}" \
| grep -E '^\+' \

.ci/scripts/tests/test_link_check_diff_selection.py:24

  • The test setup doesn't currently create any URL/xref in the merge base commit, so it can't detect the difference between git diff BASE..HEAD and git diff BASE...HEAD. To exercise the regression, the merge base should contain a URL/xref that main later removes (so BASE..HEAD would see it as an added line in HEAD, even though it predates the branch).

Without that, these tests can pass even if the scripts still use .. diffing.

This issue also appears in the following locations of the same file:

  • line 74
  • line 90
 self.write("README.md", "base\n")
self.write("docs/feature-target.md", "feature target\n")
self.write("docs/main-target.md", "main target\n")
self.run_cmd("git add README.md docs/feature-target.md docs/main-target.md")

@psiddh

Copy link
Copy Markdown
Contributor

Closing this PR in favir of #21765

@psiddhpsiddh closed this Aug 12, 2026
psiddh added a commit that referenced this pull request Aug 12, 2026
…21765)
### Summary
The URL, xref, and file-size linters diffed `base..head`, where `base`
is the **tip** of the base branch rather than the point the branch left
it. A branch cut before recent commits still carries the lines those
commits replaced, so against the newer tip its old copies read as
additions, and the branch gets blamed for links someone else already
repaired.
#21729 failed exactly this way. It touches 18 files and adds no URLs at
all, but the two-dot diff scoped the lint to 143 files and flagged nine
links that #21694 had already fixed or `@lint-ignore`d. #21707 is
starker: one Python file with no URLs in it, 199 files linted, the same
nine failures.
```
base..head 143 files <- what CI linted
base...head 18 files <- what the PR actually changes
```
The stray `jq: parse error` lines in #21729's log are the same symptom
from the other direction: the job runs the branch's own pre-#21694 copy
of `lint_urls.sh`.
### Fix
The workflow resolves the merge base and passes it down. That is the
half that matters for branches already open: on a `pull_request` the
reusable workflow resolves from the merge commit, so it carries this fix
even though `scripts/` still comes from the branch itself. The scripts
also switch to three-dot ranges so `./scripts/lint_urls.sh main HEAD` by
hand behaves the same. `lint_xrefs.sh` and `lint_file_size.sh` had the
identical bug and get the identical change.
**Cost, stated plainly.** A merge base needs real history, so this
restores `fetch-depth: 0` on the `pull_request` path — line for line
what #17682 removed in February for speed. Measured on this branch, that
is ~25s of added wall clock and ~79s of runner time across the three
concurrent jobs. #17682's 6min → 10s was mostly the runner and Docker
change rather than the fetch depth, though the commit changes both at
once and I can't fully separate them. Pushes and the nightly whole-tree
scan cannot use a merge base and stay shallow:
```yaml
fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
```
The quotes matter — bare `0` is falsy in GitHub expressions, so `&& 0 ||
1` always yields `1` and would silently disable the fix.
**No silent fallback.** When there is no merge base there is no usable
range, so the range is left unset and the linters scan the whole tree.
Substituting the base tip instead produces a range the scripts fail on
quietly: verified against unrelated histories, `lint_urls.sh` and
`lint_xrefs.sh` both exit **0** having checked nothing, while
`lint_file_size.sh` exits 128 and the wrapper reports "some files exceed
the 1 MB limit", which is not what happened. Unset args are loud instead
— verified rc=1 on a tree containing a dead link.
**`--no-color`.** Both `git diff` calls now pass it, matching the `git
grep --no-color` two lines below. With `color.ui = always` in a
developer's config, added lines arrive wrapped in escape sequences,
`grep -E '^\+'` matches nothing, and the check passes having found
nothing (measured: 2 matches → 0).
### Test plan
`.ci/scripts/tests/test_link_check_diff_selection.py` builds a diverged
history where `main` repairs bad links and shrinks an oversized file
while the feature branch simply predates all of it. Four fixture files
each pin a different part, and `curl` is stubbed so there is no network:
| Fixture | Pins |
|---|---|
| `both_sides.md` — edited on both branches | the per-file diff range |
| `big.bin` — oversized at the branch point, shrunk on main |
`lint_file_size.sh`'s range |
| `colorful.gitconfig` — `color.ui = always` | `--no-color` |
| `base_only.md` / `feature_only.md` | that main-only changes stay
invisible and the branch's own additions are still checked |
```
pytest .ci/scripts/tests/test_link_check_diff_selection.py # 4 passed
```
Mutating each changed line individually:
```
inner per-file range -> .. 3 failed caught
lint_file_size range -> .. 1 failed caught
drop --no-color 1 failed caught
file-selection range -> .. 4 passed equivalent mutant, see below
```
The file-selection range is not pinned because it cannot be: with the
per-file diff at three dots, the extra files it selects produce empty
diffs. Replayed against #21707's real 199-file range, both variants emit
byte-identical output. It is changed for consistency, not behaviour.
**Narrowing the scope must not blunt the check**, so each linter also
has a positive control where the branch itself adds the bad thing:
```
lint_urls adds a dead URL -> rc=1, reports example.invalid/dead
lint_xrefs adds a broken reference -> rc=1, reports sub/missing.md
lint_file_size adds a 1MB+ file -> rc=1, reports feature_big.bin
```
End to end on real content: a synthetic commit on top of `main` that
puts #21694's nine repaired links back, as if a PR had added them, gives
`rc=1` with 9 FAIL and 5 OK against the live network. Incidentally
`musl.cc` answered this time where CI saw `000`, and
`pybind/cmake_example` now 404s where CI saw `301` — which is the
retry-then-WARN path earning its keep.
Also replayed #21729's and #21707's exact CI refs through the fixed
scripts (exit 0 each), and ran all three linters plus `lintrunner`
against this PR's own diff.
### Known limitations
- The checkout is `head.sha`, so references still resolve against the
branch tree rather than the merge result — the mirror image of the bug
fixed here. Pre-existing and not worsened by this PR.
- The whole-tree branch swallows a `git grep` failure and exits 0, so on
a git built without PCRE the scan silently checks nothing. Reachable
locally, not on the runners, which is why the nightly scan works.
Pre-existing; worth a follow-up rather than widening this PR.
### Rollout
This does **not** repair a currently red run. A rerun keeps the original
`GITHUB_SHA`, and advancing the base alone does not fire `synchronize`,
so an already-open PR picks the fix up only on its next newly triggered
pull-request run — any push, or close/reopen. That is still cheaper than
a content rebase: no history rewrite, no conflicts, nothing to
re-review.
Supersedes #21762, which fixes the same bug but leaves the checkout
shallow and misses `lint_urls.sh`. Its reviewer's shallow-checkout point
is exactly right: a `--depth=1` fetch writes a shallow graft even into
an otherwise complete clone, after which `git merge-base` fails and
`A...B` is fatal, so the two halves of this change are a pair. The
regression test here is adapted from that PR.
Authored with Claude Code (Claude Opus 5).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@psiddh