Skip to content

gh-79638: Treat an unreachable robots.txt as "disallow all" - #138555

Merged
serhiy-storchaka merged 5 commits into
python:mainfrom
serhiy-storchaka:robotparser-unreachable
May 7, 2026
Merged

gh-79638: Treat an unreachable robots.txt as "disallow all"#138555
serhiy-storchaka merged 5 commits into
python:mainfrom
serhiy-storchaka:robotparser-unreachable

Conversation

@serhiy-storchaka

@serhiy-storchakaserhiy-storchaka commented Sep 5, 2025•
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Member

Disallow all access in urllib.robotparser if the robots.txt file is unreachable due to server or network errors.

@serhiy-storchakaserhiy-storchaka added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Sep 5, 2025
@bedevere-appbedevere-appBot mentioned this pull request Sep 5, 2025
Closed
@serhiy-storchaka
pythongh-79638: Treat an unreachable robots.txt as "disallow all"
317c116
Disallow all access in urllib.robotparser if the robots.txt file is
unreachable due to server or network errors.
@github-actions

github-actionsBot commented Apr 29, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

@github-actionsgithub-actionsBot added the stale Stale PR or inactive for long period of time. label Apr 29, 2026
@serhiy-storchaka
serhiy-storchaka enabled auto-merge (squash) May 4, 2026 19:45
@serhiy-storchaka
serhiy-storchaka enabled auto-merge (squash) May 5, 2026 09:09
@serhiy-storchaka
serhiy-storchaka merged commit 310fe88 into python:mainMay 7, 2026
49 of 52 checks passed
@miss-islington-app

miss-islington-appBot commented May 7, 2026

Copy link
Copy Markdown

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

@bedevere-app

bedevere-appBot commented May 7, 2026

Copy link
Copy Markdown

GH-149512 is a backport of this pull request to the 3.14 branch.

@bedevere-appbedevere-appBot removed the needs backport to 3.14 bugs and security fixes label May 7, 2026
@bedevere-app

bedevere-appBot commented May 7, 2026

Copy link
Copy Markdown

GH-149513 is a backport of this pull request to the 3.13 branch.

@bedevere-appbedevere-appBot removed the needs backport to 3.13 bugs and security fixes label May 7, 2026
hugovk added a commit to hugovk/cpython that referenced this pull request May 7, 2026
@hugovk
Revert "pythongh-79638: Treat an unreachable robots.txt as "disallow …
fdf75b3
…all" (pythonGH-138555)"
This reverts commit 310fe88.
@hugovk

hugovk commented May 7, 2026

Copy link
Copy Markdown
Member

This had been set to automerge but was failing CI, so wasn't meant to merge right now.

I'd temporarily removed branch protection to be able to merge #149500 which was needed to fix configure, which meant this PR also merged.

I've reverted it in #149514 to get main back to green. Sorry about this.

@serhiy-storchaka Would you like me to re-open a fresh version of this PR once that is sorted?

@serhiy-storchaka

serhiy-storchaka commented May 7, 2026

Copy link
Copy Markdown
MemberAuthor

I'll recreate it tomorrow.

@serhiy-storchaka
serhiy-storchaka deleted the robotparser-unreachable branch May 8, 2026 08:24
@serhiy-storchaka

serhiy-storchaka commented May 8, 2026

Copy link
Copy Markdown
MemberAuthor

Restored as #149541.

@serhiy-storchakaserhiy-storchaka mentioned this pull request Jun 28, 2026
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@berkerpeksagberkerpeksagAwaiting requested review from berkerpeksag

Assignees

No one assigned

Labels

staleStale PR or inactive for long period of time.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@serhiy-storchaka@hugovk