fix: stop passing a schemeless URL to the HTTP client (#90) - #158
Open
manantlerio wants to merge 1 commit into
Open
manantlerio wants to merge 1 commit into
manantlerio wants to merge 1 commit into
Conversation
A request for a root-relative path, eg /cdn-cgi/challenge-platform/... , is
resolved against the referer, because the referer is the proxied page and
therefore the only thing that says which site the path belongs to. When there
is no referer, or the referer is not a proxied page, extractUrl still returned
the bare path with a nil error:
extractUrl("/cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1")
-> "cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1", nil
ProxySite only logged the extraction error and carried on regardless, so that
path reached http.Client, which failed on it several layers later with
Get "cdn-cgi/challenge-platform/...?ray=...": unsupported protocol scheme ""
That is the error in everywall#90, and it names neither the page being proxied nor the
real problem. Cloudflare challenge pages hit it repeatedly because the
challenge fetches root-relative URLs and the reported redirect target
(?__cf_chl_rt_tk=...) is root-relative too.
extractUrl now fails when it cannot build an absolute http(s) URL, and
ProxySite returns 400 with that message instead of fetching something it
cannot fetch. Two related gaps in the same function are closed:
- the referer path is trimmed of BASE_PATH before the proxied URL is read
out of it, so a ladder running on a subpath can resolve relative requests
at all
- an absolute URL with a scheme other than http/https, or with no host, is
rejected by name rather than handed on to fail as a protocol scheme error
This does not get a Cloudflare-protected page to load; that is what
FLARESOLVERR_HOST is for. It replaces an error that points at the wrong thing
with one that says what happened.
handlers/extracturl_test.go covers the two request shapes from the issue plus
the referer, BASE_PATH and scheme cases. Against unpatched main the new tests
fail, and the ProxySite case reproduces the reported log line verbatim.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
manantlerio
force-pushed
the
fix/relative-url-without-referer
branch
from
August 29, 2026 05:42
3a27625 to
c622792
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #90.
What happens
extractUrlresolves a root-relative request path against the referer, because on a proxy that serves the target URL off its own path the referer is the only thing that says which site the path belongs to. When there is no referer, or the referer is not itself a proxied page, it returned the bare path with a nil error:ProxySiteonly logged the extraction error and carried on regardless, so that path went tohttp.Clientand failed several layers later with the message in the issue:Cloudflare challenge pages hit this repeatedly: the challenge fetches root-relative URLs, and the redirect target reported in the thread (
?__cf_chl_rt_tk=...) is root-relative too. Both reported URLs reproduce.What this changes
extractUrlreturns an error when it cannot build an absolute http(s) URL, instead of returning something unfetchable witherr == nil.ProxySitereturns 400 with that message instead of ignoring the error and fetching anyway.BASE_PATHbefore the proxied URL is read out of it. Without this, a ladder on a subpath cannot resolve a relative request at all:/mypath/https://site/xnever parses as a URL, so it produced the same error.What this does not change
It does not get a Cloudflare-protected page to load. That is what
FLARESOLVERR_HOSTis for, and the "Just a moment" part of the issue is bot protection, not URL handling. This replaces an error that points at the wrong thing with one that says what actually happened, and stops the proxy issuing a request it has no target for.handlers/raw.goandhandlers/api.gopassc.Params("*")straight tofetchSiteand can produce the same message. They do no referer resolution at all, so giving them one is a behaviour change rather than a fix, and I left them out of this PR.Tests
handlers/extracturl_test.gocovers both request shapes from the issue, the working referer case, the non-proxied referer,BASE_PATH, and the scheme/host checks.Four of the six fail against unpatched
main. The other two (TestExtractUrlAbsolute,TestExtractUrlRelativeWithProxiedReferer) pass before and after on purpose: they pin the behaviour that already works so this change does not quietly break it.The
ProxySitecase reproduces the reported log line verbatim on unpatchedmain:With the fix,
gofmt,go vet ./...,go build ./...andgo test ./...are clean on golang:1.26. (gofmt -lreportscmd/main.goboth before and after; that is pre-existing onmainand untouched here.)The test file is named
_test.goso it actually runs. #154 covers the existing*.test.gofiles that do not; this PR does not depend on it, and does not touch the same files as #153, #155 or #157.