Skip to content

fix: stop passing a schemeless URL to the HTTP client (#90) - #158

Open
manantlerio wants to merge 1 commit into
everywall:mainfrom
manantlerio:fix/relative-url-without-referer
Open

manantlerio wants to merge 1 commit into
everywall:mainfrom
manantlerio:fix/relative-url-without-referer

Conversation

@manantlerio

@manantlerio manantlerio commented Aug 29, 2026

Copy link
Copy Markdown

Fixes #90.

What happens

extractUrl resolves a root-relative request path against the referer, because on a proxy that serves the target URL off its own path the referer is the only thing that says which site the path belongs to. When there is no referer, or the referer is not itself a proxied page, it returned the bare path with a nil error:

extractUrl("/cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1")
  -> "cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1", nil

ProxySite only logged the extraction error and carried on regardless, so that path went to http.Client and failed several layers later with the message in the issue:

ERROR: Get "cdn-cgi/challenge-platform/...?ray=...": unsupported protocol scheme ""

Cloudflare challenge pages hit this repeatedly: the challenge fetches root-relative URLs, and the redirect target reported in the thread (?__cf_chl_rt_tk=...) is root-relative too. Both reported URLs reproduce.

What this changes

  • extractUrl returns an error when it cannot build an absolute http(s) URL, instead of returning something unfetchable with err == nil.
  • ProxySite returns 400 with that message instead of ignoring the error and fetching anyway.
  • The referer path is trimmed of BASE_PATH before the proxied URL is read out of it. Without this, a ladder on a subpath cannot resolve a relative request at all: /mypath/https://site/x never parses as a URL, so it produced the same error.
  • An absolute URL with a scheme other than http/https, or with no host, is rejected by name rather than handed on to fail as a protocol scheme error.

What this does not change

It does not get a Cloudflare-protected page to load. That is what FLARESOLVERR_HOST is for, and the "Just a moment" part of the issue is bot protection, not URL handling. This replaces an error that points at the wrong thing with one that says what actually happened, and stops the proxy issuing a request it has no target for.

handlers/raw.go and handlers/api.go pass c.Params("*") straight to fetchSite and can produce the same message. They do no referer resolution at all, so giving them one is a behaviour change rather than a fix, and I left them out of this PR.

Tests

handlers/extracturl_test.go covers both request shapes from the issue, the working referer case, the non-proxied referer, BASE_PATH, and the scheme/host checks.

Four of the six fail against unpatched main. The other two (TestExtractUrlAbsolute, TestExtractUrlRelativeWithProxiedReferer) pass before and after on purpose: they pin the behaviour that already works so this change does not quietly break it.

The ProxySite case reproduces the reported log line verbatim on unpatched main:

ERROR: Get "cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1?ray=92fef3e48ae3f7e5": unsupported protocol scheme ""
--- FAIL: TestProxySiteRejectsUnresolvableRelativePath
        expected: 400
        actual  : 500

With the fix, gofmt, go vet ./..., go build ./... and go test ./... are clean on golang:1.26. (gofmt -l reports cmd/main.go both before and after; that is pre-existing on main and untouched here.)

The test file is named _test.go so it actually runs. #154 covers the existing *.test.go files that do not; this PR does not depend on it, and does not touch the same files as #153, #155 or #157.

A request for a root-relative path, eg /cdn-cgi/challenge-platform/... , is
resolved against the referer, because the referer is the proxied page and
therefore the only thing that says which site the path belongs to. When there
is no referer, or the referer is not a proxied page, extractUrl still returned
the bare path with a nil error:

    extractUrl("/cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1")
      -> "cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1", nil

ProxySite only logged the extraction error and carried on regardless, so that
path reached http.Client, which failed on it several layers later with

    Get "cdn-cgi/challenge-platform/...?ray=...": unsupported protocol scheme ""

That is the error in everywall#90, and it names neither the page being proxied nor the
real problem. Cloudflare challenge pages hit it repeatedly because the
challenge fetches root-relative URLs and the reported redirect target
(?__cf_chl_rt_tk=...) is root-relative too.

extractUrl now fails when it cannot build an absolute http(s) URL, and
ProxySite returns 400 with that message instead of fetching something it
cannot fetch. Two related gaps in the same function are closed:

  - the referer path is trimmed of BASE_PATH before the proxied URL is read
    out of it, so a ladder running on a subpath can resolve relative requests
    at all
  - an absolute URL with a scheme other than http/https, or with no host, is
    rejected by name rather than handed on to fail as a protocol scheme error

This does not get a Cloudflare-protected page to load; that is what
FLARESOLVERR_HOST is for. It replaces an error that points at the wrong thing
with one that says what happened.

handlers/extracturl_test.go covers the two request shapes from the issue plus
the referer, BASE_PATH and scheme cases. Against unpatched main the new tests
fail, and the ProxySite case reproduces the reported log line verbatim.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@manantlerio
manantlerio force-pushed the fix/relative-url-without-referer branch from 3a27625 to c622792 Compare August 29, 2026 05:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Error: Unsupported protocol scheme when specific website is put in

1 participant