[BUG REPORT] - "Too many requests"

[joe024132]

Description

When downloading any thread, most of the files will be downloaded as raw text files containing the text: "Too Many Requests"

Steps to reproduce

I used https://boards.4chan.org/wsg/thread/5731328 as a test.

Expected behavior

All of the images should correctly download.

Actual behavior

Only some images correctly download, and the rest are just raw text files.

OS version

Windows 11

Confirmation

• I performed a search of the issue tracker to avoid opening a duplicate issue
• I understand that not filling out this template correctly may lead to the issue being closed

[github-actions]

Thank you for your first issue. To better understand your request or the problem you've encountered, please provide as many details as possible. If the behavior changes or if you have new information about your request, don't hesitate to add it. It will be reviewed ASAP.

[SegoCode]

Hi there,

Well that happens because you have reached a certain limit from your IP, you can connect to a VPN and continue downloading from there or wait a while. Some solutions would be to implement some “interval” between downloads or to be able to define proxies. Any other ideas?

[M0ller]

I got this to and added a interval that paused for a few seconds and then tried again for a couple of times. It works better. But also adding a millisecond wait before each go routine adds a bit more help as well.

Also did a check to only download the file if it is not already downloaded and if the request succeeds.

Could add this as a param to the command if it should apply the wait's.

[SegoCode]

I got this to and added a interval that paused for a few seconds and then tried again for a couple of times. It works better. But also adding a millisecond wait before each go routine adds a bit more help as well.

Also did a check to only download the file if it is not already downloaded and if the request succeeds.

Could add this as a param to the command if it should apply the wait's.

Yeah, definitely dont download the text “too many request” has to be implemented. And some proxy and interval parameters...

The problem with adding so many flags is that it becomes unmaintainable, and you would have to use the flag package, the problem arises because of how Go's flag package parses command-line arguments. By default, the flag package stops parsing flags when it encounters the first non-flag argument (in my case, the URL) then, we would always have to define --url, and I think it is already breaking little by little the simplicity...

[M0ller]

Yeah true. Maybe a config json file that have some default values for wait timer etc and then later a proxy config or such could be added in it if / when it is developed. That maybe is the way to go?

[SegoCode]

I made this snipped, in theory, if the flags are at the beginning they can be parsed until they reach the string URL

	// Manually parse flags and positional arguments
	var args []string
	for i := 1; i < len(os.Args); i++ {
		arg := os.Args[i]
		if arg == "--" {
			// All remaining args are positional
			args = append(args, os.Args[i+1:]...)
			break
		}
		if strings.HasPrefix(arg, "-") {
			// Flag
			fs.Parse(os.Args[i:])
			break
		}
		// Positional argument
		args = append(args, arg)
	}

	// After parsing flags, any remaining arguments are positional
	args = append(args, fs.Args()...)

I made a branch https://github.com/SegoCode/4cget/tree/develop-flags to test and keep coding this implementation. 888dc23

[SegoCode]

Restarting development on this 🎉

4cget now reads the status first and skips 403/429 instead of calling io.Copy. On those statuses it prints a Cloudflare hint: low IP trust score, browser challenge required. 429 also mentions --sleep.

Cloudflare scores each GET on boards.4chan.org from IP, ASN, TLS fingerprint, and User-Agent. Normal browser traffic from the same IP raises that score. Open the thread in a browser, browse a bit, then retry the CLI.

--sleep N serializes downloads with N seconds between files. That is the interval we talked about here. --monitor already waits between thread polls, so the two flags cannot run together.

I dropped proxy support. 429 on i.4cdn.org is a rate limit on that IP. Spacing requests is the control we have without another hop.

If 429 shows up again, raise --sleep and retry.