fix(client): surface HTTP status, bound requests, and type API errors by karaposu · Pull Request #29 · brightdata/cli

karaposu · 2026-08-27T07:19:45Z

Three defects in src/utils/client.ts — the one function every API call in the CLI goes through. They're bundled because they're the same root cause (the boundary hides what it knows) and touch the same 30 lines.

1. The client discards the HTTP status, so polling guesses

On success the client returned only the parsed body. 202 is res.ok, so a caller could not distinguish "job accepted, still building" from "here is your data" — both arrived as a body with no status attached.

Snapshot polling therefore infers readiness from the body's shape (extract_status + RUNNING_STATUSES). That works while the body is a parsed object, i.e. --format json. It silently fails for --format csv|ndjson|jsonl, where the client returns text:

constextract_status=(result: unknown)=>{if(!result||typeofresult!='object')// a string body exits herereturnundefined;// → "not running" → treated as data

So a not-ready snapshot requested as JSONL gets printed as if it were the dataset, and the CLI exits 0. Silent wrong data is the worst outcome for exactly the ETL workflows those formats exist to serve.

Fix: an opt-in get_with_status() returning {status, headers, body}. request() / get() / post() still return the body, so no existing call site changes — only code that must reason about the protocol opts in. Readiness now checks, in order of authority: HTTP 202 → parsed-object status → text body that parses to a status. The predicate returns the real status string, so progress output still shows starting → building → running instead of collapsing to one token.

Headers are exposed too — that's the plumbing a future Retry-After-aware backoff needs.

2. No request timeout — the CLI could hang forever

fetch() has no default timeout and was called without a signal. A black-holed connection (VPN drop, hung LB) produced no error at all, so the retry loop never engaged: spinner spinning, no output, Ctrl-C the only exit.

Fix:AbortSignal.timeout per attempt (fresh signal each time — hoisting it would abort retries instantly), default 120s, overridable via Request_opts.timeout_ms.

Two deliberate choices worth reviewing:

120s, not 30s. The abort must bound hung connections without cutting off slow-but-alive work — protected-site scrapes and large snapshot bodies legitimately run long. Happy to lower it if you have real numbers on the long tail.
A timed-out attempt retries once, not 3×. Reusing the generic retry budget would mean 4 × 120s ≈ 8 minutes of silent stall — multiplying the very wait this fix exists to bound. The retry is also narrated rather than silent.

3. Retry was decided by error message prose

if(einstanceofError&&e.message.startsWith('Error:'))throwe;// treat as final API error, don't retry

Retry semantics were coupled to message wording: rewording a template silently changes behavior, and a network failure whose message happens to start with Error: was misclassified as final and never retried.

Fix: a Client_api_error class (carrying status and hint) discriminated with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (REALTIME_LIMIT_MARKER = 'realtime job limit', clean_error_message), so that contract is preserved and now has a test pinning it.

Testing

22 new tests, 395 total. The request() loop had no test coverage at all before this — these are the first.
The readiness predicate is table-tested across every combination of {200, 202} × {object body, text body}, because that matrix is precisely where the silent-corruption case lives.
vi.mocks utils/config so URL assertions don't depend on the developer's local config.json (load_config() runs on every request and can override api_url).
Verified by running the built CLI.

Scope

Deliberately not included: retry policy/backoff changes (that's #20's area), migrating other call sites (status.ts is correctly excluded — for /datasets/v3/progress the status metadata is the payload), and retiring scraper.ts's parallel fetch_raw helper. That helper exists only because the shared client couldn't expose status codes; get_with_status removes its reason to exist, but that's a follow-up, not this diff.

Independent of #28 (Node 20 fix) — different files, no conflicts, either can merge first.

Note: this PR reports no CI checks because main has no CI workflow yet (#28 adds one — type-check + tests on a Node 20.17.0 / 24 matrix). Verified locally instead: tsc --noEmit clean, 391 tests passing, and the built CLI smoke-run.

Three defects in src/utils/client.ts, the function every API call goes through: 1. Protocol erasure. On success the client returned only the parsed body and discarded the status code, so a caller could not tell HTTP 202 (job accepted, still building) from 200 (this is your data) — 202 is res.ok, so both looked identical. Snapshot polling therefore inferred readiness from the body's *shape*. That misses the not-ready case whenever the body is text, which is exactly what --format csv/ndjson/ jsonl produce: the status stub is printed as if it were the dataset and the CLI exits 0. Silent wrong data is the worst failure mode for the ETL use case these formats exist for. 2. No request timeout. fetch() has no default timeout and was called without a signal, so a black-holed connection (VPN drop, hung load balancer) hung forever — no error ever arrived for the retry loop to react to. The only way out was Ctrl-C. 3. Retry decided by message prose. The catch block told API errors from network errors with message.startsWith('Error:'), so rewording an error template silently changed retry behavior, and a network failure worded that way was misclassified as final and never retried. Changes: - Add Response_envelope {status, headers, body} and an opt-in get_with_status(). request()/get()/post() keep returning the body, so all existing call sites are untouched; only pollers that must reason about the protocol opt in. Headers are exposed too, which is what a future Retry-After-aware backoff would need. - Add Client_api_error (carrying status and hint) and discriminate with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (e.g. 'realtime job limit'), so that contract is preserved and now covered by a test. - Abort each attempt with AbortSignal.timeout (default 120s, override via Request_opts.timeout_ms). The default is deliberately generous so it cannot cut off slow-but-alive work such as protected-site scrapes or large snapshot downloads. A timed-out attempt retries at most once rather than reusing the 3-retry budget, because timeout x attempts multiplies the stall the fix exists to bound, and the retry is narrated instead of being silent. - Migrate the pipelines snapshot poller to decide readiness from the protocol first, keeping body-shape checks as fallback and adding the text-body case that the object-only check missed. The predicate returns the real status string, so progress output still distinguishes starting/building/running. Tests: 22 new (395 total). The client request loop had no coverage at all before this; the readiness predicate is table-tested across every combination of {200, 202} x {object body, text body} because that matrix is where the silent-corruption case lives.

karaposu · 2026-08-27T07:19:45Z

Three defects in src/utils/client.ts — the one function every API call in the CLI goes through. They're bundled because they're the same root cause (the boundary hides what it knows) and touch the same 30 lines.

1. The client discards the HTTP status, so polling guesses

On success the client returned only the parsed body. 202 is res.ok, so a caller could not distinguish "job accepted, still building" from "here is your data" — both arrived as a body with no status attached.

Snapshot polling therefore infers readiness from the body's shape (extract_status + RUNNING_STATUSES). That works while the body is a parsed object, i.e. --format json. It silently fails for --format csv|ndjson|jsonl, where the client returns text:

constextract_status=(result: unknown)=>{if(!result||typeofresult!='object')// a string body exits herereturnundefined;// → "not running" → treated as data

So a not-ready snapshot requested as JSONL gets printed as if it were the dataset, and the CLI exits 0. Silent wrong data is the worst outcome for exactly the ETL workflows those formats exist to serve.

Fix: an opt-in get_with_status() returning {status, headers, body}. request() / get() / post() still return the body, so no existing call site changes — only code that must reason about the protocol opts in. Readiness now checks, in order of authority: HTTP 202 → parsed-object status → text body that parses to a status. The predicate returns the real status string, so progress output still shows starting → building → running instead of collapsing to one token.

Headers are exposed too — that's the plumbing a future Retry-After-aware backoff needs.

2. No request timeout — the CLI could hang forever

fetch() has no default timeout and was called without a signal. A black-holed connection (VPN drop, hung LB) produced no error at all, so the retry loop never engaged: spinner spinning, no output, Ctrl-C the only exit.

Fix:AbortSignal.timeout per attempt (fresh signal each time — hoisting it would abort retries instantly), default 120s, overridable via Request_opts.timeout_ms.

Two deliberate choices worth reviewing:

120s, not 30s. The abort must bound hung connections without cutting off slow-but-alive work — protected-site scrapes and large snapshot bodies legitimately run long. Happy to lower it if you have real numbers on the long tail.
A timed-out attempt retries once, not 3×. Reusing the generic retry budget would mean 4 × 120s ≈ 8 minutes of silent stall — multiplying the very wait this fix exists to bound. The retry is also narrated rather than silent.

3. Retry was decided by error message prose

if(einstanceofError&&e.message.startsWith('Error:'))throwe;// treat as final API error, don't retry

Retry semantics were coupled to message wording: rewording a template silently changes behavior, and a network failure whose message happens to start with Error: was misclassified as final and never retried.

Fix: a Client_api_error class (carrying status and hint) discriminated with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (REALTIME_LIMIT_MARKER = 'realtime job limit', clean_error_message), so that contract is preserved and now has a test pinning it.

Testing

22 new tests, 395 total. The request() loop had no test coverage at all before this — these are the first.
The readiness predicate is table-tested across every combination of {200, 202} × {object body, text body}, because that matrix is precisely where the silent-corruption case lives.
vi.mocks utils/config so URL assertions don't depend on the developer's local config.json (load_config() runs on every request and can override api_url).
Verified by running the built CLI.

Scope

Deliberately not included: retry policy/backoff changes (that's #20's area), migrating other call sites (status.ts is correctly excluded — for /datasets/v3/progress the status metadata is the payload), and retiring scraper.ts's parallel fetch_raw helper. That helper exists only because the shared client couldn't expose status codes; get_with_status removes its reason to exist, but that's a follow-up, not this diff.

Independent of #28 (Node 20 fix) — different files, no conflicts, either can merge first.

Note: this PR reports no CI checks because main has no CI workflow yet (#28 adds one — type-check + tests on a Node 20.17.0 / 24 matrix). Verified locally instead: tsc --noEmit clean, 391 tests passing, and the built CLI smoke-run.

Three defects in src/utils/client.ts, the function every API call goes through: 1. Protocol erasure. On success the client returned only the parsed body and discarded the status code, so a caller could not tell HTTP 202 (job accepted, still building) from 200 (this is your data) — 202 is res.ok, so both looked identical. Snapshot polling therefore inferred readiness from the body's *shape*. That misses the not-ready case whenever the body is text, which is exactly what --format csv/ndjson/ jsonl produce: the status stub is printed as if it were the dataset and the CLI exits 0. Silent wrong data is the worst failure mode for the ETL use case these formats exist for. 2. No request timeout. fetch() has no default timeout and was called without a signal, so a black-holed connection (VPN drop, hung load balancer) hung forever — no error ever arrived for the retry loop to react to. The only way out was Ctrl-C. 3. Retry decided by message prose. The catch block told API errors from network errors with message.startsWith('Error:'), so rewording an error template silently changed retry behavior, and a network failure worded that way was misclassified as final and never retried. Changes: - Add Response_envelope {status, headers, body} and an opt-in get_with_status(). request()/get()/post() keep returning the body, so all existing call sites are untouched; only pollers that must reason about the protocol opt in. Headers are exposed too, which is what a future Retry-After-aware backoff would need. - Add Client_api_error (carrying status and hint) and discriminate with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (e.g. 'realtime job limit'), so that contract is preserved and now covered by a test. - Abort each attempt with AbortSignal.timeout (default 120s, override via Request_opts.timeout_ms). The default is deliberately generous so it cannot cut off slow-but-alive work such as protected-site scrapes or large snapshot downloads. A timed-out attempt retries at most once rather than reusing the 3-retry budget, because timeout x attempts multiplies the stall the fix exists to bound, and the retry is narrated instead of being silent. - Migrate the pipelines snapshot poller to decide readiness from the protocol first, keeping body-shape checks as fallback and adding the text-body case that the object-only check missed. The predicate returns the real status string, so progress output still distinguishes starting/building/running. Tests: 22 new (395 total). The client request loop had no coverage at all before this; the readiness predicate is table-tested across every combination of {200, 202} x {object body, text body} because that matrix is where the silent-corruption case lives.

karaposu · 2026-08-27T07:19:45Z

Three defects in src/utils/client.ts — the one function every API call in the CLI goes through. They're bundled because they're the same root cause (the boundary hides what it knows) and touch the same 30 lines.

1. The client discards the HTTP status, so polling guesses

On success the client returned only the parsed body. 202 is res.ok, so a caller could not distinguish "job accepted, still building" from "here is your data" — both arrived as a body with no status attached.

Snapshot polling therefore infers readiness from the body's shape (extract_status + RUNNING_STATUSES). That works while the body is a parsed object, i.e. --format json. It silently fails for --format csv|ndjson|jsonl, where the client returns text:

constextract_status=(result: unknown)=>{if(!result||typeofresult!='object')// a string body exits herereturnundefined;// → "not running" → treated as data

So a not-ready snapshot requested as JSONL gets printed as if it were the dataset, and the CLI exits 0. Silent wrong data is the worst outcome for exactly the ETL workflows those formats exist to serve.

Fix: an opt-in get_with_status() returning {status, headers, body}. request() / get() / post() still return the body, so no existing call site changes — only code that must reason about the protocol opts in. Readiness now checks, in order of authority: HTTP 202 → parsed-object status → text body that parses to a status. The predicate returns the real status string, so progress output still shows starting → building → running instead of collapsing to one token.

Headers are exposed too — that's the plumbing a future Retry-After-aware backoff needs.

2. No request timeout — the CLI could hang forever

fetch() has no default timeout and was called without a signal. A black-holed connection (VPN drop, hung LB) produced no error at all, so the retry loop never engaged: spinner spinning, no output, Ctrl-C the only exit.

Fix:AbortSignal.timeout per attempt (fresh signal each time — hoisting it would abort retries instantly), default 120s, overridable via Request_opts.timeout_ms.

Two deliberate choices worth reviewing:

120s, not 30s. The abort must bound hung connections without cutting off slow-but-alive work — protected-site scrapes and large snapshot bodies legitimately run long. Happy to lower it if you have real numbers on the long tail.
A timed-out attempt retries once, not 3×. Reusing the generic retry budget would mean 4 × 120s ≈ 8 minutes of silent stall — multiplying the very wait this fix exists to bound. The retry is also narrated rather than silent.

3. Retry was decided by error message prose

if(einstanceofError&&e.message.startsWith('Error:'))throwe;// treat as final API error, don't retry

Retry semantics were coupled to message wording: rewording a template silently changes behavior, and a network failure whose message happens to start with Error: was misclassified as final and never retried.

Fix: a Client_api_error class (carrying status and hint) discriminated with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (REALTIME_LIMIT_MARKER = 'realtime job limit', clean_error_message), so that contract is preserved and now has a test pinning it.

Testing

22 new tests, 395 total. The request() loop had no test coverage at all before this — these are the first.
The readiness predicate is table-tested across every combination of {200, 202} × {object body, text body}, because that matrix is precisely where the silent-corruption case lives.
vi.mocks utils/config so URL assertions don't depend on the developer's local config.json (load_config() runs on every request and can override api_url).
Verified by running the built CLI.

Scope

Deliberately not included: retry policy/backoff changes (that's #20's area), migrating other call sites (status.ts is correctly excluded — for /datasets/v3/progress the status metadata is the payload), and retiring scraper.ts's parallel fetch_raw helper. That helper exists only because the shared client couldn't expose status codes; get_with_status removes its reason to exist, but that's a follow-up, not this diff.

Independent of #28 (Node 20 fix) — different files, no conflicts, either can merge first.

Note: this PR reports no CI checks because main has no CI workflow yet (#28 adds one — type-check + tests on a Node 20.17.0 / 24 matrix). Verified locally instead: tsc --noEmit clean, 391 tests passing, and the built CLI smoke-run.

Three defects in src/utils/client.ts, the function every API call goes through: 1. Protocol erasure. On success the client returned only the parsed body and discarded the status code, so a caller could not tell HTTP 202 (job accepted, still building) from 200 (this is your data) — 202 is res.ok, so both looked identical. Snapshot polling therefore inferred readiness from the body's *shape*. That misses the not-ready case whenever the body is text, which is exactly what --format csv/ndjson/ jsonl produce: the status stub is printed as if it were the dataset and the CLI exits 0. Silent wrong data is the worst failure mode for the ETL use case these formats exist for. 2. No request timeout. fetch() has no default timeout and was called without a signal, so a black-holed connection (VPN drop, hung load balancer) hung forever — no error ever arrived for the retry loop to react to. The only way out was Ctrl-C. 3. Retry decided by message prose. The catch block told API errors from network errors with message.startsWith('Error:'), so rewording an error template silently changed retry behavior, and a network failure worded that way was misclassified as final and never retried. Changes: - Add Response_envelope {status, headers, body} and an opt-in get_with_status(). request()/get()/post() keep returning the body, so all existing call sites are untouched; only pollers that must reason about the protocol opt in. Headers are exposed too, which is what a future Retry-After-aware backoff would need. - Add Client_api_error (carrying status and hint) and discriminate with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (e.g. 'realtime job limit'), so that contract is preserved and now covered by a test. - Abort each attempt with AbortSignal.timeout (default 120s, override via Request_opts.timeout_ms). The default is deliberately generous so it cannot cut off slow-but-alive work such as protected-site scrapes or large snapshot downloads. A timed-out attempt retries at most once rather than reusing the 3-retry budget, because timeout x attempts multiplies the stall the fix exists to bound, and the retry is narrated instead of being silent. - Migrate the pipelines snapshot poller to decide readiness from the protocol first, keeping body-shape checks as fallback and adding the text-body case that the object-only check missed. The predicate returns the real status string, so progress output still distinguishes starting/building/running. Tests: 22 new (395 total). The client request loop had no coverage at all before this; the readiness predicate is table-tested across every combination of {200, 202} x {object body, text body} because that matrix is where the silent-corruption case lives.

karaposu · 2026-08-27T07:19:45Z

Three defects in src/utils/client.ts — the one function every API call in the CLI goes through. They're bundled because they're the same root cause (the boundary hides what it knows) and touch the same 30 lines.

1. The client discards the HTTP status, so polling guesses

On success the client returned only the parsed body. 202 is res.ok, so a caller could not distinguish "job accepted, still building" from "here is your data" — both arrived as a body with no status attached.

Snapshot polling therefore infers readiness from the body's shape (extract_status + RUNNING_STATUSES). That works while the body is a parsed object, i.e. --format json. It silently fails for --format csv|ndjson|jsonl, where the client returns text:

constextract_status=(result: unknown)=>{if(!result||typeofresult!='object')// a string body exits herereturnundefined;// → "not running" → treated as data

So a not-ready snapshot requested as JSONL gets printed as if it were the dataset, and the CLI exits 0. Silent wrong data is the worst outcome for exactly the ETL workflows those formats exist to serve.

Fix: an opt-in get_with_status() returning {status, headers, body}. request() / get() / post() still return the body, so no existing call site changes — only code that must reason about the protocol opts in. Readiness now checks, in order of authority: HTTP 202 → parsed-object status → text body that parses to a status. The predicate returns the real status string, so progress output still shows starting → building → running instead of collapsing to one token.

Headers are exposed too — that's the plumbing a future Retry-After-aware backoff needs.

2. No request timeout — the CLI could hang forever

fetch() has no default timeout and was called without a signal. A black-holed connection (VPN drop, hung LB) produced no error at all, so the retry loop never engaged: spinner spinning, no output, Ctrl-C the only exit.

Fix:AbortSignal.timeout per attempt (fresh signal each time — hoisting it would abort retries instantly), default 120s, overridable via Request_opts.timeout_ms.

Two deliberate choices worth reviewing:

120s, not 30s. The abort must bound hung connections without cutting off slow-but-alive work — protected-site scrapes and large snapshot bodies legitimately run long. Happy to lower it if you have real numbers on the long tail.
A timed-out attempt retries once, not 3×. Reusing the generic retry budget would mean 4 × 120s ≈ 8 minutes of silent stall — multiplying the very wait this fix exists to bound. The retry is also narrated rather than silent.

3. Retry was decided by error message prose

if(einstanceofError&&e.message.startsWith('Error:'))throwe;// treat as final API error, don't retry

Retry semantics were coupled to message wording: rewording a template silently changes behavior, and a network failure whose message happens to start with Error: was misclassified as final and never retried.

Fix: a Client_api_error class (carrying status and hint) discriminated with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (REALTIME_LIMIT_MARKER = 'realtime job limit', clean_error_message), so that contract is preserved and now has a test pinning it.

Testing

22 new tests, 395 total. The request() loop had no test coverage at all before this — these are the first.
The readiness predicate is table-tested across every combination of {200, 202} × {object body, text body}, because that matrix is precisely where the silent-corruption case lives.
vi.mocks utils/config so URL assertions don't depend on the developer's local config.json (load_config() runs on every request and can override api_url).
Verified by running the built CLI.

Scope

Deliberately not included: retry policy/backoff changes (that's #20's area), migrating other call sites (status.ts is correctly excluded — for /datasets/v3/progress the status metadata is the payload), and retiring scraper.ts's parallel fetch_raw helper. That helper exists only because the shared client couldn't expose status codes; get_with_status removes its reason to exist, but that's a follow-up, not this diff.

Independent of #28 (Node 20 fix) — different files, no conflicts, either can merge first.

Note: this PR reports no CI checks because main has no CI workflow yet (#28 adds one — type-check + tests on a Node 20.17.0 / 24 matrix). Verified locally instead: tsc --noEmit clean, 391 tests passing, and the built CLI smoke-run.

Three defects in src/utils/client.ts, the function every API call goes through: 1. Protocol erasure. On success the client returned only the parsed body and discarded the status code, so a caller could not tell HTTP 202 (job accepted, still building) from 200 (this is your data) — 202 is res.ok, so both looked identical. Snapshot polling therefore inferred readiness from the body's *shape*. That misses the not-ready case whenever the body is text, which is exactly what --format csv/ndjson/ jsonl produce: the status stub is printed as if it were the dataset and the CLI exits 0. Silent wrong data is the worst failure mode for the ETL use case these formats exist for. 2. No request timeout. fetch() has no default timeout and was called without a signal, so a black-holed connection (VPN drop, hung load balancer) hung forever — no error ever arrived for the retry loop to react to. The only way out was Ctrl-C. 3. Retry decided by message prose. The catch block told API errors from network errors with message.startsWith('Error:'), so rewording an error template silently changed retry behavior, and a network failure worded that way was misclassified as final and never retried. Changes: - Add Response_envelope {status, headers, body} and an opt-in get_with_status(). request()/get()/post() keep returning the body, so all existing call sites are untouched; only pollers that must reason about the protocol opt in. Headers are exposed too, which is what a future Retry-After-aware backoff would need. - Add Client_api_error (carrying status and hint) and discriminate with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (e.g. 'realtime job limit'), so that contract is preserved and now covered by a test. - Abort each attempt with AbortSignal.timeout (default 120s, override via Request_opts.timeout_ms). The default is deliberately generous so it cannot cut off slow-but-alive work such as protected-site scrapes or large snapshot downloads. A timed-out attempt retries at most once rather than reusing the 3-retry budget, because timeout x attempts multiplies the stall the fix exists to bound, and the retry is narrated instead of being silent. - Migrate the pipelines snapshot poller to decide readiness from the protocol first, keeping body-shape checks as fallback and adding the text-body case that the object-only check missed. The predicate returns the real status string, so progress output still distinguishes starting/building/running. Tests: 22 new (395 total). The client request loop had no coverage at all before this; the readiness predicate is table-tested across every combination of {200, 202} x {object body, text body} because that matrix is where the silent-corruption case lives.

karaposu · 2026-08-27T07:19:45Z

Three defects in src/utils/client.ts — the one function every API call in the CLI goes through. They're bundled because they're the same root cause (the boundary hides what it knows) and touch the same 30 lines.

1. The client discards the HTTP status, so polling guesses

On success the client returned only the parsed body. 202 is res.ok, so a caller could not distinguish "job accepted, still building" from "here is your data" — both arrived as a body with no status attached.

Snapshot polling therefore infers readiness from the body's shape (extract_status + RUNNING_STATUSES). That works while the body is a parsed object, i.e. --format json. It silently fails for --format csv|ndjson|jsonl, where the client returns text:

constextract_status=(result: unknown)=>{if(!result||typeofresult!='object')// a string body exits herereturnundefined;// → "not running" → treated as data

So a not-ready snapshot requested as JSONL gets printed as if it were the dataset, and the CLI exits 0. Silent wrong data is the worst outcome for exactly the ETL workflows those formats exist to serve.

Fix: an opt-in get_with_status() returning {status, headers, body}. request() / get() / post() still return the body, so no existing call site changes — only code that must reason about the protocol opts in. Readiness now checks, in order of authority: HTTP 202 → parsed-object status → text body that parses to a status. The predicate returns the real status string, so progress output still shows starting → building → running instead of collapsing to one token.

Headers are exposed too — that's the plumbing a future Retry-After-aware backoff needs.

2. No request timeout — the CLI could hang forever

fetch() has no default timeout and was called without a signal. A black-holed connection (VPN drop, hung LB) produced no error at all, so the retry loop never engaged: spinner spinning, no output, Ctrl-C the only exit.

Fix:AbortSignal.timeout per attempt (fresh signal each time — hoisting it would abort retries instantly), default 120s, overridable via Request_opts.timeout_ms.

Two deliberate choices worth reviewing:

120s, not 30s. The abort must bound hung connections without cutting off slow-but-alive work — protected-site scrapes and large snapshot bodies legitimately run long. Happy to lower it if you have real numbers on the long tail.
A timed-out attempt retries once, not 3×. Reusing the generic retry budget would mean 4 × 120s ≈ 8 minutes of silent stall — multiplying the very wait this fix exists to bound. The retry is also narrated rather than silent.

3. Retry was decided by error message prose

if(einstanceofError&&e.message.startsWith('Error:'))throwe;// treat as final API error, don't retry

Retry semantics were coupled to message wording: rewording a template silently changes behavior, and a network failure whose message happens to start with Error: was misclassified as final and never retried.

Fix: a Client_api_error class (carrying status and hint) discriminated with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (REALTIME_LIMIT_MARKER = 'realtime job limit', clean_error_message), so that contract is preserved and now has a test pinning it.

Testing

22 new tests, 395 total. The request() loop had no test coverage at all before this — these are the first.
The readiness predicate is table-tested across every combination of {200, 202} × {object body, text body}, because that matrix is precisely where the silent-corruption case lives.
vi.mocks utils/config so URL assertions don't depend on the developer's local config.json (load_config() runs on every request and can override api_url).
Verified by running the built CLI.

Scope

Deliberately not included: retry policy/backoff changes (that's #20's area), migrating other call sites (status.ts is correctly excluded — for /datasets/v3/progress the status metadata is the payload), and retiring scraper.ts's parallel fetch_raw helper. That helper exists only because the shared client couldn't expose status codes; get_with_status removes its reason to exist, but that's a follow-up, not this diff.

Independent of #28 (Node 20 fix) — different files, no conflicts, either can merge first.

Note: this PR reports no CI checks because main has no CI workflow yet (#28 adds one — type-check + tests on a Node 20.17.0 / 24 matrix). Verified locally instead: tsc --noEmit clean, 391 tests passing, and the built CLI smoke-run.

Three defects in src/utils/client.ts, the function every API call goes through: 1. Protocol erasure. On success the client returned only the parsed body and discarded the status code, so a caller could not tell HTTP 202 (job accepted, still building) from 200 (this is your data) — 202 is res.ok, so both looked identical. Snapshot polling therefore inferred readiness from the body's *shape*. That misses the not-ready case whenever the body is text, which is exactly what --format csv/ndjson/ jsonl produce: the status stub is printed as if it were the dataset and the CLI exits 0. Silent wrong data is the worst failure mode for the ETL use case these formats exist for. 2. No request timeout. fetch() has no default timeout and was called without a signal, so a black-holed connection (VPN drop, hung load balancer) hung forever — no error ever arrived for the retry loop to react to. The only way out was Ctrl-C. 3. Retry decided by message prose. The catch block told API errors from network errors with message.startsWith('Error:'), so rewording an error template silently changed retry behavior, and a network failure worded that way was misclassified as final and never retried. Changes: - Add Response_envelope {status, headers, body} and an opt-in get_with_status(). request()/get()/post() keep returning the body, so all existing call sites are untouched; only pollers that must reason about the protocol opt in. Headers are exposed too, which is what a future Retry-After-aware backoff would need. - Add Client_api_error (carrying status and hint) and discriminate with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (e.g. 'realtime job limit'), so that contract is preserved and now covered by a test. - Abort each attempt with AbortSignal.timeout (default 120s, override via Request_opts.timeout_ms). The default is deliberately generous so it cannot cut off slow-but-alive work such as protected-site scrapes or large snapshot downloads. A timed-out attempt retries at most once rather than reusing the 3-retry budget, because timeout x attempts multiplies the stall the fix exists to bound, and the retry is narrated instead of being silent. - Migrate the pipelines snapshot poller to decide readiness from the protocol first, keeping body-shape checks as fallback and adding the text-body case that the object-only check missed. The predicate returns the real status string, so progress output still distinguishes starting/building/running. Tests: 22 new (395 total). The client request loop had no coverage at all before this; the readiness predicate is table-tested across every combination of {200, 202} x {object body, text body} because that matrix is where the silent-corruption case lives.

karaposu · 2026-08-27T07:19:45Z

Three defects in src/utils/client.ts — the one function every API call in the CLI goes through. They're bundled because they're the same root cause (the boundary hides what it knows) and touch the same 30 lines.

1. The client discards the HTTP status, so polling guesses

On success the client returned only the parsed body. 202 is res.ok, so a caller could not distinguish "job accepted, still building" from "here is your data" — both arrived as a body with no status attached.

Snapshot polling therefore infers readiness from the body's shape (extract_status + RUNNING_STATUSES). That works while the body is a parsed object, i.e. --format json. It silently fails for --format csv|ndjson|jsonl, where the client returns text:

constextract_status=(result: unknown)=>{if(!result||typeofresult!='object')// a string body exits herereturnundefined;// → "not running" → treated as data

So a not-ready snapshot requested as JSONL gets printed as if it were the dataset, and the CLI exits 0. Silent wrong data is the worst outcome for exactly the ETL workflows those formats exist to serve.

Fix: an opt-in get_with_status() returning {status, headers, body}. request() / get() / post() still return the body, so no existing call site changes — only code that must reason about the protocol opts in. Readiness now checks, in order of authority: HTTP 202 → parsed-object status → text body that parses to a status. The predicate returns the real status string, so progress output still shows starting → building → running instead of collapsing to one token.

Headers are exposed too — that's the plumbing a future Retry-After-aware backoff needs.

2. No request timeout — the CLI could hang forever

fetch() has no default timeout and was called without a signal. A black-holed connection (VPN drop, hung LB) produced no error at all, so the retry loop never engaged: spinner spinning, no output, Ctrl-C the only exit.

Fix:AbortSignal.timeout per attempt (fresh signal each time — hoisting it would abort retries instantly), default 120s, overridable via Request_opts.timeout_ms.

Two deliberate choices worth reviewing:

120s, not 30s. The abort must bound hung connections without cutting off slow-but-alive work — protected-site scrapes and large snapshot bodies legitimately run long. Happy to lower it if you have real numbers on the long tail.
A timed-out attempt retries once, not 3×. Reusing the generic retry budget would mean 4 × 120s ≈ 8 minutes of silent stall — multiplying the very wait this fix exists to bound. The retry is also narrated rather than silent.

3. Retry was decided by error message prose

if(einstanceofError&&e.message.startsWith('Error:'))throwe;// treat as final API error, don't retry

Retry semantics were coupled to message wording: rewording a template silently changes behavior, and a network failure whose message happens to start with Error: was misclassified as final and never retried.

Fix: a Client_api_error class (carrying status and hint) discriminated with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (REALTIME_LIMIT_MARKER = 'realtime job limit', clean_error_message), so that contract is preserved and now has a test pinning it.

Testing

22 new tests, 395 total. The request() loop had no test coverage at all before this — these are the first.
The readiness predicate is table-tested across every combination of {200, 202} × {object body, text body}, because that matrix is precisely where the silent-corruption case lives.
vi.mocks utils/config so URL assertions don't depend on the developer's local config.json (load_config() runs on every request and can override api_url).
Verified by running the built CLI.

Scope

Deliberately not included: retry policy/backoff changes (that's #20's area), migrating other call sites (status.ts is correctly excluded — for /datasets/v3/progress the status metadata is the payload), and retiring scraper.ts's parallel fetch_raw helper. That helper exists only because the shared client couldn't expose status codes; get_with_status removes its reason to exist, but that's a follow-up, not this diff.

Independent of #28 (Node 20 fix) — different files, no conflicts, either can merge first.

Note: this PR reports no CI checks because main has no CI workflow yet (#28 adds one — type-check + tests on a Node 20.17.0 / 24 matrix). Verified locally instead: tsc --noEmit clean, 391 tests passing, and the built CLI smoke-run.

Three defects in src/utils/client.ts, the function every API call goes through: 1. Protocol erasure. On success the client returned only the parsed body and discarded the status code, so a caller could not tell HTTP 202 (job accepted, still building) from 200 (this is your data) — 202 is res.ok, so both looked identical. Snapshot polling therefore inferred readiness from the body's *shape*. That misses the not-ready case whenever the body is text, which is exactly what --format csv/ndjson/ jsonl produce: the status stub is printed as if it were the dataset and the CLI exits 0. Silent wrong data is the worst failure mode for the ETL use case these formats exist for. 2. No request timeout. fetch() has no default timeout and was called without a signal, so a black-holed connection (VPN drop, hung load balancer) hung forever — no error ever arrived for the retry loop to react to. The only way out was Ctrl-C. 3. Retry decided by message prose. The catch block told API errors from network errors with message.startsWith('Error:'), so rewording an error template silently changed retry behavior, and a network failure worded that way was misclassified as final and never retried. Changes: - Add Response_envelope {status, headers, body} and an opt-in get_with_status(). request()/get()/post() keep returning the body, so all existing call sites are untouched; only pollers that must reason about the protocol opt in. Headers are exposed too, which is what a future Retry-After-aware backoff would need. - Add Client_api_error (carrying status and hint) and discriminate with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (e.g. 'realtime job limit'), so that contract is preserved and now covered by a test. - Abort each attempt with AbortSignal.timeout (default 120s, override via Request_opts.timeout_ms). The default is deliberately generous so it cannot cut off slow-but-alive work such as protected-site scrapes or large snapshot downloads. A timed-out attempt retries at most once rather than reusing the 3-retry budget, because timeout x attempts multiplies the stall the fix exists to bound, and the retry is narrated instead of being silent. - Migrate the pipelines snapshot poller to decide readiness from the protocol first, keeping body-shape checks as fallback and adding the text-body case that the object-only check missed. The predicate returns the real status string, so progress output still distinguishes starting/building/running. Tests: 22 new (395 total). The client request loop had no coverage at all before this; the readiness predicate is table-tested across every combination of {200, 202} x {object body, text body} because that matrix is where the silent-corruption case lives.

karaposu · 2026-08-27T07:19:45Z

Three defects in src/utils/client.ts — the one function every API call in the CLI goes through. They're bundled because they're the same root cause (the boundary hides what it knows) and touch the same 30 lines.

1. The client discards the HTTP status, so polling guesses

On success the client returned only the parsed body. 202 is res.ok, so a caller could not distinguish "job accepted, still building" from "here is your data" — both arrived as a body with no status attached.

Snapshot polling therefore infers readiness from the body's shape (extract_status + RUNNING_STATUSES). That works while the body is a parsed object, i.e. --format json. It silently fails for --format csv|ndjson|jsonl, where the client returns text:

constextract_status=(result: unknown)=>{if(!result||typeofresult!='object')// a string body exits herereturnundefined;// → "not running" → treated as data

So a not-ready snapshot requested as JSONL gets printed as if it were the dataset, and the CLI exits 0. Silent wrong data is the worst outcome for exactly the ETL workflows those formats exist to serve.

Fix: an opt-in get_with_status() returning {status, headers, body}. request() / get() / post() still return the body, so no existing call site changes — only code that must reason about the protocol opts in. Readiness now checks, in order of authority: HTTP 202 → parsed-object status → text body that parses to a status. The predicate returns the real status string, so progress output still shows starting → building → running instead of collapsing to one token.

Headers are exposed too — that's the plumbing a future Retry-After-aware backoff needs.

2. No request timeout — the CLI could hang forever

fetch() has no default timeout and was called without a signal. A black-holed connection (VPN drop, hung LB) produced no error at all, so the retry loop never engaged: spinner spinning, no output, Ctrl-C the only exit.

Fix:AbortSignal.timeout per attempt (fresh signal each time — hoisting it would abort retries instantly), default 120s, overridable via Request_opts.timeout_ms.

Two deliberate choices worth reviewing:

120s, not 30s. The abort must bound hung connections without cutting off slow-but-alive work — protected-site scrapes and large snapshot bodies legitimately run long. Happy to lower it if you have real numbers on the long tail.
A timed-out attempt retries once, not 3×. Reusing the generic retry budget would mean 4 × 120s ≈ 8 minutes of silent stall — multiplying the very wait this fix exists to bound. The retry is also narrated rather than silent.

3. Retry was decided by error message prose

if(einstanceofError&&e.message.startsWith('Error:'))throwe;// treat as final API error, don't retry

Retry semantics were coupled to message wording: rewording a template silently changes behavior, and a network failure whose message happens to start with Error: was misclassified as final and never retried.

Fix: a Client_api_error class (carrying status and hint) discriminated with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (REALTIME_LIMIT_MARKER = 'realtime job limit', clean_error_message), so that contract is preserved and now has a test pinning it.

Testing

22 new tests, 395 total. The request() loop had no test coverage at all before this — these are the first.
The readiness predicate is table-tested across every combination of {200, 202} × {object body, text body}, because that matrix is precisely where the silent-corruption case lives.
vi.mocks utils/config so URL assertions don't depend on the developer's local config.json (load_config() runs on every request and can override api_url).
Verified by running the built CLI.

Scope

Deliberately not included: retry policy/backoff changes (that's #20's area), migrating other call sites (status.ts is correctly excluded — for /datasets/v3/progress the status metadata is the payload), and retiring scraper.ts's parallel fetch_raw helper. That helper exists only because the shared client couldn't expose status codes; get_with_status removes its reason to exist, but that's a follow-up, not this diff.

Independent of #28 (Node 20 fix) — different files, no conflicts, either can merge first.

Note: this PR reports no CI checks because main has no CI workflow yet (#28 adds one — type-check + tests on a Node 20.17.0 / 24 matrix). Verified locally instead: tsc --noEmit clean, 391 tests passing, and the built CLI smoke-run.

Three defects in src/utils/client.ts, the function every API call goes through: 1. Protocol erasure. On success the client returned only the parsed body and discarded the status code, so a caller could not tell HTTP 202 (job accepted, still building) from 200 (this is your data) — 202 is res.ok, so both looked identical. Snapshot polling therefore inferred readiness from the body's *shape*. That misses the not-ready case whenever the body is text, which is exactly what --format csv/ndjson/ jsonl produce: the status stub is printed as if it were the dataset and the CLI exits 0. Silent wrong data is the worst failure mode for the ETL use case these formats exist for. 2. No request timeout. fetch() has no default timeout and was called without a signal, so a black-holed connection (VPN drop, hung load balancer) hung forever — no error ever arrived for the retry loop to react to. The only way out was Ctrl-C. 3. Retry decided by message prose. The catch block told API errors from network errors with message.startsWith('Error:'), so rewording an error template silently changed retry behavior, and a network failure worded that way was misclassified as final and never retried. Changes: - Add Response_envelope {status, headers, body} and an opt-in get_with_status(). request()/get()/post() keep returning the body, so all existing call sites are untouched; only pollers that must reason about the protocol opt in. Headers are exposed too, which is what a future Retry-After-aware backoff would need. - Add Client_api_error (carrying status and hint) and discriminate with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (e.g. 'realtime job limit'), so that contract is preserved and now covered by a test. - Abort each attempt with AbortSignal.timeout (default 120s, override via Request_opts.timeout_ms). The default is deliberately generous so it cannot cut off slow-but-alive work such as protected-site scrapes or large snapshot downloads. A timed-out attempt retries at most once rather than reusing the 3-retry budget, because timeout x attempts multiplies the stall the fix exists to bound, and the retry is narrated instead of being silent. - Migrate the pipelines snapshot poller to decide readiness from the protocol first, keeping body-shape checks as fallback and adding the text-body case that the object-only check missed. The predicate returns the real status string, so progress output still distinguishes starting/building/running. Tests: 22 new (395 total). The client request loop had no coverage at all before this; the readiness predicate is table-tested across every combination of {200, 202} x {object body, text body} because that matrix is where the silent-corruption case lives.

karaposu · 2026-08-27T07:19:45Z

Three defects in src/utils/client.ts — the one function every API call in the CLI goes through. They're bundled because they're the same root cause (the boundary hides what it knows) and touch the same 30 lines.

1. The client discards the HTTP status, so polling guesses

On success the client returned only the parsed body. 202 is res.ok, so a caller could not distinguish "job accepted, still building" from "here is your data" — both arrived as a body with no status attached.

Snapshot polling therefore infers readiness from the body's shape (extract_status + RUNNING_STATUSES). That works while the body is a parsed object, i.e. --format json. It silently fails for --format csv|ndjson|jsonl, where the client returns text:

constextract_status=(result: unknown)=>{if(!result||typeofresult!='object')// a string body exits herereturnundefined;// → "not running" → treated as data

So a not-ready snapshot requested as JSONL gets printed as if it were the dataset, and the CLI exits 0. Silent wrong data is the worst outcome for exactly the ETL workflows those formats exist to serve.

Fix: an opt-in get_with_status() returning {status, headers, body}. request() / get() / post() still return the body, so no existing call site changes — only code that must reason about the protocol opts in. Readiness now checks, in order of authority: HTTP 202 → parsed-object status → text body that parses to a status. The predicate returns the real status string, so progress output still shows starting → building → running instead of collapsing to one token.

Headers are exposed too — that's the plumbing a future Retry-After-aware backoff needs.

2. No request timeout — the CLI could hang forever

fetch() has no default timeout and was called without a signal. A black-holed connection (VPN drop, hung LB) produced no error at all, so the retry loop never engaged: spinner spinning, no output, Ctrl-C the only exit.

Fix:AbortSignal.timeout per attempt (fresh signal each time — hoisting it would abort retries instantly), default 120s, overridable via Request_opts.timeout_ms.

Two deliberate choices worth reviewing:

120s, not 30s. The abort must bound hung connections without cutting off slow-but-alive work — protected-site scrapes and large snapshot bodies legitimately run long. Happy to lower it if you have real numbers on the long tail.
A timed-out attempt retries once, not 3×. Reusing the generic retry budget would mean 4 × 120s ≈ 8 minutes of silent stall — multiplying the very wait this fix exists to bound. The retry is also narrated rather than silent.

3. Retry was decided by error message prose

if(einstanceofError&&e.message.startsWith('Error:'))throwe;// treat as final API error, don't retry

Retry semantics were coupled to message wording: rewording a template silently changes behavior, and a network failure whose message happens to start with Error: was misclassified as final and never retried.

Fix: a Client_api_error class (carrying status and hint) discriminated with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (REALTIME_LIMIT_MARKER = 'realtime job limit', clean_error_message), so that contract is preserved and now has a test pinning it.

Testing

22 new tests, 395 total. The request() loop had no test coverage at all before this — these are the first.
The readiness predicate is table-tested across every combination of {200, 202} × {object body, text body}, because that matrix is precisely where the silent-corruption case lives.
vi.mocks utils/config so URL assertions don't depend on the developer's local config.json (load_config() runs on every request and can override api_url).
Verified by running the built CLI.

Scope

Deliberately not included: retry policy/backoff changes (that's #20's area), migrating other call sites (status.ts is correctly excluded — for /datasets/v3/progress the status metadata is the payload), and retiring scraper.ts's parallel fetch_raw helper. That helper exists only because the shared client couldn't expose status codes; get_with_status removes its reason to exist, but that's a follow-up, not this diff.

Independent of #28 (Node 20 fix) — different files, no conflicts, either can merge first.

Note: this PR reports no CI checks because main has no CI workflow yet (#28 adds one — type-check + tests on a Node 20.17.0 / 24 matrix). Verified locally instead: tsc --noEmit clean, 391 tests passing, and the built CLI smoke-run.

Three defects in src/utils/client.ts, the function every API call goes through: 1. Protocol erasure. On success the client returned only the parsed body and discarded the status code, so a caller could not tell HTTP 202 (job accepted, still building) from 200 (this is your data) — 202 is res.ok, so both looked identical. Snapshot polling therefore inferred readiness from the body's *shape*. That misses the not-ready case whenever the body is text, which is exactly what --format csv/ndjson/ jsonl produce: the status stub is printed as if it were the dataset and the CLI exits 0. Silent wrong data is the worst failure mode for the ETL use case these formats exist for. 2. No request timeout. fetch() has no default timeout and was called without a signal, so a black-holed connection (VPN drop, hung load balancer) hung forever — no error ever arrived for the retry loop to react to. The only way out was Ctrl-C. 3. Retry decided by message prose. The catch block told API errors from network errors with message.startsWith('Error:'), so rewording an error template silently changed retry behavior, and a network failure worded that way was misclassified as final and never retried. Changes: - Add Response_envelope {status, headers, body} and an opt-in get_with_status(). request()/get()/post() keep returning the body, so all existing call sites are untouched; only pollers that must reason about the protocol opt in. Headers are exposed too, which is what a future Retry-After-aware backoff would need. - Add Client_api_error (carrying status and hint) and discriminate with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (e.g. 'realtime job limit'), so that contract is preserved and now covered by a test. - Abort each attempt with AbortSignal.timeout (default 120s, override via Request_opts.timeout_ms). The default is deliberately generous so it cannot cut off slow-but-alive work such as protected-site scrapes or large snapshot downloads. A timed-out attempt retries at most once rather than reusing the 3-retry budget, because timeout x attempts multiplies the stall the fix exists to bound, and the retry is narrated instead of being silent. - Migrate the pipelines snapshot poller to decide readiness from the protocol first, keeping body-shape checks as fallback and adding the text-body case that the object-only check missed. The predicate returns the real status string, so progress output still distinguishes starting/building/running. Tests: 22 new (395 total). The client request loop had no coverage at all before this; the readiness predicate is table-tested across every combination of {200, 202} x {object body, text body} because that matrix is where the silent-corruption case lives.

karaposu force-pushed the fix/http-boundary-hardening branch from 1bf0d98 to c9f5a0dCompare August 27, 2026 07:21

karaposu mentioned this pull request Aug 27, 2026
feat(marketplace): query Bright Data's pre-collected datasets #30
Open