Summary
The CLI is convenient for one-URL probes, but its x_posts and
linkedin_posts commands only accept a URL:
brightdata pipelines x_posts <url>
brightdata pipelines linkedin_posts <url>
This prevents the CLI from reproducing Web Scraper API and Control Panel jobs
that use discovery modes and structured inputs such as start_date,
end_date, only_authored_posts, and record limits.
I had to bypass the CLI and implement direct /datasets/v3/trigger calls to
make the collection reproducible.
Environment
- Windows 11
- PowerShell
@brightdata/cli v0.3.1- X posts dataset:
gd_lwxkxvnf1cynvib9co - LinkedIn posts dataset:
gd_lyy3tktm25m4avu764
Current behavior
Running either command without a URL shows only:
Usage: brightdata pipelines x_posts <url>
Usage: brightdata pipelines linkedin_posts <url>
There is no documented way to select:
type=discover_newdiscover_by=profile_urldiscover_by=urlstart_dateend_dateonly_authored_posts- per-input record limit
include_errors- asynchronous trigger/poll/download control
- JSONL/NDJSON raw output with snapshot metadata
The discovery modes have different input schemas. For LinkedIn:
discover_by=profile_url accepts date fields and
only_authored_posts.discover_by=url accepts a record limit but rejects date fields and
only_authored_posts.
The CLI does not surface this distinction.
Why this matters
The same collection performed through the API and Control Panel returned:
- X: 454 records from six profile inputs for equivalent date windows.
- LinkedIn: 31 authored records from four profile inputs.
The CLI cannot express those jobs, inspect the exact submitted input, or
replay them later. This makes it unsuitable as the primary interface for
versioned ETL despite being useful for individual probes.
The CLI also cannot reveal whether a low record count is caused by:
- an internal discovery limit;
- exhausted pagination;
- platform visibility;
- author filtering;
- a requested record cap; or
- another stop condition.
Requested improvement
Please add either generic dataset triggering:
bdata datasets trigger \
--dataset-id gd_lwxkxvnf1cynvib9co \
--type discover_new \
--discover-by profile_url \
--input-file inputs.jsonl \
--include-errors \
--format jsonl
or scraper-specific options:
bdata pipelines x_posts \
--url https://x.com/sama \
--discover-by profile_url \
--start-date 2022-11-30T00:00:00Z \
--end-date 2026-06-08T23:59:59Z \
--record-limit 15000 \
--async \
--format jsonl
The command should save or print:
- dataset ID and discovery mode;
- exact submitted input;
- snapshot ID;
- progress/status;
- delivered and error record counts;
- estimated and actual cost when available;
- output checksum;
- API validation errors without truncation;
- collector stop/completeness metadata when supplied by the backend.
Expected outcome
Every job that can be configured in the Control Panel should be reproducible
through the CLI without requiring users to reverse-engineer the REST API.
Summary
The CLI is convenient for one-URL probes, but its
x_postsandlinkedin_postscommands only accept a URL:This prevents the CLI from reproducing Web Scraper API and Control Panel jobs
that use discovery modes and structured inputs such as
start_date,end_date,only_authored_posts, and record limits.I had to bypass the CLI and implement direct
/datasets/v3/triggercalls tomake the collection reproducible.
Environment
@brightdata/cli v0.3.1gd_lwxkxvnf1cynvib9cogd_lyy3tktm25m4avu764Current behavior
Running either command without a URL shows only:
There is no documented way to select:
type=discover_newdiscover_by=profile_urldiscover_by=urlstart_dateend_dateonly_authored_postsinclude_errorsThe discovery modes have different input schemas. For LinkedIn:
discover_by=profile_urlaccepts date fields andonly_authored_posts.discover_by=urlaccepts a record limit but rejects date fields andonly_authored_posts.The CLI does not surface this distinction.
Why this matters
The same collection performed through the API and Control Panel returned:
The CLI cannot express those jobs, inspect the exact submitted input, or
replay them later. This makes it unsuitable as the primary interface for
versioned ETL despite being useful for individual probes.
The CLI also cannot reveal whether a low record count is caused by:
Requested improvement
Please add either generic dataset triggering:
or scraper-specific options:
The command should save or print:
Expected outcome
Every job that can be configured in the Control Panel should be reproducible
through the CLI without requiring users to reverse-engineer the REST API.