fix(collector): speed up Polymarket tape upload with zstd threads and ossutil multipart tuning

[proerror77]

Change contract

Make the Polymarket tape upload pipeline fast enough for tick-level tapes (20–25 GB each): zstd thread count is now configurable (ZSTD_THREADS, default 0 = auto, was hardcoded -T1), and every aliyun ossutil cp in the pipeline (upload, pre-check download, readback verify download) now carries configurable multipart tuning (OSS_PARALLEL, default 8; OSS_PART_SIZE, default 32Mi). Measured on the production host: default ossutil settings achieve ~50 MiB/s vs ~100+ MiB/s with --parallel 12 --part-size 32Mi; single-thread zstd was the compress-phase bottleneck under CPU quota.

Issue relationship

Refs #655

Out of scope

• Deployment env-file changes and the binary cutover on monday-trade-data-26 (separate rollout unit; code defaults already deliver the speedup, ops can set OSS_PARALLEL=12 later).
• Recorder-side tape size cap and the market-recorder rotate-on-limit fix (separate PRs).
• ACK-side processing offload (architecture follow-up).

Dependencies and merge order

None

Focused validation

• 6 new unit tests assert exact zstd/ossutil argument lists for defaults and env overrides, that oss_upload_command keeps --ignore-existing, and that UploadConfig::validate() rejects oss_parallel = 0 and a blank part size (counterexample for the new safety boundary; env parse errors fail closed via the existing env_u64).
• cargo test -p hft-collector --locked: 286 passed / 0 failed (lib) plus all bin integration suites green.
• cargo clippy -p hft-collector --locked --all-targets: no warnings. git diff --check clean; agent-worktree preflight verdict=ok.

Rollout and rollback

No production behavior changes until a new polymarket-raw-ops binary is built and delivered through the collector cutover procedure; the previous binary remains available for rollback (/opt/monday/bin/polymarket-raw-ops.rollback-*). New flags are appended after positional args, matching the existing flag placement that production already runs.

Scope exception

None

Summary by CodeRabbit

• New Features

  • Added configurable compression and multipart upload performance settings.
  • Upload tuning can now be controlled through environment variables, with sensible defaults.

• Bug Fixes

  • Added validation to reject invalid upload settings.
  • Preserved no-clobber behavior during uploads.

[coderabbitai]

📝 Walkthrough

Walkthrough

The upload command now reads zstd and OSS multipart tuning from environment variables. UploadConfig validates the values and applies them to compression and OSS copy commands. Tests cover defaults, overrides, no-clobber behavior, and invalid settings.

Changes

Polymarket upload tuning

Layer / File(s)	Summary
Configuration and validation rust_hft/tools/collector/src/polymarket_upload.rs, rust_hft/tools/collector/src/bin/polymarket-raw-ops.rs	UploadConfig stores zstd thread count, OSS parallelism, and OSS part size. The upload command reads environment values with defaults. Validation rejects zero OSS parallelism and blank part sizes.
Configurable compression and OSS upload rust_hft/tools/collector/src/polymarket_upload.rs	zstd uses the configured thread count. OSS copy commands use the configured --parallel and --part-size arguments.
Defaults and command validation rust_hft/tools/collector/src/polymarket_upload.rs	Tests cover default and explicit command arguments, --ignore-existing, and invalid OSS tuning values.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

• fix(collector): Polymarket tape recorder size cap + uploader throughput/reliability #655 — The PR implements configurable zstd threading and OSS multipart upload settings for uploader throughput.

Possibly related PRs

• proerror77/monday#34 — This PR extends the UploadConfig and CLI upload handling introduced there.
• proerror77/monday#393 — Both PRs modify upload command configuration and execution in polymarket_upload.rs.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: configurable zstd threading and ossutil multipart tuning for faster Polymarket uploads.
Description check	✅ Passed	The description completes every required template section and provides clear scope, validation, rollout, and rollback details.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/uploader-throughput

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust_hft/tools/collector/src/polymarket_upload.rs`:
- Around line 129-134: Update UploadConfig::validate to parse oss_part_size
using ossutil’s canonical size suffixes, reject malformed or unsupported part
sizes, and enforce ossutil’s valid oss_parallel range before tape preparation;
update rust_hft/tools/collector/src/polymarket_upload.rs lines 2709-2711 and
2740-2817 to expect canonical suffixes instead of 32Mi/64Mi and cover invalid
and supported-edge values; update
rust_hft/tools/collector/src/bin/polymarket-raw-ops.rs lines 278-280 to use a
canonical default such as 32M.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f5fe68b5-8c3a-4ea5-bb93-bdce30b257e5

📥 Commits

Reviewing files that changed from the base of the PR and between a80e486 and 2066d67.

📒 Files selected for processing (2)

• rust_hft/tools/collector/src/bin/polymarket-raw-ops.rs
• rust_hft/tools/collector/src/polymarket_upload.rs

[proerror77]

Verified on the production host monday-trade-data-26 (ossutil v2.3.0, invoked via the aliyun wrapper exactly as the uploader does): aliyun ossutil cp --help lists --parallel int ("Amount of concurrency tasks when work with a file") and --part-size SizeSuffix ("in the range 100Ki - 5Gi"), alongside --bigfile-threshold (default 100Mi) and --checkpoint-dir. The same flags were used live during the 2026-08-03 incident drain: --parallel 12 --part-size 32Mi uploaded 20–26 GB tapes at ~100 MiB/s vs ~50 MiB/s with defaults. Runtime smoke test with the new binary (one real tape) is planned before cutover, as noted in the PR body.