feat: support compression when importing/exporting events (gzip/xz)

[Anshumancanrock]

Description

This PR adds practical compression support to event import/export, without changing the existing JSONL workflow.

• Export now supports gzip and xz output in a stream pipeline.
• Import now auto-detects compression (extension + magic bytes) and decompresses on the fly.
• Export CLI includes compression flags plus end-of-run size/throughput summary.
• Docs and tests were updated for the new paths.

Related Issue

Closes#407

Motivation and Context

People already compress exported event files for storage and transfer. This change removes the manual steps around that and keeps the flow memory-safe by streaming throughout.

How Has This Been Tested?

Tested locally on Linux.

• npm run build:check
• npm run test:unit (889 passing)
• Compression integration scenario is covered and step wiring was validated with dry-run

Screenshots (if appropriate):

N/A (CLI/backend changes)

Types of changes

• Non-functional change (docs, style, minor refactor)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my code changes.
• I added a changeset, or this is docs-only and I added an empty changeset.
• All new and existing tests passed.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	lzma-native@​8.0.6

View full report

[coveralls]

Coverage is 73.05% — Anshumancanrock:feature/event-import-export-compression into cameri:main. No base build found for cameri:main.

[Copilot]

Pull request overview

Adds streaming gzip/xz compression support to the event import/export CLIs while preserving the existing JSONL-based workflow, improving portability and storage/transfer efficiency for large exports.

Changes:

• Introduces src/utils/compression.ts with format parsing, detection (extension + magic bytes), and streaming (de)compression helpers.
• Updates export-events to optionally compress output, add CLI flags, and print end-of-run size/throughput metrics.
• Updates import-events to auto-detect compressed inputs and decompress on-the-fly; adds unit + integration coverage and docs.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/utils/compression.ts	New shared compression utilities (format parsing, detection, gzip/xz streams, xz tuning).
src/scripts/export-events.ts	Adds compression flags, streaming compression pipeline, and export metrics/usage handling.
src/import-events.ts	Removes .jsonl-only restriction; adds compression auto-detection + decompression stream creation and updated CLI parsing/help.
src/services/event-import-service.ts	Refactors importer to accept a readable stream (importFromReadable) and keeps file-path import as a wrapper.
test/unit/utils/compression.spec.ts	Unit tests for compression helpers, detection, env tuning, and stream round-trips.
test/unit/scripts/export-events.spec.ts	Unit tests for export CLI arg parsing and metrics formatting helpers.
test/unit/import-events.spec.ts	Unit tests for import CLI arg parsing and updated error cases.
test/unit/services/event-import-service.spec.ts	Adds coverage for importing from a readable stream.
test/integration/features/compression/compression-roundtrip.feature	New cucumber feature for gzip/xz export+import roundtrip.
test/integration/features/compression/compression-roundtrip.feature.ts	Step definitions to seed DB, run CLIs, and verify roundtrip restore.
README.md	Documents compressed import support, export compression flags, and xz tuning env vars.
package.json	Adds lzma-native dependency for xz support.
package-lock.json	Locks lzma-native and related transitive dependencies.
.knip.json	Ignores lzma-native dependency (used via dynamic require).
.changeset/quick-maps-marry.md	Changeset documenting the new compression feature.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cameri]

@Anshumancanrock how did you manually test this?

[Anshumancanrock]

Hi @cameri , so I tested this locally before submitting.

1. I created a small deterministic 3-event JSONL fixture.
2. Checked that gzip and xz detection works , both extension+header and header-only cases + the empty .xz fallback.
3. Made sure mismatches fail correctly , renaming a gzip payload to .xz and vice versa both throw as expected.
4. Ran the import/export help commands to sanity check the CLI still looks right.
5. Ran the existing compression unit tests directly and all passing, including the invalid payload cases.

It looked solid on my side, happy to dig into anything specific if you want.