OneIO - all-in-one IO library for Rust

OneIO is a Rust library providing unified IO operations for reading and writing compressed files from local and remote sources with both synchronous and asynchronous support.

Quick Start

oneio = "0.25"# Default: gz, bz, https

Feature Selection Guide

Common Use Cases

Local files only:

oneio = { version = "0.25", default-features = false, features = ["gz", "bz"] }

HTTPS with default rustls:

oneio = { version = "0.25", default-features = false, features = ["https", "gz"] }

HTTPS with custom TLS backend:

# With rustlsoneio = { version = "0.25", default-features = false, features = ["http", "rustls", "gz"] }
# With native-tls (recommended for corporate proxies/VPNs)oneio = { version = "0.25", default-features = false, features = ["http", "native-tls", "gz"] }

S3-compatible storage:

oneio = { version = "0.25", default-features = false, features = ["s3", "https", "gz"] }

Async operations:

oneio = { version = "0.25", features = ["async"] }

Available Features

Compression (choose only what you need):

• gz - Gzip via flate2
• bz - Bzip2
• lz - LZ4
• xz - XZ
• zstd - Zstandard (balanced)

Protocols:

• http - HTTP-only support (no TLS)
• https - HTTP/HTTPS with rustls TLS backend (equivalent to http + rustls)
• ftp - FTP support (requires http + TLS backend)
• s3 - S3-compatible storage
• reqwest-gzip - Opt-in HTTP gzip content-encoding: advertises Accept-Encoding: gzip and transparently decodes gzipped responses (distinct from gz, which is URL-suffix-based file decompression)

TLS Backends (for HTTPS - mutually exclusive):

• rustls - Pure Rust TLS (use with http). Uses both system certificates and bundled Mozilla certificates for maximum compatibility.
• native-tls - Platform native TLS (use with http). Recommended for corporate proxies and VPNs (Cloudflare WARP, etc.) as it uses the OS trust store.

Additional:

• async - Async support (limited to gz, bz, zstd for compression)
• json - JSON parsing
• digest - SHA256 digest calculation
• cli - Command-line tool

Working with Corporate Proxies (Cloudflare WARP, etc.)

If you're behind a corporate proxy or VPN like Cloudflare WARP that uses custom TLS certificates:

[dependencies]
oneio = { version = "0.25", default-features = false, features = ["http", "native-tls", "gz"] }

The native-tls feature uses your operating system's TLS stack with its trust store, which includes custom corporate certificates. This works for both HTTP/HTTPS and S3 operations.

Alternatively, you can add custom CA certificates:

use oneio::OneIo;let oneio = OneIo::builder().add_root_certificate_pem(&std::fs::read("company-ca.pem")?)?
.build()?;

Or set the ONEIO_CA_BUNDLE environment variable:

export ONEIO_CA_BUNDLE=/path/to/company-ca.pem

Environment Variables:

• ONEIO_ACCEPT_INVALID_CERTS=true - Accept invalid TLS certificates (insecure, for development only)
• ONEIO_CA_BUNDLE=/path/to/ca.pem - Add custom CA certificate to trust store

Library Usage

Basic Reading and Writing

Read all content into a string (works with compression and remote files automatically):

use oneio;let content = oneio::read_to_string_lossy("https://spaces.bgpkit.org/oneio/test_data.txt.gz")?;println!("{}", content);

Read line by line:

use oneio;let lines = oneio::read_lines_lossy("https://spaces.bgpkit.org/oneio/test_data.txt.gz")?
.map(|line| line.unwrap()).collect::<Vec<String>>();for line in lines {println!("{}", line);}

Get a reader for streaming:

use oneio;use std::io::Read;letmut reader = oneio::get_reader("tests/test_data.txt.gz")?;letmut buffer = Vec::new();
reader.read_to_end(&mut buffer)?;

Resumable HTTP Reader

For long HTTP(S) downloads, get_resumable_http_reader reconnects with Range requests after a dropped connection and continues from the last byte read. It validates Content-Range, ETag, and Last-Modified when available to avoid combining data from different resource versions. Existing readers and download retain their current behavior.

use std::io::Read;letmut reader = oneio::get_resumable_http_reader("https://example.com/large-file.gz")?;letmut buffer = Vec::new();
reader.read_to_end(&mut buffer)?;

Write with automatic compression:

use oneio;use std::io::Write;letmut writer = oneio::get_writer("output.txt.gz")?;
writer.write_all(b"Hello, compressed world!")?;drop(writer);// Important: close the writer// Read it backlet content = oneio::read_to_string_lossy("output.txt.gz")?;

Reusable OneIo Clients

The OneIo client allows you to configure headers, TLS certificates, timeouts, and other options once, then reuse the configuration across multiple operations:

use oneio::OneIo;// reqwest is re-exported for naming HTTP types without a direct reqwest dependencyuse oneio::reqwest::header::{HeaderName,HeaderValue};// Build a reusable client with custom headers and certificateslet oneio = OneIo::builder().header_str("Authorization","Bearer TOKEN").add_root_certificate_pem(&std::fs::read("company-ca.pem")?)?
.timeout(std::time::Duration::from_secs(30)).connect_timeout(std::time::Duration::from_secs(10)).build()?;// Reuse the same configuration for multiple requestslet content1 = oneio.read_to_string_lossy("https://api.example.com/data1.json")?;let content2 = oneio.read_to_string_lossy("https://api.example.com/data2.json")?;

Builder Methods:

• .header(name, value) - Add a typed header (infallible, uses HeaderName and HeaderValue)
• .header_str(name, value) - Add a string header (panics on invalid input)
• .user_agent(value) - Set User-Agent header
• .add_root_certificate_pem(pem) - Add custom CA certificate (PEM format)
• .add_root_certificate_der(der) - Add custom CA certificate (DER format)
• .danger_accept_invalid_certs(true) - Accept invalid certificates
• .timeout(duration) - Set request timeout
• .connect_timeout(duration) - Set connection timeout
• .proxy(proxy) - Set HTTP proxy
• .no_proxy() - Disable system proxy
• .redirect(policy) - Set redirect policy
• .configure_http(f) - Escape hatch for direct reqwest configuration

Compression Override

For URLs with query parameters or non-standard extensions, use explicit compression type:

use oneio::OneIo;let oneio = OneIo::new()?;// URL has query params, so we specify compression explicitlylet reader = oneio.get_reader_with_type("https://api.example.com/data?format=gzip","gz")?;

Progress Tracking

Track download/read progress with callbacks:

use oneio::OneIo;let oneio = OneIo::new()?;// Callback receives (bytes_read, total_bytes).// total_bytes is 0 when the server does not provide a Content-Length.// The returned Option<u64> is Some(total) when the size was known upfront.let(mut reader, total_size) = oneio.get_reader_with_progress("https://example.com/largefile.gz",
|bytes_read, total_bytes| {if total_bytes > 0{let percent = (bytes_read asf64 / total_bytes asf64)*100.0;println!("Progress: {:.1}%", percent);}else{println!("Downloaded: {} bytes", bytes_read);}})?;

Async Support (Feature: async)

use oneio;#[tokio::main]asyncfnmain() -> Result<(),Box<dyn std::error::Error>>{let content = oneio::read_to_string_lossy_async("https://example.com/data.json.gz").await?;
oneio::download_async("https://example.com/data.csv.gz","local_data.csv.gz").await?;Ok(())}

Note: Async compression is limited to gz, bz, zstd. LZ4/XZ return NotSupported.

S3 Operations (Feature: s3)

use oneio::s3::*;// Upload file to S3// Files < 8MB use single PUT; larger files use multipart uploads3_upload("my-bucket","path/to/file.txt","local/file.txt")?;// Download file from S3s3_download("my-bucket","path/to/file.txt","downloaded.txt")?;// Copy object within the same buckets3_copy("my-bucket","path/to/file.txt","path/to/copy.txt")?;// Delete objects3_delete("my-bucket","path/to/copy.txt")?;// Read S3 directly using OneIOlet oneio = oneio::OneIo::new()?;let content = oneio.read_to_string_lossy("s3://my-bucket/path/to/file.txt")?;// Check existence and get metadataifs3_exists("my-bucket","path/to/file.txt")? {let stats = s3_stats("my-bucket","path/to/file.txt")?;println!("Size: {} bytes", stats.content_length);}// List objectslet objects = s3_list("my-bucket","path/",Some("/".to_string()),false)?;

Required environment variables for S3:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_REGION - Use "auto" for Cloudflare R2
• AWS_ENDPOINT - e.g. https://xxx.r2.cloudflarestorage.com

Optional environment variables for S3:

• AWS_SESSION_TOKEN - Temporary session token
• ONEIO_S3_CHUNK_SIZE - Multipart part size in bytes (default: 8MB)
• ONEIO_S3_MULTIPART_THRESHOLD - File size threshold for multipart upload (default: 5MB)

Error Handling

OneIO uses a simplified error enum with #[non_exhaustive] for forward compatibility:

use oneio::OneIoError;match oneio::get_reader("file.txt"){Ok(reader) => {/* use reader */},Err(OneIoError::Io(e)) => {/* filesystem error */},Err(OneIoError::Network(e)) => {/* network error */},Err(OneIoError::NetworkWithContext{ source, url }) => {// Network error with URL context for debuggingeprintln!("Failed to fetch {}: {}", url, source);}Err(OneIoError::Status{ service, code, .. }) => {/* remote status error */},Err(OneIoError::InvalidCertificate(msg)) => {/* TLS cert error */},Err(OneIoError::NotSupported(msg)) => {/* feature not compiled */},
_ => {/* handle future error variants */}}

Crypto Provider Initialization (Rustls)

When using HTTPS, S3, or FTP features with rustls, oneio automatically initializes a crypto provider (AWS-LC or ring) on first use. For more control, initialize it explicitly:

use oneio;fnmain() -> Result<(),Box<dyn std::error::Error>>{// Initialize crypto provider explicitly at startup
oneio::crypto::ensure_default_provider()?;// Now all HTTPS/S3/FTP operations will worklet content = oneio::read_to_string_lossy("https://example.com/data.txt")?;Ok(())}

Command Line Tool

Install the CLI tool:

cargo install oneio --features cli

Basic Usage

Read and print a remote compressed file:

$ oneio https://spaces.bgpkit.org/oneio/test_data.txt.gz
OneIO test file.
This is a test.

Read local compressed file:

$ oneio tests/test_data.txt.gz
OneIO test file.
This is a test.

Get file statistics:

$ oneio tests/test_data.txt --stats
lines: 2
chars: 31

Download with Progress Bar

Download a file with automatic progress bar (shown when stderr is a terminal):

$ oneio -d https://example.com/largefile.bz2
downloaded to largefile.bz2

When stderr is piped or redirected the progress bar is suppressed.

Custom HTTP Headers

Add custom headers for API authentication:

$ oneio -H "Authorization: Bearer TOKEN" -H "X-Custom-Header: value" https://api.example.com/data.json

Compression Override

For URLs with query parameters where extension detection fails:

$ oneio --compression gz "https://api.example.com/data?format=gzip"

Caching

Cache remote files locally for repeated reads:

$ oneio --cache-dir /tmp/cache https://example.com/largefile.gz
# Second read uses cache
$ oneio --cache-dir /tmp/cache https://example.com/largefile.gz

Force re-download even if cache exists:

$ oneio --cache-dir /tmp/cache --cache-force https://example.com/largefile.gz

S3 Operations

Upload file to S3:

$ oneio s3 upload local-file.txt my-bucket path/in/s3.txt
uploaded to s3://my-bucket/path/in/s3.txt

Download file from S3:

$ oneio s3 download my-bucket path/in/s3.txt -o local-file.txt
downloaded s3://my-bucket/path/in/s3.txt to local-file.txt

List S3 bucket:

$ oneio s3 list my-bucket path/ --delimiter "/"

List directories only:

$ oneio s3 list my-bucket path/ --dirs

Generate SHA256 Digest

$ oneio digest tests/test_data.txt
a3f5c8e9d2b1... (64 hex characters)

CLI Help Output

$ oneio --help
oneio reads files from local or remote locations with any compression
Usage: oneio [OPTIONS] [FILE] [COMMAND]
Commands:
s3 S3-related subcommands
digest Generate SHA256 digest
help Print this message or the given subcommand(s)
Arguments:
[FILE] file to open, remote or local
Options:
-d, --download download the file to the current directory
-o, --outfile <OUTFILE> output file path
--cache-dir <CACHE_DIR> cache reading to a specified directory
--cache-force force re-caching if a local cache already exists
--cache-file <CACHE_FILE> specify cache file name
-s, --stats read through the file and only print out stats
-H, --header <HEADERS> Add HTTP header (format: "Name: Value"), can be repeated
--compression <COMPRESSION> Override compression type (gz, bz2, lz4, xz, zst)
-h, --help Print help
-V, --version Print version

Supported Formats

Compression Detection

OneIO detects compression algorithm by the file extensions:

• Gzip: .gz, .gzip, .tgz
• Bzip2: .bz, .bz2
• LZ4: .lz4, .lz
• XZ: .xz, .xz2, .lzma
• Zstandard: .zst, .zstd

For URLs with query parameters, use --compression flag or get_reader_with_type().

Protocol Support

• Local files: /path/to/file.txt
• HTTP/HTTPS: https://example.com/file.txt.gz
• FTP: ftp://ftp.example.com/file.txt (requires ftp feature)
• S3: s3://bucket/path/file.txt (requires s3 feature)

License

MIT