filelayer

filelayer is a small Python package that provides a simple file abstraction over:

• local filesystem
• S3-compatible object storage such as Wasabi

It exposes a minimal API:

• read_file(filepath) -> str
• write_file(filepath, file_content) -> None
• read_file_bytes(filepath) -> bytes
• write_file_bytes(filepath, file_bytes) -> None
• exists(filepath) -> bool
• resolve_path(filepath) -> str

For S3-compatible backends, filepath is treated as the object key. For local storage, filepath is resolved relative to the configured local base path.

Installation

pip install filelayer

For development:

pip install -e .[dev]

Quick start

No configuration needed — local filesystem is the default:

fromfilelayerimportStorageServicestorage=StorageService.from_settings()
storage.write_file("documents/example.txt", "Hello from local storage")
content=storage.read_file("documents/example.txt")
print(content)
storage.write_file_bytes("documents/example.bin", b"\x00\x01\x02")
print(storage.read_file_bytes("documents/example.bin"))
print(storage.exists("documents/example.txt"))
print(storage.resolve_path("documents/example.txt"))
# → /absolute/path/to/data/storage/documents/example.txt

By default, files are stored under ./data/storage. You can customize this and other settings via environment variables or a .env file:

STORAGE_PROVIDER=local# defaultSTORAGE_DEFAULT_PREFIX=my-app# optional path prefixSTORAGE_ENCODING=utf-8# defaultLOCAL_STORAGE_BASE_PATH=./data/storage# default

Wasabi / S3-compatible example

Environment:

STORAGE_PROVIDER=s3STORAGE_DEFAULT_PREFIX=my-appSTORAGE_ENCODING=utf-8S3_ENDPOINT_URL=https://s3.eu-central-1.wasabisys.comS3_ACCESS_KEY_ID=your-access-keyS3_SECRET_ACCESS_KEY=your-secret-keyS3_REGION_NAME=eu-central-1S3_BUCKET=your-bucketS3_USE_SSL=trueS3_VERIFY_SSL=trueS3_ADDRESSING_STYLE=virtualS3_CONNECT_TIMEOUT=10S3_READ_TIMEOUT=60S3_MAX_ATTEMPTS=5

Usage:

fromfilelayerimportStorageServicestorage=StorageService.from_settings()
storage.write_file("documents/example.txt", "Hello from Wasabi")
print(storage.read_file("documents/example.txt"))
print(storage.exists("documents/example.txt"))
print(storage.resolve_path("documents/example.txt"))
# → s3://your-bucket/my-app/documents/example.txt

S3 caching

The S3 provider caches downloaded objects on the local filesystem to save bandwidth. Caching is enabled by default and uses ETag-based revalidation — on repeated reads, a conditional GET is sent to S3. If the object hasn't changed (304 Not Modified), the cached copy is used. Writes are write-through: after a successful upload, the content is stored in the cache immediately.

S3_CACHE_ENABLED=true# default, set to false to disableS3_CACHE_DIR=/tmp/filelayer_cache# default: system temp directory

Notes

• STORAGE_DEFAULT_PREFIX is prepended to all paths or keys.
• write_file() stores text using STORAGE_ENCODING.
• write_file_bytes() stores raw bytes unchanged.
• Local provider prevents path traversal outside the configured storage root.

Contributing

Contributions are welcome! See CONTRIBUTING.md for development setup and guidelines.

License

This project is licensed under the MIT License.