Afero: The Universal Filesystem Abstraction for Go

Afero is a powerful and extensible filesystem abstraction system for Go. It provides a single, unified API for interacting with diverse filesystems—including the local disk, memory, archives, and network storage.

Afero acts as a drop-in replacement for the standard os package, enabling you to write modular code that is agnostic to the underlying storage, dramatically simplifies testing, and allows for sophisticated architectural patterns through filesystem composition.

Why Afero?

Afero elevates filesystem interaction beyond simple file reading and writing, offering solutions for testability, flexibility, and advanced architecture.

🔑 Key Features:

• Universal API: Write your code once. Run it against the local OS, in-memory storage, ZIP/TAR archives, or remote systems (SFTP, GCS).
• Ultimate Testability: Utilize MemMapFs, a fully concurrent-safe, read/write in-memory filesystem. Write fast, isolated, and reliable unit tests without touching the physical disk or worrying about cleanup.
• Powerful Composition: Afero's hidden superpower. Layer filesystems on top of each other to create sophisticated behaviors:
  • Sandboxing: Use CopyOnWriteFs to create temporary scratch spaces that isolate changes from the base filesystem.
  • Caching: Use CacheOnReadFs to automatically layer a fast cache (like memory) over a slow backend (like a network drive).
  • Security Jails: Use BasePathFs to restrict application access to a specific subdirectory (chroot).
• os Package Compatibility: Afero mirrors the functions in the standard os package, making adoption and refactoring seamless.
• io/fs Compatibility: Fully compatible with the Go standard library's io/fs interfaces.

Installation

go get github.com/spf13/afero

import"github.com/spf13/afero"

Quick Start: The Power of Abstraction

The core of Afero is the afero.Fs interface. By designing your functions to accept this interface rather than calling os.* functions directly, your code instantly becomes more flexible and testable.

1. Refactor Your Code

Change functions that rely on the os package to accept afero.Fs.

// Before: Coupled to the OS and difficult to test// func ProcessConfiguration(path string) error {// data, err := os.ReadFile(path)// ...// }import"github.com/spf13/afero"// After: Decoupled, flexible, and testablefuncProcessConfiguration(fs afero.Fs, pathstring) error {
// Use Afero utility functions which mirror os/ioutildata, err:=afero.ReadFile(fs, path)
// ... process the datareturnerr
}

2. Usage in Production

In your production environment, inject the OsFs backend, which wraps the standard operating system calls.

funcmain() {
// Use the real OS filesystemAppFs:=afero.NewOsFs()
ProcessConfiguration(AppFs, "/etc/myapp.conf")
}

3. Usage in Testing

In your tests, inject MemMapFs. This provides a blazing-fast, isolated, in-memory filesystem that requires no disk I/O and no cleanup.

funcTestProcessConfiguration(t*testing.T) {
// Use the in-memory filesystemAppFs:=afero.NewMemMapFs()
// Pre-populate the memory filesystem for the testconfigPath:="/test/config.json"afero.WriteFile(AppFs, configPath, []byte(`{"feature": true}`), 0644)
// Run the test entirely in memoryerr:=ProcessConfiguration(AppFs, configPath)
iferr!=nil {
t.Fatal(err)
}
}

Afero's Superpower: Composition

Afero's most unique feature is its ability to combine filesystems. This allows you to build complex behaviors out of simple components, keeping your application logic clean.

Example 1: Sandboxing with Copy-on-Write

Create a temporary environment where an application can "modify" system files without affecting the actual disk.

// 1. The base layer is the real OS, made read-only for safety.baseFs:=afero.NewReadOnlyFs(afero.NewOsFs())
// 2. The overlay layer is a temporary in-memory filesystem for changes.overlayFs:=afero.NewMemMapFs()
// 3. Combine them. Reads fall through to the base; writes only hit the overlay.sandboxFs:=afero.NewCopyOnWriteFs(baseFs, overlayFs)
// The application can now "modify" /etc/hosts, but the changes are isolated in memory.afero.WriteFile(sandboxFs, "/etc/hosts", []byte("127.0.0.1 sandboxed-app"), 0644)
// The real /etc/hosts on disk is untouched.

Example 2: Caching a Slow Filesystem

Improve performance by layering a fast cache (like memory) over a slow backend (like a network drive or cloud storage).

import"time"// Assume 'remoteFs' is a slow backend (e.g., SFTP or GCS)varremoteFs afero.Fs// 'cacheFs' is a fast in-memory backendcacheFs:=afero.NewMemMapFs()
// Create the caching layer. Cache items for 5 minutes upon first read.cachedFs:=afero.NewCacheOnReadFs(remoteFs, cacheFs, 5*time.Minute)
// The first read is slow (fetches from remote, then caches)data1, _:=afero.ReadFile(cachedFs, "data.json")
// The second read is instant (serves from memory cache)data2, _:=afero.ReadFile(cachedFs, "data.json")

Example 3: Security Jails (chroot)

Restrict an application component's access to a specific subdirectory.

osFs:=afero.NewOsFs()
// Create a filesystem rooted at /home/user/public// The application cannot access anything above this directory.jailedFs:=afero.NewBasePathFs(osFs, "/home/user/public")
// To the application, this is reading "/"// In reality, it's reading "/home/user/public/"dirInfo, err:=afero.ReadDir(jailedFs, "/")
// Attempts to access parent directories fail_, err=jailedFs.Open("../secrets.txt") // Returns an error

Real-World Use Cases

Build Cloud-Agnostic Applications

Write applications that seamlessly work with different storage backends:

typeDocumentProcessorstruct {
fs afero.Fs
}
funcNewDocumentProcessor(fs afero.Fs) *DocumentProcessor {
return&DocumentProcessor{fs: fs}
}
func (p*DocumentProcessor) Process(inputPath, outputPathstring) error {
// This code works whether fs is local disk, cloud storage, or memorycontent, err:=afero.ReadFile(p.fs, inputPath)
iferr!=nil {
returnerr
}
processed:=processContent(content)
returnafero.WriteFile(p.fs, outputPath, processed, 0644)
}
// Use with local filesystemprocessor:=NewDocumentProcessor(afero.NewOsFs())
// Use with Google Cloud Storageprocessor:=NewDocumentProcessor(gcsFS)
// Use with in-memory filesystem for testingprocessor:=NewDocumentProcessor(afero.NewMemMapFs())

Treating Archives as Filesystems

Read files directly from .zip or .tar archives without unpacking them to disk first.

import (
"archive/zip""github.com/spf13/afero/zipfs"
)
// Assume 'zipReader' is a *zip.Reader initialized from a file or memoryvarzipReader*zip.Reader// Create a read-only ZipFsarchiveFS:=zipfs.New(zipReader)
// Read a file from within the archive using the standard Afero APIcontent, err:=afero.ReadFile(archiveFS, "/docs/readme.md")

Serving Any Filesystem over HTTP

Use HttpFs to expose any Afero filesystem—even one created dynamically in memory—through a standard Go web server.

import (
"net/http""github.com/spf13/afero"
)
funcmain() {
memFS:=afero.NewMemMapFs()
afero.WriteFile(memFS, "index.html", []byte("<h1>Hello from Memory!</h1>"), 0644)
// Wrap the memory filesystem to make it compatible with http.FileServer.httpFS:=afero.NewHttpFs(memFS)
http.Handle("/", http.FileServer(httpFS.Dir("/")))
http.ListenAndServe(":8080", nil)
}

Testing Made Simple

One of Afero's greatest strengths is making filesystem-dependent code easily testable:

funcSaveUserData(fs afero.Fs, userIDstring, data []byte) error {
filename:=fmt.Sprintf("users/%s.json", userID)
returnafero.WriteFile(fs, filename, data, 0644)
}
funcTestSaveUserData(t*testing.T) {
// Create a clean, fast, in-memory filesystem for testingtestFS:=afero.NewMemMapFs()
userData:= []byte(`{"name": "John", "email": "john@example.com"}`)
err:=SaveUserData(testFS, "123", userData)
iferr!=nil {
t.Fatalf("SaveUserData failed: %v", err)
}
// Verify the file was saved correctlysaved, err:=afero.ReadFile(testFS, "users/123.json")
iferr!=nil {
t.Fatalf("Failed to read saved file: %v", err)
}
ifstring(saved) !=string(userData) {
t.Errorf("Data mismatch: got %s, want %s", saved, userData)
}
}

Benefits of testing with Afero:

• ⚡ Fast - No disk I/O, tests run in memory
• 🔄 Reliable - Each test starts with a clean slate
• 🧹 No cleanup - Memory is automatically freed
• 🔒 Safe - Can't accidentally modify real files
• 🏃 Parallel - Tests can run concurrently without conflicts

Backend Reference

Type	Backend	Constructor	Description	Status
Core	OsFs	afero.NewOsFs()	Interacts with the real operating system filesystem. Use in production.	✅ Official
	MemMapFs	afero.NewMemMapFs()	A fast, atomic, concurrent-safe, in-memory filesystem. Ideal for testing.	✅ Official
Composition	CopyOnWriteFs	afero.NewCopyOnWriteFs(base, overlay)	A read-only base with a writable overlay. Ideal for sandboxing.	✅ Official
	CacheOnReadFs	afero.NewCacheOnReadFs(base, cache, ttl)	Lazily caches files from a slow base into a fast layer on first read.	✅ Official
	BasePathFs	afero.NewBasePathFs(source, path)	Restricts operations to a subdirectory (chroot/jail).	✅ Official
	ReadOnlyFs	afero.NewReadOnlyFs(source)	Provides a read-only view, preventing any modifications.	✅ Official
	RegexpFs	afero.NewRegexpFs(source, regexp)	Filters a filesystem, only showing files that match a regex.	✅ Official
Utility	HttpFs	afero.NewHttpFs(source)	Wraps any Afero filesystem to be served via http.FileServer.	✅ Official
Archives	ZipFs	zipfs.New(zipReader)	Read-only access to files within a ZIP archive.	✅ Official
	TarFs	tarfs.New(tarReader)	Read-only access to files within a TAR archive.	✅ Official
Network	GcsFs	gcsfs.NewGcsFs(...)	Google Cloud Storage backend.	⚡ Experimental
	SftpFs	sftpfs.New(...)	SFTP backend.	⚡ Experimental
3rd Party Cloud	S3Fs	fclairamb/afero-s3	Production-ready S3 backend built on official AWS SDK.	🔹 3rd Party
	MinioFs	cpyun/afero-minio	MinIO object storage backend with S3 compatibility.	🔹 3rd Party
	DriveFs	fclairamb/afero-gdrive	Google Drive backend with streaming support.	🔹 3rd Party
	DropboxFs	fclairamb/afero-dropbox	Dropbox backend with streaming support.	🔹 3rd Party
3rd Party Specialized	GitFs	tobiash/go-gitfs	Git repository filesystem (read-only, Afero compatible).	🔹 3rd Party
	DockerFs	unmango/aferox	Docker container filesystem access.	🔹 3rd Party
	GitHubFs	unmango/aferox	GitHub repository and releases filesystem.	🔹 3rd Party
	FilterFs	unmango/aferox	Filesystem filtering with predicates.	🔹 3rd Party
	IgnoreFs	unmango/aferox	.gitignore-aware filtering filesystem.	🔹 3rd Party
	FUSEFs	JakWai01/sile-fystem	Generic FUSE implementation using any Afero backend.	🔹 3rd Party

Afero vs. io/fs (Go 1.16+)

Go 1.16 introduced the io/fs package, which provides a standard abstraction for read-only filesystems.

Afero complements io/fs by focusing on different needs:

• Use io/fs when: You only need to read files and want to conform strictly to the standard library interfaces.
• Use Afero when:
  • Your application needs to create, write, modify, or delete files.
  • You need to test complex read/write interactions (e.g., renaming, concurrent writes).
  • You need advanced compositional features (Copy-on-Write, Caching, etc.).

Afero is fully compatible with io/fs. You can wrap any Afero filesystem to satisfy the fs.FS interface using afero.NewIOFS:

import"io/fs"// Create an Afero filesystem (writable)varmyAferoFs afero.Fs=afero.NewMemMapFs()
// Convert it to a standard library fs.FS (read-only view)varmyIoFs fs.FS=afero.NewIOFS(myAferoFs)

Third-Party Backends & Ecosystem

The Afero community has developed numerous backends and tools that extend the library's capabilities. Below are curated, well-maintained options organized by maturity and reliability.

Featured Community Backends

These are mature, reliable backends that we can confidently recommend for production use:

Amazon S3 - fclairamb/afero-s3

Production-ready S3 backend built on the official AWS SDK for Go.

import"github.com/fclairamb/afero-s3"s3fs:=s3.NewFs(bucket, session)

MinIO - cpyun/afero-minio

MinIO object storage backend providing S3-compatible object storage with deduplication and optimization features.

import"github.com/cpyun/afero-minio"minioFs:=miniofs.NewMinioFs(ctx, "minio://endpoint/bucket")

Community & Specialized Backends

Cloud Storage

• Google Drive - fclairamb/afero-gdrive
  Streaming support; no write-seeking or POSIX permissions; no files listing cache

• Dropbox - fclairamb/afero-dropbox
  Streaming support; no write-seeking or POSIX permissions

Version Control Systems

• Git Repositories - tobiash/go-gitfs
  Read-only filesystem abstraction for Git repositories. Works with bare repositories and provides filesystem view of any git reference. Uses go-git for repository access.

Container and Remote Systems

• Docker Containers - unmango/aferox
  Access Docker container filesystems as if they were local filesystems

• GitHub API - unmango/aferox
  Turn GitHub repositories, releases, and assets into browsable filesystems

FUSE Integration

• Generic FUSE - JakWai01/sile-fystem
  Mount any Afero filesystem as a FUSE filesystem, allowing any Afero backend to be used as a real mounted filesystem

Specialized Filesystems

• FAT32 Support - aligator/GoFAT
  Pure Go FAT filesystem implementation (currently read-only)

Interface Adapters & Utilities

Cross-Interface Compatibility:

• jfontan/go-billy-desfacer - Adapter between Afero and go-billy interfaces (for go-git compatibility)
• Maldris/go-billy-afero - Alternative wrapper for using Afero with go-billy
• c4milo/afero2billy - Another Afero to billy filesystem adapter

Working Directory Management:

• carolynvs/aferox - Working directory-aware filesystem wrapper

Advanced Filtering:

• unmango/aferox includes multiple specialized filesystems:
  • FilterFs - Predicate-based file filtering
  • IgnoreFs - .gitignore-aware filtering
  • WriterFs - Dump writes to io.Writer for debugging

Developer Tools & Utilities

nhatthm Utility Suite - Essential tools for Afero development:

• nhatthm/aferocopy - Copy files between any Afero filesystems
• nhatthm/aferomock - Mocking toolkit for testing
• nhatthm/aferoassert - Assertion helpers for filesystem testing

Ecosystem Showcase

Windows Virtual Drives - balazsgrill/potatodrive
Mount any Afero filesystem as a Windows drive letter. Brilliant demonstration of Afero's power!

Modern Asset Embedding (Go 1.16+)

Instead of third-party tools, use Go's native //go:embed with Afero:

import (
"embed""github.com/spf13/afero"
)
//go:embed assets/*varassetsFS embed.FSfuncmain() {
// Convert embedded files to Afero filesystemfs:=afero.FromIOFS(assetsFS)
// Use like any other Afero filesystemcontent, _:=afero.ReadFile(fs, "assets/config.json")
}

Contributing

We welcome contributions! The project is mature, but we are actively looking for contributors to help implement and stabilize network/cloud backends.

• 🔥 Microsoft Azure Blob Storage
• 🔒 Modern Encryption Backend - Built on secure, contemporary crypto (not legacy EncFS)
• 🐙 Canonical go-git Adapter - Unified solution for Git integration
• 📡 SSH/SCP Backend - Secure remote file operations
• Stabilization of existing experimental backends (GCS, SFTP)

To contribute:

1. Fork the repository
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

📄 License

Afero is released under the Apache 2.0 license. See LICENSE.txt for details.

🔗 Additional Resources

• 📖 Full API Documentation
• 🎯 Examples Repository
• 📋 Release Notes
• ❓ GitHub Discussions

---

Afero comes from the Latin roots Ad-Facere, meaning "to make" or "to do" - fitting for a library that empowers you to make and do amazing things with filesystems.