PixelSieve

A darkroom for datasets. You feed it a tray of image-like assets; it develops each one under the safelight, reads the grain, and sifts the batch into three prints: keep, review, and drop.

PixelSieve is a rapid, dependency-light multimodal dataset curation engine. It pairs a pure-standard-library Python CLI that reasons over image metadata and captions with a zero-dependency Zig companion that reads binary PPM/PGM files and emits signatures. Both tools speak one plain-text record format, so their output stacks together like negatives in a contact sheet.

There is no pixel decoding in the hot path, no NumPy, no Pillow, no model weights. That is deliberate. Curation at the manifest level — dimensions, byte hashes, caption tokens, coarse sample signatures — is where most of the cheap wins live, and it runs anywhere Python 3.8 runs.

---

The darkroom metaphor, briefly

Curating a scraped or synthesized image corpus feels a lot like standing over three developing trays in red light:

• The developer tray — everything arrives here. Some frames are crisp, some are blank, some are the same shot exposed twice.
• The sieve — a stack of fine meshes. Big grit (broken dimensions, empty captions) is caught immediately. Finer meshes catch the near-duplicates that only differ by a few grains.
• The three prints — what survives is sorted onto the shelf: the keepers, the maybes that want a second human look, and the rejects.

PixelSieve is that sieve, mechanized. The vocabulary in the code — develop, grain, sift, safelight — is not decoration; it maps onto real stages.

---

What it actually does

Given a mix of manifest files and directories, the engine:

1. Ingests assets from native .psman records, .jsonl/.json manifests, or by scanning loose binary PPM/PGM files (parsing their headers directly).
2. Fingerprints each asset with an exact SHA-256 byte hash and a portable psig1 perceptual-style sample signature.
3. Judges each asset against dimension, aspect, file-size, and caption-token rules, producing a verdict and a trail of reasons.
4. Clusters literal duplicates (same byte hash) and near-duplicates (signatures within a Hamming threshold), keeping one representative per group.
5. Emits three manifests — keep.psman, review.psman, drop.psman — each carrying both the asset metadata and the verdict line so the decision is fully auditable.

---

Install & run

The Python engine needs nothing beyond a standard interpreter.

# from the project root, no install required:export PYTHONPATH=src
python -m pixelsieve version
# or install the console script:
pip install .
pixelsieve version

Sieve the bundled samples

python -m pixelsieve sieve samples -o out

pixelsieve: 6 assets -> keep=4 review=0 drop=2 across 2 clusters
pixelsieve: manifests written to out

Peek at why something was dropped:

$ cat out/drop.psman
# PSV1 manifest emitted by pixelsieve sieve
PSV1	asset	id=gradient_copy	...	sig=psig1:02802a82aafaebff	cap=a smooth diagonal color gradient reference tile
PSV1	verdict	id=gradient_copy	verdict=drop	cluster=exact-0000	why=keep:clean;drop:exact-dup-of(gradient);review:near-dup-of(gradient)
PSV1	asset	id=tiny	...	w=16	h=16	...
PSV1	verdict	id=tiny	verdict=drop	cluster=	why=drop:below-min-dim(16x16);drop:below-min-pixels(256);drop:caption-too-short(1)

gradient_copy is a byte-for-byte duplicate of gradient, so it is dropped in favor of the representative. tiny fails three separate meshes at once.

Other subcommands

python -m pixelsieve inspect samples # dump parsed assets as PSV1 records
python -m pixelsieve sig samples/rings.pgm # print a psig1 signature
python -m pixelsieve sieve samples/remote_manifest.jsonl -o out --no-signatures

Tunable knobs on sieve:

Flag	Default	Meaning
--min-width / --min-height	64	Minimum accepted dimensions.
--max-width / --max-height	16384	Maximum accepted dimensions.
--min-tokens	2	Captions below this many tokens are dropped.
--sig-threshold	6	Max Hamming distance (of 64 bits) to call two assets near-duplicates.
--no-signatures	off	Skip psig1 computation (useful for metadata-only manifests).

---

The Zig companion

The Python engine never opens raw pixels. When you do have binary PPM/PGM files on disk and want their metadata and signatures computed at native speed, the pixelsieve-zig CLI handles it — with zero dependencies beyond the Zig standard library.

cd zig
zig build
./zig-out/bin/pixelsieve-zig meta ../samples/gradient.ppm ../samples/rings.pgm
./zig-out/bin/pixelsieve-zig sig ../samples/checker.ppm

It emits the exact same PSV1 records the Python side does:

PSV1	asset	id=gradient.ppm	path=../samples/gradient.ppm	w=96	h=96	fmt=ppm	bytes=27661	sig=psig1:02802a82aafaebff
PSV1	sig	path=../samples/checker.ppm	psig=psig1:b04066020701a809

Because the psig1 algorithm is integer-only in both languages (see docs/psig1.md), the signatures are byte-identical for the same input. You can pipe Zig asset records straight into a manifest and let Python's sieve render verdicts over them.

Build note: Zig is not installed in this project's local environment, so the binary is compiled and smoke-tested in CI (see .github/workflows/ci.yml). The Python engine is compiled and exercised locally against the bundled samples.

---

The shared line format (PSV1)

One tab-separated record per line, a PSV1 tag, a verb, then key=value fields with a tiny percent-escaping table. That is the whole contract, and it is why the two languages never need to negotiate a schema.

PSV1 <TAB> verdict <TAB> id=rings <TAB> verdict=keep <TAB> cluster= <TAB> why=keep:clean

Full grammar, escaping table, and the verb catalogue live in docs/format.md. The format is designed to be mergeable (concatenate outputs freely), auditable (a human reads the why), and portable (no JSON in Zig, no binary framing to keep in sync).

---

How the sieve decides

Verdicts combine with strict precedence: any drop wins over any review wins over keep. Every rule that fires leaves a reason tag, so a verdict line reads like a developing log.

Mesh	Signal	Example reason
Dimensions	drop below min / above max	drop:below-min-dim(16x16)
Pixel count	drop under min_pixels	drop:below-min-pixels(256)
Aspect ratio	review when extreme	review:extreme-aspect(0.09)
File size	drop for tiny files	drop:tiny-file(80B)
Caption tokens	drop/review when thin	drop:caption-too-short(1)
Exact hash	drop non-representative dupes	drop:exact-dup-of(gradient)
Signature	review near-duplicates	review:near-dup-of(gradient)

The caption pipeline also exposes a Jaccard agreement helper, so you can ask how well two captions overlap in token space — handy for spotting assets that share an image but disagree on their text.

---

Project layout

pixelsieve/
├── src/pixelsieve/
│ ├── lineformat.py # PSV1 read/write + escaping
│ ├── ingest.py # manifests, JSON/JSONL, PPM/PGM headers, scanning
│ ├── signatures.py # psig1 + hamming distance
│ ├── rules.py # dimension / aspect / caption quality meshes
│ ├── cluster.py # exact + signature (union-find) clustering
│ ├── engine.py # ingest -> rules -> cluster -> verdict pipeline
│ ├── cli.py # argparse front end
│ └── __main__.py # python -m pixelsieve
├── zig/
│ ├── build.zig # zig build -> pixelsieve-zig
│ └── src/
│ ├── lineformat.zig # PSV1 writer, mirrors the Python escaping table
│ ├── psig.zig # integer-only psig1, byte-identical to Python
│ ├── pnm.zig # P5/P6 header parser (comments tolerated)
│ └── main.zig # sig / meta / version subcommands
├── samples/ # real PPM/PGM images + captions + a JSONL manifest
├── docs/
│ ├── format.md # the PSV1 spec
│ ├── psig1.md # the signature algorithm + worked examples
│ ├── banner.svg # animated darkroom banner
│ └── pipeline.svg # animated sieve-flow diagram
├── pyproject.toml
├── Makefile
├── CHANGELOG.md
├── ROADMAP.md
└── LICENSE

---

The sample tray

samples/ ships real, valid files so every claim above is reproducible:

• gradient.ppm and gradient_copy.ppm — an exact-duplicate pair (same bytes, same hash, same psig1), used to demonstrate collapse.
• checker.ppm — a red/blue checkerboard.
• rings.pgm — concentric grayscale rings.
• stripes.pgm — a left-to-right ramp whose psig1 is a clean 00000000ffffffff.
• tiny.ppm — a 16×16 frame with a one-word caption, guaranteed to be sieved out on three separate meshes.
• remote_manifest.jsonl — a metadata-only manifest exercising the JSON path and the dimension/caption rules without any files on disk.

Each raster has a paired .txt caption, matched by filename stem exactly as the ingest layer expects.

---

A full pass, narrated

Here is what actually happens when you run python -m pixelsieve sieve samples over the bundled tray, mesh by mesh.

Loading the tray.ingest.collect walks samples/, finds six PPM/PGM files, and parses each header without decoding pixels. For every raster it looks for a sibling <stem>.txt caption and reads it. It also computes the SHA-256 byte hash while the file is open. Six Asset records drop into the developer tray.

Reading the grain.signatures.psig_from_file samples 64 evenly spaced bytes from each file, takes their integer mean, and packs a 64-bit signature. gradient.ppm and gradient_copy.ppm are byte-identical, so they produce the same psig1:02802a82aafaebff. stripes.pgm, a monotone ramp, produces the tidy psig1:00000000ffffffff — the first 32 samples sit below the mean, the last 32 above it.

The coarse mesh (rules).rules.evaluate runs each asset past the dimension, aspect, size, and caption checks. tiny.ppm (16×16, caption "tiny") trips three at once and is stamped drop. The other five pass clean.

The fine mesh (dedup).cluster.exact_clusters groups by byte hash and finds the gradient pair; the representative (gradient) is kept, the other is escalated to drop with reason drop:exact-dup-of(gradient). cluster.signature_clusters runs a union-find over Hamming distances and finds the same pair within threshold, leaving a review:near-dup-of breadcrumb on the already-dropped copy.

Three prints.engine.run returns ordered verdicts; the CLI splits them into keep.psman (checker, gradient, rings, stripes), an empty review.psman, and drop.psman (gradient_copy, tiny). Each print carries the asset line beside its verdict line, so the shelf is self-documenting.

The final tally printed to the terminal:

pixelsieve: 6 assets -> keep=4 review=0 drop=2 across 2 clusters

---

Why not just use an existing tool?

pixelsieve is intentionally narrow. It is worth being honest about where it sits relative to the alternatives.

• vs. Pillow / OpenCV pipelines — those decode pixels and are the right choice when you need real perceptual hashing, resizing, or format conversion. pixelsieve deliberately stops at the header and the byte stream, trading fidelity for zero dependencies and portability. Use it as the cheap first pass; reach for a decoder only on the survivors.
• vs. a pile of shell scripts — find, sha256sum, and jq can approximate the exact-dup pass, but they cannot cluster near-duplicates, cannot explain a verdict, and drift the moment two people write them. The PSV1 contract keeps the decision reproducible.
• vs. a database — for millions of rows you will want one. The roadmap's streaming milestone (M5) is the bridge; today's engine is tuned for the thousands-to-low-millions manifest that fits comfortably in memory.

If your problem is "I have a directory of scraped rasters and a pile of captions and I need a defensible keep/drop list by lunch," this is the tool.

---

Extending the sieve

The engine is a plain function pipeline, so extension is mostly a matter of adding a stage or a rule rather than subclassing anything.

• A new rule. Add a check inside rules.evaluate that appends a (level, reason) tuple. Precedence handles the rest — you never have to reason about ordering, only about which mesh your check belongs to.
• A new input format. Teach ingest.load_manifest a new extension branch that yields Asset records. Everything downstream is format-agnostic.
• A new fingerprint. Add a function to signatures.py and a matching verb to the line format. If you want Zig parity, mirror the integer math in zig/src/ exactly — no floating point, or the two tools will drift.
• A new output split. The CLI writes one manifest per verdict level; adding a fourth bucket is a loop entry, not a refactor.

Because every stage communicates through Asset and Verdict dataclasses and the PSV1 text format, you can also bolt on external stages in any language: read the manifest, do your work, write more PSV1 lines, concatenate.

---

FAQ

Is psig1 a real perceptual hash? No, and the docs are careful to call it "perceptual-style." It is a coarse byte-sample fingerprint. It catches exact and byte-near duplicates well; it will not recognize the same photo re-encoded at a different quality. That is a deliberate scope choice — the point is a portable, integer-only fingerprint two languages can agree on.

Why tab-separated instead of JSON? Because Zig ships JSON, but keeping two JSON serializers byte-identical is more fragile than keeping one five-character escaping table identical. Tabs also make the manifests trivially greppable and mergeable.

Does it modify my files? Never. pixelsieve reads inputs and writes manifests into the output directory you name. Deciding what to do with a dropped asset is left to you.

What Python versions? 3.8 and up, standard library only. CI runs 3.8, 3.11, and 3.12.

Where are the tests? There are none by design. Correctness is shown through the reproducible sample runs documented here and the CI smoke tests that compile the Zig binary and exercise both CLIs.

---

Design principles

• Boring on purpose. Plain text, integer math, standard library. Nothing here needs a lockfile to reproduce.
• Auditable over clever. Every drop is explained. A dataset you cannot explain is a dataset you cannot defend.
• Two languages, one contract. Python for reach and readability, Zig for the byte-level work, and a shared record format so neither owns the truth alone.
• Truthful milestones. The roadmap marks four milestones done because they are — each is exercised by the shipped samples or CI.

---

Development

make check # compile-check every Python module
make sieve # run the pipeline over samples/
make inspect # dump parsed assets
make zig # build the Zig companion (needs zig on PATH)
make clean # remove generated manifests and Zig caches

There are no unit tests in this repository by design; correctness is demonstrated through the reproducible sample runs and the CI smoke tests.

---

License

MIT. See LICENSE.

_{PixelSieve — develop · hash · sift · keep. Work under the safelight; ship under scrutiny.}