The concentrated inversions are pdfimages, not our decoding — and 144 of the 173 are this repository's size matcher

[tannevaled]

[2026-08-30 22:31:42 CEST]

The baseline's finding 3 said the inversions are concentrated and that two of
the three concentrations are "not yet explained": all 173 (samples)
inversions in fr-cerfa and nowhere else, and 28 of us-opm's 29 masks. I
chased both to a cause and measured which side is wrong.

Our pictures are right. pdfimages is the odd one, in two separate ways —
and most of the 173 are not inversions at all, they are this repository's
matcher pairing pictures that have nothing to do with each other. Nothing in
go-pdfkit/render or the codecs needs changing; what needs changing is here.

Everything below is a count over the corpus or a reading of a document I built
for the purpose. Nothing is inferred from what would explain it.

Conditions

Same as the baseline: render v0.19.0, reader v0.6.0, gfx v0.16.0,
pdfimages/pdftoppm from poppler 26.04.0, page 1 only, corpus
/Users/Shared/pdfforms. The 173 and the 28 both reproduce exactly.

1. 144 of the 173 are the size matcher, not a decoder

The 173 live in 8 documents of the 237 in fr-cerfa that draw anything on
page 1. 168 of the 173 are 2x2 pixels, 4 are 4x1 and 1 is 1x1. Not one is
larger than four pixels.

These documents set coloured text by stretching a tiny solid-colour image under
a large /SMask that carries the glyph shapes. Page 1 of cerfa_10074.pdf
draws 211 distinct 2x2 images; pdfimages -list reports the same 211
objects. All 211 have the same two stream bytes, 00 00 — every sample is
palette index 0 — so every one of them is a uniform four-pixel swatch, all
ink or all paper, and nothing else.

images.match pairs a picture of ours with the first unclaimed picture of
theirs of the same size. With 211 same-size uniform swatches on one page and
the two sides walking them in different orders, a black swatch of ours is
routinely paired with a white swatch of theirs. difference then reads 1.0 and
Judge writes Inverted: true. On cerfa_10074.pdf that produces 84
"inversions".

Matched by object identity instead — resolving each of our resource names to
its object number and pairing it with the pdfimages -list row of the same
object — the same 3449 pictures of the same 8 documents come out:

	this repository (size and order)	matched by object identity
agreeing	2906	3414
exact complements	173	29
otherwise differing	370	6

cerfa_10074.pdf alone goes from 84 reported inversions to 1.

So 144 of the 173 are an artefact of the pairing, and the matcher is also
wrong about 508 pictures in total on these 8 documents. The doc comment on
match already says the guess "is right as often as it is wrong and is
reported either way" — it is not reported either way, and this is what it
costs.

2. The 29 that survive: pdfimages writes an /Indexed palette as black only when it is exactly black

29 of 29 of the surviving complements are the same shape:

• 29 of 29 are 2x2, /BitsPerComponent 1, /ColorSpace [/Indexed /DeviceRGB 1 <...>]
• 29 of 29 carry no /Decode array
• 29 of 29 have every sample bit zero, so every pixel is palette entry 0
• palette entry 0 is one of five saturated dark colours: #174A9A (4),
  #174A9B (10), #1B2B4E (7), #213A7E (4), #333333 (4)

A 661-byte document with one such image and nothing else, swept over palette
entry 0 from #000000 to #FFFFFF, says what pdfimages does:

palette entry 0	pdfimages writes
gray 0	black
gray 1, 2, 4, 8, 16, 32, 64, 96, 127, 128, 129, 160, 200, 254, 255	white

It is not a luminance threshold. pdfimages writes the pixel black only when
the palette entry is exactly (0,0,0); #111111 and #333333 come out as
paper. Poppler's own renderer does not agree with it: pdftoppm -cropbox -aa no paints the page (33, 58, 126) for the #213A7E case, and go-pdfkit reads
that picture as ink. Page 1 of cerfa_12946.pdf has 74 116 pixels of that
navy on it — the second most common colour on the page after white.

3. us-opm: pdfimages writes every stencil with the opposite polarity

The 28 are real: matched by object identity, 28 of 28 are exact
complements, none agree and none merely differ. All 28 are the same shape:
/ImageMask true, /BitsPerComponent 1, no /Decode array (27
/CCITTFaxDecode, 1 /FlateDecode).

A 644-byte document settles which side is wrong. Its 8x8 mask has every row
0xF0 — the four left bits 1, the four right bits 0 — and PDF 32000-1 8.9.6.2
says a sample of 0 marks the area that shall be painted, so the right half
must be black:

	left half	right half
pdftoppm (what a person sees)	white	black
pdfimages	black	white
go-pdfkit	white	black

The filter is not the cause — this document has no filter at all — and neither
is /Decode, which it does not carry.

What decides it is what kind of mask it is, and that holds across the corpus
with no exceptions:

population	/ImageMask stencils	/Mask stencils	/SMask soft masks
fr-cerfa	13 of 13 inverted	2 of 2 inverted	0 of 733 inverted
us-opm	28 of 28 inverted	—	0 of 1 inverted

4. Two published claims here are false

baseline/README.md and the doc comment on images.Result.Inverted both say:

pdfimages writes a JBIG2 mask's bitmap with the opposite polarity when that
stream is a soft mask; that is a convention, not a disagreement.

Both halves are wrong. It is not about JBIG2 — 27 of the 28 us-opm cases
are CCITT, one is Flate, and the minimal document has no filter. And it is
not soft masks — soft masks are the one kind poppler does not invert, 0
of 734 of them across these two populations. The behaviour is: pdfimages
writes a stencil with the opposite polarity to the samples, whatever filter
it arrived in.

That claim is exactly the kind that was published and disproved once before on
this subject, over /Decode arrays. It should not be left standing.

What to do

1. Correct the two false claims. Small, and I have a PR for it.
2. Do not pair pictures by size when the size is ambiguous. When more than
   one of theirs has the size of one of ours, the pairing is a guess and the
   result is not evidence, so it belongs in a column of its own beside
   remapped and unmatched rather than in exact or inverted. This
   repository already refuses to count what could not be compared; this is the
   same rule applied to a case it currently lets through. It moves every number
   in the baseline, so it is not a change to slip in beside a documentation
   fix.
3. Consider whether inverted should be a column at all, now that no
   instance of it in this corpus is a disagreement about a decoded picture. In
   both mechanisms above the picture we produce is the one on the page.

Reproduction

Both minimal documents, and the three readings of each, come out of one script
that needs only poppler and Pillow. It is at
/Users/Shared/pdfneg/repro/polarity.py on the machine this was measured on,
and it writes:

• stencil.pdf, 644 bytes, sha256 2549fd4821fa9f9cc88ba39e955b15ada3d3d7b7db4b98b8f1c1c88a0c6bb73f
• indexed.pdf, 661 bytes, sha256 811903933236b85d30e2f2e843f1e970fcb3791b373a2ddcfead61ff4b84dc60

Smallest corpus documents that carry each defect:

	path	bytes	sha256
indexed swatch	/Users/Shared/pdfforms/fr-cerfa/cerfa_12946.pdf	212 649	1fbbac90ba93135c3460b8daf4f8b5418b4c7efe4d86f6f06fa3c5592379cc4b
stencil polarity	/Users/Shared/pdfforms/us-opm/sf3103.pdf	95 325	589e6e2ec85cd162c3df71e7ebe18bf9062d509daf5a1e02f544091502e23d59

What is not established

• I did not check the other two inversion concentrations the baseline names —
  JBIG2Decode mask (199, mostly ia-medical) and (samples) mask in
  ia-uscourts (34 of 51). Finding 3 predicts both are the same stencil
  polarity, and that prediction is untested here.
• The 6 pictures that still differ after identity matching are not explained.
  They are 390x133 and similar, not the tiny swatches, and they are a separate
  question.
• Whether poppler considers either behaviour a defect is not something I
  measured. Both are its extractor disagreeing with its own renderer, which is
  a reason to stop treating pdfimages as the last word on a picture's
  polarity, not a claim about poppler's intent.

[tannevaled]

[2026-08-30 22:47 CEST] Evidence for point 2 (do not pair pictures by size when the size is ambiguous), found while auditing my own tail in #14 rather than looking for it.

The matcher does not only manufacture inversions. It manufactures the tail of the lossy distribution too, and on this corpus it reaches far past the 211 swatches named above.

Auditing fr-cerfa's 105 differing DCTDecode pictures against what pdfimages -list says each page holds:

	pictures	≤ 0.0087	above 1%
pairing unambiguous	95	94	0.094990
pairing ambiguous	10	6	0.250000, 0.375000, 0.571429, 0.750000

The four largest shares in that column — including the 0.750000 that baseline/ records as the population's worst DCTDecode disagreement — are 7×1 and 8×1 slivers on page 1 of cerfa_12626.pdf. On that page pdfimages extracts 2122 pictures of size 8×1 and 81 of size 7×1, so match pairs them at random among thousands of identical sizes, and the shares are two, three, four and six differing pixels out of eight.

The only picture in the population above 1% whose pairing is unambiguous is cerfa_12711.pdf's single 418×357 Im0, at 0.094990. fr-cerfa's worst DCT disagreement is 0.095, not 0.750.

Two things this adds:

1. The ambiguity is not confined to (samples) or to tiny uniform swatches — it reaches DCTDecode, and 2122 candidates of one size is an order of magnitude past the 211 above. A page of 1×N slivers is a much commoner shape than a page of colour swatches.
2. It moves a published worst-case, not only an inversion count. Anyone reading baseline/ for the tail of the lossy filters is reading the matcher in at least this one place.

Measured at render v0.20.0 with the name-collapsing removed (#14), poppler 26.04.0, page 1, one population — fr-cerfa, 450 documents. It is one population and I have not audited the other nineteen.

[tannevaled]

Confirmed from poppler's own source, and it is not a PBM convention

[2026-08-30 23:10:14 CEST]

The claim above was published on behaviour alone. I have now read the code that
produces the behaviour. The claim stands, and the "we misread PBM's 1 = black"
escape route is closed: the same polarity comes out of -png and -tiff,
which are luminance formats with the opposite convention to PBM.

Source read: poppler/poppler at gitlab.freedesktop.org, tag
poppler-26.04.0 — the exact version the corpus runs — cross-checked against
master (e8631df8, 2026-08-29), where the lines are unchanged but shifted by
three.

1. The mechanism: two inversions, and for a stencil they cancel

utils/ImageOutputDev.cc:427-437 (26.04.0), inside writeImageFile:

    // PDF masks use 0 = draw current color, 1 = leave unchanged.
    // We invert this to provide the standard interpretation of alpha
    // (0 = transparent, 1 = opaque). If the colorMap already inverts
    // the mask we leave the data unchanged.
    invert_bits = 0xff;
    if (colorMap) {
        memset(zero, 0, sizeof(zero));
        colorMap->getGray(zero, &gray);
        if (colToByte(gray) == 0) {
            invert_bits = 0x00;
        }
    }

applied at utils/ImageOutputDev.cc:529:

                row[x] = str->getChar() ^ invert_bits;

A stencil reaches this with colorMap == nullptr — drawImageMask passes a
null colour map at utils/ImageOutputDev.cc:715-723:

void ImageOutputDev::drawImageMask(GfxState *state, Object *ref, Stream *str, int width, int height, bool /*invert*/, bool interpolate, bool inlineImg)
...
        writeImage(state, ref, str, width, height, nullptr, inlineImg);

so the if (colorMap) guard never fires and invert_bits stays 0xff. The row
handed to the writer is therefore the alpha of the stencil: 1 = opaque.

Then goo/NetPBMWriter.cc:66-70 inverts it a second time, on the stated
assumption that the row is luminance, not alpha:

        // PBM uses 0 = white, 1 = black so we need to invert the colors
        int size = (width + 7) / 8;
        for (int i = 0; i < size; i++) {
            fputc((*row)[i] ^ 0xff, file);
        }

Two ^ 0xff cancel. The PBM body is the PDF sample bytes verbatim, and PBM
reads a 1 as black — so a sample of 1, which PDF 32000-1 §8.9.6.2 says leaves
the page unchanged, becomes ink.

That is measurable on the minimal document. pdfimages 26.04.0, rows 0xF0:

$ xxd out/pbm-000.pbm
00000000: 5034 0a38 2038 0af0 f0f0 f0f0 f0f0 f0    P4.8 8.........

Every body byte is f0, the stream byte unchanged.

Two comments in one pipeline contradict each other: one says the row is alpha,
the next says the row is colour. Whichever is right, they cannot both be.

2. The PBM convention is not the explanation

If the double inversion were the whole story, -png and -tiff — which are
written once, from the post-invert_bits row, and whose 1-bit forms are
luminance with 1 = white (PNG_COLOR_TYPE_GRAY at goo/PNGWriter.cc:145-148,
PHOTOMETRIC_MINISBLACK at goo/TiffWriter.cc:132-136) — would come out the
other way round. They do not. Same document, same poppler:

output	first row, left→right
pdfimages (default, PBM)	black black black black · white white white white
pdfimages -png	0 0 0 0 255 255 255 255 — black left, white right
pdfimages -tiff	0 0 0 0 255 255 255 255 — black left, white right
pdftoppm -gray -aa no	ff ff ff ff 00 00 00 00 — white left, black right

All three extraction formats agree with each other and disagree with the
renderer. A format convention cannot explain a result that is identical across
formats of opposite convention. The comparison in this issue did not misread
PBM.

3. Why pdftoppm differs: it honours /Decode, pdfimages discards it

poppler/Gfx.cc:4321-4328 reads the array and sets invert:

            // Table 4.39 says /Decode must be [1 0] or [0 1]. Adobe
            // accepts [1.0 0.0] as well.
            if (obj2.isNum() && obj2.getNum() >= 0.9) {
                invert = true;
            }

SplashOutputDev — the device behind pdftoppm — uses it
(poppler/SplashOutputDev.cc:2607, then :2574):

    imgMaskData.invert = !invert;
...
        *q++ = *p++ ^ imgMaskData->invert;

With no /Decode, invert == false, so the source byte is XORed with 1 and a
sample of 0 becomes the painted pixel. That is §8.9.6.2 exactly.

ImageOutputDev::drawImageMask names the same parameter bool /*invert*/ and
throws it away. So pdfimages does not merely invert; it is blind to
/Decode on a stencil. Adding /Decode [1 0] to the minimal document produces
a byte-identical PBM (f0 f0 …) while pdftoppm flips to
00 00 00 00 ff ff ff ff. The same discard appears at
utils/ImageOutputDev.cc:733 for /Mask (bool /*maskInvert*/), which is why
the two /Mask stencils in fr-cerfa behave the same way.

/SMask escapes because drawSoftMaskedImage (:744) passes the mask's real
maskColorMap, so the soft mask never enters the colorMap == nullptr branch —
0 of 734 inverted, as measured.

4. Deliberate?

Half. The comment at :428 is an explicit intent — write alpha, 1 = opaque —
added by Adrian Johnson in 8f466775 (2013-08-18, "Change PNGWriter
monochrome format to be 8 pixels/byte"), replacing a bare description of the
mechanism from 8670bfed (Vasile Gaburici, 2008-08-30, "Fix extraction of
images", which cited freedesktop bug 16999 and only ever touched the
colorMap path). Nothing downstream honours that intent: the file is declared
as PBM, as PNG_COLOR_TYPE_GRAY, or as PHOTOMETRIC_MINISBLACK — three
luminance declarations — and every viewer renders it as luminance. pdfimages.1
documents stencil as "a monochrome mask image used for painting a color or
pattern" and says nothing about polarity or about alpha.

The behaviour predates the intent. Before the 2013 refactor e53aec2c,
writeMask wrote the stream straight into the PBM
(fputc(str->getChar(), f)), which is the same visible result. This has been
poppler's output for the whole of its recorded history; the refactor preserved
it.

5. The same function explains finding 2, in one line

colToByte(gray) == 0 at :435 is an exact-black test, and it is applied to
every 1-component 1-bit image, not only to masks. Measured on three
one-image documents that differ only in palette entry 0:

palette entry 0	PBM body	shown as
#000000	ff ff	black
#213A7E	00 00	white
#333333	00 00	white

getGray of #213A7E is ≈58, not 0, so invert_bits stays 0xff, the second
inversion cancels it, and the darkest entry in the palette is written as paper.
The correct test would compare entry 0 against entry 1, not against black. The
29 /Indexed findings need no re-examination; this is their cause.

6. Upstream

Not reported for stencils. Searching the poppler tracker (project 882) for
invert, inverted, polarity, stencil, pbm, ImageMask returns one
live candidate:

• poppler#1452,
  "pdfimages -png and -tiff give inverted colour output", opened 2023-12-30.
  user_notes_count = 0, updated_at == created_at: no reply in the two years and eight months since. It reports the symptom on a scanned page and does not name
  /ImageMask, /Decode, or a mechanism.

The historical Launchpad bug 134313, "pdfimages produces inverted image for
black & white image" (Fix Released against poppler), is a CCITT G4 black-and-white
image — the colorMap path — addressed by 8670bfed in 2008. The stencil path
was never touched by it.

Worth reporting upstream, and narrowly: not "output is inverted" but the two
concrete defects above — drawImageMask discarding invert, and the exact-black
test at :435 — each with a minimal document. #1452 shows that a symptom report
alone gets nowhere.

7. Other implementations read §8.9.6.2 the way we do

pdf.js (9785821, 2026-08-30). src/core/image.js:388-389 sets
const inverseDecode = decode?.[0] > 0; — false when /Decode is absent — and
src/shared/image_utils.js:39-41 maps it:
const [zeroMapping, oneMapping] = inverseDecode ? [nonBlackColor, black] : [black, nonBlackColor];
Default decode, sample 0 → black, i.e. painted. Agrees.

pdfium (6f2272e1, 2026-08-28). core/fpdfapi/page/cpdf_dib.cpp:353:
default_decode_ = !pDecode || !pDecode->GetIntegerAt(0); and :1172-1178:
if (image_mask_ && default_decode_) { … dest = ~src; } — under the default
decode the bits are complemented so a sample of 0 becomes a set coverage bit.
Agrees.

MuPDF (5fe54cef, 2026-08-24). source/fitz/image.c:704-712 states the
rule in a comment and implements it:

		/* Invert 1-bit image masks */
		if (image->imagemask)
		{
			/* 0=opaque and 1=transparent so we need to invert */

Agrees.

Three independent implementations, and poppler's own renderer, all treat a
stencil sample of 0 as the painted one under the default /Decode. Only
poppler's extraction device does otherwise.

What this changes here

Nothing in the plan above. Finding 3 stands as written and now has a citation
rather than a measurement; finding 2 has a one-line cause. The proposals — fix
the two false claims, stop pairing by size when the size is ambiguous, and
reconsider whether inverted is a column at all — are unaffected.

Sources read, cloned read-only into /Users/Shared/biblio/, outside every work
tree: poppler poppler-26.04.0 and e8631df8, pdf.js 9785821, pdfium
6f2272e1, MuPDF 5fe54cef.

[tannevaled]

Bibliography for findings 2 and 3, read in /Users/Shared/biblio/poppler. They are not two quirks. They are one code path, and it is deliberate.

utils/ImageOutputDev.cc:429-441, poppler's own comment first:

// PDF masks use 0 = draw current color, 1 = leave unchanged.
// We invert this to provide the standard interpretation of alpha
// (0 = transparent, 1 = opaque). If the colorMap already inverts
// the mask we leave the data unchanged.
invert_bits = 0xff;
if (colorMap) {
    memset(zero, 0, sizeof(zero));
    colorMap->getGray(zero, &gray);
    if (colToByte(gray) == 0) {
        invert_bits = 0x00;
    }
}

applied at :529-533 as row[x] = str->getChar() ^ invert_bits.

Finding 3 (us-opm, every stencil inverted). A stencil reaches writeImage with colorMap == nullptr, so invert_bits stays 0xff and every byte is flipped. Unconditional, by design: pdfimages is normalising to alpha polarity, not decoding to paint. drawImageMask is declared bool /*invert*/ at :718 and drawMaskedImage bool /*maskInvert*/ at :742 — the flag the PDF carries is not merely unused, it is commented out of the signature. So our side agreeing with pdftoppm and with PDF 32000-1 8.9.6.2 is not a disagreement with poppler at all; the two tools answer different questions.

Finding 2 (the /Indexed palette). :642 sends any 1-component, 1-bit image to PNGWriter::MONOCHROME — an /Indexed picture included, whatever its palette holds. It is then subject to the SAME inversion, exempted only when palette entry 0 is already black. So the measured table is exactly colToByte(gray) == 0, and the reason it is not a luminance threshold is GfxState.h:149:

// 255 * x + 0.5  =  256 * x - x + 0x8000
return static_cast<unsigned char>(((x << 8) - x + 0x8000) >> 16);

It rounds, so the test passes only below 1/510. #111111 is grey 0.0667, colToByte 17, inverted. That is why #000000 came out black and gray 1 white in the sweep, with nothing in between.

What this changes. The empirical finding said pdfimages "writes the pixel black only when the palette entry is exactly (0,0,0)". That is true and it is a consequence, not the rule. The rule is: a 1-bit picture is treated as a mask and normalised to alpha polarity, and the only exemption is a palette that already reads as ink. Same eleven lines for both. Anyone reporting this upstream should report it as one thing.

Finding 1 is fixed and merged as #28 — 68 of fr-cerfa's inversions were the size matcher, and the 15 that survive are this.

🤖 Generated with Claude Code