Make .quantize(palette=...) use exact colors when possible, and improve dithering quality

[akx]

Refs #1852 (fixes the reproducer bug: we now select exact colors when available, but we're still not necessarily selecting the mathematically nearest color when quantizing to a given palette).

• Refactors the topalette() conversion function to a separate file for ease of understanding
• Refactors the topalette() function to multiple functions for less nesting and ease of reading
• Improves internal documentation/constants around palettes
• Fixes a couple possible but implausible NULL pointer derefs (that could occur if there was less than ~1100 bytes of memory available for Python)
• The usual™️ hoist + restrict optimizations for internals
• Adds benchmarks for quantize-into-palette

... and then the main thing:

• Adds a small linear-probing hash table for finding exact matches in the palette before looking a close-enough one up in the cache "cube" in the palette. In other words, if a palette specifies a color, we will use it, guaranteed.
• ImagingPaletteCache* functions are made private, since the structure of the cache was undocumented and private anyway; we need to add a new flag in the cache entries to have a better guess of when we might have an exact match for a color.

EDIT: this now also includes the changes from #9338 as the last commit, since that touches the same files as this jiggles around.

[codspeed-hq]

Merging this PR will degrade performance by 7.15%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 10 regressed benchmarks
✅ 589 untouched benchmarks
⏩ 335 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
❌	test_quantize_to_palette[1237x811-exact-flower2-P-floyd-steinberg]	60.8 ms	77.9 ms	-21.9%
❌	test_quantize_to_palette[1237x811-exact-synthetic-P-floyd-steinberg]	60.6 ms	77.1 ms	-21.48%
❌	test_quantize_to_palette[1237x811-exact-flower2-P-none]	20.6 ms	26 ms	-20.82%
❌	test_quantize_to_palette[1237x811-exact-synthetic-P-none]	20.3 ms	25.4 ms	-20.31%
❌	test_quantize_to_palette[1237x811-grayscale-flower2-P-floyd-steinberg]	61.8 ms	71.7 ms	-13.74%
❌	test_quantize_to_palette[1237x811-grayscale-synthetic-P-floyd-steinberg]	64.5 ms	74.2 ms	-13.16%
❌	test_quantize_to_palette[1237x811-web-synthetic-P-floyd-steinberg]	66.5 ms	76.2 ms	-12.82%
❌	test_quantize_to_palette[1237x811-web-flower2-P-floyd-steinberg]	67 ms	76.8 ms	-12.7%
❌	test_quantize_to_palette[1237x811-exact-flower2-PA-floyd-steinberg]	73.8 ms	83 ms	-10.99%
❌	test_quantize_to_palette[1237x811-exact-synthetic-PA-floyd-steinberg]	73.6 ms	82.3 ms	-10.51%
⚡	test_quantize_to_palette[1237x811-web-flower2-PA-none]	31.1 ms	26 ms	+19.37%
⚡	test_quantize_to_palette[1237x811-grayscale-flower2-PA-none]	31.1 ms	26 ms	+19.32%
⚡	test_quantize_to_palette[1237x811-grayscale-synthetic-PA-none]	31.7 ms	26.7 ms	+18.92%
⚡	test_quantize_to_palette[1237x811-web-synthetic-PA-none]	31.7 ms	26.7 ms	+18.83%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing akx:palettize-fixes (22dfe86) with main (6b5a7db)}

Footnotes

1. 335 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[akx]

I think we'll have to accept some performance degradation in exchange for actually making conversions accurate...

[radarhere]

Suggested change

	/* The c array now contains the closest match for each
	* cache slot in the box. Update the cache. */
	/* The C array now contains the closest match for each
	* cache slot in the box. Update the cache. */

[akx]

This is also from 2006 and was just moved over to this file...