Ignore fully opaque alpha channels, preserve alpha when writing grayscale images

[illwieckz]

Ignore fully opaque alpha channels (when loaded by crn_mipmapped_texture.cpp).

This allows opaque images to use formats without alpha channels when converting to CRN, DDS, or KTX.

This matches the existing behavior when converting to PNG or TGA.

---

Preserve alpha when writing grayscale images (when processed in crn_image_utils.cpp).

Images marked as grayscale with a valid alpha component were written
without alpha, causing transparent grayscale images to become opaque.

Preserve the alpha channel by writing such images as grayscale+alpha,
while continuing to discard alpha for output format without alpha channel
support like JPEG.

[illwieckz]

It looks like this was an overlook, as the tool has clear intention to do heuristics by default:

• heuristics like -detectNormalMap is enabled by default,

and to select the smallest format for the output file:

• converting to PNG or TGA already drops the alpha channel when opaque.

Tools like Urcheon had to implement some image pre-processing that did the same alpha channel opaque look-up and had to write the stripped image as a temporary file before calling crunch.

[illwieckz]

Can be tested with this file as input (1×1 RGBA TGA image with opaque alpha channel).

printf '\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x01\x00\x20\x08\xFF\xFF\xFF\xFF' > opaque.tga

Before:

$ crunch -file opaque.tga -out out/opaque.tga | grep 'Format:'
Source texture: 1x1, Levels: 1, Faces: 1, Format: A8L8
Input texture: 1x1, Levels: 1, Faces: 1, Format: A8L8
Output texture: 1x1, Levels: 1, Faces: 1, Format: L8
$ crunch -file opaque.tga -out out/opaque.png | grep 'Format:'
Source texture: 1x1, Levels: 1, Faces: 1, Format: A8L8
Input texture: 1x1, Levels: 1, Faces: 1, Format: A8L8
Output texture: 1x1, Levels: 1, Faces: 1, Format: L8
$ crunch -file opaque.tga -out out/opaque.crn | grep 'Format:'
Source texture: 1x1, Levels: 1, Faces: 1, Format: A8L8
Input texture: 1x1, Levels: 1, Faces: 1, Format: A8L8
Output texture: 1x1, Levels: 1, Faces: 1, Format: DXT5
$ crunch -file opaque.tga -out out/opaque.dds | grep 'Format:'
Source texture: 1x1, Levels: 1, Faces: 1, Format: A8L8
Input texture: 1x1, Levels: 1, Faces: 1, Format: A8L8
Output texture: 1x1, Levels: 1, Faces: 1, Format: DXT5
$ crunch -file opaque.tga -out out/opaque.ktx | grep 'Format:'
Source texture: 1x1, Levels: 1, Faces: 1, Format: A8L8
Input texture: 1x1, Levels: 1, Faces: 1, Format: A8L8
Output texture: 1x1, Levels: 1, Faces: 1, Format: ETC2A

After:

$ crunch -file opaque.tga -out out/opaque.tga | grep 'Format:'
Source texture: 1x1, Levels: 1, Faces: 1, Format: L8
Input texture: 1x1, Levels: 1, Faces: 1, Format: L8
Output texture: 1x1, Levels: 1, Faces: 1, Format: L8
$ crunch -file opaque.tga -out out/opaque.png | grep 'Format:'
Source texture: 1x1, Levels: 1, Faces: 1, Format: L8
Input texture: 1x1, Levels: 1, Faces: 1, Format: L8
Output texture: 1x1, Levels: 1, Faces: 1, Format: L8
$ crunch -file opaque.tga -out out/opaque.crn | grep 'Format:'
Source texture: 1x1, Levels: 1, Faces: 1, Format: L8
Input texture: 1x1, Levels: 1, Faces: 1, Format: L8
Output texture: 1x1, Levels: 1, Faces: 1, Format: DXT1
$ crunch -file opaque.tga -out out/opaque.dds | grep 'Format:'
Source texture: 1x1, Levels: 1, Faces: 1, Format: L8
Input texture: 1x1, Levels: 1, Faces: 1, Format: L8
Output texture: 1x1, Levels: 1, Faces: 1, Format: DXT1
$ crunch -file opaque.tga -out out/opaque.ktx | grep 'Format:'
Source texture: 1x1, Levels: 1, Faces: 1, Format: L8
Input texture: 1x1, Levels: 1, Faces: 1, Format: L8
Output texture: 1x1, Levels: 1, Faces: 1, Format: ETC1

[illwieckz]

And for control to make sure it didn't break transparent images, tested with this file as input (2×2 RGBA TGA image with color variation and non-opaque alpha channel):

printf '\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02\x00\x20\x08\x00\x00\x00\xff\xff\xff\x00\xff\x00\xff\x00\x00\xff\xff\x00\x00\xff\x00\x00\x00\x00\x00\x00' > alpha_test.tga

Before:

$ crunch -file alpha_test.tga -out out/alpha_test.tga | grep 'Format:'
Source texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Input texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Output texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
$ crunch -file alpha_test.tga -out out/alpha_test.png | grep 'Format:'
Source texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Input texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Output texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
$ crunch -file alpha_test.tga -out out/alpha_test.crn | grep 'Format:'
Source texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Input texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Output texture: 2x2, Levels: 2, Faces: 1, Format: DXT5
$ crunch -file alpha_test.tga -out out/alpha_test.dds | grep 'Format:'
Source texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Input texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Output texture: 2x2, Levels: 2, Faces: 1, Format: DXT5
$ crunch -file alpha_test.tga -out out/alpha_test.ktx | grep 'Format:'
Source texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Input texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Output texture: 2x2, Levels: 2, Faces: 1, Format: ETC2A

After:

$ crunch -file alpha_test.tga -out out/alpha_test.tga | grep 'Format:'
Source texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Input texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Output texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
$ crunch -file alpha_test.tga -out out/alpha_test.png | grep 'Format:'
Source texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Input texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Output texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
$ crunch -file alpha_test.tga -out out/alpha_test.crn | grep 'Format:'
Source texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Input texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Output texture: 2x2, Levels: 2, Faces: 1, Format: DXT5
$ crunch -file alpha_test.tga -out out/alpha_test.dds | grep 'Format:'
Source texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Input texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Output texture: 2x2, Levels: 2, Faces: 1, Format: DXT5
$ crunch -file alpha_test.tga -out out/alpha_test.ktx | grep 'Format:'
Source texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Input texture: 2x2, Levels: 1, Faces: 1, Format: A8R8G8B8
Output texture: 2x2, Levels: 2, Faces: 1, Format: ETC2A

[illwieckz]

As explained, Urcheon already did the opaque alpha channel stripping, so this will not reduce Unvanquished game file size. But for other games using the crunch tool to convert their images without pre-processing them, this may reduce their game file size!

[illwieckz]

I added test samples to the test suite.

[slipher]

The output from processing test-white-alpha-transparent is wrong. I ran test.py and the outputs match your checksums, but they are opaque.

[illwieckz]

The output from processing test-white-alpha-transparent is wrong. I ran test.py and the outputs match your checksums, but they are opaque.

Good catch! I did a mistake. I did a 32-bit TGA image but without the alpha attribute saying the 4th byte is an alpha channel. Now all the images have the alpha attribute. I also committed the generator.

Actually, this disclosed a feature request: 32-bit images should be converted to 32-bit images, even if the 4th channel isn't said to be alpha. In Crunch context, alpha means nothing. An file can store normal map in RGB and height map in alpha channel, that's not transparency information. So, every conversion from an input with 4 channels should produce an output with 4 channels, and if the output format only recognize RGBA, so it should be it. So we discovered the current Crunch drops the alpha channel when the TGA has a 4th channel but that 4th channel isn't flagged as alpha. I don't know yet if that's something to fix in Crunch or something to fix in stb_image. That can be implemented later. The sample generator is ready to produce those other test files but for now those are disabled.

[illwieckz]

Hu, no, it looks like the image is still opaque when the color is white, but transparent when the color is red…

[illwieckz]

@slipher I made all the converted test files from this test suite being written to either tga-to-all-opaque or tga-to-all-transparent according to how they should look. It means the output files of transparent TGA files converted to JPEG are written in tga-to-all-opaque. It now makes very easy and obvious that all the files that should be opaque are opaque and that all the files that should have some transparent pixels have some.

Also it means we can actually consider JPEG output of transparent TGA to be clones of any output of opaque TGA. Clones, in this test tool, designates files whose output should be exactly the same to the bit, whatever the input format or the conversion operation, for example:

--- 664b4db91a red_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-rgb8-noalpha-red-opaque-1x1.jpg
Yes 664b4db91a red_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-rgb8-alpha8-red-opaque-1x1.jpg
Yes 664b4db91a red_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-rgb8-alpha8-red-transparent-1x1.jpg
Yes 664b4db91a red_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-rgb8-other8-red-opaque-1x1.jpg
Yes 664b4db91a red_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-rgb8-other8-red-transparent-1x1.jpg

So we now not only validate that converting TGA to JPG actually produces a file, that converting each TGA to JPG produces the file we expect for each conversion, but that converting a transparent TGA to JPG produces the exact same file as converting an opaque TGA to JPG.

[slipher]

LGTM

[illwieckz]

I rebased over:

• Rework the sample testing #103
• Accept grayscale JPEGs with non-1x1 sampling factors #99

[illwieckz]

I actually exported this commit:

• test: call wineserver -k before every wine run to avoid random errors

to:

• Rework the sample testing #103

As it's generic test rewrite, so it better belongs there.

[illwieckz]

OK, so everything is implemented in all those PRs:

• Rework the sample testing #103
• Ignore fully opaque alpha channels, preserve alpha when writing grayscale images #98
• Accept grayscale JPEGs with non-1x1 sampling factors #99
• Endianness fixes #102

I will rename some symbols and some produced path, but otherwise it's ready.

The endianness PR, not only adding endianness fixes, also adds tests for renormalization, conversion tests for DXN, and conversion to and from big endian KTX (including DXN).

We're now running 814 conversion tests… 😎️ it goes brrrrrrrrrrrrrrrtttt!

[illwieckz]

Note to myself:

TODO: crunch-*
TODO: base_name/sample_name

[slipher]

We're now running 814 conversion tests

A larger number of tests is not necessarily better. Now it is so many that no one is going to want to inspect the outputs to make sure they are correct. By doing so, I already found tests enforcing incorrect outputs in two previous revisions of the PR.

Maybe this could be made more manageable by reducing the number of output formats supported by the tool. The point of Crunch is to produce DXT-compressed textures, which can be shipped in the dds, ktx, or crn containers. Beyond that it makes sense to have some widely supported lossless image format to do reverse conversions or debug input issues. But there is no use in supporting as many non-DXT outputs as possible. I think it would make sense to remove JPEG and BMP output formats. There is no use in converting from CRN to JPG and there is no use in testing the exact hash of some JPEG outputs with obvious artifacts, which could very well change just from upgrading the library.

[illwieckz]

A larger number of tests is not necessarily better. Now it is so many that no one is going to want to inspect the outputs to make sure they are correct. By doing so, I already found tests enforcing incorrect outputs in two previous revisions of the PR.

I agree that the amount of tests can be a drawback, so if we can reduce them while not losing too much coverage it would be nice. But I prefer to tests many combinations.

The reasons is that it is needed that the outcome be predictable and not randomized according to the compiler, the operating system, the architecture, etc. Then, if later we discover that a specific file or format conversion from or to is producing something invalid, then we can fix it once for all.

For example I'm not excluding the possibility we discover that some format conversion may significantly degrade the RGB when the whole alpha is fully transparent. This is unwanted as game image channels are not necessarily colors. If that happens, then we would be able to fix that for the specific conversion, once for all.

Maybe this could be made more manageable by reducing the number of output formats supported by the tool. The point of Crunch is to produce DXT-compressed textures, which can be shipped in the dds, ktx, or crn containers. Beyond that it makes sense to have some widely supported lossless image format to do reverse conversions or debug input issues. But there is no use in supporting as many non-DXT outputs as possible.

I see crunch not only as a CRN producer but as a reliable tool we can recommend for image conversion, and so all the images currently supported by crunch are wanted (except PSD). This need surfaced because of the ImageMagick fiasco. I have not documented in that paper yet my more recent findings in that BMP ImageMagick does the contrary of TGA (meaning that when I will tell them, they will likely break BMP like they broke TGA, to be consistent).

I've learnt that Crunch started to be packaged in Debian as an effort to package in Debian the tools required to build games, and here Crunch was found in the build system of Xonotic. So Crunch is packaged in Debian as a dependency to build Xonotic. Xonotic switched to our Crunch for their DDS production some years ago when dropping their own custom (now expired) patent-avoiding S2TC encoder.

Xonotic suffers from broken image conversion tools like we do, especially from the ImageMagick fiasco:

• https://gitlab.com/xonotic/mediasource/-/merge_requests/42#note_2637893699
• https://gitlab.com/xonotic/xonotic/-/work_items/331
• https://gitlab.com/xonotic/xonotic/-/work_items/341
• https://gitlab.com/xonotic/xonotic-maps.pk3dir/-/merge_requests/159

So in a near future I will recommend them to switch to Crunch for non-DXT format conversions as well. For that, I will first change the default of TGA output from RLE to flat. This is a bad default because not all tools support RLE, for example NetRadiant did not for decades. Only recently Garux implemented it in its fork and I started merging that. So I want to make flat TGA the default and add an option for RLE output. This will add more tests.

The current image conversion pipeline in Urcheon is super crazy. Let's for example consider PNG to CRN, here is what Crunch does:

• convert PNG to lossless exact WebP with cwebp
• convert WebP to RGBA using Pillow
• walk the RGBA stream and convert to RGB if A is opaque using a custom code
• convert RGB(A) to TGA using a custom TGA writer
• convert TGA to CRN using Crunch

Converting a TGA to CRN does:

• convert TGA to RGBA using Pillow
• walk the RGBA stream and convert to RGB if A is opaque using a custom code
• convert RGB(A) to TGA using a custom TGA writer
• convert TGA to CRN using Crunch

You'll see we do a lot of round-trips converting images from formats to others to others to others, sometime to itself.

This is to sanitize the image formats to convert them to more popular variants we know are working as input. For example Pillow has a bug (I still have to report) that converts 8-bit grayscale PNG to 1-bit black-and-white image when converting to RGB, so we workaround the issue by converting PNG images to lossless WebP first, to get an RGB WebP that will be properly loaded and converted to other RGB formats by Pillow. The TGA-to-TGA round trip was there because of the stb_image bundled with Crunch not supporting all TGA orientations, something I fixed.

We need a conversion tool we can trust in our pipeline. The fact Crunch can read all the current input formats is very wanted (except PSD, this one we don't care), so Urcheon can simplify its pipeline by just feeding the input files to itself. And for formats like WebP as input, then we would just use dwebp to convert the WebP image to PNG as a lingua franca before feeding Crunch. I'm not excluding to add WebP support to Crunch in the future. The only reason I may not do it is that Crunch is fully autonomous and I don't want it to rely on build-time or run-time external libs.

In the past I used ImageMagick in Urcheon instead of Pillow, until we stumbled onto the ImageMagick problem of them going “we didn't like how TGA was specified so we decided the specification didn't applied to us because we want all the images formats in the world be top-left origin so we hereby declare that TGA is now top-left origin by default” (BMP not being top-left origin by default as well can be broken anytime when they discover it). Then, I discovered that Pillow has a bug. At least, unlike ImageMagick, I have good reasons to believe it's only a bug, not something they want and will fight for.

So I definitely want Crunch to be “the trusted image convertor we recommend”. We need it to be.

I don't mind the PNG output not being crazy in compression or not being a clone of oxipng at selecting the best format for the data, but a minimum should be done (see below), I don't mind the JPEG output not offering compression knobs. But I mind the PNG and JPEG output being valid and tested for regressions.

The minimum “optimization” I care in whatever the output format (including PNG and TGA) is not turning L8 into RG8G8B8 when L8 is supported in the output format and dropping the Alpha channel in output when the Alpha channel is opaque in input.

I mind for CRN, DDS, and KTX output being crazy in compression and good at selecting the best format for the data.

If a lib update changes JPEG output (because of, for example, a better compression algorithm), then we will just register the newly created JPEGs. The same happens if we find the JPEG output can be improved by turning a knob already available to us.

It still matters to check the JPEG output to catch for random breakage because of compiler, system or architecture variation.

All the libs used in JPEG input and output processing are vendored anyway, so they cannot be updated in our back. If we update them and they change the output checksum but the output is valid, then we will just re-register the new output files as updated references and call it a day.

We have seen in the recent days that for testing an input, we need to test an output. For example we have seen that testing PNG as an input when outputing X doesn't validate the PNG to Y conversion. We also have seen that modifying PNG as an input can break X or Y as an output. That's why the amount of tests skyrocketed because before, some tests only tested a collection of input with a single output, now we also test all of the output for every input.

Having 800 tests doesn't mean we have 800 files to check visually and by hand one by one.

First, for the visual check, those recent changes reshape the test image folders to make it easier for a single view of them over them to visually validate many of them at once just scrolling in a file browser. For Linux systems I recommend crunch-thumbnailer.

Then I have to remind the concept of “clones” used in the test suite. Clones are output files that should produce the exact same checksum whatever their input format (PNG, TGA…) or storage variant (RGB, L…). Right now on the 814 tests, 280 are clones.

So there are 534 remaining you will say. But there are two tiers of checking. Some images like jpg only have to be checked visually. For this a simple overview in a file browser is good enough and can be very fast.

Then for some other formats (especially the ones meant to be shipped in the game as the best optimized format possible) we better check that the storage variant is expected. But that's not hundreds of files to check.

One thing is that by adding more tests we remove the need to visually check some files or check their format by hand.

For example let's imagine we have a PNG with RGBA data. We do a PNG to PNG conversion and manually validate that the output is as expected and preserved the alpha channel.

Then we can do PNG-to-BMP-to-PNG and PNG-to-TGA-to-PNG and since the final PNG should be a clone of PNG-to-PNG, we don't have to manually inspect the PNG-to-BMP and PNG-to-TGA and BMP-to-PNG and TGA-to-PNG ouput to know that the alpha channel has been preserved in all those conversions.

So I may even add more tests that will increase the amount of clones, especially if they increase the clone/total ratio even if that increases the total.

[illwieckz]

We can do things on purpose to increase the clone/total ratio, with the current tests.

For example, here we have 2 collections of 10 clones, we can make it one collection of 20 clones just by making all the images 2×2. Verifying one JPEG output would validate 20 conversions.

In fact we can add generators for all the PNG and BMP input formats as well that are supposed to produce 2×2 white image when converted to JPEG, and then it would increase the amount of tests but just add clones, without adding anything to validate by hand.

--- 7297a8b3c1 white_opaque_2_tga_jpg build/test/tga-to-all-opaque/sample-storage-rle-grayscale8-noalpha-white-opaque-2x2.jpg
Yes 7297a8b3c1 white_opaque_2_tga_jpg build/test/tga-to-all-opaque/sample-storage-rle-rgb8-noalpha-white-opaque-2x2.jpg
Yes 7297a8b3c1 white_opaque_2_tga_jpg build/test/tga-to-all-opaque/sample-storage-rle-grayscale8-alpha8-white-opaque-2x2.jpg
Yes 7297a8b3c1 white_opaque_2_tga_jpg build/test/tga-to-all-opaque/sample-storage-rle-rgb8-alpha8-white-opaque-2x2.jpg
Yes 7297a8b3c1 white_opaque_2_tga_jpg build/test/tga-to-all-opaque/sample-storage-rle-grayscale8-alpha8-white-transparent-2x2.jpg
Yes 7297a8b3c1 white_opaque_2_tga_jpg build/test/tga-to-all-opaque/sample-storage-rle-rgb8-alpha8-white-transparent-2x2.jpg
Yes 7297a8b3c1 white_opaque_2_tga_jpg build/test/tga-to-all-opaque/sample-storage-rle-grayscale8-other8-white-opaque-2x2.jpg
Yes 7297a8b3c1 white_opaque_2_tga_jpg build/test/tga-to-all-opaque/sample-storage-rle-rgb8-other8-white-opaque-2x2.jpg
Yes 7297a8b3c1 white_opaque_2_tga_jpg build/test/tga-to-all-opaque/sample-storage-rle-grayscale8-other8-white-transparent-2x2.jpg
Yes 7297a8b3c1 white_opaque_2_tga_jpg build/test/tga-to-all-opaque/sample-storage-rle-rgb8-other8-white-transparent-2x2.jpg
--- 80c1441c7a white_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-grayscale8-noalpha-white-opaque-1x1.jpg
Yes 80c1441c7a white_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-rgb8-noalpha-white-opaque-1x1.jpg
Yes 80c1441c7a white_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-grayscale8-alpha8-white-opaque-1x1.jpg
Yes 80c1441c7a white_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-rgb8-alpha8-white-opaque-1x1.jpg
Yes 80c1441c7a white_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-grayscale8-alpha8-white-transparent-1x1.jpg
Yes 80c1441c7a white_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-rgb8-alpha8-white-transparent-1x1.jpg
Yes 80c1441c7a white_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-grayscale8-other8-white-opaque-1x1.jpg
Yes 80c1441c7a white_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-rgb8-other8-white-opaque-1x1.jpg
Yes 80c1441c7a white_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-grayscale8-other8-white-transparent-1x1.jpg
Yes 80c1441c7a white_opaque_1_tga_jpg build/test/tga-to-all-opaque/sample-storage-flat-rgb8-other8-white-transparent-1x1.jpg

[illwieckz]

We can also re-generate the TGA and BMP orientation samples to match the exact output pixels of the current TGA color samples, this way all TGA and BMP orientation samples will become clones of the TGA-to-anything conversions.

This will add no new tests, but will increase the clone/total ratio.

That means only validating a handful of outputs (opaque, transparent…) would not only validate image format conversion but also orientation processing.

[slipher]

So in a near future I will recommend them to switch to Crunch for non-DXT format conversions as well.

What is an example of using non-DXT conversions in the pipeline?

Then I have to remind the concept of “clones” used in the test suite. Clones are output files that should produce the exact same checksum whatever their input format (PNG, TGA…) or storage variant (RGB, L…). Right now on the 814 tests, 280 are clones.

That doesn't really help, unless I would go to the trouble of making a script to delete the duplicates.

So far my workflow was to use a viewer that lets you iterate through a directory with arrow keys, which can view PNG, JPG, BMP, and TGA. Then for DDS and KTX I used more specialized programs that only open one file at a time, so I only checked a few random samples of those since it is too boring to open all the files. So far I didn't see any PRs adding CRN outputs, but it would be painful to check those for sure.

Also for the the images where the entire area uses transparency, it would have been nicer to use alpha=0.5 or something, instead of 0 so that it is possible to check the color components with a normal viewer. One of the programs I used has an invert alpha function, but it would be too boring to open all of them with that one.

[illwieckz]

What is an example of using non-DXT conversions in the pipeline?

There are conversion pipelines when releasing a game, but also when creating an asset repository.

For the asset repository pipeline, when creating dpkdir repositories for legacy maps, I convert all the original files to modern formats, so TGA to PNG (now I use exact lossless WebP instead), WAV to FLAC, etc. Even if I now convert TGA to WebP, I need TGA to PNG to feed cwebp. So Crunch is welcome as a reliable non-DXT conversion tool here.

For the releasing game pipeline, I always designed Urcheon to be usable with other games done the pk3/pk4/dpk way, and I want Crunch to be reliable enough to be usable this way. So for example converting TGA to PNG for an ioquake3-derivated game is a pretty valid release pipeline, being by using Urcheon or Crunch the direct way.

Also for the the images where the entire area uses transparency, it would have been nicer to use alpha=0.5 or something, instead of 0 so that it is possible to check the color components with a normal viewer. One of the programs I used has an invert alpha function, but it would be too boring to open all of them with that one.

I want to extend the testing of colors in transparent channels but that's for later. Actually converting to jpg any image supporting transparency is an easy way to do it, and then the clone checking mechanism would take the wheel.

The thing is that with alpha=0.5 some compressors may preserve the color, while with alpha=0 they would do optimizations like flat-out the whole image to a single color. So we explicitly need to test the full transparency option.

[illwieckz]

That doesn't really help, unless I would go to the trouble of making a script to delete the duplicates.

I can add an option to the test tool to delete the clones once the whole test run is finished (when the clones are valid).