Jpeg downscaling decoding

[br3aker]

Prerequisites

• I have written a descriptive pull-request title
• I have verified that there are no overlapping pull-requests open
• I have verified that I am following the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
• I have provided test coverage for my change (where applicable)

TODO

• API
• 1/2 scaling implementation
• 1/4 scaling implementation
• Fix resulting image size remainder artifacts
• Rollback ProfilingSandbox/Program.cs

Description

This PR implements built-in scaling of the Jpeg decoder via scaled IDCT. Currently only 8-to-8 and 8-to-1 (1/8) scalings are supported, I'll implement 1/2 and 1/4 scalings as part of this PR a bit later. Current code is rather stable in terms of internal API, 1/2 and 1/4 scaling will be based on implemented abstract class.

This code is WIP, contains development leftovers.

API

I'm proposing two new methods for the IImageDecoder interface:

/// <summary>/// Decodes the image from specified stream to an <see cref="Image"/> with specified size./// </summary>/// <typeparam name="TPixel">The pixel format.</typeparam>/// <param name="configuration">The configuration for the image.</param>/// <param name="stream">The <see cref="Stream"/> containing image data.</param>/// <param name="targetSize">Target size of the output image.</param>/// <param name="resampler">Resampler used for resizing.</param>/// <param name="cancellationToken">The token to monitor for cancellation requests.</param>/// <returns>The <see cref="Image"/>.</returns>Image<TPixel>DecodeInto<TPixel>(Configurationconfiguration,Streamstream,SizetargetSize,IResamplerresampler,CancellationTokencancellationToken)whereTPixel: unmanaged,IPixel<TPixel>;ImageDecodeInto(Configurationconfiguration,Streamstream,SizetargetSize,IResamplerresampler,CancellationTokencancellationToken);

And new method to the Image<TPixel>:

// same docs as for DecodeInto abovepublicstaticImage<TPixel>LoadInto<TPixel>(Streamstream,SizetargetSize,IResamplerresampler,IImageDecoderdecoder)whereTPixel: unmanaged,IPixel<TPixel>{}

I've certainly missed other endpoints affected by proposed new API, reason for this proposal is to start a discussion :)

Why are new methods needed?
For code versatility. As ImageSharp already supports quite some image formats I'd expect this piece of code to be very popular:

usingvarimg=Image.Load(input);img.Mutate(ctx =>ctx.Resize(thumbnailSize));img.Save(output);

To use new Jpeg resizing API users would need to somehow identify image format first and only then call it directly parsing image stream twice. Proposed API eliminates this performance loss:

usingvarimg=Image.LoadInto(input,thumbnailSize);img.Save(output);

Benchmarks

Master

Method	Threads	Mean	Error	StdDev	Ratio
SystemDrawing	1	3,728.3 ms	2,858.88 ms	156.70 ms	2.25
ImageSharp	1	1,654.3 ms	204.45 ms	11.21 ms	1.00
Magick	1	4,040.4 ms	210.98 ms	11.56 ms	2.44
MagicScaler	1	888.4 ms	165.17 ms	9.05 ms	0.54
SkiaBitmap	1	2,873.0 ms	135.61 ms	7.43 ms	1.74
SkiaBitmapDecodeToTargetSize	1	888.0 ms	108.37 ms	5.94 ms	0.54
NetVips	1	805.6 ms	163.64 ms	8.97 ms	0.49
SystemDrawing	8	1,246.6 ms	565.78 ms	31.01 ms	2.82
ImageSharp	8	442.8 ms	341.56 ms	18.72 ms	1.00
Magick	8	1,082.2 ms	172.97 ms	9.48 ms	2.45
MagicScaler	8	346.6 ms	164.54 ms	9.02 ms	0.78
SkiaBitmap	8	746.7 ms	176.11 ms	9.65 ms	1.69
SkiaBitmapDecodeToTargetSize	8	235.5 ms	41.48 ms	2.27 ms	0.53
NetVips	8	232.3 ms	71.85 ms	3.94 ms	0.53

This PR

Method	Threads	Mean	Error	StdDev	Ratio
SystemDrawing	1	3,600.3 ms	534.99 ms	29.32 ms	3.22
ImageSharp	1	1,116.5 ms	126.91 ms	6.96 ms	1.00
Magick	1	4,034.3 ms	243.95 ms	13.37 ms	3.61
MagicScaler	1	890.1 ms	20.20 ms	1.11 ms	0.80
SkiaBitmap	1	2,912.9 ms	353.28 ms	19.36 ms	2.61
SkiaBitmapDecodeToTargetSize	1	890.6 ms	81.24 ms	4.45 ms	0.80
NetVips	1	803.1 ms	99.84 ms	5.47 ms	0.72
SystemDrawing	8	1,396.3 ms	471.21 ms	25.83 ms	4.41
ImageSharp	8	316.8 ms	32.03 ms	1.76 ms	1.00
Magick	8	1,164.4 ms	636.66 ms	34.90 ms	3.68
MagicScaler	8	356.2 ms	278.55 ms	15.27 ms	1.12
SkiaBitmap	8	732.9 ms	142.91 ms	7.83 ms	2.31
SkiaBitmapDecodeToTargetSize	8	236.7 ms	44.69 ms	2.45 ms	0.75
NetVips	8	232.9 ms	128.47 ms	7.04 ms	0.74

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (main@2c42e41). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 75dc96a differs from pull request most recent head 7a9cf87. Consider uploading reports for the commit 7a9cf87 to get more accurate results

@@ Coverage Diff @@## main #2076 +/- ##
======================================
Coverage ? 88% ======================================
Files ? 997 Lines ? 53874 Branches ? 6891 ======================================
Hits ? 47444 Misses ? 5250 Partials ? 1180 

Flag	Coverage Δ
unittests	88% <0%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c42e41...7a9cf87. Read the comment docs.

[br3aker]

@JimBobSquarePants it's actually done and downscaling IDCT is covered by tests. Do we need tests with actual image comparison between decode -> resize and decode_downscale to prove correctness? It's almost 100% identical to Box resizing resizing.

The only blocking thing left is API, what's the best approach to discuss it?

[tocsoft]

I've had a little idea on how we could handle this as an api... basically i think we could handle this by convention during load.

The premise is, we add an Action<ImageProcessingContext> callback into the Image.Load APIs to allow for applying mutations during loading, as part of that we can detect if the first operation applied to the context will be to apply a ResizeProcessor if it is, and the decoder support target size decoding, then we trigger that code path and apply the remaining processors to the now decoded/resized image so that all processor will work in that context.

Limitation here would be that the resize would have to be the first operation but apart from that it would work...and I think for the fact this feel like an optimisation target its not too hard an a limitation.

This benefit of this approach is would allow us to add the feature without adding lot of new APIs (only new overloads of existing exposing more existing APIs).

Here is a spike on how it could be implemented.
br3aker/ImageSharp@dp/jpeg-downscaling-decode...tocsoft:tocsoft/pipeline-load#diff-f6a7a772c9597206b85c603ae51aa6947ba4c98dfd23441ed62e05ba8934ee1c

[tocsoft]

so instead of calling

usingvarimg=Image.Load("/path.jpg");img.Mutate(ctx=>ctx.Resize(120,100));img.Save("/out.png");

you can instead call

usingvarimg=Image.Load("/path.jpg", ctx=>ctx.Resize(120,100));img.Save("/out.png");

but this wouls also work for any mutation set applied i.e.

usingvarimg=Image.Load("/path.jpg", ctx=>ctx.Resize(120,100).Pixelate(...).DrawText(...).Resize(20,10));img.Save("/out.png");

which would functionally translate to

usingvarimg=Image.Load("/path.jpg",newResizeProcessor(120,100));img.Mutate(ctx=>ctx.Pixelate(...).DrawText(...).Resize(20,10));//notice only the first resize is striped off and in lined into loadingimg.Save("/out.png");

[JimBobSquarePants]

I thought we would just provide an overload for Load\LoadAsync that takes a Size.

IImageDecoder could get a default implementation for Decode with a Size if we didn't fancy implementing the approach for all the types.

Decoding to a specific size is is a very specific and unique function. I can't ever see us implementing other mutations on load so I think it's best to keep things locked down and specific. I wouldn't even allow setting a custom IResampler since we want the output to be roughly equivalent for all formats.

[br3aker]

I wouldn't even allow setting a custom IResampler since we want the output to be roughly equivalent for all formats.

I don't think it's a problem.

Interface:

ImageLoadResize(Stream,Sizesize,IResamplerresampler);

Jpeg implementation:

ImageLoadResize(Streamstream,Sizesize,IResamplerresampler){Imageimg;// jpeg idct resizing is very close to Box resamplerif(resampler==KnownResamplers.Box){img=this.DecodeResizeImplementation(stream,size);}else{img=this.Decode(stream);}// post decode resize if decoding didn't resize due to unsupported resampler // or didn't resize to exact size user asked forif(img.Size!=size){// Note: MagicScaler provides a way to use two different resamplers// For example, it can use low-quality Box resampler for rough downscaling// and then use something better for final downscaling to target size// // IMO this is a very cool niche thingy but it's very specific to the jpeg decoder // that it would be better to cut it or put into JpegDecoder as a specific method at maximg.Mutate(ctx =>ctx.Resize(size,resampler));}returnimg;}

If any other format would provide a 'native' resizing operation - we can use this boilerplate with any other resampler check for native scaling support. And if there's no such support from the format:

Imageimg=this.Decode(stream);img.Mutate(ctx =>ctx.Resize(size,resampler));returnimg;

The only limitation is the obligation to provide a resampler, providing anything but KnowResamplers.Box in a thumbnail scenario would cut a huge chunk of performance.

[br3aker]

Soooo how do we decide?

[br3aker]

your decoder is simply decoding to the closest possible scaled size to the target yeah?

Yes.

If so I'm happy to get this merged once the conflict is fixed.

Resolved.

I think we can provide a best of both worlds scenario btw.

1. Global Decode with options (Decode to closest target, 2nd pass resize)
2. JpegDecoder specific (Decode to closest target only, user decides what to do next)

I like it!

[br3aker]

Hmmm, I think I saw this error somewhere before, lastest commit only added comments and previous commit passed all tests. Didn't we have a net7 only bug somewhere?

[brianpopow]

Hmmm, I think I saw this error somewhere before, lastest commit only added comments and previous commit passed all tests. Didn't we have a net7 only bug somewhere?

Yes , that is #2117 haunting us again. I will try restart the CI.

[JimBobSquarePants]

Very excited about this! I have plans for a general API I've already started working on.