Skip to content

Conversation

@brianpopow
Copy link
Collaborator

@brianpopow brianpopow commented Nov 4, 2021

Prerequisites

  • I have written a descriptive pull-request title
  • I have verified that there are no overlapping pull-requests open
  • I have verified that I am following the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
  • I have provided test coverage for my change (where applicable)

Description

This PR makes sure magick.net and imagesharp use the same configuration for encoding benchmarks, so we comparing the same thing.
Related to #1786

NearLossless = 100
};

this.webpMagick.Settings.SetDefine(MagickFormat.WebP, "exact", false);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlemstra the exact and quality parameters seem to be missing in WebPWriteDefines. Should they maybe be added to WebPWriteDefines? If you think that makes sense, i could open a PR at Magick.Net.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if we need to exact option. You can use this.webpMagick.Settings.Quality to set the quality. Setting the quality too 100 might enable this?

Copy link
Collaborator Author

@brianpopow brianpopow Nov 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If i dont set it, the libwebp default will be used, right? The default is false.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The IMagickSettings interface does not seem to have a Quality Property.

Copy link
Member

@dlemstra dlemstra Nov 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry my bad, that should be webpMagick.Quality instead. And I just added the Exact property to the WebPWriteDefines. This will become available in the next release. And setting the Quality to 100 will make it lossless.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry my bad, that should be webpMagick.Quality instead. And I just added the Exact property to the WebPWriteDefines. This will become available in the next release. And setting the Quality to 100 will make it lossless.

No problem, thanks for adding exact to WebPWriteDefines

@codecov
Copy link

codecov bot commented Nov 4, 2021

Codecov Report

Merging #1808 (53f278d) into master (a53ebaa) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1808      +/-   ##
==========================================
- Coverage   87.30%   87.30%   -0.01%     
==========================================
  Files         936      936              
  Lines       48015    48015              
  Branches     6034     6034              
==========================================
- Hits        41921    41920       -1     
- Misses       5095     5096       +1     
  Partials      999      999              
Flag Coverage Δ
unittests 87.30% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/ImageSharp/Formats/Webp/WebpEncoder.cs 100.00% <100.00%> (ø)
...ImageSharp/Formats/Webp/Lossless/Vp8LBitEntropy.cs 98.80% <0.00%> (-1.20%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a53ebaa...53f278d. Read the comment docs.

@JimBobSquarePants
Copy link
Member

@brianpopow does this affect the benchmarks?

@brianpopow
Copy link
Collaborator Author

@brianpopow does this affect the benchmarks?

As I already said in #1786, the benchmarks there cannot be trusted and I need to re-run them again at the commit we merged webp into master. I first wanted to make sure we agree on the settings here. The settings i have choosen are the default values we are using. Most of them should be also the default settings for libwebp, but i wanted to make it explicit. Maybe we decide to change the defaults at some point. Then the benchmarks should not change.

Here are the current benchmark results.

Encode:

BenchmarkDotNet=v0.13.0, OS=Windows 10.0.19043.1320 (21H1/May2021Update)
Intel Core i7-6700K CPU 4.00GHz (Skylake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100-rc.2.21505.57
  [Host]     : .NET 5.0.11 (5.0.1121.47308), X64 RyuJIT
  Job-WQLXJO : .NET 5.0.11 (5.0.1121.47308), X64 RyuJIT
  Job-OJJAMD : .NET Core 3.1.20 (CoreCLR 4.700.21.47003, CoreFX 4.700.21.47101), X64 RyuJIT
  Job-OMFOAS : .NET Framework 4.8 (4.8.4420.0), X64 RyuJIT

IterationCount=3  LaunchCount=1  WarmupCount=3

|                     Method |        Job |              Runtime |             Arguments |    TestImage |      Mean |     Error |   StdDev | Ratio | RatioSD |       Gen 0 |     Gen 1 |     Gen 2 |  Allocated |
|--------------------------- |----------- |--------------------- |---------------------- |------------- |----------:|----------:|---------:|------:|--------:|------------:|----------:|----------:|-----------:|
|        'Magick Webp Lossy' | Job-WQLXJO |             .NET 5.0 | /p:DebugType=portable | Png/Bike.png |  23.33 ms |  1.491 ms | 0.082 ms |  0.15 |    0.00 |           - |         - |         - |      67 KB |
|    'ImageSharp Webp Lossy' | Job-WQLXJO |             .NET 5.0 | /p:DebugType=portable | Png/Bike.png | 245.80 ms | 24.288 ms | 1.331 ms |  1.53 |    0.01 | 135000.0000 |         - |         - | 552,713 KB |
|     'Magick Webp Lossless' | Job-WQLXJO |             .NET 5.0 | /p:DebugType=portable | Png/Bike.png | 160.36 ms | 11.131 ms | 0.610 ms |  1.00 |    0.00 |           - |         - |         - |     518 KB |
| 'ImageSharp Webp Lossless' | Job-WQLXJO |             .NET 5.0 | /p:DebugType=portable | Png/Bike.png | 313.93 ms | 45.605 ms | 2.500 ms |  1.96 |    0.01 |  34000.0000 | 5000.0000 | 2000.0000 | 161,670 KB |
|                            |            |                      |                       |              |           |           |          |       |         |             |           |           |            |
|        'Magick Webp Lossy' | Job-OJJAMD |        .NET Core 3.1 |               Default | Png/Bike.png |  23.36 ms |  2.289 ms | 0.125 ms |  0.15 |    0.00 |           - |         - |         - |      67 KB |
|    'ImageSharp Webp Lossy' | Job-OJJAMD |        .NET Core 3.1 |               Default | Png/Bike.png | 254.64 ms | 19.620 ms | 1.075 ms |  1.59 |    0.00 | 135000.0000 |         - |         - | 552,713 KB |
|     'Magick Webp Lossless' | Job-OJJAMD |        .NET Core 3.1 |               Default | Png/Bike.png | 160.30 ms |  9.549 ms | 0.523 ms |  1.00 |    0.00 |           - |         - |         - |     518 KB |
| 'ImageSharp Webp Lossless' | Job-OJJAMD |        .NET Core 3.1 |               Default | Png/Bike.png | 320.35 ms | 22.924 ms | 1.257 ms |  2.00 |    0.01 |  34000.0000 | 5000.0000 | 2000.0000 | 161,669 KB |
|                            |            |                      |                       |              |           |           |          |       |         |             |           |           |            |
|        'Magick Webp Lossy' | Job-OMFOAS | .NET Framework 4.7.2 |               Default | Png/Bike.png |  23.37 ms |  0.908 ms | 0.050 ms |  0.15 |    0.00 |           - |         - |         - |      68 KB |
|    'ImageSharp Webp Lossy' | Job-OMFOAS | .NET Framework 4.7.2 |               Default | Png/Bike.png | 378.67 ms | 25.540 ms | 1.400 ms |  2.36 |    0.01 | 135000.0000 |         - |         - | 554,351 KB |
|     'Magick Webp Lossless' | Job-OMFOAS | .NET Framework 4.7.2 |               Default | Png/Bike.png | 160.13 ms |  5.115 ms | 0.280 ms |  1.00 |    0.00 |           - |         - |         - |     520 KB |
| 'ImageSharp Webp Lossless' | Job-OMFOAS | .NET Framework 4.7.2 |               Default | Png/Bike.png | 379.01 ms | 71.192 ms | 3.902 ms |  2.37 |    0.02 |  34000.0000 | 5000.0000 | 2000.0000 | 162,119 KB |

Decode:

|                     Method |        Job |              Runtime |             Arguments |        TestImageLossy |        TestImageLossless |       Mean |     Error |  StdDev |    Gen 0 | Gen 1 | Gen 2 | Allocated |
|--------------------------- |----------- |--------------------- |---------------------- |---------------------- |------------------------- |-----------:|----------:|--------:|---------:|------:|------:|----------:|
|        'Magick Lossy Webp' | Job-HLWZLL |             .NET 5.0 | /p:DebugType=portable | Webp/earth_lossy.webp | Webp/earth_lossless.webp |   107.9 ms |  28.91 ms | 1.58 ms |        - |     - |     - |     25 KB |
|    'ImageSharp Lossy Webp' | Job-HLWZLL |             .NET 5.0 | /p:DebugType=portable | Webp/earth_lossy.webp | Webp/earth_lossless.webp |   282.3 ms |  25.40 ms | 1.39 ms | 500.0000 |     - |     - |  2,428 KB |
|     'Magick Lossless Webp' | Job-HLWZLL |             .NET 5.0 | /p:DebugType=portable | Webp/earth_lossy.webp | Webp/earth_lossless.webp |   106.3 ms |  11.99 ms | 0.66 ms |        - |     - |     - |     16 KB |
| 'ImageSharp Lossless Webp' | Job-HLWZLL |             .NET 5.0 | /p:DebugType=portable | Webp/earth_lossy.webp | Webp/earth_lossless.webp |   280.2 ms |   6.21 ms | 0.34 ms |        - |     - |     - |  2,092 KB |
|        'Magick Lossy Webp' | Job-ALQPDS |        .NET Core 3.1 |               Default | Webp/earth_lossy.webp | Webp/earth_lossless.webp |   106.2 ms |   9.32 ms | 0.51 ms |        - |     - |     - |     15 KB |
|    'ImageSharp Lossy Webp' | Job-ALQPDS |        .NET Core 3.1 |               Default | Webp/earth_lossy.webp | Webp/earth_lossless.webp |   295.8 ms |  21.25 ms | 1.16 ms | 500.0000 |     - |     - |  2,427 KB |
|     'Magick Lossless Webp' | Job-ALQPDS |        .NET Core 3.1 |               Default | Webp/earth_lossy.webp | Webp/earth_lossless.webp |   106.5 ms |   4.07 ms | 0.22 ms |        - |     - |     - |     15 KB |
| 'ImageSharp Lossless Webp' | Job-ALQPDS |        .NET Core 3.1 |               Default | Webp/earth_lossy.webp | Webp/earth_lossless.webp |   464.0 ms |  55.70 ms | 3.05 ms |        - |     - |     - |  2,090 KB |
|        'Magick Lossy Webp' | Job-RYVVNN | .NET Framework 4.7.2 |               Default | Webp/earth_lossy.webp | Webp/earth_lossless.webp |   108.0 ms |  29.60 ms | 1.62 ms |        - |     - |     - |     32 KB |
|    'ImageSharp Lossy Webp' | Job-RYVVNN | .NET Framework 4.7.2 |               Default | Webp/earth_lossy.webp | Webp/earth_lossless.webp |   564.9 ms |  29.69 ms | 1.63 ms |        - |     - |     - |  2,436 KB |
|     'Magick Lossless Webp' | Job-RYVVNN | .NET Framework 4.7.2 |               Default | Webp/earth_lossy.webp | Webp/earth_lossless.webp |   106.2 ms |   4.74 ms | 0.26 ms |        - |     - |     - |     18 KB |
| 'ImageSharp Lossless Webp' | Job-RYVVNN | .NET Framework 4.7.2 |               Default | Webp/earth_lossy.webp | Webp/earth_lossless.webp | 1,767.5 ms | 106.33 ms | 5.83 ms |        - |     - |     - |  9,729 KB |

@dlemstra
Copy link
Member

dlemstra commented Nov 4, 2021

I wonder if Magick.NET should be changed to Q8 because Rgba32 seems to be used. Q16 is using 16 bit per channel instead of 8.

@JimBobSquarePants
Copy link
Member

Thanks @brianpopow I misread, I thought you had reran the benchmarks and were making configuration changes upon the result to ensure that the parameters matched. I was wondering if the changes you had made affected before and after benchmarks for the current state.

@brianpopow
Copy link
Collaborator Author

@JimBobSquarePants I think the benchmarks in the other PR's are comparable, but to really make sure we should also re-run those with these settings.

I wonder if Magick.NET should be changed to Q8 because Rgba32 seems to be used. Q16 is using 16 bit per channel instead of 8.

If no other benchmark is relying on Q16, we probably should consider changing to Q8.

Copy link
Member

@JimBobSquarePants JimBobSquarePants left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job 👍

@JimBobSquarePants JimBobSquarePants merged commit 6f0e825 into master Nov 9, 2021
@JimBobSquarePants JimBobSquarePants deleted the bp/webpbenchmarks branch November 9, 2021 05:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants