Comprehensive Test Suite of Images

[phoboslab]

I have now assembled a pretty comprehensive suite of test images. These all come with the proper license information (CC or public domain). It includes:

• an icon set in 64px & 512px (thanks @nigeltao)
• various photo sets
• various texture sets
• game screenshots
• website screenshots
• a random sample from https://pngimg.com/

Here's the full set:
https://phoboslab.org/files/qoibench/qoi_benchmark_suite.tar (1.1 GB) — very proudly excluding lenna.jpg

All images in this test suite are PNGs. I will add QOI images once the specification has been finalized (related #20).

To make it a bit easier to test tweaks for qoi, qoibench.c (in the experimental branch) can now descend into subdirectories, prints a grand total and has gained various options:

Usage: qoibench <iterations> <directory> [options]
Options:
    --nowarmup ... don't perform a warmup run
    --nopng ...... don't run png encode/decode
    --noverify ... don't verify qoi roundtrip
    --noencode ... don't run encoders
    --nodecode ... don't run decoders
    --norecurse .. don't descend into directories
Examples
    qoibench 10 images/textures/
    qoibench 1 images/textures/ --nopng --nowarmup

E.g. if you just want to check the overall compression ratio for qoi as fast as possible:
./qoibench 1 images/ --nowarmup --nopng --noverify --nodecode

[ikskuh]

This is very cool! I'll check it out and run my impl against it 😊

qoiconv.c currently has trouble encoding Ballerburg.png, OXO_emulated_portion.png, Sega-Mega-Drive-regional-lockout-error.png, thug_life_PNG15.png, silver_PNG17125.png, razor_blade_PNG17804.png, snapchat_PNG35.png, bone_PNG30.png, cursor_PNG68.png, milk_PNG12713.png, brain_PNG13.png, and cursor_PNG65.png

The reason is that they all are grayscale or grayscale with alpha, and stb_image.h decodes those as buffers of 1 and 2 channels respectively.

It was commented in #48 (comment) that the filename pngimg/сhevrolet_PNG46.png wasn't cooperating, and indeed the "c" is actually U+0441 CYRILLIC SMALL LETTER ES.

[phoboslab]

Ha, amazing! I fixed the name in the .tar archive. Thanks!

Decoding of the greyscale images mentioned above should be fixed with 525f32c

[notnullnotvoid]

I put together a collection of transparent sprites from our game to test the effectiveness of qoi at encoding transparent images. In general they don't have many hard alpha edges, they have mostly soft and messy alpha edges like you would expect from digitally painted assets. Here's the gist with the zip and a full benchmark result. I've also put a summary of the results below.

It performs worse than on opaque images, as expected, but still surprisingly well compared to PNG, except on a few pathological images with deliberately gradual and uniform alpha gradients, which are 40x larger than their PNG counterparts. These aren't artificial images, we actually use all of them in the game. That said, the worst compression ratio on one of these is 35.3%, which isn't terrible, but it still leaves me wanting for at least one DIFF opcode that includes alpha.

## Total for images-lance/backgrounds
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
spng        133.3      2234.3        126.36          7.54      7439     11.3%    13.3%     1.20x
stbi        160.0      1932.8        105.31          8.72     10982     16.7%    19.7%     1.78x
qoi          92.3        99.1        182.57        169.99      9143     13.9%    16.4%     1.48x

## Total for images-lance/misc
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
spng         15.2       136.3         92.44         10.30       321      5.9%     6.0%     1.04x
stbi         12.7       147.7        110.59          9.50       495      9.0%     9.2%     1.61x
qoi           4.4         5.6        322.11        249.40       503      9.2%     9.4%     1.63x

## Total for images-lance/particles
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
spng          3.5        62.4        122.48          6.94       176     10.4%    10.4%     1.11x
stbi          3.7        50.5        118.35          8.59       279     16.5%    16.5%     1.76x
qoi           2.1         2.5        204.76        176.79       340     20.1%    20.1%     2.14x

## Total for images-lance/screenshots
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
spng         37.0       753.6         56.03          2.75      3577     44.2%    44.2%     0.68x
stbi         46.6       463.9         44.53          4.47      5279     65.2%    65.2%     1.00x
qoi          16.9        20.1        122.78        103.15      3627     44.8%    44.8%     0.69x

## Total for images-lance/splats
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
spng         73.1       798.3         89.80          8.23      2064      8.0%     8.0%     1.05x
stbi         78.7       782.1         83.41          8.40      3067     12.0%    12.0%     1.56x
qoi          25.7        33.0        255.96        199.26      3331     13.0%    13.0%     1.70x

## Total for images-lance/sprites
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
spng         23.0       155.8         91.54         13.50       177      2.2%     2.2%     1.03x
stbi         13.9       194.4        151.21         10.83       287      3.5%     3.5%     1.67x
qoi           4.2         6.8        496.32        308.57       236      2.9%     2.9%     1.37x

# Grand total for images-lance
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
spng         43.8       510.7         92.66          7.95      1422      9.0%     9.1%     1.00x
stbi         46.2       482.8         87.82          8.41      2119     13.4%    13.6%     1.50x
qoi          16.6        20.8        244.70        195.55      2109     13.3%    13.6%     1.49x

[Wulf0x67E7]

@notnullnotvoid Tested @wbd73's idea of replacing QOI_OP_RGBA with QOI_OP_A on your images. It certainly is helping images with more complex alpha! Also, @phoboslab, wouldn't consecutive QOI_OP_As, which shouldn't normally happen and also don't decode into additional pixels, make for a good QOI_END_IMAGE?

EDIT: Another small tweak that allows for QOI_OP_RUN -> QOI_OP_A -> QOI_OP_RUN instead of QOI_OP_RUN -> QOI_OP_A -> QOI_OP_DIFF(0) -> QOI_OP_RUN.

EDIT: Re-run produced smaller performance loss (PC must have done something in the background the first time), updated qoi_op_a_tweak throughput.

Grand total for images-lance	decode ms	encode ms	decode mbps	encode mbps	size kb	rate
qoi_master	17.6	17.4	231.01	232.87	2109	13.6%
qoi_op_a	18.7	19.4	216.78	208.89	1987	12.8%
qoi_op_a_tweak	18.8	19.2	215.60	211.35	1948	12.5%

qoi_op_a.h.txt
qoi_op_a_tweak.h.txt

[notnullnotvoid]

Thanks for the info! On my end I tried stealing a couple 8-bit opcodes from QOI_OP_RUN and using them for 2-bit-per-channel and 4-bit-per-channel RGBA diffs. I got improvements on the lance image set, at the cost of some speed:

## Total for images-lance/backgrounds
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi          92.5        98.5        182.23        171.02      9143     13.9%    16.4%     1.48x
op_a         96.1        99.8        175.26        168.77      9142     13.9%    16.4%     1.48x
qoi4         93.3        99.2        180.57        169.85      9141     13.9%    16.4%     1.48x
qoi24       104.9        99.9        160.67        168.68      9142     13.9%    16.4%     1.48x
demo10       84.7       123.7        198.82        136.16      9249     14.1%    16.6%     1.50x
demo20       79.9       100.6        210.75        167.44      9055     13.8%    16.2%     1.46x

## Total for images-lance/misc
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           4.4         5.6        320.62        248.53       503      9.2%     9.4%     1.63x
op_a          4.6         5.9        308.22        238.83       434      7.9%     8.1%     1.41x
qoi4          4.3         5.7        322.88        244.65       458      8.4%     8.5%     1.49x
qoi24         4.7         5.8        296.13        242.73       445      8.1%     8.3%     1.44x
demo10        3.6         7.4        386.02        189.09       436      8.0%     8.1%     1.41x
demo20        3.9         5.9        359.80        238.78       434      7.9%     8.1%     1.41x

## Total for images-lance/particles
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           2.1         2.5        202.84        175.09       340     20.1%    20.1%     2.14x
op_a          2.2         2.5        200.86        173.64       283     16.8%    16.8%     1.78x
qoi4          2.2         2.5        199.86        173.48       286     16.9%    16.9%     1.80x
qoi24         2.4         2.6        178.97        167.00       270     16.0%    16.0%     1.70x
demo10        1.7         2.9        258.83        150.98       265     15.7%    15.7%     1.66x
demo20        1.9         2.7        229.20        162.95       287     17.0%    17.0%     1.80x

## Total for images-lance/screenshots
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi          16.8        20.2        123.59        102.45      3627     44.8%    44.8%     0.69x
op_a         16.7        20.5        124.31        101.04      3627     44.8%    44.8%     0.69x
qoi4         17.2        20.9        120.78         99.01      3627     44.8%    44.8%     0.69x
qoi24        17.3        20.8        119.93         99.79      3627     44.8%    44.8%     0.69x
demo10       14.0        25.2        148.08         82.25      4166     51.4%    51.4%     0.79x
demo20       15.4        22.5        134.91         92.31      3603     44.5%    44.5%     0.68x

## Total for images-lance/splats
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi          26.0        33.0        252.79        198.81      3331     13.0%    13.0%     1.70x
op_a         27.2        33.9        241.29        194.01      2829     11.0%    11.0%     1.44x
qoi4         26.8        34.2        245.12        192.18      2811     11.0%    11.0%     1.43x
qoi24        28.1        35.3        233.43        186.19      2665     10.4%    10.4%     1.36x
demo10       23.5        41.3        279.76        159.15      2709     10.6%    10.6%     1.38x
demo20       24.2        36.0        271.76        182.49      2829     11.0%    11.0%     1.44x

## Total for images-lance/sprites
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           4.3         6.8        491.41        310.04       236      2.9%     2.9%     1.37x
op_a          4.5         7.1        469.07        295.48       219      2.7%     2.7%     1.27x
qoi4          4.3         6.9        487.99        303.36       224      2.7%     2.7%     1.30x
qoi24         4.4         7.0        479.50        302.53       221      2.7%     2.7%     1.29x
demo10        3.9         9.4        543.42        223.83       196      2.4%     2.4%     1.14x
demo20        4.0         6.9        528.50        303.82       221      2.7%     2.7%     1.28x

# Grand total for images-lance
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi          16.8        20.8        242.42        195.31      2109     13.3%    13.6%     1.49x
op_a         17.5        21.3        232.59        190.84      1862     11.7%    12.0%     1.31x
qoi4         17.2        21.4        236.48        189.84      1857     11.7%    11.9%     1.31x
qoi24        18.2        21.9        222.68        185.19      1786     11.3%    11.5%     1.26x
demo10       15.0        26.1        270.25        155.69      1830     11.5%    11.8%     1.29x
demo20       15.3        22.4        264.57        181.49      1859     11.7%    12.0%     1.31x

qoi is the current master branch, qoi24 is using both the 2-bit and 4-bit diffs, and qoi4 is using only the 4-bit diff. It's not apparent from the totals, but qoi24 also lowered the size of the worst-case pathological file I mentioned from 40x PNG down to 20x PNG.

I also ran the main test suite to make sure there weren't any major regressions. It looks like it helps a bit on the icons, but not on other transparent images which mostly have hard alpha. As expected, none of the images got significantly larger.

## Total for images/icon_512
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           0.8         1.1        315.22        243.74        85      8.4%     8.4%     1.67x
op_a          0.8         1.1        322.02        240.00        79      7.8%     7.8%     1.56x
qoi4          0.8         1.1        322.12        243.41        80      7.8%     7.8%     1.57x
qoi24         0.9         1.1        304.68        238.29        78      7.6%     7.6%     1.53x
demo10        0.7         1.4        368.37        189.29        76      7.5%     7.5%     1.50x
demo20        0.7         1.1        383.09        243.25        79      7.8%     7.8%     1.56x

## Total for images/icon_64
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           0.0         0.0        251.95        210.65         4     28.7%    28.7%     1.21x
op_a          0.0         0.0        252.45        208.93         4     27.2%    27.2%     1.15x
qoi4          0.0         0.0        251.46        204.47         4     27.8%    27.8%     1.17x
qoi24         0.0         0.0        232.45        197.01         4     27.7%    27.7%     1.17x
demo10        0.0         0.0        334.37        161.84         4     27.6%    27.6%     1.16x
demo20        0.0         0.0        289.06        201.47         4     26.8%    26.8%     1.13x

## Total for images/photo_kodak
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           3.3         3.7        118.77        105.63       671     43.7%    58.3%     1.07x
op_a          3.2         3.6        121.89        108.27       671     43.7%    58.3%     1.07x
qoi4          3.4         3.7        116.40        106.33       671     43.7%    58.3%     1.07x
qoi24         3.4         3.7        116.28        106.14       671     43.7%    58.3%     1.07x
demo10        2.7         4.6        147.38         86.16       772     50.3%    67.0%     1.23x
demo20        3.0         3.8        130.86        104.38       669     43.6%    58.1%     1.07x

## Total for images/photo_tecnick
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi          11.7        14.1        122.72        102.21      2527     44.9%    59.9%     1.09x
op_a         11.4        13.9        126.22        103.85      2527     44.9%    59.9%     1.09x
qoi4         12.1        14.1        119.17        102.31      2527     44.9%    59.9%     1.09x
qoi24        12.2        14.0        117.58        102.83      2527     44.9%    59.9%     1.09x
demo10        9.3        16.4        154.81         87.80      2737     48.7%    64.9%     1.18x
demo20       10.7        14.3        135.13        100.37      2523     44.9%    59.8%     1.09x

## Total for images/photo_wikipedia
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           8.9        10.7        121.39        100.95      2102     49.6%    66.2%     1.19x
op_a          8.6        10.6        126.10        102.31      2102     49.6%    66.2%     1.19x
qoi4          9.0        10.7        119.86        101.09      2102     49.6%    66.2%     1.19x
qoi24         9.5        10.7        114.20        101.27      2102     49.6%    66.2%     1.19x
demo10        7.1        12.5        153.52         86.82      2289     54.0%    72.1%     1.30x
demo20        8.1        10.9        133.16         99.57      2099     49.6%    66.1%     1.19x

## Total for images/pngimg
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           8.3        10.5        218.94        171.61      1436     20.3%    20.3%     1.20x
op_a          7.9        10.6        227.83        170.33      1423     20.1%    20.2%     1.19x
qoi4          8.3        10.8        217.23        168.15      1420     20.1%    20.1%     1.19x
qoi24         8.7        11.1        207.32        163.47      1418     20.1%    20.1%     1.18x
demo10        6.8        13.2        265.36        137.06      1429     20.2%    20.2%     1.19x
demo20        7.1        11.1        253.46        163.11      1419     20.1%    20.1%     1.18x

## Total for images/screenshot_game
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           3.5         4.0        182.25        156.98       519     21.0%    24.5%     1.23x
op_a          3.4         4.0        185.86        157.80       519     21.0%    24.5%     1.23x
qoi4          3.5         4.0        180.83        156.93       519     21.0%    24.5%     1.23x
qoi24         3.6         4.1        177.49        156.14       519     21.0%    24.5%     1.23x
demo10        2.8         4.8        226.13        130.95       535     21.7%    25.3%     1.27x
demo20        3.0         4.1        208.63        154.04       512     20.7%    24.2%     1.22x

## Total for images/screenshot_web
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi          25.4        33.3        319.66        244.13      2649      8.3%     8.3%     0.94x
op_a         23.9        33.9        340.37        239.64      2651      8.4%     8.4%     0.94x
qoi4         25.0        33.6        324.44        241.70      2651      8.4%     8.4%     0.94x
qoi24        26.0        34.4        312.66        235.89      2653      8.4%     8.4%     0.94x
demo10       21.0        43.7        387.60        185.92      2680      8.4%     8.4%     0.95x
demo20       22.2        34.4        365.97        236.27      2627      8.3%     8.3%     0.94x

## Total for images/textures_photo
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           8.2         8.9        127.41        117.36      1981     48.4%    64.5%     1.07x
op_a          8.0         8.8        131.16        118.76      1981     48.4%    64.5%     1.07x
qoi4          8.5         8.8        123.46        118.69      1981     48.4%    64.5%     1.07x
qoi24         8.4         8.8        124.45        118.66      1981     48.4%    64.5%     1.07x
demo10        6.9        11.9        151.11         88.42      2506     61.2%    81.6%     1.36x
demo20        7.6         9.2        137.07        113.42      1984     48.5%    64.6%     1.07x

## Total for images/textures_pk
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           0.3         0.4        129.44        109.71        75     43.5%    56.5%     1.78x
op_a          0.3         0.4        131.42        110.31        75     43.5%    56.5%     1.78x
qoi4          0.3         0.4        127.33        111.43        75     43.5%    56.4%     1.78x
qoi24         0.4         0.4        125.53        110.78        75     43.5%    56.4%     1.78x
demo10        0.3         0.5        153.28         91.99        78     45.1%    58.5%     1.85x
demo20        0.3         0.4        150.08        107.91        73     42.0%    54.6%     1.72x

## Total for images/textures_pk01
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           0.8         1.0        166.30        131.48       178     35.2%    35.2%     1.09x
op_a          0.8         1.0        169.51        129.55       178     35.2%    35.2%     1.09x
qoi4          0.8         1.0        164.47        129.61       178     35.2%    35.2%     1.09x
qoi24         0.8         1.0        161.33        127.60       178     35.2%    35.2%     1.09x
demo10        0.6         1.2        215.45        112.21       180     35.6%    35.6%     1.10x
demo20        0.7         1.0        184.23        126.63       178     35.2%    35.2%     1.09x

## Total for images/textures_pk02
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           2.3         2.6        131.02        118.70       479     40.4%    40.4%     1.12x
op_a          2.3         2.6        134.35        118.63       479     40.4%    40.4%     1.12x
qoi4          2.4         2.6        128.76        115.94       478     40.3%    40.3%     1.12x
qoi24         2.5         2.7        123.31        114.34       478     40.3%    40.3%     1.12x
demo10        1.8         3.0        167.35         99.69       492     41.5%    41.5%     1.15x
demo20        2.1         2.8        147.11        107.11       482     40.7%    40.7%     1.13x

## Total for images/textures_plants
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           4.6         6.2        233.37        171.80       922     22.2%    22.2%     1.09x
op_a          4.5         6.3        238.77        169.68       922     22.2%    22.2%     1.09x
qoi4          4.6         6.3        230.69        167.97       920     22.1%    22.1%     1.09x
qoi24         4.9         6.6        218.30        161.84       920     22.1%    22.1%     1.09x
demo10        3.8         7.9        279.30        134.17       957     23.0%    23.0%     1.14x
demo20        4.1         6.4        257.10        166.21       920     22.1%    22.1%     1.09x

# Grand total for images
        decode ms   encode ms   decode mpps   encode mpps   size kb   vs rgba   vs raw   vs disk
qoi           2.6         3.1        180.24        149.26       463     25.6%    28.2%     1.18x
op_a          2.5         3.1        185.55        149.51       462     25.5%    28.1%     1.18x
qoi4          2.6         3.1        178.13        148.26       462     25.5%    28.1%     1.18x
qoi24         2.7         3.2        172.95        146.52       462     25.5%    28.1%     1.18x
demo10        2.1         3.8        222.34        122.70       484     26.7%    29.5%     1.23x
demo20        2.3         3.2        204.34        144.44       459     25.4%    28.0%     1.17x

Here's the exact code (diff) for these benchmarks. I would strongly advocate for including something like this in the spec, although I understand it's not free gains, as there's a performance and complexity cost. I will point out however, the impact on opaque images is significantly smaller than on transparent images - there's no noticeable cost for encoding opaque images (makes sense), and only a small cost for decoding opaque images, which could probably be mitigated by reordering branches in the decoder.

Edit: included results for demo10, demo20, and qoi-op-a.

[nigeltao]

Any chance you could do the images-lance numbers for demo10 (nigeltao/qoi2-bikeshed#22)?

[notnullnotvoid]

I've added the numbers for demo10. It seems to do a bit better at encoding transparent images, at the cost of being a bit worse at opaque images, and significantly worse in the case of grainier images. Encoding is significantly slower, and decoding is significantly faster. How feasible do you think it would be to apply the decoder optimizations in demo10 to the other encodings? That seems like it could be the best of both worlds.

[nigeltao]

How feasible do you think it would be to apply the decoder optimizations in demo10 to the other encodings?

That's essentially what demo2e is: #71 (comment)

[nigeltao]

replacing QOI_OP_RGBA with QOI_OP_A

Replacing QOI_OP_RGBA surprises me, especially if, in some sense, your QOI_OP_A combines with the next op - it's no longer the case that each op emits at least one pixel.

Here's my numbers for tweaking phoboslab/master (commit aefa0f7, 2021-12-17) to add my stand-alone QOI_OP_A alongside QOI_OP_RGBA (QOI_OP_A replaces "QOI_OP_RUN with run length 62", like as in #71 (comment)):

        decode ms   encode ms   decode mpps   encode mpps   size kb    rate

# Grand total for images
qoi-aefa0f7:  4.8         5.6         97.28         83.08       463   25.6%
qoi-op-a:     4.7         4.8         98.25         96.80       462   25.5%

# Grand total for images-lance
qoi-aefa0f7: 35.4        32.4        114.61        125.29      2109   13.3%
qoi-op-a:    35.8        29.1        113.38        139.46      1862   11.7%

337a338
> #define QOI_OP_A      0xfd /* 11111101 */
439c440
< 			if (run == 62 || px_pos == px_end) {
---
> 			if (run == 61 || px_pos == px_end) {
489a491,498
> 				else if (
> 					(px.rgba.r == px_prev.rgba.r) &&
> 					(px.rgba.g == px_prev.rgba.g) &&
> 					(px.rgba.b == px_prev.rgba.b)
> 				) {
> 					bytes[p++] = QOI_OP_A;
> 					bytes[p++] = px.rgba.a;
> 				}
593a603,605
> 			}
> 			else if (b1 == QOI_OP_A) {
> 				px.rgba.a = bytes[p++];

qoi-op-a-alongside.h.txt