Negative input values resulting in unexpected output values on some CPUs

[mjtitchener-fn]

I've been investigating a report of negative input values resulting in unexpected output values on some CPUs.
We have been able to reproduce the issue on one of our Windows machines and it appears to be happening in the OCIO library.

Reproduction steps:

1. Using input values R: 0.00532 G: 0.00075 B: -0.00293
2. Transform from ACEScg to ACEScc.

Actual result:

R: 0.11312 G: -0.07889 B: 1.50000

Expected result:

R: 0.11312 G: -0.07889 B: -0.31065

Here's a visual example of the differences seen. Top is actual result, bottom is expected result.

The user experiences this issue on machines with Dual Intel Xeon E5-2680 v2 @ 2.80GHz but doesn’t on machines with Dual Intel Xeon E5-2640 v3 @ 2.60GHz

The machine we can reproduce the issue on has Dual Intel Xeon E5-2640 0 @ 2.50GHz

One difference between the CPUs that stood out to me is Intel AVX (reproducible) vs. Intel AVX2 (not reproducible). Not sure if this is relevant though.

So far we’ve only seen this on Windows. We can only reproduce this in our latest Nuke release (14.0) but the report says they’ve encountered it since Nuke was upgraded to use OCIO v2.
It’s reproducible with the ACES 1.2 and ACES Studio configs.

Workaround:

I've found you can workaround the issue by setting the environment variable OCIO_OPTIMIZATION_FLAGS=0 to turn optimization off, or OCIO_OPTIMIZATION_FLAGS=20479 to set optimization to lossless.
In the case where the ACES Studio config is being used switching from ACEScc to ACEScct also fixes the issue.

[doug-walker]

@mjtitchener-fn , I'm not able to reproduce the bug.

I'm uploading a short Python script. Could you please run it on the machine that shows the problem and post the result here?
bug_1764.py.zip

[mjtitchener-fn]

Sure, no problem @doug-walker.
I'm on vacation until Feb 27th, so I'll do this when I'm back.

[mjtitchener-fn]

@doug-walker Result from your Python script below. I've also attached the CTF file that was generated.

bug_1764.ctf.zip

Output was R: 0.113119, G: -0.078894, B: 1.500000

The bug was reproduced.

Wrote color transform file.

Please send in the file "bug_1764.ctf" from this directory.

Please send in the following info.

<MatrixTransform direction=forward, fileindepth=unknown, fileoutdepth=unknown, matrix=0.6954522413574519 0.1406786964702942 0.1638690621722541 0 0.04479456337203772 0.8596711184564216 0.0955343181715404 0 -0.005525882558113544 0.004025210305978659 1.001500672252135 0 0 0 0 1, offset=0 0 0 0>
[0.0033251785207539797, 0.0006031448720023036, -0.0029607759788632393]

<RangeTransform direction=forward, fileindepth=unknown, fileoutdepth=unknown, minInValue=0, minOutValue=0>
[0.0033251785207539797, 0.0006031448720023036, 0.0]

<MatrixTransform direction=forward, fileindepth=unknown, fileoutdepth=unknown, matrix=1.451439316145665 -0.2365107468937402 -0.2149285692519253 0 -0.07655377339602057 1.176229699833573 -0.09967592643755223 0 0.008316148425697719 -0.006032449791021028 0.9977163013653234 0 0 0 0 1, offset=0 0 0 0>
[0.004683644510805607, 0.00045488192699849606, 2.4014236259972677e-05]

<Lut1DTransform direction=forward, fileoutdepth=unknown, interpolation=default, inputhalf=1, outputrawhalf=0, hueadjust=0, length=65536, minrgb=[0 0 0], maxrgb=[1 1 1]>
[0.2543649673461914, 0.15113228559494019, 1.0]

<RangeTransform direction=forward, fileindepth=unknown, fileoutdepth=unknown, minInValue=0, maxInValue=1, minOutValue=-0.36, maxOutValue=1.5>
[0.11311882734298706, -0.07889395952224731, 1.5]


Thanks for your help!

[doug-walker]

Thank you @mjtitchener-fn, that helped narrow down where the problem is. I wrote another script to explore the relevant area. I'm attaching it here. Please run it just like the previous one and post the output. Thanks again!
bug_1764_lut1d.py.zip

[mjtitchener-fn]

No problem @doug-walker I'm glad it helped. Output from bug_1764_lut1d.py is below and I've attached the related CTF.

bug_1764_lut1d.ctf.zip

Output was R: 0.113119, G: -0.078894, B: 1.500000

The bug was reproduced.

Wrote color transform file.

Please send in the file "bug_1764_lut1d.ctf" from this directory.

Please send in the following info.

Index      In Value           Out Value
    0              0.0     0.0008345198
    1   1.96695328e-06    0.00359982532
    2   1.07288361e-06      0.002364235
    3   1.13248825e-06    0.00244781817
    4   7.15255737e-07    0.00186017284
    5   7.74860382e-07    0.00194471597
    6   8.34465027e-07    0.00202884292
    7   8.94069672e-07    0.00211292109
    8   9.53674316e-07    0.00219699927
    9    1.1920929e-06     0.0025309741
   10   1.43051147e-06    0.00286262087
   11   1.66893005e-06    0.00319187157
   12   9.53674316e-07    0.00219699927
   13    1.1920929e-06     0.0025309741
   14   1.43051147e-06    0.00286262087
   15   1.66893005e-06    0.00319187157
   16   3.05175781e-05      0.031538289
   17   4.57763672e-05      0.049472101
   18   3.05175781e-05      0.031538289
   19   4.57763672e-05      0.049472101
   20   0.000122070312     0.0928952172
   21   0.000183105469      0.110846005
   22   0.000122070312     0.0928952172
   23   0.000183105469      0.110846005
   24    0.00048828125       0.15426904
   25   0.000732421875      0.172219723
   26    0.00048828125       0.15426904
   27   0.000732421875      0.172219723
   28      0.001953125      0.215642929
   29     0.0029296875      0.233593479
   30      0.001953125      0.215642929
   31     0.0029296875      0.233593479
   32   5.96046448e-08   0.000920949038
   33   5.96046448e-08   0.000920949038
   34   2.38418579e-07    0.00117911759
   35   2.38418579e-07    0.00117911759
   36   9.53674316e-07    0.00219699927
   37   9.53674316e-07    0.00219699927
   38   3.81469727e-06    0.00604903558
   39   3.81469727e-06    0.00604903558
   40   1.52587891e-05     0.0187853221
   41   1.52587891e-05     0.0187853221
   42   6.10351562e-05     0.0622083507
   43   6.10351562e-05     0.0622083507
   44   0.000244140625      0.123582229
   45   0.000244140625      0.123582229
   46     0.0009765625       0.18495591
   47     0.0009765625       0.18495591
   48   5.96046448e-08   0.000920949038
   49   5.96046448e-08   0.000920949038
   50   2.38418579e-07    0.00117911759
   51   2.38418579e-07    0.00117911759
   52   9.53674316e-07    0.00219699927
   53   9.53674316e-07    0.00219699927
   54   3.81469727e-06    0.00604903558
   55   3.81469727e-06    0.00604903558
   56   1.52587891e-05     0.0187853221
   57   1.52587891e-05     0.0187853221
   58   6.10351562e-05     0.0622083507
   59   6.10351562e-05     0.0622083507
   60   0.000244140625      0.123582229
   61   0.000244140625      0.123582229
   62     0.0009765625       0.18495591
   63     0.0009765625       0.18495591
   64   1.01412048e+31              1.0
   65   4.05648192e+31              1.0
   66   1.62259277e+32              1.0
   67   6.49037107e+32              1.0

Thanks for your help!

[doug-walker]

Thanks @mjtitchener-fn , that narrowed it down a lot. This is a very surprising result but apparently on your test machine, the generation of a half-domain identity Lut1D is producing bogus values for the first 1024 entries. Surprisingly, this seems to imply a problem with Imath's half.h.

Here's the place in the OCIO code where the values for the identity Lut1D are generated:
https://github.com/AcademySoftwareFoundation/OpenColorIO/blob/main/src/OpenColorIO/ops/lut1d/Lut1DOpData.cpp#L56

It initializes a half object, sets an integer value, and then casts it to a float. There doesn't seem like much could be going wrong on the OCIO side, so it seems like Imath must not be doing the half to float conversion properly. I searched the open issues on the Imath GitHub site, but did not find a known bug.

The first 1024 values represent denormalized numbers, so the algorithm for converting them to float is a special case relative to the normal half-float values, though it's not apparent why that would matter (they are not denormalized when represented as 32-bit floats, so FTZ/DAZ floating-point unit registers should not matter).

Imath does provide some CMake options that control how the half to float conversion is done. What I would like to propose is that someone build OCIO locally on that machine using the three different options the Imath library provides for half to float conversion and re-run the Python script bug_1764_lut1d.py.

OCIO sets those options in OpenColorIO/share/cmake/modules/FindImath.cmake. So someone could edit this function, re-run CMake, and build OCIO. Make sure to use the -DOCIO_INSTALL_EXT_PACKAGES=ALL option so that it is building Imath from source (or build it separately, and point OCIO at your copy).

Currently the OCIO FindImath.cmake sets -DIMATH_HALF_USE_LOOKUP_TABLE=OFF. So the next option would be to set that on. And then the last option would be to add "-m16fc" to the Imath_CXX_FLAGS. More detail on the three options is available here:
https://imath.readthedocs.io/en/latest/classes/half_conversion.html

Mark, is this something that someone on your side could do? I'd be happy to chat if more info is needed about what I'm suggesting.

For reference, I'm attaching the correct version of the ctf file generated by the above Python script. The first 67 values of the Lut1D are also the correct values for the middle column of numbers that the script prints. These are the first 67 values of a half float number.
bug_1764_lut1d.ctf.zip

[mjtitchener-fn]

That's really interesting. Thanks for your investigation into this and the detailed reply @doug-walker.

The further tests you've described make sense. I haven't built OCIO on Windows before (or anything else for that matter) but there should be plenty of people here that can help.

Edit: To be clearer I have built OCIO many times on macOS so I'm familiar with the build process/options but I've never done it on Windows.

It might take a little while but I'll let you know the results when I can.

[carolalynn]

Hey @mjtitchener-fn - did this get resolved to your satisfaction? Can we close this out?

[doug-walker]

I noticed that ilmbase has had some bug fixes to fix incorrect values on some CPUs, so updating to the latest version of that may fix this. I don't have a way of testing that though.

[mjtitchener-fn]

Hey @carolalynn22 this hasn't been resolved as yet. We've had reports of it from a handful of customers. It only happens on older CPUs (typically found in their render farms). Unfortunately one of the machines I could reproduce it on went out of action recently. I've found one other that I can reproduce the problem on, so I can see if I can get some time on there.
@doug-walker I also noticed a bug fix in the recent Imath release that I thought might be related, I assume you're referring to the following?
Fix half to float giving wrong results on older x86_64 CPUs on Windows

[doug-walker]

Yes, that's the one I was thinking of Mark (sorry, meant to write imath rather than ilmbase).

[mjtitchener-fn]

Good news! 🎉

It appears this bug is being caused by the issue in Imath that was fixed here AcademySoftwareFoundation/Imath#358 (in 3.1.10).

I've finally been able to test a build of Nuke with a version of OCIO using Imath 3.1.12 and the artefacts no longer appear.

@doug-walker OCIO currently recommends (and downloads) Imath 3.1.6. Perhaps this should be bumped to the latest version of Imath, 3.1.12 (or at least 3.1.10)?