f16 performance is abysmal, u8, u16, f32 and casting f16 to f32 performance is excellent

[adworacz]

Zig Version

0.12.0-dev.3518+d2be725e4

Steps to Reproduce and Observed Behavior

I'm working on an open source video processing library that supports video stored in various bit depths. This specifically includes u8, u16, f16, and f32 bit depths.

While writing code, I was finding a massive performance disparity between f16 and all other bit depths. What's even more interesting is that I am getting much better performance if I manually cast my f16 data to f32, perform an operation, and then manually cast back to f16.

I'd expect that the compiler would effectively do this conversion/casting for me.

The following code shows a good example of what I'm talking about:

//const T = u8;
//const T = u16;
//const T = f16;
//const T = f32;
//const T = @Vector(32, u8);
//const T = @Vector(16, u16);
const T = @Vector(16, f16);
//const T = @Vector(16, f32);

export fn clamp(c: T, a1: T, a2: T, a3: T, a4: T, a5: T, a6: T, a7: T, a8:T) T {
    const min = @min(a1, a2, a3, a4, a5, a6, a7, a8);
    const max = @max(a1, a2, a3, a4, a5, a6, a7, a8);

    return @max(min, @min(c, max));
}

The vector typed versions best demonstrate the problem.

Using Godbolt (with ReleaseFast), the same code takes the following number of instructions:

• Vector u8: 25
• Vector u16: 25
• Vector f16: 2304
• Vector f32: 56

That's a massive difference in the number of instructions for f16, and it seems like the optimizer is really missing out on the ability to convert f16 to f32 once, run an operation, and then convert back.

Full Godbolt link: https://zig.godbolt.org/z/baP7P6s8o

I have lots of other examples of poor f16 behavior in different video processing algorithms, this is just a nice succinct example.

As a point of reference - this same code is able to process u8, u16, and f32 bit depths at 1920x1080 video on a cheap laptop in several hundred frames per second, but f16 drops that down to 5-10 frames per second.

Expected Behavior

f16 performance to meet or exceed f32 performance.

[xdBronch]

im guessing this is an llvm problem? the ir looks reasonable and if you tell it to target a cpu with avx512fp16 it generates what you expect https://zig.godbolt.org/z/nEGrKsbsb

[adworacz]

Interesting - I don't have an AVX512 CPU, so I haven't been able to test that case, but you have a point in that CPUs with native f16 support might be fine, but CPUs that require something like F16C instructions to convert between f16 and f32 may need some work.

[Pyrolistical]

It wouldn't be bit perfect to substitute f16 operations by upcasting to f32.

I suspect why f16 is so slow on hardware that doesn't have native support is the fallback is to do all f16 calculations using integers. This allows one to calculate bit perfect f16s, but with a huge performance hit.

An application can make the f16 to f32 upcast tradeoff if they choose, but a compiler could never do that.

[adworacz]

You say that, but the compiler is absolutely converting f16 to f32 and back again. It's just doing it a lot in this code.

You can see this in my Godbolt link. There's over 100 vcvtph2ps operations and over 100 vcvtps2ph operations. So it's constantly converting back and forth between f16 and f32.

[rohlem]

It wouldn't be bit perfect to substitute f16 operations by upcasting to f32.

@Pyrolistical Would you consider it acceptable if the compiler introduced such conversions in @setFloatMode(.optimized)? (It currently doesn't FWICT.)
Or could/should we add a new, even laxer FloatMode?

[Pyrolistical]

I don't have a horse in this race so I will defer

[andrewrk]

The compiler is allowed to use f32 for f16 operations with @setFloatMode(.optimized). It is not allowed to otherwise.

Whether LLVM figures out to do this optimization... I wouldn't count on it. Likely Zig will need to do that even in the LLVM backend.