Fix sanitizers reports about octahedral tangents in RenderingServer

[nklbdev]

I found three places in RenderingServer with wrong type casting from float clamped by range [0..65535] to int to int16_t[-32768..32767] instead uint16_t[0..65535].
In rarely situations it leads conversion from big positive number to negative.
In some cases it can break the check Linux / Editor with doubles and GCC sanitizers.

[akien-mga]

Looks great! Could you update the commit message to be more explicit (e.g. like I did with the PR title)?

Would this happen to be this issue? #78749

[RedworkDE]

No, that issue is about a free without malloc / memory corruption issue, while the linked CI fail is an index out of bounds access to a vector.

[nklbdev]

Looks great! Could you update the commit message to be more explicit (e.g. like I did with the PR title)?

I fixed the commit message as you requested.))

[akien-mga]

LGTM.

[akien-mga]

For the record, even when the CI step doesn't error out, the following errors are reported which relate to the code here:

servers/rendering_server.cpp:406:21: runtime error: nan is outside the range of representable values of type 'short int'
servers/rendering_server.cpp:407:21: runtime error: nan is outside the range of representable values of type 'short int'
servers/rendering_server.cpp:426:22: runtime error: nan is outside the range of representable values of type 'short int'
servers/rendering_server.cpp:427:22: runtime error: 32768.5 is outside the range of representable values of type 'short int'

This change fixes the last one, but the first three errors about nan are still raised. Could be interesting to look into in a follow-up PR, or this one if you want and the fix is similarly trivial.

[nklbdev]

Okay, now I'll try to find the causes of the rest of the errors.

[nklbdev]

I think I've found the cause of the rest of the errors, but I'm not sure if I substituted the correct values when forming the mesh. Please correct me as I haven't worked much with 3D scenes and coordinates.

I found the problem place by setting a breakpoints with a conditions, and then examining the call stack

[clayjohn]

This looks great!

The defaults in SpriteBase3D just need to be normalized. They won't be used as-is as the values are always overwritten, so moving from 0,0,0 to any normalized vector is fine.

[nklbdev]

This looks great!

The defaults in SpriteBase3D just need to be normalized. They won't be used as-is as the values are always overwritten, so moving from 0,0,0 to any normalized vector is fine.

I'm very happy) Does this mean that the current state of the code is valid?

[clayjohn]

This looks great!
The defaults in SpriteBase3D just need to be normalized. They won't be used as-is as the values are always overwritten, so moving from 0,0,0 to any normalized vector is fine.

I'm very happy) Does this mean that the current state of the code is valid?

Yes, as far as I can tell!

[akien-mga]

Thanks!

[YuriSizov]

Cherry-picked for 4.1.1.