BUG: `TestDataFrameSetItem.test_setitem_dtype[uint64]` and `TestSeriesNLargestNSmallest.test_nlargest_nullable[UInt8]` fail on aarch64

[ggardet]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Run tests on openSUSE Tumbleweed aarch64

Issue Description

TestDataFrameSetItem.test_setitem_dtype[uint64] and TestSeriesNLargestNSmallest.test_nlargest_nullable[UInt8] fail on aarch64:

[ 7109s] =================================== FAILURES ===================================
[ 7109s] _______________ TestDataFrameSetItem.test_setitem_dtype[uint64] ________________
[ 7109s] [gw1] linux -- Python 3.11.3 /usr/bin/python3.11
[ 7109s] 
[ 7109s] self = <pandas.tests.frame.indexing.test_setitem.TestDataFrameSetItem object at 0xffff55491b50>
[ 7109s] dtype = 'uint64'
[ 7109s] float_frame =                    A         B         C         D
[ 7109s] Pn96zxFOj3 -0.530699  0.350158 -0.311547  0.707319
[ 7109s] 2cjtkii5Zw -0.41....252126  1.341911
[ 7109s] ts3SGMnjfe -0.077150  2.134881  1.451667 -0.589377
[ 7109s] 43mi5CIWi3 -0.751679  1.275031 -0.521072  0.631581
[ 7109s] 
[ 7109s]     @pytest.mark.parametrize(
[ 7109s]         "dtype", ["int32", "int64", "uint32", "uint64", "float32", "float64"]
[ 7109s]     )
[ 7109s]     def test_setitem_dtype(self, dtype, float_frame):
[ 7109s]         arr = np.random.randn(len(float_frame))
[ 7109s]     
[ 7109s] >       float_frame[dtype] = np.array(arr, dtype=dtype)
[ 7109s] E       RuntimeWarning: invalid value encountered in cast
[ 7109s] 
[ 7109s] pandas/tests/frame/indexing/test_setitem.py:66: RuntimeWarning
[ 7109s] __________ TestSeriesNLargestNSmallest.test_nlargest_nullable[UInt8] ___________
[ 7109s] [gw1] linux -- Python 3.11.3 /usr/bin/python3.11
[ 7109s] 
[ 7109s] self = <pandas.tests.series.methods.test_nlargest.TestSeriesNLargestNSmallest object at 0xffff3ed83b10>
[ 7109s] any_numeric_ea_dtype = 'UInt8'
[ 7109s] 
[ 7109s]     def test_nlargest_nullable(self, any_numeric_ea_dtype):
[ 7109s]         # GH#42816
[ 7109s]         dtype = any_numeric_ea_dtype
[ 7109s] >       arr = np.random.randn(10).astype(dtype.lower(), copy=False)
[ 7109s] E       RuntimeWarning: invalid value encountered in cast
[ 7109s] 
[ 7109s] pandas/tests/series/methods/test_nlargest.py:220: RuntimeWarning

Expected Behavior

Tests should pass.

Installed Versions

2.0.2 from openSUSE Tumbleweed package

[ggardet]

Cherry picking 8023225#diff-3260c8ed438c0a25d389ff455bc6760c58f551d79f8f208bbd2a56b73fd763a6 and 8023225#diff-3483b62771f6155ebc3ae9a1eb87afb662f42ce1e341e6c06b59c71756cae1bc fix the test issue.
Could we get this fix backported to 2.x branch, please?

[lithomas1]

That PR was kinda reverted. I'll try to manual backport the test fixes though, sometime soonish.