fixing pytest skip for only test_floatx.py

[HDCharles]

Summary: trying to fix fbcode CI, pytest skip is acting flaky

Test Plan: see CI

Reviewers:

Subscribers:

Tasks:

Tags:

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1251

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Unrelated Failures

As of commit 0433d7a with merge base e41ca4e ():

NEW FAILURE - The following job has failed:

• Run Regression Tests / test (CUDA 2.5, linux.g5.12xlarge.nvidia.gpu, torch==2.5.0 --index-url https://download.pytorch.o... / linux-job (gh)
  RuntimeError: Command docker exec -t d80e8eb25bd5c9e2195e4b272d318fa7ea7dcbb1d0f6eaae358890b125dbc014 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Run Regression Tests / test (CPU Nightly, linux.4xlarge, --pre torch --index-url https://download.pytorch.org/whl/nightl... / linux-job (gh) (trunk failure)
• Run Regression Tests / test (CUDA Nightly, linux.g5.12xlarge.nvidia.gpu, --pre torch --index-url https://download.pytorc... / linux-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@HDCharles has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[msaroufim]

should we remove pytest from the whole codebase? i still see it in loads of places

base) ➜  ao git:(main) grep -r "pytest.skip" .
./test/float8/test_fsdp2/test_fsdp2.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/float8/test_fsdp2/test_fsdp2.py:    pytest.skip("Unsupported CUDA device capability version", allow_module_level=True)
./test/float8/test_fsdp2/test_fsdp2_fp8_comm_only.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/float8/test_fsdp2/test_fsdp2_fp8_comm_only.py:    pytest.skip("Unsupported CUDA device capability version", allow_module_level=True)
./test/float8/test_fsdp.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/float8/test_numerics_integration.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/float8/test_compile.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/float8/test_base.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/float8/test_base.py:            pytest.skip()
./test/float8/test_fsdp_compile.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/float8/test_dtensor.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/dora/test_dora_fusion.py:    pytest.skip("requires Python >= 3.11", allow_module_level=True)
./test/dora/test_dora_fusion.py:        pytest.skip("Either store_acc or epilogue_norm must be True")
./test/dora/test_dora_layer.py:    pytest.skip("requires Python >= 3.11", allow_module_level=True)
./test/prototype/test_autoround.py:    pytest.skip("AutoRound is not available", allow_module_level=True)
./test/prototype/mx_formats/test_mx_linear.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/prototype/mx_formats/test_mx_linear.py:            pytest.skip("CUDA capability >= 8.9 required for float8 in triton")
./test/prototype/mx_formats/test_mx_linear.py:            pytest.skip("CUDA capability >= 8.9 required for float8 in triton")
./test/prototype/mx_formats/test_mx_tensor.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/prototype/mx_formats/test_mx_tensor.py:            pytest.skip("unsupported configuration")
./test/prototype/mx_formats/test_mx_tensor.py:        pytest.skip("unsupported configuration")
./test/prototype/mx_formats/test_mx_tensor.py:            pytest.skip("CUDA capability >= 8.9 required for float8 in triton")
./test/prototype/test_low_bit_optim.py:                pytest.skip("FP8 CUDA requires PyTorch >= 2.4")
./test/prototype/test_low_bit_optim.py:                pytest.skip("FP8 CUDA requires compute capability >= 8.9")
./test/prototype/test_quantized_training.py:    pytest.skip("Requires torch>=2.4", allow_module_level=True)
./test/prototype/test_awq.py:        pytest.skip("uint4 not supported on cpu")
./test/dtypes/test_bitnet.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/dtypes/test_affine_quantized_float.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/dtypes/test_uint2.py:    pytest.skip("Unsupported PyTorch version", allow_module_level=True)
./test/quantization/test_galore_quant.py:    pytest.skip("triton is not installed", allow_module_level=True)
./test/test_ops.py:    pytest.skip("Skipping the test in fbcode since we don't have TARGET file for kernels")
./test/test_ops.py:    pytest.skip("torchao.ops not available")
./test/kernel/test_fused_kernels.py:    pytest.skip("triton is not installed", allow_module_level=True)
./test/kernel/test_galore_downproj.py:    pytest.skip("triton is not installed", allow_module_level=True)
(base) ➜  ao git:(main) 

[HDCharles]

@msaroufim I spent multiple hours yesterday trying to get rid of as much of it as i can. my first attempts were more ambitious i.e.
#1245
#1243

but I basically ran into annoying to bypass issues and pared back my goal until it was just this single file to fix fbcode CI.
to actually get rid of all pytest.skip i think we'd need to get rid of pytest as the test runner which basically means getting rid of everything. pytest skip, pytest parametrize, pytest fixture....etc.

its definitely doable but probably not between the hours of 1-3am, at least not by me.
I tried getting chatgpt to do it for me but the best way to do it would be mostly using torch.testing._internal.common_utils for the test runner and parametrize and the solutions they recommended didn't actually work.