Fix integration test expected values for cuda sm_86 (Mar 15 regressions) - #48168
Open
ydshieh wants to merge 9 commits into
Open
Fix integration test expected values for cuda sm_86 (Mar 15 regressions)#48168ydshieh wants to merge 9 commits into
ydshieh wants to merge 9 commits into
Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tests Add cuda sm_86 expected values for test_multi_speaker_smart_voice, test_zero_shot_voice_cloning, test_multi_speaker_voice_cloning, and test_batched_inference. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…r cuda sm_86 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…sm_86 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
CI recapDashboard: View test results in Grafana |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
- LlamaIntegrationTest::test_llama_3_1_hard: add ("cuda", 8) entry;
sm_86 generates "radical social and political" vs older hw "radical
political and social"
- SeedOssIntegrationTest::test_model_36b_eager/sdpa: convert to
Expectations; sm_86 generates different second-prompt continuation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: aya_vision, cwm, glm, helium, higgs_audio_v2, llama, mistral, seed_oss, swiftformer, swin2sr, voxtral, voxtral_realtime |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes stale expected values in integration tests for models that regressed on March 15, 2026 on CUDA sm_86 (A10G/A100) hardware.
Models fixed:
("cuda", 8)expected text intest_small_model_integration_generate_chat_template("cuda", 8)expected logits intest_cwm_sliding_window_long_sequence("cuda", 8)expected text intest_model_9b_eager("cuda", 8)expected text intest_model_2b("cuda", 8)expected token tensors for 4 integration tests("cuda", 8)logit value at index 13 intest_model_7b_logits("cuda", 8)expected slice intest_inference_fp16Expectationswith("cuda", 8)entry intest_inference_image_classification_head("cuda", 8)expected outputs for 4 failing integration tests("cuda", 8)expected output fortest_batched_longformRemaining (not fixed in this PR):
llama::test_llama_3_1_hard— gated model, needs HF_TOKEN on runnerseed_oss::test_model_36b_eager/sdpa— 36B model, OOM on single A10GNote
Internvl, Mistral3, and Qwen2.5-Omni regressions from the same date were already fixed in separate PRs (#48153, #48161, #48164).
🤖 Generated with Claude Code