testIntegration > Autocomplete.testAutocomplete fails at Autocomplete.kt:42 (latent bug masked by FFmpeg 7.1 404)

[santhiprakash]

Summary

:testIntegration > Autocomplete.testAutocomplete() fails at Autocomplete.kt:42 on every PR that successfully passes the Setup FFmpeg step. The test has been latent for ~11 months — it was added on 2025-09-19 via PR #7690 ("test: use intellij-ide-starter for e2e tests", CON-3723) but has effectively never run to completion in CI because the AnimMouse/setup-ffmpeg@v1 version: 7.1 pin in .github/actions/run-jetbrains-tests/action.yml returns 404 first (issue #13164).

Failure detail

• Test class: com.github.continuedev.continueintellijextension.Autocomplete
• Test method: testAutocomplete()
• Failing assertion (Autocomplete.kt:42): assertTrue(text.contains("TEST_LLM_RESPONSE_0"))
• Error: expected: <true> but was: <false> (org.opentest4j.AssertionFailedError)
• Duration to failure: 1m04s (the test runs to completion; the assertion fails at the end)
• IDE under test: IC-243.21565.193 (IntelliJ Community 2024.3)
• Test framework: intellij-ide-starter (TestFrameworkType.Starter), pinned in extensions/intellij/build.gradle.kts
• JUnit: Jupiter 5.7.1
• Artifact (video + test reports): jb-failure-report from run 32468648181 — video at video/testAutocomplete_recording_2026_21_08_09_43_36.avi

Why this surfaces now

Issue #13164 reported that the FFmpeg 7.1 pin returns 404. PR #13169 fixes it by pinning to 8.1 + adding a preflight check. Once the FFmpeg step passes on the PR head (2e6fadcbd), testIntegration runs and this assertion fires.

Why this is pre-existing (not caused by the FFmpeg fix)

• The PR diff is only .github/actions/run-jetbrains-tests/action.yml (parameterized version + preflight check). No JetBrains plugin source is touched.
• The test framework, IntelliJ version, and TestLLM fixture have not changed since 2025-09-19.
• The test was effectively dead code in CI for the entire window between test: use intellij-ide-starter for e2e tests #7690 and [CI/CD] JetBrains tests fail: pinned FFmpeg 7.1 artifact no longer published by BtbN, surfacing as a cryptic tar error #13164: main-branch version: 7.1 always 404s the FFmpeg step first, so testIntegration is never reached.
• The closest prior PR (fix(ci): bump pinned FFmpeg to a version BtbN still publishes #13165 by ScrewTSW) was self-closed on 2026-08-19 before CI ran.

Likely root causes (not exhaustive)

1. Autocomplete never fires in the new IDE under test. The tab() key after enterText("TEST_USER_MESSAGE_0") + space() may not trigger Continue's inline completion — could be a default-settings issue (autocomplete disabled) or a keystroke routing issue.
2. TestLLM not loaded — the test fixture config in extensions/intellij/src/testIntegration/kotlin/com/github/continuedev/continueintellijextension/test-continue/config.json (inferred from CONTINUE_GLOBAL_DIR in build.gradle.kts) may not be picked up by the 2024.3 IDE under test.
3. Prompt-to-response mismatch — TestLLM.findResponse() requires the regex [^0-9]*TEST_USER_MESSAGE_(\d+)[^0-9]*. If the IDE injects surrounding text that breaks the digit boundaries, findResponse returns undefined and the stream yields PROMPT: <full prompt> instead of TEST_LLM_RESPONSE_0.
4. Timing — the test waits 2.seconds after typing, which may not be enough for the autocomplete provider to compute a response under the new test framework.

Suggested next step

Pick one:

• (A) Investigate and fix the test (preferred — the test should be a working CI gate).
• (B) Mark the test @Disabled with a TODO referencing this issue until a fix lands.
• (C) Accept that this test is flaky in the new framework and remove it.

Related

• Fix PR (unblocks this test from being reached): fix(ci): parameterize pinned FFmpeg version with explicit preflight check (BtbN dropped 7.1) #13169 "fix(ci): parameterize pinned FFmpeg version with explicit preflight check (BtbN dropped 7.1)"
• Root-cause bug: [CI/CD] JetBrains tests fail: pinned FFmpeg 7.1 artifact no longer published by BtbN, surfacing as a cryptic tar error #13164
• Original test introduction: test: use intellij-ide-starter for e2e tests #7690 (CON-3723), commit 507cdfc61a on 2025-09-19
• Failed CI run with artifacts: https://github.com/continuedev/continue/actions/runs/32468648181