Skip to content

[Bug]: Antigravity health checks leave large _MEI folders in Windows temp #9650

Description

@WellyngtonF

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Run T3 Code on Windows with the managed Antigravity provider enabled.
  2. Use a background activity profile with provider health checks enabled. The Performance profile checks every minute.
  3. Leave T3 running for several minutes.
  4. Inspect %LOCALAPPDATA%\Temp for _MEI* directories.
  5. Try to remove the directories while T3 and its Antigravity health checks are active.

Expected behavior

The temporary PyInstaller extraction is removed when an Antigravity health probe ends. Repeated health checks do not consume unbounded disk space.

Actual behavior

Each disposable Antigravity health probe extracts the one-file ACP binary into a new _MEI* directory. T3 force-stops the process on Windows, so PyInstaller does not run its own cleanup. Some directories stay locked by live ACP processes and the orphaned directories accumulate.

On one machine, the folder count increased from 5 to 7 while T3 was running. Total use increased from 3.53 GiB to 5.86 GiB. Each large orphan used about 1.19 GiB.

Impact

Major degradation or frequent failure

Version or commit

main @ 617edab

Environment

Windows x64, T3 Code desktop, managed Antigravity 1.1.1 (agy_acp_server.exe)

Logs or stack traces

The Antigravity provider uses makeManagedServerProvider, which starts a fresh ACP probe at the configured provider health-check interval.

The official ACP executable is a PyInstaller one-file bundle. On Windows, the child-process scope closes it with taskkill /T /F. Forced termination prevents PyInstaller from removing its _MEI extraction directory.

Loaded-module inspection mapped the locked _MEI directories to live agy_acp_server.exe processes. Other large _MEI directories remained after their probe processes ended.

Related issue #9432 increased the Antigravity startup timeout to 90 seconds. That fixes slow startup, but it does not contain or remove the extraction directories.

Suggested fix:
- Keep the Antigravity startup probe and explicit manual refresh.
- Disable its periodic process-based health probe.
- Set TMPDIR, TEMP, and TMP to a T3-owned scoped directory for each ACP runtime.
- Remove that directory after the child process is closed.

Screenshots, recordings, or supporting files

No response

Workaround

Set Settings -> Providers -> Advanced -> Provider health check interval to 0, fully exit T3 Code, then remove the orphaned _MEI* directories.

Activity

  1. Kevin-nav commented on Sep 4, 2026

    @Kevin-nav

    Confirmed with a severe reproduction on Windows 11 10.0.26200.9168, using T3 Code Nightly 0.0.39-nightly.20260904.1280 and managed Antigravity runtime agy_acp_server_20260818_01_RC01.

    I found 123 _MEI* directories under %LOCALAPPDATA%\Temp; 107 were created during the same day. More than 100 contained separate copies of the approximately 577 MB local harness payload. The accumulated Antigravity extractions consumed roughly 82 GB and reduced a 1 TB drive to about 140 MB free.

    T3 logs showed repeated Antigravity health checks followed by ACP protocol terminated, model-refresh timeouts, and authentication-support errors. After removing the orphaned directories, T3 created another approximately 850 MB _MEI directory within minutes while the provider remained enabled.

    Setting the provider health-check interval to 0 and disabling Antigravity stops the immediate recurrence. This appears consistent with the forced-termination/PyInstaller cleanup problem described here. PR #9626’s owned-runtime-directory cleanup would address the observed failure mode.

  2. TomShtern commented on Sep 5, 2026

    @TomShtern

    im on windows11, t3-nightly.
    it used about 30-ish GB on my laptop. about 239GB on my desktop(231 _mie folders give or take).
    i can give more info and details if anyone needs them, just tell me what do you need/want to know and ill give it to you.

    btw, the antigravity acp integration was janky and glitchy from the start (for me), and wasnt even working when it came out. i think that after some fixes were made so i can use it - then it started to accumulate. i could be wrong, i dont want to present assumptions as facts.

    waiting eagerly for a fix/recognition of the issue.

  3. TomShtern commented on Sep 5, 2026

    @TomShtern

    im on windows11, t3-nightly. it used about 30-ish GB on my laptop. about 239GB on my desktop(231 _mie folders give or take). i can give more info and details if anyone needs them, just tell me what do you need/want to know and ill give it to you.

    btw, the antigravity acp integration was janky and glitchy from the start (for me), and wasnt even working when it came out. i think that after some fixes were made so i can use it - then it started to accumulate. i could be wrong, i dont want to present assumptions as facts.

    waiting eagerly for a fix/recognition of the issue.

    i asked astra-low to give some details that might help the maintainers, i hope this is helpful:
    "
    Additional Windows evidence: 29 GiB of retained Antigravity _MEI files

    Environment

    • Windows 11 x64, build 26200.
    • Installed T3 Code: 0.0.39-nightly.20260904.1280.
    • Active managed Antigravity runtime: agy_acp_server_1.1.1.
    • An older managed runtime, agy_acp_server_20260818_01_RC01, was also present.

    Observed storage

    • 42 directories under %LOCALAPPDATA%\Temp_MEI*.
    • Combined file lengths: 31,171,296,544 bytes (29.03 GiB).
    • 31 directories were created during an approximately 3-hour-36-minute
      period and account for effectively all of that storage.
    • Repeated files at
      google3\third_party\jetski_prod\localharness\localharness
      totalled 23,028,469,112 bytes (21.45 GiB).
    • Complete newer extraction directories were approximately 1.16 GiB each.
    • No matching T3/Antigravity processes were running at inspection.
      The directories remained on disk.

    Source and timing evidence

    • Read-only inspection of the installed agy_acp_server.exe PyInstaller
      archive directories found the same internal localharness path.
    • Archive entries specify uncompressed sizes of 577,028,952 bytes for the
      older runtime and 922,603,056 bytes for the newer runtime, matching
      corresponding extracted files.
    • Both installed ACP executables had valid Google LLC signatures.
    • Multiple checkAntigravityProvider trace start times preceded matching
      _MEI directory creation times by less than one second.
    • A sequence of checks started five minutes after the preceding check
      completed, matching the installed provider-health refresh loop.
    • Logs also contained interrupted checks and a model-refresh timeout.
      These observations do not establish how each individual process ended.

    Expected behavior
    Temporary runtime extractions should be reclaimed after they are no
    longer needed, without repeated checks accumulating large amounts of
    retained data.

    Investigation limits
    This was a read-only examination of existing files, installed code, and
    logs. No controlled reproduction or historical process-termination
    capture was performed. File totals are logical lengths, not an NTFS
    allocated-space measurement. The precise cleanup failure has not been
    independently established.

    These measurements provide additional evidence for #9650. No files or
    settings were changed during the investigation.
    "

  4. ElliotDrel commented on Sep 5, 2026

    @ElliotDrel

    Another Windows data point, and I think the largest so far: this filled a 930 GB system volume to the point of hard ENOSPC.

    Environment

    • Windows 11 x64, build 10.0.26200
    • T3 Code Nightly 0.0.39-nightly.20260905.1287
    • Managed Antigravity runtime, agy_acp_server.exe (410 MB on disk)
    • backgroundActivityProfile: "performance", providerHealthRefreshInterval: 60000

    Scale

    526 orphaned _MEI* directories under %LOCALAPPDATA%\Temp, totalling 390.61 GB. That single folder was 390 of the 907 GB used on a 930.65 GB volume. Complete extractions measured ~1.16 GiB each, matching the numbers reported above.

    At one probe per 60 s that works out to roughly 70 GB/hour.

    Onset correlates with install to the second

    • agy_acp_server.exe written to ~/.t3/tools/antigravity-acp/... at 2026-09-04 12:11:12 local
    • First orphaned _MEI directory created at 2026-09-04 12:11:38, 26 seconds later

    Directory creation counts per hour over the incident were 40, 50, 43, 54, 35, 14, 49, 51, 57, 58, 33, 40. That tracks the 60 s health interval about 1:1. There is a clean 12-hour gap from 2026-09-04 21:00 to 2026-09-05 09:00 where zero directories were created, which is exactly when the machine was asleep.

    The leak breaks the check that causes it

    Once the volume filled, checkAntigravityProvider started failing on its own cache write:

    ENOSPC (errno -4055) writing
      C:\Users\2supe\.t3\caches\antigravity.json.<tmp>\contents.tmp
      at checkAntigravityProvider (bin.mjs:190794)
      at checkAntigravityProvider (definition) (bin.mjs:190747)
    

    Worth flagging separately: that cache write is a write-temp-then-rename, and it is the same non-atomic pattern as #4750. On a full disk it fails partway. I did not hit cache corruption, but the failure mode looks reachable.

    Contents of the orphans confirm the source: agy_acp_licenses.txt, python310.dll, google3/third_party/jetski_prod/localharness/, googleapiclient/, pydantic_core/.

    Live capture caught the spawns 60 to 68 s apart, parented to the server process:

    agy_acp_server.exe  PID 16892 / 18256
      PARENT: "T3 Code (Nightly).exe" resources\server.asar\apps\server\dist\bin.mjs --bootstrap-fd 3
    

    Negative control

    After setting the Antigravity provider to enabled: false, I watched %LOCALAPPDATA%\Temp for 200 seconds: 0 new _MEI directories, against a baseline of 3 in the same window. Deleting the 526 orphans returned the volume from 23.58 GB free to 399.18 GB free.

    Note on diagnosis difficulty

    This is hard to catch with process-level write-rate sampling. The extraction is a roughly 10 second burst once a minute, so per-minute snapshots of write I/O land in the idle gap and show almost nothing. Directory size walking is what surfaces it. Might be worth a line in the workaround section for anyone else chasing unexplained disk growth.

    #9626 looks like the right fix and is still open as of this comment. Happy to test a nightly once it lands.

  5. GorlikItsMe commented on Sep 5, 2026

    @GorlikItsMe

    My windows VM just ran out of memory. I'm leaving a comment so I can follow this PR

  6. kharitonov-egor commented on Sep 5, 2026

    @kharitonov-egor

    I can reproduce this on Windows 11 Pro x64, build 26200.

    T3 Code version: 0.0.39-nightly.20260905.1289
    Antigravity ACP version: agy_acp_server_1.1.1

    I found 45 _MEI* directories under %LOCALAPPDATA%\Temp, totaling 51.59 GiB. A complete directory is about 1.16 GiB and contains the bundled Python runtime plus google3/third_party/jetski_prod/localharness.

    The directories were created between September 3 at 10:40 PM and September 5 at 7:37 PM. Their repeated structure and timestamps match the health-check leak described in this issue.

    Prepared with OpenAI Codex.

  7. YashasVM commented on Sep 6, 2026

    @YashasVM

    Another confirmed Windows data point:

    Environment

    • Windows 11 x64, build 10.0.26200
    • T3 Code: 0.0.39-nightly.20260906.1293
    • Node: v25.6.1
    • Antigravity provider enabled

    Observed impact

    • 17 orphaned _MEI* directories under %LOCALAPPDATA%\Temp
    • Combined logical size: 20,520,125,462 bytes (~19.11 GiB)
    • Each complete extraction contained the bundled Google runtime and google3/third_party/jetski_prod/localharness/localharness
    • The repeated harness file was approximately 922,603,056 bytes

    Investigation

    • The directories were created while T3 Code was open.
    • T3 Code was closed before inspection, so no Antigravity process was running when the files were checked.
    • The _MEI* directories stopped accumulating after T3 Code was closed.
    • Removing the 17 directories reclaimed approximately 19.11 GiB.
    • The current T3 installation stores the managed Antigravity executable under its managed tools directory, but invoking the bundled PyInstaller executable still creates _MEI* extraction directories.

    Workaround applied

    • Closed T3 Code before cleanup.
    • Removed the 17 stale _MEI* directories after confirming no Antigravity process was running.
    • No T3 source or database files were modified.

    This reproduces the storage leak described in this issue. Please consider prioritizing the cleanup/fix so repeated Antigravity health checks cannot accumulate roughly 1.16 GiB per check.

    Prepared with OpenAI Codex via T3 triage.

  8. mtdewwolf commented on Sep 7, 2026

    @mtdewwolf

    Additional reproduction: 256 orphaned extractions and ~228 GB reclaimed

    I reproduced this on Windows x64 with T3 Code Nightly and the managed Antigravity ACP provider (agy_acp_server_1.1.1). This is an additional data point confirming that the leak is caused by the T3 Code Antigravity health-check integration, not by ordinary Antigravity CLI usage.

    Observed configuration

    • T3 Code server setting: providerHealthRefreshInterval: 60000
    • Persisted provider configuration: providerInstances.antigravity.enabled: true
    • Managed executable: T3 Code's agy_acp_server.exe under .t3/tools/antigravity-acp/...
    • T3's Antigravity status cache reported status: error and:
      Antigravity did not respond to its local health check within 90 seconds.

    Failure evidence

    T3 server traces repeatedly showed:

    • checkAntigravityProvider
    • AntigravityDriver.makeRuntime
    • prepareAntigravityProfile
    • AcpTransportError: ACP transport operation failed

    The failed checks occurred on the provider-health refresh loop. The extracted _MEI* directories contained the expected PyInstaller/Antigravity payload, including agy_acp_licenses.txt, python310.dll, Cython/Python packages, and the bundled local harness.

    Scale and impact

    • 256 _MEI* directories accumulated under %LOCALAPPDATA%\Temp.
    • The system had approximately 28 GB free on a roughly 1 TB volume.
    • After disabling the provider and cleaning the orphaned directories, the system had approximately 256 GB free.
    • Approximately 228 GB was reclaimed.

    Mitigation that stopped recurrence

    1. Set providerInstances.antigravity.enabled to false in T3's persisted settings.
    2. Restarted the T3 server process so it reloaded the setting.
    3. Confirmed there were no running AGY/Antigravity processes.
    4. Removed the orphaned _MEI* directories.

    After the restart, checkAntigravityProvider completed almost immediately without the previous transport failures, and no new _MEI* directories were created. The final cleanup reduced the count from 256 to 0.

    Diagnosis

    The immediate runtime failure is in the Antigravity ACP process, but the disk-space incident is an integration failure in T3 Code: T3 repeatedly launches the failing PyInstaller one-file runtime for health checks and does not reliably reclaim the extraction directory after the probe terminates. Disabling the provider prevents the repeated launches.

    Suggested fix

    In addition to the scoped-runtime cleanup proposed in this issue:

    • Do not run provider health checks for an explicitly disabled provider instance.
    • Treat repeated Antigravity health-check transport failures as a degraded/disabled provider state instead of retrying indefinitely every minute.
    • Use a T3-owned per-probe temporary directory and remove it after process termination, including forced Windows termination paths.
    • Add a disk-safe guard or backoff so a failing provider cannot create approximately 1 GiB of retained files per check.

    No credentials or source files were exposed or modified; only the Antigravity provider setting was changed and stale temporary extraction directories were removed.

  9. andybergon commented on Sep 7, 2026

    @andybergon

    We hit this on Windows x64 with T3 Code Nightly 0.0.40-nightly.20260907.1346 and managed Antigravity agy_acp_server_1.1.1.

    Before manual cleanup, we found 204 _MEI* folders with modification dates spanning 2026 Sep 05-07. Temp totalled approximately 202 GiB, and C: reached zero free space, making the PC difficult to use. Many complete extractions were 1,250,259,477 bytes, including a 922,603,056-byte google3/third_party/jetski_prod/localharness/localharness payload.

    We preserved folder metadata and sampled SHA-256 hashes before deleting the accumulated files. The installed managed runtime's release checksum matches T3's Windows x64 ACP 1.1.1 package.

    This confirms the matching accumulated-files symptom on the September 7 Nightly. We have not independently reproduced the exact termination sequence or tested recurrence after the documented workaround.

  10. mtdewwolf commented on Sep 8, 2026

    @mtdewwolf

    Confirmed that the extraction-retention symptom still reproduces on 0.0.41-nightly.20260908.1387 (September 8). Updating to this nightly did not resolve it in this test.

    Environment and method

    • Windows 11 Home x64, build 10.0.26200.
    • Installed nightly's shipped server and managed agy_acp_server_1.1.1 executable; no source build or patched binaries.
    • Separate server profile bound to localhost, Antigravity enabled, performance profile, providerHealthRefreshInterval: 60000, and binaryPath pointing to the existing managed ACP executable.
    • Five-minute disabled baseline: zero _MEI* directories and no AGY processes.
    • Two independent server starts, with process ancestry, extraction creation/size, and checkAntigravityProvider trace timing recorded.

    Measured result

    Completed startup probe Duration Extraction bytes retained after AGY exited
    1 69.541 seconds 1,250,259,477
    2 75.663 seconds 1,250,259,477

    At 08:26:21 UTC on September 8, both full extraction directories remained and no AGY processes were running: 2,500,518,954 logical bytes (2.33 GiB) retained. The first directory also survived stopping and restarting the isolated T3 server. These are file-length totals from the two directories, not a free-space delta.

    The installed makeAntigravityAcpRuntime does not contain #9626's owned temporary-directory allocation and cleanup implementation. That PR was still open/unmerged when checked (head a2cab15cb222c39b715d743dca2e6c05dbd6e31d). This is a failure observed in the shipped nightly, not a failed test of the proposed PR implementation.

    Reproduction scope

    This confirms repeated startup health-probe retention. The isolated profile was unpaired and returned “Google account access is not checked yet”; it did not exercise ten periodic probes or an authenticated desktop session. Successful authentication, injected timeouts, graceful app shutdown, and the PR's marked-orphan recovery/active-runtime protection were not tested. An initial measurement-script error interrupted a preliminary launch; that attempt was excluded from the measurements above.

    For binary identification:

    • server.asar SHA-256: 173EBF2BEA42B5EAB4169DB09C2D3271DC656BFEB7A0018CDC866A9A4FFD5519
    • agy_acp_server.exe SHA-256: 74EE0984927AF43BC9E7917004CD1AC6674C7D8C13420BE2AFA0452856A56818

    The test server was stopped and all test-created extractions removed afterward. The normal desktop profile remained Antigravity-disabled throughout.

    Correction to my earlier comment above: the approximately 228 GiB free-space increase also included concurrent worktree cleanup and should not have been attributed entirely to AGY. The 2.33 GiB measured here is isolated to the two completed probes.

  11. JDeffner commented on Sep 8, 2026

    @JDeffner

    Additional Windows confirmation: 74.3 GiB of extractions created in one day

    Confirmed on T3 Code Nightly 0.0.40-nightly.20260907.1372. On September 8, 2026, a read-only inspection found 69 _MEI* directories totaling approximately 74.3 GiB, all created that day. The C: volume reported effectively no free space. This adds a Windows case on a newer nightly than the original report.

    Environment

    • Windows Home x64
    • Desktop and server package version: 0.0.40-nightly.20260907.1372, read from the installed app.asar and server.asar package manifests.
    • Release tag commit: 09e8de9c655ae85410bf6b00446f272a01da81c7.
    • Managed executable present at %USERPROFILE%\.t3\tools\antigravity-acp\win32-x64\versions\47cb50eef14f0a4655d78cfcfda869bcea7aaee5f9787e936bc2935ea612c3b8\agy_acp_server.exe. The installed T3 bundle specifies managed release agy_acp_server_1.1.1; the executable was not launched to query its version.
    • All times below are local time in Europe/Berlin, CEST (UTC+02:00).

    Disk measurements

    The affected path is %LOCALAPPDATA%\Temp\_MEI*.

    Measurement Observation
    Retained directories 69
    Combined logical file lengths Approximately 74.3 GiB
    Complete extraction size Approximately 1.164 GiB per directory
    First directory creation September 8, 02:12:06
    Last directory creation in the snapshot September 8, 19:03:31
    Entire user Temp directory Approximately 83.1 GiB in an earlier scan

    One uninterrupted sequence starts at 02:12:06, 02:17:44, 02:23:49, 02:29:40, 02:35:23, 02:41:12, 02:47:06, 02:52:53, and 02:58:32. Each of these directories contains about 1.164 GiB. Similar sequences recur later in the day. This is roughly one new extraction every 5–6 minutes while checks are active.

    The owner had freed approximately 90 GB the previous day by disabling hibernation and clearing temporary files. That earlier free-space figure is user-reported. Inspection confirmed that C:\hiberfil.sys remained absent and that the 48 GiB pagefile was on D:. The measurements account for most of the reported loss, but do not attribute every byte of the free-space change.

    Log correlation

    The following checkAntigravityProvider spans were read from .t3/userdata/logs/server.trace.ndjson. Trace start times were converted from Unix nanoseconds to CEST. Times are displayed to whole seconds.

    Check started Duration Directory created Directory
    18:58:31 91.280 s 18:58:32 _MEI00009bec2
    19:00:02 6.909 s 19:00:05 _MEI000093d82
    19:00:09 1.748 s 19:00:10 _MEI000083982
    19:03:31 3.009 s 19:03:31 _MEI000085c82

    These later directories were smaller than complete extractions. Their creation times closely match the health-check starts. A trace span returning Success is not proof that the provider was healthy: the check converts probe failures into provider status data.

    Installed-code analysis

    The inspected bin.mjs was byte-for-byte identical to apps/server/dist/bin.mjs inside the installed native server.asar. Its SHA-256 is 4f1c6090d6dda287ead9c6bb8cc27d4d454db6391df06978138a322b1e808656.

    The bundle and accompanying source map show this sequence:

    1. The provider refresh loop uses a default interval of five minutes. The observed timing is consistent with that interval plus probe duration; the effective user setting was not independently read.
    2. AntigravityDriver.ts creates a process scope for the probe, starts a runtime, calls runtime.initialize(), and closes the scope through a finalizer.
    3. AntigravityProvider.ts gives the health probe a 90-second timeout. Increasing this timeout does not address retained extraction directories.
    4. The bundled Windows process-group termination code calls taskkill /pid <pid> /T /F (NodeServices-3SOZTEi2.mjs, lines 979–995).
    5. AntigravityAcpSupport.ts in this installed release does not allocate and reclaim a runtime-owned temporary directory.

    The retained directories contain agy_acp_licenses.txt and Google's google3 package tree. Combined with the timestamp matches, this supports the Antigravity attribution. The force-termination mechanism described in this issue is also consistent with PyInstaller's documented cleanup limitation: forcibly terminating the process can leave its extraction directory behind.

    This inspection did not capture the historical termination of each process, establish that every directory was unused, or run a controlled reproduction. Sizes are sums of logical file lengths, not NTFS allocated-space measurements. No temporary files, provider settings, or processes were changed during the investigation.

  12. R2bEEaton commented on Sep 9, 2026

    @R2bEEaton

    Reproduces on the stable v0.0.40 release (not just nightly), with default health-check settings

    Adding this data point because the existing reports are all on Nightly builds with an explicitly configured providerHealthRefreshInterval. This machine is on the stable release channel and has never set that value, so the leak occurs on the shipped defaults.

    Environment

    • Windows 11 Pro x64, build 10.0.26200
    • T3 Code v0.0.40 — stable channel (resources/app-update.yml → releaseType: release); this is the current Latest release, published 2026-09-08
    • Managed Antigravity runtime agy_acp_server_1.1.1 (agy_acp_server.exe, 410 MB; localharness_external.exe, 125 MB)
    • providerHealthRefreshInterval: not set — falls back to the bundled DEFAULT_PROVIDER_HEALTH_REFRESH_INTERVAL of 5 minutes
    • backgroundActivityProfile: not set (default balanced)
    • Antigravity provider enabled, authMethod: oauth-personal

    Scale

    • 133 orphaned _MEI* directories, totalling 86.25 GB
    • That was 98% of a 88.07 GB %LOCALAPPDATA%\Temp (631,758 files total)
    • Each complete extraction: 1.16 GB / 8,247 files, containing the bundled Python 3.10 runtime plus google3/third_party/jetski_prod/localharness/localharness
    • Creation dates: 5 on 2026-06-06, 53 on 2026-06-29, 53 on 2026-09-08, 22 on 2026-09-09 — i.e. ~87 GB accrued in roughly two days
    • 0 of the 133 were held open by any running process (checked every process's loaded modules before deleting); all 133 deleted cleanly

    Install-time correlation

    ~/.t3/tools/antigravity-acp/ was created 2026-09-08 12:36:29 local. The first leaked _MEI directory of that day appeared at 12:37:15 — 46 seconds later. Nothing before 2026-06 exists on this machine.

    The extractions are not driven by agent sessions

    This is the part I think is worth adding to the diagnosis. Across 2026-09-08 and 09-09 the provider event logs (~/.t3/userdata/logs/provider/events.*.log) contain only 4 Antigravity session logs with 9 total session.started events:

    thread starts exits
    f0c8475b… 6 1× {"exitKind":"error","reason":"Antigravity process stopped."}, rest graceful
    760f782e… 1 graceful
    72726751… 1 graceful
    236c514e… 1 graceful

    9 session starts, but 75 leaked directories over the same window. So sessions account for at most ~12% of the extractions.

    Direct confirmation that a spawn happens while the provider is completely idle: after I deleted all 133 directories, a new one (_MEI000074302, 1.16 GB) was created at 11:14:20 local. At that moment:

    • no Antigravity session was running — the last one had exited at 09:48 local (13:48:32Z)
    • agy_acp_server.exe's LastAccessTime was exactly 11:14:20, matching the directory's creation time to the second
    • the directory was written for 14 seconds (LastWriteTime 11:14:34) and then abandoned

    That is the periodic health probe starting the full 410 MB one-file bundle, reading a few KB, and being force-killed — consistent with the taskkill /T /F teardown named in the original report.

    Corroborating signal in the traces: server.trace.ndjson logs 1,056 antigravityAuthSupport.handleStderr spans plus AntigravityAdapter.handleEvent, applyAntigravityAcpModelSelection etc., far out of proportion to 9 real sessions.

    Note on the shipped mitigation

    resources/server.asar already carries an acknowledgement of the underlying hazard in collectUint8StreamText:

    keep draining after truncation so the child process can exit normally. its a know issue that on windows killing after the output cap can force an expensive taskkill operation and hurt performance

    That drain-instead-of-kill mitigation only covers the output-cap path, so the health-probe teardown still force-kills and still orphans the extraction.

    Suggested scope note

    Since this reproduces on v0.0.40 stable with no custom interval, the blast radius is wider than the nightly-only reports suggest — any Windows user who enables Antigravity on the current release will accumulate ~1.16 GB per probe indefinitely.

  13. yabswannalearn commented on Sep 11, 2026

    @yabswannalearn

    Independent confirmation + two extra findings from #11204 (closing as duplicate):

    Reproduced controlled: spawned the bundled \�gy_acp_server.exe\ directly, waited for extraction (1,192.3 MB _MEI*), force-killed ~ orphan remained. Second run: closing stdin does NOT exit the server (ignores EOF), so it also ends in a kill ~ orphan. Every launch orphans, graceful or not.

    Ghost processes: killing the bootloader leaves the server CHILD running headless (observed 2 live \�gy_acp_server.exe\ holding the _MEI\ DLL locks, blocking even manual deletion until killed). So orphaned probes/sessions linger as ghost processes burning RAM/CPU too, not just dead disk. Worth a \ asklist | findstr agy\ check for anyone cleaning up.

    Impact data point: 39 orphans (~37 GB) filled C: to 0 bytes free on a Windows 11 box; orphans appeared during unrelated desktop use, consistent with the periodic-probe theory here rather than only explicit sessions.

  14. R2bEEaton commented on Sep 11, 2026

    @R2bEEaton

    Follow-up on the v0.0.40 stable case: accrual rate, and the stopgap I'm running

    Two days after my earlier comment, here's the refill rate on the stable v0.0.40 release with providerHealthRefreshInterval left at its default:

    Date New _MEI* dirs
    2026-09-09 12
    2026-09-10 73
    2026-09-11 (partial) 2

    I deleted 133 directories (86.25 GB) on 09-09. By 09-11 there were 87 directories totalling 100.45 GB — so roughly 84 GB/day, or one ~1.15 GB orphan every ~20 minutes. Again, none of the 87 were locked by a live process, and all 87 deleted cleanly.

    Worth stressing for triage: that's the default probe interval on the shipped stable release, not a tuned-down nightly setting.

    Stopgap: scheduled cleanup that is safe to run while T3 Code is open

    Rather than disabling the provider, I'm running an hourly scheduled task. It applies two independent safety checks so it can run unattended without racing the probe:

    1. Skip any directory backing a loaded module in a live process — enumerate every process's modules and exclude those _MEI names.
    2. Skip any directory written to in the last 15 minutes — so a bundle still extracting is never touched, even if it hasn't loaded a DLL yet.

    Deleting only on both checks passing matters: check 1 alone has a window where a freshly spawned bundle is mid-extraction with no modules loaded yet.

    [CmdletBinding(SupportsShouldProcess)]
    param(
        [int]    $MinAgeMinutes = 15,
        [string] $LogPath = (Join-Path $env:LOCALAPPDATA 'T3MeiCleanup\cleanup.log')
    )
    
    $temp = Join-Path $env:LOCALAPPDATA 'Temp'
    
    # Safety check 1: directories backing a loaded module in any live process
    $pattern = '\\' + 'Temp' + '\\(_MEI[^\\]+)\\'
    $inUse = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::OrdinalIgnoreCase)
    foreach ($proc in Get-Process -ErrorAction SilentlyContinue) {
        try {
            foreach ($mod in $proc.Modules) {
                if ($mod.FileName -match $pattern) { [void]$inUse.Add($Matches[1]) }
            }
        } catch { }   # access denied, or process exited mid-enumeration
    }
    
    $cutoff = (Get-Date).AddMinutes(-$MinAgeMinutes)
    $deleted = 0; $freed = 0L; $skipLocked = 0; $skipYoung = 0
    
    foreach ($dir in Get-ChildItem -LiteralPath $temp -Force -Directory -Filter '_MEI*' -ErrorAction SilentlyContinue) {
        if ($inUse.Contains($dir.Name)) { $skipLocked++; continue }
    
        $files  = @(Get-ChildItem -LiteralPath $dir.FullName -Force -Recurse -File -ErrorAction SilentlyContinue)
        $newest = ($files | Measure-Object LastWriteTime -Maximum).Maximum
        if ($null -eq $newest) { $newest = $dir.LastWriteTime }
    
        # Safety check 2: still being extracted?
        if ($newest -gt $cutoff) { $skipYoung++; continue }
    
        $size = ($files | Measure-Object Length -Sum).Sum
        if ($PSCmdlet.ShouldProcess($dir.FullName, 'Remove orphaned _MEI extraction')) {
            try {
                Remove-Item -LiteralPath $dir.FullName -Recurse -Force -ErrorAction Stop
                $deleted++; $freed += $size
            } catch { }
        }
    }
    
    'found={0} deleted={1} freed={2:N2}GB skipped_locked={3} skipped_recent={4}' -f `
        (Get-ChildItem -LiteralPath $temp -Force -Directory -Filter '_MEI*' -EA SilentlyContinue).Count,
        $deleted, ($freed / 1GB), $skipLocked, $skipYoung

    Save as Clear-OrphanedMEI.ps1 and register it hourly (runs as the logged-on user, no stored credentials, no elevation):

    $action = New-ScheduledTaskAction -Execute "C:\Program Files\PowerShell\7\pwsh.exe" `
      -Argument '-NoProfile -NonInteractive -WindowStyle Hidden -ExecutionPolicy Bypass -File "C:\path\to\Clear-OrphanedMEI.ps1"'
    
    $t1 = New-ScheduledTaskTrigger -AtLogOn -User $env:USERNAME
    $t2 = New-ScheduledTaskTrigger -Once -At (Get-Date).Date.AddMinutes(5) `
            -RepetitionInterval (New-TimeSpan -Hours 1) -RepetitionDuration (New-TimeSpan -Days 3650)
    
    $principal = New-ScheduledTaskPrincipal -UserId "$env:USERDOMAIN\$env:USERNAME" `
                   -LogonType Interactive -RunLevel Limited
    
    $settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries `
                  -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Minutes 30) -MultipleInstances IgnoreNew
    
    Register-ScheduledTask -TaskName 'T3Code-Clear-Orphaned-MEI' `
      -Action $action -Trigger $t1,$t2 -Principal $principal -Settings $settings

    Supports -WhatIf for a dry run, and logs one summary line per run to %LOCALAPPDATA%\T3MeiCleanup\cleanup.log:

    2026-09-11 07:47:11  found=1 deleted=0 freed=0.00GB skipped_locked=0 skipped_recent=1 failed=0
    

    (That run correctly declined to delete the single directory present, because it was inside the 15-minute window — both guards behaving as intended.)

    To be clear, this is damage control, not a fix — it just caps the leak at roughly one probe interval's worth of disk instead of unbounded growth. The real fix still belongs in the ACP process teardown, or in reusing one long-lived process instead of spawning the 410 MB bundle per probe. Sharing it because at ~84 GB/day on the stable channel, a lot of people are going to need something in place before the fix lands.

  15. MiloAgudelo commented on Sep 13, 2026

    @MiloAgudelo

    Another Windows confirmation, and the disk impact here is pretty extreme.

    Environment

    • Windows 11 x64, 10.0.26200
    • Antigravity provider enabled (agy_acp_server_1.1.1, authMethod: oauth-personal)
    • No custom providerHealthRefreshInterval — default 5 minutes

    What it did to this machine

    • 148 orphaned _MEI* directories under %LOCALAPPDATA%\Temp
    • Combined size ~167 GB
    • That single leak pushed a 454 GB C: to 90% full / 45 GB free
    • Each extraction is the full PyInstaller payload (google3, antigravity, genai, grpc, bundled Python)

    This is not “a few leftover temps.” It is the biggest occupant on the system drive. One provider health loop ate more disk than Adobe + Windows + npm cache combined.

    Timing matches the 5-minute probe

    • First leftover: 2026-09-09 12:15
    • Still creating them today while T3 is open
    • Median interval between consecutive _MEI* folders: 329.5 s (300 s refresh + ~30 s extract)
    • Clean sequence on 2026-09-10 21:51–22:19: 328s, 337s, 338s, 338s, 331s
    • Night/idle gaps create zero new folders
    • Latest folder at 19:58:31 local; Antigravity cache checkedAt is 19:59:03. Same probe.

    Disabling the provider / setting the health interval to 0 plus deleting the orphans is the only thing that stops it. Please land a fix — at ~1.16 GB per check this will fill a typical Windows C: in a couple of days of leaving T3 open.

  16. vinay-veerappa commented on Sep 13, 2026

    @vinay-veerappa

    Stable-channel confirmation (default settings) — 194 orphans / 222.7 GB, 216 GB reclaimed + stopgap script

    Environment

    • Windows 11 x64
    • T3 Code stable v0.0.40.0 (resources/app-update.yml → releaseType: release); providerHealthRefreshInterval never set — leak occurs on shipped defaults
    • Managed runtime agy_acp_server_1.1.1: ~\.t3\tools\antigravity-acp\win32-x64\versions\47cb50ee…\agy_acp_server.exe (410.8 MB) + localharness_external.exe (124.9 MB)

    Scale on this machine (inspected 2026-09-13)

    Day New _MEI* dirs
    2026-09-11 22
    2026-09-12 108
    2026-09-13 (partial) 58

    194 directories totalling 222.7 GB under %LOCALAPPDATA%\Temp; each complete extraction measured 1,192.3 MB, matching prior reports. Deleting the 187 folders not held by a live process freed 216.1 GB (C: went 91 GB → 361 GB free). Zero deletion failures — none of the orphans were locked.

    Mechanism confirmed independently

    • Folder contents byte-match the managed runtime's PyInstaller bundle (agy_acp_licenses.txt, googleapiclient dist-infos, grpc, websockets, bundled VCRUNTIME) — the extractions come from T3's agy_acp_server.exe, not Antigravity CLI usage.
    • Leak continued while T3 Code sat idle with the provider enabled: a fresh ~1.2 GB _MEI* dir appeared during a 2-minute observation window, consistent with the ~1 orphan / 20 min rate measured above on stable.

    Stopgap for affected users — scheduled purge that only removes _MEI* dirs older than 2 h; dirs locked by a live runtime fail harmlessly and are retried next run:

    # clean-t3-mei.ps1
    $cut = (Get-Date).AddHours(-2)
    Get-ChildItem "$env:LOCALAPPDATA\Temp" -Directory -Filter "_MEI*" |
      Where-Object { $_.LastWriteTime -lt $cut } |
      ForEach-Object {
        try { Remove-Item $_.FullName -Recurse -Force -ErrorAction Stop } catch {}
      }
    $action  = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-NoProfile -WindowStyle Hidden -ExecutionPolicy Bypass -File `"$env:USERPROFILE\clean-t3-mei.ps1`""
    $trigger = New-ScheduledTaskTrigger -Once -At (Get-Date) -RepetitionInterval (New-TimeSpan -Minutes 30) -RepetitionDuration (New-TimeSpan -Days 3650)
    Register-ScheduledTask -TaskName "Clean T3 _MEI orphans" -Action $action -Trigger $trigger

    Status check (2026-09-13): the fix in #9626 is still not in main — apps/server/src/provider/acp/AntigravityAcpSupport.ts has no temp-directory isolation or cleanup, so current nightlies still ship the leak. #9626 has been open since Sept 4 with green CI and no maintainer review; given the severity (~70–85 GB/day on default settings, multiple ENOSPC reports in this thread), could a maintainer take a look? Happy to help test the fix on the stable channel.

  17. niVoque commented on Sep 14, 2026

    @niVoque

    Still reproduces on 0.0.41-nightly.20260913.1646 (Windows 11 Pro x64, build 26200, managed agy_acp_server.exe from versions/47cb50…).

    • 235 complete ~1.2 GB _MEI* dirs (~246 GB of Temp) created 13–14 Sept, ~50 per hour while T3 was open, providerHealthRefreshInterval: 60000
    • In server.trace.ndjson, each checkAntigravityProvider → AntigravityDriver.makeRuntime span lines up to the second with a new _MEI* dir (08:21:31, 08:22:42, 08:23:52)

    A second spawn path I haven't seen mentioned here yet: with textGenerationModelSelection set to Antigravity, the first turn of a new thread goes through maybeGenerateThreadTitleForFirstTurn → AntigravityTextGeneration.generateThreadTitle → AntigravityTextGeneration.runJson → AntigravityDriver.makeRuntime. That starts its own agy_acp_server.exe and leaks another dir, independent of the health probe. #11657 removes the spawn from probe only, so this path still needs the TEMP isolation/cleanup to apply.

    Also: 106 additional _MEI* dirs contained only a Cython/ subfolder (~200 KB each). It looks like PyInstaller's own cleanup did run for those launches but couldn't remove that subfolder, so even processes that exit cleanly leave something behind. A sweep that only targets complete extractions would miss these.

  18. VitaCodez commented on Sep 14, 2026

    @VitaCodez
    Contributor

    Thanks for flagging the textGeneration path and the Cython/ observation @niVoque

    Just to clarify on #11657: the PR does not only fix probe — it also includes the Windows TEMP/TMP isolation and cleanup:

    1. Text generation & chat sessions are isolated: buildAntigravityAcpSpawnInput redirects TEMP and TMP on Windows to <profile>/antigravity-acp/tmp. Because AntigravityTextGeneration launches via makeRuntime, it inherits this environment and unpacks into the isolated profile folder rather than polluting the user's system %TEMP%.
    2. Session cleanup: AntigravityTextGeneration already invokes removeAntigravitySessionFiles in its finalizer, which fix(antigravity): avoid spawning process on probe and isolate temp di… #11657 updated to sweep _MEI* folders in the profile's tmp directory on exit.
    3. Startup sweep: On startup, cleanOrphanedAntigravityTempDirs sweeps any leftover _MEI* directories in the profile tmp folder as well as previous Antigravity orphans in system %TEMP%.

    So both the recurring probe leak and the text generation path should be covered.

  19. AalishMS commented on Sep 15, 2026

    @AalishMS

    Adding another stable-channel Windows data point with measured timing and scale.

    Environment

    • Windows 11 Pro x64, build 10.0.26200
    • T3 Code Desktop 0.0.40.0 (Winget stable)
    • Antigravity with personal OAuth; Binary path left empty/default
    • Resolved agy.exe: 193,006,232 bytes, valid Google LLC Authenticode signature

    Observed impact

    • 127 _MEI* directories under %LOCALAPPDATA%\Temp, totaling 137.78 GB
    • 126 contained agy_acp_licenses.txt
    • A complete extraction contained 8,247 files and used about 1.16 GB
    • The 475.6 GB system volume fell to approximately 0.68 GB free

    Creation counts and approximate totals:

    Date Directories Size
    2026-09-11 47 51.42 GB
    2026-09-12 13 15.14 GB
    2026-09-13 18 20.53 GB
    2026-09-14 34 39.18 GB
    2026-09-15 15 11.51 GB

    Timing evidence

    New _MEI* creation times matched T3 Antigravity provider-log activity to within about one second:

    _MEI 10:30:25 | provider log 10:30:26
    _MEI 10:33:33 | provider log 10:33:34
    _MEI 11:22:32 | provider log 11:22:33
    _MEI 11:22:55 | provider log 11:22:55
    

    Several provider event logs also recorded session.exited with exitKind: "graceful", while the extracted directories remained. The cached provider state reported status: error and Antigravity could not complete its local health check.

    Disabling Antigravity stopped the immediate risk. After confirming that no AGY/Antigravity process was running, the orphaned directories were removed successfully. No full logs are attached because they may contain prompts, diffs, and local paths.

  20. indishere commented on Sep 16, 2026

    @indishere

    Same root cause, adding a severity data point and one aggravating factor that isn't in the report yet.

    Severity: this took a 952 GB drive to 0 bytes free

    Over 7 days (Sept 8–15) on Windows 11, the managed Antigravity provider orphaned 3,878 _MEI* directories totalling 741 GB, which filled C: completely — not "degraded", but 0 bytes free, at which point the machine could no longer create a temp file at all. The report's example (5 → 7 folders, 3.53 → 5.86 GiB) undersells where this ends up on a long-running install.

    %LOCALAPPDATA%\Temp alone measured 741 GB of the 869 GB total in C:\Users.

    Aggravating factor: the probe interval can be shorter than the probe timeout

    #9432 raised HEALTH_CHECK_TIMEOUT to "90 seconds", but providerHealthRefreshInterval is independently configurable and has no floor tied to it:

    Setting Value
    HEALTH_CHECK_TIMEOUT 90 s
    battery-saver preset 15 min
    DEFAULT_PROVIDER_HEALTH_REFRESH_INTERVAL (balanced) 5 min
    performance preset 1 min
    custom override (what this machine had) 30 s

    At a 30 s interval against a 90 s timeout, up to 3 probes overlap concurrently, each extracting its own copy of the 430 MB agy_acp_server.exe bundle. That's ~2,880 spawn attempts/day.

    Worth considering clamping providerHealthRefreshInterval to >= HEALTH_CHECK_TIMEOUT for process-based probes regardless of how the extraction issue is fixed — overlapping probes of a one-file PyInstaller bundle are expensive even when cleanup works correctly.

    Orphan sizes vary wildly, so folder count is a misleading metric

    Because the kill lands mid-extraction, orphans are partial. Sizes ranged from a few KB (one contained only Cython/Compiler/) up to the full bundle, averaging ~191 MB rather than the ~1.19 GiB cited. Total bytes is the metric to watch, not folder count.

    The probe never converges, so it retries forever

    ~/.t3/caches/antigravity.json sat at:

    "installed": false,
    "version": null,
    "status": "warning",
    "message": "Checking Antigravity availability."

    checkAntigravityProvider sets installed: !missingInstallation from a probe that is interrupted by timeoutOption(HEALTH_CHECK_TIMEOUT) before it can report — so the state never reaches ready and the loop never backs off. Related: #7230.

    Confirmed by elimination: with T3 Code closed, _MEI creation stops dead. The standalone agy.exe CLI (180 MB, same PyInstaller packaging) exits cleanly and leaks nothing — the orphans come specifically from the force-terminated probe path.

    Note on recovery

    Once the disk is full this is unusually hard to dig out of, because the tooling you'd normally use to diagnose it also can't allocate temp space. Anything that makes the periodic probe opt-in, or scopes TMPDIR/TEMP/TMP per ACP runtime as suggested, would prevent the terminal state rather than just slowing the climb.

    +1 on the suggested fix, particularly the scoped temp directory removed after child close.


    Generated by Opus 5 xhigh

  21. bompus commented on Sep 16, 2026

    @bompus
    Contributor

    Thank you for fixing this. I had 785GB used by these folders over the past few days.

  22. WojakGra commented on Sep 25, 2026

    @WojakGra

    Thanks god i found this post, because i saw the agy text file and had been left with 2GB of storage on my disk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions