memory usage keeps increasing after upgrading from 3.12 to 3.18

[wliang368-leon]

Hi APISIX team,

I’m investigating a memory usage issue after upgrading Apache APISIX from 3.12 to 3.18.

Our main use case is LLM/AI Gateway, and we are using the ai-proxy-multi plugin.

Environment

• APISIX: 3.18
• Previous version: 3.12
• Deployment: Kubernetes
• Main plugin: ai-proxy-multi
• Traffic: production LLM traffic
• Models/upstream endpoints: essentially unchanged before and after the upgrade

Problem

After upgrading from 3.12 to 3.18, APISIX worker memory usage gradually increases.
We did not observe this behavior with the same workload on APISIX 3.12.

Restarting the APISIX pods temporarily brings memory usage back down, but it gradually increases again.

Questions

1. Are there any known memory issues/regressions related to ai-proxy-multi between 3.12 and 3.18?
2. Did the upstream/connection-pool handling of ai-proxy-multi change between these versions?
3. Can multiple ai-proxy-multi instances/routes targeting the same upstream share the same keepalive connection pool?
4. Is there any known issue where ai-proxy-multi or its upstream connection pools retain Lua objects/connections and cause memory to grow over several days?
5. Are there any recommended metrics or debugging methods to distinguish:
   • Lua/worker memory growth
   • connection pool growth
   • APISIX/OpenResty memory leak
   • Linux page cache caused by log files
6. If anyone has investigated a similar issue in APISIX 3.18, what would be the recommended way to isolate the root cause?

I can provide configuration snippets, memory metrics, worker information, and relevant logs if needed.

Thanks！

[chakravarthyBatna]

I couldn't find a reported leak in ai-proxy-multi between 3.12 and 3.18. But two changes in 3.18 can explain memory that keeps growing slowly after the upgrade.

1. Bigger Prometheus shared dict (#13688). The default size of prometheus-metrics went from 10m/15m to 128m. Linux only commits that memory as the dict fills, so pod memory climbs slowly as new metric series appear. 3.18 also adds new LLM histograms (#13487), which create more series.

Check apisix_shared_dict_free_space_bytes{name="prometheus-metrics"}. If pod memory grows while this value goes down, this is the cause, and it should level off. To limit it, set a smaller size in nginx_config.http.lua_shared_dict (and meta.lua_shared_dict) for prometheus-metrics.

2. New HTTP client for AI requests (#13778). Since 3.18, ai-proxy and ai-proxy-multi send LLM requests through the C module ngx_http_ffi_client instead of lua-resty-http, so upstream connections are handled differently. That also answers your Q2. To test it, switch one pod back to the old client and compare:

plugin_attr:
  ai-proxy:
    http_client: lua-resty-http

Lua heap vs other memory: call the Control API on a pod: curl -X POST http://127.0.0.1:9090/v1/gc (forces a full Lua GC). If memory drops, it was just uncollected Lua garbage. If it doesn't, look at the shared dicts or the new C client.

[wliang368-leon]

Thanks for the suggestion. I performed an A/B test on APISIX 3.18 to compare the two HTTP clients under our real production workload.

Test environment

• APISIX version: 3.18
• Same production environment and configuration
• Same traffic pattern and workload
• Same ai-proxy / ai-proxy-multi usage
• 3 pods using the default C-based HTTP client (ngx_http_ffi_client)
• 3 pods using lua-resty-http
• The only intentional difference was the HTTP client configuration
• The traffic consists entirely of LLM requests
• Approximately 4 million LLM requests per day across the gateway
• The traffic used in this test is real production LLM traffic, not synthetic benchmark traffic

The Lua client was configured as:

plugin_attr:
ai-proxy:
http_client: lua-resty-http

A/B test results

After running the test for approximately 24 hours, the difference was very clear:

• lua-resty-http: memory remained stable at around 1.2 GB
• ngx_http_ffi_client: memory continued to increase and reached around 2.2 GB
• Based on our previous observations, the C client configuration consistently increases memory by approximately 1 GB per day

The same memory growth pattern has been observed repeatedly in our previous tests.

Additional investigation

We also performed a full Lua GC using:

curl -X POST http://127.0.0.1:9090/v1/gc

There was no significant decrease in memory usage after the GC.

We checked the Nginx worker processes as well. The memory was mainly anonymous memory (RssAnon), rather than file-backed memory. For example, the worker processes showed approximately 2 GB RSS each, with RssAnon accounting for almost all of the RSS.

We also checked TCP connections:

• ESTABLISHED: 73
• CLOSE_WAIT: 4

There was no obvious accumulation of CLOSE_WAIT connections.

Conclusion

Based on the A/B test, the memory growth appears to be strongly correlated with the new ngx_http_ffi_client used by APISIX 3.18.

The fact that:

1. Both groups run the same APISIX 3.18 version
2. Both groups handle the same type of LLM traffic
3. Both groups handle approximately the same workload
4. lua-resty-http remains stable
5. ngx_http_ffi_client consistently grows by approximately 1 GB/day
6. The workload is large-scale, with approximately 4 million LLM requests per day
7. The test uses real production traffic rather than synthetic benchmark traffic

makes the HTTP client implementation the primary suspect in our investigation.

Could you please help investigate whether there is a memory leak or memory that is not being released correctly in ngx_http_ffi_client, especially under long-running, high-volume LLM traffic and streaming requests with ai-proxy-multi?

If there are any recommended debugging options, metrics, or tools for investigating the native/C memory used by ngx_http_ffi_client, we would be happy to collect additional information.

Thanks again for your help. We really appreciate any guidance or suggestions on how we can further narrow down the root cause.