chore(ingestion): optionally serialize the calls to loadPlugin to limit memory usage

[xvello]

Problem

We can observe prod-us plugin-ingestion memory usage spiking up to 12GiB at startup, and sometimes OOMIng. This does not happen on EU nor other roles of plugin-server. overflow spikes lower, but still around 5GiB, even when there's no significant traffic to the overflow topic.

We currently have 31k active pluginconfig entries on US, and we can observe that this creates a significant amount of heap allocations on startup:

#16329 tried to delay the call to loadPlugin until an event comes in, but that behaviour would break scheduler, and the PR has failing tests due to side-effects. The current PR is a way to reduce the impact, instead of completely negating it, by hopefully reducing the GC pressure.

Changes

• Add a PLUGIN_LOAD_SEQUENTIALLY boolean option to call loadPlugin sequentially instead of in parallel for all pluginconfigs
• While loadPlugin is async, its only awaits are frontend app transpilation (only executed on scheduler pods) and processError when the plugin is invalid. I think this should not increase startup times significantly, but I'm keeping it disabled by default for now.

How did you test this code?

[tiina303]

not fully sure on the pnpm changes, but 👍