Large amount of `GetEntityJITConfig` operations when rate limit is reached

[SystemKeeper]

The green line is the remaining rate limit, the yellow line is GetEntityJITConfig on github operations.

In the logs I see a huge number of

garm.log:time=2026-02-11T16:06:57.699+01:00 level=ERROR msg="could not add runner to pool" error="failed to add new instance for pool f2a3662a-bc35-4c0a-9c3a-db0bb944f03c: failed to generate JIT config: failed to get JIT config: POST https://api.github.com/orgs/<org>/actions/runners/generate-jitconfig: 403 API rate limit of 5000 still exceeded until 2026-02-11 16:14:55 +0100 CET, not making remote request. [rate reset in 7m57s]" pool_id=f2a3662a-bc35-4c0a-9c3a-db0bb944f03c pool_mgr=<org> endpoint=github.com pool_type=organization
garm.log:time=2026-02-11T16:06:57.701+01:00 level=ERROR msg="could not add runner to pool" error="failed to add new instance for pool f2a3662a-bc35-4c0a-9c3a-db0bb944f03c: failed to generate JIT config: failed to get JIT config: POST https://api.github.com/orgs/<org>/actions/runners/generate-jitconfig: 403 API rate limit of 5000 still exceeded until 2026-02-11 16:14:55 +0100 CET, not making remote request. [rate reset in 7m57s]" pool_id=f2a3662a-bc35-4c0a-9c3a-db0bb944f03c pool_mgr=<org> endpoint=github.com pool_type=organization
garm.log:time=2026-02-11T16:06:57.704+01:00 level=ERROR msg="could not add runner to pool" error="failed to add new instance for pool f2a3662a-bc35-4c0a-9c3a-db0bb944f03c: failed to generate JIT config: failed to get JIT config: POST https://api.github.com/orgs/<org>/actions/runners/generate-jitconfig: 403 API rate limit of 5000 still exceeded until 2026-02-11 16:14:55 +0100 CET, not making remote request. [rate reset in 7m57s]" pool_id=f2a3662a-bc35-4c0a-9c3a-db0bb944f03c pool_mgr=<org> endpoint=github.com pool_type=organization
garm.log:time=2026-02-11T16:06:57.710+01:00 level=ERROR msg="could not add runner to pool" error="failed to add new instance for pool f2a3662a-bc35-4c0a-9c3a-db0bb944f03c: failed to generate JIT config: failed to get JIT config: POST https://api.github.com/orgs/<org>/actions/runners/generate-jitconfig: 403 API rate limit of 5000 still exceeded until 2026-02-11 16:14:55 +0100 CET, not making remote request. [rate reset in 7m57s]" pool_id=f2a3662a-bc35-4c0a-9c3a-db0bb944f03c pool_mgr=<org> endpoint=github.com pool_type=organization
...

I guess we should slow that down? I am not sure how to interpret "not making remote request." - I was unable to find that part in the code?
Are we actually hitting the API or not? The message says no, but the POST 403 could suggest otherwise?

[gabriel-samfira]

not making remote request

this is from the github client package.

The green line is the remaining rate limit, the yellow line is GetEntityJITConfig on github operations.

That spike is huge. Within the course of 5 minutes it finished your rate limit. There must be some logs. Would you mind setting log level to debug?

Were you using regular pools or trying out scale sets?

[SystemKeeper]

Within the course of 5 minutes it finished your rate limit.

No, the rate limit was done (green line at 0) and then the spike occurred. But yea, it seems higher than usual.

Regular pools are used. I'll send you the logs I currently have (also for the other issue), but I surely can do try debug.

[gabriel-samfira]

Ahh. I understand now. Let me create a PR with a backoff when rate limits are reached. In main we now have a better view of rate limits. We should be able to make better choices.