Skip to content

Manager Shutdown/Cache Establishment low-impact race-thing #1457

Closed
@DirectXMan12

Description

@DirectXMan12

If you end up stopping the manager just after you start it, you end up manifesting a weird shutdown "race" in which you're still in the middle of controller startup/cache establish.

Fortunately as of newer revisions of controller-runtime (0.7x+ AFAICT), this manifests as a "error received after stop sequence was engaged" log line, which is ok but not great (v0.6.x manifests it as a graceful shutdown timeout, which is 😞 ).

It's not fatal, but it'd be nice if there were a way to avoid this, since it tends to show up a bit in certain test scenarios (e.g. manager start in beforeach, some tests that don't actually gate on controller actions).

AFAICT, you can't just wait for cache establishment, cause you'll race with actually having the controllers register themselves.

We could perhaps avoid this with a concept of runnable readiness, and a corresponding way to get the overall manager readiness. Then, you can wait in your test code for readiness to finish (nice side effect: we get a readyz endpoint "for free").

Reproducer: https://gist.github.com/DirectXMan12/6edc4c1d917b121339e2a4324432d9e7

Metadata

Metadata

Assignees

No one assigned

    Labels

    lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions