Skip to content

Some persistence callers have no deadline, so they stay blocked on a silent Postgres connection #12449

Description

@sangkyoonnam

Follow-up to #12246, outside the shard io semaphore that #12441 covers.

Expected Behavior

Persistence calls end once a bounded deadline passes.

Actual Behavior

Three background callers pass no per-call deadline, so on a connection that goes silent without a reset they can stay blocked on either driver:

  • nsregistry.(*registry).refreshNamespaces (starts from context.Background())
  • ringpop.(*monitor).upsertMyMembership (lifecycle context)
  • matching taskManagerImpl.CreateTasks (WithCancel(Background()))

In my #12246 repro with postgres12_pgx, the matching writer still stalled and workflows stopped. With lib/pq, calls that do have a deadline block too (lib/pq#620).

Steps to Reproduce the Problem

  1. Run the server against Postgres through HAProxy with a steady workflow load.
  2. Black-hole established client connections without a reset.
  3. Dump goroutines after a few minutes.

Specifications

I'd add per-call deadlines to those three callers and a docs note recommending postgres12_pgx for PostgreSQL. OK to send a PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions