How to determine when new container is truly "ready"

[rarkins]

To wait for a postgres container to be "ready", I've been using a script like this:

# Wait for PG to be ready
until $PSQL -c "select version()" &> /dev/null
do
    echo "waiting for postgres container..."
    sleep 2
done

But it seems even this is too early. Is there any unequivocal way that another container can tell once the postgres container is ready to be provisioned, without resorting to more arbitrary wait timers?

[spmacdonald]

I have done this:

while ! pg_isready 
do
    echo "$(date) - waiting for database to start"
    sleep 10
done

and it seems to work, but I am still learning...

[rarkins]

Hi @spmacdonald,
Your sleep 10 is still what I'd call an "arbitrary wait timer". The challenge (problem?) with the current postgres image is that it is essentially "ready" twice - so the first time it reports ready it's not really ready:

Here's the first time:

postgres          | LOG:  database system is ready to accept connections
postgres          | LOG:  autovacuum launcher started
postgres          |  done
postgres          | server started
postgres          | ALTER ROLE

Then it runs its entrypoints, e.g. here is an example if adding postgis:

postgres          | /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/postgis.sh
postgres          | CREATE DATABASE
postgres          | UPDATE 1
postgres          | Loading PostGIS extensions into template_postgis
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION
postgres          | Loading PostGIS extensions into postgres
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION

And then the restart:

postgres          | LOG:  received fast shutdown request
postgres          | LOG:  aborting any active transactions
postgres          | LOG:  autovacuum launcher shutting down
postgres          | waiting for server to shut down....LOG:  shutting down
postgres          | LOG:  database system is shut down
postgres          |  done
postgres          | server stopped
postgres          |
postgres          | PostgreSQL init process complete; ready for start up.
postgres          |
postgres          | LOG:  database system was shut down at 2016-04-29 03:31:41 UTC
postgres          | LOG:  MultiXact member wraparound protections are now enabled
postgres          | LOG:  database system is ready to accept connections
postgres          | LOG:  autovacuum launcher started

And only now is it truly ready. If you kicked off a seeding/provisioning script once you saw PG "ready" the first time, you're likely to hit a failure when it restarts. I think all containers need to have a deterministic way to indicate when they're actually-truly "won't restart on you" ready without resorting to sleep timers.

[md5]

@rarkins What is the $PSQL in your command? Is it connecting to a separate container? If so, then I'm surprised that it's "too early". The initialization process you're talking about in this container should not be exposing ports in a way that anything outside the container can connect.

[rarkins]

@md5 sorry, $PSQL is just an alias defined earlier in my file:

PSQL="psql -h localhost -p 5432 -U postgres -v ON_ERROR_STOP=1"

I am using host networking in docker, which might explain the difference?

[md5]

@rarkins You're going to have a hard time with a lot of assumptions in various Docker images if you're using host networking.

[yosifkit]

The way the container is designed is that when it is listening on the network port via its non-loopback interface, it is ready to accept connections. Accepting connections on the loopback connection is for the scripts in /docker-entrypoint-initdb.d/; once the scripts are done, the server is killed and then started as pid 1 of the container.

[gravis]

The best way to wait for your container to be up and ready is this:

until docker run -it --rm --link [YOUR-PG-CONTAINER-NAME]:pg postgres:9.5 psql -U postgres -h pg -c "select 1" -d postgres; do sleep 1; done

That's what we're using at Gemnasium, it avoid the "double start" issue.

[sydcanem]

We do sleep 2 after done to accomodate the next restart.

until pg_isready
do
    echo "."
    sleep 1
done
sleep 2

[yelizariev]

improved version of @gravis's solution

 until docker run --rm --link [YOUR-PG-CONTAINER-NAME]:pg postgres:9.5 pg_isready -U postgres -h pg; do sleep 1; done

it works even if password is required

Update as noted by @enumag if you use network, then you need to add --net too:

until docker run --rm --link [CONTAINER]:pg --net [NETWORK] postgres:9.5 pg_isready -U postgres -h pg; do sleep 1; done

[jasonkeene]

@gravis @yelizariev Thank you for your solutions!

[fdietze]

Is the -w option for pg_ctl helpful here?
https://www.postgresql.org/docs/current/static/app-pg-ctl.html

-w
Wait for the startup or shutdown to complete. Waiting is the default option for shutdowns, but not startups. When waiting for startup, pg_ctl repeatedly attempts to connect to the server. When waiting for shutdown, pg_ctl waits for the server to remove its PID file. This option allows the entry of an SSL passphrase on startup. pg_ctl returns an exit code based on the success of the startup or shutdown.

[valk8]

As @rarkins I've struggled with the server restart too. pg_isready is not helpful if you apply a schema, seed a data right away (e.g CI workflow).
I got inspired by https://github.com/vishnubob/wait-for-it
which Docker mentions in its docs here https://docs.docker.com/compose/startup-order/

HOST=db
PORT=5432
RETRIES=10

until docker-compose exec container bash -c '(echo > /dev/tcp/$HOST/$PORT) > /dev/null 2>&1' || [ $RETRIES -eq 0 ]; do
    echo "Waiting for Postgres server, $((RETRIES--)) remaining attempts..."
        sleep 1
    done

[cjlint]

The simplest solution for me was watching the logs for PostgreSQL init process complete; ready for start up, and then watching the logs for database system is ready to accept connections

[vitorreis]

thanks for the solution @gravis