Allow the application to start without a database connection

[pietroalbini]

This PR implements the last bit of #3541, making sure the server binary can start without a database connection.

To avoid serving the initial requests without a connection, the server will now wait for 5 seconds at startup to initialize the database connections, and if it can't do that in that timeframe it will treat the database as unhealthy and continue starting up the server.

This is critical to our continued availability, so to make sure this doesn't regress I also implemented a test that fails if a database connection becomes a requirement to start the server again. The test is implemented by starting the server binary with the right environment variables and doing proper HTTP requests to it.

Fixes #3541
r? @jtgeibel

[pietroalbini]

Need to investigate the tests failing later, but the PR is still ready for review. My guess is that CI doesn't have the right environment variables configured.

[jtgeibel]

Looks great to me! The tests passed for me locally, but if I comment out either GH_CLIENT_ID or GH_CLIENT_SECRET in my .env then the server fails to start. Interestingly, these can be set to an empty string and the tests will pass.

Eventually we could probably hook GitHub and S3 requests into our existing HTTP recording code, but for now its fine if we just document that these tests should not hit endpoints that could trigger external HTTP requests.

[Turbo87]

Eventually we could probably hook GitHub and S3 requests into our existing HTTP recording code

I think GitHub and S3 are exactly what is already hooked up, or am I missing something? 🤔

[jtgeibel]

I think GitHub and S3 are exactly what is already hooked up, or am I missing something?

The 2 new ServerBin tests actually boot the server, configured via its environment variables. Our existing tests link to the library crate, and configure an App instances for each test. In particular, those tests configure the S3 uploader (api_protocol) and GitHub API (gh_base_url) to use plaintext HTTP to a per-test proxy. (As far as I can tell, we don't plaintext proxy GitHub OAuth requests in our current tests, because the actual navigation to GitHub happens client side in the frontend.)

My understanding is that for now, if we tried to hit say the crate publish endpoint in a ServerBin test, then the server binary would try to make HTTPS requests to S3 and GitHub using whatever credentials it has available (possibly picking them up from a local .env file). We can't leak test requests out into the wild, but I also don't want to build a configuration switch into release binaries that effectively switches all our outgoing requests to plaintext.

We should probably add an environment variable that disables all outgoing requests, returning an error instead. But I don't think it is necessary to block this PR on that.

[bors]

☔ The latest upstream changes (presumably #3596) made this pull request unmergeable. Please resolve the merge conflicts.

[pietroalbini]

Resolved the conflict and (hopefully!) fixed CI.

[jtgeibel]

LGTM!! @bors r+

[bors]

📌 Commit 1523948 has been approved by jtgeibel

[bors]

⌛ Testing commit 1523948 with merge 509d7a2...

[bors]

☀️ Test successful - checks-actions
Approved by: jtgeibel
Pushing 509d7a2 to master...