Verify deployments

[mmarchetti]

This PR introduces a new feature that verifies that content items are live and accessible after deployment. This is most useful for apps and APIs. Accessing the content causes Connect to launch a worker process to service the request; this will catch apps that fail to start up because of import errors or exceptions raised on import.

There is a new CLI flag --no-verify to prevent this behavior in cases where it's not wanted (for example, if app startup is expensive or might have side effects).

Intent

Fixes #503

Type of Change

• Bug Fix
• New Feature
• Breaking Change

Automated Tests

There are integration tests that cover the new behavior and the flag.

Directions for Reviewers

Deploy all kinds of content, including some apps that fail to start. Check that the verification catches those. Also deploy with --no-verify and check that even faulty apps can be deployed without rsconnect detecting the error.

Also deploy to a server with a path prefix (like Yeti or Colorado), since the path prefix is used when accessing the app.

Checklist

• I have updated CHANGELOG.md to cover notable changes.
• I have updated all related GitHub issues to reflect their current state.

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
4268	2794	65%	0%	🟢

New Files

No new covered files...

Modified Files

File	Coverage	Status
rsconnect/api.py	68%	🟢
rsconnect/main.py	55%	🟢
TOTAL	62%	🟢

updated for commit: 9fbc8f3 by action🐍

[kgartland-rstudio]

Verified on a number of apps.

Launching Dash application...
Deployment completed successfully.
         Dashboard content URL: https://rsc.radixu.com/connect/#/apps/54798b06-793a-4fe5-9ffc-28861b6b16ec
         Direct content URL: https://rsc.radixu.com/content/54798b06-793a-4fe5-9ffc-28861b6b16ec/
Verifying deployed content...   [ERROR]: Could not access the deployed content. The app might not have started successfully. Visit it in Connect to view the logs.

It also found a potential bug! When deploying a sanic api to Connect, we don't display anything at the root directory but we can hit the endpoint via curl so our current testing has been passing. I may need to set --no-verify on those tests until we get that fixed. Other than that I think this looks good. I'll have a PR out on the Connect side to avoid failures in the meantime.

[kgartland-rstudio]

Also concerning the issue I hit above, I wonder if we want to make no-verify the default behavior and have a flag for --verify so we don't suddenly impact customer's current workflow.

[kgartland-rstudio]

One more note...sanic actually does eventually load but the sainc-routing can take its time. So we're really calling something a failure that will eventually succeed. I'll test this further with longer running apps and notebooks.

We may want to hold off on this or adjust the default behavior as I mentioned.

[nealrichardson]

Personally, I think there are lots of benefits to verifying such that it should be the default. But if we wanted to do a release now where it is opt-in, and then change the default to opt-out in a following release after we've had more time to catch issues, that seems reasonable.

[mmarchetti]

The verify check should honor CONNECT_REQUEST_TIMEOUT, so there is a way to control how patient it is when waiting for the app to come up/respond.

[kgartland-rstudio]

How about we adjust the failure from an ERROR to a WARNING:

#   Verifying deployed content... 	[ERROR]: Could not access the deployed content. The app might not have started successfully. Visit it in Connect to view the logs.
#   Error: Could not access the deployed content. The app might not have started successfully. Visit it in Connect to view the logs.

[mmarchetti]

I'll test this further with longer running apps and notebooks.

Notebooks should be OK, since we wait until the rendering is complete so the "verify" step is loading the rendered notebook HTML. You would need to set CONNECT_TASK_TIMEOUT if the total deployment process, including rendering the notebook, will take >24 hours (the default).

[kgartland-rstudio]

Okay all looks good. Also looks like my sanic issue is only occuring in the Dockerized Connect instance, it works as expected on dogfood. We can merge this and then i'll add the --no-verify flag for the sanic deploy in Connect to avoid failures in CI.