Description

What happened:

We are attempting a POC using the teleport-cluster Helm chart running multiple replicas. For more context we are running in AWS and using Dynamo+S3 for state and sessions.

I was noticing that upon opening an app from the UI we would randomly get a 403 in the client and a blank page.

Upon looking at the proxy logs we could see the following WARN related to the requests that would 403:

WARN [APP:WEB]   Request failed: session does not exist. app/fragment.go:99

Upon further digging I could see the following happening:

1. User hits server1 and the session is created
2. Next request hits server 2 and the "session does not exist" error is present

I could see that the session does exist in Dynamo so it made little sense. After further debugging I could see that Teleport was failing to find the session in the sqlite backend (which I assume is the cache).

We had not turned on caching explicitly but after finding #3216 I decided to turn off caching and see what would happen.

With caching turned off I did not receive the "session does not exist" error again with multiple replicas and everything appears to work without issue.

What you expected to happen:

I would expect that everything would work OOTB, and that is multiple replicas to work fine with the default caching turned on.

Reproduction Steps

As minimally and precisely as possible, describe step-by-step how to reproduce the problem.

1. Deploy the teleport-cluster helm chart into AWS with multiple replicas
2. Attempt to load an application a good number of times

Server Details

• Teleport version (run teleport version): 6.2.1
• Server OS (e.g. from /etc/os-release): 6.2.1
• Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): AWS
• Additional details: Using Dynamo+S3 as required for HA configuration

Client Details

• Tsh version (tsh version): 6.2.1
• Computer OS (e.g. Linux, macOS, Windows): Linux
• Browser version (for UI-related issues): n/a
• Installed via (e.g. apt, yum, brew, website download):
• Additional details:

Debug Logs

Please include or attach debug logs, when appropriate. Obfuscate sensitive information!

• Start Teleport with --debug flag (teleport --debug)
• Run tsh with --debug flag (tsh --debug)