Add monitoring for login failures

[dianakhuang]

Follow up to:

• Draft RCA timeline and data for June 2 login issue #1064

We would like to add observability and monitoring in this particular situation. It could possibly be duplicated in a development environment by replicating the broken CORS mismatch issue where the old code (pre-openedx/edx-platform#36757) was still being run but the settings were updated to the post-merge version (https://github.com/edx/edx-internal/pull/12829). This might be possible due to the way docker networking works, but unclear.

Another possibility would be to simply add more observability around login failures and then add monitoring to track when there are large spikes.

Acceptance Criteria

• A monitor in Datadog to detect this issue
• (reach) Observability to discover the root cause more quickly

[robrap]

@vgulati-apphelix: This was newly added to the board. Let me know if you have questions about it. Thank you.

[ktyagiapphelix2u]

@dianakhuang @robrap I’ve been trying to reproduce the login issue related to the CORS mismatch from #36757. I have edx-platform checked out at the Merge pull request openedx/edx-platform#36570 from Talha-Rizwan/calculate_grades_csv-to-drf, and my settings include the following CSRF_TRUSTED_ORIGINS:

['http://localhost:2002', 'http://localhost:2001', 'http://localhost:1997', 'http://localhost:1995', 'http://localhost:1992', 'http://localhost:2002', 'http://localhost:1991', 'http://localhost:8734', 'http://localhost:1999', 'http://localhost:18450', 'http://localhost:1994', 'http://localhost:1996']
My CSRF_TRUSTED_ORIGINS_WITH_SCHEME is currently an empty list.

Despite this setup, I’m not able to trigger the CORS-related login failure that occurred during the June 2 incident. Is there something subtle I might be missing? Possibly related to how origin headers are handled in newer Django versions or how Docker networking behaves in this context?

Just want to make sure I’m not overlooking a mismatch that’s necessary to recreate the failure. Appreciate any insights you might have!

[robrap]

What if CSRF_TRUSTED_ORIGINS_WITH_SCHEME is non-empty? I'm just guessing. Diana might know better.

[dianakhuang]

It could also be that this isn't easy or possible when all the services are running on the same host, since that is not the case on production.

[robrap]

If what @dianakhuang says is a blocked, then we could discuss trying to break this in Stage after I return. That would be complicated, but possible (using a paused pipeline and the security pipeline), and something to discuss when I return.

[ktyagiapphelix2u]

@dianakhuang @robrap @UsamaSadiq

I was able to reproduce the issue by changing the host to 127.0.0.1. I can confirm that the relevant logs are visible on screen. Please see the attached screenshot. A specific monitor can be put in place to track them.

[jristau1984]

This has been migrated to https://2u-internal.atlassian.net/browse/BOMS-67