DEPR and move code owner monitoring code with updates

[robrap]

• DEPR of code owner code should happen because this is likely used by 2U-only, so it is just the right thing to do.
• Also, having 2U-only-code would allow us to refactor more easily:
  • We could drop theme. (e.g.code_owner and code_owner_squad would have the same value, and code_owner_theme would be dropped. In theory, we could also drop code_owner_squad, but that might force too much unnecessary work on other teams.
  • We could refactor how celery tasks get code_owner without having to support anything other than Datadog:
    • See Automatically set code owner for celery tasks #729

Implementation thoughts:

• We should experiment with a replacement solution before creating a DEPR ticket, so we know if we have any blockers before we sign ourselves up.
• CodeOwnerMonitoringMiddleware could no longer be added directly into edx-platform here.
  • We could try using edx-platform's mechanism for adding additional middleware, assuming it doesn't run too late. Note: we could experimentally set a temporary span tag and see how many spans get code_owner, but don't get our new test tag.
  • If this didn't work out, we could consider introducing an openedx-event from DeploymentMonitoringMiddleware that allows operators to add their own custom monitoring.
• Could be a separate ticket, but we used to have a New Relic search script for alerts and dashboards. It would be nice to have the same for DD for this type of work, and other monitoring refactors that have come up, like service name changes.
  • Also, see this separate and related ticket: Duplicate New Relic Dashboard and Alert search script for Datadog #786

ACs:

• Code has been moved from edx-django-utils to edx-arch-experiments plugin, so it is no longer public.
• Theme functionality is removed from the codebase, as it is no longer valuable.

[robrap]

Giving this a P3 because it is important, but it is not urgent.

[robrap]

Presumably this was marked blocked while I was out. Marking in-progress again.