Traceback from agent on max spans

[ccorbacho]

Describe the bug: ...

We are seeing the following traceback coming out of the ElasticAPM agent via Sentry frequently on many of our environments.

KeyError: 'TRANSACTION_MAX_SPANS'
  File "elasticapm/utils/threading.py", line 80, in run
    rval = self._function(*self._args, **self._kwargs)
  File "elasticapm/conf/__init__.py", line 701, in update_config
    self.reset()
  File "elasticapm/conf/__init__.py", line 650, in reset
    if self._config.values[key] != self._first_config.values[key]:

To Reproduce

Unknown. However, Sentry shows a lot of POST requests going to the APM server before this happens. Our assumption is that we are hitting TRANSACTION_MAX_SPANS, but this is not being handled gracefully, and causing this exception instead.

Environment (please complete the following information)

• OS: Linux
• Python version: 3.8
• Framework and version: Django 3.1.7
• APM Server version: 7.6
• Agent version: 6.0.0

[beniwohli]

Hey @ccorbacho. Good find! The problem here occurs in our remote configuration updater. Do you per chance set the transaction_max_spans configuration via remote config in Kibana?

Specifically, from the looks of it, this would happen if a setting is configured via Kibana, but then later on removed. If we detect that a remote configuration has been removed, we revert to the original configuration, and hit this buggy code path. Does this sound like something that could have happened in your setup? If not, we'll have to figure out why else this code path could be hit.

The bug itself should be fairly easy to fix. I pushed a test for it in #1068, and will add a fix in a bit.

[ccorbacho]

Yes, we do currently have TRANSACTION_MAX_SPANS set via the remote config as well.