Description
- Package Name: azure-monitor-opentelemetry
- Package Version: 1.3.0
- Operating System: MacOS
- Python Version: 3.10.13
Describe the bug
With default configuration (absence of configuration), time of execution of configure_azure_monitor()
takes abnormally long time: ~10 seconds.
To Reproduce
long.py:
import time
from azure.monitor.opentelemetry import configure_azure_monitor
start = time.monotonic()
configure_azure_monitor()
print(time.monotonic() - start)
APPLICATIONINSIGHTS_CONNECTION_STRING="..." python long.py
Expected behavior
Reasonable time to configure (< 1 s).
Additional context
After running in debugger I discovered two main code places contributing to the delay, and both are related to checking the fact of running in an Azure VM.
- Resource detection.
The way code gets there:
2. Statsbeat metrics
Location:
Call stack:
In both cases the delay is related to requests to this endpoint:
http://169.254.169.254/metadata/instance/compute
though, to different API versions. The first place has request timeout of 4 seconds, and the second place has 5 seconds, which together constitute almost the entire time of the startup delay.
Workarounds
- Exclude Azure resource detectors with help of setting
OTEL_EXPERIMENTAL_RESOURCE_DETECTORS=otel
environment variable. If not set, the library sets the default value, that includes App Service and Azure VM. - Disable sending statsbeat using
APPLICATIONINSIGHTS_STATSBEAT_DISABLED_ALL=TRUE
The above tweaks bring the configuration time down to ~0.8 s (and with OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
set to azure_sdk,django,fastapi,flask,psycopg2,requests,urllib,urllib3
it completes under 30 ms).
It took me hours to find the above options for fixing the startup time without touching the code. I think we need to make the library friendlier to running in non-Azure environments.