Logging large JSON passed in extra blocks main thread

[algokun]

Description

Summary

When passing very large JSON data via the extra parameter, log calls block the main thread because serialization and/or network send happen synchronously in the logging path.

Reproduction

Context: We're using logfire to log the responses we get from API and when the data is too large (~10MB) it's blocking the main thread and the caller has to wait 2x the time because this instrumentation is blocking this.

try:
    response_parsing_start_time = datetime.now()
    response_body = json.loads(response_data)
    response_parsing_duration_sec = (datetime.now() - response_parsing_start_time).total_seconds()
    print(f"Response parsing duration: {response_parsing_duration_sec} seconds")
    request_log["response"]["body"] = response_body

    # Log successful JSON response
    logfire_logging_start_time = datetime.now()
    logger.info(
        "Request completed successfully with JSON response",
        extra={
            "status_code": response.status,
            "duration_ms": attempt_duration,
            "response_size_bytes": len(response_data),
            "response_body": response_body,
            "total_attempts": retry_count + 1,
        },
    )
    logfire_logging_duration_sec = (datetime.now() - logfire_logging_start_time).total_seconds()
    print(f"Logfire logging duration: {logfire_logging_duration_sec} seconds")

    # Success - break out of while loop and go to finally
    break

Here are my logs when this function ran -

Response parsing duration: 0.082300 seconds
Logfire logging duration: 118.229488 seconds

I know removing this can solve this entirely and coming up with other ways to take this forward, but I'm interested in knowing why the implementation isn't following the OTEL Standards.

From the OTEL docs, I could find the key principles to operate this with, so just curious to understand this deeper.

- Library should not block end-user application by default.
- Library should not consume unbounded memory resource.

Python, Logfire & OS Versions, related packages (not required)

logfire="3.21.2"
platform="macOS-26.2-arm64-arm-64bit"
python="3.10.16 (main, Dec 11 2024, 10:22:29) [Clang 14.0.6 ]"
[related_packages]
requests="2.32.2"
pydantic="2.9.0"
fastapi="0.115.0"
openai="1.93.3"
protobuf="4.25.7"
rich="13.6.0"
tomli="2.2.1"
executing="2.2.0"
opentelemetry-api="1.27.0"
opentelemetry-exporter-otlp-proto-common="1.27.0"
opentelemetry-exporter-otlp-proto-http="1.27.0"
opentelemetry-instrumentation="0.48b0"
opentelemetry-instrumentation-aiohttp-client="0.48b0"
opentelemetry-instrumentation-asgi="0.48b0"
opentelemetry-instrumentation-asyncpg="0.48b0"
opentelemetry-instrumentation-fastapi="0.48b0"
opentelemetry-instrumentation-httpx="0.48b0"
opentelemetry-instrumentation-pymongo="0.48b0"
opentelemetry-instrumentation-redis="0.48b0"
opentelemetry-instrumentation-requests="0.48b0"
opentelemetry-instrumentation-system-metrics="0.48b0"
opentelemetry-proto="1.27.0"
opentelemetry-sdk="1.27.0"
opentelemetry-semantic-conventions="0.48b0"
opentelemetry-util-http="0.48b0"

[pipinstalled]

@alexmojaki is this free to work on it?