Skip to content

Conversation

@43jay
Copy link
Collaborator

@43jay 43jay commented Oct 9, 2025

📜 Description

Initalise DefaultReplayBreadcrumbConverter as the SDK-wide BeforeBreadcrumbCallback. It is responsible for delegating to a user-provided BeforeBreadcrumb.

Introduce new utility/data classes (io.sentry.util.network pkg), following format in the javascript SDK.

In BeforeBreadcrumb, NetworkDetailCaptureUtils extracts 'Network Details' data objects (NetworkRequestData, ReplayNetworkRequestOrResponse) and inserts back into the breadcrumb Hint (only sentry-okhttp for now).

When converting Breadcrumb to RRWebSpanEvent, extract NetworkRequestData data from Hint (name="replay:networkDetails") and insert into the replay performanceSpan when creating the replay segment.

TODOs

Feedback:

  • Are we handling 'enough' body types? Given there are so many diff types of request bodies ->
    Current impl handles JSON, UrlFormEncoded, skips these binary content types, and falls back to raw String as possible

  • Decide on keeping changes to sentry-samples-android

  • Will this work when the gradle plugin is used to add the SentryOkHttpInterceptor? This code implies we only send the breadcrumb w/ network details when that path isn't active.

Implementation:

  • Restrict size of breadcrumbsMap entries after data has been added to replay
  • Respect networkDetail* SDK Options flags
    • Will handle in a Future PR
  • handle case where SentryOkHttpEventListener handles http request instrumentation
    • Will handle in a Future PR
  • More unit tests after addressing the PR feedback, e.g.:
    • SentryOkHttpInterceptor
    • AndroidOptionsInitializer
    • NetworkBodyParser (at least the json part)

Pre-Land:

  • Go through commits and remove any testing-oriented ones
    • Remove FAKE_OPTIONS in SentryOkHttpInterceptor before landing // there for testing
  • Remove remaining unnecessary debug log statements

💡 Motivation and Context

Part of [Mobile Replay] Capture Network request/response bodies

Initially, we were trying to keep SDK changes simple and re-use the existing OKHTTP_REQUEST|RESPONSE hint data.
However, the :sentry-android-replay gradle module doesn't compile against any of the http libs (makes sense).

Because these okhttp3.Request, etc types don't exist in :sentry-android-replay, replay accesses the NetworkDetails data via Hint data ("replay:networkDetails") on the Breadcrumb, when the SDKOptions constraints have been met.

💚 How did you test it?

  1. DefaultReplayBreadcrumbConverterTest unit tests:

./gradlew clean :sentry-android-replay:testDebugUnitTest --no-build-cache --rerun-tasks --tests="DefaultReplayBreadcrumbConverterTest"

  1. Manually creating replay sessions:

this replay session is an example recording with no networkDetail* SDK options enabled
(replay sessions below were captured without commit 4bc5b7714).

recording segment data shows sample replay payload .

replay session respects networkDetail[Request|Response]Headers

https://sentry-sdks.sentry.io/explore/replays/f89535ea0e79499ca8d75e34e6270925

image image
replay session captures GET request bodies as null

replay session
image

replay session captures JSON bodies

1. Request body formatted as JSON key/values
ref
image

2. Response body formatted as JSON key/values
ref
image

replay session captures plaintext bodies

replay session
image

replay session captures form bodies (formurlencoded)

replay session
image

replay session captures binary bodies

1. binary body under size limit (valid response size; summarized body content

https://sentry-sdks.sentry.io/explore/replays/baf899dd12c14718bc5d8b877a9fff3d
image

2. binary body over size limit (invalid response size, summarized body content
(https://sentry-sdks.sentry.io/explore/replays/94dc96ae2b8c4f4d8c23e2e0c69e5d1d)
image

replay session truncates plaintext bodies over 150KB (`MAX_NETWORK_BODY_SIZE`)

https://sentry-sdks.sentry.io/explore/replays/e11711a70c164a8fa1a2ed39e8632a05
image

replay sessions properly capture one-shot okhttp request bodies

One-shot HTTP Request Body request
image

Test: read(..) okhttp3.Request.body in an OkHttpInterceptor after cloning body into new request causes the request to succeed ✅ (current impl)

replay session

Valid response body logged to sentry:
image

Sanity Test: read(..) okhttp3.Request.body in an OkHttpInterceptor without cloning body into a new request causes the request to fail ✅

replay session

Invalid response body logged to sentry:
image

@linear
Copy link

linear bot commented Oct 9, 2025

@43jay 43jay marked this pull request as draft October 9, 2025 21:28
cursor[bot]

This comment was marked as outdated.

43jay added 4 commits October 13, 2025 10:52
added some unit tests:
 ./gradlew :sentry-android-replay:testDebugUnitTest --tests="*DefaultReplayBreadcrumbConverterTest*"
Breadcrumb.java has several timestamp fields:
`timestamp: Date`, `timestampMs: Long`, `nanos: Long`

`hashcode` was relying solely on `timestamp`, which can be null depending on which constructor was used.

=> Change to use getTimestamp as 1. this is what equals does (consistency) 2. getTimestamp initialises timestamp if null.
@43jay 43jay changed the title RFF(replay): Adding OkHttp Request/Response bodies for sentry-java replay(feature): Adding OkHttp Request/Response bodies for sentry-java Oct 24, 2025
@43jay 43jay marked this pull request as ready for review October 24, 2025 15:50
Comment on lines +33 to +42
private val httpNetworkDetails =
Collections.synchronizedMap(
object : LinkedHashMap<Breadcrumb, NetworkRequestData>() {
override fun removeEldestEntry(
eldest: MutableMap.MutableEntry<Breadcrumb, NetworkRequestData>?
): Boolean {
return size > MAX_HTTP_NETWORK_DETAILS
}
}
)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Breadcrumb.equals() omits data map, causing HashMap key collisions for concurrent requests, leading to incorrect network data attribution.
Severity: CRITICAL | Confidence: 1.00

🔍 Detailed Analysis

The Breadcrumb.equals() method omits the data map from its comparison. This allows distinct Breadcrumb objects, particularly those generated by HTTP requests occurring within the same millisecond but having different URLs or other data map contents, to be considered equal. When these Breadcrumb objects are used as keys in the httpNetworkDetails LinkedHashMap, one entry can unintentionally overwrite another, causing network details from one request to be incorrectly associated with a different request during session replay processing.

💡 Suggested Fix

Modify Breadcrumb.equals() to include the data map in its comparison, or use a different key for httpNetworkDetails that uniquely identifies each network request, such as a UUID or a composite key including relevant data fields.

🤖 Prompt for AI Agent
Fix this bug. In
sentry-android-replay/src/main/java/io/sentry/android/replay/DefaultReplayBreadcrumbConverter.kt
at lines 33-42: The `Breadcrumb.equals()` method omits the `data` map from its
comparison. This allows distinct `Breadcrumb` objects, particularly those generated by
HTTP requests occurring within the same millisecond but having different URLs or other
`data` map contents, to be considered equal. When these `Breadcrumb` objects are used as
keys in the `httpNetworkDetails` `LinkedHashMap`, one entry can unintentionally
overwrite another, causing network details from one request to be incorrectly associated
with a different request during session replay processing.

Did we get this right? 👍 / 👎 to inform future reviews.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romtsn is there any issue with updating Breadcrumb#equals and Breadcrumb#hashmap to include data in the comparison?

I left it out in my commit cuz assumed we want Breadcrumbs to be equal if they have different data (e.g. it's the same breadcrumb if some more data gets added by a different caller at some point later)

Copy link
Member

@romtsn romtsn Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, that's a tricky one. The problem with data is that people tend to put there any kind of stuff and it can be a deeply-nested data structure which will result in a super expensive equals/hashCode call, so I think doing this generally for all breadcrumbs is gonna backfire.

I guess I see 3 options to address this:

  • Override contains/equals for the LinkedHashMap.Entry (if possible), and define our equals logic there specifically for http breadcrumbs
  • Introduce a transient id to the Breadcrumb object and use it for comparison (or use it as a key in the httpNetworkDetails dictionary)
  • Switch the entire approach to always rely on BeforeBreadcrumbCallback, i.e. we wouldn't grab the breadcrumbs from the Scope here, but instead store them from within our ReplayBeforeBreadcrumbCallback directly and use it later for creating a segment/sending to Sentry. That way we wouldn't need to associate crumbs <-> hints, as we would have them in one place. But that's certainly a bigger lift

Maybe there's some other option I didn't think of - let me know!

Copy link
Collaborator Author

@43jay 43jay Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, yeah - i see there will be an issue if multiple HTTP requests go out in the same millisecond, we will only record one of them.

Given it's an impl detail of Breadcrumb, which already has helpers to create http breadcrumbs - wyt about adding custom equals/hashcode logic for http breadcrumb i.e. something like

@Override
  public boolean equals(Object o) {
    if (this == o) return true;
    if (o == null || getClass() != o.getClass()) return false;
    Breadcrumb that = (Breadcrumb) o;
     
    switch (type) {
      case  'http':
        return httpBreadcrumbEquals(this, that);
      default:
         return breadcrumbEquals(this, that);
    }
  }

private static boolean breadcrumbEquals(Breadcrumb a, Breadcrumb b) {
return a.getTimestamp().getTime() == that.getTimestamp().getTime()
        && Objects.equals(message, that.message)
        && Objects.equals(type, that.type)
        && Objects.equals(category, that.category)
        && Objects.equals(origin, that.origin)
        && level == that.level;
}

private static boolean httpBreadcrumbEquals(Breadcrumb a, Breadcrumb b) {
    return breadcrumbEquals(a, b) 
        && a.getData("status_code") == b.getData("status_code")
        && a.getData("url") == b.getData("url")
        && a.getData("method") == b.getData("method")
        && a.getData("http.fragment") == b.getData("http.fragment")
        && a.getData("http.query") == b.getData("http.query")
}

cursor[bot]

This comment was marked as outdated.

Entrypoint is NetworkDetailCaptureUtils (initializeForUrl) called from SentryOkHttpInterceptor
- common logic to handle checking sdk options.
- Accept data from http client via NetworkBodyExtractor, NetworkHeaderExtractor interfaces
that can be reused in future (if needed)

Placeholder impl for req/resp bodies.

From https://docs.sentry.io/platforms/javascript/session-replay/configuration/
- networkDetailAllowUrls, networkDetailDenyUrls,
- networkCaptureBodies
- networkRequestHeaders, networkResponseHeaders

These SDKOptions don't exist yet => impl acts as if they do, but have not been enabled.
@43jay 43jay requested a review from romtsn October 27, 2025 21:26
@romtsn romtsn changed the title replay(feature): Adding OkHttp Request/Response bodies for sentry-java feat(replay): Adding OkHttp Request/Response bodies Oct 29, 2025
import org.jetbrains.annotations.NotNull;
import org.jetbrains.annotations.Nullable;

/**
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so I brainstormed this a bit, and thinking if it'd make more sense to use composition here -- currently this is a breaking binary change for our hybrid SDKs (RN and Flutter that inherit from DefaultReplayBreadcrumbConverter). I was thinking of ways to avoid the breaking change and came up with the following:

public open class DefaultReplayBreadcrumbConverter() : ReplayBreadcrumbConverter {

  private var options: SentryOptions? = null

  public constructor(options: SentryOptions) : this() {
    this.options = options
    this.options?.beforeBreadcrumb = ReplayBeforeBreadcrumbCallback(options.beforeBreadcrumb)
  }

  private class ReplayBeforeBreadcrumbCallback(
    val delegate: BeforeBreadcrumbCallback?
  ) : BeforeBreadcrumbCallback {
    override fun execute(breadcrumb: Breadcrumb, hint: Hint): Breadcrumb? {
      TODO()
    }
  }

this way the hybrid SDKs wouldn't be broken once we ship this and they'd be just missing on the native req/resp bodies because they are not using the new ctor with options. Once they use the new ctor they automatically would get req/resp bodies for okhttp-originated requests. Wdyt?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

options.setCompositePerformanceCollector(new DefaultCompositePerformanceCollector(options));
}

ReplayBreadcrumbConverter replayBreadcrumbConverter = options.getReplayController().getBreadcrumbConverter();
Copy link
Member

@romtsn romtsn Oct 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this currently has a bug if someone excludes the -replay module from the build, then this would return a NoOpReplayBreadcrumbConverter implementation which does not currently delegate to the user-defined beforeBreadcrumb callback. So we'd be effectively swallowing the breadcrumb in here.

Given my comment here, if we go for that impl, we could change to instantiating and setting DefaultReplayBreadcrumbConverter here:

final @NotNull  converter = options.getReplayController().getBreadcrumbConverter();
if (converter instanceof NoOpReplayBreadcrumbConverter) {
  options.getReplayController().setBreadcrumbConverter(new DefaultReplayBreadcrumbConverter(options));
}

I guess we still have to check for no-op here, because RN and Flutter define their own converters, e.g. see this, so we shouldn't be overwriting whatever they have set.

Copy link
Collaborator Author

@43jay 43jay Nov 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes a lot of sense!

It looks tho like the sdk always overrides the NoopBreadcrumbConverter =>
instanceof NoOpReplayBreadcrumbConverter here will always returns false.
(^? this is what i was uncertain of... but reading the code and adding a log line here confirms at least for my test app)

One option: unwind that behaviour by moving the DefaultReplayBreadcrumbConverter initialization out of installDefaultIntegrations so that the instanceof NoOpReplayBreadcrumbConverter can evaluate to true here + DefaultReplayBreadcrumbConverter can be initialized here.

Alternatively, leave that code alone (which effectively means the NoOpReplayBreadcrumbConverter is dead code? unless there are tests, or other configurations that don't come through installDefaultIntegrations🤔): instead refactor the DefaultReplayBreadcrumbConverter to have post-initialization logic something like

final @NotNull  beforeBreadcrumb = options.getReplayController().getBreadcrumbConverter().getBeforeBreadcrumb();
if (beforeBreadcrumb == null) {
  options.getReplayController().getBreadcrumbConverter()
    .setBeforeBreadcrumb(new ReplayBeforeBreadcrumbCallback(options));
}

Copy link
Collaborator Author

@43jay 43jay Nov 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving the initialization of DefaultReplayBreadcrumbConverter out of installDefaultIntegrations entirely and depending on instanceof NoOpReplayBreadcrumbConverter seems cleaner (first option).

Assuming the initialization was put there b/c previously DefaultReplayBreadcrumbConverter was independent of SDKOptions... but "now it is" => move the entire thing out, instead of try to keep some initialization in installDefaultIntegrations and then do some more initialization later (here in initializeIntegrationsAndProcessors).

I've started working on this (and the refactor to use composition to not change the ReplayBreadcrumbConverter API as well)... any idea if there were other motivations for putting DefaultReplayBreadcrumbConverter initialization inside installDefaultIntegrations, that moving it down to initializeIntegrationsAndProcessors could break?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

/**
* Maximum size in bytes for network request/response bodies to be captured in replays. Bodies
* larger than this will be truncated or replaced with a placeholder message. Default is 150KB
* (153600 bytes).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* (153600 bytes).
* (153600 bytes).
* Aligned with JS: https://github.com/getsentry/sentry-javascript/blob/98de756506705b60d1ca86cbbcfad3fd76062f8f/packages/replay-internal/src/constants.ts#L33

maybe worth linking the JS part here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

* @param value The map representing the JSON object
* @return A NetworkBody instance for the JSON object
*/
static @NotNull NetworkBody fromJsonObject(@NotNull Map<String, Object> value) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we have a tendency to mark all parameters (and members) as final everywhere - would you mind following that pattern? We probably should've adopted checkstyle's FinalParam check a long time ago 😅

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

/**
* Fake options for testing network detail capture
*/
private val FAKE_OPTIONS = object {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this will go away in a future PR, right?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes my plan was to remove it in the next PR when I would introduce the actual SentryOptions API.

I can take it out and in-line emptyArray, etc everywhere if you think that's cleaner though?

safeExtractRequestBody(bodyBytes, originalBody.contentType())
} catch (e: Exception) {
scopes.options.logger.log(
io.sentry.SentryLevel.DEBUG,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we import this? Also, probably worth to change the level to WARNING or ERROR

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

response?.let { resp -> it[OKHTTP_RESPONSE] = resp }

if (networkDetailData != null) {
it.set("replay:networkDetails", networkDetailData)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe worth to add this to TypeCheckHint as a const? I'd also change it to sentry:replayNetworkDetails to align with other hint keys

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}

/** Extracts headers from OkHttp Headers object into a map */
private fun okhttp3.Headers.toMap(): Map<String, String> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering if we can make this potentially faster with less allocations, e.g.:

private fun okhttp3.Headers.toMap(): Map<String, String> {
  val headers = LinkedHashMap<String, String>(size)
  for (i in 0 until size) {
    headers[name(i)] = value(i)
  }
  return headers
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still pending

)
} catch (e: Exception) {
scopes.options.logger.log(
io.sentry.SentryLevel.DEBUG,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably also could be a warning or error

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


// Original breadcrumb http data
for ((key, value) in breadcrumb.data) {
if (key in supportedNetworkData) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i guess some of these keys can override what you set above right? (e.g. responseBodySize). But I guess it should be the same value, so doesn't matter much probably

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that was my take as well - and left it in so that the old impl would 'win' in case there is some issue in the new one

Copy link
Member

@romtsn romtsn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great overall, hats off for all that different body types parsing logic! I left one overall design question and a few of nits.

Also, would be great to cover some of this with more tests after addressing the PR feedback, e.g.:

  • SentryOkHttpInterceptor
  • AndroidOptionsInitializer
  • NetworkBodyParser (at least the json part)

43jay added 12 commits November 4, 2025 12:21
Claude decided to format these files ¯\_(ツ)_/¯
use composition via optional constructor on DefaultReplayBreadcrumbConverter
Instead of modifying the parent interface ReplayBreadcrumbConverter to force all subclasses to implement BeforeBreadcrumbCallback (breaking / harder to manage change)
…lization

DefaultReplayBreadcrumbConverter depends on SDKOptions as it now
 needs to delegate to any user-provided BeforeBreadcrumbCallback
Also got rid of the VisibleForTesting constructor in NetworkRequestData

review comment - https://github.com/getsentry/sentry-java/pull/4796/files#r2474350569
return parse(content, contentType, logger);
} catch (UnsupportedEncodingException e) {
if (logger != null) {
logger.log(SentryLevel.WARNING, "Failed to decode bytes: " + e.getMessage());
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: API contract clash in NetworkBodyParser fromBytes

The NetworkBodyParser.fromBytes method's implementation takes an ILogger parameter, which conflicts with the public API contract expecting SentryOptions. This also creates an inconsistency where the @NotNull annotated logger parameter is unnecessarily null-checked within the method.

Fix in Cursor Fix in Web

// Only capture developer-provided urls.
if (networkDetailAllowUrls == null) {
return false;
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Null-Driven URL Capture Mismatch in Javadoc

The shouldCaptureUrl method's Javadoc for networkDetailAllowUrls indicates null means "allow all", but the code treats null as "allow none". This prevents URL capture when networkDetailAllowUrls is not explicitly set.

Additional Locations (1)

Fix in Cursor Fix in Web

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants