-
-
Notifications
You must be signed in to change notification settings - Fork 461
feat(replay): Adding OkHttp Request/Response bodies #4796
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…dcrumbCallback to extract NetworkRequestData from Hint -> NetworkRequestData is landing on DefaultReplayBreadcrumbConverter via Hint.get(replay:networkDetails)
...ry-android-replay/src/main/java/io/sentry/android/replay/DefaultReplayBreadcrumbConverter.kt
Outdated
Show resolved
Hide resolved
sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java
Outdated
Show resolved
Hide resolved
sentry-okhttp/src/main/java/io/sentry/okhttp/SentryOkHttpInterceptor.kt
Outdated
Show resolved
Hide resolved
sentry-okhttp/src/main/java/io/sentry/okhttp/SentryOkHttpInterceptor.kt
Outdated
Show resolved
Hide resolved
...ry-android-replay/src/main/java/io/sentry/android/replay/DefaultReplayBreadcrumbConverter.kt
Outdated
Show resolved
Hide resolved
These get set on the breadcrumb Hint, and then hang around in DefaultReplayBreadcrumbConverter until the replay segment has been sent off See https://github.com/getsentry/sentry-javascript/blob/632f0b953d99050c11b0edafb9f80b5f3ba88045/packages/replay-internal/src/types/performance.ts#L133-L140 https://github.com/getsentry/sentry-javascript/blob/develop/packages/replay-internal/src/types/request.ts#L12
Manually set networkDetailHasUrls to pass this check https://github.com/getsentry/sentry-javascript/blob/d1646c8a281dd8795c5a6a3b8e18f2e7069e7fa9/packages/replay-internal/src/util/handleRecordingEmit.ts#L134 https://github.com/getsentry/sentry/blob/master/static/app/views/replays/detail/network/details/getOutputType.tsx#L33 https://github.com/getsentry/sentry/blob/master/static/app/views/replays/detail/network/details/content.tsx#L55-L56
added some unit tests: ./gradlew :sentry-android-replay:testDebugUnitTest --tests="*DefaultReplayBreadcrumbConverterTest*"
Breadcrumb.java has several timestamp fields: `timestamp: Date`, `timestampMs: Long`, `nanos: Long` `hashcode` was relying solely on `timestamp`, which can be null depending on which constructor was used. => Change to use getTimestamp as 1. this is what equals does (consistency) 2. getTimestamp initialises timestamp if null.
| private val httpNetworkDetails = | ||
| Collections.synchronizedMap( | ||
| object : LinkedHashMap<Breadcrumb, NetworkRequestData>() { | ||
| override fun removeEldestEntry( | ||
| eldest: MutableMap.MutableEntry<Breadcrumb, NetworkRequestData>? | ||
| ): Boolean { | ||
| return size > MAX_HTTP_NETWORK_DETAILS | ||
| } | ||
| } | ||
| ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bug: Breadcrumb.equals() omits data map, causing HashMap key collisions for concurrent requests, leading to incorrect network data attribution.
Severity: CRITICAL | Confidence: 1.00
🔍 Detailed Analysis
The Breadcrumb.equals() method omits the data map from its comparison. This allows distinct Breadcrumb objects, particularly those generated by HTTP requests occurring within the same millisecond but having different URLs or other data map contents, to be considered equal. When these Breadcrumb objects are used as keys in the httpNetworkDetails LinkedHashMap, one entry can unintentionally overwrite another, causing network details from one request to be incorrectly associated with a different request during session replay processing.
💡 Suggested Fix
Modify Breadcrumb.equals() to include the data map in its comparison, or use a different key for httpNetworkDetails that uniquely identifies each network request, such as a UUID or a composite key including relevant data fields.
🤖 Prompt for AI Agent
Fix this bug. In
sentry-android-replay/src/main/java/io/sentry/android/replay/DefaultReplayBreadcrumbConverter.kt
at lines 33-42: The `Breadcrumb.equals()` method omits the `data` map from its
comparison. This allows distinct `Breadcrumb` objects, particularly those generated by
HTTP requests occurring within the same millisecond but having different URLs or other
`data` map contents, to be considered equal. When these `Breadcrumb` objects are used as
keys in the `httpNetworkDetails` `LinkedHashMap`, one entry can unintentionally
overwrite another, causing network details from one request to be incorrectly associated
with a different request during session replay processing.
Did we get this right? 👍 / 👎 to inform future reviews.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@romtsn is there any issue with updating Breadcrumb#equals and Breadcrumb#hashmap to include data in the comparison?
I left it out in my commit cuz assumed we want Breadcrumbs to be equal if they have different data (e.g. it's the same breadcrumb if some more data gets added by a different caller at some point later)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm, that's a tricky one. The problem with data is that people tend to put there any kind of stuff and it can be a deeply-nested data structure which will result in a super expensive equals/hashCode call, so I think doing this generally for all breadcrumbs is gonna backfire.
I guess I see 3 options to address this:
- Override
contains/equalsfor the LinkedHashMap.Entry (if possible), and define our equals logic there specifically for http breadcrumbs - Introduce a transient
idto theBreadcrumbobject and use it for comparison (or use it as a key in thehttpNetworkDetailsdictionary) - Switch the entire approach to always rely on
BeforeBreadcrumbCallback, i.e. we wouldn't grab the breadcrumbs from the Scope here, but instead store them from within ourReplayBeforeBreadcrumbCallbackdirectly and use it later for creating a segment/sending to Sentry. That way we wouldn't need to associate crumbs <-> hints, as we would have them in one place. But that's certainly a bigger lift
Maybe there's some other option I didn't think of - let me know!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, yeah - i see there will be an issue if multiple HTTP requests go out in the same millisecond, we will only record one of them.
Given it's an impl detail of Breadcrumb, which already has helpers to create http breadcrumbs - wyt about adding custom equals/hashcode logic for http breadcrumb i.e. something like
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Breadcrumb that = (Breadcrumb) o;
switch (type) {
case 'http':
return httpBreadcrumbEquals(this, that);
default:
return breadcrumbEquals(this, that);
}
}
private static boolean breadcrumbEquals(Breadcrumb a, Breadcrumb b) {
return a.getTimestamp().getTime() == that.getTimestamp().getTime()
&& Objects.equals(message, that.message)
&& Objects.equals(type, that.type)
&& Objects.equals(category, that.category)
&& Objects.equals(origin, that.origin)
&& level == that.level;
}
private static boolean httpBreadcrumbEquals(Breadcrumb a, Breadcrumb b) {
return breadcrumbEquals(a, b)
&& a.getData("status_code") == b.getData("status_code")
&& a.getData("url") == b.getData("url")
&& a.getData("method") == b.getData("method")
&& a.getData("http.fragment") == b.getData("http.fragment")
&& a.getData("http.query") == b.getData("http.query")
}
sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java
Outdated
Show resolved
Hide resolved
...ry-android-replay/src/main/java/io/sentry/android/replay/DefaultReplayBreadcrumbConverter.kt
Outdated
Show resolved
Hide resolved
Entrypoint is NetworkDetailCaptureUtils (initializeForUrl) called from SentryOkHttpInterceptor - common logic to handle checking sdk options. - Accept data from http client via NetworkBodyExtractor, NetworkHeaderExtractor interfaces that can be reused in future (if needed) Placeholder impl for req/resp bodies. From https://docs.sentry.io/platforms/javascript/session-replay/configuration/ - networkDetailAllowUrls, networkDetailDenyUrls, - networkCaptureBodies - networkRequestHeaders, networkResponseHeaders These SDKOptions don't exist yet => impl acts as if they do, but have not been enabled.
| import org.jetbrains.annotations.NotNull; | ||
| import org.jetbrains.annotations.Nullable; | ||
|
|
||
| /** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so I brainstormed this a bit, and thinking if it'd make more sense to use composition here -- currently this is a breaking binary change for our hybrid SDKs (RN and Flutter that inherit from DefaultReplayBreadcrumbConverter). I was thinking of ways to avoid the breaking change and came up with the following:
public open class DefaultReplayBreadcrumbConverter() : ReplayBreadcrumbConverter {
private var options: SentryOptions? = null
public constructor(options: SentryOptions) : this() {
this.options = options
this.options?.beforeBreadcrumb = ReplayBeforeBreadcrumbCallback(options.beforeBreadcrumb)
}
private class ReplayBeforeBreadcrumbCallback(
val delegate: BeforeBreadcrumbCallback?
) : BeforeBreadcrumbCallback {
override fun execute(breadcrumb: Breadcrumb, hint: Hint): Breadcrumb? {
TODO()
}
}this way the hybrid SDKs wouldn't be broken once we ship this and they'd be just missing on the native req/resp bodies because they are not using the new ctor with options. Once they use the new ctor they automatically would get req/resp bodies for okhttp-originated requests. Wdyt?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
| options.setCompositePerformanceCollector(new DefaultCompositePerformanceCollector(options)); | ||
| } | ||
|
|
||
| ReplayBreadcrumbConverter replayBreadcrumbConverter = options.getReplayController().getBreadcrumbConverter(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this currently has a bug if someone excludes the -replay module from the build, then this would return a NoOpReplayBreadcrumbConverter implementation which does not currently delegate to the user-defined beforeBreadcrumb callback. So we'd be effectively swallowing the breadcrumb in here.
Given my comment here, if we go for that impl, we could change to instantiating and setting DefaultReplayBreadcrumbConverter here:
final @NotNull converter = options.getReplayController().getBreadcrumbConverter();
if (converter instanceof NoOpReplayBreadcrumbConverter) {
options.getReplayController().setBreadcrumbConverter(new DefaultReplayBreadcrumbConverter(options));
}I guess we still have to check for no-op here, because RN and Flutter define their own converters, e.g. see this, so we shouldn't be overwriting whatever they have set.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
makes a lot of sense!
It looks tho like the sdk always overrides the NoopBreadcrumbConverter =>
instanceof NoOpReplayBreadcrumbConverter here will always returns false.
(^? this is what i was uncertain of... but reading the code and adding a log line here confirms at least for my test app)
One option: unwind that behaviour by moving the DefaultReplayBreadcrumbConverter initialization out of installDefaultIntegrations so that the instanceof NoOpReplayBreadcrumbConverter can evaluate to true here + DefaultReplayBreadcrumbConverter can be initialized here.
Alternatively, leave that code alone (which effectively means the NoOpReplayBreadcrumbConverter is dead code? unless there are tests, or other configurations that don't come through installDefaultIntegrations🤔): instead refactor the DefaultReplayBreadcrumbConverter to have post-initialization logic something like
final @NotNull beforeBreadcrumb = options.getReplayController().getBreadcrumbConverter().getBeforeBreadcrumb();
if (beforeBreadcrumb == null) {
options.getReplayController().getBreadcrumbConverter()
.setBeforeBreadcrumb(new ReplayBeforeBreadcrumbCallback(options));
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moving the initialization of DefaultReplayBreadcrumbConverter out of installDefaultIntegrations entirely and depending on instanceof NoOpReplayBreadcrumbConverter seems cleaner (first option).
Assuming the initialization was put there b/c previously DefaultReplayBreadcrumbConverter was independent of SDKOptions... but "now it is" => move the entire thing out, instead of try to keep some initialization in installDefaultIntegrations and then do some more initialization later (here in initializeIntegrationsAndProcessors).
I've started working on this (and the refactor to use composition to not change the ReplayBreadcrumbConverter API as well)... any idea if there were other motivations for putting DefaultReplayBreadcrumbConverter initialization inside installDefaultIntegrations, that moving it down to initializeIntegrationsAndProcessors could break?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done!
| /** | ||
| * Maximum size in bytes for network request/response bodies to be captured in replays. Bodies | ||
| * larger than this will be truncated or replaced with a placeholder message. Default is 150KB | ||
| * (153600 bytes). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| * (153600 bytes). | |
| * (153600 bytes). | |
| * Aligned with JS: https://github.com/getsentry/sentry-javascript/blob/98de756506705b60d1ca86cbbcfad3fd76062f8f/packages/replay-internal/src/constants.ts#L33 |
maybe worth linking the JS part here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
| * @param value The map representing the JSON object | ||
| * @return A NetworkBody instance for the JSON object | ||
| */ | ||
| static @NotNull NetworkBody fromJsonObject(@NotNull Map<String, Object> value) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so we have a tendency to mark all parameters (and members) as final everywhere - would you mind following that pattern? We probably should've adopted checkstyle's FinalParam check a long time ago 😅
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
| /** | ||
| * Fake options for testing network detail capture | ||
| */ | ||
| private val FAKE_OPTIONS = object { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess this will go away in a future PR, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes my plan was to remove it in the next PR when I would introduce the actual SentryOptions API.
I can take it out and in-line emptyArray, etc everywhere if you think that's cleaner though?
| safeExtractRequestBody(bodyBytes, originalBody.contentType()) | ||
| } catch (e: Exception) { | ||
| scopes.options.logger.log( | ||
| io.sentry.SentryLevel.DEBUG, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we import this? Also, probably worth to change the level to WARNING or ERROR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
| response?.let { resp -> it[OKHTTP_RESPONSE] = resp } | ||
|
|
||
| if (networkDetailData != null) { | ||
| it.set("replay:networkDetails", networkDetailData) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe worth to add this to TypeCheckHint as a const? I'd also change it to sentry:replayNetworkDetails to align with other hint keys
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
| } | ||
|
|
||
| /** Extracts headers from OkHttp Headers object into a map */ | ||
| private fun okhttp3.Headers.toMap(): Map<String, String> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wondering if we can make this potentially faster with less allocations, e.g.:
private fun okhttp3.Headers.toMap(): Map<String, String> {
val headers = LinkedHashMap<String, String>(size)
for (i in 0 until size) {
headers[name(i)] = value(i)
}
return headers
}There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
still pending
| ) | ||
| } catch (e: Exception) { | ||
| scopes.options.logger.log( | ||
| io.sentry.SentryLevel.DEBUG, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
probably also could be a warning or error
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
...ry-android-replay/src/main/java/io/sentry/android/replay/DefaultReplayBreadcrumbConverter.kt
Outdated
Show resolved
Hide resolved
|
|
||
| // Original breadcrumb http data | ||
| for ((key, value) in breadcrumb.data) { | ||
| if (key in supportedNetworkData) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i guess some of these keys can override what you set above right? (e.g. responseBodySize). But I guess it should be the same value, so doesn't matter much probably
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that was my take as well - and left it in so that the old impl would 'win' in case there is some issue in the new one
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking great overall, hats off for all that different body types parsing logic! I left one overall design question and a few of nits.
Also, would be great to cover some of this with more tests after addressing the PR feedback, e.g.:
- SentryOkHttpInterceptor
- AndroidOptionsInitializer
- NetworkBodyParser (at least the json part)
Claude decided to format these files ¯\_(ツ)_/¯
This reverts commit 4bc5b77.
use composition via optional constructor on DefaultReplayBreadcrumbConverter Instead of modifying the parent interface ReplayBreadcrumbConverter to force all subclasses to implement BeforeBreadcrumbCallback (breaking / harder to manage change)
…lization DefaultReplayBreadcrumbConverter depends on SDKOptions as it now needs to delegate to any user-provided BeforeBreadcrumbCallback
Also got rid of the VisibleForTesting constructor in NetworkRequestData review comment - https://github.com/getsentry/sentry-java/pull/4796/files#r2474350569
This reverts commit 13c1da8.
| return parse(content, contentType, logger); | ||
| } catch (UnsupportedEncodingException e) { | ||
| if (logger != null) { | ||
| logger.log(SentryLevel.WARNING, "Failed to decode bytes: " + e.getMessage()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bug: API contract clash in NetworkBodyParser fromBytes
The NetworkBodyParser.fromBytes method's implementation takes an ILogger parameter, which conflicts with the public API contract expecting SentryOptions. This also creates an inconsistency where the @NotNull annotated logger parameter is unnecessarily null-checked within the method.
| // Only capture developer-provided urls. | ||
| if (networkDetailAllowUrls == null) { | ||
| return false; | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bug: Null-Driven URL Capture Mismatch in Javadoc
The shouldCaptureUrl method's Javadoc for networkDetailAllowUrls indicates null means "allow all", but the code treats null as "allow none". This prevents URL capture when networkDetailAllowUrls is not explicitly set.
📜 Description
Initalise
DefaultReplayBreadcrumbConverteras the SDK-wideBeforeBreadcrumbCallback. It is responsible for delegating to a user-providedBeforeBreadcrumb.Introduce new utility/data classes (
io.sentry.util.network pkg), following format in the javascript SDK.In
BeforeBreadcrumb,NetworkDetailCaptureUtilsextracts 'Network Details' data objects (NetworkRequestData,ReplayNetworkRequestOrResponse) and inserts back into the breadcrumb Hint (only sentry-okhttp for now).When converting Breadcrumb to
RRWebSpanEvent, extractNetworkRequestDatadata from Hint (name="replay:networkDetails") and insert into the replayperformanceSpanwhen creating the replay segment.TODOs
Feedback:
Are we handling 'enough' body types? Given there are so many diff types of request bodies ->
Current impl handles JSON, UrlFormEncoded, skips these binary content types, and falls back to raw String as possible
Decide on keeping changes to sentry-samples-android
Will this work when the gradle plugin is used to add the SentryOkHttpInterceptor? This code implies we only send the breadcrumb w/ network details when that path isn't active.
Implementation:
networkDetail*SDK Options flagsPre-Land:
💡 Motivation and Context
Part of [Mobile Replay] Capture Network request/response bodies
Initially, we were trying to keep SDK changes simple and re-use the existing OKHTTP_REQUEST|RESPONSE hint data.
However, the
:sentry-android-replaygradle module doesn't compile against any of the http libs (makes sense).Because these
okhttp3.Request, etc types don't exist in:sentry-android-replay, replay accesses the NetworkDetails data via Hint data ("replay:networkDetails") on the Breadcrumb, when the SDKOptions constraints have been met.💚 How did you test it?
this replay session is an example recording with no
networkDetail*SDK options enabled(replay sessions below were captured without commit 4bc5b7714).
recording segment data shows sample replay payload .
replay session respects networkDetail[Request|Response]Headers
https://sentry-sdks.sentry.io/explore/replays/f89535ea0e79499ca8d75e34e6270925
replay session captures GET request bodies as null
replay session

replay session captures JSON bodies
1. Request body formatted as JSON key/values

ref
2. Response body formatted as JSON key/values

ref
replay session captures plaintext bodies
replay session

replay session captures form bodies (formurlencoded)
replay session

replay session captures binary bodies
1. binary body under size limit (valid response size; summarized body content
https://sentry-sdks.sentry.io/explore/replays/baf899dd12c14718bc5d8b877a9fff3d

2. binary body over size limit (invalid response size, summarized body content

(https://sentry-sdks.sentry.io/explore/replays/94dc96ae2b8c4f4d8c23e2e0c69e5d1d)
replay session truncates plaintext bodies over 150KB (`MAX_NETWORK_BODY_SIZE`)
https://sentry-sdks.sentry.io/explore/replays/e11711a70c164a8fa1a2ed39e8632a05

replay sessions properly capture one-shot okhttp request bodies
One-shot HTTP Request Body request

Test:
read(..)okhttp3.Request.body in an OkHttpInterceptor after cloning body into new request causes the request to succeed ✅ (current impl)replay session
Valid response body logged to sentry:

Sanity Test:
read(..)okhttp3.Request.body in an OkHttpInterceptor without cloning body into a new request causes the request to fail ✅replay session
Invalid response body logged to sentry:
