How to scrub/sanitise transactions

[i-am-david-fernandez]

I have only just begun evaluating Sentry (in a python/django backend context, to be expanded to cover a javascript/angular frontend/client), and so far I am very impressed. One thing stopping me from using it fully, however, and as CISO recommending its use in our products, is the seeming inability to scrub/sanitise performance data (transactions, I believe). The before_send and before_breadcrumb callbacks appear to be more than flexible enough to scrub events and breadcrumbs before they leave the application, but there appears to be no equivalent for scrubbing performance data. As a result, the data available via sentry.io contains things I do not wish it to (e.g., sensitive data within request bodies). The result is that I simply cannot use performance monitoring outside, perhaps, development environments with toy/non-sensitive data.

The documentation (https://develop.sentry.dev/sdk/performance/#interaction-with-beforesend-and-event-processors) states this:

Transactions should not go through beforeSend.

which makes me confident I haven't simply missed something (though I'd be more than happy to be corrected here).

My request, then, is for the SDK to provide an equivalent to before_send that allows complete control over what data gets sent via performance monitoring. More broadly, it should be possible to scrub any and all data that is being sent (which at this stage appears to be only events, breadcrumbs and transactions, but perhaps the list is longer). I can't consider Sentry to be a security-compliant tool without such facilities, and quite frankly, given the obvious awareness of the importance of managing PII, I'm surprised that this problem exists.

I'd love to make full use of Sentry, as I think it's both a brilliant idea and, above issue not-withstanding, well executed, so I hope this request is thoughtfully considered.

Thank-you.

[AbhiPrasad]

Hey thanks for reaching out.

You can use the event processor API to update events (both errors and transactions) before they are sent to Sentry. You can see the schema for an event here: https://develop.sentry.dev/sdk/event-payloads/

from sentry_sdk.scope import add_global_event_processor

@add_global_event_processor
def processor(event, hint):
    # type: (Event, Any) -> Dict[str, Any]
    if event.get("type") == "transaction":
        remove_event_details(event)
        return event

    if is_bad_event(event):
        # Don't send event
        return None

    return event

We are working on fleshing out a better API for transaction in particular - stay tuned for that.

If you're interested, we also have advanced server-side controls for PII stripping that are configurable through the Sentry UI: https://docs.sentry.io/product/data-management-settings/scrubbing/advanced-datascrubbing/

[i-am-david-fernandez]

@AbhiPrasad Thanks for the rapid response and for pointing me towards a possible solution.

My particular use revolves around a graphql API (though the problems will be similar for other forms). If you're not familiar with it, graphql manages all operations (both reads and writes) via POSTs with the query (and parameters) embedded in a JSON request body. Roughly speaking, what I wish to do is retain the "operational" part of the request body (e.g., the query or mutation structure) and strip the "data" part (the variables). This means I can discern operations (all operations hit the same URL, unlike REST, so knowledge of the in-body operation is necessary) while protecting sensitive data (the parameters of the operations). As a result, to be of any real use, I need to retain part of the request bodies, but to ensure data security, I need to remove other parts. Also, as useful as it may be in many cases, sentry-sided stripping is not acceptable for me; I require stripping to be performed in-app, such that I can guarantee sensitive data never leaves our systems.

I've spent a bit of time playing with the event processor (as per your guide), but I'm not sure how to make it work the way I need. In particular, it appears (from debugging the sentry code) that, in part, the event is built up by the actions of the event processors, possibly related to the integrations that are configured. For my needs, specifically, the request field of an event is added by one of the event processors (I believe attached by the Request Interface), and, unfortunately, it is added after my event processor is being called. This means this field is not available when my handler is called.

I really need my handler to be the last that is called (ensuring any and all interface-specific fields are populated), but this doesn't seem possible. This line is particularly problematic:

sentry-python/sentry_sdk/scope.py

Line 392 in cad2f65

for event_processor in chain(global_event_processors, self._event_processors):

It means that handlers registered via add_global_event_processor cannot be guaranteed to be called last under all circumstances (i.e., if self._event_processors is non-empty, its handlers will be called after any global handlers). I will note that what appears to be the entire event is available in the before_send handler but, as previously noted, this is not called for transactions.

I think that, to be reliable, effective, and secure, transaction/event stripping must occur immediately prior to data being sent to sentry (or perhaps more importantly, at a point in the flow where it is guaranteed that no additional data will be attached to the event/transaction).

Unless there's another way, in its current form, sentry-based performance monitoring is simply a non-starter for me (or other users similarly interested in protecting data). I hope that whatever you're planning by way of a transactions API addresses these concerns/requirements and that its arrival is imminent.

[rhcarvalho]

@i-am-david-fernandez

This line is particularly problematic:

sentry-python/sentry_sdk/scope.py

Line 392 in cad2f65

for event_processor in chain(global_event_processors, self._event_processors):

As you noted, that line means global event processors run before scope event processors (as designed/intended).

For the Python SDK, I see that the web framework integrations like WSGI (used by Django) are setting an event processor in the scope:

sentry-python/sentry_sdk/integrations/wsgi.py

Lines 119 to 123 in cad2f65

	scope.add_event_processor(
	_make_wsgi_event_processor(
	environ, self.use_x_forwarded_for
	)
	)

That could be revisited, as it might be possible to make them global, but that doesn't change the solution for your use case.

The outgoing flow is:

  Global Event Processors   ->  Scope Event Processors   ->  Before Send   ->  Transport
       (all events)                (all events)              (errors-only)     (everything)

So you have two possibilities:

1. Scope Event Processor can be added from a middleware, making sure every request's scope gets the event processor as the last processor in the chain.
2. Custom Transport gives you total control on how data goes out of the SDK.

I'd suggest you start with (1), by adding a Django Middleware as the last middleware.

[i-am-david-fernandez]

Thanks for the details @rhcarvalho . Could you please provide a pointer (perhaps a link to the relevant documentation) to how I might add a scope event processor from/via middleware? I'm familiar enough with middleware in Django, but I'm assuming you're referring to Sentry middleware (is that the same as Integrations?). It's a bit unclear from your particular wording above.

Thank-you.

[rhcarvalho]

Sorry for the delay --

I meant Django middleware.

@sentry_sdk.Hub.current.scope.add_event_processor
def sanitizer(event, hint):
    # TODO: sanitize event here
    return event

[i-am-david-fernandez]

@rhcarvalho Certainly no apologies are needed! Conversely, thank-you for the continued support.

I think I've got things working according to you suggestion (until something more robust is introduced in the SDK).

For future reference (or hopefully to help others), my django middleware is this:

class SanitisationMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):

        ## Pass down to any other middlewares
        response = self.get_response(request)

        ## We are now in the middleware unwinding phase.
        ## We add/register the event sanitiser as late as possible to ensure it
        ## has access to all and any event fields added by other processors,
        ## allowing it to properly clean.
        sentry_sdk.Hub.current.scope.add_event_processor(self.event_sanitiser)

        return response

    def event_sanitiser(self, event, hint):
        if event.get("type") == "transaction":
            ## Sanitise/handle here

        return event

This middleware is registered as the first middleware in the django middleware stack (as best as one can do). This is easier to manage, as I can more easily control the initial order in the configuration (other things may append to this list, though this is imperfect as other things could potentially insert earlier middlewares). To ensure the processor is registered last, I add it in the middleware unwinding phase.

This does appear to work, in as much as the fields I'm currently concerned with have been populated by the time my handler is hit, though I'd still very much like something more robust. When it comes to security, I really want auditable guarantees, not just "it happens to work for the moment if things are lined up just so.".

I can see in the documentation, code and commit comments that it is the design intent that before_send is not applied to transactions, but I'd like to understand why. I have created a fork of this repo where I've replicated the logic in _Client._prepare_event(...) to apply a second processor (before_send_transaction) to transactions only. While this appears to work, I can't help but wonder why transactions were excluded from before_send, and consequently I assume that there's something I have missed that means my changes could break something. To put it another way, it seems the "fix" I've put in place is too easy and obvious to be correct!

Again, thank-you for the support and for helping me find an interim solution. Your time is appreciated.

[github-actions]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀