Sensitive Data Redaction

text/0255-sensitive-data-redaction.md

@@ -0,0 +1,276 @@
+# Sensitive Data Redaction
+
+This is a proposal for adding treatment of sensitive data to the OpenTelemetry (OTel) Specification and semantic conventions.
+
+## Motivation
+
+When collecting data from an application, there is always the possibility, that the data contains information that shouldn't be collected, because it is either leaking (parts of) credentials (passwords, tokens, usernames, credit card information) or can be used to uniquely identify a person (name, IP, email, credit card information) which may be protected through certain regulations. By adding OTel to a library or instrumentation an end-user of OTel is facing exactly this challenge: values of an [Attribute](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/common/README.md#attribute) may carry such sensitive data.
+
+While it’s ultimately the responsibility of the legal entity operating an application to protect sensitive data, end-users of OpenTelemetry (developers, operators working for that entity) are turning to the authors of OpenTelemetry – or to those of libraries that implement OpenTelemetry, like the Azure SDK – to have means in place to redact/filter sensitive data. Without that capability provided, they will raise security issues and/or will drop OpenTelemetry eventually due to it not meeting their security/legal requirements.
+
+In this OTEP you will find a proposal for adding treatment of sensitive data to OpenTelemetry.
+
+By adding the proposed features, OpenTelemetry will provide its end-users the tooling needed to make sure that sensitive data is treated according to their requirements.
+
+This will make sure that these end-users can use OpenTelemetry within their secure and legal requirements and that OpenTelemetry (and implementing libraries) are able to avoid vulnerabilities.
+
+## Explanation
+
+### Overview
+
+This proposal aims to provide the following features:
+
+- methods for consumers of the OpenTelemetry API to enrich collected Attributes with sensitivity information and hooks to apply different levels of redaction.
+- related to those methods a similar way to enrich attributes in the semantic conventions with sensitivity information and ways of redaction.
+- a consistent way to configure sensitivity requirements for end-users of the OpenTelemetry SDK and in instrumentation libraries, including predefined “configuration profiles” and ways for fine grain configuration.
+- A redactor implements the logic to apply redaction and that owns predefined helpers for redaction in the SDK (URLParams filtering, Zeroing IPs, etc.).
+
+The following limitations apply:
+
+- Only Attributes are treated, although it is possible that sensitive data is contained in other data generated by OpenTelemetry as well (e.g. the span name, or the instrumentation scope name could contain sensitive data)
+
+### Annotate attributes with sensitivity information
+
+As a first building block the OpenTelemetry API needs to provide capability to enrich collected attributes with sensitivity details and with potential hooks to redact those. Those API changes then also can be used to describe sensitivity information in the semantic conventions.
+
+#### API
+
+Every API that sets an attribute consisting of a key and a value, needs to be enhanced by an additional functionality that allows to add details about the potential sensitivity of this data and a hooks how it may be redacted. This can be an additional set of parameters for an existing method or a method that can be called after the attribute has been set, if adding parameters to a method signature would lead to a breaking change.
+
+An additional method for setting the sensitivity information for a span attribute might look like the following:
+
+```
+span.setAttribute("url.query", url.toString(), <SENSITIVITY_DETAILS>);
+```
+
+Or, if the setAttribute method can not be extended:
+
+```
+span.setAttribute("url.query", url.toString());
+span.redactAttribute(“url.query”, <SENSITIVITY_DETAILS>);
+```
+
+The content of `<SENSITIVITY_DETAILS>` may look like the following example:
+
+```
+{
+  <REDACTION_LEVEL>: <REDACTION_RULE>
+}
+```
+
+The key `<REDACTION_LEVEL>` is one of multiple available pre-defined levels of redaction requirements that an end-user may choose, e.g.
+
+- `DEFAULT`: What should be the default redaction applied to this value
+- `STRICT`: What should be a basic redaction that is applied to this value
+- `STRICTER`: What should be an advanced redaction that is applied to this value
+- …
+
+The value `<REDACTION_RULE>` can be one of the following:
+
+- A regular expression (or sed expression?) which when applied turns parts of a string into their redacted version,e.g. `s/([0-9]+\.[0-9]+\.[0-9]+\.)[0-9]+/\10/` applied on an IP address will replace the last octet with 0: `1.2.3.4` becomes `1.2.3.0`
+- A constant that represents pre-defined redaction methods (see below for Redaction helpers), e.g. `REDACT_INSECURE_URL_PARAM_VALUES` will apply what is required for [#971](https://github.com/open-telemetry/semantic-conventions/pull/971)
+- A callback that will call a function on that value and apply advanced redaction
+
+It should be recommended to either use the expression or the constant and only in rare circumstances a callback function should be applied.
+
+Note that those API calls are no-op and will be implemented by the SDK (as we do it with other API methods as well), this way (almost?) no additional overhead will be created by introducing those annotations.
+
+Below you will find details on how the `<SENSITIVITY_DETAILS>` are included in the definitions of attributes in the Semantic Conventions. By pre-loading the details for known attributes in the SDK configuration a call to `span.setAttribute("url.query", url.toString());` can apply the redaction internally. An end user may append/overwrite those details.
+
+### Semantic Conventions
+
+With the definitions above values in the semantic conventions can be annotated with `<SENSITIVITY_DETAILS>` as outlined above, with the exception that no callback functions can be supplied as redaction rules.
+
+Example:
+
+| Attribute | ... existing columns ... | sensitivity details |
+|-----------|--------------------------|---------------------|
+| `url.query`|                         | Rationale: Some verbatim wording why this is the way it is below<br>Type: `mixed`<br>`DEFAULT`: `REDACT_INSECURE_URL_PARAM_VALUES`<br>STRICT: `REDACT_ALL_URL_VALUES`<br>`STRICTER`: `DROP` |
+| `client.address`|                    | Rationale: some reasons why dropping octets may be required<br>Type: `maybe_pii`<br>`DEFAULT`: `NONE`<br>`STRICT`: `'s/([0-9]+\.[0-9]+\.[0-9]+\.)[0-9]+/\10/'`<br>`STRICTER`: `'s/([0-9]+\.[0-9]+\.)[0-9]+\.[0-9]+/\10.0/'` |
+| `enduser.creditCardNumber`**[1]** |     | Rationale: ...<br>Type: `always_pii`<br>DEFAULT: `EXTRACT_IIN`<br>`STRICT`: `DROP`|
+
+**[1]**: _This is a made-up example for demonstration purpose, it’s not part of the current semantic conventions. It gives a more nuanced example, e.g. that extracting the IIN might be an option over dropping the number completely. It also demonstrated the value of “type”, which can enable Data lineage use cases_
+
+The `DROP` keyword means that the value is replaced with `REDACTED` (or set to null, …).
+
+It is the responsibility of the OpenTelemetry SDK to implement those sensitivity details provided by the semantic conventions. This means that an instrumentation library does not need to add `<sensitivityDetails>` when calling an `setAttribute` method or does not need to call the additional `redactAttribute` method as outlined above. An instrumentation library may choose to apply additional redactions (leveraging the OpenTelemetry APIs or doing it before calling `setAttribute` in their own business logic).
+
+### SDK Configuration of sensitivity requirements
+
+The annotations and APIs as outlined above will allow SDK users to provide their sensitive requirements as configuration (here: environment variable, but can be encoded in future config files as well), e.g.
+
+```
+env OTEL_SENSITIVE_DATA_PROFILE=”STRICT” ./myInstrumentedApp
+```
+
+This call will make sure that redactions applied follow the `STRICT` profile. If not set the `DEFAULT` will be used. Additionally there are 2 levels that can not be used in sensitivity details:
+
+- `NEVER`: No redaction is applied, SDKs may choose to log a warning that this is a risky choice
+- `ALWAYS`: All values with sensitivity details will be dropped
+
+Additionally SDK end users can provide advanced configuration (through code, configuration file, probably not environment variable) to add specific needs:
+
+```
+{
+  <ATTRIBUTE_KEY> => <SENSITIVITY_DETAILS>
+}
+```
+
+e.g.
+
+```
+{
+  "url.query" => { DEFAULT: REDACT_ALL_URL_VALUES },
+  "com.example.customer.name" => { DEFAULT: x => { /* do something in this callback * } },
+  "com.example.customer.email" => { ‘s/^[^@]*@/REDACTED@/’ },
+}
+```
+
+### Redactor
+
+To accomplish redaction the SDK needs a component (similar to a sampler) that inspects attributes when they are set and applies the required redactions:
+
+- `Redactor::setup(profile)` will setup the redactor with the given profile, maybe a constructor? depends on the language
+- `Redactor::redact(value, <sensitivityDetails>)` will return the redacted value using the provided profile
+
+The redactor will also have all the methods to apply predefined redactions (`REDACT_ALL_URL_VALUES`, `REDACT_INSECURE_URL_PARAM_VALUES`, `DROP`, etc.). If a method is not implemented (either by the SDK or by the end-user choosing one that does not exist), it will default to apply `DROP` to avoid leakage of any sensitive data.
+
+## Internal details
+
+**tbd**
+
+## Additional context
+
+Treating sensitive data properly is a very complex multi-dimensional topic. Below you will find some additional context on that subject.
+
+### Types of sensitive data
+
+There are different kinds of “sensitivity” that may apply to data. The ones most relevant in this proposal are “security” and “privacy”. They may overlap but we distinguish them as follows:
+
+- security-relevant sensitive data: any information that when exposed [weakens the overall security of the device/system](https://en.wikipedia.org/wiki/Vulnerability_(computing)).
+- privacy-relevant sensitive data: any information that when exposed [can adversely affect the privacy or welfare of an individual](https://en.wikipedia.org/wiki/Information_sensitivity).
+
+Note, that there are other kinds of sensitive data (business information, classified information), which are not covered extensively by this proposal.
+
+### Level of Sensitivity
+
+The level of sensitivity of an information can also be different and that sensitivity can be contextual, e.g.
+
+- The password of a user without privileges is less sensitive than the password of an administrator
+- The "client IP" in a server-to-server communication is less sensitive than the "client IP" in an client-to-server communication, where the client can be linked to a human.
+- API tokens of a demo system are less sensitive than API tokens for a production system
+- The license plate of an individual’s car is less sensitive than their social security number
+- The full name of a user in a social network is less sensitive than the full name of a user in a medical research database
+
+Depending on the sensitivity data an end-user of an observability system may weigh up if collecting this data is worth it.
+
+### Regulatory and other requirements
+
+Due to the negative effects that the exposure of sensitive data can have (see above in "Types of sensitive data"), different entities have created regulations for the collection of sensitive data, among them:
+
+- GDPR
+- CPRA
+- PIPEDA
+- HIPAA
+- [more…](https://en.wikipedia.org/wiki/Information_privacy)
+
+Additionally the entities operating the applications who leverage OpenTelemetry may have their own requirements for treating certain sensitive data.
+
+Finally end-users may want to apply recommendations for [Data Minimization](https://en.wikipedia.org/wiki/Data_minimization), to avoid "unnecessary risks for the data subject".
+
+**Note 1**: it is not (and can not be) the responsibility of the OpenTelemetry project to provide compliance with any of those regulations, this is a responsibility of the OTel end-user. OTel can only facilitate parts of those requirements.
+
+**Note 2**: Those requirements are subject of change and outside of the control of the OpenTelemetry community.
+
+## Trade-offs and mitigations
+
+### Performance Impact
+
+By adding an extra layer of processing every time an attribute value gets set, might have an impact on the performance. There might be ways to reduce that overhead, e.g. by only redacting values which are finalized and ready to exported such that updated values or sampled data does not need to be handled.
+
+## Prior art and alternatives
+
+### OTEPS
+
+- [OTEP 100 - Sensitive Data Handling](https://github.com/open-telemetry/oteps/pull/100)
+- [OTEP 187 - Data Classification for resources and attributes](https://github.com/open-telemetry/oteps/pull/187)
+
+### Spec & SemConv Issues
+
+This problem has been discussed multiple times, here is a list of existing issues in the OpenTelemetry repositories:
+
+- [Http client and server span default collection behavior for url.full and url.query attributes](https://github.com/open-telemetry/semantic-conventions/issues/860)
+- [URL query string values should be redacted by default](https://github.com/open-telemetry/semantic-conventions/pull/961)
+- [Specific URL query string values should be redacted](https://github.com/open-telemetry/semantic-conventions/pull/971)
+- [Allow url.path sanitization](https://github.com/open-telemetry/semantic-conventions/pull/676)
+- [Guidance requested: static SQL queries may contain sensitive values](https://github.com/open-telemetry/semantic-conventions/issues/436)
+- [Semantic conventions vs GDPR](https://github.com/open-telemetry/semantic-conventions/issues/128)
+- [Guidelines for redacting sensitive information](https://github.com/open-telemetry/semantic-conventions/issues/877)
+- [DB sanitization uniform format](https://github.com/open-telemetry/semantic-conventions/issues/717)
+- [Add db.statement sanitization/masking examples](https://github.com/open-telemetry/semantic-conventions/issues/708)
+- [TC Feedback Request: View attribute filter definition in Go](https://github.com/open-telemetry/opentelemetry-specification/issues/3664)
+
+### SemConv Pages
+
+The semantic conventions already contains notes around treating sensitive data (search for "sensitive" on the linked pages if not stated otherwise):
+
+- [gRPC SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/rpc/grpc.md)
+- [Sensitive Information in URL SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/url/url.md#sensitive-information)
+- [GraphQL Spans SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/graphql/graphql-spans.md)
+- [Container Resource SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/resource/container.md)
+- [Database Spans SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/database/database-spans.md)
+- [HTTP Spans SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/http/http-spans.md)
+- [General Attributes SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/attributes.md)
+- [LLM Spans SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/llm-spans.md)
+- [ElasticSearch SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/database/elasticsearch.md)
+- [Redis SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/database/redis.md)
+- [Connect RPC SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/rpc/connect-rpc.md)
+- [Device SemConv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/resource/device.md) (search for GDPR)
+
+### Existing Solutions within OpenTelemetry
+
+The following solutions for OpenTelemetry already exist:
+
+- [MrAlias/redact](https://github.com/MrAlias/redact) for OpenTelemetry Go
+- Collector processors, including
+  - [Redaction Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/redactionprocessor)
+  - [Transform Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor)
+  - [Filter Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor)
+
+### Alternative 0: Do nothing
+
+It’s always good to analyze this option. If we do nothing end users will still need to satisfy their requirements for treating sensitive data accordingly:
+
+- Instrumentation library authors are required to manage redaction before using the OpenTelemetry API
+- Application developers will do the same or are forced to join the instrumented application with a collector for redaction/filtering
+- Third party solutions will be implemented.
+
+### Alternative 1: OpenTelemetry Collector
+
+As listed above there are multiple processors available that can be used to redact or filter sensitive data with the OpenTelemetry collector. The challenge with that is that it is unknown to an application (owner) if data is processed in the collector as expected. Also, the data leaving the application might already be a risk (non-encrypted or compromised network) or may not be allowed (collector is hosted in a different country, which may conflict with a regulation)
+
+Ideally a combination is used.
+
+### Alternative 2: Backend
+
+The backend consuming the OpenTelemetry data can provide processing for filtering and redaction as well. The same objection as for the collector apply.
+
+### Existing Solutions outside OpenTelemetry
+
+There are many solutions outside OpenTelemetry that help to filter or redact sensitive data based on security and privacy requirements:
+
+- [sanitize_field_name in Elastic Java](https://www.elastic.co/guide/en/apm/agent/java/1.x/config-core.html#config-sanitize-field-names)
+- [Filter sensitive data in AppDynamics Java Agent](https://docs.appdynamics.com/appd/24.x/24.4/en/application-monitoring/install-app-server-agents/java-agent/administer-the-java-agent/filter-sensitive-data)
+- [GA4 data redaction](https://support.google.com/analytics/answer/13544947?sjid=3336918779004544977-EU)
+- [Configure Privacy Settings in Matamo](https://matomo.org/faq/general/configure-privacy-settings-in-matomo/)
+- [DataDog Sensitive Data Redaction](https://docs.datadoghq.com/observability_pipelines/sensitive_data_redaction/)
+
+## Open questions
+
+- **Question 1**: Should sensitivity details for an attribute in the semantic conventions be excluded from the stability guarantees? This means, updating them for a **stable** attribute is not a breaking change. The idea behind excluding them from the stability guarantees is that the requirements are subject of change due to changes in technology (see [#971](https://github.com/open-telemetry/semantic-conventions/pull/971), the list of query string values will evolve over time) or changes in regulatory requirements or both.
+
+## Future possibilities
+
+Attributes are most likely to carry sensitive information, but as stated in the overview section of the explanation other user-set properties may carry sensitive information as well. In a later iteration we might want to review them as well.
+
+The proposal puts the configuration of sensitivity requirements into the hands of the person operating an application. In a future iteration we can look into providing end-users of instrumented applications to provide their _consent_ of which and how data related to them is tracked, see [Do Not Track](https://en.wikipedia.org/wiki/Do_Not_Track), [Global Privacy Control](https://privacycg.github.io/gpc-spec/) and the requirements of certain local regulations.