Guidance on anonymization/pseudonymization

[loekvangool]

I'd like to propose that ECS adds guidance for anonymization and pseudonymization. Some thoughts:

Definitions

• anonymization: Irreversible data obfuscation.
• pseudonymization: Reversible data obfuscation.

PII model
The NIST 800-122 publication on PII identifies levels of personal identifiable information:

• High (4): publication has severe/catastrophic effects
• Medium (3): publication has serious adverse effects
• Low (2): publication has limited adverse effects
• Public (1): not part of PII, but describes non-personal data

Typically if one is allowed to see PII level X, one can also see PII levels < X (the Air Force One uses the same method: walk freely towards the rear, but never walk forward of your own seat). We could also imagine putting pii_<level> as a pre- or postfix in field names to easily manage Field Level Security (because it supports access based on wildcards (*)).

Varying levels of obfuscation
We should also recognize that various versions of the same field can (and should) exist in harmony. Perhaps the Dutch postal code system is a good example:

• postalcode: 1234AB

The system is set up so that each character to the right is adding more precision to the location.

Perhaps in Elasticsearch this becomes:

• customer.postalcode.raw: 1234AB
• customer.postalcode.city: 12
• customer.postalcode.obfuscated: E32DB25A9BAAA6AF655FE65A861C9BD35AF1868229E0E9D738236B4500626AFB

Or, implementing PII:

• customer.postalcode.pii4: 1234AB <-- perhaps enough to identify the customer
• customer.postalcode.pii2: 12 <-- not enough to identify the customer
• customer.postalcode.pii1: E32DB25A9BAAA6AF655FE65A861C9BD35AF1868229E0E9D738236B4500626AFB <-- not enough to identify the customer, but based on PII 4 data hence we can bucket customers of the same street without knowing which street it is.

The above would allow various users to access the postal code at an appropriate level for their usage (in case Business Analytics, for example, uses non-PII 3 or 4 data only due to laws on personal data like GDPR).

[ruflin]

Thanks for filing this issue with all the details. I assume by default the "original" field would always contain the original object?

If we would have a field a and a field b and want to use field security, the fields would have to be called pii1_a and pii1_b so we could give someone access to all pii1_* fields? Would be nice if we could have a_pii1 instead so assuming has access to pii1 and pii2 he could start typing a and would see an auto complete for all the fields available to him.

[loekvangool]

I assume by default the "original" field would always contain the original object?

We have to allow for that, while also allowing the user to remove the original data. Depends on the business needs and applicable compliance rules.

If we would have a field a and a field b and want to use field security, the fields would have to be called pii1_a and pii1_b so we could give someone access to all pii1_* fields? Would be nice if we could have a_pii1 instead so assuming has access to pii1 and pii2 he could start typing a and would see an auto complete for all the fields available to him.

It can be a prefix or a postfix, does not make a difference w.r.t. how much effort it takes to create the Field Level Security rules. I also like postfix more because of the reasons you mentioned.

[ruflin]

Is there an easy way to do postfix field level security? Only found prefix so far: https://www.elastic.co/guide/en/elastic-stack-overview/current/field-level-security.html

[loekvangool]

I tested it real quick, seems to work just fine, luckily :).

PUT flstest

PUT flstest/doc/1
{
  "name.pii1": "secret",
  "name.pii2": "fd9adsfSADF"
}

GET flstest/doc/1

POST /_xpack/security/role/flsrole
{
  "indices": [
    {
      "names": [
        "flstest"
      ],
      "privileges": [
        "read"
      ],
      "field_security": {
        "grant": [
          "*.pii2"
        ]
      }
    }
  ]
}

POST /_xpack/security/user/flsuser
{
  "password": "s3cr3t",
  "roles": [
    "flsrole",
    "kibana_user"
  ]
}

POST /_xpack/security/role_mapping/flsmapping
{
  "roles": [
    "flsrole"
  ],
  "enabled": true,
  "rules": {
    "field": {
      "username": "flsuser"
    }
  },
  "metadata": {
    "version": 1
  }
}

As flsuser:

GET flstest/doc/1

Response:

{
  "_index": "flstest",
  "_type": "doc",
  "_id": "1",
  "_version": 3,
  "found": true,
  "_source": {
    "name.pii2": "fd9adsfSADF"
  }
}

DELETE /_xpack/security/role_mapping/flsmapping
DELETE /_xpack/security/role/flsrole
DELETE /_xpack/security/user/flsuser
DELETE flstest

[ruflin]

Thanks for testing. Good to hear prefixes also work. I assume it doesn't make a difference if it is a.pii2 or a_pii2?

[loekvangool]

Getting the same results on _pii2, yes.

[ruflin]

Nice. So now we should tackle the bigger question: How do we fit this into ECS? So far ECS purely focuses on the field definitions. I definitively see like how this makes sense "on top of ECS" and should be part of it. But where do we put it and how do we describe it?

@MikePaquette Ideas?

[MikePaquette]

@ruflin @loekvangool Sorry this has lagged so long without comment.

I wonder if this could be an "ECS Extension for Personal Data" that we can define, rather the defining the actual fields? I definitely like the postfix approach better than prefix for the reasons you've agreed upon.

Fields with the *.pii1 postfix could fit under our definition of ECS Extended fields, although we would not define them all. Having multiple levels is a great idea, as it can satisfy NIST and also GDPR, depending on the jurisdiction of the instantiation.

If this is acceptable, I could fold this into a big 'ol PR on the README.md

[loekvangool]

@MikePaquette sounds good. Reading my own post again, I think there is one additional item:

We should consider to add guidance on naming conventions for anonymized fields. Anonymized data cannot be reversed, and should not have a constant salt, hence referential integrity is lost, and probably index should be disabled on them.

[MikePaquette]

@loekvangool I think we agree that ECS should not attempt to specify what constitutes anonymization, but rather it should define a convention for storing pseudonymized and/or anonymized data in fields with a defined set of postfixes and a defined order of increasing anonymity.

So are you saying that the fourth level (assuming we define 4 of them) would be defined as anonymized? And thus the index mapping template should not bother to index fields with this fourth postfix?

[loekvangool]

@MikePaquette I agree we should not take a stand in that discussion, such a complex area with opinions on definitions etc.

We also can't say that PII4 is anonymized. It might as well be non-personal data altogether.

No I guess I'm proposing to add a line like "if you anonymize data, and there is no consistent hashing used, append .anonymized to the field name before .piiX We could cover index: false with a dynamic mapping template.

It doesn't take long to see that this line of thinking could lead to multiple flags being added to the field names leading to the usage of wildcards on both sides in the dynamic mapping (such as *.pii1*) which is not great and I have no clever way to fix this right now. The only thing I can think of to prevent this is to add the attributes to the field definition of the mapping:

"source.ip.pii2.anonymized": {
  "type": "ip",
  "index": false
}

vs:

"source.ip": {
  "type": "ip",
  "pii": "2",
  "anonymized": true
}

[MikePaquette]

Thanks @loekvangool I see. I am wondering if this (special treatment for anonymized fields) might be an optimization that is too complex, or beyond the scope of ECS. I will attempt to document this concept for the README.md, and would welcome your feedback.

[loekvangool]

Sounds good