Feature: Long term storage of Webform submission data

[Adnan-cds]

What does this change?

Adds a submodule to store Webform submission data into a separate database for long term storage (LTS).

Personally identifiable information (PII) is optionally redacted before copying into the LTS database.

How to test

Please see the Setup and Inspection sections of the README.

How can we measure success?

Webform submissions should get copied into the LTS database during cron runs.

Have we considered potential risks?

PII redaction may miss relevant pieces information. Current redaction rules are as follows:

• All Webform elements of type email, telephone, number, address, localgov_forms_dob.
• Any element with the following happening in its machine id: name, mail, phone, date_of_birth, personal, title, gender, sex, ethnicity, post_code.
• All textareas are cleaned of email, postcode, and any number. Email detection uses a simple regex which may miss complex email address formats.

[finnlewis]

Very cool @Adnan-cds .

One question, if we're removing / redacting personal information, what's the motivation for storing the data in a separate database?

[Adnan-cds]

if we're removing / redacting personal information, what's the motivation for storing the data in a separate database?

This is primarily for reporting. The idea is, you connect the separate database with a reporting or data warehousing tool for further reporting and analysis.

[finnlewis]

@stephen-cox @ekes any chance of a quick code review on this one for @Adnan-cds ?

[finnlewis]

@ekes is happy to read review the code this week.

[finnlewis]

Discussing with @ekes and @Adnan-cds

The code looks good in general!

Discussing use cases, we think that it might be usefult to separate the redaction from the storage in a separate database:

a) some people might want redaction before storing int he Drupal database.
b) some people might want to store submissions in a separate LTS database without redacting

Also noting that this module was released today: https://www.drupal.org/project/webform_sanitize_submissions and there is also https://www.drupal.org/project/webform_sanitize

[Adnan-cds]

Changes since last review:

• Added a settings form for enabling or disabling LTS storage. This should be useful in dev/stage sites where there's no need for LTS. By default, LTS storage is disabled. Settings form path: /admin/structure/webform/config/submissions-lts
• PII redaction functionality is now available as a plugin. The existing PII redaction code is present as a selectable plugin in the above mentioned settings form. By default, no plugin is selected meaning no PII redaction happens while Webform submissions are copied to LTS. I also looked at the two suggested modules (webform_sanitize_submissions and webform_sanitize) and found that their code are not really reusable. So added a plugin manager instead.
• Removed the Drush deploy hook in favour of a Drush command drush localgov_forms_lts:copy. This gives some more control over the deployment process.
• Updated the README accordingly.

Ready for review once again. @finnlewis @ekes

[ekes]

🤯

Well it works, and it was copying stuff over to my database. Code review we've already discussed. So we're good there. I was wondering if it was possible to use the sample plugin PII redaction per-form; but I'm guessing not yet.

[Adnan-cds]

Thanks for the review and approval. Must have taken a good few hours :(

I was wondering if it was possible to use the sample plugin PII redaction per-form; but I'm guessing not yet.

That's right. There is no per-form config settings. At some point, I would like to write another plugin that can be activated for each field in a Webform. But at this point, we just want to find out if this submodule runs into any unforeseen issues at scale.