dev/core#2258 - Read+write SMTP password using 'crypto.token'

[totten]

Overview

This converts the smtpPassword subfield from CRM_Utils_Crypt to the crypto.token service, which means:

• It supports more robust key-management.
• When storing smtpPassword in SQL, the on-disk format is less ambiguous.
• When storing smtpPassword in civicrm.settings.php, the on-disk format is less ambiguous and more forgiving.

See also: https://lab.civicrm.org/dev/core/-/issues/2258

(NOTE: This depends on #19236 an #19251.)

(NOTE: A few edits added to the description after-the-fact. It originally spoke of the old plaintext as, well, plaintext. It was actually Base64-encoded plaintext.)

Before

The format of the smtpPassword subfield depends on the available PHP/PECL extensions:

• The field is Base64 plaintext... if and only if PHP has mcrypt disabled
• The field is encrypted... if and only if PHP has mcrypt enabled

Additionally, note that:

• There is no way to discern (based on the data) if a value like YWJjZGVm is random password or ciphertext. It remains legible only as long as both the PHP configuration and SITE_KEY stay the same.
• There is no sensible way to rotate the encryption key.

After

The format of the smtpPassword subfield is specified by crypto.token which means:

• The field is normal plaintext... if and only if the content begins with a printable character (not chr(2)).
• The field is encrypted... if and only if the content begins with chr(2) (in which case the string looks like ^CTK?k=<keyid>&t=<ciphertext>)

Additionally, note that:

• One can unambiguously distinguish between between plaintext and ciphertext (and, among ciphertexts, you can identify the key needed for decoding).
• The System.rotateKey API can be used to phase-out old keys (per (dev/core#2258) Add API+hook to rotate keys for encrypted fields #19251).

Technical Details: Review of upgrade scenarios

There are three variables which determine different upgrade scenarios:

• Storage: Is the password stored in the DB (civicrm_settings) or in PHP (civicrm.settings.php)?
• Old Crypto: Does the environment have mcrypt (ie the old crypto system) enabled?
• New Crypto: Has the administrator pre-configured CIVICRM_CRED_KEYS (ie the new crypto system)?

Here are how the scenarios work out:

#	Storage?	Old crypto?	New crypto?	Comments
1	DB (civicrm_settings)	No	No	smtpPassword starts as Base64 plain-text and ends as normal plain-text.
2	DB (civicrm_settings)	No	Yes	smtpPassword converts from plain-text to new-encryption.
3	DB (civicrm_settings)	Yes	No	smtpPassword converts from old-encryption to plain-text.
4	DB (civicrm_settings)	Yes	Yes	smtpPassword converts from old-encryption to new-encryption.
5	PHP (civicrm.settings.php)	No	*	smtpPassword was stored in PHP as plain-text, and the same plain-text will work post-upgrade. ⚠️ smtpPassword was stored in PHP with the old Base64 plain-text, and the upgrader cannot change it. After upgrading, you must manually edit civicrm.settings.php to set smtpPassword as normal plain-text.
6	PHP (civicrm.settings.php)	Yes	*	⚠️ smtpPassword was stored in PHP with the old encrypted format, and the upgrader cannot change it. After upgrading, you must manually edit civicrm.settings.php to set smtpPassword using plain-text or the new encryption format.

Scenarios 1-4 all end with a working/readable variant of smtpPassword. After upgrade, one has the option to change the crypto by updating CIVICRM_CRED_KEYS and calling System.rotateKeys.

Scenarios 5-6 are annoying, but there's not much we can do besides show a warning. However, I don't expect this to be common because it has always been tricky to understand/setup. If you have managed to figure it out before, then you're probably clever enough to sort out the migration - as long as you see the warning provided by the upgrader. (There are more comments in ThirtyFour.php alongside the relevant warning.)

Technical Details: Sysadmin usability

This patch incidentally makes it easier to set the SMTP password via civicrm.settings.php / $civicrm_settings.

$civicrm_setting['domain']['mailing_backend'] =  array(
        'outBound_option' => '0',
        'sendmail_path' => '',
        'sendmail_args' => '',
        'smtpServer' => 'localhost',
        'smtpPort' => '1025',
        'smtpAuth' => '1',
        'smtpUsername' => '',
        'smtpPassword' => '********',
      );

Compare:

• With CRM_Utils_Crypt, the format of ******** was determined by the PECL configuration, which is usually out-of-sight/out-of-mind. Consequently, I would wager that the behavior seems unreliable/confusing to administrators -- on one deployment/server, ******** is interpreted as plaintext, and it works intuitively. On another, the ******** is interpreted as ciphertext, and it's very difficult to set correctly.
• With crypto.token, the PECL configuration is not decisive. The sysadmin can put the smtpPassword as plain-text ('smtpPassword' => 'mySecret'), and that will work on any server. If they prefer to enter the override as an encrypted value, that works too ('smtpPassword' => chr(2).'CTK?k=...&t=...' -- although I think the audience for this would be quite limited).

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[totten]

jenkins, test this please

[totten]

This was "WIP" because it needed a mechanism to re-enable support for encryption. That is now available in #19251, so I've rebased and updated both title+description.

Of course, the PR still depends on others. (Recommended sequence of reading/merging: #19236 => #19251 => #19239.)

[seamuslee001]

@totten can you rebase this please

[seamuslee001]

Is there a reason why you don't want to call crypto service and encrypt using the new system here?

[totten]

For posterity, we discussed further at https://chat.civicrm.org/civicrm/pl/r8pbskapc7gybngdceqxg9sksh

Basically, we're evaluating trade-off between:

• SITE_KEY as fallback crypto key
  • How: Register SITE_KEY as low-priority crypto-key for use with CRED. The upgrader uses this for initial encryption of smtpPassword
  • Pro: The smtpPassword is promptly (re)encrypted. No interregnum.
  • Con: Creates more edge-cases / complexity
• Temporary decryption
  • How: The upgrader decrypts smtpPassword. After upgrade, sysadmin defines CRED_KEYS and runs rotateKey
  • Pro: Long-term, system is simpler to understand / has fewer edge-cases (for admin/dev/etc)
  • Con: There's an interregnum (between the upgrade and encryption) during which smtpPassword is stored as plaintext.

[seamuslee001]

Based on discussion with Tim I'm ok with this now, MOP

[totten]

Merging per MOP