CiviMail - Restore support for preview of "mailing"/"action" tokens via TokenProcessor/Flexmailer

[totten]

Overview

When using TokenProcessor to generate a mailing (e.g. as with Flexmailer/Mosaico), the mailing/action-tokens (e.g. {mailing.subject} and {action.optOutUrl}) are generated via CRM_Mailing_Tokens and CRM_Mailing_ActionTokens. To properly generate them, CRM_Mailing_Tokens and CRM_Mailing_ActionTokens check the mailing/job ID. However, that information is no longer available when performing a "Preview" -- leading to misbehavior in previews. In combination with civicrm/org.civicrm.flexmailer#38, this restores parity for token/preview support.

Before (Pre-5.6; OK)

• When a user begins composing a mailing, CiviMail creates a draft mailing with a concrete ID (e.g. mailing #123).
• To preview the mailing, the UI calls Mailing.preview API with the ID of the mailing.
• Flexmailer/Mosaico generates the preview by calling TokenProcessor and therefore CRM_Mailing_ActionTokens.
• CRM_Mailing_ActionTokens has strictness checks. These pass because the ID is available.

Before (5.6-5.12; Not OK)

As a performance enhancement, CiviCRM 5.6 (PR #12509; dev/mail#20) revised the signature for Mailing.preview API to allow previews without having a specific mailing record/job/ID. Consequently:

• When a user begins composing a mailing, CiviMail creates a draft mailing with a concrete ID (e.g. mailing #123).
• To preview the mailing, the UI calls Mailing.preview API with without the ID of the mailing.
• Flexmailer/Mosaico generates the preview by calling TokenProcessor and therefore CRM_Mailing_ActionTokens.
• CRM_Mailing_ActionTokens has strictness checks. These pass fail because the ID is available unavailable.

After (OK)

• When a user begins composing a mailing, CiviMail creates a draft mailing with a concrete ID (e.g. mailing #123).
• To preview the mailing, the UI calls Mailing.preview API with without the ID of the mailing.
• Flexmailer/Mosaico generates the preview by calling TokenProcessor and therefore CRM_Mailing_ActionTokens.
• CRM_Mailing_ActionTokens has strictness less strict checks. These pass because the context[schema] hints that a mailing ID will be available when it's really needed.

Comments

The change in contract for Mailing.preview API has produced multiple overlapping issues -- including this issue in civicrm-core and some other issues in flexmailer. The flexmailer issues generally involve getting the wrong information about a previewed mailing because the id is unavailable. Unfortunately, because of the overlapping issues, it can be hard to produce a clear/singular symptom. (Ex: The symptoms change significantly depending on the template_type of the first mailing in the DB and/or depending on whether you enable the hidden setting experimentalFlexMailerEngine.)

With regard to QA for this PR, some important mitigating circumstances to note:

• CiviMail's BAO-based delivery engine (in civicrm-core) doesn't use TokenProcessor or CRM_*_Tokens. That change was done as "leap by extension" (flexmailer).
• civicrm-core does use TokenProcessor for scheduled reminders - but they don't involve mailings.
• For some other/overlapping issues in, there's (dev/mail#20) Mailing.preview - Fix regression circa Civi 5.6.x org.civicrm.flexmailer#38. That patch also expands test-coverage, and it will be included in the CiviCRM-Ext-Matrix once it's merged. (However, that PR currently fails because it depends on having this change!)

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[eileenmcnaughton]

@seamuslee001 @colemanw @mattwire @monishdeb @lcdservices do any of you feel placed to review this? some of you were involved in the pr that causes this so might have some background

[totten]

Aside: @seamuslee001 @eileenmcnaughton Given the age of the underlying regression, it's tempting to put this master first - however, (a) I view this as a continuation of the CiviMail preview problems in 5.12.2 and 5.12.4, and (b) it affects a very widely deployed extension.

[seamuslee001]

Code looks fine to me and I am a +1 for this being in the RC

[seamuslee001]

@totten @monishdeb test failure relates https://test.civicrm.org/job/CiviCRM-Core-PR/26011/testReport/junit/(root)/CRM_Mailing_TokensTest/testTokensWithoutMailing/

[seamuslee001]

The code look sensible to me, I think we should merge this into the RC. As we have seen we have tests already in this area and the tests are being altered sensibly tbh.