[Task]: verify Cinder override decisions work as expected

[eviljeff]

Description

Cinder has an override function to make another decision on a job where a decision was already made (which closed the job). AMO has supported this from the beginning but it wasn't called out as supported feature so may not have been fully QA'd.

Acceptance Criteria

Milestones/checkpoints

Beta Give feedback

1. A job for an add-on that was resolved with (policy leading to) ADDON_DISABLE can be overridden with a different decision to APPROVE; the add-on owner gets a correct email informing them their add-on is now re-enabled because we reconsidered our decision.
2. an APPROVE can be overridden with a ADDON_DISABLE and the developer gets disable email (as if the override was the first decision, because they wouldn't have been aware of the the APPROVE)
3. a policy can be overridden with a different policy that still leads to ADDON_DISABLE - the developer gets the disable email with the new policies
4. (lower priority) repeat with non-addon content types

Checks

• If I have identified that the work is specific to a repository, I have removed "repository:addons-server" or "repository:addons-frontend"

┆Issue is synchronized with this Jira Task

[ioanarusiczki]

So far I tried the following:

• a theme disabled , went to the job to override with approve. Theme was not enabled, a second email at approval was not sent.
  For traceability
  https://reviewers.addons-dev.allizom.org/en-US/reviewers/review/633963
  https://stage.cinder.nonprod.webservices.mozgcp.net/investigate/amo_addon/633963/decisions?filters=%7B%7D

• an extension , resolved with approve and then override to disable. Wasn't disabled, I did not see an email sent.
  https://reviewers.addons-dev.allizom.org/en-US/reviewers/review/633961
  https://stage.cinder.nonprod.webservices.mozgcp.net/investigate/amo_addon/633961/decisions?filters=%7B%7D

• override a policy with another (both take down content). No email sent the second time.
  https://reviewers.addons-dev.allizom.org/en-US/reviewers/review/633962
  https://stage.cinder.nonprod.webservices.mozgcp.net/investigate/amo_addon/633962/decisions?filters=%7B%7D

[eviljeff]

From looking at the webhook response I've identified the problem - Cinder is not passing the job id of the overridden job, only the decision id.

[ioanarusiczki]

I've a strange thing going on with emails. They stopped being sent to real inboxes so I've decided to check this on fakemail.

It took a while to understand what is going on in there, it seems that an email keeps being generated for the same action, for example at submission.

For example for an extension https://stage.cinder.nonprod.webservices.mozgcp.net/investigate/amo_addon/634059/decisions?filters=%7B%7D initially approved from Cinder

same email at addon submission is generated twice (as far as I've checked)
https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196169/change/
https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196173/change/

the auto-approval email is again generated at least twice
https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196174/change/
https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196179/change/

first email at approve from Cinder sent to reporter
https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196178/change/
https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196180/change/
https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196182/change/
https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196183/change/

and the emails sent after override to disable content
reporter https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196184/change/
developer https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196185/change/
reporter https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196187/change/
developer https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196186/change/
reporter https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196189/change/
developer https://addons-internal.dev.mozaws.net/en-US/admin/models/amo/fakeemail/196188/change/

When checking #15104 I had to do an auto-approval, block, delete block, force enable and unreject -> on stage I've received in the real inbox all emails. On dev, I did not see today emails sent for auto-approval, block or force enable (they're on dev, don't remember seeing them more than once but I wouldn't be 100% sure)

[ioanarusiczki]

forgot -> at the moment it's a bit confusing what's going on so I can't tell if the correct emails are sent or not. What's different than before at override it's that the actions have an effect, for example if something is disabled then overridden with approve, content is re-enabled.

https://stage.cinder.nonprod.webservices.mozgcp.net/investigate/amo_addon/634054/decisions?filters=%7B%7D
https://stage.cinder.nonprod.webservices.mozgcp.net/investigate/amo_addon/634053/decisions?filters=%7B%7D

[eviljeff]

@ioanarusiczki it seems the duplicate emails are caused by (or a symptom of) #15160 - though I understand it may be hard/impractical to complete QA on this change with all the extra email noise in the meantime!

[ioanarusiczki]

I tried this on -dev and -stage today

Scenario 1 : content is disabled, override with approve , I tried with an extension, theme, rating, user

• Emails: when content is disabled 2 emails are sent, one to the reporter and the other to the author. After override, only one email is sent to the author (that the initial decision was incorrect).

• Author's appeal: the url from the email throws a 404. (when I tried to send it after override)

Scenario2: an approval overridden with disable, tested with a theme, an extension, a collection

• Emails: at approval an email is sent to the reporter (with an appeal URL). At disable a different email is sent (that content has been disabled) to the reporter and the email to the author (with an appeal URL).
• Reporter's appeal: the url throws a 404.

Scenario3: a disable decision overridden with another policy that disables, tested with an extension and a theme.

• Emails: initially the reporter and the author get an email. When overridden a new email is sent only to the author.
• Author's first appeal url - throws a 404.

That should be all expected. I didn't make screenshots with the emails because 1. would make this tltr 2. I think it's the correct emails sent.

I did not try overriding appeals (I'll check if that's possible tomorrow). Only now while writing test results down, I've realized maybe that's possible too.

[ioanarusiczki]

There's no such thing for appeals 🙃😄

I forgot to add that I've seen an error message together with success! in Cinder while testing this -> but as I said, things are happening as it's expected.