📖Update release process docs & create new cherry pick template (#15911)

* add notes for new release process

* minor fixes

* add clarifications to respond to review

* change 1% day to Wednesday

.github/ISSUE_TEMPLATE/cherry_pick_template.md

@@ -0,0 +1,26 @@
+**Replace *everything* in angle brackets in the title AND body of this issue.  If you have any questions see the [cherry pick documentation](https://github.com/ampproject/amphtml/blob/master/contributing/release-schedule.md#cherry-picks).**
+
+*GitHub issue your cherry pick is fixing:*
+
+Issue #<ISSUE_NUMBER>
+
+*PR that you are requesting a cherry pick for:*
+*(Put N/A if you do not yet have a PR with a fix if you are seeking pre-approval for a cherry pick.  Edit this issue's body to add the PR number once it is available.)*
+
+PR #<PR_NUMBER>
+
+*[Type: Release](https://github.com/ampproject/amphtml/labels/Type%3A%20Release) issue for the release build you are requesting a cherry pick into:*
+
+Release Issue #<RELEASE_ISSUE_NUMBER>
+
+*Is this request for a cherry pick into a canary or a production release?*
+*(If you are requesting a cherry pick into a production release you may need to file a separate cherry pick request for the current canary as well.  See the [cherry pick documentation](https://github.com/ampproject/amphtml/blob/master/contributing/release-schedule.md#cherry-picks).)*
+
+<CANARY_OR_PRODUCTION>
+
+*Why does this issue meet the [cherry pick criteria](https://github.com/ampproject/amphtml/blob/master/contributing/release-schedule.md#cherry-pick-criteria)?  Be specific.*
+
+<YOUR_REASONS>
+
+*Assign this issue to the current TL (cramforce) if you have permission to, otherwise leave this cc line in.*
+/cc @cramforce

contributing/release-schedule.md

@@ -1,39 +1,67 @@
 # Release Schedule
 
-We push a new release of AMP to all AMP pages every week on Tuesday.
+- [Detailed schedule](#detailed-schedule)
+- [Determining if your change is in production](#determining-if-your-change-is-in-production)
+- [Cherry picks](#cherry-picks)
+	- [Cherry pick criteria](#cherry-pick-criteria)
+	- [Process for requesting a cherry pick](#process-for-requesting-a-cherry-pick)
+- [AMP Dev Channel](#amp-dev-channel)
+- [Release cadence](#release-cadence)
 
-**It will typically take 1-2 weeks for a change in AMP to be live for all users.**  After a change is submitted it will be included in the next canary build cut on Mondays.  This canary build will be tested on our [Dev Channel](#amp-dev-channel) opt-in and with 1% of users for just over a week.  If no problems are found the canary build will then be pushed to all users.
+A new release of AMP is pushed to all AMP pages every week on Tuesday.  **Once a change in AMP is merged into the master branch of the amphtml repository, it will typically take 1-2 weeks for the change be live for all users.**
 
-You can determine what is in a given build using:
+["Type: Release" GitHub issues](https://github.com/ampproject/amphtml/labels/Type%3A%20Release) are used to track the status of current and past releases (from the initial cut to canary testing to production).  Announcements about releases are made on the [AMP Slack #release channel](https://amphtml.slack.com/messages/C4NVAR0H3/) ([sign up for Slack](https://bit.ly/amp-slack-signup)).
 
-- the [releases page](https://github.com/ampproject/amphtml/releases); the release used by most users is marked as `Latest release` and the current Dev Channel release is marked as `Pre-release`
-- the [PR Use: In Canary](https://github.com/ampproject/amphtml/issues?utf8=%E2%9C%93&q=label%3A%22PR%20use%3A%20In%20Canary%22) and [PR Use: In Production](https://github.com/ampproject/amphtml/issues?utf8=%E2%9C%93&q=label%3A%22PR%20use%3A%20In%20Production%22) labels; these are added to PRs when they've made it into a canary/production build (though note there may be a delay between the build is created and when it goes live)
+## Detailed schedule
 
-Announcements regarding releases will be made on the [AMP Slack #release channel](https://amphtml.slack.com/messages/C4NVAR0H3/) ([sign up for Slack](https://docs.google.com/forms/d/e/1FAIpQLSd83J2IZA6cdR6jPwABGsJE8YL4pkypAbKMGgUZZriU7Qu6Tg/viewform?fbzx=4406980310789882877)).
+We try to stick to this schedule as closely as possible, though complications may cause delays.  You can track the latest status about any release in the ["Type: Release" GitHub issues](https://github.com/ampproject/amphtml/labels/Type%3A%20Release) and the [AMP Slack #release channel](https://amphtml.slack.com/messages/C4NVAR0H3/) ([sign up for Slack](https://bit.ly/amp-slack-signup)).
 
-## Release cadence
+- Tuesday @ [11am Pacific](https://www.google.com/search?q=11am+pacific+in+current+time+zone): a new canary release build is created from the [latest master build that passes all of our tests](https://travis-ci.org/ampproject/amphtml/branches) and is pushed to users of AMP who opted into the [AMP Dev Channel](#amp-dev-channel)
+- Wednesday:  we check bug reports for Dev Channel users and if everything looks fine, we push the canary to 1% of AMP pages
+- Thursday-Monday: we continue to monitor error rates and bug reports for Dev Channel users and the 1% of pages with the canary build
+- Tuesday (about a week after the canary release build was cut): the canary is fully pushed to production (i.e. all AMP pages will now use this build)
 
-We are intentionally cautious with our release cadence.
+## Determining if your change is in production
 
-In determining how often we should push new versions of AMP to everyone, we have to weigh many factors including:
+You can determine what is in a given build using one of the following:
 
-- stability for the millions of sites/billions of pages built using AMP
-- cache busting that might happen when we push a new version
-- the desire to get new features out quickly
+- The ["Type: Release" GitHub issues](https://github.com/ampproject/amphtml/labels/Type%3A%20Release) for each release build will include a link to the specific [release page](https://github.com/ampproject/amphtml/releases) which lists the PRs in that release.
+  - The most recent "Type: Release" issue with `(Production)` in the title is the build currently in production.
+  - The "Type: Release" issue with `(Canary)` in the title tracks the current canary.
+- The [PR Use: In Canary](https://github.com/ampproject/amphtml/issues?utf8=%E2%9C%93&q=label%3A%22PR%20use%3A%20In%20Canary%22) and [PR Use: In Production](https://github.com/ampproject/amphtml/issues?utf8=%E2%9C%93&q=label%3A%22PR%20use%3A%20In%20Production%22) labels are added to PRs when they've made it into a canary/production build.  There may be a delay between when the build is created and when it goes live.
 
-After considering all of these factors we have arrived at the 1-2 week push cycle.  Thus far we have found this to be a reasonable compromise, but we will continue to evaluate all of these factors and may make changes in the future.
+## Cherry picks
 
-## Detailed schedule
+We have a well-defined process for handling requests for changes to the canary release build or to the current production release build.  These changes are known as "cherry picks".
+
+### Cherry pick criteria
+
+**The bar for getting a cherry pick into canary or production is very high** because our goal is to produce high quality launches on a predictable schedule:
+- In general only fixes for [P0 issues](https://github.com/ampproject/amphtml/blob/master/contributing/issue-priorities.md) (causing "an outage or a critical production issue") may be cherry picked.  P0 issues are those that:
+  - cause privacy or security issues
+  - cause user data loss
+  - break existing AMP web pages in a significant way
+  - or would otherwise cause a significant harm to AMP's reputation if left unresolved
+- Regressions found in the canary that are not P0 *may* be approved if they can be resolved with a rollback.  Fixes other than rollbacks--no matter how simple they may seem--will not be approved because these can cause cascading problems and delay the release of canary to production for everyone.
+
+### Process for requesting a cherry pick
+
+Use the following process to request a cherry pick if you have a change that you believe meets the [cherry pick criteria](#cherry-pick-criteria).
+
+If you run into any issues or have any questions when requesting a cherry pick, please use the [AMP Slack #release channel](https://amphtml.slack.com/messages/C4NVAR0H3/) ([sign up for Slack](https://bit.ly/amp-slack-signup)).
+
+- Ensure there is a GitHub issue filed for the problem that needs to be resolved *before* filing the cherry pick request issue.
+- File the cherry pick request issue using the [Cherry pick request template](https://github.com/ampproject/amphtml/issues/new?title=%F0%9F%8C%B8%20Cherry%20pick%20request%20for%20%3CYOUR_ISSUE_NUMBER%3E%20into%20%3CRELEASE_ISSUE_NUMBER%3E%20%28Pending%29&template=cherry_pick_template).  Follow the instructions in the template, providing all of the requested data.  **Make sure you fill out this issue completely or your cherry pick may not be seen or acted upon.**
+- **The [TL](../GOVERNANCE.md) or their designate is the only person who may approve cherry picks.**  The TL/designate will update the issue with their decision about whether your fix warrants a cherry pick.  You should be available to respond to any questions the TL/designate has regarding your request.
+- If the TL/designate approves the cherry pick, the person currently handling releases (onduty) will work with you to ensure the cherry pick is made.
+- **Once the cherry pick is made you are responsible for verifying that the cherry pick you requested fixes the reported issue and that it does not cause other issues.**
+
+**If you are requesting a cherry pick to fix an issue in production** it is likely you will *also* need a cherry pick into the canary release build.  If this is needed file the cherry pick request for the production release build first and once it is approved file a separate cherry pick request for the canary release build.
 
-- Monday:  we create a canary build from a green release at the latest commit that passed all tests
-- Tuesday:  this canary is pushed to users of AMP who opted into the [AMP Dev Channel](#amp-dev-channel)
-- Thursday:  we check error rates for opt-in users and bug reports and if everything looks fine, we push the canary to 1% of AMP pages
-- Friday-Monday: we continue to monitor error rates and bug reports
-- Tuesday (about a week after the canary build was cut): the canary is pushed to all users
 
 ## AMP Dev Channel
 
-AMP Dev Channel is a way to opt a browser into using a newer version of the AMP JS libraries.  This release **may be less stable** and it may contain features not available to all users. 
+The AMP Dev Channel is a way to opt a browser into using the canary release build of the AMP JS libraries.  The Dev Channel **may be less stable** and it may contain features not yet available to all users.
 
 Opting into the Dev Channel is great to:
 
@@ -44,4 +72,20 @@ When you opt into the AMP Dev Channel you are only affecting the AMP JS librarie
 
 To opt your browser into the AMP Dev Channel, go to [the AMP experiments page](https://cdn.ampproject.org/experiments.html) and activate the "AMP Dev Channel" experiment. Please subscribe to our [low-volume announcements](https://groups.google.com/forum/#!forum/amphtml-announce) mailing list to get notified about important/breaking changes about AMP.
 
-If you find an issue that appears to only occur in the Dev Channel version of AMP, please [file an issue](https://github.com/ampproject/amphtml/issues/new) with a description of the problem. Please always include a URL to a page that reproduces the issue.
+**If you find an issue that appears to only occur in the Dev Channel version of AMP**:
+- please [file an issue](https://github.com/ampproject/amphtml/issues/new) with a description of the problem
+  - include a note that the problem is new to the Dev Channel build so that it can be properly prioritized
+  - include a URL to a page that reproduces the problem
+- ping the [AMP Slack #release channel](https://amphtml.slack.com/messages/C4NVAR0H3/) ([sign up for Slack](https://bit.ly/amp-slack-signup)) with the issue you filed so we can delay the push of the Dev Channel version to production if needed
+
+## Release cadence
+
+We are intentionally cautious with our release cadence.
+
+In determining how often we should push new versions of AMP to everyone, we have to weigh many factors including:
+
+- stability for the millions of sites/billions of pages built using AMP
+- cache busting that might happen when we push a new version
+- the desire to get new features out quickly
+
+After considering all of these factors we have arrived at the 1-2 week push cycle.  Thus far we have found this to be a reasonable compromise, but we will continue to evaluate all of these factors and may make changes in the future.