Merge branch 'master' of https://github.com/preset-io/superset into f…

…eat-add-timezone

RELEASING/README.md

@@ -67,6 +67,32 @@ need to be done at every release.
     svn update
 ```
 
+To minimize the risk of mixing up your local development environment, it's recommended to work on the
+release in a different directory than where the devenv is located. In this example, we'll clone
+the repo directly from the main `apache/superset` repo to a new directory `superset-release`:
+
+```bash
+cd <MY PROJECTS PATH>
+git clone git@github.com:apache/superset.git superset-release
+cd superset-release
+```
+
+We recommend setting up a virtual environment to isolate the python dependencies from your main
+setup:
+
+```bash
+virtualenv venv
+source venv/bin/activate
+```
+
+In addition, we recommend using the [`cherrytree`](https://pypi.org/project/cherrytree/) tool for
+automating cherry picking, as it will help speed up the release process. To install `cherrytree`
+and other dependencies that are required for the release process, run the following commands:
+
+```bash
+pip install -r RELEASING/requirements.txt
+```
+
 ## Setting up the release environment (do every time)
 
 As the vote process takes a minimum of 72h, sometimes stretching over several weeks
@@ -78,35 +104,41 @@ the wrong files/using wrong names. There's a script to help you set correctly al
 necessary environment variables. Change your current directory to `superset/RELEASING`
 and execute the `set_release_env.sh` script with the relevant parameters:
 
-Usage (BASH):
+Usage (MacOS/ZSH):
+
 ```bash
-. set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
+cd RELEASING
+source set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
 ```
 
-Usage (ZSH):
+Usage (BASH):
+
 ```bash
-source set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
+. set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
 ```
 
 Example:
+
 ```bash
-source set_release_env.sh 0.38.0rc1 myid@apache.org
+source set_release_env.sh 1.5.1rc1 myid@apache.org
 ```
 
-The script will output the exported variables. Here's example for 0.38.0rc1:
+The script will output the exported variables. Here's example for 1.5.1rc1:
 
 ```
+-------------------------------
 Set Release env variables
-SUPERSET_VERSION=0.38.0
+SUPERSET_VERSION=1.5.1
 SUPERSET_RC=1
-SUPERSET_GITHUB_BRANCH=0.38
-SUPERSET_PGP_FULLNAME=myid@apache.org
-SUPERSET_VERSION_RC=0.38.0rc1
-SUPERSET_RELEASE=apache-superset-0.38.0
-SUPERSET_RELEASE_RC=apache-superset-0.38.0rc1
-SUPERSET_RELEASE_TARBALL=apache-superset-0.38.0-source.tar.gz
-SUPERSET_RELEASE_RC_TARBALL=apache-superset-0.38.0rc1-source.tar.gz
-SUPERSET_TMP_ASF_SITE_PATH=/tmp/superset-site-0.38.0
+SUPERSET_GITHUB_BRANCH=1.5
+SUPERSET_PGP_FULLNAME=villebro@apache.org
+SUPERSET_VERSION_RC=1.5.1rc1
+SUPERSET_RELEASE=apache-superset-1.5.1
+SUPERSET_RELEASE_RC=apache-superset-1.5.1rc1
+SUPERSET_RELEASE_TARBALL=apache-superset-1.5.1-source.tar.gz
+SUPERSET_RELEASE_RC_TARBALL=apache-superset-1.5.1rc1-source.tar.gz
+SUPERSET_TMP_ASF_SITE_PATH=/tmp/incubator-superset-site-1.5.1
+-------------------------------
 ```
 
 ## Crafting a source release
@@ -116,41 +148,133 @@ a branch named with the release MAJOR.MINOR version (on this example 0.37).
 This new branch will hold all PATCH and release candidates
 that belong to the MAJOR.MINOR version.
 
+### Creating an initial minor release (e.g. 1.5.0)
+
 The MAJOR.MINOR branch is normally a "cut" from a specific point in time from the master branch.
-Then (if needed) apply all cherries that will make the PATCH.
+When creating the initial minor release (e.g. 1.5.0), create a new branch:
+
+```bash
+git checkout master
+git pull
+git checkout -b ${SUPERSET_GITHUB_BRANCH}
+git push origin $SUPERSET_GITHUB_BRANCH
+```
+
+Note that this initializes a new "release cut", and is NOT needed when creating a patch release
+(e.g. 1.5.1).
+
+### Creating a patch release (e.g. 1.5.1)
+
+When getting ready to bake a patch release, simply checkout the relevant branch:
+
+```bash
+git checkout master
+git pull
+git checkout ${SUPERSET_GITHUB_BRANCH}
+```
+
+### Cherry picking
+
+It is customary to label PRs that have been introduced after the cut with the label
+`v<MAJOR>.<MINOR>`. For example, for any PRs that should be included in the 1.5 branch, the
+label `v1.5` should be added.
+
+To see how well the labelled PRs would apply to the current branch, run the following command:
+
+```bash
+cherrytree bake -r apache/superset -m master -l v${SUPERSET_GITHUB_BRANCH} ${SUPERSET_GITHUB_BRANCH}
+```
+
+This requires the presence of an environment variable `GITHUB_TOKEN`. Alternatively,
+you can pass the token directly via the `--access-token` parameter (`-at` for short).
+
+#### Happy path: no conflicts
+
+This will show how many cherries will apply cleanly. If there are no conflicts, you can simply apply all cherries
+by adding the `--no-dry-run` flag (`-nd` for short):
 
 ```bash
-git checkout -b $SUPERSET_GITHUB_BRANCH
-git push upstream $SUPERSET_GITHUB_BRANCH
+cherrytree bake -r apache/superset -m master -l v${SUPERSET_GITHUB_BRANCH} -nd ${SUPERSET_GITHUB_BRANCH}
 ```
 
+#### Resolving conflicts
+
+If there are conflicts, you can issue the following command to apply all cherries up until the conflict automatically, and then
+break by adding the `-error-mode break` flag (`-e break` for short):
+
+```bash
+cherrytree bake -r apache/superset -m master -l v${SUPERSET_GITHUB_BRANCH} -nd -e break ${SUPERSET_GITHUB_BRANCH}
+```
+
+After applying the cleanly merged cherries, `cherrytree` will specify the SHA of the conflicted cherry. To resolve the conflict,
+simply issue the following command:
+
+```bash
+git cherry-pick <SHA>
+```
+
+Then fix all conflicts, followed by
+
+```bash
+git add -u  # add all changes
+git cherry-pick --continue
+```
+
+After this, rerun all the above steps until all cherries have been picked, finally pushing all new commits to the release branch
+on the main repo:
+
+```bash
+git push
+```
+
+### Updating changelog
+
 Next, update the `CHANGELOG.md` with all the changes that are included in the release.
-Make sure the branch has been pushed to `upstream` to ensure the changelog generator
+Make sure the branch has been pushed to `origin` to ensure the changelog generator
 can pick up changes since the previous release.
-Change log script requires a github token and will try to use your env var GITHUB_TOKEN.
-you can also pass the token using the parameter `--access_token`.
+Similar to `cherrytree`, the change log script requires a github token, either as an env var
+(`GITHUB_TOKEN`) or as the parameter `--access_token`.
+
+#### Initial release (e.g. 1.5.0)
+
+When generating the changelog for an initial minor relese, you should compare with
+the previous release (in the example, the previous release branch is `1.4`, so remember to
+update it accordingly):
 
-Example:
 ```bash
-python changelog.py --previous_version 0.37 --current_version 0.38 changelog
+python changelog.py --previous_version 1.4 --current_version ${SUPERSET_GITHUB_BRANCH} changelog
 ```
 
 You can get a list of pull requests with labels started with blocking, risk, hold, revert and security by using the parameter `--risk`.
 Example:
+
 ```bash
 python changelog.py --previous_version 0.37 --current_version 0.38 changelog --access_token {GITHUB_TOKEN} --risk
 ```
 
-The script will checkout both branches and compare all the PR's, copy the output and paste it on the `CHANGELOG.md`
+The script will checkout both branches, compare all the PRs, and output the lines that are needed to be added to the
+`CHANGELOG.md` file in the root of the repo. Remember to also make sure to update the branch id (with the above command
+`1.5` needs to be changed to `1.5.0`)
 
 Then, in `UPDATING.md`, a file that contains a list of notifications around
 deprecations and upgrading-related topics,
 make sure to move the content now under the `Next Version` section under a new
 section for the new release.
 
-Finally bump the version number on `superset-frontend/package.json` (replace with whichever version is being released excluding the RC version):
+#### Patch release (e.g. 1.5.1)
+
+To compare the forthcoming patch release with the latest release from the same branch, set
+`--previous_version` as the tag of the previous release (in this example `1.5.0`; remember to update accordingly)
+
+```bash
+python changelog.py --previous_version 1.5.0 --current_version ${SUPERSET_GITHUB_BRANCH} changelog
+```
+
+### Set version number
+
+Finally, bump the version number on `superset-frontend/package.json` (replace with whichever version is being released excluding the RC version):
 
-```json
+```
 "version": "0.38.0"
 ```
 
@@ -162,9 +286,13 @@ git add ...
 git commit ...
 # push new tag
 git tag ${SUPERSET_VERSION_RC}
-git push upstream ${SUPERSET_VERSION_RC}
+git push origin ${SUPERSET_VERSION_RC}
 ```
 
+### Create a release on Github
+
+After submitting the tag, follow the steps [here](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository) to create the release. Use the vote email text as the content for the release description. Make sure to check the "This is a pre-release" checkbox for release canditates. You can check previous releases if you need an example.
+
 ## Preparing the release candidate
 
 The first step of preparing an Apache Release is packaging a release candidate
@@ -180,14 +308,15 @@ the tag and create a signed source tarball from it:
 
 Note that `make_tarball.sh`:
 
-- By default assumes you have already executed an SVN checkout to `$HOME/svn/superset_dev`.
-This can be overridden by setting `SUPERSET_SVN_DEV_PATH` environment var to a different svn dev directory
+- By default, the script assumes you have already executed an SVN checkout to `$HOME/svn/superset_dev`.
+  This can be overridden by setting `SUPERSET_SVN_DEV_PATH` environment var to a different svn dev directory
 - Will refuse to craft a new release candidate if a release already exists on your local svn dev directory
 - Will check `package.json` version number and fails if it's not correctly set
 
 ### Build and test the created source tarball
 
 To build and run the **local copy** of the recently created tarball:
+
 ```bash
 # Build and run a release candidate tarball
 ./test_run_tarball.sh local
@@ -209,6 +338,7 @@ svn update
 ### Build and test from SVN source tarball
 
 To build and run the recently created tarball **from SVN**:
+
 ```bash
 # Build and run a release candidate tarball
 ./test_run_tarball.sh
@@ -217,6 +347,7 @@ To build and run the recently created tarball **from SVN**:
 ```
 
 ### Voting
+
 Now you're ready to start the [VOTE] thread. Here's an example of a
 previous release vote thread:
 https://lists.apache.org/thread.html/e60f080ebdda26896214f7d3d5be1ccadfab95d48fbe813252762879@<dev.superset.apache.org>
@@ -225,17 +356,10 @@ To easily send a voting request to Superset community, still on the `superset/RE
 
 ```bash
 # Note: use Superset's virtualenv
-(venv)$ python send_email.py vote_pmc
+(venv)$ python generate_email.py vote_pmc
 ```
 
-The script will interactively ask for extra information so it can authenticate on the Apache Email Relay.
-The release version and release candidate number are fetched from the previously set environment variables.
-
-```
-Sender email (ex: user@apache.org): your_apache_email@apache.org
-Apache username: your_apache_user
-Apache password: your_apache_password
-```
+The script will generate the email text that should be sent to dev@superset.apache.org using an email client. The release version and release candidate number are fetched from the previously set environment variables.
 
 Once 3+ binding votes (by PMC members) have been cast and at
 least 72 hours have past, you can post a [RESULT] thread:
@@ -245,23 +369,20 @@ To easily send the result email, still on the `superset/RELEASING` directory:
 
 ```bash
 # Note: use Superset's virtualenv
-python send_email.py result_pmc
+python generate_email.py result_pmc
 ```
 
 The script will interactively ask for extra information needed to fill out the email template. Based on the
 voting description, it will generate a passing, non passing or non conclusive email.
-here's an example:
+Here's an example:
 
 ```
-Sender email (ex: user@apache.org): your_apache_email@apache.org
-Apache username: your_apache_user
-Apache password: your_apache_password
 A List of people with +1 binding vote (ex: Max,Grace,Krist): Daniel,Alan,Max,Grace
 A List of people with +1 non binding vote (ex: Ville): Ville
 A List of people with -1 vote (ex: John):
 ```
 
-Following the result thread, yet another [VOTE] thread should be
+The script will generate the email text that should be sent to dev@superset.apache.org using an email client. The release version and release candidate number are fetched from the previously set environment variables.
 
 ### Validating a release
 
@@ -270,6 +391,7 @@ https://www.apache.org/info/verification.html
 ## Publishing a successful release
 
 Upon a successful vote, you'll have to copy the folder into the non-"dev/" folder.
+
 ```bash
 cp -r ~/svn/superset_dev/${SUPERSET_VERSION_RC}/ ~/svn/superset/${SUPERSET_VERSION}/
 cd ~/svn/superset/
@@ -281,6 +403,7 @@ svn update
 ```
 
 Then tag the final release:
+
 ```bash
 # Go to the root directory of the repo, e.g. `~/src/superset`
 cd ~/src/superset/
@@ -289,7 +412,7 @@ git branch
 # Create the release tag
 git tag -f ${SUPERSET_VERSION}
 # push the tag to the remote
-git push upstream ${SUPERSET_VERSION}
+git push origin ${SUPERSET_VERSION}
 ```
 
 ### Update CHANGELOG and UPDATING on superset
@@ -312,9 +435,11 @@ Once it's all done, an [ANNOUNCE] thread announcing the release to the dev@ mail
 
 ```bash
 # Note use Superset's virtualenv
-python send_email.py announce
+python generate_email.py announce
 ```
 
+The script will generate the email text that should be sent to dev@superset.apache.org using an email client. The release version is fetched from the previously set environment variables.
+
 ### GitHub Release
 
 Finally, so the GitHub UI reflects the latest release, you should create a release from the

RELEASING/changelog.py

@@ -164,6 +164,12 @@ def _is_risk_pull_request(self, labels: List[Any]) -> bool:
         return False
 
     def _get_changelog_version_head(self) -> str:
+        if not len(self._logs):
+            print(
+                f"No changes found between revisions. "
+                f"Make sure your branch is up to date."
+            )
+            sys.exit(1)
         return f"### {self._version} ({self._logs[0].time})"
 
     def _parse_change_log(

RELEASING/email_templates/announce.j2

@@ -17,7 +17,7 @@
   under the License.
 -#}
 To: {{ receiver_email }}
-From: {{ sender_email }}
+
 Subject: [ANNOUNCE] Apache {{ project_name }} version {{ version }} Released
 
 Hello Community,

RELEASING/email_templates/result_pmc.j2

@@ -17,7 +17,7 @@
   under the License.
 -#}
 To: {{ receiver_email }}
-From: {{ sender_email }}
+
 Subject: [RESULT] [VOTE] Release Apache {{ project_name }} {{ version }} based on Superset {{ version_rc }}
 
 Thanks to everyone that participated. The vote to release