Generate partner tarballs that are identical to those generated by github

[omajid]

Describe the Problem

This is a follow-up from #3717. In that issue, we decided to publish tarballs to the partner site for partners to consume. However, those tarballs are different than the release tarballs generated by GitHub.

That means that some distros are using a different release tarball than others, even if they both build/ ship the same .NET version.

From a security, auditing and consistency point of view, it would be fantastic if the tarballs were identical between the two sets of releases.

Describe the Solution

https://wiki.debian.org/Creating%20signed%20GitHub%20releases#Creating_GnuPG-signed_releases_on_GitHub_-_alternative_local_workflow describes how to generate a tarball locally from a git repo that matches the one generated by GithHub.

I tested it out with the partner git repo:

$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="39 (Workstation Edition)"
ID=fedora
VERSION_ID=39
VERSION_CODENAME=""
PLATFORM_ID="platform:f39"
PRETTY_NAME="Fedora Linux 39 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:39"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f39/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=39
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=39
SUPPORT_END=2024-11-12
VARIANT="Workstation Edition"
VARIANT_ID=workstation
$ git --version
git version 2.44.0
$ git remote -v
origin  git@ssh.dev.azure.com:<removed> (fetch)
origin  git@ssh.dev.azure.com:<removed> (push)
$ ver=8.0.3; tag=v${ver}; git -c tar.tar.gz.command='gzip -cn' archive --format=tar.gz --prefix="dotnet-${ver}/" -o "../private-dotnet-${ver}.tar.gz" "${tag}"

The checksum on this private-dotnet-8.0.3.tar.gz matches the one from GitHub:

$ sha256sum *tar.gz
c8e1c21bfc9e69e01e99e9288da4d39ab7ecf7dbb33885d12c2f744a1ec3b149  dotnet-sdk-source-8.0.103.tar.gz   # original partner tarball
1cf467f47168572aae8cbcbf8fede972b31c8c1177f6584560a7fe4aecd66d19  private-dotnet-8.0.3.tar.gz        # self-generated release tarball
1cf467f47168572aae8cbcbf8fede972b31c8c1177f6584560a7fe4aecd66d19  v8.0.3.tar.gz                      # github's release tarball

Additional Context

This also lets everyone easily verify the private tarball matches the git repositories (public or partner-private) and lets us check for things like the xz source-tarball backdoor.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[oleksandr-didyk]

Hey Omair, thanks a lot of looking into this and opening an issue!

One question that I have is about validation - is comparing the checksum of the two tarballs as you have done in the description enough or would you see additional verification needed? Thanks!

[omajid]

Hey @oleksandr-didyk ! I am not sure I understand the question. Are you asking what validation can be done to ensure the tarball is identical to GitHub's? Or is there some other property you are looking to validate on the tarball?

For validating that the tarballs generated by the commands are identical to GitHub, comparing using sha256 (maybe sha512 if you want to be safer) hashes for a release or two is good enough for me, specially given that it's basically impossible to make two files have the same sha256 hash.

[oleksandr-didyk]

Are you asking what validation can be done to ensure the tarball is identical to GitHub's?

Yeah, that's what I meant, sorry for the confusion.

Thanks! I'll make sure to add the sha256 comparison as part of this issue so we don't accidentally produce different tarballs in the future