Find alternatives to Mozilla cert list on Linux

[davidanthoff]

I think this should probably be reverted once the pemfile dependency is upgraded to include rustls/pemfile#7. Yes, the Mozilla certs a known good, so why not use them? Because many users are behind MITM proxying firewalls and the connection will fail using only public certs. On such systems, a sysadmin will typically have added the CA root cert for the firewall to the system so that secure connections can be made. By using Mozilla certs, that will stop working.

Originally posted by @StefanKarpinski in #335 (comment)

[davidanthoff]

@StefanKarpinski and I just had another chat, and for now the strategy would be to a) use system certs, but b) detect Debian and on Debian don't use them by default, but have users opt-in to using system certs via some env var.

[davidanthoff]

https://crates.io/crates/os_info might be useful (thanks to @wolthom for finding it!).

[StefanKarpinski]

Here's the logic we use in NetworkOptions:

https://github.com/JuliaLang/NetworkOptions.jl/blob/791fa05928f03b7f18ff5/src/ca_roots.jl#L85-L110

I'm not sure if juliaup uses OpenSSL on Linux; if it does, it can probably skip the logic in there that skips over pem files that only contain OpenSSL-specific BEGIN TRUSTED CERTIFICATE blocks. The list of places where we look is here:

https://github.com/JuliaLang/NetworkOptions.jl/blob/791fa05928f03b7f18ff5/src/ca_roots.jl#L58-L68

I based this off of a combination of what Rust and Go do when looking at these files and it seems to work pretty well (we needed to do some reordering at some point). If you wanted, we could put this list somewhere common that you could download it during the rustup build process and likewise NetworkOptions could grab it from there. Otherwise we could just try to keep the lists in sync—they shouldn't have to change often. A comment saying to change the other if one of theses changes might suffice.

[StefanKarpinski]

Looking at this issue in an attempt to close out the public release for Linux & MacOS milestone.

[wolthom]

@StefanKarpinski @davidanthoff
I'm a bit out of depth regarding the certificate roots / native certificate stores vs bundled ones. If the points below do not actually address this issue, feel free to skip over this comment.

After searching around in the ureq and rustls docs for a bit, I wonder if this could be an option:
https://docs.rs/ureq/2.5.0/ureq/#trusted-roots
This section references:
https://docs.rs/rustls-native-certs/latest/rustls_native_certs/

Here is an explanation of how to use rustls_native_certs with ureq:
algesten/ureq#386 (comment)

If I read the rustls_native_certs landing page correctly, this crate is designed to take care of the platform-specific certificate handling. It does seem to be quite widely used in the Rust ecosystem: https://crates.io/crates/rustls-native-certs

[Firionus]

@wolthom Indeed, we should be able to just use the native-certs option in ureq to fix this issue. This is equivalent to reverting #337.

As long as pemfile is used in version >=1.0.1, #335 will not appear again (we should check that we use that new version).

---

There's the remaining issue of whether one would want to rely on the ca-certificates package in Debian. The author of rustls-native-certs thinks it's poorly maintained (source):

The quality of the ca-certificates package on debian-based Linux distributions is poor. At the time of writing, this ships many certificates not included in the Mozilla set, either because they failed an audit and were withdrawn or were removed for mississuance.

The mentioned problematic root certs were removed in versions 20190110 and 20200601 of ca-certificates (https://salsa.debian.org/debian/ca-certificates/-/blob/master/debian/changelog), roughly 7 and 12 months after the issues were resolved on BugZilla. Sounds a bit long, but I'm not sure juliaup could beat that with its release cycle, especially once it becomes a bit more stable.
Also, I only see two open certificate bugs in the ca-certificates bug tracker: One recently about two expired certificates, one very recent where the situation is still developing. Doesn't look too bad to me, but I don't know anything about security.

It seems to me the only problems with system root certs on Debian that could arise for juliaup are:

• ca-certificates ships a compromised certificate. If the download location were to be signed with a certificate chain leading to that root CA, MITM attackers could inspect and change the Julia download of juliaup users.
  With the standard download location https://julialang-s3.julialang.org, this means the certificate chain of GlobalSign or Julia would be compromised, in which case Julia ought to quickly change their certificates anyway.
  If a non-default download location is specified via JULIAUP_SERVER, the user has to trust that server in the same sense they would trust Julia to keep their certificates secure. Since the user installs executables from that source, they have to trust it in a much stronger sense already.
  Overall, I don't see how this scenario would be a problem for us.
• ca-certificates does not ship the root certificate with which the Julia download is signed, so people cannot download Julia. Currently the root CA for the default location seems to be GlobalSign - if you're missing that root certificate, something is seriously wrong with your system certificates.
  The main problem then is in support: If we rely on system certificates more strongly, people with broken system certificates might open issues against juliaup, needing help with their system certificates. Or worse, they could stop using Julia since the installer doesn't work for them.

We could attempt to get the best of both worlds by merging system certificates (allowing the use case behind MITM firewalls) with builtin certificates (as fallback for bad system certificates), but I'm not sure that's worth the effort.

TL;DR

From my view, I think changing to system certificates is the way to go. I don't see why Debian's ca-certificates is significantly worse than other system certificate stores. However I'd expect some people to have problems with their system certificates, which is the price to pay for more flexibility.

[davidanthoff]

So, this is simply a policy decision. Right now, it is using the Mozilla list on Linux. #368 would change it to use system certificate list on Linux, i.e. align things with the state on Windows and Mac. If there is a consensus that that is OK from a security point of view, we can just merge #368 and that should be all that is needed, AFAIK.

[StefanKarpinski]

I agree with @Firionus: it may be the case that Debian system certs are not good, but that's frankly not our problem. Debian should get their shit together, or admins/users should stop using Debian, or they should at least replace the system certs with better ones. In all cases, not Julia's responsibility.