Kestrel on Debian/Ubuntu not sending private intermediate cert

[jack4it]

Is your feature request related to a problem? Please describe.

I'm having a TLS keypair (site.key, site.crt) issued by an internal intermediate CA which in turn is issued by an internal root CA. On Linux Kestrel is not sending the intermediate cert even the intermediate CA cert is in the site.crt.

This was reported before (#10971) but the issue got closed without a conclusion, unfortunately. By reading the discussion, I understood this is a lower-layer issue.

But the issue is really annoying and I think this deserves another look. The workaround would be adding the intermediate CA cert into the machine CA store with root user privilege. For scenarios like inside a docker container, this means sudo capability needs to be there and imposes a serious security issue.

Describe the solution you'd like

The Kestrel server should give the intermediate CA cert if it is baked in the TLS cert chain.

Additional context

I tried with both golang and python3 with the same TLS keypair. Both give out the intermedia CA cert

Following are snippets of my tests

// .net 5; DOES NOT WORK
        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>()
                        .ConfigureKestrel(_ => _.ListenAnyIP(7443,
                            options =>
                            {
                                options.UseHttps(new HttpsConnectionAdapterOptions
                                {
                                    ServerCertificate = X509Certificate2.CreateFromPemFile("site.crt", "site.key")
                                });
                            }));
                });

// .net 6; DOES NOT WORK
var builder = WebApplication.CreateBuilder(args);

builder.WebHost
    .ConfigureKestrel(_ => _.ListenAnyIP(7443,
        options =>
        {
            options.UseHttps(new HttpsConnectionAdapterOptions
            {
                ServerCertificate = X509Certificate2.CreateFromPemFile("../site.crt", "../site.key")
            });
        }));

// go: WORKS
	err := http.ListenAndServeTLS(":6443", "site.crt", "site.key", nil)
	if err != nil {
		log.Fatal("ListenAndServe: ", err)
	}

# python: WORKS
httpd = HTTPServer(('localhost', 4443), BaseHTTPRequestHandler)

httpd.socket = ssl.wrap_socket(httpd.socket,
                               keyfile="site.key",
                               certfile='site.crt', server_side=True)

httpd.serve_forever()

[jack4it]

// nodejs: WORKS
const https = require('https');
const fs = require('fs');

const options = {
  key: fs.readFileSync('site.key'),
  cert: fs.readFileSync('site.crt')
};

https.createServer(options, function (req, res) {
  res.writeHead(200);
  res.end("hello world\n");
}).listen(8000);

[davidfowl]

What's in your crt file?

[jack4it]

What's in your crt file?

Two certs, the leaf cert + the intermediate CA cert.

Kestrel returns only the leaf cert without explicitly adding the intermediate CA cert to the CA store. E.g. in Ubuntu, sudo cp $CERT /usr/local/share/ca-certificates && sudo update-ca-certificates

The other popular servers/techs, return the entire chain (both certs). The full chain is crucial for the client to successfully validate the cert with the internal root CA cert.

[davidfowl]

Yep, this was that PR that I never merged 😢 (#24935). The full chain should be resolved at startup before any requests are made (but only if the cert contains an AIA extension). How did you verify that it does not send the full cert chain?

This is a dupe of #21513. It seems like we should also just support reading the intermediates from the same file since other stacks do that. You should be able to work around this one of the more advanced HTTPS overloads but it's not easy to use...

[jack4it]

I use step CLI to do cert verification. E.g. this is a sample output from one of the stacks that emit the full chain.

# command
step certificate inspect https://localhost:8000 --insecure --bundle --short             

# output

X.509v3 TLS Certificate (ECDSA P-256) [Serial: 1551...2374]
  Subject:     abc-service-api-service
               abc-service-api-service.abc-service
               abc-service-api-service.abc-service.svc
               abc-service-api-service.abc-service.svc.cluster.local
               localhost
               10.0.15.157
               172.16.64.15
               127.0.0.1
  Issuer:      Blah Intermediate CA
  Provisioner: autocert [ID: 4w8W...XroE]
  Valid from:  2021-09-05T21:37:12Z
          to:  2021-09-06T21:38:12Z
X.509v3 Intermediate CA Certificate (ECDSA P-256) [Serial: 2374...3149]
  Subject:     Blah Intermediate CA
  Issuer:      Blah Root CA
  Valid from:  2021-08-20T01:26:57Z
          to:  2031-08-18T01:26:57Z

[jack4it]

You should be able to work around this one of the more advanced HTTPS overloads but it's not easy to use...

It seems that this would be a .NET 7 candidate. I'd definitely take a look the workaround if you can point me to a few proper places to begin with :)

[davidfowl]

I'm writing you a workaround to try right now. This will be some trial and error (I don't have any test cert chains hanging around).

This is a .NET 6 project:

using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.AspNetCore.Server.Kestrel.Https;

var builder = WebApplication.CreateBuilder(args);

builder.WebHost.ConfigureKestrel(options =>
{
    options.ConfigureEndpointDefaults(listenOptions =>
    {
        listenOptions.UseHttpsWithFullChain("site.crt", "site.key");
    });
});

var app = builder.Build();

public static class TlsListenOptionsExtensions
{
    public static ListenOptions UseHttpsWithFullChain(this ListenOptions listenOptions, string certPath, string keyPath)
    {
        var leafCertWithKey = X509Certificate2.CreateFromPemFile(certPath, keyPath);

        // Import the full cert chain
        var fullChain = new X509Certificate2Collection();
        fullChain.ImportFromPemFile(certPath);

        var options = new SslServerAuthenticationOptions
        {
            // Don't go online to retrieve cert chain
            ServerCertificateContext = SslStreamCertificateContext.Create(leafCertWithKey, fullChain, offline: true)
        };

        return listenOptions.UseHttps(new TlsHandshakeCallbackOptions
        {
            OnConnection = context => new(options)
        });
    }
}

[jack4it]

Yay! thx @davidfowl for the .NET 6 sample. I was able to use the idea on a .NET 5 project. Following codes work

options.UseHttps((_, _, _, _) =>
    {
        var cert = X509Certificate2.CreateFromPemFile("site.crt", "site.key");
        var chain = new X509Certificate2Collection();
        chain.ImportFromPemFile("site.crt");
        return new ValueTask<SslServerAuthenticationOptions>(new SslServerAuthenticationOptions
        {
            ServerCertificateContext = SslStreamCertificateContext.Create(cert, chain),
        });
    },
    null);

[davidfowl]

Nice! @jack4it I'd recommend making sure you aren't recreating all of that state per connection. Set a break point and make sure that's all happening once per application.

[davidfowl]

Closing this as a duplicate

[jack4it]

Nice! @jack4it I'd recommend making sure you aren't recreating all of that state per connection. Set a break point and make sure that's all happening once per application.

Absolutely. I'm caching it. Unrelated, but this key pair is renewed every 24 hours roughly. I'm having a file watcher on it and reloading/caching after the renewal

Thanks again!