[Bug]: Exception after pruning CRL

[saz]

Is this a critical security issue?

• This is not a security issue.

Describe the Bug

The following command was executed after stopping openvox-server:

puppetserver ca prune --remove-expired

Trying to start the service again wasn't working, due to this exception:

java.security.cert.CRLException: not enough content
        at java.base/sun.security.x509.CRLExtensions.parseExtension(CRLExtensions.java:128)
        at java.base/sun.security.x509.CRLExtensions.init(CRLExtensions.java:100)
        at java.base/sun.security.x509.CRLExtensions.<init>(CRLExtensions.java:81)
        at java.base/sun.security.x509.X509CRLImpl$TBSCertList.<init>(X509CRLImpl.java:259)
        at java.base/sun.security.x509.X509CRLImpl.parse(X509CRLImpl.java:1164)
        at java.base/sun.security.x509.X509CRLImpl.<init>(X509CRLImpl.java:380)
        at java.base/sun.security.provider.X509Factory.parseX509orPKCS7CRL(X509Factory.java:527)
        at java.base/sun.security.provider.X509Factory.engineGenerateCRLs(X509Factory.java:424)
        at java.base/java.security.cert.CertificateFactory.generateCRLs(CertificateFactory.java:553)
        at org.eclipse.jetty.util.security.CertificateUtils.loadCRL(CertificateUtils.java:66)
        at org.eclipse.jetty.util.ssl.SslContextFactory.loadCRL(SslContextFactory.java:1157)
        at com.puppetlabs.trapperkeeper.services.webserver.jetty10.utils.InternalSslContextFactory.loadCRL(InternalSslContextFactory.java:30)
        at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:295)
        at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:213)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121)
        at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:112)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:367)
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:75)
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:228)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
        at org.eclipse.jetty.server.Server.doStart(Server.java:428)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:196)
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:159)
        at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:467)
        at puppetlabs.trapperkeeper.services.webserver.jetty10_core$fn__30211$start_webserver_BANG___30216$fn__30217$fn__30218.invoke(jetty10_core.clj:788)
        at puppetlabs.trapperkeeper.services.webserver.jetty10_core$fn__30211$start_webserver_BANG___30216$fn__30217.invoke(jetty10_core.clj:787)
        at puppetlabs.trapperkeeper.services.webserver.jetty10_core$fn__30211$start_webserver_BANG___30216.invoke(jetty10_core.clj:780)
        at puppetlabs.trapperkeeper.services.webserver.jetty10_core$start_server_single_default.invokeStatic(jetty10_core.clj:1037)
        at puppetlabs.trapperkeeper.services.webserver.jetty10_core$start_server_single_default.invoke(jetty10_core.clj:1034)
        at puppetlabs.trapperkeeper.services.webserver.jetty10_core$fn__30679$start_BANG___30684$fn__30685.invoke(jetty10_core.clj:1137)
        at puppetlabs.trapperkeeper.services.webserver.jetty10_core$fn__30679$start_BANG___30684.invoke(jetty10_core.clj:1132)
        at puppetlabs.trapperkeeper.services.webserver.jetty10_service$reify__31079$service_fnk__5265__auto___positional$reify__31086.start(jetty10_service.clj:53)
        at puppetlabs.trapperkeeper.services$fn__5115$G__5083__5118.invoke(services.clj:7)
        at puppetlabs.trapperkeeper.services$fn__5115$G__5082__5122.invoke(services.clj:7)
        at puppetlabs.trapperkeeper.internal$fn__14305$run_lifecycle_fn_BANG___14312$fn__14313.invoke(internal.clj:242)
        at puppetlabs.trapperkeeper.internal$fn__14305$run_lifecycle_fn_BANG___14312.invoke(internal.clj:225)
        at puppetlabs.trapperkeeper.internal$fn__14335$run_lifecycle_fns__14340$fn__14341.invoke(internal.clj:275)
        at puppetlabs.trapperkeeper.internal$fn__14335$run_lifecycle_fns__14340.invoke(internal.clj:252)
        at puppetlabs.trapperkeeper.internal$fn__14970$build_app_STAR___14979$fn$reify__14991.start(internal.clj:663)
        at puppetlabs.trapperkeeper.internal$fn__15021$boot_services_for_app_STAR__STAR___15028$fn__15029$fn__15031.invoke(internal.clj:699)
        at puppetlabs.trapperkeeper.internal$fn__15021$boot_services_for_app_STAR__STAR___15028$fn__15029.invoke(internal.clj:697)
        at puppetlabs.trapperkeeper.internal$fn__15021$boot_services_for_app_STAR__STAR___15028.invoke(internal.clj:691)
        at clojure.core$partial$fn__5931.invoke(core.clj:2647)
        at puppetlabs.trapperkeeper.internal$fn__14381$initialize_lifecycle_worker__14392$fn__14393$fn__14556$state_machine__11619__auto____14581$fn__14584.invoke(internal.clj:295)
        at puppetlabs.trapperkeeper.internal$fn__14381$initialize_lifecycle_worker__14392$fn__14393$fn__14556$state_machine__11619__auto____14581.invoke(internal.clj:295)
        at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:58)
        at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:57)
        at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:62)
        at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:60)
        at clojure.core.async$ioc_alts_BANG_$fn__11851.invoke(async.clj:453)
        at clojure.core.async$do_alts$fn__11786$fn__11789.invoke(async.clj:320)
        at clojure.core.async.impl.channels$appm$fn__6412.invoke(channels.clj:36)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Expected Behavior

Getting rid of expired things within the CRL, without preventing openvox-server from starting again

Steps to Reproduce

systemctl stop puppetserver
puppetserver ca prune --remove-expired
systemctl start puppetserver

Environment

Package: openvox-server
Version: 8.12.1-1+ubuntu22.04

Additional Context

No response

Relevant log output

[saz]

And I'm not the only one:

[saz]

Restoring ca_crl.pem from a backup helped.

[corporate-gadfly]

Presumably stack trace comes from JDK's DerValue class while loading the CRL:

DerValue(byte[] buf, int offset, int len, boolean allowBER, boolean allowMore)
            throws IOException {

        if (len < 2) {
            throw new IOException("Too short");
        }
        int pos = offset;
        tag = buf[pos++];
        if ((tag & 0x1f) == 0x1f) {
            throw new IOException("Tag number over 30 at " + offset + " is not supported");
        }
        int lenByte = buf[pos++];

        int length;
        if (lenByte == (byte) 0x80) { // indefinite length
            if (!allowBER) {
                throw new IOException("Indefinite length encoding " +
                        "not supported with DER");
            }
            if (!isConstructed()) {
                throw new IOException("Indefinite length encoding " +
                        "not supported with non-constructed data");
            }

            // Reconstruct data source
            buf = DerIndefLenConverter.convertStream(
                    new ByteArrayInputStream(buf, pos, len - (pos - offset)), tag);
            offset = 0;
            len = buf.length;
            pos = 2;

            if (tag != buf[0]) {
                throw new IOException("Indefinite length encoding not supported");
            }
            lenByte = buf[1];
            if (lenByte == (byte) 0x80) {
                throw new IOException("Indefinite len conversion failed");
            }
        }

        if ((lenByte & 0x080) == 0x00) { // short form, 1 byte datum
            length = lenByte;
        } else {                     // long form
            lenByte &= 0x07f;
            if (lenByte > 4) {
                throw new IOException("Invalid lenByte");
            }
            if (len < 2 + lenByte) {
                throw new IOException("Not enough length bytes");
            }
            length = 0x0ff & buf[pos++];
            lenByte--;
            if (length == 0 && !allowBER) {
                // DER requires length value be encoded in minimum number of bytes
                throw new IOException("Redundant length bytes found");
            }
            while (lenByte-- > 0) {
                length <<= 8;
                length += 0x0ff & buf[pos++];
            }
            if (length < 0) {
                throw new IOException("Invalid length bytes");
            } else if (length <= 127 && !allowBER) {
                throw new IOException("Should use short form for length");
            }
        }
        // pos is now at the beginning of the content
        if (len - length < pos - offset) {
            throw new EOFException("not enough content");
        }
        if (len - length > pos - offset && !allowMore) {
            throw new IOException("extra data at the end");
        }
        this.buffer = buf;
        this.start = pos;
        this.end = pos + length;
        this.allowBER = allowBER;
        this.data = data();
    }

[corporate-gadfly]

Using attached CrlVerifier.java (which was authored by Claude), I have some evidence that the generated CRL is malformed.

=== CRL Verifier ===
File : /Users/haroon/Downloads/crl-verifier/ca_crl.pem-2026-01-27
Size : 1966 bytes

--- PEM structure ---
Has BEGIN marker : true
Has END marker   : true
PEM blocks found : 2
  Block 1: label='BEGIN X509 CRL' (~699 DER bytes)
  Block 2: label='BEGIN X509 CRL' (~681 DER bytes)

=== Block 1 of 2 : BEGIN X509 CRL ===

--- Base64 / DER ---
DER bytes decoded: 697

--- DER structure walk ---
Outer tag        : 0x30 (SEQUENCE)
Outer length     : 693
Buffer available : 693 (need 693)
Outer length OK  : exact fit

  Children of outer SEQUENCE:
  [0] tag=0x30 (SEQUENCE)  length=158     available=690     OK
  [1] tag=0x30 (SEQUENCE)  length=13      available=530     OK
  [2] tag=0x03 (BIT STRING)  length=513     available=513     OK

--- DerValue / DerInputStream attempt ---
DerValue outer   : OK  tag=0x30
TBSCertList      : OK  length=161
AlgorithmId      : OK  tag=0x30
Signature        : OK  tag=0x03
DerValue parse   : PASSED

--- TBSCertList deep walk ---
  [0] version         : 2
  [1] signature alg   : tag=0x30 len=15
  [2] issuer          : tag=0x30 len=59
  [3] thisUpdate      : tag=0x17
  [4] nextUpdate      : tag=0x17
  [5] revokedCerts    : SEQUENCE len=2
  [6] crlExtensions   : CONTEXT[0] len=49
    Extensions SEQUENCE len=47
    ext[0] OID=2.5.29.20 critical=false valueBytes=3
      inner DerValue FAILED: EOFException: not enough content
      *** THIS is the extension causing CRLException: not enough content ***
      Raw bytes (3):
        35 36 39 
    ext[1] OID=2.5.29.35 critical=false valueBytes=24
      inner DerValue OK  tag=0x30 len=24

--- CertificateFactory parse ---
Parse FAILED    : CRLException: not enough content

=== Block 2 of 2 : BEGIN X509 CRL ===

--- Base64 / DER ---
DER bytes decoded: 679

--- DER structure walk ---
Outer tag        : 0x30 (SEQUENCE)
Outer length     : 675
Buffer available : 675 (need 675)
Outer length OK  : exact fit

  Children of outer SEQUENCE:
  [0] tag=0x30 (SEQUENCE)  length=140     available=672     OK
  [1] tag=0x30 (SEQUENCE)  length=13      available=530     OK
  [2] tag=0x03 (BIT STRING)  length=513     available=513     OK

--- DerValue / DerInputStream attempt ---
DerValue outer   : OK  tag=0x30
TBSCertList      : OK  length=143
AlgorithmId      : OK  tag=0x30
Signature        : OK  tag=0x03
DerValue parse   : PASSED

--- TBSCertList deep walk ---
  [0] version         : 2
  [1] signature alg   : tag=0x30 len=15
  [2] issuer          : tag=0x30 len=43
  [3] thisUpdate      : tag=0x17
  [4] nextUpdate      : tag=0x17
  [5] crlExtensions   : CONTEXT[0] len=49
    Extensions SEQUENCE len=47
    ext[0] OID=2.5.29.35 critical=false valueBytes=24
      inner DerValue OK  tag=0x30 len=24
    ext[1] OID=2.5.29.20 critical=false valueBytes=3
      inner DerValue OK  tag=0x02 len=3

--- CertificateFactory parse ---
Parse           : OK
Issuer          : CN=Puppet Root CA: 09e1dcdd38e229
This update     : Wed Feb 05 12:56:36 EST 2020
Next update     : Fri Feb 02 12:56:39 EST 2035
Revoked entries : 0
Sig algorithm   : SHA256withRSA

According to Claude, block 2 is good.

Block 2 (good): valueBytes=3 → raw bytes 02 01 XX → tag=0x02 (INTEGER), which is correct. CRL Number is an INTEGER.
Block 1 (bad): valueBytes=3 → raw bytes 35 36 39 → that's the ASCII characters 5, 6, 9. There is no DER tag, no length byte — it's the literal ASCII string "569" stored directly instead of a DER-encoded INTEGER.
So PuppetServer wrote the CRL Number extension value as the ASCII text 569 instead of DER-encoding it as INTEGER 569 (02 02 02 39). The correct DER encoding of 569 would be:

02 02 02 39
│  │  └──── 569 = 0x0239
│  └─────── length: 2 bytes
└────────── tag: INTEGER

Instead it contains:

35 36 39
│  │  └── '9' (0x39)
│  └───── '6' (0x36)
└──────── '5' (0x35)

Now we need a way to quickly figure out in which version this regression was introduced.

Compile with (assuming Java 17):

javac --add-exports java.base/sun.security.util=ALL-UNNAMED CrlVerifier.java

Run with:

java  --add-exports java.base/sun.security.util=ALL-UNNAMED \
      --add-opens  java.base/sun.security.util=ALL-UNNAMED \
      CrlVerifier ca_crl.pem

[nmburgan]

I believe the problem is OpenVoxProject/openvoxserver-ca@9ac9a73.

Before the change, the CRL number was updated and ASN1 encoded properly via OpenSSL::ASN1::Integer(updated_crl_number). After the change, it's doing OpenSSL::X509::Extension.new("crlNumber", ext.value.next) which is just a plain string. (I think ext.value.next really shouldn't be used here, because that just increments it string-ily, not as an integer, which kinda just works because String.next mostly works fine for plain integer strings, but there might be edge cases). Shoving a plain string in there probably kinda works because Ruby's OpenSSL binding is lenient enough so that the prune command works, but Java's parser probably chokes on it.

[corporate-gadfly]

Fix is in OpenVoxProject/openvoxserver-ca#26

[corporate-gadfly]

I tested manually on lima with ubuntu-lts template with a local copy of openvoxserver-ca-3.1.0.gem.

# install openvox-server
curl -LOR https://apt.voxpupuli.org/openvox8-release-ubuntu24.04.deb
dpkg -i openvox8-release-ubuntu24.04.deb
apt update
apt install -y openvox-server

# remember to add puppet to /etc/hosts

# setup CA
/opt/puppetlabs/server/bin/puppetserver ca setup
# install gem and remove old gem
/opt/puppetlabs/puppet/bin/gem install --install-dir "/opt/puppetlabs/puppet/lib/ruby/vendor_gems" openvoxserver-ca-3.1.0.gem
rm -rf /opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/openvoxserver-ca-3.0.0/

# start server
systemctl start puppetserver
# Generate a few certs
/opt/puppetlabs/server/bin/puppetserver ca generate --certname=node1.local
/opt/puppetlabs/server/bin/puppetserver ca generate --certname=node2.local
/opt/puppetlabs/server/bin/puppetserver ca generate --certname=node3.local
# Clean some certs, out of order
/opt/puppetlabs/server/bin/puppetserver ca clean --certname=node3.local
/opt/puppetlabs/server/bin/puppetserver ca clean --certname=node2.local
# prepare to prune some serial numbers
systemctl stop puppetserver
# get some serial numbers
openssl crl -inform PEM -in /etc/puppetlabs/puppetserver/ca/ca_crl.pem -text -noout | grep Serial | awk '{print $NF}' | paste -sd ','
# prune some entries
/opt/puppetlabs/server/bin/puppetserver ca prune --remove-entries --serial 04,03
# and you should get output similar to:
# Total number of certificates found in Puppet's CRL is: 2.
# Removed 2 certs from Puppet's CRL.`
systemctl start puppetserver

• At this point, the next command will fail unless you are using the new (fixed) gem:
  • /opt/puppetlabs/server/bin/puppetserver ca list --all
• With original gem, you will get:

  Error:
      code: 500
      body: Internal Server Error: java.security.cert.CRLException: Too short
  Error while getting certificates