Fix #77411: large serialNumber returned as hex with OpenSSL 1.1

[HypeMC]

Starting with OpenSSL version 1.1 i2s_ASN1_INTEGER returns large serial numbers in hex form. Since this function is used by openssl_x509_parse to calculate the serialNumber field, it is no longer guaranteed to be an integer.

  ["serialNumber"]=>
  string(34) "0xCB0F6A32FD527F6C0001000056603DD9"
  ["serialNumberHex"]=>
  string(32) "CB0F6A32FD527F6C0001000056603DD9"

[nikic]

Looks reasonable to me. @bukka Any concerns?

[bukka]

I'm thinking if it's not actually better to keep it as it is with that 0x. As can be seen in the linked commit message, it takes quadratic time to pretty print it and the value of conversion is not really clear. Is there any reason why it needs to be printed in decimal?

[HypeMC]

@bukka In my particular case, yes, I need the decimal format since the third party system I'm using requires so.

In any case, the problem I see here is that this breaking change happened silently without any warning. If printing the decimal format really is slow and should be removed, then the breaking change should be done intentionally and should be documented. Otherwise, it's just an unplanned and undocumented breaking change that possibly might have optimize some code by accident. That's at least how I see it.

[bukka]

Well I'm not sure if it was ever specified that it's a decimal number - we don't even document the result anywhere as far as I know. That said I kind of get that it might still break some code.

I think it would be probably good to see if it can really cause some kind of issue and if the slow down is really noticeable. Are you able to find out how big can the serial numbers be and do a quick analysis of the impact if we do the decimal conversion. We can consider the openssl_x509_parse as not a perf critical function so if it can't cause big issue, it might be sensible to accept this code. I would like to just have some idea if doing decimal conversion might cause some issues in the future.

[cmb69]

From the docs:

The structure of the returned data is (deliberately) not yet documented, as it is still subject to change.

[github-actions]

There has not been any recent activity in this PR. It will automatically be closed in 7 days if no further action is taken.

[parallels999]

The structure of the returned data is (deliberately) not yet documented, as it is still subject to change.

but although it is subject to change, it should have a fixed return type, sometimes it is integer, other times it is hex

<complexType name="X509IssuerSerialType"> 
  <sequence> 
    <element name="X509IssuerName" type="string"/> 
    <element name="X509SerialNumber" type="integer"/> 
  </sequence>
</complexType>

Signing XMLs i found this problem, X509SerialNumber expects integer, throws fatal error on hex chars

CC: @bukka