Fix RSA signature verification issue

[hoihochan]

Both verifyKeySignature() and verifyCertificateVerify() has a bug when handling RSA signature as it looks at the signature algorithm of the certificate to determine whether to verify with RSA PKCSv1.5. This will cause issues if the certificate's issuing CA uses something other than RSA (e.g. ECDSA) to sign the certificate.

Since DTLS v1.2 does not support RSA-PSS [1], we can just use RSA PKCSv1.5 verification directly if the public key of the certificate is RSA.

[1] https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-16

[codecov]

Codecov Report

Attention: Patch coverage is 50.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 78.23%. Comparing base (98a05d6) to head (d2d2fee).
Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
crypto.go	50.00%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #679      +/-   ##
==========================================
- Coverage   80.18%   78.23%   -1.95%     
==========================================
  Files         101      101              
  Lines        5374     6567    +1193     
==========================================
+ Hits         4309     5138     +829     
- Misses        695     1058     +363     
- Partials      370      371       +1     

Flag	Coverage Δ
go	78.26% <50.00%> (-1.95%)	⬇️
wasm	62.92% <25.00%> (-1.56%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Sean-Der]

Thank you so much @hoihochan !

How was the library otherwise? Any things I can improve/change?

[hoihochan]

Thank you so much @hoihochan !

How was the library otherwise? Any things I can improve/change?

It's been good. We had an old fork of the library with this fix and I wanted to upstream the change so we can use the upstream library.

When I get time I would like to help on #524

[Sean-Der]

I would love your help!

Reach out if you need anything. Really appreciate any help :)