Mono 5.18 SSL store problems on aarch64 platforms

[bjorn1981]

For new Package Requests, see the guidelines

Setup

Package Name: Mono
Package Version: 5.18

NAS Model: DS118
NAS Architecture: ARM rtd1296
DSM version: 6.1

Expected behavior

https connection excepted

Actual behavior

any https connection is refused since updating Mono 5.8 -> 5.18, for example torrentapi or telegram. Happens in sonarr and radarr. Mono, Sonarr and Radarr are all latest version by SynoCommunity.

log error:
System.Net.WebException: Error: SecureChannelFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:0f000078:elliptic curve routines:OPENSSL_internal:POINT_IS_NOT_ON_CURVE

Running /var/packages/mono/target/bin/cert-sync /etc/ssl/certs/ca-certificates.crt did not help. Also noticed there is no /usr/share/.mono/ folder

Connecting torrentapi over http instead of https works.

[ymartin59]

@m4tt075 Hello. As you can see, I have just successfully uploaded and published Mono and related application package updates.

[bjorn1981]

I might not be looking in the correct place, but the latest I see is Mono 5.18.0.240-12. I have that version installed.

[m4tt075]

@bjorn1981 Yes, that is the latest version we ship. I've just searched the net for reports of the same error. There is not a lot. I found some issues for ARM64 platforms with this, but no clear direction where it comes from. Some people blamed mono, some openssl.
EDIT: To complete my message: Please report this with the mono team as they might be able to debug this further. We'll have to see whether this is an issue that affects people more broadly or just ARM64 platforms. Sorry, but I'm afraid I'm not able to do more right now. As temporary workaround, I'd downgrade the mono package again, but please report the error first, to ensure you provide all information the mono people need.

[m4tt075]

Sorry, accidentally closed and reopened...

[silv3r23]

same issue here since mono update to 5.18.0.240-12

  at /spksrc/spk/mono/work-aarch64-6.1/mono-5.18.0.240/external/boringssl/crypto/ec/ec.c:835
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00038] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000a1] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000ff] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at Mono.Net.Security.AsyncProtocolRequest.StartOperation (System.Threading.CancellationToken cancellationToken) [0x0008b] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00252] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x00126] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at System.Net.WebOperation.Run () [0x0009a] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <63552ca6ec3f4cb28de3f8c0aa59b164>:0
  at NzbDrone.Common.Http.Dispatchers.ManagedHttpDispatcher.GetResponse (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookies) [0x0012f] in <a1d4cd371e3f48118cccea718b1ace8d>:0
  at NzbDrone.Common.Http.Dispatchers.FallbackHttpDispatcher.GetResponse (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookies) [0x0009d] in <a1d4cd371e3f48118cccea718b1ace8d>:0
  at NzbDrone.Common.Http.HttpClient.ExecuteRequest (NzbDrone.Common.Http.HttpRequest request, System.Net.CookieContainer cookieContainer) [0x0007e] in <a1d4cd371e3f48118cccea718b1ace8d>:0
  at NzbDrone.Common.Http.HttpClient.Execute (NzbDrone.Common.Http.HttpRequest request) [0x00008] in <a1d4cd371e3f48118cccea718b1ace8d>:0
  at NzbDrone.Common.Http.HttpClient.Get (NzbDrone.Common.Http.HttpRequest request) [0x00007] in <a1d4cd371e3f48118cccea718b1ace8d>:0
  at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider.FetchCapabilities (NzbDrone.Core.Indexers.Newznab.NewznabSettings indexerSettings) [0x00052] in <b938542dd9a14cfc824f9cd70d8df65b>:0

[m4tt075]

@silv3r23 You are on aarch64 as well, right?

[m4tt075]

@kaso17 May I please ask you to look at the above? Can you make sense out of this or do you know where this is coming from? Any advice appreciated...

[kaso17]

I never had time or any aarch64 device to track this down further.

I've never seen this on any other platform than aarch64.

mono/mono#8648 indicates some problems with older acc versions. If you used a new toolchain for the mono 5.18 release you might want to try the previous one first.

might be a gcc regression.

if that doesn't make the source of the problem clear, I would build boringssl (used by mono for TLS) and use the included command line tools to try to reproduce this without mono to find out if it's a mono or boringssl bug.

[m4tt075]

Many thanks, @kaso17. Good to know (though not good) that the issue is apparently known. I've just cross-checked,the toolchains: The one we've used to build 5.18 is the same that we used for 5.8 at the time. No change in that regard. Looking through the thread you have referenced and references therein, the problem really appears to be a bit random, i.e. occurring with some mono version but not with others. I'm considering to build a couple of aarch64 builds of different mono versions to see whether we can at least "bisect" to some extent.

@bjorn1981 @silv3r23 What happens if you manually downgrade your mono installation to our previous 5.8 version? You can get it from here:
https://synocommunity.com/package/mono
Does it work then?

[CraigInBrisbane]

I have the same issue. As stated in the stack trace above. I'm using a Synology DS418J. It looks like the only way to get things working again, would be a downgrade. I'll try the link above - but it's happening on the Synology DS418J. Unfortunately, it looks like you need to uninstall Radarr and Sonarr (The two apps that use Mono) before you can downgrade. Actually, downgrade is forbidden. You need to uninstall Mono, and then do the manual install.

Edit: I'm unable to do this now, as when I try reinstalled Radarr (Had to uninstall to remove mono), it forces me to update to the latest mono version (after downgrading).

[m4tt075]

@CraigInBrisbane Right. Unfortunately, some of the newer *arr packages require newer mono versions, so just reverting the mono update is no option either.
Anyways, I think we have sufficient evidence that the aarch64 version of mono is broken for now. @ymartin59 For that reason, I've deactiveted it (just aarch64) on our repository.

For aarch64 users (@CraigInBrisbane et. al.), who updated already, downgrading mono is the only workaround for now. That requires to downgrade Radarr, too. On https://synocommunity.com/packages you can click on each package and get access to older package versions, which you can manually download and install. Sorry for the hassle.

My debugging capabilities are limited, but I'll see whether I can build other, newer mono versions for aarch64 that do not exhibit this problem. I'll post them here so that you can try. I don't have an aarch64 test platform myself.

[bjorn1981]

Thanks for looking into it. Downgrading was my first idea as well, but I did not have time to backup and downgrade Sonarr and Radarr.

I can test new releases when available.