InvalidCastException thrown after TimerQueueTimer.CallCallback

[tipa]

Description

After upgrading my iOS app from classic Xamarin.iOS to .NET7 I started observing frequent InvalidCastExeptions in my crash reporting tool (AppCenter). I haven't changed much dependency-wise and I am unable to reproduce this locally.

Reproduction Steps

Unfortunately, I don't know what part of my code causes this behavior.
For the first stack trace, I assume that my code (or some dependencies code) is creating a CancellationTokenSource with a delay parameter.
For the second stack trace below, I assume that some call to Task.Delay is causing the crash.

Expected behavior

No crash :)

Actual behavior

SIGABRT: Arg_InvalidCastException

System.Threading.CancellationTokenSource.TimerCallback(Object )
System.Threading.TimerQueueTimer.CallCallback(Boolean )
System.Threading.TimerQueueTimer.Fire(Boolean )
System.Threading.TimerQueue.FireNextTimers()
System.Threading.TimerQueue.System.Threading.IThreadPoolWorkItem.Execute()
System.Threading.ThreadPoolWorkQueue.DispatchItemWithAutoreleasePool(Object , Thread )

this is here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs

Line 35 in 215b39a

((CancellationTokenSource)state!).NotifyCancellation(throwOnFirstException: false); // skip ThrowIfDisposed() check in Cancel()

SIGABRT: Arg_InvalidCastException

System.Threading.Tasks.Task.DelayPromise.TimerCallback(Object )
System.Threading.TimerQueueTimer.CallCallback(Boolean )
System.Threading.TimerQueueTimer.Fire(Boolean )
System.Threading.TimerQueue.FireNextTimers()
System.Threading.TimerQueue.System.Threading.IThreadPoolWorkItem.Execute()
System.Threading.ThreadPoolWorkQueue.DispatchItemWithAutoreleasePool(Object , Thread )

this is here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs

Line 5632 in 26b58b9

private static void TimerCallback(object? state) => ((DelayPromise)state!).CompleteTimedOut();

Both crashes originate from here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs

Line 706 in 26b58b9

_timerCallback(_state);

The "_state" variable is nullable, and based on the reports, the crashes are caused by the variable to be indeed null.

Regression?

I did not see those crashes when using "old" Xamarin.iOS

Known Workarounds

No response

Configuration

.net7-ios
happens on various iPhone models and iOS versions

Other information

No response

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

After upgrading my iOS app from classic Xamarin.iOS to .NET7 I started observing frequent InvalidCastExeptions in my crash reporting tool (AppCenter). I haven't changed much dependency-wise and I am unable to reproduce this locally.

Reproduction Steps

Unfortunately, I don't know what part of my code causes this behavior.
For the first stack trace, I assume that my code (or some dependencies code) is creating a CancellationTokenSource with a delay parameter.
For the second stack trace below, I assume that some call to Task.Delay is causing the crash.

Expected behavior

No crash :)

Actual behavior

System.Threading.CancellationTokenSource.TimerCallback(Object )
System.Threading.TimerQueueTimer.CallCallback(Boolean )
System.Threading.TimerQueueTimer.Fire(Boolean )
System.Threading.TimerQueue.FireNextTimers()
System.Threading.TimerQueue.System.Threading.IThreadPoolWorkItem.Execute()
System.Threading.ThreadPoolWorkQueue.DispatchItemWithAutoreleasePool(Object , Thread )

this is here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs

Line 35 in 215b39a

((CancellationTokenSource)state!).NotifyCancellation(throwOnFirstException: false); // skip ThrowIfDisposed() check in Cancel()

System.Threading.Tasks.Task.DelayPromise.TimerCallback(Object )
System.Threading.TimerQueueTimer.CallCallback(Boolean )
System.Threading.TimerQueueTimer.Fire(Boolean )
System.Threading.TimerQueue.FireNextTimers()
System.Threading.TimerQueue.System.Threading.IThreadPoolWorkItem.Execute()
System.Threading.ThreadPoolWorkQueue.DispatchItemWithAutoreleasePool(Object , Thread )

this is here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs

Line 5632 in 26b58b9

private static void TimerCallback(object? state) => ((DelayPromise)state!).CompleteTimedOut();

Both crashes originate from here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs

Line 706 in 26b58b9

_timerCallback(_state);

The "_state" variable is nullable, and based on the reports, the crashes are caused by the variable to be indeed null.

Regression?

I did not see those crashes when using "old" Xamarin.iOS

Known Workarounds

No response

Configuration

.net7-ios
happens on various iPhone models and iOS versions

Other information

No response

Author:	tipa
Assignees:	-
Labels:	area-System.Threading, untriaged
Milestone:	-

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

After upgrading my iOS app from classic Xamarin.iOS to .NET7 I started observing frequent InvalidCastExeptions in my crash reporting tool (AppCenter). I haven't changed much dependency-wise and I am unable to reproduce this locally.

Reproduction Steps

Unfortunately, I don't know what part of my code causes this behavior.
For the first stack trace, I assume that my code (or some dependencies code) is creating a CancellationTokenSource with a delay parameter.
For the second stack trace below, I assume that some call to Task.Delay is causing the crash.

Expected behavior

No crash :)

Actual behavior

SIGABRT: Arg_InvalidCastException

System.Threading.CancellationTokenSource.TimerCallback(Object )
System.Threading.TimerQueueTimer.CallCallback(Boolean )
System.Threading.TimerQueueTimer.Fire(Boolean )
System.Threading.TimerQueue.FireNextTimers()
System.Threading.TimerQueue.System.Threading.IThreadPoolWorkItem.Execute()
System.Threading.ThreadPoolWorkQueue.DispatchItemWithAutoreleasePool(Object , Thread )

this is here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs

Line 35 in 215b39a

((CancellationTokenSource)state!).NotifyCancellation(throwOnFirstException: false); // skip ThrowIfDisposed() check in Cancel()

SIGABRT: Arg_InvalidCastException

System.Threading.Tasks.Task.DelayPromise.TimerCallback(Object )
System.Threading.TimerQueueTimer.CallCallback(Boolean )
System.Threading.TimerQueueTimer.Fire(Boolean )
System.Threading.TimerQueue.FireNextTimers()
System.Threading.TimerQueue.System.Threading.IThreadPoolWorkItem.Execute()
System.Threading.ThreadPoolWorkQueue.DispatchItemWithAutoreleasePool(Object , Thread )

this is here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs

Line 5632 in 26b58b9

private static void TimerCallback(object? state) => ((DelayPromise)state!).CompleteTimedOut();

Both crashes originate from here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs

Line 706 in 26b58b9

_timerCallback(_state);

The "_state" variable is nullable, and based on the reports, the crashes are caused by the variable to be indeed null.

Regression?

I did not see those crashes when using "old" Xamarin.iOS

Known Workarounds

No response

Configuration

.net7-ios
happens on various iPhone models and iOS versions

Other information

No response

Author:	tipa
Assignees:	-
Labels:	area-System.Threading, untriaged, area-Threading-mono, os-ios
Milestone:	-

[akoeplinger]

/cc @lambdageek

[lambdageek]

Weird. Both CancellationTokenSource and DelayTimer pass this for the state argument of TimerQueueTimer.
So the state passed to the callback should be non-NULL, and the cast from object to the respective types should succeed.

Maybe there's a GC hole somewhere?

[lambdageek]

@tipa when you try to reproduce the issue locally, are you trying it in a simulator or on a device?

[tipa]

@lambdageek I tested my apps on both simulator and physical device numerous times - I was never able to observe this myself (edit: I now am, using a TestFlight app and repeatedly starting it).
The crash is not super-common, but also not super-rare. I updated both my apps ~1 week ago.
Since then, one of the two apps that I am observing this is installed on ~5000 devices with around 2000 app sessions per day and the crash occurred 7 times (on 7 different devices).
And the other app is installed on ~1000 devices with around 300 app sessions per day and the crash occurred 3 times (on 2 different devices).

[tipa]

I am also seeing a lot of these exceptions (220 crashes from 80 devices , all originating from DelayPromise) in an app that does a lot of (concurrent) network calls. Maybe the HttpClient uses this under the hood...

[tipa]

A lot more crash reports have been accumulated in the meantime. I expect the real number to be even higher because AppCenter only seems to report a fraction of the real crashes.
I was also able to reproduce this now in a TestFlight app. it appears to happen when the app is of spawning a few (~10) new tasks and letting them do some concurrent work (getting credentials from the key chain, doing HTTP requests, deserialize JSON (using System.Text.Json), etc).
The other app I mentioned above (now 322 crashes in the last 30 days) does something similar.

I observed a crash with a similar behavior on Android (see here + fix here), maybe this is from a similar cause?

Additionally, I also started seeing these stack traces::

SIGABRT: Object reference not set to an instance of an object

System.AggregateException..ctor(String , List`1 )
System.Threading.Tasks.TaskExceptionHolder.Finalize()

SIGABRT: Arg_ArrayTypeMismatchException

System.AggregateException..ctor(String , List`1 )
System.Threading.Tasks.TaskExceptionHolder.Finalize()

Maybe I should have opened this bug in the xamarin-macios repo?

[steveisok]

Maybe I should have opened this bug in the xamarin-macios repo?

No, it does look like a runtime issue, so you have the right place.