[mono][aot] Loader tests failing on CI with Mono LLVM FullAOT

[ivanpovazan]

Seen when running runtime-extra-platforms on linux-x64 Release AllSubsets_Mono_LLVMFullAot_RuntimeTests llvmfullaot.
Loader tests failures detected on:

• Work item Loader.classloader fails with:

  Loader/classloader/InlineArray/InlineArrayInvalid/InlineArrayInvalid.sh [FAIL]
        
        Return code:      1
        Raw output file:      /datadisks/disk1/work/A06E08C1/w/AE0D0957/uploads/Reports/Loader.classloader/InlineArray/InlineArrayInvalid/InlineArrayInvalid.output.txt
        Raw output:
        BEGIN EXECUTION
        /datadisks/disk1/work/A06E08C1/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true InlineArrayInvalid.dll ''
        Explicit_Fails...
        Xunit.Sdk.ThrowsException: Assert.Throws() Failure
        Expected: typeof(System.TypeLoadException)
        Actual:   typeof(System.ExecutionEngineException): Attempting to JIT compile method 'void Validate/<>c:<Explicit_Fails>b__1_0 ()' while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information.
                  
           at Xunit.Assert.RecordException(Action testCode)
        Expected: 100
        Actual: 101
        END EXECUTION - FAILED
  

• Work item PayloadGroup0 fails with:

    Loader/StartupHooks/StartupHookTests/StartupHookTests.sh [FAIL]
        ...
        BEGIN EXECUTION
         /datadisks/disk1/work/A06E08C1/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true StartupHookTests.dll ''
         Running ValidHookName...
         -- Hello from startup hook with overload! Call count: 1
         Running ValidHookPath...
         Failed to load AOT module '/datadisks/disk1/work/A06E08C1/w/A66308F3/e/Loader/StartupHooks/StartupHookTests/private/PrivateInitialize.dll.so' ('/datadisks/disk1/work/A06E08C1/w/A66308F3/e/Loader/StartupHooks/StartupHookTests/private/PrivateInitialize.dll') in aot-only mode.
  

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=274202
Build error leg or test failing: Loader.classloader.WorkItemExecution
Pull request: #86266

Error Message

Fill the error message using known issues guidance.

{
  "ErrorPattern": "(Loader).+(InlineArrayInvalid|StartupHookTests)(.sh\\s\\[FAIL\\])",
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Test	Pull Request
290619	dotnet/runtime	Loader.classloader.WorkItemExecution	#86883
290636	dotnet/runtime	Loader.classloader.WorkItemExecution	#86690
290501	dotnet/runtime	Loader.classloader.WorkItemExecution
290401	dotnet/runtime	Loader.classloader.WorkItemExecution	#85558
290218	dotnet/runtime	Loader.classloader.WorkItemExecution
290207	dotnet/runtime	Loader.classloader.WorkItemExecution
289544	dotnet/runtime	Loader.classloader.WorkItemExecution
289319	dotnet/runtime	Loader.classloader.WorkItemExecution
288816	dotnet/runtime	Loader.classloader.WorkItemExecution
288696	dotnet/runtime	Loader.classloader.WorkItemExecution	#86854
288661	dotnet/runtime	Loader.classloader.WorkItemExecution
288473	dotnet/runtime	Loader.classloader.WorkItemExecution	#86842
288603	dotnet/runtime	PayloadGroup0.WorkItemExecution
288489	dotnet/runtime	Loader.classloader.WorkItemExecution
288379	dotnet/runtime	Loader.classloader.WorkItemExecution
288302	dotnet/runtime	PayloadGroup0.WorkItemExecution
288186	dotnet/runtime	Loader.classloader.WorkItemExecution
287916	dotnet/runtime	Loader.classloader.WorkItemExecution
287757	dotnet/runtime	Loader.classloader.WorkItemExecution	#85692
287085	dotnet/runtime	Loader.classloader.WorkItemExecution
286854	dotnet/runtime	Loader.classloader.WorkItemExecution
286685	dotnet/runtime	Loader.classloader.WorkItemExecution
286557	dotnet/runtime	Loader.classloader.WorkItemExecution
286466	dotnet/runtime	Loader.classloader.WorkItemExecution
285749	dotnet/runtime	Loader.classloader.WorkItemExecution
285617	dotnet/runtime	Loader.classloader.WorkItemExecution
285493	dotnet/runtime	Loader.classloader.WorkItemExecution
285027	dotnet/runtime	Loader.classloader.WorkItemExecution	#86690
285287	dotnet/runtime	Loader.classloader.WorkItemExecution
284500	dotnet/runtime	Loader.classloader.WorkItemExecution
283991	dotnet/runtime	Loader.classloader.WorkItemExecution
283137	dotnet/runtime	Loader.classloader.WorkItemExecution
283140	dotnet/runtime	Loader.classloader.WorkItemExecution	#86554
283011	dotnet/runtime	Loader.classloader.WorkItemExecution	#86581
282482	dotnet/runtime	Loader.classloader.WorkItemExecution
282672	dotnet/runtime	Loader.classloader.WorkItemExecution	#86581
281884	dotnet/runtime	Loader.classloader.WorkItemExecution	#86581
281696	dotnet/runtime	Loader.classloader.WorkItemExecution	#86342
281656	dotnet/runtime	Loader.classloader.WorkItemExecution
281352	dotnet/runtime	Loader.classloader.WorkItemExecution
281259	dotnet/runtime	Loader.classloader.WorkItemExecution	#86554
281245	dotnet/runtime	Loader.classloader.WorkItemExecution	#86554
281173	dotnet/runtime	Loader.classloader.WorkItemExecution	#85643
281185	dotnet/runtime	PayloadGroup0.WorkItemExecution
281107	dotnet/runtime	Loader.classloader.WorkItemExecution
280994	dotnet/runtime	Loader.classloader.WorkItemExecution
280892	dotnet/runtime	PayloadGroup0.WorkItemExecution
280763	dotnet/runtime	Loader.classloader.WorkItemExecution
280375	dotnet/runtime	Loader.classloader.WorkItemExecution
279332	dotnet/runtime	Loader.classloader.WorkItemExecution
278778	dotnet/runtime	Loader.classloader.WorkItemExecution
278375	dotnet/runtime	Loader.classloader.WorkItemExecution	#86194
278104	dotnet/runtime	Loader.classloader.WorkItemExecution	#86430
278002	dotnet/runtime	PayloadGroup0.WorkItemExecution
277389	dotnet/runtime	PayloadGroup0.WorkItemExecution
276852	dotnet/runtime	Loader.classloader.WorkItemExecution	#86269
276623	dotnet/runtime	PayloadGroup0.WorkItemExecution	#86375
276622	dotnet/runtime	PayloadGroup0.WorkItemExecution	#85965
276603	dotnet/runtime	PayloadGroup0.WorkItemExecution
276393	dotnet/runtime	PayloadGroup0.WorkItemExecution	#86366
275963	dotnet/runtime	PayloadGroup0.WorkItemExecution
275222	dotnet/runtime	PayloadGroup0.WorkItemExecution	#86307
274202	dotnet/runtime	PayloadGroup0.WorkItemExecution	#86266
275006	dotnet/runtime	PayloadGroup0.WorkItemExecution
274405	dotnet/runtime	PayloadGroup0.WorkItemExecution
274003	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.sh	#86097

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
7	30	66

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Seen when running runtime-extra-platforms on linux-x64 Release AllSubsets_Mono_LLVMFullAot_RuntimeTests llvmfullaot.
Loader tests failures detected on:

• Work item Loader.classloader
• Work item PayloadGroup0

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=274202
Build error leg or test failing: Loader.classloader.WorkItemExecution
Pull request: #86266

Error Message

Fill the error message using known issues guidance.

{
  "ErrorPattern": "(Loader).+(InlineArrayInvalid|StartupHookTests)(.sh\s\[FAIL\])",
  "ExcludeConsoleLog": false
}

Author:	ivanpovazan
Assignees:	-
Labels:	blocking-clean-ci, area-Infrastructure-mono, Known Build Error
Milestone:	8.0.0

[kotlarmilos]

@ivanpovazan I can take a look at Loader/classloader/InlineArray/InlineArrayInvalid.

[ivanpovazan]

Sure, thanks!
Although I am not sure if ErrorPattern pattern will work - if BuildAnalysis will pick it up, so I provided error message details for reference as well.

UPDATE: I fixed the ErrorPattern which is now properly tracking the failures

[akoeplinger]

I just fixed the StartupHookTests one with #86269

[kotlarmilos]

Initially, the InlineArrayInvalid test fails due to missing lambda generic methods for error handling that belong to a struct with invalid inline array custom attribute. Type load errors occur during class init in the mono_class_layout_fields and mono_class_setup_fields functions, preventing the AOT compilation of the generics below.

Unable to compile method 'void Validate/<>c:<ZeroLength_Fails>b__3_0 ()' due to: 'Inline array length property must be positive. 
Unable to compile method 'void Validate/<>c:<TooLarge_Fails>b__5_0 ()' due to: 'Inline array struct size out of bounds, abnormally large. 
Unable to compile method 'void Validate/<>c:<NegativeLength_Fails>b__7_0 ()' due to: 'Inline array length property must be positive. 
Unable to compile method 'void Validate/<>c:<NoFields_Fails>b__9_0 ()' due to: 'Inline array struct must have a single field. 
Unable to compile method 'void Validate/<>c:<TwoFields_Fails>b__11_0 ()' due to: 'Inline array struct must have a single field. 

To fix this issue, AOT compiler can avoid setting the mono_error_set_for_class_failure for structs with the inline array attribute in

runtime/src/mono/mono/metadata/class.c

Lines 2960 to 2963 in 4db4bb5

	if (m_class_has_failure (klass)) {
	mono_error_set_for_class_failure (error, klass);
	return NULL;
	}

and

runtime/src/mono/mono/metadata/class.c

Lines 4722 to 4725 in 4db4bb5

	if (mono_class_has_failure (klass)) {
	mono_error_set_for_class_failure (error, klass);
	return NULL;
	}

By making these changes, the errors will not be thrown during the AOT compilation.

In AOT runtime, when loading structs with invalid attribute, it should throw a type load error. Since mono_class_layout_fields and mono_class_setup_fields are not invoked during the runtime (except for domain objects), some of the checks can be moved to mono_class_create_from_typedef. However, in that case, there is an issue where the error is swallowed when searching for the trampoline gsharedvt_in for one of the lambda generic methods in

runtime/src/mono/mono/mini/aot-runtime.c

Lines 4487 to 4488 in 4db4bb5

	m = decode_resolve_method_ref_with_target (code_amodule, method, p, &p, error);
	mono_error_cleanup (error); /* FIXME don't swallow the error */

Even without error propagation, the runtime will throw an exception during type load for gsharedvt_in, which is just before the try/catch block within the generic methods void Validate/<>c:<T>b__1_0().

It could be that I missed something important. Any ideas on how to proceed? In general, the current approach with reporting issues during mono_class_layout_fields and mono_class_setup_fields could be challenging in AOT scenarios.

@vargaz @lambdageek It is possible that I've missed some important details, so it would be helpful to gather additional information. In general, the current approach of throwing exceptions during class init in mono_class_layout_fields and mono_class_setup_fields can be challenging in AOT scenarios, causing attempting to JIT compile method errors.

[lambdageek]

In general, the current approach with reporting issues during mono_class_layout_fields and mono_class_setup_fields could be challenging in AOT scenarios.

@kotlarmilos Here's an idea (I'm not confident yet - I'm just making stuff up)

We could add a bit to the MonoClass (and to the cached class info) that says "the AOT compiler noticed this class has a failure, but it was not fatal enough that AOT compilation had to be aborted". In that case the AOT cache loading function could ignore the cached info and let the runtime go on the slow path of trying to setup the class layout at runtime. At that point it will fail again and we can throw the proper TLE.

I'm assuming that if we just ignore the attribute during AOT compilation and pretend it's a single normal-sized field, we can continue with the AOT-time class setup enough to get somewhat usable code?

Maybe we can wrap the manipulation of the bit and setting with a nice wrapper. So instead of

			mono_class_set_type_load_failure (klass, "Inline array struct must have a single field.");

we can have

			mono_class_set_type_load_failure_aot_warning (klass, "Inline array struct must have a single field.");

which will:

1. if we're not in the AOT compiler, just call mono_class_set_type_load_failure
2. otherwise:
   1. set the new MonoClass:has_aot_recoverable_failure bit (or something shorter)
   2. print the message with g_warning
   3. return to the caller with the type otherwise unaltered.

[kotlarmilos]

We could add a bit to the MonoClass (and to the cached class info) that says "the AOT compiler noticed this class has a failure, but it was not fatal enough that AOT compilation had to be aborted". In that case the AOT cache loading function could ignore the cached info and let the runtime go on the slow path of trying to setup the class layout at runtime. At that point it will fail again and we can throw the proper TLE.

Thanks for the feedback, it looks like a good approach! I've created a PR #86554 that addresses the issue as described. In the AOT compilation, the function set_failure_type(DEFERRED_FAILURE) is called to set a failure callback that defers a failure for runtime execution. During JIT and AOT runtime, the function set_failure_type(IMMEDIATE_FAILURE) is called to set a failure callback which handles failures during the execution.

typedef gboolean (*TypeLoadFailureCallback)(MonoClass *klass, const char * fmt, ...);
TypeLoadFailureCallback type_load_failure_callback;

gboolean
mono_class_set_type_load_deferred_failure (MonoClass *klass, const char * fmt, ...)
{
	if (!mono_class_has_deferred_failure (klass)) {
		va_list args;

		va_start (args, fmt);
		g_warning ("Warning: %s", fmt, args);
		va_end (args);

		mono_class_set_deferred_failure (klass);
	}

	return FALSE;
}

gboolean
mono_class_set_type_load_failure (MonoClass *klass, const char * fmt, ...)
{
	// error handling

	return TRUE;
}

void set_failure_type(FailureType failure_type) {
	if (failure_type == DEFERRED_FAILURE) {
		type_load_failure_callback = mono_class_set_type_load_deferred_failure;
	} else if (failure_type == IMMEDIATE_FAILURE) {
		type_load_failure_callback = mono_class_set_type_load_failure;
	} else {
		g_assert_not_reached ();
	}
}

Such an approach allows for extensibility in similar scenarios where failures should be deferred.

I'm assuming that if we just ignore the attribute during AOT compilation and pretend it's a single normal-sized field, we can continue with the AOT-time class setup enough to get somewhat usable code?

Correct. However, the fix cannot be applied in cases where the AOT compiler optimizes unused code or adds const instructions.

For example, in the following code snippet the AOT compiler removes the body since it is not used.

Assert.Throws<TypeLoadException>(() => { var t = typeof(Explicit); });

Here, the AOT compiler adds struct size to the instruction during AOT compilation.

Assert.Throws<TypeLoadException>(() => { return sizeof(Explicit); });

To address these cases, I suggest adding an instruction (or using the existing one) that will force type loading instead of the actual instruction, ensuring that an exception is thrown during runtime, which would help ensure that the code is executed as intended. Do you think it is a good approach to handle these cases?

[vargaz]

Imho it would be easier to just disable these tests. Running apps with type load errors/invalid IL does not need to be supported in full aot mode.