Performance: Reflection in ResumableUpload and ParameterUtils causes intermittent long response delays.

[baal2000]

Issue

We found a performance problem in the Google.Apis library. It causes the application to "flap" (pause intermittently) whenever a standard Gen 2 Garbage Collection happens.

What we saw

Application pauses intermittently for 30s up to a minute at a time.

A 2s perf trace collected during one of the pauses shows the application mostly idle with the only activity in Finalizer dealing with System.Reflection.Emit.DynamicResolver+DestroyScout::Finalize() that waits on a lock (could be protecting the Code Heap, not sure)

We traced the source of these dynamic method stubs by dotnet-trace that collected the JIT activity and captured 2 sources with constant, repeating JIT activity:

1. Google.Apis.Upload.ResumableUpload.SetAllPropertyValues

2. Google.Apis.Requests.Parameters.ParameterUtils.IterateParameters

In both places the library duplicates the same logic that generates a brand new collection of PropertyInfo instances, enumerates, finds which property contains RequestParameterAttribute and then invokes PropertyInfo.GetValue(). This happens non-stop every time the library processes a request.

The Summary of the Sequence of Events:

1. New Objects: The library calls Type.GetProperties(). This creates a new array of PropertyInfo objects for every single request.
2. The Optimization: When PropertyInfo.GetValue() is called, the .NET runtime creates an IL "stub" (a small dynamic method) to read the property.
3. The Tracking: The runtime tracks these stubs using a Long Weak Handle in the native C++ code.
4. Orphaned PropertyInfo objects cleanup: The PropertyInfo objects are created new each time and the Garbage Collector (GC) cleans the orphans up during its regular cycle. When they are collected, that "Weak Handle" breaks.
5. The Lag: When the handle breaks, a specific finalizer called DestroyScout runs to clean up the native memory. Even under normal memory load, destroying hundreds of these stubs at once blocks the Finalizer thread and by extension the whole application process.

Environment details

• Programming language: C#
• OS: Ubuntu 22.04 LTS
• Language runtime version: .NET 8, .NET 9
• Package version: 1.72.0, 1.73.0, main dev branch

Proposed Solution

Create a PropertyInfo cache that persists for the application lifetime and reuse it in Google.Apis.Requests.Parameters.ParameterUtils and Google.Apis.Upload.ResumableUpload.

Example Fix Pattern:

// Shared Static cache
private static readonly ConcurrentDictionary<Type, PropertyInfo[]> PropertyCache = new();
...
var properties = PropertyCache.GetOrAdd(type, t => t.GetProperties(...)); // NEW
foreach (var property in properties)
{
    ...
}

Reproduction (A Simulation Test Case)

ReflectionJitActivityTest.cs
Test setup:

1. Test case 1. Run a loop that calls GetProperties() and GetValue() just like the Google library does.
2. Test case 2. Allocate a large managed Array and let the GC run naturally.
3. Test case 3. Use the proposed solution to protect PropertyInfos from garbage collection.

Test case 1.

• Infrequent JIT events beyond the initial spike at the start of the test (expected).
• No “dynamic” method names present in the names list

            JIT vs GC CORRELATION HISTOGRAM               
==========================================================================
[00:00-00:05] (  21) ++++
[00:05-00:10] (   0) 
...
[01:40-01:45] (   1) 
==========================================================================
                        JIT IDENTITY FORENSICS                            
==========================================================================
Dynamic JIT activities (all detected)
--------------------------------------------------------------------------
     None
--------------------------------------------------------------------------

Test case 2.

• From 60 second mark onwards the test allocates and abandons arrays, causing background GC Gen2.
  That is when multiple dynamic JIT events enter the picture: get_Data, get_Id, get_Name

            JIT vs GC CORRELATION HISTOGRAM               
==========================================================================
...
[01:05-01:10] (  28) +++++ [G2]
...
[01:55-02:00] (   4)  [G2]
==========================================================================
                        JIT IDENTITY FORENSICS                            
==========================================================================
Dynamic JIT activities (all detected)
--------------------------------------------------------------------------
    8 x  dynamicClass.InvokeStub_BaseRequest`1.get_Data
    8 x  dynamicClass.InvokeStub_BaseRequest`1.get_Id
    8 x  dynamicClass.InvokeStub_ConcreteRequest.get_Name
--------------------------------------------------------------------------

Test case 3.

• Enabled caching of PropertyInfo[] result = t.GetProperties() to prevent Reflection results from being collected. The test results are similar to the test case 1 with no dynamic JIT activity reported.

            JIT vs GC CORRELATION HISTOGRAM               
==========================================================================
[00:00-00:05] (  27) +++++
...
[01:00-01:05] (  52) ++++++++++ [G2]
...
[01:45-01:50] (   8) + [G2]
==========================================================================
                        JIT IDENTITY FORENSICS                            
==========================================================================
Dynamic JIT activities (all detected)
--------------------------------------------------------------------------
     None
--------------------------------------------------------------------------

[amanda-tarafa]

Thanks for the detailed report. We'll take a look as soon as possible.