Request body too large error from onefuzzlib/Queue.cs when processing crash logs in v8.0.0

[puhley]

Information

• Onefuzz version: 8.0.0
• OS: Linux

Provide detailed reproduction steps (if any)

This issue is very similar to #2708 . However, the patch from 6.4.0 didn't catch this variant. It is the same general error as 2708:

The request body is too large and exceeds the maximum permissible limit.
RequestId:38799644-e103-002b-80a8-12345e000000
Time:2023-04-24T12:12:12.4036764Z
Status: 413 (The request body is too large and exceeds the maximum permissible limit.)
ErrorCode: RequestBodyTooLarge

Additional Information:
MaxLimit: 131584

Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit.
RequestId:38799644-e103-002b-80a8-12345e000000
Time:2023-04-24T12:12:212.4036764Z</Message><MaxLimit>131584</MaxLimit></Error>

The messages being sent are of the type:

{"target":"events","arguments":[{"event_id":"a3907a31-10aa-915e-b123-8a2684c32cd3","event_type":"regression_reported","event":{"regression_report":{"crash_test_result":{},"original_crash_test_result":{"crash_report":{"input_blob":{"account":"fuzzabf6fyns77abc","container":"oft-crashes-7c0ea4c8d123712e31e72eba08b7e36e","name":"leak-ab6e19768b89ec0de6e18266aa912af5aadce496"},"executable":"foo-asan","crash_type":"5632","crash_site":"AddressSanitizer: 5234 byte(s) leaked in .... .","call_stack":["#0 0x1fc497 in posix_memalign (...

This is the specific stack trace associated with the error. The last line of the call stack provides the location in OneFuzz where it is crashing:

parsedStack: 
[
{"assembly":"Azure.Storage.Queues, Version=12.11.0.0, Culture=neutral, PublicKeyToken=91232123e12e11c8","method":"Azure.Storage.Queues.MessagesRestClient+<EnqueueAsync>d__14.MoveNext","level":0,"line":0},
{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7abc23d7bef2218e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":1,"line":0},
{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7abc23d7bef2218e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":2,"line":0},
{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7abc23d7bef2218e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":3,"line":0},
{"assembly":"Azure.Storage.Queues, Version=12.11.0.0, Culture=neutral, PublicKeyToken=91232123e12e11c8","method":"Azure.Storage.Queues.QueueClient+<SendMessageInternal>d__82.MoveNext","level":4,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7abc23d7bef2218e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":5,"line":0},
{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7abc23d7bef2218e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":6,"line":0},
{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7abc23d7bef2218e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":7,"line":0},
{"assembly":"Azure.Storage.Queues, Version=12.11.0.0, Culture=neutral, PublicKeyToken=91232123e12e11c8","method":"Azure.Storage.Queues.QueueClient+<SendMessageAsync>d__80.MoveNext","level":8,"line":0},
{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7abc23d7bef2218e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":9,"line":0},
{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7abc23d7bef2218e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":10,"line":0},
{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7abc23d7bef2218e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":11,"line":0},
{"assembly":"ApiService, Version=8.0.0.0, Culture=neutral, PublicKeyToken=null","method":"Microsoft.OneFuzz.Service.Queue+<SendMessage>d__5.MoveNext","level":12,"line":52,"fileName":"/home/runner/work/onefuzz/onefuzz/src/ApiService/ApiService/onefuzzlib/Queue.cs"}
]

Some other details from the error messages that may be useful:

CalledBy: Microsoft.OneFuzz.Service.Queue+<SendMessage>d__5
_MS.ProcessedByMetricExtractors: (Name:'Exceptions', Ver:'1.1')
AppVersion: 8.0.0.0
Operation name: QueueFileChanges
SDKVersion: azurefunctions-netiso: 1.0.0.0

Expected result

Messages should either be shortened or broken up to fit into the queue.

Actual result

Messages are dropped due to exceeding the size limitations of the queue.

AB#140774

[Porges]

@tevoinea is looking at a general solution for this

[puhley]

Thanks for the update. I'll keep an eye out for the changes.

[tevoinea]

@puhley I'll have a patch for this in our next release.

The ultimate general solution that George was talking about will involve a breaking change in a future release. The contents of the event payload will be scoped down (so that it's impossible for the data to be too large for Azure Queue) and the consumer of the events will have to download the event payload out of band using a SAS URL.