Skip to content

Dead lettering tests #101524

Closed as not planned
Closed as not planned
@carlossanlop

Description

@carlossanlop

Build Information

Build: https://dev.azure.com/dnceng-public/public/public%20Team/_build/results?buildId=654910
Build error leg or test failing: browser-wasm windows Release LibraryTests_Smoke_AOT

Error Message

{
  "ErrorMessage" : "If you’re reading this, that means the Helix work item you’re trying to find the logs for has dead-lettered.",
  "BuildRetry" : false,
  "ExcludeConsoleLog" : false
}
If you’re reading this, that means the Helix work item you’re trying to find the logs for has dead-lettered.

What this means:

- All attempts to retry execution of this work item were unable to complete.  This can be both for infrastructure reasons (problems within Azure) or issues with the work item (for instance, causing a machine to reboot unexpectedly or killing the Helix client on the machine will force a retry).
- No further work will be done for this specific work item, and its exit code is set to an artificial -1 (since it did not complete, there is no real exit code).

Common causes:

- Disabled queue (end-of-life Helix queues are automatically forwarded to deadletter and will fail instantly)
- Unhealthy Helix Client machine(s)
- Queue has been backed up heavily by a large amount of work and was manually purged by the engineering team
- Azure issues (e.g. Service Bus is overloaded)
- Malformed payloads; if Helix cannot download and unzip all payloads successfully, work will retry until dead-lettered.

For follow up:

- Check if your Helix Queue is still enabled, either via the metadata you see by browsing to https://helix.dot.net/api/info/queues?api-version=2019-06-17 or recent emails from the .NET Engineering Infrastructure team.
- Check that all work item payloads are accessible using a browser.
- If you are sending to a non-disabled queue and find this error repeatedly occurring, please contact the dnceng team.
- If a single, specific work item dead letters and others do not, consider local debugging; it may be causing spontaneous reboot (or trigging one intentionally).

Report

Build Definition Test Pull Request
710640 dotnet/runtime System.Reflection.Tests.WorkItemExecution #103595
710441 dotnet/runtime Workloads-Wasm.Build.NativeRebuild.Tests.OptimizationFlagChangeTests.WorkItemExecution
710444 dotnet/runtime System.Threading.Tasks.Parallel.Tests.WorkItemExecution
710229 dotnet/runtime System.Security.Cryptography.Csp.Tests.WorkItemExecution #103560
709965 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
709966 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
709954 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
709957 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
709335 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
709334 dotnet/runtime System.Formats.Cbor.Tests.WorkItemExecution
709332 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
709333 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
709068 dotnet/runtime System.Threading.Tasks.Dataflow.Tests.WorkItemExecution #100041
708944 dotnet/runtime System.Runtime.Tests.WorkItemExecution #103416
708854 dotnet/runtime System.Diagnostics.TraceSource.Tests.WorkItemExecution
708846 dotnet/runtime System.Diagnostics.TraceSource.Tests.WorkItemExecution
708853 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
708851 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
708463 dotnet/runtime System.Globalization.CalendarsWithConfigSwitch.Tests.WorkItemExecution
708201 dotnet/runtime System.Diagnostics.TraceSource.Tests.WorkItemExecution
708194 dotnet/runtime JIT.Generics.WorkItemExecution #103484
708055 dotnet/runtime System.ComponentModel.TypeConverter.Tests.WorkItemExecution #102927
708065 dotnet/runtime System.Linq.Queryable.Tests.WorkItemExecution #103479
708015 dotnet/runtime System.Runtime.Serialization.BinaryFormat.Tests.WorkItemExecution #103370
708020 dotnet/runtime System.IO.Tests.WorkItemExecution #103337
708039 dotnet/runtime System.CodeDom.Tests.WorkItemExecution #103456
707938 dotnet/runtime Workloads-Wasm.Build.Tests.SatelliteAssembliesTests.WorkItemExecution #98494
706459 dotnet/runtime iOS.Simulator.LibraryMode.Test.WorkItemExecution #102748
707775 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
707765 dotnet/runtime System.Formats.Cbor.Tests.WorkItemExecution
707769 dotnet/runtime System.Diagnostics.TraceSource.Config.Tests.WorkItemExecution
707764 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
703066 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102670
707322 dotnet/runtime System.IO.Tests.WorkItemExecution #103442
707058 dotnet/runtime System.Diagnostics.Tools.Tests.WorkItemExecution
707052 dotnet/runtime Regression_3.WorkItemExecution
707057 dotnet/runtime System.Runtime.Extensions.Tests.WorkItemExecution
706980 dotnet/runtime System.IO.Tests.WorkItemExecution
706568 dotnet/runtime System.Composition.Convention.Tests.WorkItemExecution #103411
706525 dotnet/runtime System.Diagnostics.Tracing.Tests.WorkItemExecution
706528 dotnet/runtime System.Formats.Cbor.Tests.WorkItemExecution
706532 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
706522 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
706429 dotnet/runtime tvOS.Device.Aot.Test.WorkItemExecution #103232
706163 dotnet/runtime Regression_3.WorkItemExecution
706180 dotnet/runtime System.Diagnostics.TextWriterTraceListener.Tests.WorkItemExecution #103393
706187 dotnet/runtime System.Collections.Specialized.Tests.WorkItemExecution #103364
706167 dotnet/runtime System.Diagnostics.FileVersionInfo.Tests.WorkItemExecution
705894 dotnet/runtime Microsoft.Extensions.Diagnostics.Tests.WorkItemExecution #103379
705827 dotnet/runtime System.IO.Tests.WorkItemExecution #103366
2472316 dotnet-runtime System.Dynamic.Runtime.Tests.WorkItemExecution #40292
2471911 dotnet-runtime release.Partition0.WorkItemExecution
2472143 dotnet-runtime release.Partition0.WorkItemExecution
705295 dotnet/runtime System.IO.Tests.WorkItemExecution
2471843 dotnet-runtime System.Formats.Cbor.Tests.WorkItemExecution #40292
705034 dotnet/runtime System.Formats.Cbor.Tests.WorkItemExecution
705035 dotnet/runtime System.Diagnostics.Tracing.Tests.WorkItemExecution
705022 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
705026 dotnet/runtime System.Formats.Cbor.Tests.WorkItemExecution
704278 dotnet/runtime System.Collections.Specialized.Tests.WorkItemExecution #99747
704203 dotnet/runtime Regression_3.WorkItemExecution
704267 dotnet/runtime System.Globalization.Tests.WorkItemExecution
704117 dotnet/runtime System.Data.Odbc.Tests.WorkItemExecution
703927 dotnet/runtime System.Diagnostics.Tools.Tests.WorkItemExecution
703923 dotnet/runtime Regression_3.WorkItemExecution
703814 dotnet/runtime System.Runtime.Serialization.Schema.Tests.WorkItemExecution #103283
703591 dotnet/runtime System.Diagnostics.TraceSource.Tests.WorkItemExecution
703592 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
703588 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
703582 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
703614 dotnet/runtime Invariant.Tests.WorkItemExecution #100030
703283 dotnet/runtime Regression_3.WorkItemExecution
703219 dotnet/runtime iOS.Simulator.LibraryMode.Test.WorkItemExecution
703291 dotnet/runtime System.IO.Tests.WorkItemExecution #103266
703287 dotnet/runtime System.Diagnostics.DiagnosticSource.Switches.Tests.WorkItemExecution
703222 dotnet/runtime System.Globalization.Calendars.Tests.WorkItemExecution
703069 dotnet/runtime JIT_Math.WorkItemExecution
703205 dotnet/runtime System.Formats.Cbor.Tests.WorkItemExecution
703121 dotnet/runtime Regression_3.WorkItemExecution
703077 dotnet/runtime System.Globalization.Tests.WorkItemExecution
702874 dotnet/runtime iOS.Simulator.LibraryMode.Test.WorkItemExecution
702836 dotnet/runtime System.Console.Tests.WorkItemExecution
702918 dotnet/runtime Regression_3.WorkItemExecution
703011 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
702973 dotnet/runtime System.IO.Compression.Tests.WorkItemExecution
702832 dotnet/runtime Regression_3.WorkItemExecution
702967 dotnet/runtime System.IO.Compression.Tests.WorkItemExecution
702934 dotnet/runtime System.Collections.Specialized.Tests.WorkItemExecution #103138
702711 dotnet/runtime Regression_3.WorkItemExecution
702715 dotnet/runtime System.Console.Tests.WorkItemExecution
2470077 dotnet-runtime System.Formats.Tar.Manual.Tests.WorkItemExecution #40222
702615 dotnet/runtime System.Diagnostics.TextWriterTraceListener.Tests.WorkItemExecution #103159
2470076 dotnet-runtime System.IO.FileSystem.Manual.Tests.WorkItemExecution #40221
702578 dotnet/runtime tvOS.Device.Aot.Test.WorkItemExecution #103226
702185 dotnet/runtime System.Formats.Cbor.Tests.WorkItemExecution
702184 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
702186 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
702182 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
701895 dotnet/runtime ComInterfaceGenerator.Unit.Tests.WorkItemExecution
701694 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
Displaying 100 of 338 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
8 77 338

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions