Experimental.System.Net.Sockets: "socket connect" marked Error for a non-blocking Connect that succeeds

[tulior]

Description

Every synchronous physical connection Npgsql opens on our Linux pods produces a "socket connect" activity with Status=Error and error.type=_OTHER, even though the connection succeeds. TLS and the Postgres protocol proceed on that same socket. No matching success activity, ever. The minimal repro below reproduces it on Windows as well, so this isn't platform-specific.

The cause is the standard sync connect-with-timeout idiom: non-blocking socket, call Connect, catch the expected WouldBlock/InProgress SocketException, finish via Socket.Select (npgsql raises this exception on every open, see npgsql/npgsql#1183; the Select frame is visible in npgsql/npgsql#881, and #15820 discusses the same pattern). SocketsTelemetry.AfterConnect treats any non-Success SocketError as a failure and stops the activity with Error status, and GetErrorType has no entry for WouldBlock, so the tag falls through to _OTHER:

runtime/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketsTelemetry.cs

Lines 148 to 161 in ad50b41

	Debug.Assert(newCount >= 0);
	if (activity is not null)
	{
	if (error != SocketError.Success)
	{
	activity.SetStatus(ActivityStatusCode.Error);
	activity.SetTag("error.type", GetErrorType(error));
	}
	activity.Stop();
	}
	if (error == SocketError.Success)

runtime/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketsTelemetry.cs

Lines 236 to 259 in ad50b41

	// Common connect() errors expected to be seen:
	// https://learn.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-connect#return-value
	// https://man7.org/linux/man-pages/man2/connect.2.html
	SocketError.NetworkDown => "network_down",
	SocketError.AddressAlreadyInUse => "address_already_in_use",
	SocketError.Interrupted => "interrupted",
	SocketError.InProgress => "in_progress",
	SocketError.AlreadyInProgress => "already_in_progress",
	SocketError.AddressNotAvailable => "address_not_available",
	SocketError.AddressFamilyNotSupported => "address_family_not_supported",
	SocketError.ConnectionRefused => "connection_refused",
	SocketError.Fault => "fault",
	SocketError.InvalidArgument => "invalid_argument",
	SocketError.IsConnected => "is_connected",
	SocketError.NetworkUnreachable => "network_unreachable",
	SocketError.HostUnreachable => "host_unreachable",
	SocketError.NoBufferSpaceAvailable => "no_buffer_space_available",
	SocketError.TimedOut => "timed_out",
	SocketError.AccessDenied => "access_denied",
	SocketError.ProtocolType => "protocol_type",
	_ => "_OTHER"
	};

In APM these look like real database connect failures. They cost us a few days of chasing network ghosts (DNS, SNAT) before we read the instrumentation source.

Reproduction Steps

using System.Diagnostics;
using System.Net;
using System.Net.Sockets;

var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();

var stopped = new List();
ActivitySource.AddActivityListener(new ActivityListener
{
    ShouldListenTo = s => s.Name == "Experimental.System.Net.Sockets",
    Sample = (ref ActivityCreationOptions _) => ActivitySamplingResult.AllDataAndRecorded,
    ActivityStopped = stopped.Add
});

var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp) { Blocking = false };
try
{
    socket.Connect((IPEndPoint)listener.LocalEndpoint);
}
catch (SocketException e) when (e.SocketErrorCode is SocketError.WouldBlock or SocketError.InProgress)
{
    // expected for a non-blocking connect
}
socket.Poll(5_000_000, SelectMode.SelectWrite);

var a = stopped.Single();
Console.WriteLine($"connected={socket.Connected} status={a.Status} error.type={a.GetTagItem("error.type")} duration={a.Duration.TotalMicroseconds}us");

Plain net10.0 console app, no package references. Output (win-x64):

connected=True status=Error error.type=_OTHER duration=5524.1us

The duration here includes first-socket-use overhead in a cold process; in production these activities run ~50-80us. On Windows, WouldBlock is the documented result of a non-blocking connect, which matches the _OTHER bucket (WouldBlock has no GetErrorType mapping). I haven't run this standalone repro on Linux, but our production spans (linux-x64) carry error.type=_OTHER too, which rules out InProgress there (that one maps to "in_progress").

Expected behavior

I'd expect no connect activity at all for a non-blocking socket: WouldBlock is the documented "in progress" contract, not a failure, and the eventual completion isn't observable at this API layer anyway. If there's a reason to keep the activity, it shouldn't carry Error status. At minimum, WouldBlock deserves a GetErrorType mapping so it's distinguishable from genuinely unknown failures. If you settle on a direction I can send a PR; the change looks contained to AfterConnect/GetErrorType plus tests.

Actual behavior

The activity stops with Status=Error and error.type=_OTHER while the socket is connected and usable. Production spans run roughly 50-80us; the cold repro above shows ~5.5ms. One error activity per sync open, no success counterpart.

Regression?

No. The source is new in 9.0; we observe this on 10.0 and the mapping is unchanged in main as of ad50b41. I didn't test 9.0 explicitly.

Known Workarounds

Don't subscribe to the experimental source, or drop the spans in the tracing backend (we exclude on otel.library.name + network.peer.port + error.type in Datadog).

Configuration

Repro environment:

.NET SDK: 10.0.204
Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.26200
 OS Platform: Windows
 RID:         win-x64

Host:
 Version:      10.0.8
 Architecture: x64

Originally observed in production: .NET 10 on linux-x64 (Azure Container Apps), Npgsql 10.0.1, OpenTelemetry SDK 1.14, exported via OTLP to Datadog.

Other information

No response

[MihuBot]

cc: @dotnet/ncl