-
Notifications
You must be signed in to change notification settings - Fork 838
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add WebSockets telemetry #1237
Add WebSockets telemetry #1237
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like the right structure. Testing is going to be interesting.
Work with the customer to decide the right field types for the telemetry output.
src/ReverseProxy/WebSocketsTelemetry/WebSocketsTelemetryMiddleware.cs
Outdated
Show resolved
Hide resolved
src/ReverseProxy/WebSocketsTelemetry/WebSocketsTelemetryMiddleware.cs
Outdated
Show resolved
Hide resolved
samples/ReverseProxy.Metrics.Sample/WebSocketsTelemetryConsumer.cs
Outdated
Show resolved
Hide resolved
samples/ReverseProxy.Metrics.Sample/WebSocketsTelemetryConsumer.cs
Outdated
Show resolved
Hide resolved
src/ReverseProxy.TelemetryConsumption/TelemetryConsumptionExtensions.cs
Outdated
Show resolved
Hide resolved
var clientCloseTime = _readParser.CloseTime; | ||
var serverCloseTime = _writeParser.CloseTime; | ||
|
||
if (clientCloseTime.HasValue && (!serverCloseTime.HasValue || serverCloseTime.Value > clientCloseTime.Value)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The close is only graceful if both sides acknowledge it. Otherwise you can attribute the reason as a disconnect by the unacknowledged side.
if (clientCloseTime.HasValue && (!serverCloseTime.HasValue || serverCloseTime.Value > clientCloseTime.Value)) | |
if (clientCloseTime.HasValue && serverCloseTime.HasValue && serverCloseTime.Value > clientCloseTime.Value) |
https://datatracker.ietf.org/doc/html/rfc6455#section-7.1.2
Once an endpoint has both sent and
received a Close control frame, that endpoint SHOULD Close the
WebSocket Connection as defined in
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So something like
if (clientSentClose && serverSentClose)
{
return clientCloseTime < serverCloseTime ? ClientGraceful : ServerGraceful;
}
else
{
Disconnect
}
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if (clientSentClose && serverSentClose)
{
return clientCloseTime < serverCloseTime ? ClientGraceful : ServerGraceful;
}
else if (clientSentClose)
{
ServerDisconnect
}
else if (serverSentClose)
{
ClientDisconnect
}
else // check forwarder errors
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the client sends a Close message and immediately closes the connection (without waiting for the server to respond), we would be reporting that as a ServerDisconnect (and vice-versa).
Would that be the desired behavior?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting... how would you detect that? Would the read return 0? Writes might fail, depending on the timing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the logic so that it is now
if (clientSentClose && serverSentClose)
{
return clientCloseTime < serverCloseTime ? ClientGraceful : ServerGraceful;
}
// One side sent a WebSocket close, but we never saw a response from the other side
// It is possible an error occurred, but we saw a graceful close first, so that is the intiator
if (clientSentClose)
{
return ClientGraceful;
}
if (serverSentClose)
{
return ServerGraceful;
}
return forwarderError ...
While "graceful" doesn't necessarily mean "both sides sent close", I believe reporting what likely initiated the close is more useful as the WebSocketCloseReason
.
Either side may still perform protocol violations, but it's not our aim to catch that.
src/ReverseProxy.TelemetryConsumption/WebSockets/WebSocketCloseReason.cs
Outdated
Show resolved
Hide resolved
ff9a28c
to
c11308f
Compare
@@ -576,7 +576,7 @@ private static void RestoreUpgradeHeaders(HttpContext context, HttpResponseMessa | |||
var (secondResult, secondException) = await secondTask; | |||
if (secondResult != StreamCopyResult.Success) | |||
{ | |||
error = ReportResult(context, requestFinishedFirst, secondResult, secondException!); | |||
error = ReportResult(context, !requestFinishedFirst, secondResult, secondException!); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This appears to be a bug where if the request copy finishes first successfully, we will attribute potential response errors to the request side.
@MihaZupan Are you waiting for anything from anyone on this PR? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was expecting some unit tests for the WebSocketsParser?
src/ReverseProxy/WebSocketsTelemetry/WebSocketsTelemetryStream.cs
Outdated
Show resolved
Hide resolved
Added more tests |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have you tried running this in the sample with real websocket traffic?
src/ReverseProxy/WebSocketsTelemetry/WebSocketsTelemetryMiddleware.cs
Outdated
Show resolved
Hide resolved
test/ReverseProxy.Tests/WebSocketsTelemetry/WebSocketsParserTests.cs
Outdated
Show resolved
Hide resolved
Yes, I tried it with a simple demo app and the default Blazor template, and it's working as expected. |
Fixes #1209
I followed the approach outlined in #1209 of wrapping the upgrade feature and MITM the upgrade transport stream.
Public API is just the
UseWebSocketsTelemetry
extension method.I fed the data to an
EventSource
event, but it could also alternatively be something likeIWebSocketsTelemetryFeature
.I didn't know where to put the code so I made a
WebSocketsTelemetry
folder in the main project (all types are internal though).ToDo: I will add tests when we're happy with the overall approach