Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve error reporting on oversized messages #572

Merged
merged 5 commits into from
Nov 20, 2019

Conversation

eddyashton
Copy link
Member

tlsframedendpoint imposes a maximum message size of 2MB, but previously did so by closing connections with no logging. I've tried to improve this by logging a failure when we close these connections, logging more clearly when we're ignoring messages for closed connections, and trying to send a reasonable message back over the connection before we close it.

I made the e2e_logging tests deliberately increase message sizes until it hit this error, but have removed as this didn't work - the error is reported for seqno 0 so we need to fetch from the stream manually, and still we sometimes get a closed session without any preceding error message.

@eddyashton eddyashton requested a review from a team as a code owner November 19, 2019 16:28
@@ -31,7 +31,7 @@ namespace asynchost
friend class close_ptr<TCPImpl>;

static constexpr int backlog = 128;
static constexpr size_t read_size = 1024;
static constexpr size_t max_read_size = 16384;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reduces the output spam for large messages (reduces the number of reads by a factor of 16 so we), and should also improve performance for them. But this may be at the cost of performance for smaller messages, and was a deadend in tracking down the actual failure so should perhaps be removed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be a comment in the code as well?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like this doesn't actually have a noticeable impact on performance either way, so I think we can just ignore this for now (and hopefully remember it as a potentially bad choice if it comes back to bite us in future?).

I think it should actually be possible to just keep a single buffer in this object, rather than reallocating on every request, but I'll also leave that investigation on the backburner atm.

@cimetrics
Copy link

images

@eddyashton
Copy link
Member Author

I haven't reproduced this yet, but we can also hit this size limit for node-to-node communications via RPCClient. I'm not sure what the fix is there, perhaps the RPCClient endpoint shouldn't have a maximum message size?

@codecov-io
Copy link

codecov-io commented Nov 19, 2019

Codecov Report

Merging #572 into master will decrease coverage by 0.22%.
The diff coverage is 22.22%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #572      +/-   ##
==========================================
- Coverage   78.56%   78.34%   -0.22%     
==========================================
  Files         140      140              
  Lines       10567    10599      +32     
==========================================
+ Hits         8301     8303       +2     
- Misses       2266     2296      +30
Flag Coverage Δ
#e2e_BFT 50.77% <22.22%> (-0.34%) ⬇️
#e2e_CFT 73.18% <22.22%> (-0.24%) ⬇️
#unit_BFT 65.33% <0%> (-0.12%) ⬇️
#unit_CFT 72.65% <0%> (-0.1%) ⬇️
Impacted Files Coverage Δ
src/enclave/rpcendpoint.h 62.16% <0%> (-9.71%) ⬇️
src/host/tcp.h 73.19% <100%> (-0.74%) ⬇️
src/enclave/tlsendpoint.h 57.83% <12.5%> (-1.63%) ⬇️
src/enclave/tlsframedendpoint.h 66% <21.43%> (-17.78%) ⬇️

@cimetrics
Copy link

images

@eddyashton eddyashton merged commit 4195cd3 into microsoft:master Nov 20, 2019
@eddyashton eddyashton deleted the ERR_TOO_BIG branch November 20, 2019 15:21
eddyashton added a commit to eddyashton/CCF that referenced this pull request Mar 24, 2020
* Add errors for oversized messages

* Add trace of message sizes

* Restore previous max msg size

* Increase max TCP read size
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants