fix: Fix flush/close semantics for HTTP files, improve testing #1232
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Jul 13, 2023
All HTTP-based tests now use an embedded test server instead of httpbin.org, which makes them much faster and more reliable. These more reliable tests also exposed some issues that began recently with PR shaka-project#1201. HttpFile's Flush() semantics were different than those documented for files in general. Flush() used to close the file for uploading, so that no further writes were allowed, but the documentation stated that it would only flush data to its destination. PR shaka-project#1201 brought HttpFile's Flush() in line with the docs, but gave us no way to terminate a chunked upload. This adds a new method to File called CloseForWriting(), which terminates a chunked upload for HttpFile. The only other implementation that does anything is UdpFile, which uses the socket library function shutdown() to terminate writes while allowing reads. This also tweaks HttpFile::CloseWithStatus() so that it will not generate an error if the file is closed before the HTTP response is written to the download cache. This modifies the test HttpFileTest.MultipleWrites so that the file is Flushed after each chunk. This adds test coverage for the changes introduced in PR shaka-project#1201. Fixes shaka-project#1224 (missing test coverage for HttpFile::Flush)
joeyparrish
force-pushed
the
independent-http-testing
branch
from
a584a0e to
2f67ac5
Compare
kqyang
approved these changes
Jul 13, 2023
packager/file/http_file_unittest.cc
Outdated
| @@ -165,21 +214,35 @@ TEST(HttpFileTest, MultipleWrites) { | |||
|
|
|||
| ASSERT_EQ(file->Write(data1.data(), data1.size()), | |||
| static_cast<int64_t>(data1.size())); | |||
| ASSERT_TRUE(file->Flush()); | |||
There was a problem hiding this comment.
Why do we call Flush after every write in this test? Is it needed after every write?
There was a problem hiding this comment.
It adds coverage for the change in Flush() semantics from #1201. Now that Flush() only flushes, this shows the new behavior. Before #1201, you could not Write() after Flush().
In the test server, you can see those pieces written as separate chunks now. This isn't used yet, but it may be an important detail for chunked uploads for LL streaming in the future.
There was a problem hiding this comment.
I see. It is probably better to add another MultipleWritesWithFlush to test this case. Otherwise consecutive Writes without Flush is not tested.
github-actions
bot
added
the
status: archived
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
All HTTP-based tests now use an embedded test server instead of
httpbin.org, which makes them much faster and more reliable.
These more reliable tests also exposed some issues that began recently
with PR #1201. HttpFile's Flush() semantics were different than those
documented for files in general. Flush() used to close the file for
uploading, so that no further writes were allowed, but the documentation
stated that it would only flush data to its destination. PR #1201
brought HttpFile's Flush() in line with the docs, but gave us no way to
terminate a chunked upload.
This adds a new method to File called CloseForWriting(), which
terminates a chunked upload for HttpFile. The only other implementation
that does anything is UdpFile, which uses the socket library function
shutdown() to terminate writes while allowing reads.
This also tweaks HttpFile::CloseWithStatus() so that it will not
generate an error if the file is closed before the HTTP response is
written to the download cache.
This modifies the test HttpFileTest.MultipleWrites so that the file is
Flushed after each chunk. This adds test coverage for the changes
introduced in PR #1201.
Fixes #1224 (missing test coverage for HttpFile::Flush)