Skip to content

Commit

Permalink
httpfs: Strenghten code to avoid corruptions with servers sending mor…
Browse files Browse the repository at this point in the history
…e data than asked

Fixes duckdb#8930 and duckdb#8908 (avoiding the fatal crash).

HTTP server are allowed to send back more data than asked for, and be somehow unreliably.
In this case a GET request with a range headers gets more data than asked for, and with very
big files we ended up overriding the current buffer.

This is an uncommon but legal behaviour, to properly handle this situation so we will have
to improve logic elsewhere to properly handle this case.
  • Loading branch information
carlopi committed Sep 15, 2023
1 parent ba71015 commit a189055
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions extension/httpfs/httpfs.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -329,6 +329,14 @@ unique_ptr<ResponseWrapper> HTTPFileSystem::GetRangeRequest(FileHandle &handle,
hfs.state->total_bytes_received += data_length;
}
if (buffer_out != nullptr) {
if (data_length + out_offset > buffer_out_len) {
// As of v0.8.2-dev4424 we might end up here when very big files are served from servers
// that returns more data than requested via range header. This is an uncommon but legal
// behaviour, so we have to improve logic elsewhere to properly handle this case.

// To avoid corruption of memory, we bail out.
return false;
}
memcpy(buffer_out + out_offset, data, data_length);
out_offset += data_length;
}
Expand Down

0 comments on commit a189055

Please sign in to comment.