Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor zstd decoder #498

Merged
merged 35 commits into from
Feb 24, 2022
Merged

Refactor zstd decoder #498

Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
1a14c30
Refactor zstd decoder
klauspost Jan 18, 2022
b5ab519
Merge branch 'master' into refactor-zstd-decoder
klauspost Feb 9, 2022
984fc8f
Make it compile
klauspost Feb 11, 2022
dca58e6
Fix up single decodes.
klauspost Feb 11, 2022
f01cd33
Merge branch 'master' into refactor-zstd-decoder
klauspost Feb 14, 2022
3c54049
Almost working now...
klauspost Feb 16, 2022
8bf62b7
Tests pass.
klauspost Feb 18, 2022
ac20ec2
Avoid a few allocs
klauspost Feb 18, 2022
9a9ac6b
Add stream decompression with no goroutines.
klauspost Feb 18, 2022
b2951ef
Check FrameContentSize and max decoded size.
klauspost Feb 18, 2022
9d525e5
Remove unused var+func.
klauspost Feb 18, 2022
b74ecce
Tweaks and cleanup
klauspost Feb 18, 2022
e217e78
Use maxsize as documented.
klauspost Feb 19, 2022
6598a07
Merge branch 'master' into refactor-zstd-decoder
klauspost Feb 21, 2022
561b94c
Ensure history from frames cannot overlap.
klauspost Feb 21, 2022
3db2dcb
Fix deadlock on error.
klauspost Feb 21, 2022
55e4c4b
Stricter framecontent size checks and consistency.
klauspost Feb 21, 2022
d6e790a
Fix short test.
klauspost Feb 21, 2022
8b43a92
Add bench
klauspost Feb 22, 2022
d0ce155
Merge branch 'master' into refactor-zstd-decoder
klauspost Feb 22, 2022
50a135c
Reject big RLE/RAW blocks as per https://github.com/facebook/zstd/iss…
klauspost Feb 22, 2022
d731396
Use os.FileInfo for Go 1.15.
klauspost Feb 22, 2022
03b301a
Break all on errors
klauspost Feb 22, 2022
4c5a306
Move sync code to separate method.
klauspost Feb 22, 2022
d3974d3
Don't read sent error.
klauspost Feb 22, 2022
133d52c
Fix consistent error reporting and dict inits.
klauspost Feb 22, 2022
2b89e67
Fix error message
klauspost Feb 22, 2022
606373a
Check if huff0 X4 blocks match size exactly.
klauspost Feb 23, 2022
a197f86
Fix decoder leakage.
klauspost Feb 23, 2022
d1f91e2
Forward blocks, so we don't risk run-away decoding.
klauspost Feb 23, 2022
e7906eb
Fix test race
klauspost Feb 23, 2022
7a09f3d
Save last before sending.
klauspost Feb 23, 2022
f5e0961
Reuse history between async calls.
klauspost Feb 23, 2022
6cabc28
Protect local frame.
klauspost Feb 23, 2022
f20d563
Clarify error msg
klauspost Feb 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Break all on errors
  • Loading branch information
@klauspost
klauspost committed Feb 22, 2022
commit 03b301af11042d71164a1fd620f60ca0eec6b8e1
6 changes: 3 additions & 3 deletions zstd/decoder.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -843,7 +843,7 @@ decodeStream:
}
break decodeStream
}
decodeFrame:

// Go through all blocks of the frame.
for {
var dec *blockDec
Expand Down Expand Up @@ -884,10 +884,10 @@ decodeStream:
}
select {
case <-ctx.Done():
break decodeFrame
break decodeStream
case seqPrepare <- dec:
}
if err != nil {
if dec.err != nil {
break decodeStream
}
if dec.Last {
Expand Down