Skip to content

Add zip bomb tests #1300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 85 commits into from
Apr 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
0b089ef
WIP
Mar 2, 2023
e4e513b
remove max size from NewZstdCompressor
Mar 2, 2023
ef84d36
WIP support multiple compression types
Mar 2, 2023
4a659b0
rename compressionType to Type
Mar 2, 2023
c6de927
fix metric
Mar 2, 2023
13eb890
WIP remove CompressionEnabled and add --network-compression-type
Mar 2, 2023
bc55ea4
rename types
Mar 2, 2023
400a918
add zstd compression/decompression metrics
Mar 3, 2023
6f8bd3e
Merge remote-tracking branch 'origin/dev' into add-zstd-compression
Mar 3, 2023
38e7a12
don't allow 2 network compression flags
Mar 3, 2023
3850c79
remove benchmark
Mar 3, 2023
6065be8
cleanup
Mar 3, 2023
d7c5d52
don't use zstd until v1.10
Mar 7, 2023
385ba27
tweak error message
Mar 7, 2023
37bcf39
nit
Mar 7, 2023
81afb71
nits
Mar 7, 2023
4da5f21
nits
Mar 7, 2023
d8a9ccf
Merge remote-tracking branch 'origin/dev' into add-zstd-compression
Mar 8, 2023
d9b778c
flag wording nit
Mar 8, 2023
41bc6c5
add zstd tests; fix bugs
Mar 8, 2023
11c171d
consolidate metrics
Mar 8, 2023
ddd35ee
remove old todo
Mar 8, 2023
6dd3f25
update test
Mar 8, 2023
eab4879
update tests
Mar 8, 2023
1ad19c1
add tests
Mar 8, 2023
6b29cfc
appease linter
Mar 8, 2023
7c150cd
Merge branch 'dev' into add-zstd-compression
Mar 13, 2023
874632a
Merge remote-tracking branch 'origin/dev' into add-zstd-compression
Mar 15, 2023
a9a96f6
Merge branch 'add-zstd-compression' of github.com:ava-labs/avalancheg…
Mar 15, 2023
9526ba9
nits
Mar 15, 2023
e1f0d55
Merge branch 'dev' into add-zstd-compression
Mar 15, 2023
c000a83
Merge branch 'dev' into add-zstd-compression
Mar 21, 2023
483f040
use default compression type in tests
Mar 21, 2023
041c37e
Merge branch 'add-zstd-compression' of github.com:ava-labs/avalancheg…
Mar 21, 2023
6b3f9a3
flag nit
Mar 21, 2023
5325c6b
only allow zstd after cortina
Mar 21, 2023
4345b1e
Merge remote-tracking branch 'upstream/dev' into add-zstd-compression
Mar 31, 2023
51d3cbb
merged
StephenButtolph Apr 2, 2023
9dc7108
Merge remote-tracking branch 'upstream/dev' into add-zstd-compression
Apr 3, 2023
b401ea6
address PR comments
Apr 3, 2023
b0225da
remove switch
Apr 3, 2023
45d9b3f
Merge branch 'add-zstd-compression' of github.com:ava-labs/avalancheg…
Apr 3, 2023
8472760
add max message size test
Apr 3, 2023
0e61411
add max message size to zstd
Apr 3, 2023
32d8d37
add test
Apr 3, 2023
72a0b7c
Merge remote-tracking branch 'upstream/dev' into add-zstd-compression
Apr 3, 2023
db6504a
remove switch
Apr 4, 2023
53df4c6
use stream interface for Decompress to avoid unzip bomb
Apr 4, 2023
adbc82a
fix copy pasta bug
Apr 4, 2023
e510398
move switch case to default
Apr 4, 2023
82d7f68
remove impossible switch case
Apr 4, 2023
bdcf189
appease linter
Apr 4, 2023
b91159d
make reader a local var
Apr 4, 2023
c259199
return nit
Apr 4, 2023
9105741
add invalid max size check to zstd compressor creation
Apr 4, 2023
8ad8fe6
Merge branch 'dev' into add-zstd-compression
StephenButtolph Apr 4, 2023
eaec3ef
Parallelize gzip compression
StephenButtolph Apr 4, 2023
c6c6fdf
merged
StephenButtolph Apr 4, 2023
c6f4800
log warning for unknown op during metrics observation
Apr 5, 2023
8ad1cb6
use sync.Pool of gzip writer
Apr 5, 2023
8c499af
gzipWriter.Flush() --> gzipWriter.Close()
Apr 5, 2023
812a37d
test cleanup
Apr 5, 2023
084be58
Merge remote-tracking branch 'upstream/add-zstd-compression' into sim…
Apr 5, 2023
a0c1dda
appease linter
Apr 5, 2023
c1d7a98
Merge remote-tracking branch 'upstream/add-zstd-compression' into sim…
Apr 5, 2023
26e7235
remove magic number
Apr 5, 2023
f3f7785
Merge remote-tracking branch 'upstream/add-zstd-compression' into sim…
Apr 5, 2023
a2cc314
Merge branch 'dev' into add-zstd-compression
StephenButtolph Apr 5, 2023
436685b
imports nit
StephenButtolph Apr 5, 2023
d30e0a9
Merge remote-tracking branch 'upstream/add-zstd-compression' into sim…
Apr 5, 2023
b40e14b
nit
StephenButtolph Apr 5, 2023
0c0a396
test nits
Apr 5, 2023
422a840
Merge remote-tracking branch 'upstream/add-zstd-compression' into sim…
Apr 5, 2023
0fca08a
nits
StephenButtolph Apr 5, 2023
dfc909d
nit
StephenButtolph Apr 5, 2023
781023e
Merge branch 'add-zstd-compression' into simplify-gzip-compression
StephenButtolph Apr 5, 2023
e769d97
Add zipbomb tests
StephenButtolph Apr 5, 2023
c2fe2e4
nit
StephenButtolph Apr 5, 2023
e550864
merged
StephenButtolph Apr 6, 2023
dda9596
Embed the zip bombs
StephenButtolph Apr 6, 2023
65ec891
nits
StephenButtolph Apr 6, 2023
7afff07
Merge branch 'simplify-gzip-compression' into add-zip-bomb-tests
StephenButtolph Apr 6, 2023
0f43f37
Merge branch 'dev' into simplify-gzip-compression
StephenButtolph Apr 6, 2023
a074713
Merge branch 'simplify-gzip-compression' into add-zip-bomb-tests
StephenButtolph Apr 6, 2023
37ed76c
Merge branch 'dev' into add-zip-bomb-tests
StephenButtolph Apr 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 51 additions & 6 deletions utils/compression/compressor_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,11 @@ package compression
import (
"fmt"
"math"
"runtime"
"testing"

_ "embed"

"github.com/stretchr/testify/require"

"github.com/ava-labs/avalanchego/utils"
Expand All @@ -16,12 +19,54 @@ import (

const maxMessageSize = 2 * units.MiB // Max message size. Can't import due to cycle.

var newCompressorFuncs = map[Type]func(maxSize int64) (Compressor, error){
TypeNone: func(int64) (Compressor, error) { //nolint:unparam // an error is needed to be returned to compile
return NewNoCompressor(), nil
},
TypeGzip: NewGzipCompressor,
TypeZstd: NewZstdCompressor,
var (
newCompressorFuncs = map[Type]func(maxSize int64) (Compressor, error){
TypeNone: func(int64) (Compressor, error) { //nolint:unparam // an error is needed to be returned to compile
return NewNoCompressor(), nil
},
TypeGzip: NewGzipCompressor,
TypeZstd: NewZstdCompressor,
}

//go:embed gzip_zip_bomb.bin
gzipZipBomb []byte

//go:embed zstd_zip_bomb.bin
zstdZipBomb []byte

zipBombs = map[Type][]byte{
TypeGzip: gzipZipBomb,
TypeZstd: zstdZipBomb,
}
)

func TestDecompressZipBombs(t *testing.T) {
for compressionType, zipBomb := range zipBombs {
// Make sure that the hardcoded zip bomb would be a valid message.
require.Less(t, len(zipBomb), maxMessageSize)

newCompressorFunc := newCompressorFuncs[compressionType]

t.Run(compressionType.String(), func(t *testing.T) {
compressor, err := newCompressorFunc(maxMessageSize)
require.NoError(t, err)

var (
beforeDecompressionStats runtime.MemStats
afterDecompressionStats runtime.MemStats
)
runtime.ReadMemStats(&beforeDecompressionStats)
_, err = compressor.Decompress(zipBomb)
runtime.ReadMemStats(&afterDecompressionStats)

require.ErrorIs(t, err, ErrDecompressedMsgTooLarge)

// Make sure that we didn't allocate significantly more memory than
// the max message size.
bytesAllocatedDuringDecompression := afterDecompressionStats.TotalAlloc - beforeDecompressionStats.TotalAlloc
require.Less(t, bytesAllocatedDuringDecompression, uint64(10*maxMessageSize))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kinda interesting that you have to make this at least 6 to pass 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just because of how the underlying bytes buffer grows afaict.

})
}
}

func TestCompressDecompress(t *testing.T) {
Expand Down
Binary file added utils/compression/gzip_zip_bomb.bin
Binary file not shown.
Binary file added utils/compression/zstd_zip_bomb.bin
Binary file not shown.