gzip module failing to decompress valid compressed file

[EricG]

BPO	24301
Nosy	@ned-deily, @vadmium, @iritkatriel, @rhpvorderman
PRs	bpo-24301: Do not fail when gzip files have trailing garbage #29847

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-05-27.15:59:03.363>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'gzip module failing to decompress valid compressed file'
updated_at = <Date 2021-12-31.09:44:00.395>
user = 'https://bugs.python.org/Ericg'

bugs.python.org fields:

activity = <Date 2021-12-31.09:44:00.395>
actor = 'rhpvorderman'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2015-05-27.15:59:03.363>
creator = 'Ericg'
dependencies = []
files = []
hgrepos = []
issue_num = 24301
keywords = ['patch']
message_count = 9.0
messages = ['244188', '244214', '244230', '245368', '245369', '407148', '407280', '407282', '409410']
nosy_count = 6.0
nosy_names = ['ned.deily', 'martin.panter', 'Ericg', 'nczeczulin', 'iritkatriel', 'rhpvorderman']
pr_nums = ['29847']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24301'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[EricG]

I have a file whose first four bytes are 1F 8B 08 00 and if I use gunzip from the command line, it outputs:

gzip: zImage_extracted.gz: decompression OK, trailing garbage ignored

and correctly decompresses the file. However, if I use the gzip module to read and decompress the data, I get the following exception thrown:

File "/usr/lib/python3.4/gzip.py", line 360, in read
while self._read(readsize):
File "/usr/lib/python3.4/gzip.py", line 433, in _read
if not self._read_gzip_header():
File "/usr/lib/python3.4/gzip.py", line 297, in _read_gzip_header
raise OSError('Not a gzipped file')

I believe the problem I am facing is the same one described here in this SO question and answer:

http://stackoverflow.com/questions/4928560/how-can-i-work-with-gzip-files-which-contain-extra-data

This would appear to be serious bug in the gzip module that needs to be fixed.

[ned-deily]

Can you add a public copy of a file that fails this way? There are several open issues with gzip, like bpo-1159051, that might cover this but it's hard to know for sure without a test case.

[vadmium]

I suspect Eric’s file has non-zero, non-gzip garbage bytes appended to the end of it. Assuming I am right, here is way to reproduce that scenario:

>>> from gzip import GzipFile
>>> from io import BytesIO
>>> file = BytesIO()
>>> with GzipFile(fileobj=file, mode="wb") as z:
...     z.write(b"data")
... 
4
>>> file.write(b"garbage")
7
>>> file.seek(0)
0
>>> GzipFile(fileobj=file).read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/proj/python/cpython/Lib/gzip.py", line 274, in read
    return self._buffer.read(size)
  File "/home/proj/python/cpython/Lib/gzip.py", line 461, in read
    if not self._read_gzip_header():
  File "/home/proj/python/cpython/Lib/gzip.py", line 409, in _read_gzip_header
    raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b'ga')

This is a bit different to bpo-1508475. That one is about cases where the “gzip” trailer has been truncated, although the compressed data is probably intact. This case is the converse: extra data has been added.

All of the “gzip”, “bzip2” and XZ Utils (for LZMA) command-line decompressors happily extract the compressed data without an error exit status, but emit warning messages:

gzip: stdin: decompression OK, trailing garbage ignored
bzip2: (stdin): trailing garbage after EOF ignored
xz: (stdin): Unexpected end of input

In Python, the “bzip” and LZMA modules successfully extract the compressed data, and ignore the non-compressed garbage at the end without even a warning. On the other hand, the “gzip” module has special code to ignore trailing zero bytes (bpo-2846), but treats any other trailing non-gzip data as an error.

So I think a strong argument could be made for the ability to extract all the compressed data from even if there is garbage appended. The question is, how would this support be added? Perhaps the mechanism chosen could also be integrated with a fix for bpo-1508475. Some options:

• Silently ignore the condition by default like the other compression modules (consistent, but could silently swallow real errors)
• An optional new GzipFile(strict=False) mode
• Perhaps an exception deferred until close() is called

[nczeczulin]

The spec allows for multi-member files. Some libraries and utilities seem to solve this problem (incorrectly?) by simply ignoring everything past the first member -- even when valid (e.g., DotNetZip, 7-Zip)

For 2.7 and 3.4, the data that has been decompressed but not yet read before the exception was raised is still available:

Modifying Martin's example slightly:

>>> f = BytesIO()
>>> with GzipFile(fileobj=f, mode="wb") as z:
...     z.write(b"data")
...
4
>>> f.write(b"garbage")
7
>>> f.seek(0)
0
>>> with GzipFile(fileobj=f, mode="rb") as z:
...     try:
...         z.read(1)
...         z.read()
...     except OSError as e:
...         z.extrabuf[z.offset - z.extrastart:]
...         e
...
b'd'
b'ata'
OSError('Not a gzipped file',)

My issue is that catching and handling this specific exception is a little more involved because there are 3(?) different OSErrors (IOError on 2.7) that could potentially be raised during the read. But mostly:
OSError('CRC check failed 0x447ba3f9 != 0x225cb2a3',) -- would be bad one to mistake for it.

Maybe a specific Exception type to catch for an invalid header, and a better method to read the remaining buffer when handling it?

[vadmium]

Just noticed in my previous message I mentioned bpo-1508475 a few times when I meant to say bpo-1159051.

In Python 3.5, a workaround is not so easy because we would need to access the internal buffer of a BufferedReader. One potential workaround is to use read1():

>>> z.read1(1)
b'd'
>>> z.read1()
b'ata'
>>> z.read1()
OSError: Not a gzipped file (b'ga')

The only practical way to allow for an exception and read() returning all the data is to defer the exception until close() is called. Another option might be to store a list of defects, similar to “email.message.Message.defects”.

[iritkatriel]

Reproduced on 3.11:

>>> from gzip import GzipFile
>>> from io import BytesIO
>>> file = BytesIO()
>>> with GzipFile(fileobj=file, mode="wb") as z:
...     z.write(b"data")
... 
4
>>> file.write(b"garbage")
7
>>> file.seek(0)
0
>>> GzipFile(fileobj=file).read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/iritkatriel/src/cpython-654/Lib/gzip.py", line 301, in read
    return self._buffer.read(size)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-654/Lib/_compression.py", line 118, in readall
    while data := self.read(sys.maxsize):
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-654/Lib/gzip.py", line 499, in read
    if not self._read_gzip_header():
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-654/Lib/gzip.py", line 468, in _read_gzip_header
    last_mtime = _read_gzip_header(self._fp)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iritkatriel/src/cpython-654/Lib/gzip.py", line 428, in _read_gzip_header
    raise BadGzipFile('Not a gzipped file (%r)' % magic)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gzip.BadGzipFile: Not a gzipped file (b'ga')