compressed log file can't be decompressed with error "invalid compressed data--format violated"

[David-Gee]

Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.

• fluentd or td-agent version.
  td-agent-3.2.1-0.el6.x86_64
  gzip-1.3.12-24.el6.x86_64
• Environment information:
  • CentOS release 6.10 (Final)
  • 2.6.32-754.3.5.el6.x86_64
• Your configuration

   <match seiya2cn.log.**>
    @type copy
    <store>
      @type file
      path /data/from_infra/10000001/from_log/%Y%m%d/10000001_%Y%m%d
      append true
      **compress gzip**
      <format>
	@type json
      </format>
    </store>
    <store>
        @type kafka2
        brokers 10.96.29.136:19092,10.96.29.137:19092,10.96.29.138:19092
        default_topic 10000001_game_log
        <format>
          @type json
        </format>
        <buffer topic>
          flush_interval 1s
        </buffer>
    </store>
</match>

• Your problem explanation. If you have an error logs, write it together.
  In this config file, I use "compress gzip" to compress my log, but when I want to decompress my log with command "gunzip 10000001_20190101.log.gz" I always get a error:
  "gzip: 10000001_20190101.log.gz: invalid compressed data--format violated"

Could you please help to check this issue ?

[repeatedly]

I tested your configuration and it worked with small events.

% ls -al
drwxr-xr-x    6 repeatedly  staff   204B  2 27 02:29 ./
drwxr-xr-x  372 repeatedly  staff    12K  2 27 01:26 ../
drwxr-xr-x    2 repeatedly  staff    68B  2 27 01:26 10000001_%Y%m%d/
-rw-r--r--    1 repeatedly  staff    54B  2 27 01:26 10000001_20190227.log
-rw-r--r--    1 repeatedly  staff   114B  2 27 01:26 10000001_20190227.log.gz
% cat 10000001_20190227.log
{"message":"hey"}
{"message":"hey"}
{"message":"hey"}

% gunzip --version
Apple gzip 251

Could you show reproducible events and message rate?

[ganmacs]

No response from the author for a long time. I'm closing. feel free to ask us if you have additional info

[singhravi1]

Hi Guys,

I'm getting the same issue alongwith the issue mentioned here.
I recently updated the logger to log a dictionary with keys and values converted to strings. Never encoutered this error before this. I'm using python fluentd logger.
Could that tell something related to the issue?
Please let me know if you need any other information.