-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
buffer flush failed by unexpected Zlib::DataError #1903
Comments
This is very difficult problem because zlib doesn't tell detailed information. This is drawback of zlib... I will check your configuration and test traffic on my environment. |
Hi, i get this error, any suggestions? @repeatedly |
Hi, i get this error, any suggestions? @repeatedly
|
@nurse Do you have any insight to debug zlib error? |
I use @google238 config can reproduce problems. @repeatedly can try it. |
|
@repeatedly @nurse this may because https://github.com/ruby/zlib/blob/e529c315e1b568535b2e9439f82249cf4cf6c6ae/ext/zlib/zlib.c#L1025
Fluentd |
Hmm, the code you pointed out seems actually wrong... |
@nurse Do you know how this happened? |
@jl2005 If signal is received or the thread is killed, zstream_unblock_func will be called. |
I tried the scenario but it doesn't reproduce Zlib::DataError on
|
@nurse send signal or kill thread is I change
When I run this code:
I got :
|
The weird point is "retry is succeeded." |
zstream_unblock_fun will only be called when an event occurs. So try again to succeed. This event may be triggered on the on_notify event. |
Timer thread calls
|
@nurse How to fix this bug? |
@nurse @repeatedly |
In fluentd output, "Thread#kill" is not called until stop/restart. |
Bingo! |
@nurse What should we do? |
… ref #1903 Direct append causes broken gzipped file when Zlib::DataError happens. This patch uses Tempfile for gzip compression to avoid broken compression. Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Temporal fix for this: #1995 |
@repeatedly When can you really fix it. |
…gzip-append out_file: Temporal fix for broken gzipped files with gzip and append. ref #1903
@repeatedly
Any suggestions would be helpful. Also, the log says retry succeeded, does it mean that the retry created a new file with the same data? |
@hlakshmi yes and latest version should not have broken problem with |
v1.3 should resolve this problem. If you hit same problem with v1.3 or later, reopen the issue. |
facing same issue with 1.10.4
@repeatedly Shall we reopen the issue. |
No. This issue is for zlib library and thread interruption problem. |
@repeatedly |
fluted v1.0,this is my configure rule:
there are so many warn log like this:
2018-03-19 00:18:51 +0900 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2018-03-19 00:18:52 +0900 chunk="567b15a13fd0f003c72c7e0e534dbf3e" error_class=Zlib::DataError error="data error"
2018-03-19 00:18:51 +0900 [warn]: #0 suppressed same stacktrace
2018-03-19 00:18:52 +0900 [warn]: #0 retry succeeded. chunk_id="567b15a13fd0f003c72c7e0e534dbf3e"
the result file cannot open correctly:
gzip: invalid compressed data--crc error
gzip: invalid compressed data--length error
what happend??
The text was updated successfully, but these errors were encountered: