Start a new harvester when file was truncated #1882
Conversation
| @@ -74,7 +74,7 @@ func (h *Harvester) Harvest() { | |||
|
|
|||
| h.SetOffset(0) | |||
| seeker.Seek(h.getOffset(), os.SEEK_SET) | |||
| continue | |||
| return | |||
There was a problem hiding this comment.
The complete if err == errFileTruncate can be removed. Where/how does prospector detect the file being truncated? Still h.SetOffset(0) required?
There was a problem hiding this comment.
It is the harvester that detects that the file was truncated, not prospector. The prospector only sees a finished harvester and opens a new one if the offset (which is 0).
There was a problem hiding this comment.
In this case it's still too much code. There is no need to call seeker.Seek.
There was a problem hiding this comment.
Yep, will remove it.
Currently file truncation was handled by the reader. In case of truncation, the same harvester stayed open and the reader just continued reading from the beginning. This is the most efficient way to make sure the truncated file is immediately read. From a logical point of view, truncation of a file is very similar to creating a new file. Instead of letting the reader just continue, the harvester is closed and a completely new harvester is started with a new reader. This means the reader only needs to detect a truncation, but doesnt' have to decide what happens. The consequence of this change is that it can take up to `scan_frequency` to read the truncated file, but that is the same behaviour as for a new file. In addition the file handler will be closed when the old harvester is stopped.
ruflin
force-pushed
the
refactor-truncated
branch
from
f5ff342 to
c13f555
Compare
| logp.Err("can not seek source") | ||
| return | ||
| } | ||
|
|
||
| logp.Info("File was truncated. Begin reading file from offset 0: %s", h.Path) |
There was a problem hiding this comment.
Is info message enough? There is a chance we did loose lines.
There was a problem hiding this comment.
That is a good point. We should make a WARN out of this and add a comment that this means that not all content was read potentially. WDYT?
But I suggest to make this in a second PR, ok?
There was a problem hiding this comment.
@urso after our conversation yesterday about warn messages, I will leave it as info, ok?
There was a problem hiding this comment.
Didn't we remove the requirement to seek most recently?
There was a problem hiding this comment.
Yes, in this PR. But the logging that the file was truncated is still needed.
Currently file truncation was handeld by the reader. In case of truncation, the same harvester stayed open and the reader just continued reading from the beginning. This is the most efficient way to make sure the truncated file is immidiately read.
From a logical point of view, truncation of a file is very similar to creating a new file. Instead of letting the reader just continue, the harvester is closed and a completely new harvester is started with a new reader. This means the reader only needs to detect a truncation, but doesnt' have to decide what happens.
The consequence of this change is that it can take up to
scan_frequencyto read the truncated file, but that is the same behaviour as for a new file. In addition the file handler will be closed when the old harvester is stopped.