Implementation of fileProspector #21479
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
botelastic
bot
added
the
needs_team
kvch
added
the
Team:Services
|
Pinging @elastic/integrations-services (Team:Services) |
botelastic
bot
removed
the
needs_team
❕ Build Aborted
Expand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
faec
approved these changes
Oct 2, 2020
| m.Lock() | ||
| for path, src := range m.files { | ||
| isSame, err := isSameFile(path, src.info) | ||
| if m.closeBecauseRemoved(path, err) || m.closeBecauseRenamed(path, isSame) { |
There was a problem hiding this comment.
I'm trying to understand closeBecauseRenamed... elsewhere, the way the filestream input detects if something is renamed is if os.SameFile returns true for a different path than the original, indicating that the file has been moved to that new path. Here though, it seems to instead check whether os.SameFile returns false for the same path as before. To me all that indicates is that something else is where that file used to be, and isn't enough to conclude whether the original was removed or renamed.
Here's the specific case that concerns me: if closeRemoved == false and closeRenamed == true, and we remove a file, then isSameFile will return false with a non-nil error because the os.Stat call fails. Then closeBecauseRemoved will return false, because closeRemoved is false, but closeBecauseRenamed will return true even though the file wasn't renamed.
It's possible I'm mis-parsing this, but in that case some clarification in the language or comments here would be appreciated :-)
There was a problem hiding this comment.
Indeed. 🤦 🤦 🤦 I will move these checks back to the filereader. Thus, I removed it from the PR.
kvch
force-pushed
the
feature-filebeat-file-prospector
branch
from
a521568 to
2667073
Compare
💔 Build Failed
Expand to view the summary
Build stats
Test stats 🧪
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
2 tasks
kvch
added a commit
to kvch/beats
that referenced
this pull request
Oct 5, 2020
## What does this PR do? This PR adds the implementation of `fileProspector`. The prospector listens for events from the `FSWatcher` and processes them depending on the type of the event. Possible actions are starting a new Harvester to read from a file, removing an entry from the registry, etc. (cherry picked from commit d987d10)
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 5, 2020
* upstream/master: (26 commits) [Ingest Manager] Send updating state (elastic#21461) [Filebeat][New Fileset] Cisco Umbrella support (elastic#21504) [Ingest Manager] Download asc from artifact store specified in spec (elastic#21488) Implementation of fileProspector (elastic#21479) [Metricbeat] Add latency config option into aws module (elastic#20875) Skip filestream flaky tests (elastic#21490) Ignore unsupported metrics in the azure module (elastic#21486) Do not run symlink tests on Windows (elastic#21472) Map `cloud.account.id` to azure sub id (elastic#21483) Add support for app_state metricset (elastic#20639) Include original error when metricbeat fails to connect with Kafka (elastic#21484) Prompt only when agent is already enrolled (elastic#21473) Fix leftover delpoyment example (elastic#21474) Bump version to ECS 1.6 in modules without ECS updates (elastic#21455) Clarify input type configuration options (elastic#19284) Increase index pattern size check to 10MiB (elastic#21487) Migrate S3 Input to Filebeat Input V2 (elastic#20005) [libbeat] Add configurable exponential backoff for disk queue write errors (elastic#21493) Revert "Revert "[JJBB] Set shallow cloning to 10 (elastic#21409)" (elastic#21447)" (elastic#21467) Fix format of debug messages in tlscommon (elastic#21482) ...
2 tasks
kvch
added a commit
that referenced
this pull request
Oct 5, 2020
## What does this PR do? This PR adds the implementation of `fileProspector`. The prospector listens for events from the `FSWatcher` and processes them depending on the type of the event. Possible actions are starting a new Harvester to read from a file, removing an entry from the registry, etc. (cherry picked from commit d987d10)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds the implementation of
fileProspector. The prospector listens for events from theFSWatcherand processes them depending on the type of the event. Possible actions are starting a new Harvester to read from a file, removing an entry from the registry, etc.Why is it important?
Required for
filestream.Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added tests that prove my fix is effective or that my feature works- [ ] I have added an entry inCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.