Add date_prefix configuration option #81
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change adds an optional new 'date_prefix' configuration option that will allow for the creation of a date-based prefix in the S3 path. This will work in combination with the existing 'prefix' option if desired. Any value for this option is expected to be a time definition suitable for passing through strftime (e.g '%Y-%m-%d/'). The value of this option will be inserted between 'prefix' and the filename generated by logstash. This is particularly useful if you have large volumes of logs that you wish to keep for long periods of time (archival purposes). For example, a 5 minute time_file setting will result in 156,960 files for an 18 month retention period for each host sending data. It is more convenient to work with that number of files, particularly in GUI based tools, if they are distributed into smaller groups of files. This will also allow for more granular loading of files into Elasticsearch.
|
+1 |
elasticsearch-bot
pushed a commit
that referenced
this pull request
Dec 15, 2016
**Motivation** One of the most requested features was adding a way to add dynamic prefixes using the fieldref syntax for the files on the bucket and also the changes in the pipeline to support shared delegator. The S3 output by nature was always a single threaded writes but had multiples workers to process the upload, the code was threadsafe when used in the concurrency `:single` mode. This PR addresses a few problems and provide shorter and more structured code: - This Plugin now uses the V2 version of the SDK, this make sure we receive the latest updates and changes. - We now uses S3's `upload_file` instead of reading chunks, this method is more efficient and will uses the multipart with threads if the files is too big. - You can now use the `fieldref` syntax in the prefix to dynamically changes the target with the events it receives. - The Upload queue is now a bounded list, this options is necessary to allow back pressure to be communicated back to the pipeline but its configurable by the user. - If the queue is full the plugin will start the upload in the current thread. - The plugin now threadsafe and support the concurrency model `shared` - The rotation strategy can be selected, the recommended is `size_and_time` that will check for both the configured limits (`size` and `time` are also available) - The `restore` option will now use a separate threadpool with an unbounded queue - The `restore` option will not block the launch of logstash and will uses less resources than the real time path - The plugin now uses `multi_receive_encode`, this will optimize the writes to the files - rotate operation are now batched to reduce the number of IO calls. - Empty file will not be uploaded by any rotation rotation strategy - We now use Concurrent-Ruby for the implementation of the java executor - If you have finer grain permission on prefixes or want faster boot, you can disable the credentials check with `validate_credentials_on_root_bucket` - The credentials check will no longer fails if we can't delete the file - We now have a full suite of integration test for all the defined rotation Fixes: #4 #81 #44 #59 #50 Fixes #102
|
The |
thanks for the PR, ^^ makes sense - we do not want too many prefixes (would be confusing) ... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/
This change adds an optional new 'date_prefix' configuration option that
will allow for the creation of a date-based prefix in the S3 path. This
will work in combination with the existing 'prefix' option if desired.
Any value for this option is expected to be a time definition suitable
for passing through strftime (e.g '%Y-%m-%d/'). The value of this option
will be inserted between 'prefix' and the filename generated by
logstash. This is particularly useful if you have large volumes of
logs that you wish to keep for long periods of time (archival
purposes). For example, a 5 minute time_file setting will result in
156,960 files for an 18 month retention period for each host sending
data. It is more convenient to work with that number of files,
particularly in GUI based tools, if they are distributed into
smaller groups of files. This will also allow for more granular
loading of files into Elasticsearch.