Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New blog post introducing Data Prepper 1.3.0 and its new features. #729

Merged
merged 3 commits into from
Mar 22, 2022
Merged

New blog post introducing Data Prepper 1.3.0 and its new features. #729

merged 3 commits into from
Mar 22, 2022

Conversation

dlvenable
Copy link
Member

@dlvenable dlvenable commented Mar 18, 2022 •
edited
Loading

Description

This blog post introduces Data Prepper 1.3.0. We'd like it to go live on Tuesday Mar 22.

Issues Resolved

N/A

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

@dlvenable @graytaylor0
New blog post introducing Data Prepper 1.3.0 and its new features.
ea2ddff 
Co-authored-by: Taylor Gray <tylgry@amazon.com>
Co-authored-by: David Powers <ddpowers@amazon.com>
Signed-off-by: David Venable <dlv@amazon.com>
@dlvenable dlvenable requested a review from a team as a code owner March 18, 2022 21:30
@dlvenable dlvenable self-assigned this Mar 18, 2022
@stockholmux stockholmux mentioned this pull request Mar 18, 2022
Merged
1 task
stockholmux
stockholmux suggested changes Mar 21, 2022
View reviewed changes
Copy link
Member

@stockholmux stockholmux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks pretty good to me. I have a few small requests - mostly formatting and style.

_posts/2022-03-22-Introducing-Data-Prepper-1.3.0-with-New-Aggregation-Processor.md Outdated
[key-value processor](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/key-value-processor/README.md)
provides robust support for extracting keys and values from strings like these.

The following example shows how you could use the new split_string processor and key_value processor to get query
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

split_string, key_value -> put in backticks (``)

_posts/2022-03-22-Introducing-Data-Prepper-1.3.0-with-New-Aggregation-Processor.md Outdated

## Setting Timestamps on Events

Data Prepper new provides a new
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'Data Prepper new provides' -> doesn't parse. Is something missing?

_posts/2022-03-22-Introducing-Data-Prepper-1.3.0-with-New-Aggregation-Processor.md Outdated

## Other Improvements

In addition to the new features already described, we have made a few other improvements to Data Prepper.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd revise this - does 'we' refer to the authors of the blog post or the software?

_posts/2022-03-22-Introducing-Data-Prepper-1.3.0-with-New-Aggregation-Processor.md Outdated

You can see the [Data Prepper roadmap](https://github.com/opensearch-project/data-prepper/projects/1) to see other
upcoming changes. If there are any features on the roadmap that you are most interested in, please comment on the
issue to help us prioritize issues. You can also request any changes by creating a
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'help us' -> 'help the team'

_posts/2022-03-22-Introducing-Data-Prepper-1.3.0-with-New-Aggregation-Processor.md Outdated
for example, would allow a user to add in an entry to their data to help debug the flow of data:


```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to ```yaml

_posts/2022-03-22-Introducing-Data-Prepper-1.3.0-with-New-Aggregation-Processor.md Outdated
The following example shows how you could use the new split_string processor and key_value processor to get query
parameters from an Apache log line.

```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to ```yaml

_posts/2022-03-22-Introducing-Data-Prepper-1.3.0-with-New-Aggregation-Processor.md Outdated
processor, and then aggregates on those fields over a period of 30 seconds using the `aggregate` processor and
the `put_all` action. At the end of the 30 seconds, the aggregated log is sent to the OpenSearch sink.

```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to ```yaml

_posts/2022-03-22-Introducing-Data-Prepper-1.3.0-with-New-Aggregation-Processor.md Outdated

Given the following batch of logs:

```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to ```json

_posts/2022-03-22-Introducing-Data-Prepper-1.3.0-with-New-Aggregation-Processor.md Outdated

The grok processor will extract the `identification_keys` to create the following logs:

```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to ```json

_posts/2022-03-22-Introducing-Data-Prepper-1.3.0-with-New-Aggregation-Processor.md Outdated
And when the group is concluded after a duration of 30 seconds from the time that the first log is
received by the `aggregate` processor, the following aggregated log will be shipped to the sink:

```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to ```json

@dlvenable @graytaylor0
Updated the blog post for Data Prepper 1.3.0 release based on PR feed…
1d2bfeb 
…back.

Co-authored-by: Taylor Gray <tylgry@amazon.com>
Co-authored-by: David Powers <ddpowers@amazon.com>
Signed-off-by: David Venable <dlv@amazon.com>
@dlvenable
Copy link
Member Author

@stockholmux , Thank you for the feedback. I made the changes you suggested.

Thanks for noting the language support. I didn't know that the project page's tooling supported these. I think it especially helps with the YAML.

@dlvenable @graytaylor0
Corrected the publication date of the blog post for Data Prepper 1.3.…
640fe1e 
…0 release.

Co-authored-by: Taylor Gray <tylgry@amazon.com>
Co-authored-by: David Powers <ddpowers@amazon.com>
Signed-off-by: David Venable <dlv@amazon.com>
stockholmux
stockholmux approved these changes Mar 22, 2022
View reviewed changes
Copy link
Member

@stockholmux stockholmux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@stockholmux stockholmux merged commit 1a59d4c into opensearch-project:main Mar 22, 2022
@dlvenable dlvenable deleted the data-prepper-1.3.0-blog-post branch October 20, 2022 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stockholmux stockholmux stockholmux approved these changes

Assignees

@dlvenable dlvenable

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dlvenable @stockholmux