Split plugins in input, processor, sink

[sandervandegeijn]

Is your feature request related to a problem? Please describe.
All types are now in one directory: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins

Would be nice to split everything in different subdirectories for input, processor, sink, etc. Goal is to provide a better overview to the user.

Describe the solution you'd like
Create multiple directories

Describe alternatives you've considered (Optional)
None

Additional context
None

[dlvenable]

This is related to #1503 which calls for a new directory named plugins.

So I believe we'd end up with:

plugins/
  sinks/
    opensearch/
    s3/
  sources/
    opensearch/
    http/
    ...

---

One possible problem with this approach is that some plugins don't fit the pattern above. For example, kafka-plugins is a project for three Kafka related items: source, sink, buffer. There is also the common and armeria-common projects and perhaps a few others.

---

@ict-one-nl , Would you be interested in thinking through this solution more, making a proposal for handling some of the projects that don't fit neatly, and create a PR?

[sandervandegeijn]

Splitting the i.e. kafka lib is also an option, a common/core where the (probably very minimal) sink/input implementation depend on.

Although I'm not a java dev, sure. Do you have a format do write this in? Also, I'm somewhat surprised by the amount of documentation in github that's not in the documentation itself. This would be a nice to have as well.

[dlvenable]

@ict-one-nl , Yes, we could split the Kafka plugins and create a kafka-common project. I think that would be a good alternative.

Although I'm not a java dev, sure. Do you have a format do write this in?

Thanks for your interest! I'd suggest just outlining where to put the different projects here in a comment in this GitHub issue. You can follow the format to outline a directory structure that I used here and here.

Also, I'm somewhat surprised by the amount of documentation in github that's not in the documentation itself. This would be a nice to have as well.

Yes, we'd like to move this documentation out of the data-prepper GitHub project and into the https://github.com/opensearch-project/documentation-website project. See #2740 for the related issue for this task. If you are interested in helping toward that as well let us know. We can give any guidance that might be helpful.

[sandervandegeijn]

This is related to #1503 which calls for a new directory named plugins.

So I believe we'd end up with:

plugins/
  sinks/
    opensearch/
    s3/
  sources/
    opensearch/
    http/
    ...

One possible problem with this approach is that some plugins don't fit the pattern above. For example, kafka-plugins is a project for three Kafka related items: source, sink, buffer. There is also the common and armeria-common projects and perhaps a few others.

@ict-one-nl , Would you be interested in thinking through this solution more, making a proposal for handling some of the projects that don't fit neatly, and create a PR?

I took a look at it this morning, I believe your suggestion is the right thing to do. Create seperate directories for:

• common: contains shared code when necessary
• input: input plugins
• processors: processors
• sinks: sinks

The documentation should be moved to the documentation repository and the repo's should contain a reference to that, but not the documentation itself. As for creating a pull request: I'm not a java dev and I'm not knowledgeable when it comes to the setup of this project, so I'm not comfortable to reorganize large amounts of code while not breaking things. :)