Skip to content

Commit

Permalink
Created INGESTION.md
Browse files Browse the repository at this point in the history
Signed-off-by: Dan Dong <58446449+danieldong51@users.noreply.github.com>
  • Loading branch information
danieldong51 authored Aug 1, 2023
1 parent 37b1847 commit 80fab73
Showing 1 changed file with 9 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Ingestion Pipeline
To set up an ingestion pipeline, I used Docker to run Fluent-bit and an Apache fake log generator, along with an instance of OpenSearch.

In `fluent-bit.conf`:
- The input is wherever your Apache logs are being ingested from. In my case, since I was using an Apache log generator, I used the `forward` protocol.
- Use the [filter from fluent-bit](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) for apache access logs.
- Optionally, I used a [GeoIP2 Filter](https://docs.fluentbit.io/manual/pipeline/filters/geoip2-filter) to enrich the data with geolocation data.
- Finally, I used a `otel-converter.lua` script to convert the parsed data into schema-compliant data
- I outputted the data to the port my OpenSearch instance was listening on.

0 comments on commit 80fab73

Please sign in to comment.