fix: significant rework of Event<->IngestDocument marshalling

[yaauie]

This PR Fundamentally changes the model for marshalling Events to IngestDocument`s
and back again to solve three inter-related issues:

Resolves: #47

The IngestDocument metadata fields are no longer added to the top-level of
the Event, and are now separately routed to [@metadata][_ingest_document],
fixing an issue where the presence of Elasticsearch-reserved fields such as
the top-level _version would cause a downstream Elasticsearch output to be
unable to index the event

Resolves: #54

The top-level @timestamp and @version fields are no longer excluded from
the IngestDocument, as required by some existing integration pipelines

Partially-Resolves: #65

Field-type conversions have been improved by adding a two-way-mapping between
the Logstash-internal Timestamp-type object and the equivalent
ZonedDateTime-object used in several Ingest Common processors

Review Note: unfortunately, these three issues became pretty tangled, so this code is best reviewed by looking at the result of the changes to the IngestDuplexMarshaller and not the diff

[roaksoax]

LGTM!

[yaauie]

@roaksoax unfortunately this opens up a can-of-worms because ingest pipelines do intentionally set metadata fields that are used downstream for indexing (see failing integration specs), and we currently have no way to propagate them to our Elasticsearch output plugin, which must supply them separately from the event payload itself.

I have considered adding a [@metadata][_ingest_pipeline_metadata] to contain any ingest pipeline metadata (possibly _id, _index, _version, _routing, and _version_type, all of which have sprintf-powered options on the ES output), but (1) the ES output must be configured explicitly with a sprintf for each option and (2) does not behave well when a sprintf references fields that are unset.

Example usage (would work IFF all five magic fields are set):

output {
  elasticsearch {
    # ... connection details
    document_id => "%{[@metadata][_ingest_pipeline_metadata][_id]}"
    index => "%{[@metadata][_ingest_pipeline_metadata][_index]}"
    version => "%{[@metadata][_ingest_pipeline_metadata][_version]}"
    version_type => "%{[@metadata][_ingest_pipeline_metadata][_version_type]}"
    routing => "%{[@metadata][_ingest_pipeline_metadata][_routing]}"
  }
}

While we theoretically could pre-populate all five with values when they are not present, we have no way to tell Elasticsearch to do its "default" id generation or routing.

[mashhurs]

Except doc N/A to _index map change, rest of them LGTM!

[mashhurs]

Suggested change

| _N/A_ | `_index`, `

I got confused with this change. Can you please explain what it is? Basically, how non-available field can be mapped to _index?

[yaauie]

This was accidental and will be removed. Oops.

[mashhurs]

I greatly appreciate this work @yaauie, thanks so much !
This changes resolve deep-core issues of plugin-event to IngestDocument transfer safety.

I have tested with major issues with some integrations such as Apache, Nginx and LGTM!
We have to land ASAP this change and release the new 0.0.2 version.

[yaauie]

Merging now, will land a new release of the plugin on Monday.