Find a way to reliably track semantic meaning at runtime across buffers/reboot

[JeanMertz]

This issue is based on a discussion at #11344 (comment), and an async Zoom discussion between @fuchsnj and me.

This issue has to be resolved before we publicly release the event schema work

As work on internal schemas continues, there is a need to track at runtime which schema belongs to which event. This cannot be done at boot-time because a sink can accept events from multiple inputs, and for it to be able to fetch field values based on the "semantic meaning" attached to an event schema, it has to know per event where the field belonging to a given semantic meaning can be found.

In the current (not yet released) implementation, we store a u16 as a schema::Id type in the event metadata. There are two issues with this:

1. Event metadata is not propagated through disk buffers (and thus also not across restarts).
2. Even if it was, the schema IDs are pointers into a schema registry, that stores the actual schema definitions. This registry is also not persisted across reboots, and thus any events pulled from the disk buffer after a reboot would have their schema ID pointing to a non-existing entry in the registry.

A potential solution to (1) is to not use event metadata to store the event ID, but have it be stored directly on the event (but invisible to users, so not part of the external event payload). This is somewhat unfortunate, as event metadata is the appropriate location for this type of information, where it not for the fact that no event metadata survives across buffer boundaries. So, the most ideal solution here was for #7065 to be resolved independently.

Assuming (1) is solved, for (2), one potential solution to this problem is to remove the indirection of schema::Id and directly store schema::Definition on an event, but the reason we chose to store an ID is that it has no significant performance impact on Vector's runtime behaviour, whereas storing the definition would require more data to be serialized, which will have a higher (but unsure how high) impact on performance. We could minimize this overhead by storing only the subset of a schema::Definition that we need at runtime, namely the semantic meaning and the LookupBuf it points to inside an event, but either way, there will be an increase in overhead.

UPDATE another way to further reduce the serialized data size, as discussed with @fuchsnj, is to change our semantic meanings to an enum internally, and attach a "default field" with each meaning. For example, the Meaning::Message semantic meaning would by default point to our log_schema defaults (in this case message), which, if the default is used, means we don't need to serialize any data, and can reconstruct it when deserializing from that default value. This does mean changing the defaults of log_schema is a breaking change, but I believe we have never changed those in the past.

The alternative solution for (2) is to persist the schema::Registry separately from the persisted events with their schema::Id, so that that ID always points back to a known definition in the registry after boot. This does bring other complications with it, such as when to persist this data. There might be other as-of-yet undocumented complications here.

Lastly, regardless of the solutions taken, there is always the possibility that an event is read from a disk buffer, using a newer Vector version that introduced changes to the schema requirements of a sink, such that it can no longer process the events it receives from the buffer (e.g. it expects a new semantic meaning that isn't known to exist in that event), at which point there's no recourse for us, other than dropping the event and returning an error. We can mitigate this by making sure we keep our schema requirements as stable as possible, and/or change them in backward compatible ways, but there will always be a possibility for events to be dropped.

[JeanMertz]

There is a (most likely preferred) way forward that requires no data to be stored on events, and thus circumvents any of the serialization issues discussed above. It would require a "translation" layer between upstream components and sinks, and has some other constraints that need to be worked out, but @fuchsnj will explain this solution. For now, we settled on one of the above solutions potentially being the short-term solution, with the more extensive solution being a longer-term solution.

[fuchsnj]

To solve this issue, I'd like to propose a "translation" step that occurs to events after "transforms" run, but before an event is sent to a buffer (on its way to a sink).

Before the translation step, each event will still need to track semantic meaning, but since buffers aren't involved this shouldn't be an issue.

In the translation step, it effectively "applies" the semantic meaning to the event. This will re-structure the event so that the top-level fields names are the semantic meanings.

As an example, let's say we have an event:

{
   "foo": "unicorn",
   "timestamp": 12873577238,
   "bar": "some additional data",
}

This event is sent to some imaginary sink that requires the following semantic meaning

timestamp
message
payload

The "payload" here is a new concept. In order for this proposal to work, sinks would be required to only use semantic meaning to obtain data from events. For something like the HTTP sink that would want to send the entire event, this is still possible, you just add new semantic meaning that represents what I'll call the "payload", which can point to the root event object (or any other field).

The semantic meaning "schema" maps from the event fields to the actual semantic fields. This is based on the configured sources, and any transforms which can also adjust this mapping. Once this event reaches the "translation" step, the mapping of "semantic meaning" -> "field name" looks like this

   "timestamp": "timestamp",
   "message": "foo",
   "payload": "."

The "translation" step will now apply the semantic meaning mapping to the event, and produce this.

"timestamp": 12873577238,
"message": "unicorn",
"payload": {
   "foo": "unicorn",
   "timestamp": 12873577238,
   "bar": "some additional data",
}

This translated event can still be treated as a normal event, so the buffer implementations don't need to change, and sinks can accept them as-is and just work today. Only fields actually necessary for the sink will be sent to the sink. The sink no longer requires, or even cares about which schema was associated with the event, and this data would be valid across restarts or configuration changes.

Note: This example duplicated some of the fields during the translation step (increasing the size of the event). This probably isn't typical of real-world sinks, but was useful for the example to show different use cases.