Add healthchecks for transforms

[LucioFranco]

Some of our transforms need to fetch data from external systems. This means the data may not be available by the time the first event comes through the system. Currently, the behavior is to just let the event pass through without enriching it. To avoid this we can instead add healthchecks for these transforms that will block the topology from being ready until data is available. This achieves the proper healthcheck requirement but will also provide a much more consistent experience for users.

We most likely want to update the TransformConfig trait to add a healthcheck method that has a default implementation that always resolves to ready.

[binarylogic]

Thanks. Isn't a better approach to implement an initialization callback? The healthcheck shouldn't be responsible for initialization, it should only verify that it has everything it needs to function properly.

[LucioFranco]

@binarylogic I believe if you run vector to block on healthy healthchecks then you get the behavior of no events missing data. If you don't include that then it won't so I see healthchecks as aligning with the behavior we want.

As for an init future, that could work but would deviate from what we currently do with our other components. We generally prefer to lazily do things instead of upfront with an init future. I'd like to hear what @lukesteensen thinks of that.

[binarylogic]

I believe if you run vector to block on healthy healthchecks then you get the behavior of no events missing data

That's good, but I think it's misleading. It's not obvious to users that healtchecks change state. FTR, I don't think they should either.

As for an init future, that could work but would deviate from what we currently do with our other components. We generally prefer to lazily do things instead of upfront with an init future.

Sure, then I'd say the aws_ec2_metadata transform should be able to block on that first event while it fetches the metadata. Or, this should be an option.

[lukesteensen]

I agree that using a healthcheck as initialization is a confusing combination of two concerns that should be separate.

Currently, our "init" phase is just the build method, which isn't a great fit here. I don't think it'd be terribly difficult to add an optional method to Transform that we call and block on before feeding events through, but it's worth some thought.

[LucioFranco]

That's good, but I think it's misleading. It's not obvious to users that healtchecks change state. FTR, I don't think they should either.

The way I was viewing it is the metadata for the vector instance is global, so if the first fetch happens during a healthcheck we can just use that data. But I agree it is a bit confusing.

Currently, our "init" phase is just the build method, which isn't a great fit here. I don't think it'd be terribly difficult to add an optional method to Transform that we call and block on before feeding events through, but it's worth some thought.

Yeah, that could work I really wonder if that is a healthcheck or if we can change healthchecks in general to be an init and if it fails we don't run the components?

[lukesteensen]

Yeah, that could work I really wonder if that is a healthcheck or if we can change healthchecks in general to be an init and if it fails we don't run the components?

So I think this is a particular case where health checking and initialization consist of basically the same action. That certainly isn't always the case, so I think we should be cautious of drawing general conclusions based on it.

It'd be better to wait until we have a least a handful of transform use cases that would benefit from health checking and/or initialization before deciding on a design.