Investigate replacing feature flag service

[austinlparker]

Capturing a SIG meeting discussion today...

The feature flag service is kind of a frustrating dependency for us currently. It's very important, but it's also hard to maintain as we don't have a ton of expertise in Erlang. In addition, due to limitations of our build pipeline it's very out of date.

Our general idea is to investigate replacing the feature flag service with some other off-the-shelf solution, then refactoring everything to use OpenFeature. Erlang support would come back into the demo as part of a smaller, more self-contained service (possibly some sort of consumer off Kafka) that should be more amenable to the build pipeline.

[austinlparker]

From some research, I think our best bet might be flagd. It's maintained by OpenFeature, and it seems to be fairly lightweight and straightforward for our purposes. Even better, it's OpenTelemetry compatible, so we can get independent instrumentation out of it. It supports more evaluation styles than we currently do as well (such as targeting, multi-variable experiments, etc.) and can be configured with JSON files rather than requiring a separate database.

[puckpuck]

I vote we replace the product catalog service with an Erlang/Elixir equivalent. This service is part of the minimal install and is one of 3 Go-based services in the demo.

[austinlparker]

I vote we replace the product catalog service with an Erlang/Elixir equivalent. This service is part of the minimal install and is one of 3 Go-based services in the demo.

That should be vaguely easier to maintain since there's no UI component at least?

[julianocosta89]

ProductCatalog sounds like a good suggestion to be replaced.

I'm just afraid of whenever we start adding Swift and Android to the demo.
Those 2 will have to talk with ProductCatalog, and changing anything in Erlang/Elixir is a bit overwhelming when you are not used to work with it, which is the case for 98% of global population 😅

[austinlparker]

#1388

Went ahead and made a PR swapping things out so we can get an idea of what this all would look like. There's a few gaps (no Python provider for flagd, which I'll have to write or find) but overall it seems ok.

ed: looks like there is a flagd provider for python, it's just not released yet. hopefully they'll get that sorted soon.

[austinlparker]

@julianocosta89 @puckpuck After spending a bit of time trying to rewrite the product catalog in Erlang, I'm not so sure about it. Frankly, 3/4ths of this is that I simply don't grok Erlang well enough to be productive in it (and ChatGPT only gets you so far). I think our best bet would be to create a new Erlang service that reads off Kafka and try to make it as minimal as possible to avoid the need to touch it in the future outside of OpenTelemetry updates. I feel like making Product Catalog an Erlang service just creates another load-bearing dependency that'll be hard for us to work with going forward.

[austinlparker]

Also worth noting there's no OpenFeature libraries for Erlang at this point in time, so we'd have to write that as well.

[julianocosta89]

Yeah, I've figured that would be tricky!
There is also another thing, if we make a erlang/elixir service to talk with kafka, there is no instrumentation for it 😢
Not sure how great that would be

[julianocosta89]

Let me bring to the discussion @joshleecreates that has some experience in Erlang/Elixir, and also @tsloughter that was the one that initially implemented the FeatureFlagService.

[joshleecreates]

Also worth noting there's no OpenFeature libraries for Erlang at this point in time, so we'd have to write that as well.

The service doesn't need to consume feature flags in its initial version.

I think our best bet would be to create a new Erlang service that reads off Kafka and try to make it as minimal as possible to avoid the need to touch it in the future outside of OpenTelemetry updates.

I am inclined to agree with this. I like the idea of a simple stateless service, but to @julianocosta89's point, there would be no auto-instrumentation unless we also included a downstream database or some kind of HTTP server. But maybe that isn't a bad thing? This is a pretty realistic use-case and would be a good demo of combining auto- and manual-instrumentation to complete a trace.

[tsloughter]

Just saw this issue after i saw the other. I can do the product catalog. Also can do a kafka service, but seems easier to just translate an existing service that seems to just read in json and serve it over grpc?

[joshleecreates]

Personally I lean away from "batteries-inlcuded" frameworks like Phoenix in the demo since the frameworks' conventions are often at odds with the development and build needs of the demo, so removing Phoenix gets a +1 from me.

[austinlparker]

@tsloughter @joshleecreates I think the ideal replacement Erlang service would be something that meets the criteria of #912 . Reads orders off the Kafka queue, then persists them to a DB.

If we wanted to swap out a JSON service, then we could... but I'd be more in favor of doing something net-new, just to make sure it wasn't a load-bearing part of the architecture.

[austinlparker]

Replying to #1388 (comment)

The biggest advantage I can see to simplifying the build pipeline is that if we move Erlang off into its own little corner, then we don't have to deal with the weird QEMU build issues any more. FF Service was load-bearing to the entire demo and wouldn't run on AS/ARM without cross-compilation. By making a service that's kind of just off to the side, if we can't build cross-platform then that's fine, we can just publish the x86 version and have it not run without impacting the overall demo.