[FLI-856] Support Alpine/Musl

[markphelps]

Currently the engine is only compiled against GLibc which means the clients will not work in Alpine env

We should create musl compatible versions, targeting x86_64-unknown-linux-musl and aarch64-unknown-linux-musl

I think there are two paths we could take:

1. Continue to bundle these new libs along with the glibc versions so that the native clients (ie flipt-client-java) should 'just work' on all architectures
2. Create new musl/alpine versions of each flipt-client-x library, putting the responsibility on the user/integrator to select the correct SDK for their use case.

I'm leaning on 2 as it might be very difficult for us to figure out which version of the fliptengine Clib to 'load' at runtime when the clients initialize (ie likely not all languages allow you to know which libc your os supports/requires). Also each additional fliptengine we package increases the overall library side by avg of 20-30mb (perhaps we can find a way to slim these down though?)

I'd be interested to know what others think, also if there are any examples of other SDKs built with FFI and run on both libc and musl based OSes

_{From SyncLinear.com | FLI-856}

SDKs to Support

• Go (go.flipt.io/flipt-client@musl-vx.x.x)
• Node (now uses wasm)
• Python
• Ruby
• Dart
• Java

[markphelps]

See: https://github.com/flipt-io/flipt-client-sdks/actions/runs/7962787687/job/21736987230?pr=140#step:5:1395 for POC of failing test using musl based docker image

[markphelps]

cc @piclemx

[piclemx]

@markphelps I would go with the second option also.

[jalaziz]

What would option 2 look like in practice?

For example, if I import the go library, would I have to choose between an glibc version or a musl version? That would be somewhat problematic in scenarios where local development environments are glibc, but the production environment is musl.

[jalaziz]

@markphelps Spent some time looking into this.

I think a good example repo to look at is the confluent-kafka-go library since they use the librdkafka C-library under the hood. This is also generally true for all their SDKs except for the Java one.

For the Go SDK, it looks like they use Go build tags to support musl vs libc, which seems like a better option than different imports.

If you look at the librdkafka_vendor directory each included version of the library is 13MB+, so doesn't look like that's a huge issue.

Also, see the build_*.go files (e.g. build_darwin_amd64.go) for how they use build tags to include the right version of the library. The file platform and arch file suffixes act as implicit build tags that match against GOOS and GOARCH.

It seems like the biggest downside to this approach is that it will download all the different pre-built binaries during the go mod download phase, but the final binary should be statically linked to only the version the platform that you're building for needs. Obviously, it would be nice to make the pre-built libraries smaller to save on download costs, but given that's usually cached, I wouldn't over optimize yet.

Some additional useful reading:

• https://www.digitalocean.com/community/tutorials/building-go-applications-for-different-operating-systems-and-architectures
• https://www.digitalocean.com/community/tutorials/customizing-go-binaries-with-build-tags

[jalaziz]

Just checking in on this. We'd love to use the client-side evaluation SDK, but this is a blocker for us (not just musl, but also automatic arch selection).

[markphelps]

He @jalaziz sorry for the delay in replying. We can definitely take a look at building a MUSL compatible build and packaging like you mentioned above. Or if you are able to contribute we would ofcourse ❤️ that as well!

We are also actively exploring leaning more into WASM, and building a proper WASM impl of the evaluator engine (not just for JS like we have now) and then using WASM runtimes in the various languages we support to call the evaluator. The main benefit of this approach would be that it would be architecture independent.

First one would likely be Go since that's the language we use the most internally. Would this work for you and your team?

[jalaziz]

He @jalaziz sorry for the delay in replying. We can definitely take a look at building a MUSL compatible build and packaging like you mentioned above. Or if you are able to contribute we would ofcourse ❤️ that as well!

We are also actively exploring leaning more into WASM, and building a proper WASM impl of the evaluator engine (not just for JS like we have now) and then using WASM runtimes in the various languages we support to call the evaluator. The main benefit of this approach would be that it would be architecture independent.

First one would likely be Go since that's the language we use the most internally. Would this work for you and your team?

That would work great. Our main use case is Go as well.

I don't know if I'll have time to contribute the changes for the approach outlined above, but I'll definitely try!

[markphelps]

@jalaziz @piclemx we just released flipt-client-go@musl-v0.0.1, see #332 for details

Can you try:

go install go.flipt.io/flipt-client@musl-v0.0.1

It should support both x86 and arm on Linux

[willyw0nka]

I'm implementing Flipt at alpine based Java microservices for my company. Is there some workaround or a roadmap planned for the Java version? This seems to be a big blocker for now

[markphelps]

I'm implementing Flipt at alpine based Java microservices for my company. Is there some workaround or a roadmap planned for the Java version? This seems to be a big blocker for now

I can get a Java version of the sdk built that works on musl. Will work on it today and keep you posted

[markphelps]

@willyw0nka we just released https://central.sonatype.com/artifact/io.flipt/flipt-client-java-musl

could you try it and let me know if it works for you?

[SultanICQ]

Hi @markphelps , I'm a colleage of @willyw0nka and tried your solution. It still doesn't work for us.

I integrated the dependency and built the project just fine, but when I tried to run it on my development machine (OSX) it throws me these errors.

nested exception is java.lang.UnsatisfiedLinkError: Unable to load library 'fliptengine': dlopen(libfliptengine.dylib, 0x0009): tried: 'libfliptengine.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibfliptengine.dylib' (no such file), '/Users/david/.sdkman/candidates/java/17.0.10-tem/bin/./libfliptengine.dylib' (no such file), '/Users/david/.sdkman/candidates/java/17.0.10-tem/bin/../lib/libfliptengine.dylib' (no such file), '/usr/lib/libfliptengine.dylib' (no such file, not in dyld cache), 'libfliptengine.dylib' (no such file) dlopen(libfliptengine.dylib, 0x0009): tried: 'libfliptengine.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibfliptengine.dylib' (no such file), '/Users/david/.sdkman/candidates/java/17.0.10-tem/bin/./libfliptengine.dylib' (no such file), '/Users/david/.sdkman/candidates/java/17.0.10-tem/bin/../lib/libfliptengine.dylib' (no such file), '/usr/lib/libfliptengine.dylib' (no such file, not in dyld cache), 'libfliptengine.dylib' (no such file) dlopen(/Users/david/Library/Frameworks/fliptengine.framework/fliptengine, 0x0009): tried: '/Users/david/Library/Frameworks/fliptengine.framework/fliptengine' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/david/Library/Frameworks/fliptengine.framework/fliptengine' (no such file), '/Users/david/Library/Frameworks/fliptengine.framework/fliptengine' (no such file) dlopen(/Library/Frameworks/fliptengine.framework/fliptengine, 0x0009): tried: '/Library/Frameworks/fliptengine.framework/fliptengine' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Library/Frameworks/fliptengine.framework/fliptengine' (no such file), '/Library/Frameworks/fliptengine.framework/fliptengine' (no such file) dlopen(/System/Library/Frameworks/fliptengine.framework/fliptengine, 0x0009): tried: '/System/Library/Frameworks/fliptengine.framework/fliptengine' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/System/Library/Frameworks/fliptengine.framework/fliptengine' (no such file), '/System/Library/Frameworks/fliptengine.framework/fliptengine' (no such file, not in dyld cache)

Next thing I tried was to run the app.jar inside our Alpine Linux docker image and the errors are the same as with the regular java version.

Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'flagController' defined in URL [jar:file:/app.jar!/BOOT-INF/classes!/org/company/services/nexus/medias/controller/FlagController.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'featureFlagService' defined in class path resource [org/company/services/featureflags/config/FeatureFlagConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.company.services.featureflags.service.FeatureFlagService]: Factory method 'featureFlagService' threw exception; nested exception is java.lang.UnsatisfiedLinkError: Error relocating /root/.cache/JNA/temp/jna8663586253287976616.tmp: swapcontext: symbol not found

hope you can help us.

[markphelps]

Hey @SultanICQ thanks for the detailed reply!

I think there's two issues going on here:

1. for the mac local dev setup, I failed to include the mac version of the libraries (dylibs) in the musl build of our SDK. I'm rectifying that in the next version of the SDK (commit here: a02a947)
2. for the alpine version, I think we also copied the wrong library (the glibc version instead of the musl version), this should also be fixed in the next release of the SDK.

I'll be sure to test this as well before declaring the next version good

[SultanICQ]

Hi @markphelps no worries. If you need me to test something dont doubt to reach me.

[markphelps]

I just released flipt-client-java-musl-v0.0.2 which should have fixed both issues above ☝🏻

it seems to take a bit for it to make it to maven central however, so maybe can try in 30m or so? Im about to be traveling for the next couple hours but can follow up later today

[SultanICQ]

No worries! :)

[SultanICQ]

I just finished testing it and it worked like a charm in both scenarios. Well done!

[markphelps]

I just finished testing it and it worked like a charm in both scenarios. Well done!

Thanks for your patience and for testing @SultanICQ !!

[SultanICQ]

Hi @markphelps, today one of our developers with a windows machine tested the new library and, as everybody would expect, it didn't work because the windows libraries are not included.

Could it be possible for the musl version to add also de win dlls?

[markphelps]

Hey @SultanICQ yes I will work on getting a windows version built this week

[markphelps]

@SultanICQ does your colleague need 32-bit or 64-bit Windows support?

[markphelps]

oh nevermind i see x86-64 in the stack trace

[markphelps]

Work being done over 👉🏻 #433

[markphelps]

Hey @SultanICQ we just released 0.2.0-rc.1 that should have Windows support. I say should because I cant easily test since I don't have a Windows machine. We're working on getting more comprehensive testing after release via GitHub Actions though.

Can you have your coworker try 0.2.0-rc.1 on Windows? Thanks!

[SultanICQ]

Hi @markphelps !

I'll prepare a test for my colleage to try. I assume she'll do the test on monday at the earliest.

I keep you posted.

[SultanICQ]

Hi @markphelps ,

I deleted my previous message, maybe you have received an email from github.

With Windows now there is no library error but now there is a subtle error with the evaluateBoolean response. I see that now is not returning a Result.

So I think that overall it works but I have to adapt my code with this method signature change.

I keep you posted.

[markphelps]

Hi @markphelps ,

I deleted my previous message, maybe you have received an email from github.

With Windows now there is no library error but now there is a subtle error with the evaluateBoolean response. I see that now is not returning a Result.

So I think that overall it works but I have to adapt my code with this method signature change.

I keep you posted.

Yes apologies we simplified the response type by not returning the wrapped Result type re: #353

Since these are pre 1.0.0 releases we decided to bump the minor version for this change in method signature.. sorry about that

[SultanICQ]

Well, all good and working fine with this new version from my side.

Nice work!!

[jalaziz]

@jalaziz @piclemx we just released flipt-client-go@musl-v0.0.1, see #332 for details

Can you try:

go install go.flipt.io/flipt-client@musl-v0.0.1

It should support both x86 and arm on Linux

Just getting back to this, but unfortunately a musl vs glibc dependency won't work for us long term. While all our built docker images are Alpine, we generally test locally on our machines which use glibc. It would be great if we could install one dependency that is universal.

As I mentioned in a previous post, that should be doable with Go and build tags. Yes that means downloading both the musl and glibc libraries when building, but I don't think that's a major issue.

[markphelps]

Gotcha. I'll create a universal version to try this approach and ping you once ready