Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add wasm runtimes (embedded and nodejs) in kpt #3450

Merged
merged 2 commits into from
Aug 18, 2022

Conversation

mengqiy
Copy link
Contributor

@mengqiy mengqiy commented Aug 9, 2022

This PR introduces 2 commands: kpt alpha wasm push and kpt alpha wasm pull:

  • kpt alpha wasm push [path-to-local-wasm-file] [desired-OCI-image-name]
  • kpt alpha wasm pull [OCI-image-name] [desired-local-path-to-store-the-wasm-file]

This PR added 2 wasm runtimes (one embedded wasmtime and node.js) in kpt:

  • It is not a WASI runtime, since golang doesn't have a great support to compile go program to WASI and most of our KRM functions are written in go.
  • It requires the wasm to register a js function called processResourceList in the globals.

To use it in Kptfile:

apiVersion: kpt.dev/v1
kind: Kptfile
metadata:
  name: app
pipeline:
  mutators:
    - wasm: gcr.io/kpt-fn-demo/set-namespace:v0.5.0 # Note: this image has a manifest with arch: wasm and os: js
      configMap:
        namespace: staging
    - wasm: gcr.io/kpt-fn-demo/set-labels:v0.2.0 # Note: this image has a manifest with arch: wasm and os: js
      configMap:
        foo: bar

Then run kpt fn render --allow-alpha-wasm with 2 functions. KPT_FN_WASM_RUNTIME can be used to select a runtime.

Syntax embedded wasmtime node.js container
first run real 0m8.648s
user 0m44.559s
sys 0m1.047s
TODO TODO
subsequent run real 0m1.223s
user 0m1.125s
sys 0m0.498s
TODO TODO

This PR requires CGO, it seems cross compilation doesn't work nicely with CGO. We are using GitHub actions to ensure it can be compiled on other OS. #3463 blocks it from working on windows.

To produce wasm artifacts, please take a look at GoogleContainerTools/kpt-functions-catalog#898.

Depends on #3470

@mengqiy mengqiy changed the title add wasm runtime in kpt [WIP] add wasm runtime in kpt Aug 9, 2022
@mengqiy mengqiy mentioned this pull request Aug 9, 2022
@mengqiy mengqiy changed the title [WIP] add wasm runtime in kpt add wasm runtime in kpt Aug 9, 2022
@mengqiy mengqiy changed the title add wasm runtime in kpt add embedded wasm runtime in kpt Aug 9, 2022
@mengqiy mengqiy force-pushed the embedwasmruntime branch 4 times, most recently from a79c405 to d7e233a Compare August 10, 2022 06:26
@mengqiy mengqiy changed the title add embedded wasm runtime in kpt add wasm runtimes (embedded and nodejs) in kpt Aug 10, 2022
commands/wasmcmd.go Outdated Show resolved Hide resolved
commands/wasmcmd.go Outdated Show resolved Hide resolved
commands/wasmcmd.go Outdated Show resolved Hide resolved
go.mod Outdated Show resolved Hide resolved
.github/workflows/checkLicenses.yml Outdated Show resolved Hide resolved
pkg/api/kptfile/v1/validation.go Outdated Show resolved Hide resolved
pkg/wasm/client.go Show resolved Hide resolved
pkg/wasm/client.go Outdated Show resolved Hide resolved
pkg/wasm/client.go Outdated Show resolved Hide resolved
porch/api/generated/clientset/versioned/scheme/register.go Outdated Show resolved Hide resolved
@justinsb
Copy link
Contributor

Some small nits, but this is pretty close I think.

I think sending the go 1.19 and ioutil -> os changes separately would help make this much more reviewable.

@mengqiy
Copy link
Contributor Author

mengqiy commented Aug 10, 2022

I think sending the go 1.19 and ioutil -> os changes separately would help make this much more reviewable.

These changes are really from #3447. I cherry picked the commits to make CI happy.

This PR requires CGO, but it seems we cross compilation doesn't work nicely with CGO. To workaround it, We may need to build it in container for darwin and windows. @justinsb Any suggestions or prior arts in other k8s projects?

internal/fnruntime/exec.go Show resolved Hide resolved
internal/fnruntime/nodejs.go Outdated Show resolved Hide resolved
internal/fnruntime/nodejs.go Outdated Show resolved Hide resolved
internal/fnruntime/nodejs.go Outdated Show resolved Hide resolved
internal/fnruntime/wasm.go Show resolved Hide resolved
internal/fnruntime/wasmtime.go Outdated Show resolved Hide resolved
internal/cmdrender/cmdrender.go Show resolved Hide resolved
internal/util/render/executor.go Outdated Show resolved Hide resolved
@mengqiy mengqiy force-pushed the embedwasmruntime branch 3 times, most recently from 75b874e to 67aed17 Compare August 17, 2022 01:15
@mengqiy
Copy link
Contributor Author

mengqiy commented Aug 17, 2022

I just realized that a few comments have not been addressed. I missed them until I resolved other comments. I will resolve them tomorrow.
Meanwhile the PR is mostly ready for another round of review.

I have move part of porch/pkg/oci package to pkg/oci. If you think I should split it out to its own PR. Please let me know. It should not be hard to do it.

@mengqiy mengqiy force-pushed the embedwasmruntime branch 2 times, most recently from 6d3ece5 to 9342e90 Compare August 17, 2022 21:15
Copy link
Contributor

@natasha41575 natasha41575 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, defer to @justinsb for final approval

.github/workflows/go.yml Show resolved Hide resolved
internal/fnruntime/nodejs.go Outdated Show resolved Hide resolved
@justinsb
Copy link
Contributor

Two small things, but lgtm!

@mengqiy mengqiy merged commit 8b9bfcf into kptdev:main Aug 18, 2022
@mengqiy mengqiy deleted the embedwasmruntime branch August 18, 2022 06:55
chunglu-chou pushed a commit to chunglu-chou/kpt that referenced this pull request Aug 20, 2022
* add wasm runtime in kpt

* add testcase for wasm eval
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants