README.md

Function Mesh Runner Images

Function Mesh uses runner images as functions / connectors' Pod images. Each runtime runner image only contains necessary tool-chains and libraries for specified runtime.

Images

Base Runner

The base Runner is located at ./pulsar-functions-base-runner. The base runner contains basic tool-chains like /pulsar/bin, /pulsar/conf and /pulsar/lib to ensure that the pulsar-admin CLI tool works properly to support Apache Pulsar Packages.

Java Runner

The Java Runner is based on Base Runner and contains the Java function instance to run Java functions or connectors.

The streamnative/pulsar-functions-java-runner Java runner is stored at the Docker Hub and is automatically updated to align with Apache Pulsar release.

Python Runner

The Python runner is based on the base runner and contains the Python function instance to run Python functions. You can build your own Python runner to customize Python dependencies.

The streamnative/pulsar-functions-python-runner Python runner is located at the Docker Hub and is automatically updated to align with Apache Pulsar release.

Golang Runner

The Golang runner provides all the toolchain and dependencies required to run Golang functions.

The streamnative/pulsar-functions-go-runner Golang runner is located at the Docker Hub and is automatically updated to align with Apache Pulsar release.

How to use

Use with Apache Pulsar Packages

Prerequisite

• Apache Pulsar 2.8.0
• Function Mesh v0.1.3

Package your function locally

You can package Pulsar Functions in Java, Python and Go. Please refer to the official document to packaging your Pulsar Functions.

Upload your function to Pulsar Cluster

With the packaged functions, user can upload the function package into Apache Pulsar Packages.

You can upload a package to the package management service by pulsar-admin:

bin/pulsar-admin packages upload function://my-tenant/my-ns/my-function@0.1 --path package-file --description package-description

Then user can define Function Mesh CRDs to use the uploaded function package.

Submit to Function Mesh

Below is a sample CRD to create a Java function from package function://my-tenant/my-ns/my-function@0.1.

apiVersion: compute.functionmesh.io/v1alpha1
kind: Function
metadata:
  name: java-function-sample
  namespace: default
spec:
  image: streamnative/pulsar-functions-java-runner:2.7.1 # using java function runner
  className: org.apache.pulsar.functions.api.examples.ExclamationFunction
  forwardSourceMessageProperty: true
  maxPendingAsyncRequests: 1000
  replicas: 1
  maxReplicas: 5
  logTopic: persistent://public/default/logging-function-logs
  input:
    topics:
    - persistent://public/default/java-function-input-topic
    typeClassName: java.lang.String
  output:
    topic: persistent://public/default/java-function-output-topic
    typeClassName: java.lang.String
  pulsar:
    pulsarConfig: "test-pulsar"
  java:
    jar: my-function.jar # the package will download as this filename.
    jarLocation: function://my-tenant/my-ns/my-function@0.1 # function package URL

If you are not using self built runner image, you can skip image, Function Mesh manager will choose image according to the runtime defined in CRD. (for example, if user define java runtime settings, Function Mesh will use streamnative/pulsar-functions-java-runner by default)

Use with self-built Docker image

Prerequisite

• Apache Pulsar 2.7.0
• Function Mesh v0.1.3

Package your function locally

You can package Pulsar Functions in Java, Python and Go. Please refer to the official document to packaging your Pulsar Functions.

Build your function image

You can write up a Dockerfile to build your function image packed with your function executable and all dependencies. For example, with a packed Java function called example-function.jar, you can define a Dockerfile as below.

# Use pulsar-functions-java-runner since we pack Java function
FROM streamnative/pulsar-functions-java-runner:2.7.1
# Copy function JAR package into /pulsar directory  
COPY example-function.jar /pulsar/

With the Dockerfile, you can build the function image and push into image registry (like Docker Hub, or any private registry)

Submit to Function Mesh

Assume we build the image from previous step as streamnative/example-function-image:latest, and pushed into Docker Hub already.

Below is a sample CRD to create a Java function from docker image.

apiVersion: compute.functionmesh.io/v1alpha1
kind: Function
metadata:
  name: java-function-sample
  namespace: default
spec:
  image: streamnative/example-function-image:latest # using function image here
  className: org.apache.pulsar.functions.api.examples.ExclamationFunction
  forwardSourceMessageProperty: true
  maxPendingAsyncRequests: 1000
  replicas: 1
  maxReplicas: 5
  logTopic: persistent://public/default/logging-function-logs
  input:
    topics:
    - persistent://public/default/java-function-input-topic
    typeClassName: java.lang.String
  output:
    topic: persistent://public/default/java-function-output-topic
    typeClassName: java.lang.String
  pulsar:
    pulsarConfig: "test-pulsar"
  java:
    jar: /pulsar/example-function.jar # the package location in image
    jarLocation: "" # leave empty since we will not download package from Pulsar Packages

When you use Function Mesh with self build images, you need to define the executable location within CRD yaml file, below are the settings for different function runtime.

  java: # Java runtime
    jar: example-function.jar
  python: # Python runtime
    py: exclamation_function.py
  golang: # Golang runtime
    go: go_func_executable

Scripts

build.sh is a bash script used to build base, Java, Python, and Golang runner locally.

Samples

function-samples has a sample Dockerfile that copies /pulsar/examples from pulsar-all into java-runner, which can be used to test Function Mesh.