Kubernetes Framework for deploying ML and GenAI Apache Beam workflows

Explore Beamstack Documentation »
Join Beamstack Community »
Explore Blogs »
Join Discord Channel

Beamstack Features

Simplified ML Workflow Deployment:

• Beamstack simplifies the deployment of machine learning workflows on Kubernetes.

Holistic Solution:

• Beamstack offers an all-encompassing solution for managing machine learning pipelines, data processing workflows, and deployment infrastructure.

Abstraction Layers:

• Beamstack introduces abstraction layers that streamline the deployment of Apache Beam Pipelines in Kubernetes.

Leveraged Kubernetes Custom Resource Definitions (CRDs):

• Beamstack uses Kubernetes CRDs to extend the Kubernetes API, allowing smooth integration of machine learning-specific resources.

Seamless Provisioning of Spark and Flink Clusters in Kubernetes:

• Beamstack incorporates features that spin up spark and flink clusters in Kubernetes for running Apache Beam Jobs

Easily Monitor and Visualize Deployed Workflows:

• Beamstack seamlessly integrates with Prometheus and Grafana to visualize the states of the deployed workflows in real time.

---

Architecture

---

Installation

1. Prerequisite

To be able to work with beamstack-cli, an active Kubernetes cluster is required. Before you begin setup a local Kubernetes cluster using minikube

2. Start Kubernetes cluster:

minikube delete && minikube start \
    --kubernetes-version=v1.23.0 \
    --memory=6g --bootstrapper=kubeadm \
    --extra-config=kubelet.authentication-token-webhook=true \
    --extra-config=kubelet.authorization-mode=Webhook \
    --extra-config=scheduler.bind-address=0.0.0.0 \
    --extra-config=controller-manager.bind-address=0.0.0.0

3. download the helper scrip:

wget https://raw.githubusercontent.com/BeamStackProj/beamstack-cli/main/get-beamstack.sh

4. Install beamstack-cli:

sh get-beamstack.sh

---

5. Verify beamstack installation:

beamstack

---

Initializing your kubernetes cluster

To configure your Kubernetes cluster for running Beam YAML pipelines and accessing other BeamStack commands, use the init command in BeamStack.

• Step 1: View Available Flags Start by viewing the available flags and options for the init command:

beamstack init --help

• Step 2: Initialize BeamStack with Your Desired Configuration Once you've reviewed the options, initialize BeamStack with the configuration that suits your needs:

beamstack init -me

---

Components of Beamstack

• Beamstack CLI
• Beamstack Custom Transforms
• Apache Beam YAML
• Kubernetes
• Monitoring

---

Beamstack Technology

---

💪 Support, Contribution, and Community

👥 Community

Get updates on Beamstack's development and chat with project maintainers, contributors, and community members

• Visit the Community Page
• Raise feature requests, suggest enhancements, and report bugs in our GitHub Issues
• Articles, How-Tos, Tutorials - Beamstack Blogs

🤝 Contribute

Take a look at our contributing guidelines for information on how to open issues, adhere to coding standards, and understand our development processes. We greatly value your contribution.