[history server][storage] Add Azure Blob Storage support

[ikchifo]

Why are these changes needed?

Adds Azure Blob Storage support for the History Server, enabling Azure users to persist Ray cluster logs and events. Supports both connection string authentication (for dev/test) and Microsoft Entra Workload Identity (for production AKS deployments).

Configuration

Set --runtime-class-name=azureblob to enable this module.

Config via environment variables or /var/collector-config/data:

{"azureContainer": "", "azureConnectionString": "", "azureAccountURL": ""}

Variable	Description
AZURE_STORAGE_CONNECTION_STRING	Connection string (for dev/test)
AZURE_STORAGE_ACCOUNT_URL	Account URL (for workload identity)
AZURE_STORAGE_CONTAINER	Container name (default: ray-historyserver)
AZURE_STORAGE_AUTH_MODE	Auth mode: connection_string, workload_identity, default

Related issue number

Closes #4398

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

End-to-End Testing on AKS with Workload Identity

This feature was tested on Azure Kubernetes Service (AKS) using Microsoft Entra Workload Identity for secure, credential-free authentication to Azure Blob Storage.

1. Infrastructure Setup

Environment Variables

export RESOURCE_GROUP="rg-kuberay-wi-test"
export LOCATION="eastus"
export CLUSTER_NAME="aks-kuberay-wi-test"
export IDENTITY_NAME="id-kuberay-historyserver"
export STORAGE_ACCOUNT_NAME="stkuberayhistory$(date +%s | tail -c 6)"
export CONTAINER_NAME="ray-historyserver"
export SERVICE_ACCOUNT_NAME="ray-collector"
export SERVICE_ACCOUNT_NAMESPACE="default"
export ACR_NAME="kuberaycollector"

# AKS Cluster with Workload Identity
az group create --name "${RESOURCE_GROUP}" --location "${LOCATION}"

az aks create \
  --resource-group "${RESOURCE_GROUP}" \
  --name "${CLUSTER_NAME}" \
  --node-count 2 \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --generate-ssh-keys

az aks get-credentials --resource-group "${RESOURCE_GROUP}" --name "${CLUSTER_NAME}"

# User-Assigned Managed Identity

az identity create \
  --name "${IDENTITY_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --location "${LOCATION}"

export IDENTITY_CLIENT_ID=$(az identity show --name "${IDENTITY_NAME}" \
  --resource-group "${RESOURCE_GROUP}" --query "clientId" -o tsv)
export IDENTITY_PRINCIPAL_ID=$(az identity show --name "${IDENTITY_NAME}" \
  --resource-group "${RESOURCE_GROUP}" --query "principalId" -o tsv)

# Storage Account with RBAC

az storage account create \
  --name "${STORAGE_ACCOUNT_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --location "${LOCATION}" \
  --sku Standard_LRS

export STORAGE_ACCOUNT_ID=$(az storage account show \
  --name "${STORAGE_ACCOUNT_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --query "id" -o tsv)

az role assignment create \
  --assignee-object-id "${IDENTITY_PRINCIPAL_ID}" \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope "${STORAGE_ACCOUNT_ID}"

# Federated Identity Credential

export AKS_OIDC_ISSUER=$(az aks show \
  --resource-group "${RESOURCE_GROUP}" \
  --name "${CLUSTER_NAME}" \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

az identity federated-credential create \
  --name "fc-ray-collector" \
  --identity-name "${IDENTITY_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --issuer "${AKS_OIDC_ISSUER}" \
  --subject "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}" \
  --audience "api://AzureADTokenExchange"

# Create ACR and attach to AKS
az acr create --resource-group "${RESOURCE_GROUP}" --name "${ACR_NAME}" --sku Basic
az aks update --resource-group "${RESOURCE_GROUP}" --name "${CLUSTER_NAME}" --attach-acr "${ACR_NAME}"

# Build collector image
cd historyserver
az acr build --registry "${ACR_NAME}" --image collector:v0.1.0 . -f Dockerfile.collector

# Install KubeRay Operator

helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm install kuberay-operator kuberay/kuberay-operator --version 1.5.1

# Create ServiceAccount with Workload Identity

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ray-collector
  namespace: default
  annotations:
    azure.workload.identity/client-id: "${IDENTITY_CLIENT_ID}"
EOF

2. Setup RayCluster with Azure Storage-backed Collector Sidecar

kubectl apply -f - <<EOF
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-wi-test
spec:
  headGroupSpec:
    rayStartParams:
      dashboard-host: 0.0.0.0
    template:
      metadata:
        labels:
          azure.workload.identity/use: "true"
      spec:
        serviceAccountName: ray-collector
        containers:
        - name: ray-head
          image: rayproject/ray:2.50.1
          env:
          - name: RAY_enable_ray_event
            value: "true"
          - name: RAY_enable_core_worker_ray_event_to_aggregator
            value: "true"
          - name: RAY_DASHBOARD_AGGREGATOR_AGENT_EVENTS_EXPORT_ADDR
            value: "http://localhost:8084/v1/events"
          - name: RAY_DASHBOARD_AGGREGATOR_AGENT_EXPOSABLE_EVENT_TYPES
            value: "TASK_DEFINITION_EVENT,TASK_LIFECYCLE_EVENT,ACTOR_TASK_DEFINITION_EVENT,TASK_PROFILE_EVENT,DRIVER_JOB_DEFINITION_EVENT,DRIVER_JOB_LIFECYCLE_EVENT,ACTOR_DEFINITION_EVENT,ACTOR_LIFECYCLE_EVENT,NODE_DEFINITION_EVENT,NODE_LIFECYCLE_EVENT"
          resources:
            limits:
              cpu: "2"
              memory: 4Gi
            requests:
              cpu: "500m"
              memory: 1Gi
          lifecycle:
            postStart:
              exec:
                command:
                - /bin/sh
                - -c
                - |
                  while true; do
                    nodeid=\$(ps -ef | grep 'raylet.*--node_id' | grep -v grep | sed -n 's/.*--node_id=\([^ ]*\).*/\1/p' | head -1)
                    if [ -n "\$nodeid" ]; then
                      echo "\$nodeid" > /tmp/ray/raylet_node_id
                      break
                    fi
                    sleep 1
                  done
          volumeMounts:
          - name: ray-logs
            mountPath: /tmp/ray
        - name: collector
          image: kuberaycollector.azurecr.io/collector:v0.1.0
          env:
          - name: AZURE_STORAGE_ACCOUNT_URL
            value: "https://${STORAGE_ACCOUNT_NAME}.blob.core.windows.net"
          - name: AZURE_STORAGE_CONTAINER
            value: "ray-historyserver"
          command:
          - collector
          - --role=Head
          - --runtime-class-name=azureblob
          - --ray-cluster-name=raycluster-wi-test
          - --events-port=8084
          volumeMounts:
          - name: ray-logs
            mountPath: /tmp/ray
        volumes:
        - name: ray-logs
          emptyDir: {}
EOF

# Wait for pod to be ready
kubectl wait --for=condition=ready pod -l ray.io/cluster=raycluster-wi-test --timeout=120s

# Submit a Ray job to generate activity
kubectl exec -it $(kubectl get pods -l ray.io/cluster=raycluster-wi-test -o jsonpath='{.items[0].metadata.name}') \
  -c ray-head -- python -c "import ray; ray.init(); print(ray.cluster_resources())"

# Delete the cluster to trigger log upload
kubectl delete raycluster raycluster-wi-test

# Check Azure Blob Storage for uploaded files
az storage blob list \
  --account-name "${STORAGE_ACCOUNT_NAME}" \
  --container-name ray-historyserver \
  --auth-mode login \
  --output table

# Cleanup
az group delete --name "${RESOURCE_GROUP}" --yes --no-wait

3. Screenshotss

Azure Storage Browser (with the uploaded session files from the Collector)

Logs when new events were detected

Demo Video

azure_blob_recording_compressed.mp4

Note: E2E tests using Azurite will be added in a follow-up PR to keep this one focused on the core implementation.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

[ikchifo]

Ready for review. @Future-Outlier - Please take a look when you get the chance.
cc: @JiangJiaWei1103 @rueian as I a saw you've recently interacted with code for the history server.

[Future-Outlier]

thank you for this PR!
do you mind provide screenshots or a video to prove this really works in Azure Blob Storage?

[ikchifo]

thank you for this PR! do you mind provide screenshots or a video to prove this really works in Azure Blob Storage?

@Future-Outlier Updated the PR description with a video showing:

• an initially empty Azure Blob Container (S3 bucket equivalent)
• the collector sidecar logs showing that the azureblob runtimeclass is being used at start-up
• submitting a test job to the RayCluster
• showing events being emitted in the collecter sidecar after the job is submitted
• delete the RayCluster to trigger an upload of the history-server files
• show files uploaded in the Blob container

The manifests and additional details used to run the test can are also in the PR's description