Skip to content

feat(experimentation): add the Databricks warehouse provider - #8703

Draft
Zaimwa9 wants to merge 3 commits into
feat/databricks-warehouse-providerfrom
feat/databricks-warehouse-connection
Draft

Zaimwa9 wants to merge 3 commits into
feat/databricks-warehouse-providerfrom
feat/databricks-warehouse-connection

Conversation

@Zaimwa9

@Zaimwa9 Zaimwa9 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor
  • I have read the Contributing Guide.
  • I have added information to docs/ if required so people know about the feature.
  • I have filled in the "Changes" section below.
  • I have filled in the "How did you test this code" section below.

Changes

Second half of the Databricks provider, stacked on #8700 (the statement client). Adds Databricks as a bring-your-own warehouse for experiment results, behind the databricks_warehouse flag.

  • DatabricksWarehouse: config and credentials validation, verify, event name and stats lookups, exposure buckets and results, over the client from feat(experimentation): add a Databricks statement client #8700.
  • Config: host, workspace_id, region, warehouse_id, catalog, schema (default flagsmith_exp). Pasted URLs and HTTP paths are normalised, and the workspace ID can come from a pasted ?o= URL. Verify checks the workspace ID against the x-databricks-org-id response header.
  • DatabricksDialect plugs into the shared ResultsQueryBuilder. Time buckets don't depend on the session timezone.
  • Migration 0017 adds the warehouse type. The databricks_warehouse flag gates creating a Databricks connection, switching an existing one to it, and testing unsaved Databricks details.

How did you test this code?

Unit tests cover the provider, dialect, serializers and views at 100% diff coverage, including the expected SQL for every query.

Live QA against a Databricks trial workspace, using the same events as a ClickHouse baseline: 18/18 API calls returned identical results. A wrong workspace ID was also checked live.

Separately, the warehouse-delivery work ran a pipeline (ingestion → Kafka → Zerobus) into the trial workspace, then compute_experiment_exposures and compute_experiment_results on an earlier revision of this branch, without the two review fixes. The computed totals matched the generated data. The results API endpoint was not exercised over pipeline-delivered rows.

@vercel

vercel Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
docs Ignored Ignored Preview Oct 8, 2026 3:20pm UTC
flagsmith-frontend-preview Ignored Ignored Preview Oct 8, 2026 3:20pm UTC
flagsmith-frontend-staging Ignored Ignored Preview Oct 8, 2026 3:20pm UTC

Request Review

@coderabbitai

coderabbitai Bot commented Oct 8, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added api Issue related to the REST API feature New feature or request labels Oct 8, 2026
@codecov

codecov Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.86%. Comparing base (7b1f778) to head (e3de701).

Additional details and impacted files
@@                         Coverage Diff                          @@
##           feat/databricks-warehouse-provider    #8703    +/-   ##
====================================================================
  Coverage                               98.85%   98.86%            
====================================================================
  Files                                    1668     1669     +1     
  Lines                                   69013    69332   +319     
====================================================================
+ Hits                                    68223    68542   +319     
  Misses                                    790      790            

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request docs Documentation updates labels Oct 8, 2026
@Zaimwa9

Zaimwa9 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

@themis-blindfold review

@themis-blindfold

Copy link
Copy Markdown
Contributor

⚖️ Themis review: ✅ Ship it

TL;DR: The Databricks provider is correctly registered, feature-gated at the connection API, and follows the existing warehouse read, verification, caching, and error-reporting paths. The completed API unit suites and documentation checks passed; no blocking review issues found.

Area Score
🎯 Correctness 5/5
🧪 Test coverage 5/5
📐 Code quality 5/5
🚀 Product impact 4/5
📝 Walkthrough
  • Warehouse connection API - adds the Databricks type, migration, generated schema, and organisation-scoped feature gate for create, type changes, and unsaved connection tests.
  • Databricks provider - validates and normalises configuration, verifies service-principal access, and reads event metadata, exposure buckets, and experiment aggregates.
  • SQL dialect - supplies Databricks parameter binding, aggregation, array unnesting, and UTC-epoch bucket expressions for the shared query builders.
  • Tests - cover configuration, credentials, verification errors, feature-gate behaviour, generated SQL, caching, and result conversion.
🧪 How to verify
  1. Run cd api && pytest tests/unit/experimentation/warehouses/test_databricks.py.
  2. Run cd api && pytest tests/unit/experimentation/test_views.py tests/unit/experimentation/test_serializers.py.
  3. Enable experimentation_warehouse_connection and databricks_warehouse for a test organisation, then create and test a connection with valid service-principal credentials.
  4. Repeat with the Databricks flag disabled and confirm create, type-switch, and unsaved connection tests return validation errors.
  5. Create an experiment with occurrence, count, sum, and mean metrics and confirm the exposure and result refreshes match the warehouse rows.
    Automate: retain the existing provider and view unit suites in the API test matrix.

Product take: Solid capability expansion for customers running experimentation data in Databricks. The targeted flag makes the rollout appropriately controlled.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

Databricks joins the warehouse club without upsetting the query bouncer. · reviewed at e3de701

This branch was successfully deployed

1 active (outdated) deployment
Preview – docs — b3e64ead Deployed Oct 8, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api Issue related to the REST API feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant