Add Lakebase Autoscaling support for Databricks Asset Bundles #4423
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements postgres_project as a new bundle resource type with full support for direct deployment and Terraform conversion. Includes config resource, direct deployment handler, and Terraform converter. Key features: - Uses display_name field for bundle summary output - Supports project_id as immutable identifier (triggers recreation on change) - Handles long-running operations with waiter pattern - Custom state wrapper for fields not in SDK types Also adds postgres_branch config resource (implementation in progress). Includes three acceptance tests: - single-project: basic deployment happy path - without-project-id: validates required field error handling - recreate-on-id-change: verifies immutable field recreation behavior Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Implements postgres_branch as a bundle resource with full support for direct deployment and Terraform conversion. Branches represent isolated database environments with copy-on-write storage within a postgres project. Key features: - Hierarchical relationship: branches belong to projects - Immutable identifiers: branch_id and parent trigger recreation on change - Long-running operations with waiter pattern - Custom state wrapper for creation-only fields (branch_id) Includes single-branch acceptance test: - Deploys project and branch together - Verifies branch creation and READY state - Normalizes non-deterministic branch UIDs and LSN values Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Adds support for Lakebase v2 (Postgres) endpoints in Databricks Asset Bundles:
- Config resource: bundle/config/resources/postgres_endpoint.go
- Implements ConfigResource interface with hierarchical URL parsing
- Supports endpoint_id field for user-specified IDs
- Direct deployment: bundle/direct/dresources/postgres_endpoint.go
- State wrapper for creation-time fields
- Handles long-running operations with proper waiting
- Creates endpoints with parent branch reference
- Terraform converter: bundle/deploy/terraform/tfdyn/convert_postgres_endpoint.go
- Normalizes endpoint spec for Terraform provider
- Registration:
- bundle/config/resources.go: PostgresEndpoints map
- bundle/direct/dresources/all.go: Added to SupportedResources
- bundle/direct/dresources/resources.yml: Immutable fields configuration
- Acceptance test: acceptance/bundle/resources/postgres_endpoints/single-endpoint/
- Tests full stack: project + branch + endpoint deployment
- Verifies endpoint creation with ENDPOINT_TYPE_READ_WRITE
- Includes cleanup delay for backend reconciliation
Key implementation details:
- Endpoint types use full enum names: ENDPOINT_TYPE_READ_WRITE, ENDPOINT_TYPE_READ_ONLY
- Immutable fields: endpoint_id, parent, spec.endpoint_type
- Hierarchical naming: projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id}
- Endpoint UIDs follow pattern: ep-{adjective}-{noun}-{alphanum}
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Adds comprehensive test coverage for required fields and immutable field behavior: **Branches (2 new tests):** - without-branch-id: Validates that branch_id is required - recreate-on-id-change: Verifies branch_id immutability triggers recreation **Endpoints (2 new tests):** - without-endpoint-id: Validates that endpoint_id is required - recreate-on-endpoint-type-change: Verifies endpoint_type immutability **Immutable Fields Documentation:** Updated bundle/direct/dresources/resources.yml to document additional immutable fields: - Branch source fields (source_branch, source_branch_lsn, source_branch_time) These define the point-in-time data source and cannot be changed after creation - Endpoint type (endpoint_type) already documented **Test Coverage:** All 9 postgres tests now passing: - 3 project tests (5.66s, 19.92s, 27.42s) - 3 branch tests (16.95s, 19.91s, 33.74s) - 3 endpoint tests (12.85s, 29.36s, 68.89s) Total test time: ~72s (parallel execution) **Key Findings:** 1. Changing endpoint_type triggers recreation as expected 2. Test verifies recreation via DELETE/POST in request logs 3. Cannot verify "old endpoint gone" when reusing same endpoint_id (both old and new have same hierarchical name) 4. Branch source fields are immutable by nature (define data lineage) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Refactor postgres resources to embed *Spec types directly instead of
embedding the full resource types. This eliminates the unnecessary "spec:"
nesting level in bundle YAML configuration, making postgres resources
consistent with other DAB resources (jobs, clusters, pipelines, etc.).
Changes:
- PostgresProject embeds postgres.ProjectSpec (not postgres.Project)
- PostgresBranch embeds postgres.BranchSpec (not postgres.Branch)
- PostgresEndpoint embeds postgres.EndpointSpec (not postgres.Endpoint)
- Updated PrepareState() to reconstruct API types from flattened config
- Updated all 9 acceptance tests to remove spec: nesting
- Regenerated bundle schema
User-facing config changes from:
postgres_projects:
my_project:
spec:
display_name: "Test"
To:
postgres_projects:
my_project:
display_name: "Test"
All acceptance tests passing.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The Name field is output-only but was included in StateType structs via
embedding, causing variable references like ${...name} to resolve to
empty strings during planning instead of delaying until after API response.
Restructured StateType to only include user-configurable fields.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add postgres resource types to GroupToTerraformName mapping to enable
reference resolution (e.g., ${resources.postgres_projects.x.name}).
Update converters to restructure flattened bundle config fields into
nested spec blocks expected by the Terraform provider schema.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The SDK's duration.Duration type uses custom JSON marshaling to represent
durations as strings (e.g., "300s", "604800s"). Previously, the dyn/convert
package treated it as a struct, causing duration fields to be silently
dropped with "expected map, found string" warnings.
This change adds special handling for duration.Duration in:
- normalize.go: Treat as string during normalization
- to_typed.go: Convert string to duration using JSON unmarshaling
- from_typed.go: Convert duration to string using JSON marshaling
- from_type.go: Represent as string type in JSON schema
Also updates postgres acceptance tests to use protobuf duration format
("604800s" instead of "7d") which is what the SDK expects.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Track filtered postgres API requests in out.requests.json files for test verification while keeping full request logs ignored. This enables validation of postgres resource CRUD operations in acceptance tests. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Implement full CRUD operations for postgres projects, branches, and endpoints in the testserver. This enables local acceptance testing without requiring cloud connectivity. Key changes: - Add postgres.go with handlers for projects, branches, and endpoints - Add postgres_test.go with unit tests for the handlers - Register postgres routes in handlers.go - Fix WorkspaceDelete to also remove directories (not just files) - Update test.toml files with correct regex patterns for UIDs and LSNs - Add conditional sleep in endpoint recreate test for cloud mode only Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
After endpoint create/update operations complete, poll the endpoint status until PendingState is empty. Also retry DELETE operations that fail with 409 ABORTED due to reconciliation still in progress. This ensures subsequent operations don't fail because the endpoint is still reconciling internally. - Add waitForReconciliation helper to poll until PendingState is empty - Add retry loop in DoDelete for 409 ABORTED errors - Remove sleep commands from acceptance tests (no longer needed) - Deduplicate requests in test output to handle retry variability Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add postgres resource field definitions to refschema output - Update sql_warehouses destroy output to reflect no active deployment Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Implements test cases for postgres_project, postgres_branch, and postgres_endpoint resources in the no_drift test suite. The key fix is in RemapState functions: extract resource IDs from hierarchical names and populate spec fields from status (effective values) instead of the sparse spec returned by the API. This ensures state comparison accurately reflects deployed resources and prevents false drift detection. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Implements conversion support for SDK native types across the dyn package conversion layer (FromTyped, ToTyped, Normalize). Supports three SDK types that use custom JSON marshaling: - duration.Duration: Protobuf duration format (e.g., "300s") - time.Time: RFC3339 timestamp format (e.g., "2023-12-25T10:30:00Z") - fieldmask.FieldMask: Comma-separated paths (e.g., "name,age,email") The implementation uses a generic approach with all SDK-specific code in sdk_native_types.go and sdk_native_types_test.go. All SDK type imports are prefixed with "sdk" for clarity. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Adds comprehensive round-trip tests using actual SDK types from the postgres service (BranchSpec and UpdateBranchRequest) to verify that serialization and deserialization work correctly in real-world usage. Tests cover: - BranchSpec with time.Time and duration.Duration fields - UpdateRequest with fieldmask.FieldMask field - Full round-trip FromTyped -> dyn.Value -> ToTyped - Normalization with SDK types Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Update test code to follow linter recommendations including struct field alignment and using assert.True for boolean assertions. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
1. Clarified that all SDK native types marshal to JSON strings with
improved comments explaining the conversion process.
2. Fixed JSON encoding in toTypedSDKNative to use json.Marshal instead
of fmt.Sprintf("%q") for proper JSON string literal creation.
3. Fixed TestFromTypedDurationPointer to actually test with a pointer
instead of a value.
4. Added comprehensive table-driven roundtrip tests with reusable
equality check functions for all three SDK types (Duration, Time,
FieldMask) testing both value and pointer variants.
5. Removed redundant basic tests that are now covered by the roundtrip
tests, while keeping edge case tests for nil, zero, error cases,
normalization, and in-struct scenarios.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Using .id ensures that parent references resolve to stable identifiers rather than computed name values, which prevents false drift detection and unnecessary resource recreation in dependent resources. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Removed 18 tests that are now covered by the comprehensive roundtrip tests. The remaining tests focus on unique edge cases: - Nil value handling (one per type) - Zero values - Error cases (invalid format, wrong type) - Empty values (FieldMask) - Normalize with nil (important edge case) - End-to-end tests with real postgres types This keeps test coverage while improving maintainability. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Tests that display_name can be updated without recreating the project. Follows the pattern: create → verify idempotency → update → verify → restore → verify → destroy. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Converted 13 individual edge case tests into 3 table-driven tests: - TestSDKNativeTypesNilValues: Tests nil handling for FromTyped and Normalize across all 3 SDK types (6 subtests) - TestSDKNativeTypesErrors: Tests error handling for ToTyped with invalid inputs across all 3 types (4 subtests) - TestSDKNativeTypesSpecialCases: Tests zero values and empty FieldMask (3 subtests) Final test suite for SDK native types: - 1 comprehensive roundtrip test (18 subtests) - 3 edge case table tests (13 subtests) - 3 end-to-end tests with real postgres types This improves maintainability while preserving full test coverage. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This test verifies that the is_protected field on a postgres branch can be updated in place without recreating the branch. The test covers four stages: - create: Initial deployment with is_protected: false - no_change: Verify idempotency (no unnecessary updates) - update: Change is_protected to true (PATCH request) - restore: Change is_protected back to false (PATCH request) Each stage captures both the plan output (focused on the branch resource) and the HTTP requests made during deployment, allowing verification that updates use PATCH requests rather than recreate (DELETE + POST). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This test verifies that the autoscaling_limit_max_cu field on a postgres endpoint can be updated in place without recreating the endpoint. The test covers four stages: - create: Initial deployment with autoscaling_limit_max_cu: 8 - no_change: Verify idempotency (no unnecessary updates) - update: Change autoscaling_limit_max_cu to 4 (PATCH request) - restore: Change autoscaling_limit_max_cu back to 8 (PATCH request) Each stage captures both the plan output (focused on the endpoint resource) and the HTTP requests made during deployment, allowing verification that updates use PATCH requests rather than recreate (DELETE + POST). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…utput Apply the same structural improvements as the other update tests: - Add focused plan output using jq for the project resource - Rename "idempotency" to "no_change" for clarity - Update file naming to out.requests.<stage>.json and out.plan.<stage>.json - Capture plan output at all four stages (create, no_change, update, restore) This allows verification of the exact changes being made at each stage and confirms that updates use PATCH requests rather than recreate. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Use `sync.paths: []` instead of `sync.exclude: ["*"]` to prevent the "no files to sync" warning. An empty paths array is treated as explicitly set, so no default is applied and no warning is shown. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When updating postgres resources (projects, branches, endpoints), the code was using update_mask="*" which requires ALL spec fields to be provided in the request. The real cloud API is stricter than the test server and rejects updates with missing fields. Changed to derive the precise update_mask based on which fields actually changed, using the Changes map that already tracks field-level changes. Code changes: - Added collectUpdatePaths() in dresources/util.go to extract paths with action=Update from the Changes map - Modified DoUpdate() in postgres_project.go to use precise field paths - Modified DoUpdate() in postgres_branch.go to use precise field paths - Modified DoUpdate() in postgres_endpoint.go to use precise field paths Test changes: - Fixed postgres_branches/update-protected test by adding no_expiry field and updating branch UID replacement pattern to be more flexible - Fixed postgres_endpoints/update-autoscaling test by updating endpoint UID and branch UID replacement patterns - Updated expected test outputs to reflect new precise update_mask behavior All postgres acceptance tests now pass against real cloud instance: - postgres_projects: 4/4 passing - postgres_branches: 4/4 passing - postgres_endpoints: 4/4 passing Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Moved common Repl patterns from individual test.toml files to parent test.toml files in each postgres category (branches, endpoints, projects). This eliminates duplication of patterns like URL cleanup, UID normalization, and LSN values across multiple tests. Individual tests now only define test-specific patterns (e.g., TraceId in recreate-on-id-change tests). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Postgres resources (postgres_projects, postgres_branches, postgres_endpoints) use the `name` attribute as their identifier in terraform state, not `id`. Added these types to the switch case that reads from instance.Attributes.Name. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
pietern
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
The CloudSlow flag was originally added to mitigate quota limits, but is no longer needed for these tests. Also removes unnecessary header comments. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Previously, bundle summary showed "(not deployed)" for all resources without a URL, even if the resource was successfully deployed. This was misleading for resources like postgres projects, branches, and endpoints that don't have web UI URLs. Now the output distinguishes between: - "(not deployed)" - resource has no ID (not yet deployed) - "(not available)" - resource has ID but no URL (deployed but no web UI) - URL - resource has both ID and URL (deployed with web UI) Changes: - Add GetID() method to ConfigResource interface and BaseResource - Update bundle summary template to check ID before URL - Update acceptance tests for postgres resources Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
pietern
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
pietern
requested review from
andrewnester,
anton-107 and
shreyas-goenka
as code owners
This reverts commit 8c4ad10.
pietern
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
denik
reviewed
Feb 3, 2026
| }, | ||
| "spec.suspend_timeout_duration": { | ||
| "action": "skip", | ||
| "reason": "empty_struct", |
Contributor
There was a problem hiding this comment.
It's odd to see empty_struct and value is "300s". I understand that it's struct underneath but for users of plan it's confusing. There also might be unintended effects of this check.
Given that SDK native types require special handling everywhere, we might need support for that in libs/structs/structdiff as well.
Could you add a couple tests that show what action / reason is taken in the plan?
- update suspend_timeout_duration field locally
- different value: 300s -> 600s
- same value, different string: 120s -> 2m
- remove field completely
- add field
- update suspend_timeout_duration field remotely. I know it's ignored but is still handled by structdiff, we should ensure it does not error there.
pietern
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Changes the postgres resource ID fields (project_id, branch_id, endpoint_id) from optional to required, matching the API's behavior. Previously these fields were marked as omitempty but the API requires them for resource creation. Changes: - Mark project_id, branch_id, endpoint_id, and parent fields as required in bundle config - Remove manual validation from direct deployment code (now handled by config validation) - Update generated required_fields.go and JSON schema - Add validation in test server to return 400 errors matching real API behavior - Update acceptance test outputs to reflect new warning messages and error responses The test server now properly validates required ID fields and returns the same error message as the real API: "Field 'X' is required, expected non-default value (not \"\")!" All acceptance tests pass both locally and against real AWS workspace. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
pietern
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Collaborator
|
Commit: f6c4af2
25 interesting tests: 8 KNOWN, 8 FAIL, 5 SKIP, 4 RECOVERED
Top 50 slowest tests (at least 2 minutes):
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
This PR adds support for Lakebase Autoscaling resources, enabling declarative configuration of projects, branches, and endpoints. For more information about the product, see https://docs.databricks.com/aws/en/oltp/projects/.
These APIs use a "spec" field to capture the user intent in requests and a "status" field to contain the applied values in responses. To match DABs UX patterns, this PR embeds the "spec" type at the top level of the resource.
Example
Example configuration:
After deploying this, you can access the endpoint using the
psqlcommand (see #4399):Note: the default database name assumed by the command is
databricks_postgres. If this doesn't work, you can try--dbname lakebase.Tests
Known limitations