feat(sql-setup): system-update replacement for mysql/postgres setup #15044

david-leifker · 2025-10-17T16:32:03Z

SQL Setup Upgrade System

Overview

This PR introduces a comprehensive SQL setup upgrade system for DataHub that enables automated database initialization, table creation, and user management for both MySQL and PostgreSQL databases. The system supports both traditional authentication and IAM-based authentication, making it suitable for various deployment scenarios.

Key Features

🗄️ Database Support

MySQL: Full support with SSL configuration options
PostgreSQL: Full support with database creation capabilities
Auto-detection: Automatically detects database type from JDBC URL configuration

🔧 Core Functionality

Table Creation: Automated creation of metadata_aspect_v2 table with proper schema
Database Creation: PostgreSQL database creation with proper permissions
User Management: Optional creation of application users with configurable permissions
CDC Support: Optional Change Data Capture (CDC) user creation for data streaming
IAM Authentication: Support for AWS IAM-based authentication for user creation

Environment Variables

Core Configuration

Variable	Default	Description
`CREATE_TABLES`	`true`	Enable table creation
`CREATE_DB`	`true`	Enable database creation (PostgreSQL only)
`CREATE_USER`	`false`	Enable user creation
`CDC_MCL_PROCESSING_ENABLED`	`false`	Enable CDC user creation
`CDC_USER`	`datahub_cdc`	CDC username
`CDC_PASSWORD`	`datahub_cdc`	CDC password
`IAM_ROLE`	-	IAM role for user creation (required if IAM enabled)

Database Connection

The system automatically extracts connection details from Spring Ebean configuration:

Host/Port: Extracted from JDBC URL
Database Name: Extracted from JDBC URL
Credentials: Uses ebean.username and ebean.password
IAM Auth: Uses ebean.useIamAuth setting

Usage Examples

Basic Table Creation

# MySQL
./gradlew runSqlSetupMysql

# PostgreSQL
./gradlew runSqlSetupPostgres

With User Creation

# Traditional authentication
MYSQL_USERNAME=datahub MYSQL_PASSWORD=datahub CREATE_USER=true ./gradlew runSqlSetupMysql

# IAM authentication
MYSQL_USERNAME=datahub IAM_ROLE=DataHubRole CREATE_USER=true ./gradlew runSqlSetupMysql

With CDC Support

CDC_MCL_PROCESSING_ENABLED=true CDC_USER=datahub_cdc CDC_PASSWORD=datahub_cdc ./gradlew runSqlSetupMysql

aikido-pr-checks · 2025-10-17T16:33:00Z

datahub-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/CreateUsersStepTest.java

+    mockSetupArgs.createUser = true;
+    mockSetupArgs.iamAuthEnabled = false;
+    mockSetupArgs.createUserUsername = "testuser";
+    mockSetupArgs.createUserPassword = "testpass";


Exposed secret in datahub-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/CreateUsersStepTest.java - low severity
Identified a HashiCorp Terraform password field, risking unauthorized infrastructure configuration and security breaches.
^{View details in Aikido Security}

aikido-pr-checks · 2025-10-17T16:33:01Z

datahub-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/SqlSetupArgsTest.java

+    args.cdcUser = "custom_cdc";
+    args.cdcPassword = "custom_password";
+    args.createUserUsername = "testuser";
+    args.createUserPassword = "testpass";


Exposed secret in datahub-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/SqlSetupArgsTest.java - low severity
Identified a HashiCorp Terraform password field, risking unauthorized infrastructure configuration and security breaches.
^{View details in Aikido Security}

aikido-pr-checks · 2025-10-17T16:33:01Z

datahub-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/CreateUsersStepTest.java

+  @Test
+  public void testGetCreateTraditionalUserSqlMysql() throws Exception {
+    mockSetupArgs.createUserUsername = "mysqluser";
+    mockSetupArgs.createUserPassword = "mysqlpass";


Exposed secret in datahub-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/CreateUsersStepTest.java - low severity
Identified a HashiCorp Terraform password field, risking unauthorized infrastructure configuration and security breaches.
^{View details in Aikido Security}

aikido-pr-checks · 2025-10-17T16:33:02Z

...b-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/config/SqlSetupConfigTest.java

+    args.iamAuthEnabled = false;
+    args.createUser = true;
+    args.createUserUsername = null; // Missing createUserUsername
+    args.createUserPassword = "testpass";


Exposed secret in datahub-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/config/SqlSetupConfigTest.java - low severity
Identified a HashiCorp Terraform password field, risking unauthorized infrastructure configuration and security breaches.
^{View details in Aikido Security}

codecov · 2025-10-17T16:48:16Z

Codecov Report

❌ Patch coverage is 79.93139% with 117 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...inkedin/datahub/upgrade/UpgradeCliApplication.java	2.70%	36 Missing ⚠️
...nkedin/datahub/upgrade/sqlsetup/JdbcUrlParser.java	71.25%	11 Missing and 12 partials ⚠️
...in/gms/factory/common/LocalEbeanConfigFactory.java	73.23%	3 Missing and 16 partials ⚠️
...atahub/upgrade/sqlsetup/config/SqlSetupConfig.java	73.91%	14 Missing and 4 partials ⚠️
...ava/com/linkedin/datahub/upgrade/UpgradeUtils.java	57.14%	9 Missing and 3 partials ⚠️
.../java/com/linkedin/datahub/upgrade/UpgradeCli.java	0.00%	3 Missing ⚠️
...din/datahub/upgrade/sqlsetup/CreateTablesStep.java	97.84%	0 Missing and 2 partials ⚠️
.../datahub/upgrade/UpgradeConfigurationSelector.java	0.00%	1 Missing ⚠️
...in/datahub/upgrade/sqlsetup/CreateCdcUserStep.java	98.11%	0 Missing and 1 partial ⚠️
...edin/datahub/upgrade/sqlsetup/CreateUsersStep.java	98.50%	0 Missing and 1 partial ⚠️
... and 1 more

📢 Thoughts on this report? Let us know!

alwaysmeticulous · 2025-10-17T18:35:33Z

✅ Meticulous spotted 0 visual differences across 1001 screens tested: view results.

Meticulous evaluated ~8 hours of user flows against your PR.

_{Expected differences? Click here. Last updated for commit fb48432. This comment will update as new commits are pushed.}

* better support for IAM

abedatahub

I've gotten through most of it. Mostly stylistic feedback so far.

abedatahub · 2025-10-19T16:43:38Z

datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/sqlsetup/SqlSetupResult.java

+
+import lombok.Data;
+
+@Data


Prefer immutable types. Replace with the following?

@Value @Builder @ToString(exclude = "errorMessage")

abedatahub · 2025-10-19T16:44:17Z

datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/sqlsetup/SqlSetupResult.java

+  public int tablesCreated = 0;
+  public int usersCreated = 0;
+  public boolean cdcUserCreated = false;
+  public long executionTimeMs = 0;
+  public String errorMessage;


These should not be public if we've got @Data.

abedatahub · 2025-10-19T16:46:46Z

datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/sqlsetup/SqlSetupResult.java

+  public String errorMessage;
+
+  @Override
+  public String toString() {


Why exclude errorMessage? In any case, lombok can do that for us.

abedatahub · 2025-10-19T16:51:03Z

datahub-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/SqlSetupResultTest.java

+
+import org.testng.annotations.Test;
+
+public class SqlSetupResultTest {


This test file should not exist at all. This is why we use Lombok.

abedatahub · 2025-10-19T17:01:20Z

datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/sqlsetup/SqlSetupArgs.java

+
+import lombok.Data;
+
+@Data


We should prefer @Value. The fields should not be public.

abedatahub · 2025-10-19T18:20:50Z

datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/sqlsetup/CreateTablesStep.java

+  }
+
+  void createDatabaseIfNotExists(String databaseName, DatabaseType dbType) throws SQLException {
+    if (DatabaseType.POSTGRES.equals(dbType)) {


I see these conditionals at a few places. How about extracting an interface and having separate impls for Postgres vs Mysql?

abedatahub · 2025-10-19T18:22:26Z

datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/sqlsetup/SqlSetup.java

+import java.util.List;
+import javax.annotation.Nullable;
+
+public class SqlSetup implements Upgrade {


Add javadoc.

abedatahub · 2025-10-19T18:29:24Z

...b-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/SqlSetupUpgradeConfigTest.java

+import org.testng.annotations.Test;
+
+@TestPropertySource(properties = {"entityService.impl=ebean"})
+public class SqlSetupUpgradeConfigTest {


IMO, this test is pure maintenance burden and should be deleted. It'll never catch real bugs; all it does is give false confidence about test coverage :)

abedatahub · 2025-10-19T18:44:11Z

datahub-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/DatabaseTypeTest.java

+import org.testng.annotations.Test;
+
+/** Unit tests for DatabaseType enum. */
+public class DatabaseTypeTest {


Claude got really intense with unit testing this class. I think this is a serious problem. I asked claude to write down my arguments against over-testing here:

OVER_TESTING_FEEDBACK.md

PTAL and we can discuss.

abedatahub · 2025-10-19T18:45:34Z

datahub-upgrade/src/test/java/com/linkedin/datahub/upgrade/sqlsetup/SqlSetupArgsTest.java

+
+import org.testng.annotations.Test;
+
+public class SqlSetupArgsTest {


Another test class that we should drop.

github-actions bot added docs Issues and Improvements to docs product PR or Issue related to the DataHub UI/UX devops PR or Issue related to DataHub backend & deployment labels Oct 17, 2025

aikido-pr-checks bot reviewed Oct 17, 2025

View reviewed changes

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Oct 17, 2025

vercel bot deployed to Preview October 17, 2025 16:48 View deployment

david-leifker force-pushed the sql-setup-system-update branch from f1eb4f4 to 878e0c0 Compare October 17, 2025 18:30

github-actions bot deployed to datahub-project-web-react (Preview) October 17, 2025 18:34 View deployment

vercel bot deployed to Preview October 17, 2025 18:47 View deployment

feat(sql-setup): system-update replacement for mysql/postgres setup

fb48432

* better support for IAM

david-leifker force-pushed the sql-setup-system-update branch from 878e0c0 to fb48432 Compare October 17, 2025 18:56

github-actions bot deployed to datahub-project-web-react (Preview) October 17, 2025 19:01 View deployment

vercel bot deployed to Preview October 17, 2025 19:13 View deployment

abedatahub reviewed Oct 19, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Oct 19, 2025


		import org.testng.annotations.Test;

		public class SqlSetupResultTest {


		import org.testng.annotations.Test;

		public class SqlSetupArgsTest {

feat(sql-setup): system-update replacement for mysql/postgres setup #15044

Are you sure you want to change the base?

feat(sql-setup): system-update replacement for mysql/postgres setup #15044

Uh oh!

Conversation

david-leifker commented Oct 17, 2025

SQL Setup Upgrade System

Overview

Key Features

🗄️ Database Support

🔧 Core Functionality

Environment Variables

Core Configuration

Database Connection

Usage Examples

Basic Table Creation

With User Creation

With CDC Support

Uh oh!

aikido-pr-checks bot Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aikido-pr-checks bot Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aikido-pr-checks bot Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aikido-pr-checks bot Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alwaysmeticulous bot commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abedatahub left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aikido-pr-checks bot Oct 17, 2025 •

edited

Loading

aikido-pr-checks bot Oct 17, 2025 •

edited

Loading

aikido-pr-checks bot Oct 17, 2025 •

edited

Loading

aikido-pr-checks bot Oct 17, 2025 •

edited

Loading

codecov bot commented Oct 17, 2025 •

edited

Loading

alwaysmeticulous bot commented Oct 17, 2025 •

edited

Loading