Hms standalone rest server with spring boot by difin · Pull Request #6327 · apache/hive

difin · 2026-02-19T15:54:20Z

What changes were proposed in this pull request?

The Standalone REST Catalog Server is reimplemented to use Spring Boot instead of plain Java:

Server framework – Uses Spring Boot with an embedded Jetty server instead of raw servlet wiring.
Health checks – Adds Actuator liveness and readiness probes; readiness verifies HMS connectivity via Thrift.
Observability – Exposes Prometheus metrics for Kubernetes HPA and monitoring.
Configuration – Keeps port and other settings in MetastoreConf but bridges them into Spring (e.g., via system properties) so Spring Boot uses the configured port.
Graceful shutdown – Uses Spring Boot’s shutdown handling with a configurable timeout.
Standalone packaging – Adds a spring-boot-maven-plugin “exec” JAR for running the server as a standalone process.

Why are the changes needed?

Spring Boot improves how the Standalone REST Catalog Server is run and operated:

Kubernetes support – Liveness and readiness probes (/actuator/health/*) let Kubernetes reliably route traffic and restart unhealthy pods. Readiness includes an actual HMS connectivity check instead of a simple config check.
Observability – Prometheus metrics enable HPA, dashboards, and alerting, which is standard for production deployments.
Operational behavior – Graceful shutdown and a well-defined lifecycle reduce the chance of dropped requests during restarts.
Maintainability – Spring Boot replaces custom servlet wiring and configuration, and aligns with common patterns for cloud-native Java services.

Does this PR introduce any user-facing change?

If the standalone REST Catalog server is deployed in Kubernetes:

Liveness/readiness probes – Configure HTTP probes to use the new actuator endpoints:
-- Liveness: httpGet: /actuator/health/liveness
-- Readiness: httpGet: /actuator/health/readiness
Metrics/HPA – Prometheus scraping or custom metrics use /actuator/prometheus.

How was this patch tested?

Integration tests in TestStandaloneRESTCatalogServer where updated to run the Spring Boot standalone HMS rest catalog server, verify liveness and readiness probes, Prometheus metrics.

sonarqubecloud · 2026-02-19T21:54:09Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

deniskuzZ · 2026-02-20T14:29:22Z

	Suppressed: java.lang.NullPointerException: Cannot invoke "org.keycloak.admin.client.Keycloak.close()" because "this.keycloak" is null
		at org.apache.iceberg.rest.extension.OAuth2AuthorizationServer.stop(OAuth2AuthorizationServer.java:181)
		at org.apache.iceberg.rest.extension.HiveRESTCatalogServerExtension.afterAll(HiveRESTCatalogServerExtension.java:124)
		... 1 more
Caused by: java.lang.ClassNotFoundException: jakarta.annotation.Priority

deniskuzZ · 2026-02-20T14:30:48Z

.../qtest-iceberg/src/test/java/org/apache/hadoop/hive/cli/TestStandaloneRESTCatalogServer.java

+    webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
+    properties = {
+        "spring.main.allow-bean-definition-overriding=true",
+        "spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration"


do we need these properties?

this property is needed: "spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration"

without it tests fail with the following:

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.zaxxer.hikari.HikariDataSource]: Factory method 'dataSource' threw exception; nested exception is org.springframework.boot.autoconfigure.jdbc.DataSourceProperties$DataSourceBeanCreationException: Failed to determine a suitable driver class at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:641) ... 140 more Caused by: org.springframework.boot.autoconfigure.jdbc.DataSourceProperties$DataSourceBeanCreationException: Failed to determine a suitable driver class at org.springframework.boot.autoconfigure.jdbc.DataSourceProperties.determineDriverClassName(DataSourceProperties.java:186) at org.springframework.boot.autoconfigure.jdbc.DataSourceProperties.initializeDataSourceBuilder(DataSourceProperties.java:125) at org.springframework.boot.autoconfigure.jdbc.DataSourceConfiguration.createDataSource(DataSourceConfiguration.java:48) at org.springframework.boot.autoconfigure.jdbc.DataSourceConfiguration$Hikari.dataSource(DataSourceConfiguration.java:90) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ... 141 more

deniskuzZ · 2026-02-20T14:33:43Z

.../qtest-iceberg/src/test/java/org/apache/hadoop/hive/cli/TestStandaloneRESTCatalogServer.java

+  }
+
+  private static void deleteDirectoryStatic(File directory) {
+    if (directory.exists()) {


don't we have utils to do the recursive delete FileUtils.deleteDirectory(directory)

deniskuzZ · 2026-02-20T14:37:59Z

.../qtest-iceberg/src/test/java/org/apache/hadoop/hive/cli/TestStandaloneRESTCatalogServer.java

+    String livenessUrl = "http://localhost:" + port + "/actuator/health/liveness";
+    try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
+      HttpGet request = new HttpGet(livenessUrl);
+      try (CloseableHttpResponse response = httpClient.execute(request)) {


you can put both under same try and avoid nesting

deniskuzZ · 2026-02-20T14:39:22Z

.../qtest-iceberg/src/test/java/org/apache/hadoop/hive/cli/TestStandaloneRESTCatalogServer.java

-    LOG.info("=== Test: Health Check ===");
-    
-    String healthUrl = "http://localhost:" + restCatalogServer.getPort() + "/health";
+  public void testPrometheusMetrics() throws Exception {


do we use spring actuator? nice :)

deniskuzZ · 2026-02-20T14:40:45Z

.../qtest-iceberg/src/test/java/org/apache/hadoop/hive/cli/TestStandaloneRESTCatalogServer.java

-    
-    String configUrl = restCatalogServer.getRestEndpoint() + "/v1/config";
+
+    String configUrl = "http://localhost:" + port + "/iceberg/v1/config";


can we use URI template and apply format with port and path?

deniskuzZ · 2026-02-20T14:48:53Z

to support OAuth / JWT Authentication don't we need SecurityConfig?

@EnableWebSecurity
public class SecurityConfig extends WebSecurityConfigurerAdapter {
    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.oauth2ResourceServer()
            .jwt(); // validate JWT tokens
    }
}

cc @okumin

deniskuzZ · 2026-02-20T14:51:31Z

standalone-metastore/metastore-rest-catalog/pom.xml

+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-actuator</artifactId>
+      <version>${spring-boot.version}</version>


should we move versions to dependency management?

deniskuzZ · 2026-02-20T14:54:18Z

itests/qtest-iceberg/pom.xml

+  <repositories>
+    <repository>
+      <id>central</id>
+      <url>https://repo.maven.apache.org/maven2</url>


em, i am not sure about this
cc @zabetak, @abstractdog should we whitelist spring dependencies in Nexus? I originally thought that if a dependency is missing from the local repository, we always fetch it from Central."

I guess it should work without defining this repository, spring artifacts are present in central:
https://mvnrepository.com/artifact/org.springframework.boot

Yeah, the repo definition here is strange don't think we need it.

Yes, I added that to check if the build passes with this definitions, not for production.
It wasn't able to find spring boot dependencies through the wonder repo.

I guess it should work without defining this repository, spring artifacts are present in central: https://mvnrepository.com/artifact/org.springframework.boot

@abstractdog no it didn't work, you can see that previous build failed saying these dependencies were missing in wonder.

I know it sounds silly, but the same happens when I add a new tez staging artifact repo, it magically fails to fetch all artifacts for 2-3 times, and then it succeeds: I still think you should remove this, and we can check wonder artifactory logs what exactly happened when it tried to connect to the remote maven central (if it tried at all :D )
I'll check if I can provide easy access to you to the artifactory logs (e.g. kubeconfig)

deniskuzZ · 2026-02-20T14:57:30Z

...st-catalog/src/main/java/org/apache/iceberg/rest/standalone/HMSReadinessHealthIndicator.java

+ * Used by Kubernetes readiness probes to determine if the server is ready to accept traffic.
+ */
+@Component
+public class HMSReadinessHealthIndicator implements HealthIndicator {


should we move it under the health package?

deniskuzZ · 2026-02-20T15:08:49Z

...st-catalog/src/main/java/org/apache/iceberg/rest/standalone/StandaloneRESTCatalogServer.java

 * 
 * <p>Multiple instances can run behind a Kubernetes Service for load balancing.
 */
+@SpringBootApplication(exclude = DataSourceAutoConfiguration.class)


can we refactor and extract configuration like

@SpringConfiguration public class IcebergCatalogConfiguration { private static final Logger LOG = LoggerFactory.getLogger(IcebergCatalogConfiguration.class); private final Configuration conf; public IcebergCatalogConfiguration(Configuration conf) { this.conf = conf; } @Bean public Configuration hadoopConfiguration() { return conf; } @Bean public ServletRegistrationBean<HttpServlet> restCatalogServlet() { // Determine servlet path and port String servletPath = MetastoreConf.getVar(conf, ConfVars.ICEBERG_CATALOG_SERVLET_PATH); if (servletPath == null || servletPath.isEmpty()) { servletPath = "iceberg"; MetastoreConf.setVar(conf, ConfVars.ICEBERG_CATALOG_SERVLET_PATH, servletPath); } int port = MetastoreConf.getIntVar(conf, ConfVars.CATALOG_SERVLET_PORT); if (port == 0) { port = 8080; MetastoreConf.setLongVar(conf, ConfVars.CATALOG_SERVLET_PORT, port); } LOG.info("Creating REST Catalog servlet at /{}", servletPath); // Create servlet from Iceberg factory var descriptor = HMSCatalogFactory.createServlet(conf); if (descriptor == null || descriptor.getServlet() == null) { throw new IllegalStateException("Failed to create Iceberg REST Catalog servlet"); } return new ServletRegistrationBean<>(descriptor.getServlet(), "/" + servletPath + "/*"); } }

and then

@SpringBootApplication(exclude = DataSourceAutoConfiguration.class) public class StandaloneRESTCatalogServer { private static final Logger LOG = LoggerFactory.getLogger(StandaloneRESTCatalogServer.class); @Bean public Configuration hadoopConfiguration() { Configuration conf = MetastoreConf.newMetastoreConf(); // Load system properties for (String prop : System.getProperties().stringPropertyNames()) { conf.set(prop, System.getProperty(prop)); } // Validate mandatory config String thriftUris = MetastoreConf.getVar(conf, ConfVars.THRIFT_URIS); if (thriftUris == null || thriftUris.isEmpty()) { throw new IllegalArgumentException("metastore.thrift.uris must be configured to connect to HMS"); } LOG.info("Hadoop Configuration initialized, HMS Thrift URIs: {}", thriftUris); return conf; } public static void main(String[] args) { SpringApplication.run(StandaloneRESTCatalogServer.class, args); LOG.info("Standalone REST Catalog Server started successfully"); } }

deniskuzZ · 2026-02-20T15:12:23Z

standalone-metastore/metastore-rest-catalog/src/main/resources/application.properties

+
+# Server configuration
+# Port is set via MetastoreConf.CATALOG_SERVLET_PORT
+server.port=${metastore.catalog.servlet.port:8080}


should we use application.yml ?

deniskuzZ · 2026-02-20T15:13:35Z

standalone-metastore/metastore-rest-catalog/pom.xml

+  <repositories>
+    <repository>
+      <id>central</id>
+      <url>https://repo.maven.apache.org/maven2</url>


same nexus related issue

yes, build on Jenkins failed being unable to find spring boot dependencies before adding this.

we should set up jenkins in this case, where the local repository has a desired mirrored remote one, let me check the current configuration

we have all these remote repos configured for the single virtual repository (wonder), so it should work:

I would try to remove the repo from pom.xml, and we'll investigate in the next round what's happening
I can even check the logs of the artifactory if you need, in case of errors

deniskuzZ · 2026-02-20T15:14:27Z

standalone-metastore/metastore-rest-catalog/pom.xml

+      <dependency>
+        <groupId>org.eclipse.jetty</groupId>
+        <artifactId>jetty-servlets</artifactId>
+        <version>${jetty.version}</version>


isn't it defined in dependency management somewhere? if not, maybe it should be

deniskuzZ · 2026-02-20T15:18:59Z

standalone-metastore/metastore-rest-catalog/pom.xml

+      <exclusions>
+        <exclusion>
+          <groupId>org.springframework.boot</groupId>
+          <artifactId>spring-boot-starter-logging</artifactId>


why exclude logging?

deniskuzZ · 2026-02-20T15:21:28Z

standalone-metastore/metastore-rest-catalog/pom.xml

  </properties>
+  <dependencyManagement>
+    <dependencies>
+      <!-- Align all Jetty artifacts to Hive's version; Spring Boot 2.7.18 defaults to 9.4.53 -->


should we align hive to spring deps maybe?

deniskuzZ · 2026-02-20T15:22:58Z

standalone-metastore/metastore-rest-catalog/pom.xml

+        <executions>
+          <execution>
+            <goals>
+              <goal>repackage</goal>


do we even need to define the goal?

<plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <mainClass>org.apache.iceberg.rest.standalone.StandaloneRESTCatalogServer</mainClass> </configuration> </plugin>

HIVE-29468: Standalone HMS REST Catalog Server: Migrate to Spring Boot

70113b0

difin requested a review from deniskuzZ February 19, 2026 15:54

asf-ci-hive added the tests pending label Feb 19, 2026

difin mentioned this pull request Feb 19, 2026

HIVE-29468: Standalone HMS REST Catalog Server: Migrate to Spring Boot #6324

Closed

asf-ci-hive added tests unstable and removed tests pending labels Feb 19, 2026

Fixing spring-boot missing artifacts.

196fd31

difin force-pushed the hms-standalone-rest-server-with-spring-boot branch from 2e24788 to 196fd31 Compare February 19, 2026 20:19

asf-ci-hive added tests pending and removed tests unstable labels Feb 19, 2026

asf-ci-hive added tests unstable and removed tests pending labels Feb 19, 2026

deniskuzZ reviewed Feb 20, 2026

View reviewed changes

asf-ci-hive added tests pending tests failed and removed tests unstable tests pending labels Feb 25, 2026

Code redview comments 2026-Feb-24

d27f6b3

difin force-pushed the hms-standalone-rest-server-with-spring-boot branch from b48799e to d27f6b3 Compare February 25, 2026 22:45


		String configUrl = restCatalogServer.getRestEndpoint() + "/v1/config";

		String configUrl = "http://localhost:" + port + "/iceberg/v1/config";

Conversation

difin commented Feb 19, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

sonarqubecloud bot commented Feb 19, 2026

Quality Gate passed

Uh oh!

deniskuzZ commented Feb 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

difin Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

difin Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdog Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdog Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

difin Feb 25, 2026 •

edited

Loading

deniskuzZ Feb 20, 2026 •

edited

Loading

deniskuzZ Feb 20, 2026 •

edited

Loading

deniskuzZ Feb 20, 2026 •

edited

Loading

deniskuzZ commented Feb 20, 2026 •

edited

Loading

difin Feb 20, 2026 •

edited

Loading

abstractdog Feb 20, 2026 •

edited

Loading

deniskuzZ Feb 20, 2026 •

edited

Loading

abstractdog Feb 20, 2026 •

edited

Loading

deniskuzZ Feb 20, 2026 •

edited

Loading

deniskuzZ Feb 20, 2026 •

edited

Loading

deniskuzZ Feb 20, 2026 •

edited

Loading