Always builds a native-image event when explicitly disabled

[cmdjulian]

I have a Spring Boot App with Gradle and Kotlin including id("org.graalvm.buildtools.native") version "0.9.26" and a META-INF/native-image folder in Resources. When running bootBuildImage with paketo base builder, it builds a native image. This is fine for prod uses. For some debugging I want to build a java based image, not a native image. When I now set BP_NATIVE_IMAGE=false, still a native image is build. I did try to exclude the plugin and also to exclude the META-INF folder, but regardless of what I try, the builder always builds a native image with liberica nik.

Expected Behavior

When setting BP_NATIVE_IMAGE=false I would expect to not build a native-image but rather a normal jvm based image.

Current Behavior

The builder builds a native image, regardless of which variables I set.

Possible Solution

Steps to Reproduce

import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

plugins {
    val kotlinVersion = "1.9.10"
    kotlin("jvm") version kotlinVersion
    kotlin("plugin.spring") version kotlinVersion
    id("org.springframework.boot") version "3.1.3"
    id("io.spring.dependency-management") version "1.1.3"
    id("org.graalvm.buildtools.native") version "0.9.26"

    application
    id("org.flywaydb.flyway") version "9.22.0"
}

group = "com.example"

kotlin {
    jvmToolchain {
        languageVersion.set(JavaLanguageVersion.of(17))
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation(platform("org.springframework.cloud:spring-cloud-dependencies:2022.0.4"))
    implementation("org.springframework.boot:spring-boot-starter-actuator")
    implementation("org.springframework.boot:spring-boot-starter-data-r2dbc")
    implementation("org.springframework.boot:spring-boot-starter-security")
    implementation("org.springframework.boot:spring-boot-starter-thymeleaf")
    implementation("org.springframework.boot:spring-boot-starter-validation")
    implementation("org.springframework.boot:spring-boot-starter-webflux")
    implementation("org.springframework.cloud:spring-cloud-starter-gateway")
    implementation("org.thymeleaf.extras:thymeleaf-extras-springsecurity6")
    runtimeOnly("io.r2dbc:r2dbc-h2")
    runtimeOnly("com.h2database:h2")
    runtimeOnly(kotlin("reflect"))

    // Kotlin
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core")
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-reactive")
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-reactor")
    implementation("com.fasterxml.jackson.module:jackson-module-kotlin")

    // Flyway
    implementation("org.flywaydb:flyway-core")
}

tasks {
    bootBuildImage {
        builder.set("paketobuildpacks/builder:base")
        environment.set(
            mapOf(
                "BP_JVM_VERSION" to "17",
                "BP_NATIVE_IMAGE" to "false",
                "BPE_SPRING_PROFILES_ACTIVE" to "prod",
                "BP_SPRING_CLOUD_BINDINGS_DISABLED" to "true",
                "BPE_APPEND_JAVA_TOOL_OPTIONS" to "-XX:+ExtensiveErrorReports",
                "BPE_DELIM_JAVA_TOOL_OPTIONS" to " ",
            ),
        )
        imageName.set("registry.gitlab.com/etalytics/infrastructure/eta-central")
        tags.set(listOf("${project.version}"))
    }
}

tasks.withType<KotlinCompile>().configureEach {
    kotlinOptions {
        javaParameters = true
        freeCompilerArgs = listOf("-Xjsr305=strict", "-Xemit-jvm-type-annotations", "-Xjvm-default=all", "-Xcontext-receivers")
    }
}

tasks.test {
    useJUnitPlatform()
}

graalvmNative {
    agent {
        defaultMode.set("standard")
    }
    toolchainDetection.set(false)
    binaries {
        all {
            resources.autodetect()
            buildArgs("--enable-monitoring=heapdump", "-march=native", "--initialize-at-build-time=org.slf4j.LoggerFactory,ch.qos.logback,org.apache.logging")
        }
        named("main") {
            when {
                project.hasProperty("static") -> buildArgs("--static", "--libc=musl")
                else -> buildArgs("-H:+StaticExecutableWithDynamicLibC")
            }
        }
    }
    metadataRepository {
        enabled.set(true)
    }
}

Motivations

[dmikusa]

You are correct that the behavior here is a bit odd. The buildpack doesn't actually look at the value, just if it's present or not.

If you simply remove it instead of setting it to false, you'll get the desired behavior.

We can look at changing this, but it would be a behavior/breaking change, so we'd be limited in terms of when we could introduce the change. I'll leave this as a bug request, but it could be a while before we can change this.

[cmdjulian]

The reason why I did set it in the first place, was that without setting it, the builder always builds a native image.
So when using this one:

mapOf(
    "BP_JVM_VERSION" to "17",
    "BPE_SPRING_PROFILES_ACTIVE" to "prod",
    "BP_SPRING_CLOUD_BINDINGS_DISABLED" to "true",
    "BPE_APPEND_JAVA_TOOL_OPTIONS" to "-XX:+ExtensiveErrorReports",
    "BPE_DELIM_JAVA_TOOL_OPTIONS" to " ",
)

I still see the builder build a native image with liberika NIK

[dmikusa]

I'm not seeing that. The builder should include both the paketo-buildpacks/java and paketo-buildpacks/java-native-image so it should be capable of building a standard Java app as well as a Java native image app.

Also, paketobuildpacks/builder:base is the older Bionic base image set. It shouldn't be used anymore as Ubuntu Bionic is no longer supported by Canonical. The current builder is paketobuildpacks/builder-jammy-tiny. That's based on Jammy.

1. Check that your builder has both of those buildpacks. Run pack builder inspect <builder>.
2. Use the Jammy builder.
3. If that doesn't help, include your build log so I can see what is happening in the build.

Thanks

[cmdjulian]

Sorry for the late reply. I now found the time to prepare a demo app. As you can see there, the only thing I did is to add org.graalvm.buildtools.native gradle plugin to my spring starter project. Without setting anything, it still builds a native image up on running bootBuildImage task, even when I explicitly set the flag to false (you did elaborate on that, I just wanted to mention it again).
I think this happens as the plugin creates a resource folder with META-INF content for native image resources.
I can elaborate on my use case here. When using the native image plugin, Spring automatically processes the context by writing out all its proxies into more efficient builder classes. These can than be used on a regular run to speed up the startup phase, even without using native image, resulting in a faster startup phase and overall a slightly smaller memory food print.
I think it is a very valid use case to include native image plugin to trigger the aot context processing without wanting to use native image.

demo.zip

[cmdjulian]

Okay it seems like it is dependent up on the MANIFEST.MF attribute Spring-Boot-Native-Processed = true not to any META-INF stuff. When dropping this from the jar, no native image is build, but then the app does not know it has to start as aot processed as well

[dmikusa]

Oh, I see. Yes, you're right. There were some changes not too far back to auto-detect when there's a Spring Boot app that is capable of being built with native image. I think the thought was that if you were doing this then you'd likely want to have a native image app image, so we defaulted to it.

A couple of thoughts:

1. I believe the intent was that you could opt-out of this by setting BP_NATIVE_IMAGE=false so it sounds like this is not working as it should.
2. I don't believe that we documented this new criteria to auto-detect Spring Boot native. As I look at the README, it doesn't mention this. We should get that updated too.

That said, I didn't make these changes so I'm going to defer to @anthonydahanne who introduced them. He would know best the intended behavior. Hopefully he can chime in soon on this issue.

[anthonydahanne]

Hello 👋!
Sorry for the late reply 🥲

If I understand correctly, this issue, which I could reproduce using the demo.zip from you @cmdjulian (thanks for the well formulated bug report btw 🙏) is only happening with the Spring Boot Gradle plugin.

With Spring Boot and Native Image buildpacks, we need to consider 4 scenarios possible:

• Maven and Gradle Spring Boot plugins -> they each build the jar before calling the buildpacks; and
• pack CLI with either Maven or Gradle source code -> the buildpacks are just given the source code; up to them to wire the Gradle or Maven buildpacks to build the jar

Let's look at each of them in detail

• if you build a gradle project, which relies on the native plugin, with ./gradlew bootBuildImage no matter what you do with BP_NATIVE_IMAGE -> native build ❌

Like you discovered, the mere presence of manifest.Get("Spring-Boot-Native-Processed") will make the Spring Boot buildpack plans it's a native build.
Since the Spring Boot Gradle plugin automatically (because of the native plugin I believe) set BP_NATIVE_IMAGE to true, the Native Image buildpack will happily comply.

=> what are the ways to fix this?
-> either the Gradle Spring Boot plugin stop setting BP_NATIVE_IMAGE=true; the consequence being that... the users will need to set BP_NATIVE_IMAGE=true themselves... not ideal...
->or the Spring Boot buildpack checks the BP_NATIVE_IMAGE variable before setting the plan to native, the issue being... that the Spring Boot buildpack becomes aware of non strictly Spring Boot things 😖 ... which we really avoid doing

• if you build a maven project, which relies on the AOT plugin, with ./mvnw spring-boot:build-image but not the native profile -> no native ✅

The Maven BOM has a native profile, that will, when activated, not only make sure AOT plugin is enabled, but also set BP_NATIVE_IMAGE=true.
Great for native; but if you just want AOT then probably the user can set a profile, not named native that will enable the AOT plugin, without setting BP_NATIVE_IMAGE - see the pack with Maven below example

• if you use pack with gradle ❌

pack build test --env BP_JVM_VERSION=21 --env BP_SPRING_AOT_ENABLED=true  -B paketocommunity/builder-jammy-java-tiny

Well, the Gradle buildpack will gradle assemble which will do the AOT if you have the native-plugin in your gradle.kts BUT because the Gradle plugin will also add Spring-Boot-Native-Processed: true to the MANIFEST, the Spring Boot buildpack will think it's a native build, and hence won't do anything; that means it won't even add -Dspring.aot.enabled=true to the startup command.
You can counter this with docker run -it --env JAVA_TOOL_OPTIONS="-Dspring.aot.enabled=true" gradle-test but still, you will miss other Spring Boot buildpack goodness, such as Spring Cloud Binding, etc.
If you want native, of course, set --env BP_NATIVE_IMAGE=true

• if you use pack with maven✅

pack build test --env BP_JVM_VERSION=21  --env BP_SPRING_AOT_ENABLED=true -B paketocommunity/builder-jammy-java-tiny

Well, the Maven buildpack will mvn package which will do the AOT if you have the native-maven-plugin in your pom.xml and an execution configured:

			<plugin>
				<groupId>org.graalvm.buildtools</groupId>
				<artifactId>native-maven-plugin</artifactId>
			</plugin>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
				<executions>
					<execution>
						<id>process-aot</id>
						<goals>
							<goal>process-aot</goal>
						</goals>
					</execution>
				</executions>
                       </plugin>

If you want native, of course, set --env BP_NATIVE_IMAGE=true as well, or more simply, just set BP_MAVEN_ADDITIONAL_BUILD_ARGUMENTS=-Pnative

Well, I did not expect to write such a long comment...

But I think this is where we are: native builds are very easy to get automatically, no matter how you build your app; AOT jars, not so much, because it was assumed, int eh gradle path, the user must want native if they do AOT.

I will check with Spring boot team if it would be possible to have a specific profile, or signal, to hint the buildpacks what to do; because unless we start mixing concerns between the spring-boot and native-image BPs, I don't think there's a way to guess.

[mhalbritter]

I've played a bit around with it and even if we (Spring Boot) remove setting the BP_NATIVE_IMAGE variable (spring-projects/spring-boot#32884), things don't work for Gradle.

That's because when we detect that the NBT (native-build-tools) plugin is applied, we set the Spring-Boot-Native-Processed manifest entry. The Spring Boot buildpack acts on that and provides native-image-application, which the Native Image buildpack happily complies and then a native image is built.

Would it work by adding something to the native image buildpack, that, when BP_NATIVE_IMAGE is explicitly set to false the native image building is skipped?

[CodingMaxima]

Yes, thanks for raising it @mhalbritter I'm also facing the same issue. Are there any workarounds or solution for this with gradle? cc @anthonydahanne @dmikusa @cmdjulian