[SPARK-52468] Support generating both v1alpha1 and v1beta1 from CRD generator #241
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
While reviewing the PR, I'm not sure this is worthy to keep all legacy POJO itself of v1alpha1. We are going to remove v1alpha1 eventually because we can do with alpha versions.
Let me play with these and think about more because it is more convenient in principle as you mentioned in the PR description.
With this patch, our gradle task would be able to generate yaml that matches latest changes in CRD POJO, and save us from maintaining the yaml files for new version.
|
BTW, @jiangzho , SPARK-52251 is for crd-generator-cli, not this. Please create a new JIRA issue with this PR. Or, correct me if I'm wrong. |
jiangzho
changed the title
|
Thanks Dongjoon for the clarification - created SPARK-52468 for this scope |
+1 that we won't keep all legacy versions. Let me clarify - we keep the CRD version(s) that is supported by the current version of operator. We'll keep the POJO inline with what we offer in the chart & yaml, will we not ? They can be removed together when we drop the support for legacy version. |
|
Yes, I agree with you that it will be perfect when we keep them in sync. |
…enerator ### What changes were proposed in this pull request? This PR adds support to generate SparkApplication and SparkCluster CRD yaml with both v1alpha1 and v1beta1 version, with storage set to v1beta1. ### Why are the changes needed? With this patch, our gradle task would be able to generate yaml that matches latest changes in CRD POJO, and save us from maintaining the yaml files for new version. The similar pattern can be used when we promote to future (v1/v2 ...) versions. We may keep the previous version POJO in separate package, leaving them as a reference, and specify the storage version only to the latest. No ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? CIs. Also, validated the generated yaml includes both versions. ### Was this patch authored or co-authored using generative AI tooling? No
dongjoon-hyun
requested changes
Jun 13, 2025
There was a problem hiding this comment.
This PR seems to be incomplete.
Also, validated the generated yaml includes both versions.
Here is the result from my verification.
$ ./gradlew build -x test
$ ./gradlew buildDockerImage
$ ./gradlew spark-operator-api:relocateGeneratedCRD
$ git diff
diff --git a/build-tools/helm/spark-kubernetes-operator/crds/sparkapplications.spark.apache.org-v1.yaml b/build-tools/helm/spark-kubernetes-operator/crds/sparkapplications.spark.apache.org-v1.yaml
index c38270a..8a68b82 100644
--- a/build-tools/helm/spark-kubernetes-operator/crds/sparkapplications.spark.apache.org-v1.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/crds/sparkapplications.spark.apache.org-v1.yaml
@@ -1,18 +1,4 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Generated by Fabric8 CRDGenerator, manual edits might get overwritten!
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
@@ -27,7 +13,7 @@ spec:
singular: sparkapplication
scope: Namespaced
versions:
- - name: v1alpha1
+ - name: v1beta1
schema:
openAPIV3Schema:
properties:
@@ -8694,7 +8680,7 @@ spec:
type: object
type: object
served: true
- storage: false
+ storage: true
subresources:
status: {}
additionalPrinterColumns:
@@ -8704,7 +8690,7 @@ spec:
- jsonPath: .metadata.creationTimestamp
name: Age
type: date
- - name: v1beta1
+ - name: v1alpha1
schema:
openAPIV3Schema:
properties:
@@ -17371,13 +17357,6 @@ spec:
type: object
type: object
served: true
- storage: true
+ storage: false
subresources:
status: {}
- additionalPrinterColumns:
- - jsonPath: .status.currentState.currentStateSummary
- name: Current State
- type: string
- - jsonPath: .metadata.creationTimestamp
- name: Age
- type: date
diff --git a/build-tools/helm/spark-kubernetes-operator/crds/sparkclusters.spark.apache.org-v1.yaml b/build-tools/helm/spark-kubernetes-operator/crds/sparkclusters.spark.apache.org-v1.yaml
index 36b8218..bd542a6 100644
--- a/build-tools/helm/spark-kubernetes-operator/crds/sparkclusters.spark.apache.org-v1.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/crds/sparkclusters.spark.apache.org-v1.yaml
@@ -1,18 +1,4 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Generated by Fabric8 CRDGenerator, manual edits might get overwritten!
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
@@ -27,7 +13,7 @@ spec:
singular: sparkcluster
scope: Namespaced
versions:
- - name: v1alpha1
+ - name: v1beta1
schema:
openAPIV3Schema:
properties:
@@ -7305,7 +7291,7 @@ spec:
type: object
type: object
served: true
- storage: false
+ storage: true
subresources:
status: {}
additionalPrinterColumns:
@@ -7315,7 +7301,7 @@ spec:
- jsonPath: .metadata.creationTimestamp
name: Age
type: date
- - name: v1beta1
+ - name: v1alpha1
schema:
openAPIV3Schema:
properties:
@@ -14593,13 +14579,6 @@ spec:
type: object
type: object
served: true
- storage: true
+ storage: false
subresources:
status: {}
- additionalPrinterColumns:
- - jsonPath: .status.currentState.currentStateSummary
- name: Current State
- type: string
- - jsonPath: .metadata.creationTimestamp
- name: Age
- type: date
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds support to generate SparkApplication and SparkCluster CRD yaml with both v1alpha1 and v1beta1 version, with storage set to v1beta1.
Why are the changes needed?
With this patch, our gradle task would be able to generate yaml that matches latest changes in CRD POJO, and save us from maintaining the yaml files for new version.
The similar pattern can be used when we promote to future (v1/v2 ...) versions. We may keep the previous version POJO in separate package, leaving them as a reference, and specify the storage version only to the latest.
No
Does this PR introduce any user-facing change?
No
How was this patch tested?
CIs. Also, validated the generated yaml includes both versions.
Was this patch authored or co-authored using generative AI tooling?
No