Skip to content

Latest commit

 

History

History
3321 lines (3321 loc) · 62.9 KB

api-docs.md

File metadata and controls

3321 lines (3321 loc) · 62.9 KB
Raw

Packages:

sparkoperator.k8s.io/v1beta2

Package v1beta2 is the v1beta2 version of the API.

Resource Types:

ScheduledSparkApplication

Field Description
apiVersion
string		 sparkoperator.k8s.io/v1beta2
kind
string 		ScheduledSparkApplication
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ScheduledSparkApplicationSpec

schedule
string

Schedule is a cron schedule on which the application should run.
template
SparkApplicationSpec

Template is a template from which SparkApplication instances can be created.
suspend
bool 		(Optional)

Suspend is a flag telling the controller to suspend subsequent runs of the application if set to true. Defaults to false.
concurrencyPolicy
ConcurrencyPolicy

ConcurrencyPolicy is the policy governing concurrent SparkApplication runs.
successfulRunHistoryLimit
int32 		(Optional)

SuccessfulRunHistoryLimit is the number of past successful runs of the application to keep. Defaults to 1.
failedRunHistoryLimit
int32 		(Optional)

FailedRunHistoryLimit is the number of past failed runs of the application to keep. Defaults to 1.
status
ScheduledSparkApplicationStatus

SparkApplication

SparkApplication represents a Spark application running on and using Kubernetes as a cluster manager.

Field Description
apiVersion
string		 sparkoperator.k8s.io/v1beta2
kind
string 		SparkApplication
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
SparkApplicationSpec

type
SparkApplicationType

Type tells the type of the Spark application.
sparkVersion
string

SparkVersion is the version of Spark the application uses.
mode
DeployMode

Mode is the deployment mode of the Spark application.
proxyUser
string 		(Optional)

ProxyUser specifies the user to impersonate when submitting the application. It maps to the command-line flag “–proxy-user” in spark-submit.
image
string 		(Optional)

Image is the container image for the driver, executor, and init-container. Any custom container images for the driver, executor, or init-container takes precedence over this.
imagePullPolicy
string 		(Optional)

ImagePullPolicy is the image pull policy for the driver, executor, and init-container.
imagePullSecrets
[]string 		(Optional)

ImagePullSecrets is the list of image-pull secrets.
mainClass
string 		(Optional)

MainClass is the fully-qualified main class of the Spark application. This only applies to Java/Scala Spark applications.
mainApplicationFile
string 		(Optional)

MainFile is the path to a bundled JAR, Python, or R file of the application.
arguments
[]string 		(Optional)

Arguments is a list of arguments to be passed to the application.
sparkConf
map[string]string 		(Optional)

SparkConf carries user-specified Spark configuration properties as they would use the “–conf” option in spark-submit.
hadoopConf
map[string]string 		(Optional)

HadoopConf carries user-specified Hadoop configuration properties as they would use the the “–conf” option in spark-submit. The SparkApplication controller automatically adds prefix “spark.hadoop.” to Hadoop configuration properties.
sparkConfigMap
string 		(Optional)

SparkConfigMap carries the name of the ConfigMap containing Spark configuration files such as log4j.properties. The controller will add environment variable SPARK_CONF_DIR to the path where the ConfigMap is mounted to.
hadoopConfigMap
string 		(Optional)

HadoopConfigMap carries the name of the ConfigMap containing Hadoop configuration files such as core-site.xml. The controller will add environment variable HADOOP_CONF_DIR to the path where the ConfigMap is mounted to.
volumes
[]Kubernetes core/v1.Volume 		(Optional)

Volumes is the list of Kubernetes volumes that can be mounted by the driver and/or executors.
driver
DriverSpec

Driver is the driver specification.
executor
ExecutorSpec

Executor is the executor specification.
deps
Dependencies 		(Optional)

Deps captures all possible types of dependencies of a Spark application.
restartPolicy
RestartPolicy

RestartPolicy defines the policy on if and in which conditions the controller should restart an application.
nodeSelector
map[string]string 		(Optional)

NodeSelector is the Kubernetes node selector to be added to the driver and executor pods. This field is mutually exclusive with nodeSelector at podSpec level (driver or executor). This field will be deprecated in future versions (at SparkApplicationSpec level).
failureRetries
int32 		(Optional)

FailureRetries is the number of times to retry a failed application before giving up. This is best effort and actual retry attempts can be >= the value specified.
retryInterval
int64 		(Optional)

RetryInterval is the unit of intervals in seconds between submission retries.
pythonVersion
string 		(Optional)

This sets the major Python version of the docker image used to run the driver and executor containers. Can either be 2 or 3, default 2.
memoryOverheadFactor
string 		(Optional)

This sets the Memory Overhead Factor that will allocate memory to non-JVM memory. For JVM-based jobs this value will default to 0.10, for non-JVM jobs 0.40. Value of this field will be overridden by Spec.Driver.MemoryOverhead and Spec.Executor.MemoryOverhead if they are set.
monitoring
MonitoringSpec 		(Optional)

Monitoring configures how monitoring is handled.
batchScheduler
string 		(Optional)

BatchScheduler configures which batch scheduler will be used for scheduling
timeToLiveSeconds
int64 		(Optional)

TimeToLiveSeconds defines the Time-To-Live (TTL) duration in seconds for this SparkApplication after its termination. The SparkApplication object will be garbage collected if the current time is more than the TimeToLiveSeconds since its termination.
batchSchedulerOptions
BatchSchedulerConfiguration 		(Optional)

BatchSchedulerOptions provides fine-grained control on how to batch scheduling.
sparkUIOptions
SparkUIConfiguration 		(Optional)

SparkUIOptions allows configuring the Service and the Ingress to expose the sparkUI
driverIngressOptions
[]DriverIngressConfiguration 		(Optional)

DriverIngressOptions allows configuring the Service and the Ingress to expose ports inside Spark Driver
dynamicAllocation
DynamicAllocation 		(Optional)

DynamicAllocation configures dynamic allocation that becomes available for the Kubernetes scheduler backend since Spark 3.0.
status
SparkApplicationStatus

ApplicationState

(Appears on:SparkApplicationStatus)

ApplicationState tells the current state of the application and an error message in case of failures.

Field Description
state
ApplicationStateType
errorMessage
string

ApplicationStateType (string alias)

(Appears on:ApplicationState)

ApplicationStateType represents the type of the current state of an application.

Value Description

"COMPLETED"

"FAILED"

"SUBMISSION_FAILED"

"FAILING"

"INVALIDATING"

""

"PENDING_RERUN"

"RUNNING"

"SUBMITTED"

"SUCCEEDING"

"UNKNOWN"

BatchSchedulerConfiguration

(Appears on:SparkApplicationSpec)

BatchSchedulerConfiguration used to configure how to batch scheduling Spark Application

Field Description
queue
string 		(Optional)

Queue stands for the resource queue which the application belongs to, it’s being used in Volcano batch scheduler.
priorityClassName
string 		(Optional)

PriorityClassName stands for the name of k8s PriorityClass resource, it’s being used in Volcano batch scheduler.
resources
Kubernetes core/v1.ResourceList 		(Optional)

Resources stands for the resource list custom request for. Usually it is used to define the lower-bound limit. If specified, volcano scheduler will consider it as the resources requested.

ConcurrencyPolicy (string alias)

(Appears on:ScheduledSparkApplicationSpec)

Value Description

"Allow"

ConcurrencyAllow allows SparkApplications to run concurrently.

"Forbid"

ConcurrencyForbid forbids concurrent runs of SparkApplications, skipping the next run if the previous one hasn’t finished yet.

"Replace"

ConcurrencyReplace kills the currently running SparkApplication instance and replaces it with a new one.

Dependencies

(Appears on:SparkApplicationSpec)

Dependencies specifies all possible types of dependencies of a Spark application.

Field Description
jars
[]string 		(Optional)

Jars is a list of JAR files the Spark application depends on.
files
[]string 		(Optional)

Files is a list of files the Spark application depends on.
pyFiles
[]string 		(Optional)

PyFiles is a list of Python files the Spark application depends on.
packages
[]string 		(Optional)

Packages is a list of maven coordinates of jars to include on the driver and executor classpaths. This will search the local maven repo, then maven central and any additional remote repositories given by the “repositories” option. Each package should be of the form “groupId:artifactId:version”.
excludePackages
[]string 		(Optional)

ExcludePackages is a list of “groupId:artifactId”, to exclude while resolving the dependencies provided in Packages to avoid dependency conflicts.
repositories
[]string 		(Optional)

Repositories is a list of additional remote repositories to search for the maven coordinate given with the “packages” option.

DeployMode (string alias)

(Appears on:SparkApplicationSpec)

DeployMode describes the type of deployment of a Spark application.

Value Description

"client"

"cluster"

"in-cluster-client"

DriverInfo

(Appears on:SparkApplicationStatus)

DriverInfo captures information about the driver.

Field Description
webUIServiceName
string
webUIPort
int32

UI Details for the UI created via ClusterIP service accessible from within the cluster.
webUIAddress
string
webUIIngressName
string

Ingress Details if an ingress for the UI was created.
webUIIngressAddress
string
podName
string

DriverIngressConfiguration

(Appears on:SparkApplicationSpec)

DriverIngressConfiguration is for driver ingress specific configuration parameters.

Field Description
servicePort
int32

ServicePort allows configuring the port at service level that might be different from the targetPort.
servicePortName
string

ServicePortName allows configuring the name of the service port. This may be useful for sidecar proxies like Envoy injected by Istio which require specific ports names to treat traffic as proper HTTP.
serviceType
Kubernetes core/v1.ServiceType 		(Optional)

ServiceType allows configuring the type of the service. Defaults to ClusterIP.
serviceAnnotations
map[string]string 		(Optional)

ServiceAnnotations is a map of key,value pairs of annotations that might be added to the service object.
serviceLabels
map[string]string 		(Optional)

ServiceLables is a map of key,value pairs of labels that might be added to the service object.
ingressURLFormat
string

IngressURLFormat is the URL for the ingress.
ingressAnnotations
map[string]string 		(Optional)

IngressAnnotations is a map of key,value pairs of annotations that might be added to the ingress object. i.e. specify nginx as ingress.class
ingressTLS
[]Kubernetes networking/v1.IngressTLS 		(Optional)

TlsHosts is useful If we need to declare SSL certificates to the ingress object

DriverSpec

(Appears on:SparkApplicationSpec)

DriverSpec is specification of the driver.

Field Description
SparkPodSpec
SparkPodSpec

(Members of SparkPodSpec are embedded into this type.)

podName
string 		(Optional)

PodName is the name of the driver pod that the user creates. This is used for the in-cluster client mode in which the user creates a client pod where the driver of the user application runs. It’s an error to set this field if Mode is not in-cluster-client.
coreRequest
string 		(Optional)

CoreRequest is the physical CPU core request for the driver. Maps to spark.kubernetes.driver.request.cores that is available since Spark 3.0.
javaOptions
string 		(Optional)

JavaOptions is a string of extra JVM options to pass to the driver. For instance, GC settings or other logging.
lifecycle
Kubernetes core/v1.Lifecycle 		(Optional)

Lifecycle for running preStop or postStart commands
kubernetesMaster
string 		(Optional)

KubernetesMaster is the URL of the Kubernetes master used by the driver to manage executor pods and other Kubernetes resources. Default to https://kubernetes.default.svc.
serviceAnnotations
map[string]string 		(Optional)

ServiceAnnotations defines the annotations to be added to the Kubernetes headless service used by executors to connect to the driver.
serviceLabels
map[string]string 		(Optional)

ServiceLabels defines the labels to be added to the Kubernetes headless service used by executors to connect to the driver.
ports
[]Port 		(Optional)

Ports settings for the pods, following the Kubernetes specifications.

DriverState (string alias)

DriverState tells the current state of a spark driver.

Value Description

"COMPLETED"

"FAILED"

"PENDING"

"RUNNING"

"UNKNOWN"

DynamicAllocation

(Appears on:SparkApplicationSpec)

DynamicAllocation contains configuration options for dynamic allocation.

Field Description
enabled
bool

Enabled controls whether dynamic allocation is enabled or not.
initialExecutors
int32 		(Optional)

InitialExecutors is the initial number of executors to request. If .spec.executor.instances is also set, the initial number of executors is set to the bigger of that and this option.
minExecutors
int32 		(Optional)

MinExecutors is the lower bound for the number of executors if dynamic allocation is enabled.
maxExecutors
int32 		(Optional)

MaxExecutors is the upper bound for the number of executors if dynamic allocation is enabled.
shuffleTrackingTimeout
int64 		(Optional)

ShuffleTrackingTimeout controls the timeout in milliseconds for executors that are holding shuffle data if shuffle tracking is enabled (true by default if dynamic allocation is enabled).

ExecutorSpec

(Appears on:SparkApplicationSpec)

ExecutorSpec is specification of the executor.

Field Description
SparkPodSpec
SparkPodSpec

(Members of SparkPodSpec are embedded into this type.)

instances
int32 		(Optional)

Instances is the number of executor instances.
coreRequest
string 		(Optional)

CoreRequest is the physical CPU core request for the executors. Maps to spark.kubernetes.executor.request.cores that is available since Spark 2.4.
javaOptions
string 		(Optional)

JavaOptions is a string of extra JVM options to pass to the executors. For instance, GC settings or other logging.
lifecycle
Kubernetes core/v1.Lifecycle 		(Optional)

Lifecycle for running preStop or postStart commands
deleteOnTermination
bool 		(Optional)

DeleteOnTermination specify whether executor pods should be deleted in case of failure or normal termination. Maps to spark.kubernetes.executor.deleteOnTermination that is available since Spark 3.0.
ports
[]Port 		(Optional)

Ports settings for the pods, following the Kubernetes specifications.

ExecutorState (string alias)

(Appears on:SparkApplicationStatus)

ExecutorState tells the current state of an executor.

Value Description

"COMPLETED"

"FAILED"

"PENDING"

"RUNNING"

"UNKNOWN"

GPUSpec

(Appears on:SparkPodSpec)

Field Description
name
string

Name is GPU resource name, such as: nvidia.com/gpu or amd.com/gpu
quantity
int64

Quantity is the number of GPUs to request for driver or executor.

MonitoringSpec

(Appears on:SparkApplicationSpec)

MonitoringSpec defines the monitoring specification.

Field Description
exposeDriverMetrics
bool

ExposeDriverMetrics specifies whether to expose metrics on the driver.
exposeExecutorMetrics
bool

ExposeExecutorMetrics specifies whether to expose metrics on the executors.
metricsProperties
string 		(Optional)

MetricsProperties is the content of a custom metrics.properties for configuring the Spark metric system. If not specified, the content in spark-docker/conf/metrics.properties will be used.
metricsPropertiesFile
string 		(Optional)

MetricsPropertiesFile is the container local path of file metrics.properties for configuring the Spark metric system. If not specified, value /etc/metrics/conf/metrics.properties will be used.
prometheus
PrometheusSpec 		(Optional)

Prometheus is for configuring the Prometheus JMX exporter.

NameKey

(Appears on:SparkPodSpec)

NameKey represents the name and key of a SecretKeyRef.

Field Description
name
string
key
string

NamePath

(Appears on:SparkPodSpec)

NamePath is a pair of a name and a path to which the named objects should be mounted to.

Field Description
name
string
path
string

Port

(Appears on:DriverSpec, ExecutorSpec)

Port represents the port definition in the pods objects.

Field Description
name
string
protocol
string
containerPort
int32

PrometheusSpec

(Appears on:MonitoringSpec)

PrometheusSpec defines the Prometheus specification when Prometheus is to be used for collecting and exposing metrics.

Field Description
jmxExporterJar
string

JmxExporterJar is the path to the Prometheus JMX exporter jar in the container.
port
int32 		(Optional)

Port is the port of the HTTP server run by the Prometheus JMX exporter. If not specified, 8090 will be used as the default.
portName
string 		(Optional)

PortName is the port name of prometheus JMX exporter port. If not specified, jmx-exporter will be used as the default.
configFile
string 		(Optional)

ConfigFile is the path to the custom Prometheus configuration file provided in the Spark image. ConfigFile takes precedence over Configuration, which is shown below.
configuration
string 		(Optional)

Configuration is the content of the Prometheus configuration needed by the Prometheus JMX exporter. If not specified, the content in spark-docker/conf/prometheus.yaml will be used. Configuration has no effect if ConfigFile is set.

RestartPolicy

(Appears on:SparkApplicationSpec)

RestartPolicy is the policy of if and in which conditions the controller should restart a terminated application. This completely defines actions to be taken on any kind of Failures during an application run.

Field Description
type
RestartPolicyType

Type specifies the RestartPolicyType.
onSubmissionFailureRetries
int32 		(Optional)

OnSubmissionFailureRetries is the number of times to retry submitting an application before giving up. This is best effort and actual retry attempts can be >= the value specified due to caching. These are required if RestartPolicy is OnFailure.
onFailureRetries
int32 		(Optional)

OnFailureRetries the number of times to retry running an application before giving up.
onSubmissionFailureRetryInterval
int64 		(Optional)

OnSubmissionFailureRetryInterval is the interval in seconds between retries on failed submissions.
onFailureRetryInterval
int64 		(Optional)

OnFailureRetryInterval is the interval in seconds between retries on failed runs.

RestartPolicyType (string alias)

(Appears on:RestartPolicy)

Value Description

"Always"

"Never"

"OnFailure"

ScheduleState (string alias)

(Appears on:ScheduledSparkApplicationStatus)

Value Description

"FailedValidation"

"Scheduled"

ScheduledSparkApplicationSpec

(Appears on:ScheduledSparkApplication)

Field Description
schedule
string

Schedule is a cron schedule on which the application should run.
template
SparkApplicationSpec

Template is a template from which SparkApplication instances can be created.
suspend
bool 		(Optional)

Suspend is a flag telling the controller to suspend subsequent runs of the application if set to true. Defaults to false.
concurrencyPolicy
ConcurrencyPolicy

ConcurrencyPolicy is the policy governing concurrent SparkApplication runs.
successfulRunHistoryLimit
int32 		(Optional)

SuccessfulRunHistoryLimit is the number of past successful runs of the application to keep. Defaults to 1.
failedRunHistoryLimit
int32 		(Optional)

FailedRunHistoryLimit is the number of past failed runs of the application to keep. Defaults to 1.

ScheduledSparkApplicationStatus

(Appears on:ScheduledSparkApplication)

Field Description
lastRun
Kubernetes meta/v1.Time

LastRun is the time when the last run of the application started.
nextRun
Kubernetes meta/v1.Time

NextRun is the time when the next run of the application will start.
lastRunName
string

LastRunName is the name of the SparkApplication for the most recent run of the application.
pastSuccessfulRunNames
[]string

PastSuccessfulRunNames keeps the names of SparkApplications for past successful runs.
pastFailedRunNames
[]string

PastFailedRunNames keeps the names of SparkApplications for past failed runs.
scheduleState
ScheduleState

ScheduleState is the current scheduling state of the application.
reason
string

Reason tells why the ScheduledSparkApplication is in the particular ScheduleState.

SecretInfo

(Appears on:SparkPodSpec)

SecretInfo captures information of a secret.

Field Description
name
string
path
string
secretType
SecretType

SecretType (string alias)

(Appears on:SecretInfo)

SecretType tells the type of a secret.

Value Description

"GCPServiceAccount"

GCPServiceAccountSecret is for secrets from a GCP service account Json key file that needs the environment variable GOOGLE_APPLICATION_CREDENTIALS.

"Generic"

GenericType is for secrets that needs no special handling.

"HadoopDelegationToken"

HadoopDelegationTokenSecret is for secrets from an Hadoop delegation token that needs the environment variable HADOOP_TOKEN_FILE_LOCATION.

SparkApplicationSpec

(Appears on:SparkApplication, ScheduledSparkApplicationSpec)

SparkApplicationSpec describes the specification of a Spark application using Kubernetes as a cluster manager. It carries every pieces of information a spark-submit command takes and recognizes.

Field Description
type
SparkApplicationType

Type tells the type of the Spark application.
sparkVersion
string

SparkVersion is the version of Spark the application uses.
mode
DeployMode

Mode is the deployment mode of the Spark application.
proxyUser
string 		(Optional)

ProxyUser specifies the user to impersonate when submitting the application. It maps to the command-line flag “–proxy-user” in spark-submit.
image
string 		(Optional)

Image is the container image for the driver, executor, and init-container. Any custom container images for the driver, executor, or init-container takes precedence over this.
imagePullPolicy
string 		(Optional)

ImagePullPolicy is the image pull policy for the driver, executor, and init-container.
imagePullSecrets
[]string 		(Optional)

ImagePullSecrets is the list of image-pull secrets.
mainClass
string 		(Optional)

MainClass is the fully-qualified main class of the Spark application. This only applies to Java/Scala Spark applications.
mainApplicationFile
string 		(Optional)

MainFile is the path to a bundled JAR, Python, or R file of the application.
arguments
[]string 		(Optional)

Arguments is a list of arguments to be passed to the application.
sparkConf
map[string]string 		(Optional)

SparkConf carries user-specified Spark configuration properties as they would use the “–conf” option in spark-submit.
hadoopConf
map[string]string 		(Optional)

HadoopConf carries user-specified Hadoop configuration properties as they would use the the “–conf” option in spark-submit. The SparkApplication controller automatically adds prefix “spark.hadoop.” to Hadoop configuration properties.
sparkConfigMap
string 		(Optional)

SparkConfigMap carries the name of the ConfigMap containing Spark configuration files such as log4j.properties. The controller will add environment variable SPARK_CONF_DIR to the path where the ConfigMap is mounted to.
hadoopConfigMap
string 		(Optional)

HadoopConfigMap carries the name of the ConfigMap containing Hadoop configuration files such as core-site.xml. The controller will add environment variable HADOOP_CONF_DIR to the path where the ConfigMap is mounted to.
volumes
[]Kubernetes core/v1.Volume 		(Optional)

Volumes is the list of Kubernetes volumes that can be mounted by the driver and/or executors.
driver
DriverSpec

Driver is the driver specification.
executor
ExecutorSpec

Executor is the executor specification.
deps
Dependencies 		(Optional)

Deps captures all possible types of dependencies of a Spark application.
restartPolicy
RestartPolicy

RestartPolicy defines the policy on if and in which conditions the controller should restart an application.
nodeSelector
map[string]string 		(Optional)

NodeSelector is the Kubernetes node selector to be added to the driver and executor pods. This field is mutually exclusive with nodeSelector at podSpec level (driver or executor). This field will be deprecated in future versions (at SparkApplicationSpec level).
failureRetries
int32 		(Optional)

FailureRetries is the number of times to retry a failed application before giving up. This is best effort and actual retry attempts can be >= the value specified.
retryInterval
int64 		(Optional)

RetryInterval is the unit of intervals in seconds between submission retries.
pythonVersion
string 		(Optional)

This sets the major Python version of the docker image used to run the driver and executor containers. Can either be 2 or 3, default 2.
memoryOverheadFactor
string 		(Optional)

This sets the Memory Overhead Factor that will allocate memory to non-JVM memory. For JVM-based jobs this value will default to 0.10, for non-JVM jobs 0.40. Value of this field will be overridden by Spec.Driver.MemoryOverhead and Spec.Executor.MemoryOverhead if they are set.
monitoring
MonitoringSpec 		(Optional)

Monitoring configures how monitoring is handled.
batchScheduler
string 		(Optional)

BatchScheduler configures which batch scheduler will be used for scheduling
timeToLiveSeconds
int64 		(Optional)

TimeToLiveSeconds defines the Time-To-Live (TTL) duration in seconds for this SparkApplication after its termination. The SparkApplication object will be garbage collected if the current time is more than the TimeToLiveSeconds since its termination.
batchSchedulerOptions
BatchSchedulerConfiguration 		(Optional)

BatchSchedulerOptions provides fine-grained control on how to batch scheduling.
sparkUIOptions
SparkUIConfiguration 		(Optional)

SparkUIOptions allows configuring the Service and the Ingress to expose the sparkUI
driverIngressOptions
[]DriverIngressConfiguration 		(Optional)

DriverIngressOptions allows configuring the Service and the Ingress to expose ports inside Spark Driver
dynamicAllocation
DynamicAllocation 		(Optional)

DynamicAllocation configures dynamic allocation that becomes available for the Kubernetes scheduler backend since Spark 3.0.

SparkApplicationStatus

(Appears on:SparkApplication)

SparkApplicationStatus describes the current status of a Spark application.

Field Description
sparkApplicationId
string

SparkApplicationID is set by the spark-distribution(via spark.app.id config) on the driver and executor pods
submissionID
string

SubmissionID is a unique ID of the current submission of the application.
lastSubmissionAttemptTime
Kubernetes meta/v1.Time

LastSubmissionAttemptTime is the time for the last application submission attempt.
terminationTime
Kubernetes meta/v1.Time

CompletionTime is the time when the application runs to completion if it does.
driverInfo
DriverInfo

DriverInfo has information about the driver.
applicationState
ApplicationState

AppState tells the overall application state.
executorState
map[string]github.com/kubeflow/spark-operator/pkg/apis/sparkoperator.k8s.io/v1beta2.ExecutorState

ExecutorState records the state of executors by executor Pod names.
executionAttempts
int32

ExecutionAttempts is the total number of attempts to run a submitted application to completion. Incremented upon each attempted run of the application and reset upon invalidation.
submissionAttempts
int32

SubmissionAttempts is the total number of attempts to submit an application to run. Incremented upon each attempted submission of the application and reset upon invalidation and rerun.

SparkApplicationType (string alias)

(Appears on:SparkApplicationSpec)

SparkApplicationType describes the type of a Spark application.

Value Description

"Java"

"Python"

"R"

"Scala"

SparkPodSpec

(Appears on:DriverSpec, ExecutorSpec)

SparkPodSpec defines common things that can be customized for a Spark driver or executor pod. TODO: investigate if we should use v1.PodSpec and limit what can be set instead.

Field Description
cores
int32 		(Optional)

Cores maps to spark.driver.cores or spark.executor.cores for the driver and executors, respectively.
coreLimit
string

CoreLimit specifies a hard limit on CPU cores for the pod. Optional
memory
string 		(Optional)

Memory is the amount of memory to request for the pod.
memoryOverhead
string 		(Optional)

MemoryOverhead is the amount of off-heap memory to allocate in cluster mode, in MiB unless otherwise specified.
gpu
GPUSpec 		(Optional)

GPU specifies GPU requirement for the pod.
image
string 		(Optional)

Image is the container image to use. Overrides Spec.Image if set.
configMaps
[]NamePath 		(Optional)

ConfigMaps carries information of other ConfigMaps to add to the pod.
secrets
[]SecretInfo 		(Optional)

Secrets carries information of secrets to add to the pod.
env
[]Kubernetes core/v1.EnvVar 		(Optional)

Env carries the environment variables to add to the pod.
envVars
map[string]string 		(Optional)

EnvVars carries the environment variables to add to the pod. Deprecated. Consider using env instead.
envFrom
[]Kubernetes core/v1.EnvFromSource 		(Optional)

EnvFrom is a list of sources to populate environment variables in the container.
envSecretKeyRefs
map[string]github.com/kubeflow/spark-operator/pkg/apis/sparkoperator.k8s.io/v1beta2.NameKey 		(Optional)

EnvSecretKeyRefs holds a mapping from environment variable names to SecretKeyRefs. Deprecated. Consider using env instead.
labels
map[string]string 		(Optional)

Labels are the Kubernetes labels to be added to the pod.
annotations
map[string]string 		(Optional)

Annotations are the Kubernetes annotations to be added to the pod.
volumeMounts
[]Kubernetes core/v1.VolumeMount 		(Optional)

VolumeMounts specifies the volumes listed in “.spec.volumes” to mount into the main container’s filesystem.
affinity
Kubernetes core/v1.Affinity 		(Optional)

Affinity specifies the affinity/anti-affinity settings for the pod.
tolerations
[]Kubernetes core/v1.Toleration 		(Optional)

Tolerations specifies the tolerations listed in “.spec.tolerations” to be applied to the pod.
podSecurityContext
Kubernetes core/v1.PodSecurityContext 		(Optional)

PodSecurityContext specifies the PodSecurityContext to apply.
securityContext
Kubernetes core/v1.SecurityContext 		(Optional)

SecurityContext specifies the container’s SecurityContext to apply.
schedulerName
string 		(Optional)

SchedulerName specifies the scheduler that will be used for scheduling
sidecars
[]Kubernetes core/v1.Container 		(Optional)

Sidecars is a list of sidecar containers that run along side the main Spark container.
initContainers
[]Kubernetes core/v1.Container 		(Optional)

InitContainers is a list of init-containers that run to completion before the main Spark container.
hostNetwork
bool 		(Optional)

HostNetwork indicates whether to request host networking for the pod or not.
nodeSelector
map[string]string 		(Optional)

NodeSelector is the Kubernetes node selector to be added to the driver and executor pods. This field is mutually exclusive with nodeSelector at SparkApplication level (which will be deprecated).
dnsConfig
Kubernetes core/v1.PodDNSConfig 		(Optional)

DnsConfig dns settings for the pod, following the Kubernetes specifications.
terminationGracePeriodSeconds
int64 		(Optional)

Termination grace period seconds for the pod
serviceAccount
string 		(Optional)

ServiceAccount is the name of the custom Kubernetes service account used by the pod.
hostAliases
[]Kubernetes core/v1.HostAlias 		(Optional)

HostAliases settings for the pod, following the Kubernetes specifications.
shareProcessNamespace
bool 		(Optional)

ShareProcessNamespace settings for the pod, following the Kubernetes specifications.

SparkUIConfiguration

(Appears on:SparkApplicationSpec)

SparkUIConfiguration is for driver UI specific configuration parameters.

Field Description
servicePort
int32 		(Optional)

ServicePort allows configuring the port at service level that might be different from the targetPort. TargetPort should be the same as the one defined in spark.ui.port
servicePortName
string 		(Optional)

ServicePortName allows configuring the name of the service port. This may be useful for sidecar proxies like Envoy injected by Istio which require specific ports names to treat traffic as proper HTTP. Defaults to spark-driver-ui-port.
serviceType
Kubernetes core/v1.ServiceType 		(Optional)

ServiceType allows configuring the type of the service. Defaults to ClusterIP.
serviceAnnotations
map[string]string 		(Optional)

ServiceAnnotations is a map of key,value pairs of annotations that might be added to the service object.
serviceLabels
map[string]string 		(Optional)

ServiceLables is a map of key,value pairs of labels that might be added to the service object.
ingressAnnotations
map[string]string 		(Optional)

IngressAnnotations is a map of key,value pairs of annotations that might be added to the ingress object. i.e. specify nginx as ingress.class
ingressTLS
[]Kubernetes networking/v1.IngressTLS 		(Optional)

TlsHosts is useful If we need to declare SSL certificates to the ingress object

Generated with gen-crd-api-reference-docs.