Adding support for setting spark job namespaces to all namespaces

[ChenYi015]

Purpose of this PR

Add support for setting spark.jobNamespaces to all namespaces ("").

• controller/webhook should watch all namespaces if spark.jobNamespaces is empty list or contains ""
• should not create spark service account and rbac resources for empty namespace
• mutating/validating webhook configurations should not add namespaceSelector when spark.jobNamespaces is empty list or contains ""

1. If we want to watch all namespaces, then we can install the spark operator as follows:

helm install spark-operator charts/spark-operator-chart \
    --create-namespace \
    --namespace spark-operator \
    --set 'spark.jobNamespaces={}'

but this will not create rbac resources for spark in any namespaces.

2. If we want to watch all namespaces and also create rbac resources for spark in namespace default and spark-operaotr, then we can do as follows:

helm install spark-operator charts/spark-operator-chart \
    --create-namespace \
    --namespace spark-operator \
    --set 'spark.jobNamespaces={,default,spark-operator}'

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• [] Documentation update

Rationale

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

If we set spark.jobNamespaces to ["","spark-operator"], then it will failes:

$ helm install spark-operator spark-operator/spark-operator \
    --version 2.0.0-rc.0 \
    --create-namespace \
    --namespace spark-operator \
    --set 'spark.jobNamespaces={,spark-operator}'
Error: INSTALLATION FAILED: 1 error occurred:
	* serviceaccounts "spark-operator-spark" already exists

[ChenYi015]

/hold
/assign @vara-bonthu @yuchaoran2011

[ChenYi015]

Since PR #2128 has already fixed the cache issue when the spark job namespaces contains namespaceAll (""), I will rebase it once that PR is merged.

[vara-bonthu]

#2128 is now merged. Do you want to update your PR?

[ChenYi015]

#2128 is now merged. Do you want to update your PR?

@vara-bonthu Updated.

[vara-bonthu]

It would be nice to add these details to our docs as well

If we want to watch all namespaces, then we can install the spark operator as follows:
helm install spark-operator charts/spark-operator-chart
--create-namespace
--namespace spark-operator
--set 'spark.jobNamespaces={}'
but this will not create rbac resources for spark in any namespaces.

If we want to watch all namespaces and also create rbac resources for spark in namespace default and spark-operaotr, then we can do as follows:
helm install spark-operator charts/spark-operator-chart
--create-namespace
--namespace spark-operator
--set 'spark.jobNamespaces={,default,spark-operator}'

[ChenYi015]

@vara-bonthu Yes, I will update the docs once v2.0.0 is released.

[vara-bonthu]

/approve

Will wait for another approval @yuchaoran2011

[ChenYi015]

/hold cancel

[yuchaoran2011]

/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vara-bonthu, yuchaoran2011

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vara-bonthu,yuchaoran2011]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment