Skip to content

Make namespace optional for KPO #27116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dstandish merged 7 commits into from
Oct 22, 2022
Merged

Make namespace optional for KPO #27116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b73a9ee
Use current namespace if none provided in KPO
dstandish Oct 18, 2022
4a3a34c
fix tests
dstandish Oct 21, 2022
d0e88a5
document interaction with pod spec / template
dstandish Oct 21, 2022
72b8193
docs
dstandish Oct 21, 2022
473c55c
changelog clarif
dstandish Oct 21, 2022
3d9f8f4
docs
dstandish Oct 22, 2022
d0bb221
revert get_namespace change
dstandish Oct 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion airflow/providers/cncf/kubernetes/CHANGELOG.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,9 @@ Previously KubernetesPodOperator considered some settings from the Airflow confi
Features
~~~~~~~~

Previously, ``name`` was a required argument for KubernetesPodOperator (when also not supplying pod template or full pod spec). Now, if ``name`` is not supplied, ``task_id`` will be used.
* Previously, ``name`` was a required argument for KubernetesPodOperator (when also not supplying pod template or full pod spec). Now, if ``name`` is not supplied, ``task_id`` will be used.
* KubernetsPodOperator argument ``namespace`` is now optional. If not supplied via KPO param or pod template file or full pod spec, then we'll check the airflow conn,
then if in a k8s pod, try to infer the namespace from the container, then finally will use the ``default`` namespace.

4.4.0
.....
Expand Down
14 changes: 13 additions & 1 deletion airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,13 @@ def __init__(
self.pod_request_obj: k8s.V1Pod | None = None
self.pod: k8s.V1Pod | None = None

@cached_property
def _incluster_namespace(self):
from pathlib import Path

path = Path('/var/run/secrets/kubernetes.io/serviceaccount/namespace')
return path.exists() and path.read_text() or None

def _render_nested_template_fields(
self,
content: Any,
Expand Down Expand Up @@ -575,6 +582,11 @@ def build_pod_request_obj(self, context: Context | None = None) -> k8s.V1Pod:
if self.random_name_suffix:
pod.metadata.name = PodGenerator.make_unique_pod_id(pod.metadata.name)

if not pod.metadata.namespace:
hook_namespace = self.hook.conn_extras.get('extra__kubernetes__namespace')
Copy link
Contributor Author

@dstandish dstandish Oct 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a get_namespace method on the hook but it's problematic. i may resolve that in future PR and then can update this. i'll resolve that and update this in a followup PR

pod_namespace = self.namespace or hook_namespace or self._incluster_namespace or 'default'
pod.metadata.namespace = pod_namespace

for secret in self.secrets:
self.log.debug("Adding secret to task %s", self.task_id)
pod = secret.attach_to_pod(pod)
Expand All @@ -583,7 +595,7 @@ def build_pod_request_obj(self, context: Context | None = None) -> k8s.V1Pod:
pod = xcom_sidecar.add_xcom_sidecar(pod)

labels = self._get_ti_pod_labels(context)
self.log.info("Creating pod %s with labels: %s", pod.metadata.name, labels)
self.log.info("Building pod %s with labels: %s", pod.metadata.name, labels)

# Merge Pod Identifying labels with labels passed to operator
pod.metadata.labels.update(labels)
Expand Down
16 changes: 13 additions & 3 deletions docs/apache-airflow-providers-cncf-kubernetes/operators.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,7 @@ You can print out the Kubernetes manifest for the pod that would be created at r

.. code-block:: python

from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
KubernetesPodOperator,
)
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator

k = KubernetesPodOperator(
name="hello-dry-run",
Expand All @@ -73,6 +71,18 @@ You can print out the Kubernetes manifest for the pod that would be created at r

k.dry_run()

Argument precedence
^^^^^^^^^^^^^^^^^^^

When building the pod object, there may be overlap between KPO params, pod spec, template and airflow connection.
In general, the order of precedence is KPO argument > full pod spec > pod template file > airflow connection.

For ``namespace``, if namespace is not provided via any of these methods, then we'll first try to
get the current namespace (if the task is already running in kubernetes) and failing that we'll use
the ``default`` namespace.

For pod name, if not provided explicitly, we'll use the task_id. A random suffix is added by default so the pod
name is not generally of great consequence.

How to use cluster ConfigMaps, Secrets, and Volumes with Pod?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down
Loading