Use private docker repository with K8S operator and XCOM sidecar container

[mickaelgervais]

Use private docker repository with K8S operator and XCOM sidecar container

An extra parameter to KubernetesPodOperator: docker_repository, this allows to specify the repository where the sidecar container is located

My company force docker proxy usage for K8S

I need to use my company docker repository, images that are not proxifed by the company docker repository are not allowed

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

would you like to add this feature @mickaelgervais ? I am happy to review it

[mickaelgervais]

Hi @potiuk ,

I've submitted a PR, do nopt hesitate to add comment or if I've implemented this badly.
The sidecar container declaration is low level in K8S operator so I've tried to not add complexity.

Regards

[zoomzoomTnT]

Hi @mickaelgervais
To understand your issue better, are you having trouble turning on xcom because 'alpine' give you 403 error? Because in your description, you are not able to use private registry, but in my case, I can pass the whole string to the image in my KubernetesPodOperator like below. I just have problem with the xcom-sidecar, because class PodDefaults hard coded image name to be 'alpine'.
image="<private-registry>/<dir>/<dir>/<image>:<tag>"

I have left some comments in your pr for adding a explicit image tag, we have something like private-registry/alpine:3.10, and would like to add a variable for image tags.

Hi @potiuk

Is this feature going live in 1.10.13 or later versions?

[Shivarp1]

A work around would be to add this code block in the /home/airflow/airflow_local_settings.py
PodDefaults.SIDECAR_CONTAINER = k8s.V1Container(
name=PodDefaults.SIDECAR_CONTAINER_NAME,
command=['sh', '-c', PodDefaults.XCOM_CMD],
image='',
volume_mounts=[PodDefaults.VOLUME_MOUNT],
resources=k8s.V1ResourceRequirements(
requests={
"cpu": "10m",
"memory":"32Mi"
},
limits={
"cpu": "0.25",
"memory":"32Mi"
}
),
)

[fritzb]

Docker Inc started image pull rate limiting around October 30th 2020 (https://www.docker.com/blog/checking-your-current-docker-pull-rate-limits-and-status/). As a result, all KubernetesPodPerator tasks with XCOM started to fail in large deployment. Can we have the fix in Airflow 1.10 branch as well?

Based on the severity of the failure, the bug priority should be raised higher.

  - image: alpine
    imageID: ""
    lastState: {}
    name: airflow-xcom-sidecar
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: 'rpc error: code = Unknown desc = Error response from daemon: toomanyrequests:
          You have reached your pull rate limit. You may increase the limit by authenticating
          and upgrading: https://www.docker.com/increase-rate-limit'
        reason: ErrImagePull