Run Windows node CSI Drivers as HostProcess containers

[mauriciopoppe]

The HostProcess container feature became beta in 1.23, we'd like to leverage it in CSI Drivers which will run as privileged jobs in the Windows host, there'll be more details about the transition steps in the design doc.

By making the CSI Driver a HostProcess pod we no longer need the binary in the client/server model (although we will still support it). There are some items for the maintenance of the current client/server model of CSI Proxy.

Tasks

Beta Give feedback

1. Update the .github/workflows scripts to work with both v1.x and master #222
   +0
2. Update the Prow jobs to work with the v1.x branch and the master branch #223
   +0
3. Update the integration scripts to work with the v1.x and master branches #224
   +0
4. Release CSI Proxy v1.1.2 #225
   +0
5. Design doc: Using Windows HostProcess containers in CSI Drivers running in Windows nodes @mauriciopoppe created CSI Drivers in Windows as HostProcess Pods kubernetes/enhancements#3636
6. Blogpost draft: https://docs.google.com/document/d/18WwuKSONG_qkD3vaoppL_ikeDL0Ms0Wwv7HsyN8Oel0/edit

Items for the refactor of CSI Proxy to become a go library:

Tasks

Beta Give feedback

1. Refactor the FS API group to become a library #226
   +0
2. Update the integration tests to run in a HostProcess pod #227
   +0
   
3. Document the go structs used by clients when they import CSI Proxy #232
   +0
4. Refactor the Disk API group to become a library #234
   +0
5. Refactor the Volume API group to become a library #235
   +0
6. Refactor the System API group to become a library #236
   +0
7. Refactor the SMB API group to become a library #237
   +0
8. Refactor the iSCSI API group to become a library #238
   +0
9. Document the pkg/<api group>/<api>.go functions used by clients when they import CSI Proxy #239
   +0
10. Remove v1beta and v1alpha compatibility methods in the server side for the API groups that are GA #241
    +0
11. Rename API to HostAPI for v2 #242
    +0
12. Update the README, DEVELOPMENT docs for master & library development #243
    +0
13. Rename Smb and smb to SMB throughout the codebase #246
    +0
14. Rename Iscsi to iSCSI throughout the codebase #248
    +0
15. Remove v1 integration test code #249
    +0
16. Remove v1 pkg code and client library #250
    +0
17. Deprecate filesystem API group #253
    lifecycle/rotten +1
18. Rename Id to ID #265
    +0
19. Rename Iqn to IQN in iSCSI #264
    +0

Items to use the new CSI Proxy library in a CSI Driver (PD CSI Driver):

Tasks

Beta Give feedback

1. Update the Windows node manifests adding the HostProcess Pod fields kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#1061
   lifecycle/frozen +1
2. Refactor the Windows node code to use the CSI Proxy library pkg/api code instead of the client/ code kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#1062
   lifecycle/frozen +1
3. Verify that the E2E jobs are passing with the Windows node component running as a HostProcess Pod kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#1063
   lifecycle/frozen +1
4. Migrate CSI Proxy to v2 kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#1070
   +0
5. Try https://github.com/microsoft/windows-host-process-containers-base-image as a new base image after the POC is completed

/assign @mauriciopoppe

[msau42]

@mauriciopoppe if the work to migrate to hostprocess has not merged yet, can we do a release now with the existing workflow and then create a branch?

[mauriciopoppe]

@msau42 yeah we can do that, I wanted to try to release in the v1.x branch so that if we need to have a new release we would have everything set up (if that's needed although that might not happen). I'll proceed with the release in the master branch.

[mauriciopoppe]

Updates about the ongoing work:

• The migration effort is complete and it happened in the library-development branch, unit tests and integration tests are running directly in the host.
• We created an alpha tag https://github.com/kubernetes-csi/csi-proxy/releases/tag/v2.0.0-alpha.0 that can already be used for testing, it's alpha and must not be used in production yet
• Ongoing effort in the PD CSI Driver to use the alpha tag Migrate to csi proxy v2 alpha kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#1071, we haven't run presubmit tests with it properly yet (waiting on test-infra fixes)

[alexander-ding]

Following up on @mauriciopoppe 's points:

• We are drafting's a release blog post for CIS Proxy v2. Any feedback would be greatly appreiated.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mauriciopoppe]

/remove-lifecycle stale
/lifecycle frozen

[davhdavh]

For anyone interested, there is no reason to wait for v2 to use host process...

This is all the yaml you need to get it working.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: csi-proxy
  name: csi-proxy
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: csi-proxy
  template:
    metadata:
      labels:
        k8s-app: csi-proxy
    spec:
      nodeSelector:
        "kubernetes.io/os": windows
      securityContext:
        windowsOptions:
          hostProcess: true
          runAsUserName: "NT AUTHORITY\\SYSTEM"
      hostNetwork: true
      containers:
        - name: csi-proxy
          image: ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.1.2

[alexander-ding]

For anyone interested, there is no reason to wait for v2 to use host process...

This is all the yaml you need to get it working.

apiVersion: apps/v1

kind: DaemonSet

metadata:

  labels:

    k8s-app: csi-proxy

  name: csi-proxy

  namespace: kube-system

spec:

  selector:

    matchLabels:

      k8s-app: csi-proxy

  template:

    metadata:

      labels:

        k8s-app: csi-proxy

    spec:

      nodeSelector:

        "kubernetes.io/os": windows

      securityContext:

        windowsOptions:

          hostProcess: true

          runAsUserName: "NT AUTHORITY\\SYSTEM"

      hostNetwork: true

      containers:

        - name: csi-proxy

          image: ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.1.2

Hi there! Thanks for your interest in the project :)

It's been a bit since I've worked on this, but I'm fairly certain this shouldn't work, or at least this doesn't do what we are intending to do.

The CSI Proxy v1 container is a proxy server that relays client API calls to a privileged binary directly running on Windows machines (outside of Kubernetes). The point of v2 is to completely eliminate the need for the separate privileged binary. It is true that HostProcess containers (HPCs) are available already, but just running the v1 proxy server container as an HPC doesn't actually eliminate the need for the separate binary. Also, because HPCs don't support named pipes or unix sockets, the proxy server would likely fail to connect to binary completely.

[davhdavh]

ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.1.2 includes the required binary, hostProcess: true gives it privilege.

Not sure why the HPC would need to support pipes or sockets specifically, it just runs like a normal process in the host and thus native support.

I'm fairly certain this shouldn't work

It works just fine; with smb atleast

The point of v2 is to completely eliminate the need for the separate privileged binary

Sure, a totally valid goal, but as it isn't available today, I gave a solution for the thing is available today.

[mauriciopoppe]

What you propose was already implemented by the Windows team a while ago in https://github.com/kubernetes-sigs/sig-windows-tools/blob/master/hostprocess/csi-proxy/README.md, while it's running only CSI Proxy as a HostProcess Pod our goal is to leverage the HostProcessContainers feature to improve other parts of the entire solution (e.g. maintainability, parity between Linux & Windows implementations).

So while you're using HPC your CSI Driver deployment is different between Linux/Windows still, this is the problem we'd like to solve.

[alexander-ding]

ghcr.io/kubernetes-sigs/sig-windows/csi-proxy:v1.1.2 includes the required binary, hostProcess: true gives it privilege.

Not sure why the HPC would need to support pipes or sockets specifically, it just runs like a normal process in the host and thus native support.

I'm fairly certain this shouldn't work

It works just fine; with smb atleast

The point of v2 is to completely eliminate the need for the separate privileged binary

Sure, a totally valid goal, but as it isn't available today, I gave a solution for the thing is available today.

Yeah, you're totally right. For some reason I thought you were suggesting running the node plugins/registrars as HPCs, instead of running the privileged binary as an HPC. My bad on this.