Issue with user/group id mapping for projected volumes sysbox 0.6.2 on ubuntu 22.04

[nclaeys]

I am encountering issues when I want to use OIDC tokens on Azure nodes.
These tokens are mounted to the pod using a projectedVolumeSource, full yaml of the pod is as follows:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    io.kubernetes.cri-o.userns-mode: auto:size=65536
    karpenter.sh/do-not-evict: "true"
    runtime.datafy.cloud/DatafyInstanceLifecycle: on-demand
    runtime.datafy.cloud/DatafyInstanceType: mx.xlarge
  creationTimestamp: "2023-08-21T13:19:55Z"
  generateName: ide-ffffe137-1719-4b93-80d2-d356cd5f522c-
  labels:
    app.kubernetes.io/managed-by: conveyor-operator
    controller-revision-hash: ide-ffffe137-1719-4b93-80d2-d356cd5f522c-7cf75685bd
    ide.datafy.cloud/ideCRD: 0cb70806-7127-475f-b2e2-a625f6f59a9a
    ide.datafy.cloud/statefulSet: ide-ffffe137-1719-4b93-80d2-d356cd5f522c
    launched-by: datafy-ide-operator
    statefulset.kubernetes.io/pod-name: ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0
  name: ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0
  namespace: ncazure
spec:
  automountServiceAccountToken: false
  containers:
  - command:
    - /sbin/init
    - --log-level=err
    env:
    - name: SYSBOX_ALLOW_TRUSTED_XATTR
      value: "FALSE"
    - name: AZURE_CLIENT_ID
      value: e4fe099a-a07e-429f-a985-6cf795d80f00
    - name: AZURE_FEDERATED_TOKEN_FILE
      value: /var/run/secrets/tokens/azure-identity-token
    - name: AZURE_AUTHORITY_HOST
      value: https://login.microsoftonline.com/
    - name: AZURE_TENANT_ID
      value: 55226c2c-0b83-4621-a5cd-e8e0e57ec920
    image: datafydpdevanc.azurecr.io/datafy/data-plane/project/azurepython:snapshot-0cb70806-7127-475f-b2e2-a625f6f59a9a
    imagePullPolicy: Always
    name: ide
    ports:
    - containerPort: 8000
      name: ide-port
      protocol: TCP
    readinessProbe:
      failureThreshold: 6
      httpGet:
        path: /environments/ncazure/ide/0cb70806-7127-475f-b2e2-a625f6f59a9a/healthz
        port: ide-port
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "4"
        memory: 12028755Ki
      requests:
        cpu: 3400m
        memory: 12028755Ki
    securityContext:
      allowPrivilegeEscalation: true
      privileged: false
      runAsGroup: 0
      runAsNonRoot: false
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/ide-mgr/var_file
      name: configmap
      readOnly: true
      subPath: ide-mgr_var_file
    - mountPath: /etc/code-server/var_file
      name: configmap
      readOnly: true
      subPath: code-server_var_file
    - mountPath: /tmp
      name: tmp
    - mountPath: /var/run/secrets/tokens
      name: azure-identity-token
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: false
  hostname: ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0
  imagePullSecrets:
  - name: environment-ncazure-ecr-credentials
  nodeName: aks-ide4s51266-64819984-vmss000005
  nodeSelector:
    node.kubernetes.io/datafy-instance-class: mx.xlarge
    node.kubernetes.io/lifecycle: on-demand
    sysbox-runc: "true"
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  runtimeClassName: sysbox-runc
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1000
    runAsGroup: 0
    runAsNonRoot: false
    runAsUser: 0
  serviceAccount: azurepython
  serviceAccountName: azurepython
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: datafy.com/sysbox-enabled
    operator: Equal
    value: "true"
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoSchedule
    key: datafy-instance-class
    operator: Equal
    value: mx.xlarge
  - effect: NoSchedule
    key: node.kubernetes.io/lifecycle
    operator: Equal
    value: on-demand
  volumes:
  - configMap:
      defaultMode: 420
      name: ide-0cb70806-7127-475f-b2e2-a625f6f59a9a
    name: configmap
  - emptyDir: {}
    name: tmp
  - name: azure-identity-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: api://AzureADTokenExchange
          expirationSeconds: 3600
          path: azure-identity-token
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-08-21T13:19:55Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-08-21T13:20:40Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-08-21T13:20:40Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-08-21T13:19:55Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: cri-o://fe6e14cdcf26066528ec24d0f723b012118b25f03f4b8f96030c4f080c04995f
    image: datafydpdevanc.azurecr.io/datafy/data-plane/project/azurepython:snapshot-0cb70806-7127-475f-b2e2-a625f6f59a9a
    imageID: datafydpdevanc.azurecr.io/datafy/data-plane/project/azurepython@sha256:1ef4acadb9fa0ed30057f21bcce65cb97339abaf72267e9c4e02d8275a1d2d2d
    lastState: {}
    name: ide
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-08-21T13:20:34Z"
  hostIP: 10.0.4.6
  phase: Running
  podIP: 100.96.1.103
  podIPs:
  - ip: 100.96.1.103
  qosClass: Burstable
  startTime: "2023-08-21T13:19:55Z"

When I want to use the azure tokens to access azure resources I get the error that I am not allowed to read the tokens file.
Inspecting the group and user id of the respective files I see the following:

Here you see that the volumes that are mounted from configmaps/emptyDirs is working correctly but for the azureTokens it is not. For this last case the user and group id are: nobody and nogroup but I do not understand why this is happening.
Next are the logs from the node, regarding the startup of the ide pod:

fullLogsNodePodStartup.txt

I am using sysbox 0.6.2 on our Azure aks cluster, which uses ubuntu with the following details:

# uname -a
Linux aks-ide4s51266-64819984-vmss000005 5.15.0-1042-azure #49-Ubuntu SMP Tue Jul 11 17:28:46 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/lsb-release	
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"

Sysbox is installed using the sysbox-deploy-k8s daemonset.

Extra full node logs:
fullNodeLogs.txt

Any tips/suggestions on what is going wrong here?

[ctalledo]

Hi @nclaeys, thanks for giving Sysbox a shot, hope it helps.

On a quick look, the first issue I see is that /var/run/secrets/tokens/azure-identity-token appears to be a broken symlink (?), as it points to ..data/azure-identity-token but I don't see that anywhere in the directories you listed. Correct me if I am wrong please.

If that's not the case, please provide the listing (ls -la) for ..data/azure-identity-token inside the Sysbox container, so we can see it's file ownership/permissions. Also, provide the findmnt output inside the Sysbox container.

Regarding the nobody:nogroup on /var/run/secrets/tokens:

That's a host volume mounted into the Sysbox container. Since the Sysbox container is rootless, Sysbox uses a feature called ID-mapped-mounts to ensure that host mount shows up with proper ownership inside the container.

However, in your case, the host volume is defined as:

 - name: azure-identity-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: api://AzureADTokenExchange
          expirationSeconds: 3600
          path: azure-identity-token

That's does not appear to be a "real host volume" but rather programmatically served (I am not familiar with Azure tokens, but I am making that inference based on the definition above).

As a result, I suspect ID-mapped-mounts are not working, so the volume shows up as nobody:nogroup inside the container.

That could be a real problem, unless the volume has read-other permissions, so that the rootless container can read them (i.e., from the host's perspective, the container is an "other" user).

Hope that helps!

[nclaeys]

It is indeed a bit a weird path, thought that was the issue initially. The file does exist as shown:

root@ide-2cd16c09-678c-4973-906c-62b905d57b00-0:/var/run/secrets/tokens# ls -l
total 0
lrwxrwxrwx 1 nobody nogroup 27 Aug 22 07:43 azure-identity-token -> ..data/azure-identity-token
root@ide-2cd16c09-678c-4973-906c-62b905d57b00-0:/var/run/secrets/tokens# ls -la ..data/azure-identity-token 
-rw-r----- 1 nobody nogroup 1371 Aug 22 07:43 ..data/azure-identity-token
root@ide-2cd16c09-678c-4973-906c-62b905d57b00-0:/var/run/secrets/tokens# cat ..data/azure-identity-token 
cat: ..data/azure-identity-token: Permission denied

You are correct that it is the kubelet on the node that provides the token file based on the provided configuration.
You can see some info about this in the log output of kubelet:

Aug 21 12:11:30 aks-ide4s51266-64819984-vmss000005 kubelet[18243]: I0821 12:11:30.070047   18243 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"azure-identity-token\" (UniqueName: \"kubernetes.io/projected/b7b53652-6832-4e04-87d9-9b8a41473ffd-azure-identity-token\") pod \"ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0\" (UID: \"b7b53652-6832-4e04-87d9-9b8a41473ffd\") " pod="ncazure/ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0"
Aug 21 12:11:30 aks-ide4s51266-64819984-vmss000005 kubelet[18243]: I0821 12:11:30.070108   18243 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7b53652-6832-4e04-87d9-9b8a41473ffd-tmp\") pod \"ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0\" (UID: \"b7b53652-6832-4e04-87d9-9b8a41473ffd\") " pod="ncazure/ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0"
Aug 21 12:11:30 aks-ide4s51266-64819984-vmss000005 kubelet[18243]: I0821 12:11:30.070216   18243 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap\" (UniqueName: \"kubernetes.io/configmap/b7b53652-6832-4e04-87d9-9b8a41473ffd-configmap\") pod \"ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0\" (UID: \"b7b53652-6832-4e04-87d9-9b8a41473ffd\") " pod="ncazure/ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0"
Aug 21 12:11:30 aks-ide4s51266-64819984-vmss000005 kubelet[18243]: I0821 12:11:30.170577   18243 reconciler_common.go:228] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7b53652-6832-4e04-87d9-9b8a41473ffd-tmp\") pod \"ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0\" (UID: \"b7b53652-6832-4e04-87d9-9b8a41473ffd\") " pod="ncazure/ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0"
Aug 21 12:11:30 aks-ide4s51266-64819984-vmss000005 kubelet[18243]: I0821 12:11:30.170624   18243 reconciler_common.go:228] "operationExecutor.MountVolume started for volume \"configmap\" (UniqueName: \"kubernetes.io/configmap/b7b53652-6832-4e04-87d9-9b8a41473ffd-configmap\") pod \"ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0\" (UID: \"b7b53652-6832-4e04-87d9-9b8a41473ffd\") " pod="ncazure/ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0"
Aug 21 12:11:30 aks-ide4s51266-64819984-vmss000005 kubelet[18243]: I0821 12:11:30.170662   18243 reconciler_common.go:228] "operationExecutor.MountVolume started for volume \"azure-identity-token\" (UniqueName: \"kubernetes.io/projected/b7b53652-6832-4e04-87d9-9b8a41473ffd-azure-identity-token\") pod \"ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0\" (UID: \"b7b53652-6832-4e04-87d9-9b8a41473ffd\") " pod="ncazure/ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0"
Aug 21 12:11:30 aks-ide4s51266-64819984-vmss000005 kubelet[18243]: I0821 12:11:30.170990   18243 operation_generator.go:740] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7b53652-6832-4e04-87d9-9b8a41473ffd-tmp\") pod \"ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0\" (UID: \"b7b53652-6832-4e04-87d9-9b8a41473ffd\") " pod="ncazure/ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0"
Aug 21 12:11:30 aks-ide4s51266-64819984-vmss000005 kubelet[18243]: I0821 12:11:30.171485   18243 operation_generator.go:740] "MountVolume.SetUp succeeded for volume \"configmap\" (UniqueName: \"kubernetes.io/configmap/b7b53652-6832-4e04-87d9-9b8a41473ffd-configmap\") pod \"ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0\" (UID: \"b7b53652-6832-4e04-87d9-9b8a41473ffd\") " pod="ncazure/ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0"
Aug 21 12:11:30 aks-ide4s51266-64819984-vmss000005 kubelet[18243]: I0821 12:11:30.193948   18243 operation_generator.go:740] "MountVolume.SetUp succeeded for volume \"azure-identity-token\" (UniqueName: \"kubernetes.io/projected/b7b53652-6832-4e04-87d9-9b8a41473ffd-azure-identity-token\") pod \"ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0\" (UID: \"b7b53652-6832-4e04-87d9-9b8a41473ffd\") " pod="ncazure/ide-ffffe137-1719-4b93-80d2-d356cd5f522c-0"

Findmnt Azure

Findmnt returns me the following output:

{
   "filesystems": [
      {
         "target": "/",
         "source": "overlay",
         "fstype": "overlay",
         "options": "rw,relatime,lowerdir=/var/lib/containers/storage/overlay/l/6FHS6BEXIDWARV3JPGMIQLCEX4:/var/lib/containers/storage/overlay/l/6FHS6BEXIDWARV3JPGMIQLCEX4/../diff1:/var/lib/containers/storage/overlay/l/Q2ARHZV7MQBNI73WBJMFLRLE4X:/var/lib/containers/storage/overlay/l/R77I4OX3YMNO5YSSQK3ICQBQXH:/var/lib/containers/storage/overlay/l/PZ6Y4Y7FGSE5QI2TTJARCZQYNN:/var/lib/containers/storage/overlay/l/HBVRQSFOXZVTU6FF3GF7INEI6G:/var/lib/containers/storage/overlay/l/VVU4KB5JYBDE2NO3Q4IIOOABHA:/var/lib/containers/storage/overlay/l/FNJYYMEAS2U2TLJAWPP7FX27WJ:/var/lib/containers/storage/overlay/l/MRLSJLRVT5SYJ5KTB5BPTCKCNK:/var/lib/containers/storage/overlay/l/73INNRGQNV7PUQ6HGICJVGNICR:/var/lib/containers/storage/overlay/l/7GIDL5645WB63N56OFXQQRHM4V:/var/lib/containers/storage/overlay/l/UC5IVDKQ5PN5UVUPS3E7IDZJJT:/var/lib/containers/storage/overlay/l/N2UFY3MFJOUPXBMD76CYP5INGE:/var/lib/containers/storage/overlay/l/6L3KW2HULQODHZ4FHU2N6ABDWM:/var/lib/containers/storage/overlay/l/576OT35LUDUJGOVUULGCXENSNO:/var/lib/containers/storage/overlay/l/ARQTKTKRBLLIEE6VSD3TH3JVP7:/var/lib/containers/storage/overlay/l/WZBNYJQUSD6MQ55CKHU5ZBGP7R:/var/lib/containers/storage/overlay/l/AZ5AULNTJD5LUPDHLJZ4IH5G6D:/var/lib/containers/storage/overlay/l/US324DCHRBPGK7EKCTH26AX4GP:/var/lib/containers/storage/overlay/l/RXKTQPFFDX4MU7PPJ7VLR2ZOGX:/var/lib/containers/storage/overlay/l/ENGBH5ICX6OAIZYYNDXPT5KW3M:/var/lib/containers/storage/overlay/l/SN5Q774CZUTYWPJN5NRLRTHKSB:/var/lib/containers/storage/overlay/l/ZNABRTKGS6YH32JF2EYRAGTPUY:/var/lib/containers/storage/overlay/l/JP7BZLJADL6XLPKCLSEFQEFICJ:/var/lib/containers/storage/overlay/l/F5F2BAT6CCNUK42PSDXDNS4LVO:/var/lib/containers/storage/overlay/l/MQSJR2YXACUP4C5WGJGELPWWHW:/var/lib/containers/storage/overlay/l/OOHFXO3CJR6IGRXWYNXKP65YAM:/var/lib/containers/storage/overlay/l/AXUK5ECED637ZKFYMOGJVX7HSQ:/var/lib/containers/storage/overlay/l/PVXQLGJSPFTXHHVNXJ76QPQKLC:/var/lib/containers/storage/overlay/l/FQZCU5J4OAJVTV4QW2MOY7VNXJ:/var/lib/containers/storage/overlay/l/L3CBX6QQ76C53P5TACRRMYAKM6:/var/lib/containers/storage/overlay/l/CHHT5RG4T2AWE6Y4VSHNNUQVF5:/var/lib/containers/storage/overlay/l/S7LIWYUP2E2DZRWUI2UQF5SVSU:/var/lib/containers/storage/overlay/l/BZ6JUJCBJWWWNIZKR5JDPQKOTW:/var/lib/containers/storage/overlay/l/XDOTWRNQPPKEXB3BUDHQAHEMKE:/var/lib/containers/storage/overlay/l/TTF5GGD5WP3TGVN2TCXUYIDRTV:/var/lib/containers/storage/overlay/l/I7FQVBXD7CECIXIQ3C222RHVES:/var/lib/containers/storage/overlay/l/B6ZU3GABIIBPVKW7E436BWZ7I7:/var/lib/containers/storage/overlay/l/F2K5APGI7F37DX4Q5M6OA3MIME:/var/lib/containers/storage/overlay/l/EBSSLUAXL3ILC4CZRCZJPZNOV2,upperdir=/var/lib/containers/storage/overlay/ecddbf1545bccf151dabbff93e00e0af1de7b31bfe7567ce1a3e3b7fc43bf86a/diff,workdir=/var/lib/containers/storage/overlay/ecddbf1545bccf151dabbff93e00e0af1de7b31bfe7567ce1a3e3b7fc43bf86a/work,metacopy=on,volatile",
         "children": [
            {
               "target": "/sys",
               "source": "sysfs",
               "fstype": "sysfs",
               "options": "rw,nosuid,nodev,noexec,relatime",
               "children": [
                  {
                     "target": "/sys/firmware",
                     "source": "tmpfs",
                     "fstype": "tmpfs",
                     "options": "ro,relatime,uid=165536,gid=165536,inode64"
                  },{
                     "target": "/sys/fs/cgroup",
                     "source": "cgroup",
                     "fstype": "cgroup2",
                     "options": "rw,nosuid,nodev,noexec,relatime"
                  },{
                     "target": "/sys/devices/virtual",
                     "source": "sysboxfs[/sys/devices/virtual]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  },{
                     "target": "/sys/kernel",
                     "source": "sysboxfs[/sys/kernel]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  },{
                     "target": "/sys/module/nf_conntrack/parameters",
                     "source": "sysboxfs[/sys/module/nf_conntrack/parameters]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  }
               ]
            },{
               "target": "/proc",
               "source": "proc",
               "fstype": "proc",
               "options": "rw,nosuid,nodev,noexec,relatime",
               "children": [
                  {
                     "target": "/proc/bus",
                     "source": "proc[/bus]",
                     "fstype": "proc",
                     "options": "ro,nosuid,nodev,noexec,relatime"
                  },{
                     "target": "/proc/fs",
                     "source": "proc[/fs]",
                     "fstype": "proc",
                     "options": "ro,nosuid,nodev,noexec,relatime"
                  },{
                     "target": "/proc/irq",
                     "source": "proc[/irq]",
                     "fstype": "proc",
                     "options": "ro,nosuid,nodev,noexec,relatime"
                  },{
                     "target": "/proc/sysrq-trigger",
                     "source": "proc[/sysrq-trigger]",
                     "fstype": "proc",
                     "options": "ro,nosuid,nodev,noexec,relatime"
                  },{
                     "target": "/proc/acpi",
                     "source": "tmpfs",
                     "fstype": "tmpfs",
                     "options": "ro,relatime,uid=165536,gid=165536,inode64"
                  },{
                     "target": "/proc/keys",
                     "source": "devtmpfs[/null]",
                     "fstype": "devtmpfs",
                     "options": "rw,relatime,size=8182316k,nr_inodes=2045579,mode=755,inode64"
                  },{
                     "target": "/proc/timer_list",
                     "source": "devtmpfs[/null]",
                     "fstype": "devtmpfs",
                     "options": "rw,relatime,size=8182316k,nr_inodes=2045579,mode=755,inode64"
                  },{
                     "target": "/proc/scsi",
                     "source": "tmpfs",
                     "fstype": "tmpfs",
                     "options": "ro,relatime,uid=165536,gid=165536,inode64"
                  },{
                     "target": "/proc/swaps",
                     "source": "sysboxfs[/proc/swaps]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  },{
                     "target": "/proc/sys",
                     "source": "sysboxfs[/proc/sys]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  },{
                     "target": "/proc/uptime",
                     "source": "sysboxfs[/proc/uptime]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  }
               ]
            },{
               "target": "/dev",
               "source": "tmpfs",
               "fstype": "tmpfs",
               "options": "rw,nosuid,size=65536k,mode=755,uid=165536,gid=165536,inode64",
               "children": [
                  {
                     "target": "/dev/mqueue",
                     "source": "mqueue",
                     "fstype": "mqueue",
                     "options": "rw,nosuid,nodev,noexec,relatime"
                  },{
                     "target": "/dev/pts",
                     "source": "devpts",
                     "fstype": "devpts",
                     "options": "rw,nosuid,noexec,relatime,gid=165541,mode=620,ptmxmode=666"
                  },{
                     "target": "/dev/null",
                     "source": "devtmpfs[/null]",
                     "fstype": "devtmpfs",
                     "options": "rw,relatime,size=8182316k,nr_inodes=2045579,mode=755,inode64"
                  },{
                     "target": "/dev/random",
                     "source": "devtmpfs[/random]",
                     "fstype": "devtmpfs",
                     "options": "rw,relatime,size=8182316k,nr_inodes=2045579,mode=755,inode64"
                  },{
                     "target": "/dev/kmsg",
                     "source": "devtmpfs[/null]",
                     "fstype": "devtmpfs",
                     "options": "rw,relatime,size=8182316k,nr_inodes=2045579,mode=755,inode64"
                  },{
                     "target": "/dev/shm",
                     "source": "shm",
                     "fstype": "tmpfs",
                     "options": "rw,nosuid,nodev,noexec,relatime,size=65536k,inode64"
                  },{
                     "target": "/dev/termination-log",
                     "source": "/dev/root[/var/lib/kubelet/pods/b824ed2f-5da1-405f-a4d9-232a455031fe/containers/ide/ff0e3338]",
                     "fstype": "ext4",
                     "options": "rw,relatime,idmapped,discard,errors=remount-ro"
                  },{
                     "target": "/dev/full",
                     "source": "devtmpfs[/full]",
                     "fstype": "devtmpfs",
                     "options": "rw,relatime,size=8182316k,nr_inodes=2045579,mode=755,inode64"
                  },{
                     "target": "/dev/tty",
                     "source": "devtmpfs[/tty]",
                     "fstype": "devtmpfs",
                     "options": "rw,relatime,size=8182316k,nr_inodes=2045579,mode=755,inode64"
                  },{
                     "target": "/dev/zero",
                     "source": "devtmpfs[/zero]",
                     "fstype": "devtmpfs",
                     "options": "rw,relatime,size=8182316k,nr_inodes=2045579,mode=755,inode64"
                  },{
                     "target": "/dev/urandom",
                     "source": "devtmpfs[/urandom]",
                     "fstype": "devtmpfs",
                     "options": "rw,relatime,size=8182316k,nr_inodes=2045579,mode=755,inode64"
                  }
               ]
            },{
               "target": "/run",
               "source": "tmpfs",
               "fstype": "tmpfs",
               "options": "rw,nosuid,nodev,noexec,relatime,uid=165536,gid=165536,inode64",
               "children": [
                  {
                     "target": "/run/lock",
                     "source": "tmpfs",
                     "fstype": "tmpfs",
                     "options": "rw,nosuid,nodev,noexec,relatime,uid=165536,gid=165536,inode64"
                  },{
                     "target": "/run/.containerenv",
                     "source": "tmpfs[/containers/storage/overlay-containers/b062b0f778856f9cfff1087b685f88e04385869f0800800b84238ec6db6add13/userdata/.containerenv]",
                     "fstype": "tmpfs",
                     "options": "rw,nosuid,nodev,size=3274716k,nr_inodes=819200,mode=755,inode64"
                  },{
                     "target": "/run/secrets/tokens",
                     "source": "tmpfs",
                     "fstype": "tmpfs",
                     "options": "ro,relatime,size=12028756k,inode64"
                  }
               ]
            },{
               "target": "/var/log/journal",
               "source": "tmpfs",
               "fstype": "tmpfs",
               "options": "rw,nosuid,nodev,noexec,relatime,uid=165536,gid=165536,inode64"
            },{
               "target": "/etc/resolv.conf",
               "source": "tmpfs[/containers/storage/overlay-containers/b062b0f778856f9cfff1087b685f88e04385869f0800800b84238ec6db6add13/userdata/resolv.conf]",
               "fstype": "tmpfs",
               "options": "rw,nosuid,nodev,noexec,size=3274716k,nr_inodes=819200,mode=755,inode64"
            },{
               "target": "/etc/hostname",
               "source": "tmpfs[/containers/storage/overlay-containers/b062b0f778856f9cfff1087b685f88e04385869f0800800b84238ec6db6add13/userdata/hostname]",
               "fstype": "tmpfs",
               "options": "rw,nosuid,nodev,size=3274716k,nr_inodes=819200,mode=755,inode64"
            },{
               "target": "/tmp",
               "source": "/dev/root[/var/lib/kubelet/pods/b824ed2f-5da1-405f-a4d9-232a455031fe/volumes/kubernetes.io~empty-dir/tmp]",
               "fstype": "ext4",
               "options": "rw,relatime,idmapped,discard,errors=remount-ro"
            },{
               "target": "/etc/hosts",
               "source": "/dev/root[/var/lib/kubelet/pods/b824ed2f-5da1-405f-a4d9-232a455031fe/etc-hosts]",
               "fstype": "ext4",
               "options": "rw,relatime,idmapped,discard,errors=remount-ro"
            },{
               "target": "/etc/ide-mgr/var_file",
               "source": "/dev/root[/var/lib/kubelet/pods/b824ed2f-5da1-405f-a4d9-232a455031fe/volumes/kubernetes.io~configmap/configmap/..2023_08_22_07_43_15.1205058254/ide-mgr_var_file//deleted]",
               "fstype": "ext4",
               "options": "ro,relatime,idmapped,discard,errors=remount-ro"
            },{
               "target": "/etc/code-server/var_file",
               "source": "/dev/root[/var/lib/kubelet/pods/b824ed2f-5da1-405f-a4d9-232a455031fe/volumes/kubernetes.io~configmap/configmap/..2023_08_22_07_43_15.1205058254/code-server_var_file//deleted]",
               "fstype": "ext4",
               "options": "ro,relatime,idmapped,discard,errors=remount-ro"
            },{
               "target": "/var/lib/k0s",
               "source": "/dev/root[/var/lib/sysbox/k0s/6d9d4b3d79d66a3c5601c11821bfd80f888e662f960d5772d9cdfe56734572e8]",
               "fstype": "ext4",
               "options": "rw,relatime,discard,errors=remount-ro"
            },{
               "target": "/var/lib/buildkit",
               "source": "/dev/root[/var/lib/sysbox/buildkit/6d9d4b3d79d66a3c5601c11821bfd80f888e662f960d5772d9cdfe56734572e8]",
               "fstype": "ext4",
               "options": "rw,relatime,discard,errors=remount-ro"
            },{
               "target": "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs",
               "source": "/dev/root[/var/lib/sysbox/containerd/6d9d4b3d79d66a3c5601c11821bfd80f888e662f960d5772d9cdfe56734572e8]",
               "fstype": "ext4",
               "options": "rw,relatime,discard,errors=remount-ro"
            },{
               "target": "/var/lib/docker",
               "source": "/dev/root[/var/lib/sysbox/docker/6d9d4b3d79d66a3c5601c11821bfd80f888e662f960d5772d9cdfe56734572e8]",
               "fstype": "ext4",
               "options": "rw,relatime,discard,errors=remount-ro"
            },{
               "target": "/var/lib/rancher/k3s",
               "source": "/dev/root[/var/lib/sysbox/rancher-k3s/6d9d4b3d79d66a3c5601c11821bfd80f888e662f960d5772d9cdfe56734572e8]",
               "fstype": "ext4",
               "options": "rw,relatime,discard,errors=remount-ro"
            },{
               "target": "/var/lib/rancher/rke2",
               "source": "/dev/root[/var/lib/sysbox/rancher-rke2/6d9d4b3d79d66a3c5601c11821bfd80f888e662f960d5772d9cdfe56734572e8]",
               "fstype": "ext4",
               "options": "rw,relatime,discard,errors=remount-ro"
            },{
               "target": "/var/lib/kubelet",
               "source": "/dev/root[/var/lib/sysbox/kubelet/6d9d4b3d79d66a3c5601c11821bfd80f888e662f960d5772d9cdfe56734572e8]",
               "fstype": "ext4",
               "options": "rw,relatime,discard,errors=remount-ro"
            },{
               "target": "/usr/src/linux-headers-5.15.0-1042-azure",
               "source": "/dev/root[/usr/src/linux-headers-5.15.0-1042-azure]",
               "fstype": "ext4",
               "options": "ro,relatime,idmapped,discard,errors=remount-ro"
            },{
               "target": "/usr/src/linux-azure-headers-5.15.0-1042",
               "source": "/dev/root[/usr/src/linux-azure-headers-5.15.0-1042]",
               "fstype": "ext4",
               "options": "ro,relatime,idmapped,discard,errors=remount-ro"
            },{
               "target": "/usr/lib/modules/5.15.0-1042-azure",
               "source": "/dev/root[/usr/lib/modules/5.15.0-1042-azure]",
               "fstype": "ext4",
               "options": "ro,relatime,idmapped,discard,errors=remount-ro"
            }
         ]
      }
   ]
}

It does not show the /var/run/secrets/tokens mount but it does show the /run/secrets/tokens which does not seem to be idmapped if I read that correctly. Based on the link you provided about idmapping (https://github.com/nestybox/sysbox/blob/master/docs/user-guide/storage.md#mounting-storage-with-kubernetes--sysbox), it seems that ProjectedVolumes are not officially supported/tested, is that correct?

When I look at the mounts on the host, I see the file is mounted as follows:
/var/lib/kubelet/pods/b824ed2f-5da1-405f-a4d9-232a455031fe/volumes/kubernetes.io~projected/azure-identity-token tmpfs tmpfs rw,rela

ls -la /var/lib/kubelet/pods/b824ed2f-5da1-405f-a4d9-232a455031fe/volumes/kubernetes.io~projected/azure-identity-token
total 4
drwxrwsrwt 3 root azureuser  100 Aug 22 09:19 .
drwxr-xr-x 3 root root      4096 Aug 22 07:43 ..
drwxr-sr-x 2 root azureuser   60 Aug 22 09:19 ..2023_08_22_09_19_56.4256461206
lrwxrwxrwx 1 root azureuser   32 Aug 22 09:19 ..data -> ..2023_08_22_09_19_56.4256461206
lrwxrwxrwx 1 root azureuser   27 Aug 22 07:43 azure-identity-token -> ..data/azure-identity-token

AWS setup

Just as a sidenode: we have similar logic for mounting token files on AWS, which does work.
Running findmnt on AWS leads to the following result:

{
               "target": "/run",
               "source": "tmpfs",
               "fstype": "tmpfs",
               "options": "rw,nosuid,nodev,noexec,relatime,uid=362144,gid=362144,inode64",
               "children": [
                  {
                     "target": "/run/lock",
                     "source": "tmpfs",
                     "fstype": "tmpfs",
                     "options": "rw,nosuid,nodev,noexec,relatime,uid=362144,gid=362144,inode64"
                  },{
                     "target": "/run/.containerenv",
                     "source": "/var/lib/sysbox/shiftfs/e29850e7-0bc1-4daa-bda9-2703dda800c2[/.containerenv]",
                     "fstype": "shiftfs",
                     "options": "rw,relatime"
                  },{
                     "target": "/run/secrets/eks.amazonaws.com/serviceaccount",
                     "source": "/var/lib/sysbox/shiftfs/224433bd-a9dc-4636-9c56-08ca0c1a6981",
                     "fstype": "shiftfs",
                     "options": "ro,relatime"
                  }
               ]
}

From the host perspective it looks similar to the Azure setup:

/var/lib/kubelet/pods/76aa110f-ae82-44d8-b06e-437bb36651bb/volumes/kubernetes.io~projected/aws-iam-token tmpfs tmpfs rw,relatime,size=14306776k,inode64

root@ip-10-2-131-91:~# ls -la /var/lib/kubelet/pods/76aa110f-ae82-44d8-b06e-437bb36651bb/volumes/kubernetes.io~projected/aws-iam-token/
total 4
drwxrwsrwt 3 root ubuntu  100 Aug 22 10:20 .
drwxr-xr-x 3 root root   4096 Aug 22 10:20 ..
drwxr-sr-x 2 root ubuntu   60 Aug 22 10:20 ..2023_08_22_10_20_05.147953030
lrwxrwxrwx 1 root ubuntu   31 Aug 22 10:20 ..data -> ..2023_08_22_10_20_05.147953030
lrwxrwxrwx 1 root ubuntu   12 Aug 22 10:20 token -> ..data/token

OS details for AWS:

root@ip-10-2-131-91:~# uname -a
Linux ip-10-2-131-91 5.15.0-1041-aws #46~20.04.1-Ubuntu SMP Wed Jul 19 15:40:00 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
root@ip-10-2-131-91:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"

Conclusion

What is your take on this? Is this an issue with sysbox? It seems that because of the more recent kernel in Azure, shiftfs is not used, which results in the files not being mapped... But I am not an expert at all in sysbox and the id-mapping still feels a bit magic (although I do love the highlevel idea).

[ctalledo]

Hi @nclaeys, thanks for all the info.

So it looks like /run/secrets/tokens is a tmpfs mount, and ID-mapped mounts are not supported on top of tmpfs until kernel 6.3. In your case the kernel is 5.15.

A couple of options here:

• Upgrade the kernel to 6.3. The only caveat is that we've not tested Sysbox on that kernel yet, but it should work fine.

• Install the alternative shiftfs kernel module on the host. This module is the predecessor to ID-mapped mounts, and does work on tmpfs. Sysbox will automatically detect it's presence and do the right thing.

Hope either of those options work for you. Otherwise I don't believe there's a good solution, other than opening up the permission of /run/secrets/tokens to other, but that's not good since those are secret tokens.

[nclaeys]

@ctalledo

Upgrading the kernel is not really an option in our setup, so I will follow option 2.

We use the deploy-k8s daemonset to install sysbox on our Azure nodes. As far as I know, it installs shiftfs on the nodes. Here is the output when inspecting shiftfs on the node:

# modprobe shiftfs
# lsmod | grep shiftfs
shiftfs                28672  0
# modinfo shiftfs
filename:       /lib/modules/5.15.0-1042-azure/updates/dkms/shiftfs.ko
license:        GPL v2
description:    id shifting filesystem
author:         Christian Brauner <christian.brauner@ubuntu.com>
author:         Seth Forshee <seth.forshee@canonical.com>
author:         James Bottomley
alias:          fs-shiftfs
srcversion:     B0C2D82DE327B38F653B659
depends:        
retpoline:      Y
name:           shiftfs
vermagic:       5.15.0-1042-azure SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         aks-ide4s51266-21878874-vmss000 Secure Boot Module Signature key
sig_key:        3B:E6:05:0A:E3:7E:81:FB:79:6C:F5:B5:DC:14:A3:05:25:FC:6C:F5
sig_hashalgo:   sha512
signature:      23:F7:7F:B5:9F:BB:98:B7:EF:88:B2:92:5E:C1:91:D2:AD:63:75:14:
		89:AA:B7:AF:19:E0:DE:01:8E:51:A9:2E:FE:BC:CC:E4:D0:68:F0:97:
		CD:56:30:11:1E:E4:1F:38:8B:E7:2D:0B:C1:66:EA:8F:CA:9D:96:7A:
		48:15:CC:1F:9C:2C:14:50:97:96:72:E2:70:A5:76:CA:D3:4D:22:63:
		C0:2A:92:B8:F0:C1:6E:22:2B:06:D1:4D:C1:A9:49:A5:C1:92:65:EA:
		3A:86:4C:25:0C:20:58:42:41:9B:EB:DF:13:F8:68:5B:26:2A:F4:60:
		47:04:D7:BB:0C:32:6A:4F:8C:4A:26:D2:86:F8:BA:00:90:FF:85:9C:
		AB:18:CE:6E:A4:2C:FC:7B:B5:B1:55:04:9D:5A:5B:F3:19:C6:81:DD:
		8A:BB:00:11:8A:5C:00:E6:20:7A:11:E2:B1:06:1A:4A:98:A8:01:67:
		F6:03:FF:8E:7B:75:5F:DE:5D:3B:86:1E:CC:2F:C6:83:5C:B2:BE:76:
		48:CA:CB:D4:13:B7:6C:65:76:DC:83:36:2A:49:4F:41:32:EC:5E:8C:
		EC:43:01:42:E6:58:68:72:6D:F7:6D:6E:A3:C2:89:9D:E2:46:B7:2C:
		25:30:3E:35:3D:1E:D3:9B:91:55:06:10:65:D8:B3:C7

Am I missing something?

[ctalledo]

Oh cool; can you provide the sysbox-mgr log in the K8s node?

Log in to the node, and then type: sudo journalctl -u sysbox-mgr

I am interested in the first ~10 lines of the log, where sysbox-mgr reports on whether shiftfs is actually working on the host (i.e., even though it's installed, in some kernels it doesn't work well; sysbox-mgr does a quick test on startup and reports it via the log).

[nclaeys]

@ctalledo Ah nice, here are the logs. Seems like there is indeed something wrong with the shiftfs setup.

Aug 23 16:36:56 aks-ide4s51266-64819984-vmss00000B systemd[1]: Starting sysbox-mgr (part of the Sysbox container runtime)...
Aug 23 16:36:56 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:56" level=info msg="Starting ..."
Aug 23 16:36:56 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:56" level=info msg="Sysbox data root: /var/lib/sysbox"
Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:58" level=info msg="Shiftfs module found in kernel: yes"
Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:58" level=info msg="Shiftfs works properly: no"
Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:58" level=info msg="Shiftfs-on-overlayfs works properly: no"
Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:58" level=info msg="ID-mapped mounts supported by kernel: yes"
Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:58" level=info msg="Overlayfs on ID-mapped mounts supported by kernel: no"
Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:58" level=info msg="Operating in system container mode."
Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:58" level=info msg="Inner container image preloading disabled."
Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:58" level=info msg="Listening on /run/sysbox/sysmgr.sock"
Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:58" level=info msg="Ready ..."
Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B systemd[1]: Started sysbox-mgr (part of the Sysbox container runtime).
Aug 23 16:39:41 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:39:41" level=info msg="registered new container 8e2aa656f07d"

Is there a way to increase verbosity or how can I best diagnose the issue?

[ctalledo]

This is the issue:

Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:58" level=info msg="Shiftfs works properly: no"
Aug 23 16:36:58 aks-ide4s51266-64819984-vmss00000B sysbox-mgr[16846]: time="2023-08-23 16:36:58" level=info msg="Shiftfs-on-overlayfs works properly: no"

In some Ubuntu kernels (5.15->5.18 IIRC), shiftfs does not work well as Ubuntu missed a patch in the kernel :(

Sysbox detects this by running a quick test, and since it's not working it's not using shiftfs (even though it's installed).

So unfortunately you are in a tough spot: you can't change the kernel version so you have no ID-mapping on tmpfs and a non-working shiftfs :(

[nclaeys]

For future reference: in the end we fetched our own oidc tokens instead of the built in k8s features. This way we can use normal secrets/configmaps which do not use a tmpfs mount but rather a ext4, which does work. Hopefully Azure will soon provide Ubuntu images with a kernal that does support shiftfs.

[rodnymolina]

Issue ended up being a known limitation of id-mapped mounts over tmpfs partitions for kernels < 6.3. I'm closing the issue now as this is working as expected.