daemonset csi-rbdplugin pod pids.current value keep rising #4520
Description
Describe the bug
During test, we create then delete many pvc and pods(about 180) , every pods will mount one pvc. We repeat the process about 5 hours. We find that the daemonset csi-rbdplugin pod pids.current value keep rising. During our test, the pids.current value is 47 before test, the value increased to 99 after stopping the test.
The ceph-csi version is v3.9.0 and has set pidlimit=-1.
Environment details
- Image/version of Ceph CSI driver :
v3.9.0 - Helm chart version :
- Kernel version :
Linux 5.3.18-57-default - Mounter used for mounting PVC (for cephFS its
fuseorkernel. for rbd its
krbdorrbd-nbd) : krbd - Kubernetes cluster version :
v1.28.4 - Ceph cluster version :
v17
Steps to reproduce
Steps to reproduce the behavior:
- create a script to create pod and pvc
action=$1
for i in {1..180}
do
cat <<EOF | kubectl $action -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc-$i
spec:
accessModes:- ReadWriteOnce
storageClassName: "network-block"
resources:
requests:
storage: 100M
- ReadWriteOnce
apiVersion: v1
kind: Pod
metadata:
name: test-rbd-$i
spec:
containers:
- name: www
image: nginx:alpine
ports:- containerPort: 80
name: www
volumeMounts: - name: www-persistent-storage
mountPath: /usr/share/nginx/html
volumes: - name: www-persistent-storage
persistentVolumeClaim:
claimName: rbd-pvc-$i
nodeName: node-xxx
EOF
done
- containerPort: 80
- run the script to create and delete pod and pvc for about 5 hours
- Check pods pid.max
before test:
crictl exec -it 608df5831ddae cat /sys/fs/cgroup/pids/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1aea94b_2820_418f_8b33_fd51e946a442.slice/cri-containerd-608df5831ddae1b508434f20edee9b18d58fb76771c5fc4c785f99b024dc56c5.scope/pids.current
47
after test:
crictl exec -it 608df5831ddae cat /sys/fs/cgroup/pids/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1aea94b_2820_418f_8b33_fd51e946a442.slice/cri-containerd-608df5831ddae1b508434f20edee9b18d58fb76771c5fc4c785f99b024dc56c5.scope/pids.current
99
Actual results
Describe what happened
Expected behavior
Some pids should be released.
Logs
If the issue is in PVC creation, deletion, cloning please attach complete logs
of below containers.
- csi-provisioner and csi-rbdplugin/csi-cephfsplugin container logs from the
provisioner pod.
If the issue is in PVC resize please attach complete logs of below containers.
- csi-resizer and csi-rbdplugin/csi-cephfsplugin container logs from the
provisioner pod.
If the issue is in snapshot creation and deletion please attach complete logs
of below containers.
- csi-snapshotter and csi-rbdplugin/csi-cephfsplugin container logs from the
provisioner pod.
If the issue is in PVC mounting please attach complete logs of below containers.
-
csi-rbdplugin/csi-cephfsplugin and driver-registrar container logs from
plugin pod from the node where the mount is failing. -
if required attach dmesg logs.
Note:- If its a rbd issue please provide only rbd related logs, if its a
cephFS issue please provide cephFS logs.
Additional context
Add any other context about the problem here.
For example:
Any existing bug report which describe about the similar issue/behavior