README.md: describe features of raw block volumes, update example

Our documentation did not specify how exactly PMEM-CSI provides raw
block volumes and thus what users can expect from them. Actually using
them also isn't trivial. This now gets explained in the documentation
and in the updated example.

Ubuntu is used as base image because the "mknod" in busybox does not
accept hex device numbers as printed by "stat" and because mkfs.ext4
is in the image. With Clear Linux, mkfs.ext4 would have to be
installed from a rather large bundle.

Author: pohly

README.md

@@ -775,13 +775,33 @@ parameters:
 
 #### Note about raw block volumes
 
-Applications can use volumes provisioned by PMEM-CSI as 
-[raw block devices](https://kubernetes.io/blog/2019/03/07/raw-block-volume-support-to-beta/).
+Applications can use volumes provisioned by PMEM-CSI as [raw block
+devices](https://kubernetes.io/blog/2019/03/07/raw-block-volume-support-to-beta/). Such
+volumes use the same "fsdax" namespace mode as filesystem volumes
+and therefore are block devices. That mode only supports dax (=
+`mmap(MAP_SYNC)`) through a filesystem. Pages mapped on the raw block
+device go through the Linux page cache. Applications have to format
+and mount the raw block volume themselves if they want dax. The
+advantage then is that they have full control over that part.
+
+Volumes with "devdax" namespace mode are not supported. That mode
+results in a character device, something that Kubernetes does not
+expect from a storage driver and that does not work because Kubernetes
+internally tries to [bind the device to a loop
+device](https://github.com/kubernetes/kubernetes/blob/7c87b5fb55ca096c007c8739d4657a5a4e29fb09/pkg/volume/util/util.go#L531-L534).
+
 For provisioning a PMEM volume as raw block device, one has to create a 
 `PersistentVolumeClaim` with `volumeMode: Block`. See example [PVC](
 deploy/common/pmem-pvc-block-volume.yaml) and
 [application](deploy/common/pmem-app-block-volume.yaml) for usage reference.
 
+That example demonstrates how to handle some details:
+- `mkfs.ext4` needs `-b 4096` to produce volumes that support dax;
+  without it, the automatic block size detection may end up choosing
+  an unsuitable value depending on the volume size.
+- [Kubernetes bug #85624](https://github.com/kubernetes/kubernetes/issues/85624)
+  must be worked around to format and mount the raw block device.
+
 <!-- FILL TEMPLATE:
 
   ### How to extend the plugin

deploy/common/pmem-app-block-volume.yaml

@@ -3,16 +3,38 @@ apiVersion: v1
 metadata:
   name: my-csi-app
 spec:
+  initContainers:
+    # This init container is a workaround for https://github.com/kubernetes/kubernetes/issues/85624.
+    - name: store-device
+      image: ubuntu
+      command:
+        - "sh"
+        - "-c"
+        - "(echo '#!/bin/sh' && stat -c 'mknod /dev-xpmem b 0x%t 0x%T' /dev-xpmem) >/data/create-dev.sh && chmod a+x /data/create-dev.sh"
+      volumeMounts:
+        - name: data
+          mountPath: /data
+      volumeDevices:
+        - name: my-csi-device
+          devicePath: /dev-xpmem
   containers:
     - name: my-frontend
-      image: busybox
-      command: [ "sleep", "100000" ]
-      volumeDevices:
-      - devicePath: "/dev/xpmem"
-        name: my-csi-volume
+      image: ubuntu
+      securityContext:
+        privileged: True
+      command:
+        - "sh"
+        - "-c"
+        # mkfs.ext4 may fail here if the volume was already formatted before, so we ignore the return code.
+        - "if [ ! -e /dev-xpmem ]; then /data/create-dev.sh; fi && mkfs.ext4 -b 4096 /dev-xpmem; mkdir -p /mnt && mount -odax /dev-xpmem /mnt && mount | grep /mnt | grep dax && sleep 100000"
+      volumeMounts:
+        - name: data
+          mountPath: /data
   nodeSelector:
     storage: pmem
   volumes:
-  - name: my-csi-volume
+  - name: my-csi-device
     persistentVolumeClaim:
       claimName: pmem-csi-pvc-block-volume
+  - name: data
+    emptyDir: