[Flaky Test] In-tree Volumes: Error getting c-m metrics : the server could not find the requested resource

[spiffxp]

Which jobs are flaking:

• pull-kubernetes-e2e-gce
• ci-kubernetes-e2e-gci-gce

Which test(s) are flaking:

• [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly]
• [sig-storage] In-tree Volumes [Driver: emptydir] [Testpattern: Inline-volume (default fs)] subPath should support file as subpath [LinuxOnly]
• [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link] [Testpattern: Pre-provisioned PV (default fs)] volumes should allow exec of files on the volume
• [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly]
• [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data
• [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Inline-volume (default fs)] volumes should store data
• [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should be able to unmount after the subpath directory is deleted
• [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should be able to unmount after the subpath directory is deleted
• [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory
• [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should support non-existent path
• [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support existing directories when readOnly specified in the volumeSource
• [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should be able to unmount after the subpath directory is deleted
• [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource
• [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly]
• [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (ext3)] volumes should allow exec of files on the volume
• [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works
• [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (ext3)] volumes should store data

Testgrid link:

• https://testgrid.k8s.io/sig-release-master-informing#gce-cos-master-default&width=5&include-filter-by-regex=In-tree&sort-by-flakiness=
  https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-e2e-gce&include-filter-by-regex=In-tree&width=5&sort-by-flakiness=

Reason for failure:
Failure fetching metrics:

Dec 15 22:01:17.942: Error getting c-m metrics : the server rejected our request for an unknown reason (get pods kube-controller-manager-e2e-4fc8c5bbfe-674b9-master:10252)
...
Causes: [
                    {
                        Type: "UnexpectedServerResponse",
                        Message: "unknown",
                        Field: "",
                    },
                ],

Dec  4 20:06:05.021: Error getting Kubelet e2e-f19735d0af-674b9-minion-group-l4ln metrics: the server is currently unable to handle the request (get nodes e2e-f19735d0af-674b9-minion-group-l4ln:10250)
...
Causes: [
                    {
                        Type: "UnexpectedServerResponse",
                        Message: "Error trying to reach service: 'dial tcp 10.40.0.4:10250: i/o timeout'",
                        Field: "",
                    },
                ],

Anything else we need to know:
https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&text=Error%20getting%20c-m%20metrics&job=pull-kubernetes-e2e-gce%24&test=sig-storage

This appears to have restarted abruptly around 2/11-2/12

/sig storage
/priority important-soon

[liggitt]

https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/batch/pull-kubernetes-e2e-gce/1206995563719954438/

[liggitt]

xref #86312

I think we have a general failure to start containers reliably. Almost every instance of these test failures contained the pod status error of "OCI runtime start failed: container process is already dead: unknown"

• https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce/1206995563719954438
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce/1206995563719954438
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce/1206989684341739522
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce/1206963761311649796
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce/1206944256451678211
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce/1206933812953485313
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce/1206930288853127169

[liggitt]

@kubernetes/sig-node-test-failures

[liggitt]

after sweeping a bunch of the in-tree test failures, almost all seemed to have #86312 as a root cause, triggered by very short-lived containers used to check file content or permissions.

The only test I saw failing that didn't seem to fit that was:

[sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path

test/e2e/storage/testsuites/subpath.go:191
Dec 17 19:12:19.924: Unexpected error:
    <*errors.errorString | 0xc0027e44b0>: {
        s: "expected \"content of file \\\"/test-volume/provisioning-9844/test-file\\\": mount-tester new file\" in container output: Expected\n    <string>: content of file \"/test-volume/provisioning-9844/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-9844/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-9844/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-9844/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-9844/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-9844/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-9844/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-9844/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-9844/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-9844/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retrying\nto contain substring\n    <string>: content of file \"/test-volume/provisioning-9844/test-file\": mount-tester new file",
    }
    expected "content of file \"/test-volume/provisioning-9844/test-file\": mount-tester new file" in container output: Expected
        <string>: content of file "/test-volume/provisioning-9844/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-9844/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-9844/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-9844/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-9844/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-9844/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-9844/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-9844/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-9844/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-9844/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retrying
    to contain substring
        <string>: content of file "/test-volume/provisioning-9844/test-file": mount-tester new file
occurred
test/e2e/framework/util.go:829

[sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory

test/e2e/storage/testsuites/subpath.go:202
Dec 17 14:08:37.147: Unexpected error:
    <*errors.errorString | 0xc0021bf240>: {
        s: "expected \"content of file \\\"/test-volume/provisioning-5457/test-file\\\": mount-tester new file\" in container output: Expected\n    <string>: content of file \"/test-volume/provisioning-5457/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-5457/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-5457/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-5457/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-5457/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-5457/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-5457/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-5457/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-5457/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retryingcontent of file \"/test-volume/provisioning-5457/test-file\": \n    Unexpected content. Expected: mount-tester new file\n    . Retrying\nto contain substring\n    <string>: content of file \"/test-volume/provisioning-5457/test-file\": mount-tester new file",
    }
    expected "content of file \"/test-volume/provisioning-5457/test-file\": mount-tester new file" in container output: Expected
        <string>: content of file "/test-volume/provisioning-5457/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-5457/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-5457/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-5457/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-5457/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-5457/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-5457/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-5457/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-5457/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retryingcontent of file "/test-volume/provisioning-5457/test-file": 
        Unexpected content. Expected: mount-tester new file
        . Retrying
    to contain substring
        <string>: content of file "/test-volume/provisioning-5457/test-file": mount-tester new file
occurred
test/e2e/framework/util.go:829

examples:

• https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce/1207004999654051841
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce/1206930288853127169

I'd recommend repurposing this issue to address that specifically and leave the OCI issue for the short-lived container issue