Missing file assets

[cecobask]

/kind bug

1. What kops version are you running? The command kops version, will display
this information.

1.31.0

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

1.29.15

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

kops get assets --name <NAME> --state s3://c15-kops-state-store

5. What happened after the commands executed?

The command returned incomplete file assets.

6. What did you expect to happen?

When I tried to create a cluster after that, its validation failed with error:

Apr 23 19:54:20 i-0c4ee8198243ddcc9 nodeup[1864]: W0423 19:54:20.674173    1864 main.go:133] got error running nodeup (will retry in 30s): error adding asset "02990fa281c0a2c4b073c6d2415d264b682bd693aa7d86c5d8eb4b86d684a18c@https://<REDACTED>/v1/containerd/containerd/releases/download/v1.7.25/containerd-1.7.25-linux-amd64.tar.gz": error response from "https://<REDACTED>/v1/containerd/containerd/releases/download/v1.7.25/containerd-1.7.25-linux-amd64.tar.gz": HTTP 404

Our automations rely on the output of the kops get assets command to upload required assets to our corporate Artifactory. However, the containerd and other file assets are not included in the output.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2025-04-24T13:07:11Z"
  name: k8s.<REDACTED>
spec:
  additionalPolicies:
    master: '[{"Effect":"Allow","Action":["logs:*"],"Resource":"*"},{"Effect":"Allow","Action":["elasticloadbalancing:DeregisterTargets","elasticloadbalancing:ModifyTargetGroupAttributes"],"Resource":"*"},{"Effect":"Allow","Action":["kms:Decrypt","kms:Encrypt"],"Resource":"arn:aws:kms:us-east-1:<REDACTED>:key/<REDACTED>"},{"Effect":"Allow","Action":["s3:List*"],"Resource":"arn:aws:s3:::c15-kops-state-store"}]'
    node: '[{"Effect":"Allow","Action":["logs:*"],"Resource":"*"},{"Effect":"Allow","Action":["route53:ListHostedZones","route53:ListResourceRecordSets"],"Resource":["*"]},{"Effect":"Allow","Action":["route53:ChangeResourceRecordSets"],"Resource":["arn:aws:route53:::hostedzone/*"]},{"Effect":"Allow","Action":["autoscaling:DescribeAutoScalingGroups","autoscaling:DescribeAutoScalingInstances","autoscaling:SetDesiredCapacity","autoscaling:DescribeLaunchConfigurations","autoscaling:DescribeTags","autoscaling:TerminateInstanceInAutoScalingGroup"],"Resource":["*"]},{"Effect":"Allow","Action":["route53:CreateHealthCheck","route53:DeleteHealthCheck","route53:UpdateHealthCheck","route53:GetHealthCheck","route53:ListHealthChecks","route53:ChangeTagsForResource","route53:ListTagsForResource","route53:ListTagsForResources"],"Resource":["*"]},{"Effect":"Allow","Action":["lambda:InvokeFunction","lambda:InvokeAsync"],"Resource":"arn:aws:lambda:*:<REDACTED>:function:<REDACTED>-update-role-<REDACTED>"}]'
  api:
    loadBalancer:
      class: Network
      idleTimeoutSeconds: 3600
      type: Internal
  assets:
    containerRegistry: <REDACTED>/temp-docker-local/<REDACTED>/v1
    fileRepository: https://<REDACTED>/artifactory/temp-generic-local/<REDACTED>/v1
  authentication: {}
  authorization:
    rbac: {}
  channel: none
  cloudConfig:
    awsEBSCSIDriver:
      enabled: true
  cloudLabels:
    business_unit: <REDACTED>
    created_by: <REDACTED>
    k8s.io/cluster-autoscaler/enabled: "true"
    product: <REDACTED>
  cloudProvider: aws
  configBase: s3://c15-kops-state-store/k8s.<REDACTED>
  encryptionConfig: true
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a
      name: a
    - encryptedVolume: true
      instanceGroup: master-us-east-1b
      name: b
    - encryptedVolume: true
      instanceGroup: master-us-east-1c
      name: c
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a
      name: a
    - encryptedVolume: true
      instanceGroup: master-us-east-1b
      name: b
    - encryptedVolume: true
      instanceGroup: master-us-east-1c
      name: c
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: events
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a
      name: a
    - encryptedVolume: true
      instanceGroup: master-us-east-1b
      name: b
    - encryptedVolume: true
      instanceGroup: master-us-east-1c
      name: c
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: cilium
  fileAssets:
  - content: |-
      apiVersion: audit.k8s.io/v1
      kind: Policy
      rules:
        - level: Metadata
          users: ["kubecfg"]
          userGroups: ["system:masters"]

        - level: None
          users: ["system:kube-proxy"]
          verbs: ["watch"]
          resources:
            - group: "" # core
              resources: ["endpoints", "services", "services/status"]

        - level: None
          users: ["kubelet"] # legacy kubelet identity
          verbs: ["get"]
          resources:
            - group: "" # core
              resources: ["nodes", "nodes/status"]

        # Don't log authenticated requests to certain non-resource URL paths.
        - level: None
          userGroups: ["system:authenticated"]
          nonResourceURLs:
          - "/api*" # Wildcard matching.
          - "/version"

        - level: None
          userGroups: ["system:nodes"]
          verbs: ["get"]
          resources:
            - group: "" # core
              resources: ["nodes", "nodes/status"]

        - level: None
          users:
            - system:kube-controller-manager
            - system:kube-scheduler
            - system:serviceaccount:kube-system:endpoint-controller
          verbs: ["get", "update"]
          namespaces: ["kube-system"]
          resources:
            - group: "" # core
              resources: ["endpoints"]

        - level: None
          users: ["system:apiserver"]
          verbs: ["get"]
          resources:
            - group: "" # core
              resources: ["namespaces", "namespaces/status", "namespaces/finalize"]

        # Don't log HPA fetching metrics.
        - level: None
          users:
            - system:kube-controller-manager
          verbs: ["get", "list"]
          resources:
            - group: "metrics.k8s.io"

        # Don't log these read-only URLs.
        - level: None
          nonResourceURLs:
            - /healthz*
            - /version
            - /swagger*
            - /liveness
            - /inrotation.txt

        # Don't log events requests.
        - level: None
          resources:
            - group: "" # core
              resources: ["events"]

        # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
        - level: Request
          users: ["kubelet", "system:node-problem-detector", "system:serviceaccount:kube-system:node-problem-detector"]
          verbs: ["update","patch"]
          resources:
            - group: "" # core
              resources: ["nodes/status", "pods/status"]
          omitStages:
            - "RequestReceived"

        # deletecollection calls can be large, don't log responses for expected namespace deletions
        - level: Request
          users: ["system:serviceaccount:kube-system:namespace-controller"]
          verbs: ["deletecollection"]
          omitStages:
            - "RequestReceived"

        # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
        # so only log at the Metadata level.
        - level: Metadata
          resources:
            - group: "" # core
              resources: ["secrets", "configmaps"]
            - group: authentication.k8s.io
              resources: ["tokenreviews"] # Tokenreviews usually contains the token itself
          omitStages:
            - "RequestReceived"


        # Exclude read event of configmaps for controllers also it is fine for viewing
        # configmaps unlike secrets.
        - level: None
          verbs: ["get", "list", "watch"]
          resources:
            - group: "" # core
              resources: ["configmaps"]

        # Get repsonses can be large; skip them.
        - level: Request
          verbs: ["get", "list", "watch"]
          resources:
            - group: "" # core
            - group: "admissionregistration.k8s.io"
            - group: "apiextensions.k8s.io"
            - group: "apiregistration.k8s.io"
            - group: "apps"
            - group: "authentication.k8s.io"
            - group: "authorization.k8s.io"
            - group: "autoscaling"
            - group: "batch"
            - group: "certificates.k8s.io"
            - group: "extensions"
            - group: "metrics.k8s.io"
            - group: "networking.k8s.io"
            - group: "policy"
            - group: "rbac.authorization.k8s.io"
            - group: "scheduling.k8s.io"
            - group: "settings.k8s.io"
            - group: "storage.k8s.io"


        # Default level for known APIs
        # This will not capture read event (get, list, watch) which is specifically defined above
        - level: RequestResponse
          resources:
            - group: "" # core
            - group: "admissionregistration.k8s.io"
            - group: "apiextensions.k8s.io"
            - group: "apiregistration.k8s.io"
            - group: "apps"
            - group: "authentication.k8s.io"
            - group: "authorization.k8s.io"
            - group: "autoscaling"
            - group: "batch"
            - group: "certificates.k8s.io"
            - group: "extensions"
            - group: "metrics.k8s.io"
            - group: "networking.k8s.io"
            - group: "policy"
            - group: "rbac.authorization.k8s.io"
            - group: "scheduling.k8s.io"
            - group: "settings.k8s.io"
            - group: "storage.k8s.io"
          omitStages:
            - "RequestReceived"

        # A catch-all rule to log all other requests at the Metadata level.
        - level: Metadata
          omitStages:
            - "RequestReceived"
    name: audit-policy
    path: /srv/kubernetes/kube-apiserver/audit.conf
    roles:
    - ControlPlane
  - content: |2+

      apiVersion: v1
      kind: Pod
      metadata:
        labels:
          app: aws-encryption-provider
        name: aws-encryption-provider
        namespace: kube-system
      spec:
        priorityClassName: system-cluster-critical
        hostNetwork: true
        containers:
        - image: <REDACTED>/temp-docker-local/aws-encryption-provider:1.2
          name: aws-encryption-provider
          command:
          - /aws-encryption-provider
          - --key=arn:aws:kms:us-east-1:<REDACTED>:key/<REDACTED>
          - --region=us-east-1
          - --listen=/srv/kubernetes/kube-apiserver/kmsplugin/socket.sock
          - --health-port=:8083
          ports:
          - containerPort: 8083
            protocol: TCP
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8083
          volumeMounts:
          - mountPath: /srv/kubernetes/kube-apiserver/kmsplugin
            name: kmsplugin
        volumes:
        - name: kmsplugin
          hostPath:
            path: /srv/kubernetes/kube-apiserver/kmsplugin
            type: DirectoryOrCreate

    name: aws-kms-encryption-config
    path: /etc/kubernetes/manifests/aws-kms-encryption-provider.manifest
    roles:
    - ControlPlane
  - content: |
      ssh-rsa <REDACTED> unused-kops-must-have-one
    name: authorized-keys
    path: /home/ubuntu/authorized-keys.pub
    roles:
    - ControlPlane
    - Node
  hooks:
  - manifest: "[Unit]\n\t\tDescription=Limited resources slice for Kubernetes services\n\t\tDocumentation=man:systemd.special(7)\n\t\tDefaultDependencies=no\n\t\tBefore=slices.target\n\t\tRequires=-.slice\n\t\tAfter=-.slice"
    name: podruntime.slice
    useRawManifest: true
  - before:
    - kubelet.service
    manifest: "Type=oneshot\n\t\tExecStart=/bin/mkdir -p /sys/fs/cgroup/cpuset/podruntime.slice\n\t\tExecStart=/bin/mkdir
      -p /sys/fs/cgroup/hugetlb/podruntime.slice"
    name: podruntime-patch
    requires:
    - podruntime.slice
  - before:
    - local-fs.target
    - umount.target
    execContainer:
      command:
      - /usr/local/bin/pkops-hooks
      - nodeup
      - update-manifest
      - --cluster-alias
      - <REDACTED>
      - --cluster-name
      - k8s.<REDACTED>
      - --region
      - us-east-1
      - --state-bucket
      - s3://c15-kops-state-store
      - --irsa
      image: <REDACTED>/temp-docker-local/pkops-hooks:2.14
    name: modify-api-server.service
    roles:
    - Master
  - before:
    - local-fs.target
    - umount.target
    execContainer:
      command:
      - /usr/local/bin/pkops-hooks
      - nodeup
      - update-sshkeys
      - /home/ubuntu/authorized-keys.pub
      image: <REDACTED>/temp-docker-local/pkops-hooks:2.14
    name: add-ssh-keys.service
    roles:
    - Master
    - Node
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeAPIServer:
    auditLogMaxAge: 30
    auditLogMaxBackups: 10
    auditLogMaxSize: 100
    auditLogPath: /var/log/kube-apiserver-audit.log
    auditPolicyFile: /srv/kubernetes/kube-apiserver/audit.conf
    featureGates:
      RotateKubeletServerCertificate: "true"
    oidcClientID: 0oa1rgmxymfEBv7Xz1d8
    oidcGroupsClaim: groups
    oidcIssuerURL: <REDACTED>
    oidcUsernameClaim: email
    serviceAccountIssuer: <REDACTED>
    serviceAccountKeyFile:
    - /srv/kubernetes/kube-apiserver/irsa-signer-pkcs8.pub
    serviceAccountSigningKeyFile: /srv/kubernetes/kube-apiserver/irsa-signer.key
    tlsCipherSuites:
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_RSA_WITH_AES_128_GCM_SHA256
    tlsMinVersion: VersionTLS12
  kubeControllerManager:
    featureGates:
      RotateKubeletServerCertificate: "true"
    terminatedPodGCThreshold: 1000
    tlsCipherSuites:
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_RSA_WITH_AES_128_GCM_SHA256
    tlsMinVersion: VersionTLS12
  kubeDNS:
    memoryLimit: 2Gi
  kubeProxy:
    enabled: true
  kubelet:
    anonymousAuth: false
    cpuManagerPolicy: static
    enforceNodeAllocatable: pods,kube-reserved
    featureGates:
      RotateKubeletServerCertificate: "true"
    imageGCHighThresholdPercent: 80
    imageGCLowThresholdPercent: 75
    kubeReserved:
      cpu: 200m
      memory: 2Gi
    kubeReservedCgroup: /podruntime.slice
    kubeletCgroups: /podruntime.slice
    readOnlyPort: 0
    runtimeCgroups: /podruntime.slice
    streamingConnectionIdleTimeout: 1h0m0s
    tlsCipherSuites:
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_RSA_WITH_AES_128_GCM_SHA256
    tlsMinVersion: VersionTLS12
  kubernetesApiAccess:
  - 10.0.0.0/8
  - 172.18.0.0/16
  kubernetesVersion: 1.29.15
  masterKubelet:
    featureGates:
      RotateKubeletServerCertificate: "true"
    kubeletCgroups: /podruntime.slice
    runtimeCgroups: /podruntime.slice
    tlsCipherSuites:
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_RSA_WITH_AES_128_GCM_SHA256
    tlsMinVersion: VersionTLS12
  masterPublicName: api.k8s.<REDACTED>
  networkCIDR: 10.93.168.0/21
  networkID: vpc-07eaa4bc2f4ca8685
  networking:
    cilium:
      agentPrometheusPort: 9090
      enablePrometheusMetrics: true
      metrics:
      - -cilium_node_connectivity_latency_seconds
      - -cilium_node_connectivity_status
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 10.0.0.0/8
  subnets:
  - cidr: 10.93.168.64/26
    id: subnet-0633dd8960d08092c
    name: us-east-1a
    type: Private
    zone: us-east-1a
  - cidr: 10.93.168.128/26
    id: subnet-0b3613bc0a3e4eade
    name: us-east-1b
    type: Private
    zone: us-east-1b
  - cidr: 10.93.168.192/26
    id: subnet-0e3cd5576422b42a5
    name: us-east-1c
    type: Private
    zone: us-east-1c
  - cidr: 10.93.168.16/28
    id: subnet-0d6f651d91095c118
    name: utility-us-east-1a
    type: Utility
    zone: us-east-1a
  - cidr: 10.93.168.32/28
    id: subnet-04ccf76daab62a3b7
    name: utility-us-east-1b
    type: Utility
    zone: us-east-1b
  - cidr: 10.93.168.48/28
    id: subnet-0944609d26957bb28
    name: utility-us-east-1c
    type: Utility
    zone: us-east-1c
  topology:
    dns:
      type: Public
  updatePolicy: external

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2025-04-24T13:07:30Z"
  labels:
    kops.k8s.io/cluster: k8s.<REDACTED>
  name: master-us-east-1a
spec:
  cloudLabels:
    business_unit: <REDACTED>
    component: kubernetes
    created_by: k8s.<REDACTED>
    k8s.io/cluster-autoscaler/enabled: "true"
    product: <REDACTED>
  compressUserData: true
  image: <REDACTED>/k8s-<REDACTED>-ubuntu-20.04-3.22
  instanceMetadata:
    httpPutResponseHopLimit: 2
  machineType: t3.xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-us-east-1a
  role: Master
  subnets:
  - us-east-1a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2025-04-24T13:07:30Z"
  labels:
    kops.k8s.io/cluster: k8s.<REDACTED>
  name: master-us-east-1b
spec:
  cloudLabels:
    business_unit: <REDACTED>
    component: kubernetes
    created_by: k8s.<REDACTED>
    k8s.io/cluster-autoscaler/enabled: "true"
    product: <REDACTED>
  compressUserData: true
  image: <REDACTED>/k8s-<REDACTED>-ubuntu-20.04-3.22
  instanceMetadata:
    httpPutResponseHopLimit: 2
  machineType: t3.xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-us-east-1b
  role: Master
  subnets:
  - us-east-1b

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2025-04-24T13:07:30Z"
  labels:
    kops.k8s.io/cluster: k8s.<REDACTED>
  name: master-us-east-1c
spec:
  cloudLabels:
    business_unit: <REDACTED>
    component: kubernetes
    created_by: k8s.<REDACTED>
    k8s.io/cluster-autoscaler/enabled: "true"
    product: <REDACTED>
  compressUserData: true
  image: <REDACTED>/k8s-<REDACTED>-ubuntu-20.04-3.22
  instanceMetadata:
    httpPutResponseHopLimit: 2
  machineType: t3.xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-us-east-1c
  role: Master
  subnets:
  - us-east-1c

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2025-04-24T13:07:30Z"
  labels:
    kops.k8s.io/cluster: k8s.<REDACTED>
  name: nodes
spec:
  additionalUserData:
  - content: |
      #!/bin/bash

      set -o errexit
      set -o nounset
      set -o pipefail

      instance_id=$(curl http://169.254.169.254/latest/meta-data/instance-id)
      aws lambda invoke --function-name <REDACTED>-update-role-<REDACTED> --payload "{\"instance_id\": \"$instance_id\", \"region\": \"us-east-1\", \"action\":\"post-nodeup\",  \"cluster_alias\":\"<REDACTED>\"}" response.json --region us-east-1
    name: z-1-post-nodeup.sh
    type: text/x-shellscript
  cloudLabels:
    business_unit: <REDACTED>
    component: kubernetes
    created_by: k8s.<REDACTED>
    k8s.io/cluster-autoscaler/enabled: "true"
    product: <REDACTED>
  image: <REDACTED>/k8s-<REDACTED>-ubuntu-20.04-3.22
  machineType: m6a.xlarge
  maxSize: 6
  minSize: 3
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  rollingUpdate:
    maxSurge: 30
  subnets:
  - us-east-1a
  - us-east-1b
  - us-east-1c

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2025-04-24T13:07:30Z"
  labels:
    kops.k8s.io/cluster: k8s.<REDACTED>
  name: prometheus-operator
spec:
  additionalUserData:
  - content: |
      #!/bin/bash

      set -o errexit
      set -o nounset
      set -o pipefail

      instance_id=$(curl http://169.254.169.254/latest/meta-data/instance-id)
      aws lambda invoke --function-name <REDACTED>-update-role-<REDACTED> --payload "{\"instance_id\": \"$instance_id\", \"region\": \"us-east-1\", \"action\":\"post-nodeup\",  \"cluster_alias\":\"<REDACTED>\"}" response.json --region us-east-1
    name: z-1-post-nodeup.sh
    type: text/x-shellscript
  cloudLabels:
    business_unit: <REDACTED>
    component: kubernetes
    created_by: k8s.<REDACTED>
    instanceGroupService: prometheus
    k8s.io/cluster-autoscaler/enabled: "true"
    kops.k8s.io/instancegroup: prometheus-operator
    product: <REDACTED>
  image: <REDACTED>/k8s-<REDACTED>-ubuntu-20.04-3.22
  machineType: t3.large
  maxSize: 2
  minSize: 2
  nodeLabels:
    instancegroupService: prometheus
    kops.k8s.io/instancegroup: prometheus-operator
  role: Node
  rootVolumeSize: 40
  subnets:
  - us-east-1a
  taints:
  - dedicated=prometheus:NoSchedule

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

File	1.30.4	1.31.0
nodeup	✅	✅
protokube	✅	✅
channels	✅	✅
kubelet	✅	❌
kubectl	✅	❌
containerd	✅	❌
runc	✅	❌
nerdctl	✅	❌
crictl	✅	❌
... others

[hakman]

Thanks for reporting this @cecobask.
CC @justinsb @rifelpet @ameukam

[hakman]

This should be fixed in #17385 and should be part of the next release.
/close

[k8s-ci-robot]

@hakman: Closing this issue.

In response to this:

This should be fixed in #17385 and should be part of the next release.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.