Argo CD applies invalid Kubernetes manifests.

[daftping]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Argo CD swallows the error and applies invalid Kubernetes manifests. Which is inconsistent with kubectl and ServerSideApply behaviour.

To Reproduce

Apply new CRD

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct

Put the manifest with invalid filed into your repo

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
  thisfiledisnotallowed: value

Create Argo CD Application pointing to your repo with manifest

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: tmp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/daftping/tmp.git
    targetRevision: HEAD
    path: manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: cron

Click Sync in Argo CD UI.

Expected behavior

Argo CD returns an error. Similar to kubectl saying that Kubernetes manifest is not valid.

kubectl apply -f manifests/manifest.yaml 
Error from server (BadRequest): error when creating "manifests/manifest.yaml": CronTab in version "v1" cannot be handled as a CronTab: strict decoding error: unknown field "spec.thisfiledisnotallowed"

See the Workaround section for the UI behaviour I would expect.

Screenshots

Version

v2.9.3+6eba5be

Logs

time="2024-01-10T18:45:03Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:4f38d8f7c7e69f828cc9fbb935e1451ed9feed28,Prune:false,DryRun:false,SyncStrategy:&SyncStrategy{Apply:nil,Hook:&SyncStrategyHook{SyncStrategyApply:SyncStrategyApply{Force:false,},},},Resources:[]SyncOperationResource{},Source:nil,Manifests:[],SyncOptions:[],Sources:[]ApplicationSource{},Revisions:[],} {admin false} [] {0 nil}}" application=argocd/tmp
time="2024-01-10T18:45:03Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: cron)" application=argocd/tmp
time="2024-01-10T18:45:03Z" level=info msg="getRepoObjs stats" application=argocd/tmp build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=3 unmarshal_ms=3 version_ms=0
time="2024-01-10T18:45:03Z" level=info msg=Syncing application=argocd/tmp skipHooks=false started=false syncId=00016-nkCrM
time="2024-01-10T18:45:03Z" level=info msg="Tasks (dry-run)" application=argocd/tmp syncId=00016-nkCrM tasks="[Sync/0 resource stable.example.com/CronTab:cron/my-new-cron-object nil->obj (,,)]"
time="2024-01-10T18:45:03Z" level=info msg="Applying resource CronTab/my-new-cron-object in cluster: https://10.96.0.1:443, namespace: cron"
time="2024-01-10T18:45:03Z" level=info msg="Updating operation state. phase: Running -> Running, message: '' -> 'one or more tasks are running'" application=argocd/tmp syncId=00016-nkCrM
time="2024-01-10T18:45:03Z" level=info msg="Applying resource CronTab/my-new-cron-object in cluster: https://10.96.0.1:443, namespace: cron"
time="2024-01-10T18:45:03Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/tmp
time="2024-01-10T18:45:03Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: cron)" application=argocd/tmp
W0110 18:45:03.792425      12 warnings.go:70] unknown field "spec.thisfiledisnotallowed"
time="2024-01-10T18:45:03Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'crontab.stable.example.com/my-new-cron-object created'" application=argocd/tmp kind=CronTab name=my-new-cron-object namespace=cron phase=Sync syncId=00016-nkCrM
time="2024-01-10T18:45:03Z" level=info msg="Updating operation state. phase: Running -> Succeeded, message: 'one or more tasks are running' -> 'successfully synced (all tasks run)'" application=argocd/tmp syncId=00016-nkCrM
time="2024-01-10T18:45:03Z" level=info msg="sync/terminate complete" application=argocd/tmp duration=306.6959ms syncId=00016-nkCrM
time="2024-01-10T18:45:03Z" level=info msg="getRepoObjs stats" application=argocd/tmp build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=3 unmarshal_ms=2 version_ms=0
time="2024-01-10T18:45:03Z" level=info msg="updated 'argocd/tmp' operation (phase: Succeeded)" appNamespace=argocd application=tmp project=default
time="2024-01-10T18:45:03Z" level=info msg="Sync operation to 4f38d8f7c7e69f828cc9fbb935e1451ed9feed28 succeeded" application=tmp dest-namespace=cron dest-server="https://kubernetes.default.svc" reason=OperationCompleted type=Normal
time="2024-01-10T18:45:03Z" level=info msg="Update successful" application=argocd/tmp
time="2024-01-10T18:45:03Z" level=info msg="Reconciliation completed" application=argocd/tmp dedup_ms=0 dest-name= dest-namespace=cron dest-server="https://kubernetes.default.svc" diff_ms=0 fields.level=1 git_ms=3 health_ms=0 live_ms=0 patch_ms=15 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=20
time="2024-01-10T18:45:03Z" level=info msg="Refreshing app status (controller refresh requested), level (2)" application=argocd/tmp
time="2024-01-10T18:45:03Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: cron)" application=argocd/tmp
time="2024-01-10T18:45:03Z" level=info msg="getRepoObjs stats" application=argocd/tmp build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=3 unmarshal_ms=3 version_ms=0
time="2024-01-10T18:45:03Z" level=info msg="Update successful" application=argocd/tmp
time="2024-01-10T18:45:03Z" level=info msg="Reconciliation completed" application=argocd/tmp dedup_ms=0 dest-name= dest-namespace=cron dest-server="https://kubernetes.default.svc" diff_ms=0 fields.level=2 git_ms=3 health_ms=0 live_ms=0 patch_ms=8 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=12

Workaround
Use server-side apply

spec:
  syncPolicy:
    syncOptions:
    - ServerSideApply=true

Argo CD does not sync invalid manifest and returns a clear error.

[rumstead]

Argo CD does client-side apply before deploying manifests which don't catch the error. As you said, the server-side apply would and is the future default.

❯ k apply -f cron-tab.yaml
customresourcedefinition.apiextensions.k8s.io/crontabs.stable.example.com created                                                                                                                                                                                                                             
❯ k apply --dry-run=client -f invalid-cron.yaml
crontab.stable.example.com/my-new-cron-object created (dry run)

[vhosakot]

I hit this issue too today, argoCD deployed an invalid k8s manifest in helm chart, and this invalid manifest failed with both kubectl create ... and helm install ... due to error below:

Error from server: error when creating "gw.yaml": admission webhook "validation.istio.io" denied the request: configuration is invalid: server config must contain at least one host

[daftping]

I hit this issue too today, argoCD deployed an invalid k8s manifest in helm chart, and this invalid manifest failed with both kubectl create ... and helm install ... due to error below:

Error from server: error when creating "gw.yaml": admission webhook "validation.istio.io" denied the request: configuration is invalid: server config must contain at least one host

In the client side apply, Argo CD does not interact with ValidationWebhook therefore can not predict the response.
If you don't want to switch fully to Server Side Apply, I'd recommend checking the new Server-Side Diif strategy
https://argo-cd.readthedocs.io/en/stable/user-guide/diff-strategies/#server-side-diff