CNI chaining fails and does not self-heal in AKS

[tnn-simon]

Describe the bug
When Antrea agent pods start up before the pods of the default network stack of AKS, the CNI chaining occasionally fails and pod creation is blocked for the given nodes. Manually restarting the install-cni container, after the AKS network pods have started successfully, leads to a sufficient recovery of the CNI chaining.

To Reproduce

1. Create an AKS cluster with Azure CNI Overlay as network plugin.
2. Install Antrea in network-policy-only mode.
3. Scale up the cluster with a couple of new nodes and hope for Antrea agent pods to start before the Azure CNI overlay pods.
4. Watch new node(s) being marked as NotReady with error message: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized.
5. Open a shell into the install-cni container and print its content: cat /host/etc/cni/net.d/05-antrea.conflist. Which shows that it is empty.

Expected
Expected Antrea not to leave the CNI plugin setup in an invalid state, but resolve the issue automatically.

Actual behavior
Node is marked as NotReady until the AKS engine decides to replace the node, due to it being stuck in NotReady state. The issue also blocks AKS clusters from properly autoscaling, as the new nodes are not usable and can not host pods (as they are NotReady).

Versions:
Antrea: 2.0.1
Kubernetes: 1.28.5
Container runtime: Containerd 1.7.15-1
Linux kernel version: 5.15.0-1061-azure

Additional context

I have applied the following workaround:

    # fail if the content is empty (custom)
    if [[ "$content" =~ ^[[:space:]]*$ ]]; then
        log_error "install_cni_chaining" "Chaining failed. Proposed content is empty."
        exit 1
    fi

    # fail if the content is not valid JSON (custom)
    echo "$content" | jq empty >/dev/null 2>&1
    if [[ $? != 0 ]]; then
        log_error "install_cni_chaining" "CNI conf file is not valid JSON"
        exit 1
    fi

to this script and line number:

antrea/build/images/scripts/install_cni_chaining

Line 90 in fc40157

[tnqn]

@tnn-simon thanks for reporting the issue and the proposed workaround. If I understand correctly, the problem is because antrea-agent pod may start before azure-cni pod on a new Node, while the latter creates the cni conf.

I think we can fix it by v2.1 and backport it.

[tnn-simon]

@tnn-simon thanks for reporting the issue and the proposed workaround. If I understand correctly, the problem is because antrea-agent pod may start before azure-cni pod on a new Node, while the latter creates the cni conf.

I think we can fix it by v2.1 and backport it.

Your understanding is correct. 2.1 and backport sounds magnificent.