Bridge interface not recognized for newly added host if they don't have a physical interface attached #10804

deajan · 2025-04-15T13:14:52Z

deajan
Apr 15, 2025

problem

So I added a KVM hypervisor running AlmaLinux 9.5 to a Cloudstack Management Server via the UI, which failed with error Unable to add the host: Cannot find the server resources at <host>

While digging in the cloudstack management-server.log file, I noticed that my bridge br_npf0 is not found according to the management server:

2025-04-15 14:47:43,865 INFO  [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) PingMap for agent: 1 will not be updated because agent is no longer in the PingMap
2025-04-15 14:47:43,866 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Not processing PingRoutingCommand for agent id=0; can't find the host in the DB
2025-04-15 14:47:43,867 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:[]) (logid:) Seq 1-9217179587367141377: Processing:  { Ans: , MgmtId: 90520739542428, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.CheckNetworkAnswer":{"_reconnect":"false","result":"false","details":"Can not find network: br_npf0","wait":"0","bypassHostMaintenance":"false"}}] }
2025-04-15 14:47:43,867 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:[]) (logid:) Seq 1-9217179587367141377: No more commands found
2025-04-15 14:47:43,867 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) Seq 1-9217179587367141377: Received:  { Ans: , MgmtId: 90520739542428, via: 1(reacted_host_name.local), Ver: v1, Flags: 110, { CheckNetworkAnswer } }
2025-04-15 14:47:43,867 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) Details from executing class com.cloud.agent.api.CheckNetworkCommand: Can not find network: br_npf0
2025-04-15 14:47:43,867 WARN  [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) Unable to setup agent 1 due to Can not find network: br_npf0
2025-04-15 14:47:43,870 WARN  [c.c.a.AlertManagerImpl] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) alertType=[7] dataCenterId=[1] podId=[1] clusterId=[null] message=[Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: br_npf0].
2025-04-15 14:47:43,878 WARN  [c.c.a.AlertManagerImpl] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: br_npf0] and content [Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: br_npf0].
2025-04-15 14:47:43,878 INFO  [c.c.u.e.CSExceptionErrorCode] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) Could not find exception: com.cloud.exception.ConnectionException in error code list for exceptions
[...]
2025-04-15 14:47:43,890 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) Failed to handle host connection: com.cloud.exception.ConnectionException: Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: br_npf0
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.processConnect(NetworkOrchestrator.java:4321)
        at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:553)
        at com.cloud.agent.manager.AgentManagerImpl.sendReadyAndGetAttache(AgentManagerImpl.java:1116)
        at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1135)
        at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1227)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

Looking at my configuration on the KVM host, the bridge br_npf0 exists, is up, has an IP, and can ping the management server.

ip a | grep br_npf0
10: br_npf0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.13.37.2/24 brd 10.13.37.255 scope global noprefixroute br_npf0
28: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br_npf0 state UP group default qlen 1000

# pinging the cloudstack management server
ping cloudstack01i.npf.local -c 1
PING cloudstack01i.npf.local (10.13.37.250) 56(84) bytes of data.
64 bytes from cloudstack01i.npf.local (10.13.37.250): icmp_seq=1 ttl=64 time=0.687 ms

--- cloudstack01i.npf.local ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.687/0.687/0.687/0.000 ms

The management server zone is setup with that exact same bridge name:

Is there any direction to point me to ?

versions

Cloudstack 4.20 running on AlmaLinux 9.5
KVM host AlmaLinux 9.5 with bridge setup via NetworkManager

The steps to reproduce the bug

What to do about it?

No response

weizhouapache · 2025-04-15T13:25:03Z

weizhouapache
Apr 15, 2025
Collaborator

@deajan
can you run some commands on the kvm host ?

bridge=brXXX
ls -l /sys/devices/virtual/net/$bridge/brif
ls -ld /sys/class/net/$bridge/

0 replies

deajan · 2025-04-15T13:34:46Z

deajan
Apr 15, 2025
Author

Sure

export bridge=br_npf0
ls -l /sys/devices/virtual/net/${bridge}/brif
total 0
lrwxrwxrwx. 1 root root 0 Apr 15 15:32 eno2 -> ../../../../pci0000:00/0000:00:1c.0/0000:02:00.1/net/eno2/brport
ls -ld /sys/class/net/${bridge}/
drwxr-xr-x. 7 root root 0 Apr 15 15:28 /sys/class/net/br_npf0/

Dates are recent because I disconnected / reconnected eno2 just to make sure (and yes, the management server still complains)

0 replies

weizhouapache · 2025-04-15T13:48:47Z

weizhouapache
Apr 15, 2025
Collaborator

@deajan
looks good
can you share the agent.log ?

0 replies

deajan · 2025-04-15T13:52:00Z

deajan
Apr 15, 2025
Author

Sure:
It looks like it complains about lscpu (and iscsi I haven't setup, going to use local storage for tests), but nothing about the bridges.

2025-04-15 15:49:59,027 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching CPU speed from "host capabilities"
2025-04-15 15:49:59,031 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Retrieved value [2095080000] from "host capabilities". This corresponds to a CPU speed of [2095] MHz.
2025-04-15 15:49:59,031 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching host CPU arch
2025-04-15 15:49:59,063 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Host uses control group [cgroup2fs].
2025-04-15 15:49:59,063 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Calculating the max shares of the host.
2025-04-15 15:49:59,063 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) The max shares of the host is [167600].
2025-04-15 15:49:59,256 WARN  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Execution of process [28600] for command [sudo grep InitiatorName= /etc/iscsi/initiatorname.iscsi ] failed.
2025-04-15 15:49:59,256 WARN  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Process [28600] for command [sudo grep InitiatorName= /etc/iscsi/initiatorname.iscsi ] encountered the error: [grep: /etc/iscsi/initiatorname.iscsi: Aucun fichier ou dossier de ce type].
2025-04-15 15:49:59,429 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Attempting to create storage pool 3fd9d9aa-40fc-42d5-b99e-12fe590fd42d (Filesystem) in libvirt
2025-04-15 15:49:59,431 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Found existing defined storage pool 3fd9d9aa-40fc-42d5-b99e-12fe590fd42d, using it.
2025-04-15 15:49:59,431 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Trying to fetch storage pool 3fd9d9aa-40fc-42d5-b99e-12fe590fd42d from libvirt
2025-04-15 15:49:59,549 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Process agent startup answer, agent id = 0
2025-04-15 15:49:59,549 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Startup Response Received: agent id = 0
2025-04-15 15:49:59,703 WARN  [cloud.agent.Agent] (UgentTask-5:[]) (logid:) Unable to send request: null
2025-04-15 15:50:03,833 INFO  [cloud.agent.Agent] (Agent-Handler-4:[]) (logid:) Connected to the host: 10.131.37.250
2025-04-15 15:50:04,663 INFO  [cloud.agent.Agent] (Agent-Handler-4:[]) (logid:) Lost connection to host: 10.131.37.250. Attempting reconnection while we still have 0 commands in progress.
2025-04-15 15:50:04,663 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:) NioClient connection closed
2025-04-15 15:50:04,663 INFO  [cloud.agent.Agent] (Agent-Handler-4:[]) (logid:) Reconnecting to host:10.131.37.250
2025-04-15 15:50:04,664 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:) Connecting to 10.131.37.250:8250
2025-04-15 15:50:04,664 INFO  [utils.nio.Link] (Agent-Handler-4:[]) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2025-04-15 15:50:04,786 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:) SSL: Handshake done
2025-04-15 15:50:04,786 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:) Connected to 10.131.37.250:8250
2025-04-15 15:50:04,878 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching CPU speed from command "lscpu".
2025-04-15 15:50:05,000 ERROR [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Unable to retrieve the CPU speed from lscpu. java.lang.NullPointerException

2025-04-15 15:50:05,000 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching CPU speed from file [/sys/devices/system/cpu/cpu0/cpufreq/base_frequency].
2025-04-15 15:50:05,000 ERROR [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Unable to retrieve the CPU speed from file [/sys/devices/system/cpu/cpu0/cpufreq/base_frequency] java.io.FileNotFoundException: /sys/devices/system/cpu/cpu0/cpufreq/base_frequency (Aucun fichier ou dossier de ce type)
        at java.base/java.io.FileInputStream.open0(Native Method)
        at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
        at java.base/java.io.FileReader.<init>(FileReader.java:60)
        at org.apache.cloudstack.utils.linux.KVMHostInfo.getCpuSpeedFromFile(KVMHostInfo.java:152)
        at org.apache.cloudstack.utils.linux.KVMHostInfo.getCpuSpeed(KVMHostInfo.java:121)
        at org.apache.cloudstack.utils.linux.KVMHostInfo.getHostInfoFromLibvirt(KVMHostInfo.java:196)
        at org.apache.cloudstack.utils.linux.KVMHostInfo.<init>(KVMHostInfo.java:69)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.initialize(LibvirtComputingResource.java:3672)
        at com.cloud.agent.Agent.sendStartup(Agent.java:458)
        at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1128)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2025-04-15 15:50:05,001 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching CPU speed from "host capabilities"
2025-04-15 15:50:05,004 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Retrieved value [2095080000] from "host capabilities". This corresponds to a CPU speed of [2095] MHz.
2025-04-15 15:50:05,004 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching host CPU arch
2025-04-15 15:50:05,038 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Host uses control group [cgroup2fs].
2025-04-15 15:50:05,038 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Calculating the max shares of the host.
2025-04-15 15:50:05,038 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) The max shares of the host is [167600].
2025-04-15 15:50:05,243 WARN  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Execution of process [32149] for command [sudo grep InitiatorName= /etc/iscsi/initiatorname.iscsi ] failed.
2025-04-15 15:50:05,244 WARN  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Process [32149] for command [sudo grep InitiatorName= /etc/iscsi/initiatorname.iscsi ] encountered the error: [grep: /etc/iscsi/initiatorname.iscsi: Aucun fichier ou dossier de ce type].
2025-04-15 15:50:05,436 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Attempting to create storage pool 3fd9d9aa-40fc-42d5-b99e-12fe590fd42d (Filesystem) in libvirt
2025-04-15 15:50:05,438 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Found existing defined storage pool 3fd9d9aa-40fc-42d5-b99e-12fe590fd42d, using it.
2025-04-15 15:50:05,439 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Trying to fetch storage pool 3fd9d9aa-40fc-42d5-b99e-12fe590fd42d from libvirt
2025-04-15 15:50:05,558 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Process agent startup answer, agent id = 0
2025-04-15 15:50:05,558 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Startup Response Received: agent id = 0
2025-04-15 15:50:05,720 WARN  [cloud.agent.Agent] (UgentTask-5:[]) (logid:) Unable to send request: null
2025-04-15 15:50:09,786 INFO  [cloud.agent.Agent] (Agent-Handler-4:[]) (logid:) Connected to the host: 10.131.37.250
2025-04-15 15:50:10,681 INFO  [cloud.agent.Agent] (Agent-Handler-4:[]) (logid:) Lost connection to host: 10.131.37.250. Attempting reconnection while we still have 0 commands in progress.
2025-04-15 15:50:10,682 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:) NioClient connection closed
2025-04-15 15:50:10,682 INFO  [cloud.agent.Agent] (Agent-Handler-4:[]) (logid:) Reconnecting to host:10.131.37.250
2025-04-15 15:50:10,682 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:) Connecting to 10.131.37.250:8250
2025-04-15 15:50:10,683 INFO  [utils.nio.Link] (Agent-Handler-4:[]) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2025-04-15 15:50:10,815 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:) SSL: Handshake done
2025-04-15 15:50:10,815 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:) Connected to 10.131.37.250:8250
2025-04-15 15:50:10,889 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching CPU speed from command "lscpu".
2025-04-15 15:50:11,005 ERROR [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Unable to retrieve the CPU speed from lscpu. java.lang.NullPointerException

2025-04-15 15:50:11,005 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching CPU speed from file [/sys/devices/system/cpu/cpu0/cpufreq/base_frequency].
2025-04-15 15:50:11,006 ERROR [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Unable to retrieve the CPU speed from file [/sys/devices/system/cpu/cpu0/cpufreq/base_frequency] java.io.FileNotFoundException: /sys/devices/system/cpu/cpu0/cpufreq/base_frequency (Aucun fichier ou dossier de ce type)
        at java.base/java.io.FileInputStream.open0(Native Method)
        at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
        at java.base/java.io.FileReader.<init>(FileReader.java:60)
        at org.apache.cloudstack.utils.linux.KVMHostInfo.getCpuSpeedFromFile(KVMHostInfo.java:152)
        at org.apache.cloudstack.utils.linux.KVMHostInfo.getCpuSpeed(KVMHostInfo.java:121)
        at org.apache.cloudstack.utils.linux.KVMHostInfo.getHostInfoFromLibvirt(KVMHostInfo.java:196)
        at org.apache.cloudstack.utils.linux.KVMHostInfo.<init>(KVMHostInfo.java:69)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.initialize(LibvirtComputingResource.java:3672)
        at com.cloud.agent.Agent.sendStartup(Agent.java:458)
        at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1128)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2025-04-15 15:50:11,006 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching CPU speed from "host capabilities"
2025-04-15 15:50:11,009 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Retrieved value [2095080000] from "host capabilities". This corresponds to a CPU speed of [2095] MHz.
2025-04-15 15:50:11,009 INFO  [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching host CPU arch
2025-04-15 15:50:11,036 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Host uses control group [cgroup2fs].
2025-04-15 15:50:11,036 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Calculating the max shares of the host.
2025-04-15 15:50:11,036 INFO  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) The max shares of the host is [167600].
2025-04-15 15:50:11,237 WARN  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Execution of process [35023] for command [sudo grep InitiatorName= /etc/iscsi/initiatorname.iscsi ] failed.
2025-04-15 15:50:11,237 WARN  [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Process [35023] for command [sudo grep InitiatorName= /etc/iscsi/initiatorname.iscsi ] encountered the error: [grep: /etc/iscsi/initiatorname.iscsi: Aucun fichier ou dossier de ce type].
2025-04-15 15:50:11,398 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Attempting to create storage pool 3fd9d9aa-40fc-42d5-b99e-12fe590fd42d (Filesystem) in libvirt
2025-04-15 15:50:11,400 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Found existing defined storage pool 3fd9d9aa-40fc-42d5-b99e-12fe590fd42d, using it.
2025-04-15 15:50:11,400 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Trying to fetch storage pool 3fd9d9aa-40fc-42d5-b99e-12fe590fd42d from libvirt
2025-04-15 15:50:11,514 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Process agent startup answer, agent id = 0
2025-04-15 15:50:11,515 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Startup Response Received: agent id = 0
2025-04-15 15:50:11,666 WARN  [cloud.agent.Agent] (UgentTask-5:[]) (logid:) Unable to send request: null

0 replies

deajan · 2025-04-15T13:54:53Z

deajan
Apr 15, 2025
Author

@weizhouapache While digging into my problem, I am thinking of a potential issue

Do the bridges need to be physically connected ?

I have one bridge (br_bgp0) that is "fed" by a vxlan interface (not connected to a physical interface but to a wireguard instance in order to transport some public IPs into my lab) on this KVM host.
This allows me to have my public IPs directly bridged from my bgp routers into my lab host.

Is this "allowed" ?

If not, that could perhaps explain why cloudstack complains about my bridges, even though they work.

0 replies

weizhouapache · 2025-04-15T13:55:59Z

weizhouapache
Apr 15, 2025
Collaborator

change the log level and restart cloudstack-agent ?

sed -i "s/INFO/DEBUG/g" /etc/cloudstack/agent/log4j-cloud.xml

Is this "allowed" ?

yes, that's ok

0 replies

deajan · 2025-04-15T14:11:01Z

deajan
Apr 15, 2025
Author

Hmmm... interesting.

2025-04-15 15:59:16,309 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:[]) (logid:04967b5d) failing to get physical interface from bridge br_bgp0, did not find an eth*, bond*, team*, vlan*, em*, p*p*, ens*, eno*, enp*, or enx* in /sys/devices/virtual/net/br_bgp0/brif

Looks like indeed using a bridge in cloudstack is limited to physical interfaces.
I've done a (stupid) check:

nmcli c add type dummy ifname ethdummy0 con-name ethdummy0
nmcli c mod ethdummy0 master br_bgp0
nmcli c up ethdummy0

Now the agent doesn't complain anymore, since there's now a eth* in /sys/devices/virtual/net/br_bgp0/brif

I guess that indeed the bridge tests are a bit too restrictive.
I wonder if the test should be less restrictive, like checking if there is at least one non vtnet* interface in the bridge, so people would be allowed to use whatever they want behind the bridge (like vxlans, gretap or whatever).
This would also allow make cloudstack future proof for ethernet driver names. Perhaps one day we'll have igb* or whatever driver names.

I can do python and bash PRs, but I really am not java fluent.
Is there anyway this could be discussed for perhaps some next release ?

@weizhouapache Big thanks for the hints.

0 replies

weizhouapache · 2025-04-15T14:30:43Z

weizhouapache
Apr 15, 2025
Collaborator

Now the agent doesn't complain anymore, since there's now a eth* in /sys/devices/virtual/net/br_bgp0/brif

that's strange.
in your previous reply, it showed that the eno2 is present

ls -l /sys/devices/virtual/net/${bridge}/brif
total 0
lrwxrwxrwx. 1 root root 0 Apr 15 15:32 eno2 -> ../../../../pci0000:00/0000:00:1c.0/0000:02:00.1/net/eno2/brport

0 replies

deajan · 2025-04-15T14:45:23Z

deajan
Apr 15, 2025
Author

in your previous reply, it showed that the eno2 is present

I think I mixed up br_npf0 and br_bgp0 in my tests (deleted & recreated bridges multiple times in order to diagnose my issue).

Point is, my bgp interface, the bridge br_bgp0 with a vxlan interface, wasn't accepted by Cloudstack agent since the bridge test complained that there wasn't a physical interface. There's still no physical interface yet, but I fooled the agent test by creating a dummy interface slave in that bridge and naming it ethdummy0.

ls /sys/devices/virtual/net/br_bgp0/brif
ethdummy0  vx_wg0

I think those tests should be relaxed, as I think they only should test that there are other interfaces than vnet* connected to the bridge.

0 replies

weizhouapache · 2025-04-15T15:44:21Z

weizhouapache
Apr 15, 2025
Collaborator

in your previous reply, it showed that the eno2 is present

I think I mixed up br_npf0 and br_bgp0 in my tests (deleted & recreated bridges multiple times in order to diagnose my issue).

Point is, my bgp interface, the bridge br_bgp0 with a vxlan interface, wasn't accepted by Cloudstack agent since the bridge test complained that there wasn't a physical interface. There's still no physical interface yet, but I fooled the agent test by creating a dummy interface slave in that bridge and naming it ethdummy0.
ls /sys/devices/virtual/net/br_bgp0/brif
ethdummy0  vx_wg0
I think those tests should be relaxed, as I think they only should test that there are other interfaces than vnet* connected to the bridge.

hmm, has the kvm host been added to cloudstack ?

0 replies

deajan · 2025-04-15T16:20:53Z

deajan
Apr 15, 2025
Author

Yes, host is now visible and manageable in cloudstack.

Anything I need to check ?

0 replies

weizhouapache · 2025-04-16T06:07:31Z

weizhouapache
Apr 16, 2025
Collaborator

Anything I need to check ?

Are the system vms Running and agent states are Up ?

0 replies

weizhouapache · 2025-04-16T09:33:02Z

weizhouapache
Apr 16, 2025
Collaborator

@deajan
just to confirm, is br_bgp0 used as the kvm network label of a physical network ?

0 replies

deajan · 2025-04-16T10:46:01Z

deajan
Apr 16, 2025
Author

@weizhouapache You mean like this ?

I don't have system VMs yet (it's a fresh cloudstack lab setup), but host is up and running according to management server.
I've redone a specific check:

Disconnect eno2 physical interface and remove it from bridge br_npf0
Host status goes to disabled
Add ethdummy to br_npf0
Host status goes to enabled

Of course, this will prevent me from using cloudstack properly until I reconnect eno2, but at it validates the assumption.

0 replies

weizhouapache · 2025-04-16T11:13:56Z

weizhouapache
Apr 16, 2025
Collaborator

br_npf0

I am a bit confused about br_npf0 and br_bgp0
anyway, all bridges used by cloudstack (check kvm network label) should have a physical ointerface (or virtual or dummy interface with proper name)

do you think anything we could improve or fix ?

0 replies

deajan · 2025-04-16T11:22:49Z

deajan
Apr 16, 2025
Author

Sorry, as I said, at one moment, I think I have mixed both while doing tests creating/deleting bridges to find what the issue is.
For instance, br_npf has eno2 as slave interface, br_bgp0 has a vxlan as slave interface.

So basically the test to see whether there is eth*, bond*, team*, vlan*, em*, p*p*, ens*, eno*, enp*, or enx* in /sys/devices/virtual/net/${bridge}/brif is not sufficient.

I would change that check to report successful if at least one slave different than "vnet*" "or "vmbr*" is present (so we outrule running VM interfaces from uplinks).

This would allow to have bridge slaves interfaces being VXLan, GRETAP, Geneve or whatever fancy interfaces someone would like to use. It would also allow Cloudstack to be future proof if someday a new ethernet driver naming comes out.

0 replies

weizhouapache · 2025-04-16T13:09:39Z

weizhouapache
Apr 16, 2025
Collaborator

I would change that check to report successful if at least one slave different than "vnet*" "or "vmbr*" is present (so we outrule running VM interfaces from uplinks).

as far as I know, vnet* is used by vm instances on kvm host in cloudstack, vmbr* (vm bridge ?) is default bridge name of proxmox.

0 replies

deajan · 2025-04-16T16:40:13Z

deajan
Apr 16, 2025
Author

You are right, I actually ment virbr* which would be NATed vm instances on kvm instead of vmbr*... I don't use them often enough to remember the right name. Had to create one to check.

1 reply

DaanHoogland May 1, 2025
Collaborator

@deajan is this the answer to your original query ? I ask so we can mark it as such for future reference and close the issue.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bridge interface not recognized for newly added host if they don't have a physical interface attached #10804

{{title}}

Replies: 18 comments 1 reply

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Bridge interface not recognized for newly added host if they don't have a physical interface attached #10804

deajan Apr 15, 2025

problem

versions

The steps to reproduce the bug

What to do about it?

Replies: 18 comments · 1 reply

weizhouapache Apr 15, 2025 Collaborator

deajan Apr 15, 2025 Author

weizhouapache Apr 15, 2025 Collaborator

deajan Apr 15, 2025 Author

deajan Apr 15, 2025 Author

weizhouapache Apr 15, 2025 Collaborator

deajan Apr 15, 2025 Author

weizhouapache Apr 15, 2025 Collaborator

deajan Apr 15, 2025 Author

weizhouapache Apr 15, 2025 Collaborator

deajan Apr 15, 2025 Author

weizhouapache Apr 16, 2025 Collaborator

weizhouapache Apr 16, 2025 Collaborator

deajan Apr 16, 2025 Author

weizhouapache Apr 16, 2025 Collaborator

deajan Apr 16, 2025 Author

weizhouapache Apr 16, 2025 Collaborator

deajan Apr 16, 2025 Author

DaanHoogland May 1, 2025 Collaborator

deajan
Apr 15, 2025

Replies: 18 comments 1 reply

weizhouapache
Apr 15, 2025
Collaborator

deajan
Apr 15, 2025
Author

weizhouapache
Apr 15, 2025
Collaborator

deajan
Apr 15, 2025
Author

deajan
Apr 15, 2025
Author

weizhouapache
Apr 15, 2025
Collaborator

deajan
Apr 15, 2025
Author

weizhouapache
Apr 15, 2025
Collaborator

deajan
Apr 15, 2025
Author

weizhouapache
Apr 15, 2025
Collaborator

deajan
Apr 15, 2025
Author

weizhouapache
Apr 16, 2025
Collaborator

weizhouapache
Apr 16, 2025
Collaborator

deajan
Apr 16, 2025
Author

weizhouapache
Apr 16, 2025
Collaborator

deajan
Apr 16, 2025
Author

weizhouapache
Apr 16, 2025
Collaborator

deajan
Apr 16, 2025
Author

DaanHoogland May 1, 2025
Collaborator