Default behavior issues after `tdnf install cloud-network-setup`

[dcasota]

Describe the bug

Hi Susant,
here some findings with the default behavior of cloud-network-setup.

See vmware/photon#1390 F. cloud-network fails to start. Is this expected because cloud-network isn't configured?

a) /etc/cloud-network/cloud-network.toml content looks different as in https://vmware.github.io/photon/docs/administration-guide/managing-network-configuration/configuring-a-secondary-network-interface-using-cloud-network/. See port in [Network].

root@ph4 [ ~ ]# cat /etc/cloud-network/cloud-network.toml
[System]
RefreshTimer="300s"
LogLevel="info"
LogFormat="text"

[Network]
Listen="127.0.0.1:5209"
# Supplementary="eth0 eth1"
root@ph4 [ ~ ]#

b) cnctl status network fails and exposes some internal programming informations. cloud-network-setup-cnctl.txt

c) In the onprem constellation, cnctl status system doesn't show any information.

root@ph4 [ ~ ]# cnctl status system
root@ph4 [ ~ ]#

Reproduction steps

1. build make of slipstreamed x86_64 photon-4.0-5347f73fc.iso, it includes cloud-network-setup 0.2.1. cloud-network-setup-info.txt
2. Intentionally, not a cloud network (Azure, AWS) environment, but onprem x86_64 ESXi 8.0 vm, default vm settings for Photon OS, boot from iso, realtime flavor installed.
3. tdnf install cloud-network-setup, reboot afterwards.
   ...

Expected behavior

• No fail default behavior of cloud-network daemon, if no metadata server has been found.
• Matching docs content
• no internal programming information exposure
• cnctl show: features for Azure ?

Additional context

No response

[ssahani]

Failed to fetch instance metadata: 'Get "http://127.0.0.1:5209/api/cloud/network": dial tcp 127.0.0.1:5209: connect: connection refused ```

Is cloud-network-setup is running in azure cloud machine? what is the status of 

``` sudo systemctl status cloud-network.service```

If you clone and install from github does it run correctly ?

[ssahani]

Also please run in debug mode

[System]
RefreshTimer="300s"
LogLevel="debug"
LogFormat="text"

[Network]
Listen="127.0.0.1:5209"
#Supplementary="eth0 eth1"
❯ sudo /usr/bin/cloud-network
INFO[0000] cloud-network: v0.2.1 (built 'go1.19.3')
FATA[0000] Failed to detect cloud environment, Aborting ...

[dcasota]

case distinction
In a metadata server all paths down constellation WITH reboot, the cloud-network.service fails to start.
cloud-network-setup_issue19_1.txt

[dcasota]

case distinction (Azure)
Right after the installation, it remains "dead". Manual restart and it works.

oot@ph4 [ ~ ]# tdnf install cloud-network-setup

Installing:
cloud-network-setup                        x86_64                0.2.1-8.ph4                  photon-updates         15.82M 16587500

Total installed size:  15.82M 16587500
Is this ok [y/N]: y

Downloading:
cloud-network-setup                    4811483 100%
Testing transaction
Running transaction
Installing/Updating: cloud-network-setup-0.2.1-8.ph4.x86_64
Created symlink /etc/systemd/system/multi-user.target.wants/cloud-network.service → /usr/lib/systemd/system/cloud-network.service.

Complete!
root@ph4 [ ~ ]# systemctl status cloud-network.service
● cloud-network.service - Configures network in cloud environment from metadata
     Loaded: loaded (/usr/lib/systemd/system/cloud-network.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
root@ph4 [ ~ ]# 

Replace info with debug in .toml file and restart daemon.

root@ph4 [ ~ ]# systemctl status cloud-network.service
● cloud-network.service - Configures network in cloud environment from metadata
     Loaded: loaded (/usr/lib/systemd/system/cloud-network.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
root@ph4 [ ~ ]# vi /etc/cloud-network/cloud-network.toml 
root@ph4 [ ~ ]# systemctl restart cloud-network.service
root@ph4 [ ~ ]# systemctl status cloud-network.service
● cloud-network.service - Configures network in cloud environment from metadata
     Loaded: loaded (/usr/lib/systemd/system/cloud-network.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-03-08 10:48:28 UTC; 4s ago
   Main PID: 948 (cloud-network)
      Tasks: 10 (limit: 77166)
     Memory: 7.1M
     CGroup: /system.slice/cloud-network.service
             └─948 /usr/bin/cloud-network

Mar 08 10:48:28 ph4 cloud-network[948]: level=debug msg="Configuring network from (azure) metadata"
Mar 08 10:48:28 ph4 cloud-network[948]: level=info msg="Successfully added address='192.168.1.4/24 on link='eth0' ifindex='2'"
Mar 08 10:48:28 ph4 cloud-network[948]: level=info msg="Received address update: {{192.168.1.4 ffffff00} 2 128 0 4294967295 4294967295 true}"
Mar 08 10:48:28 ph4 cloud-network[948]: level=info msg="Address='192.168.1.4/24' added to link ifindex='2'"
Mar 08 10:48:28 ph4 cloud-network[948]: level=info msg="Successfully added default gateway='192.168.1.1' for link='eth0' ifindex='2' table='10
001'"
Mar 08 10:48:28 ph4 cloud-network[948]: level=info msg="Link='eth0' ifindex='2' is now configured"
Mar 08 10:48:28 ph4 cloud-network[948]: level=info msg="Successfully added routing policy rule 'from' in route table='10001' for link='eth0' i
findex='2'"
Mar 08 10:48:28 ph4 cloud-network[948]: level=info msg="Successfully added routing policy rule 'to' in route table='10001' for link='eth0' ifi
ndex='2'"
Mar 08 10:48:28 ph4 cloud-network[948]: level=debug msg="Saving (azure) metadata"
Mar 08 10:48:28 ph4 cloud-network[948]: level=info msg="Local instance metadata cache Server listening at '127.0.0.1':'5209'"
root@ph4 [ ~ ]# 

In this constellation with a reboot, the daemon still works successfully.

git clone from repo, build and start /usr/bin/cloud-network works successfully.
cloud-network-setup_issue19_2.txt

tdnf install cloud-network-setup does not implicitly start the daemon. It needs a daemon restart to make the daemon run.
cloud-network-setup_issue19_3.txt

[ssahani]

strange the same code works after reboot

root@ph5beta [ ~ ]# root@ph5beta [ ~ ]# /usr/bin/cloud-network
INFO[0000] cloud-network: v0.2.1 (built 'go1.19.3')
FATA[0000] Failed to detect cloud environment, Aborting ...^C

It's unable to detect the cloud environment in first go . We need to update a version same as upstream.

[dcasota]

fyi it's a different environment, one on ph5beta-on-ESXi and one on ph4rev2-on-Azure. Tested with slightly different cloud-network-setup versions. It would be nice to solve the issue for the ph4rev2-on-Azure environment.

In a non-privileged user account context, the install process seems to finish successfully the daemon configuration but the activation remains as issue. cloud-network-setup_issue19_4.txt
ExecStart with ambient process capabilities might be worth to have a look at it ? -> more tdnf debug information

[ssahani]

b56df98 should fix cnctl

[ssahani]

fyi it's a different environment, one on ph5beta-on-ESXi and one on ph4rev2-on-Azure. Tested with slightly different cloud-network-setup versions. It would be nice to solve the issue for the ph4rev2-on-Azure environment.

In a non-privileged user account context, the install process seems to finish successfully the daemon configuration but the activation remains as issue. cloud-network-setup_issue19_4.txt

Activation. Well cloud-network-setup only runs in cloud environment. yes to start the daemon privileged user is required

[dcasota]

Tested on ph5beta-on-ESXi

b56df98 should fix cnctl

Solves it! Please add carriage return+line feed.

Tested on ph4rev2-on-Azure

With e.g. networkctl down eth0 as test case start, cnctl still exits with a non-user friendly message.
cloud-network-setup_issue19_5.txt

[dcasota]

fyi same source version ergo same 'cloud daemon inactive (dead)' + 'network down eth0' sub issues in slipstreamed photon-4.0-bff8291af.iso vhd'ified to Azure.

[ssahani]

fyi same source version ergo same 'cloud daemon inactive (dead)' + 'network down eth0' sub issues in slipstreamed photon-4.0-bff8291af.iso vhd'ified to Azure.

It requires at least one interface to be up and a ipv4 address to communicate with the meta data server. Note it does not configure the primary address. Hence it will not work. #22 should drop the too much backtrace