Could not reload dnsmasq (LXD 5.20)

[jardon]

Required information

• Distribution: Ubuntu
• Distribution version: 23.10
• The output of "lxc info" or if that fails:
  lxd-info.txt

Issue description

A brief description of the problem. Should include what you were
attempting to do, what you did, what happened and what you expected to
see happen.

$ snapcraft pack --verbosity debug
2024-01-31 13:46:48.090 Starting Snapcraft 8.0.2.post24+git2d3fecb4                                                                                                                         
2024-01-31 13:46:48.090 Logging execution to '/home/jardon/.local/state/snapcraft/log/snapcraft-20240131-134648.089582.log'                                                                 
2024-01-31 13:46:48.091 lifecycle command: 'pack', arguments: Namespace(destructive_mode=False, use_lxd=False, debug=False, enable_manifest=False, manifest_image_information=None, bind_ssh=False, build_for=None, http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None)                                                              
2024-01-31 13:46:48.091 command: pack, arguments: Namespace(destructive_mode=False, use_lxd=False, debug=False, enable_manifest=False, manifest_image_information=None, bind_ssh=False, build_for=None, http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None)                                                                          
2024-01-31 13:46:48.099 CPU count (from process affinity): 16                                                                                                                               
2024-01-31 13:46:48.099 Invalid SNAPCRAFT_MAX_PARALLEL_BUILD_COUNT ''                                                                                                                       
2024-01-31 13:46:48.099 Running on amd64 for amd64                                                                                                                                          
2024-01-31 13:46:48.101 Checking build provider availability                                                                                                                                
2024-01-31 13:46:48.106 Retrieved snap config: {}                                                                                                                                           
2024-01-31 13:46:48.106 Using default provider 'lxd' on linux system.                                                                                                                       
2024-01-31 13:46:48.222 Executing on host: lxc --project default profile show local:default                                                                                                 
2024-01-31 13:46:48.308 Using hostname 'snapcraft-qt-framework-5-15-core22-on-amd64-for-amd64-22843192'                                                                                     
2024-01-31 13:46:48.308 Launching instance...                                                                                                                                               
2024-01-31 13:46:48.308 Executing on host: lxc remote list --format=yaml                                                                                                                    
2024-01-31 13:46:48.389 Remote 'craft-com.ubuntu.cloud-buildd' already exists.                                                                                                              
2024-01-31 13:46:48.389 Executing on host: lxc project list local: --format=yaml                                                                                                            
2024-01-31 13:46:48.483 Set LXD instance name to 'snapcraft-qt-framework-5-15-core22-on-amd64-for-amd64-22843192'                                                                           
2024-01-31 13:46:48.483 Checking for instance 'snapcraft-qt-framework-5-15-core22-on-amd64-for-amd64-22843192' in project 'snapcraft' in remote 'local'                                     
2024-01-31 13:46:48.483 Executing on host: lxc --project snapcraft list local: --format=yaml                                                                                                
2024-01-31 13:46:48.563 Instance 'snapcraft-qt-framework-5-15-core22-on-amd64-for-amd64-22843192' does not exist.                                                                           
2024-01-31 13:46:48.563 Set LXD instance name to 'base-instance-snapcraft-buildd-base-v60--689084613984e0adb44d'                                                                            
2024-01-31 13:46:48.563 Checking for base instance 'base-instance-snapcraft-buildd-base-v60--689084613984e0adb44d' in project 'snapcraft' in remote 'local'                                 
2024-01-31 13:46:48.563 Executing on host: lxc --project snapcraft list local: --format=yaml                                                                                                
2024-01-31 13:46:48.646 Base instance 'base-instance-snapcraft-buildd-base-v60--689084613984e0adb44d' does not exist.                                                                       
2024-01-31 13:46:48.646 Creating new instance from remote                                                                                                                                   
2024-01-31 13:46:48.646 Creating new base instance from remote                                                                                                                              
2024-01-31 13:46:48.647 Creating new base instance from image 'core22' from remote 'craft-com.ubuntu.cloud-buildd'                                                                          
2024-01-31 13:46:48.647 Executing on host: lxc --project snapcraft info local:                                                                                                              
2024-01-31 13:46:48.760 Executing on host: lxc --project snapcraft launch craft-com.ubuntu.cloud-buildd:core22 local:base-instance-snapcraft-buildd-base-v60--689084613984e0adb44d --config 'raw.idmap=both 1000 0' --config security.syscalls.intercept.mknod=true --config user.craft_providers.status=STARTING --config user.craft_providers.timer=2024-01-31T18:46:48.760284+00:00 --config user.craft_providers.pid=172197                                                                                                                                                       
2024-01-31 13:46:48.876 Failed to launch instance base-instance-snapcraft-buildd-base-v60--689084613984e0adb44d, retrying 0.                                                                
2024-01-31 13:46:48.876 Command '['lxc', '--project', 'snapcraft', 'launch', 'craft-com.ubuntu.cloud-buildd:core22', 'local:base-instance-snapcraft-buildd-base-v60--689084613984e0adb44d', '--config', 'raw.idmap=both 1000 0', '--config', 'security.syscalls.intercept.mknod=true', '--config', 'user.craft_providers.status=STARTING', '--config', 'user.craft_providers.timer=2024-01-31T18:46:48.760284+00:00', '--config', 'user.craft_providers.pid=172197']' returned non-zero exit status 1.                                                                                
2024-01-31 13:46:48.876 craft-providers error: Failed to launch instance 'base-instance-snapcraft-buildd-base-v60--689084613984e0adb44d'.
* Command that failed: "lxc --project snapcraft launch craft-com.ubuntu.cloud-buildd:core22 local:base-instance-snapcraft-buildd-base-v60--689084613984e0adb44d --config 'raw.idmap=both 1000 0' --config security.syscalls.intercept.mknod=true --config user.craft_providers.status=STARTING --config user.craft_providers.timer=2024-01-31T18:46:48.760284+00:00 --config user.craft_providers.pid=172197"
* Command exit code: 1
* Command output: b'Creating base-instance-snapcraft-buildd-base-v60--689084613984e0adb44d\n'
* Command standard error output: b'Error: Failed instance creation: Failed creating instance record: Failed initialising instance: Failed to add device "eth0": Could not reload dnsmasq: Could not reload process: permission denied\n'                                                           
2024-01-31 13:46:48.876 Full execution log: '/home/jardon/.local/state/snapcraft/log/snapcraft-20240131-134648.089582.log'

Steps to reproduce

1. Install lxd version 5.20
2. Install snapcraft
3. Build any snap project

Information to attach

• dmesg.txt
• Container log (lxc info NAME --show-log)
• container-config.txt
• lxd-info.txt
• Output of the client with --debug
• Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

[tomponline]

Hi what is your kernel version?

Also please show output of "sudo sysctl -a" and "sudo snap list"

I'm thinking we may need to cherry pick this fix into latest/stable as it seems that the snapd snap has been updated.

#12713

Cc @alexmurray

[simondeziel]

Hi, could you please also check dmesg is there are Apparmor denial around the time dnsmasq is being reloaded?

Update: sorry I missed the attached file.

[tomponline]

Looks like it

6519.604675] audit: type=1400 audit(1706729169.512:2432): apparmor="DENIED" operation="signal" class="signal" profile="lxd_dnsmasq-lxdbr0_</var/snap/lxd/common/lxd>" pid=307957 comm="lxd" requested_mask="receive" denied_mask="receive" signal=kill peer="snap.lxd.daemon"

[alexmurray]

It might be sufficient to just pull the change to lxd/apparmor/network_dnsmasq.go from that PR (although I suspect the whole change is needed since I expect you are using the snapd from the edge channel).

What version of snapd are you using @jardon ? Can you post the output of snap info snapd?

[tomponline]

Confirmed this is an issue only in latest/stable and 5.20/stable because 5.19/stable they only have canonical/lxd-pkg-snap#189 and not #12713 (whereas 5.19 has neither).

We need to cherry-pick #12713 into latest/stable, 5.20/stable and the upcoming 5.0/* channels.

[tomponline]

@jardon you should be able to get something work using snap refresh lxd --channel=latest/edge for now.

[tomponline]

LXD 5.0.3 in 5.0/candidate isn't affected, same as LXD 5.19 so will consider whether we need to cherry-pick those changes just yet.

[tomponline]

Fix is in latest/candidate now:

lxc launch ubuntu-daily:23.10 vmantic --vm
lxc shell vmantic
root@vmantic:~# snap list
Name    Version       Rev    Tracking         Publisher   Notes
core22  20231123      1033   latest/stable    canonical✓  base
lxd     5.19-8635f82  26200  latest/stable/…  canonical✓  -
snapd   2.61.1        20671  latest/stable    canonical✓  snapd
root@vmantic:~# snap refresh lxd --channel=latest/stable
lxd 5.20-c4bbef1 from Canonical✓ refreshed
root@vmantic:~# lxd init --auto
root@vmantic:~# lxc launch ubuntu-minimal:22.04 c1
Creating c1
Starting c1                                 
root@vmantic:~# snap refresh snapd --channel=latest/edge
2024-02-01T15:03:38Z INFO Waiting for automatic snapd restart...
2024-02-01T15:03:39Z INFO Waiting for automatic snapd restart...
2024-02-01T15:03:40Z INFO Waiting for automatic snapd restart...
2024-02-01T15:03:41Z INFO Waiting for automatic snapd restart...
snapd (edge) 2.61.1+git1679.gcd608ef from Canonical✓ refreshed
root@vmantic:~# snap stop lxd
2024-02-01T15:03:54Z INFO Waiting for "snap.lxd.daemon.service" to stop.
Stopped.
root@vmantic:~# snap start lxd
Started.
root@vmantic:~# lxc launch ubuntu-minimal:22.04 c2
Creating c2
Error: Failed instance creation: Failed creating instance record: Failed initialising instance: Failed to add device "eth0": Could not reload dnsmasq: Could not reload process: permission denied
root@vmantic:~# snap refresh lxd --channel=latest/candidate
2024-02-01T15:04:27Z INFO Waiting for "snap.lxd.daemon.service" to stop.
lxd (candidate) 5.20-95ec499 from Canonical✓ refreshed
root@vmantic:~# lxc launch ubuntu-minimal:22.04 c2
Creating c2
Starting c2                               
root@vmantic:~# lxc ls
+------+---------+---------------------+----------------------------------------------+-----------+-----------+
| NAME |  STATE  |        IPV4         |                     IPV6                     |   TYPE    | SNAPSHOTS |
+------+---------+---------------------+----------------------------------------------+-----------+-----------+
| c1   | STOPPED |                     |                                              | CONTAINER | 0         |
+------+---------+---------------------+----------------------------------------------+-----------+-----------+
| c2   | RUNNING | 10.8.228.214 (eth0) | fd42:ed5:f839:e64a:216:3eff:fec1:8ab3 (eth0) | CONTAINER | 0         |
+------+---------+---------------------+----------------------------------------------+-----------+-----------+

[tomponline]

Once riscv build is done ill push out to 5.20/stable and latest/stable.

[jardon]

Ya I am on latest/edge for snapd.

I will check the fix when I get home and report back.

Thanks for the quick fix!

[tomponline]

This fix is rolling out now to latest/stable as 5.20-51a2393

Thanks!