Unprivileged LXD containers with large filesets takes hours to start

[divanikus]

Description
LXD driver does not honor LXD idmap values which causes full disk remapping on each start. While it is not a problem on a small fileset, it causes really huge time to start a container with a large fileset.

The problem lies within two values in the container's config, which are
config.volatile.idmap.next
config.volatile.last_state.idmap

If one of these is empty, but the container has an idmap, it causes LXD to run a full remap of all underlying disks. On a large fileset (millions of files) it is a costly operation and might take hours to complete.

To Reproduce
Steps to reproduce the behavior.

1. Create a regular container
2. Add LOTS of files into it
3. Stop the container
4. Try to start it again

Expected behavior
If those values are maintained properly, container starts immediately.

Details

• Affected Component: VMM
• Hypervisor: LXD
• Version: 5.8.4

Suggestion
I think that LXD driver should save the successful config somewhere and reuse it's values to rebuild newer deployments.

[dann1]

Hi @divanikus could you test #3892 ?

[divanikus]

Hello @dann1. Unfortunately I cannot test it right now, because I'm already using my own patch in our production. Regarding the issue, you should really store and restore only config.volatile.last_state.idmap, because config.volatile.idmap.next is calculated and maintaned by lxd daemon by itself.

My patch against the current master is looking like that:

diff --git a/src/vmm_mad/remotes/lib/lxd/container.rb b/src/vmm_mad/remotes/lib/lxd/container.rb
index 526ef4b7b..10117309b 100644
--- a/src/vmm_mad/remotes/lib/lxd/container.rb
+++ b/src/vmm_mad/remotes/lib/lxd/container.rb
@@ -486,6 +486,8 @@ class Container
 
         @lxc = @client.get("#{CONTAINERS}/#{name}")['metadata']
 
+        @one.from_lxc(@lxc)
+
         status
     end
 
diff --git a/src/vmm_mad/remotes/lib/lxd/opennebula_vm.rb b/src/vmm_mad/remotes/lib/lxd/opennebula_vm.rb
index 47df5c64b..d94a32857 100644
--- a/src/vmm_mad/remotes/lib/lxd/opennebula_vm.rb
+++ b/src/vmm_mad/remotes/lib/lxd/opennebula_vm.rb
@@ -65,6 +65,15 @@ class OpenNebulaVM
         sysds_id = @xml['//HISTORY_RECORDS/HISTORY/DS_ID']
         @sysds_path = "#{@lxdrc[:datastore_location]}/#{sysds_id}"
 
+        @prev = if File.exist?("#{@sysds_path}/#{@vm_id}/previous")
+            begin
+                YAML.load(File.read("#{@sysds_path}/#{@vm_id}/previous"))
+            rescue => e
+            end
+        else
+            nil
+        end
+
         return if wild?
 
         # Sets the DISK ID of the root filesystem
@@ -104,6 +113,14 @@ class OpenNebulaVM
         lxc
     end
 
+    def from_lxc(lxc)
+        file = "#{@sysds_path}/#{@vm_id}/previous"
+        begin
+            File.write(file, lxc.to_yaml)
+        rescue => e
+        end
+    end
+
     #---------------------------------------------------------------------------
     # Container Attribute Mapping
     #---------------------------------------------------------------------------
@@ -354,6 +371,10 @@ class OpenNebulaVM
             end
         end
 
+        if !@prev.nil? && @prev.key?('config') && @prev['config'].key?('volatile.last_state.idmap')
+            raw_data['volatile.last_state.idmap'] = @prev['config']['volatile.last_state.idmap']
+        end
+
         hash.merge!(raw_data) unless raw_data.empty?
     end
 
-- 

I'm just saving the whole lxc definition to a previous file on each state change and reusing it on container start to get and set the config.volatile.last_state.idmap.