ubuntu-16.04: unattended updates locking dpkg

[tpetchel]

Test Kitchen fails to converge bento/ubuntu-16.04 version 2.2.7 when multiple CPUs are specified in .kitchen.yml.

The error message is:

dpkg: error: dpkg status database is locked by another process

Repro steps

step 1. Create a cookbook. For example:

$ chef generate cookbook ubuntu-test

step 2. Modify .kitchen.yml to specify 2 CPUs.

---
driver:
  name: vagrant

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-16.04
    driver:
      customize:
        cpus: 2

suites:
  - name: default
    run_list:
      - recipe[ubuntu-test::default]
    attributes:

step 3. Run kitchen converge

Result

https://gist.github.com/tpetchel/0b38bb72f296e330c4d90d5c002a63b7

More info

This was reported on Learn Chef: https://learn.chef.io/manage-a-web-app/ubuntu/run-your-web-application-on-a-node/#comment-2717043449.

After seeing the user's .kitchen.yml, I was able to repo on an empty cookbook. The issue does not repro when you do not specify the customize section. That's not to say that it's strictly caused by running more than 1 CPU, but perhaps the timing with multi CPUs just happens to make it repro more consistently.

[mjuarez]

I'm experiencing the same issue however I've manually set the platform's driver to use 1 cpu.

Relevant output:
https://gist.github.com/mjuarez/ac6bfff8b140ad82399ec93549b5faf1

[cheeseplus]

Thanks for the report @mjuarez - when we this was initially reported I expected the CPU count was probably unrelated and more an issue of timing which this seems to support.

[cblecker]

+1 to @mjuarez. Am seeing the same thing with 16.04, using 2.2.7 of the bento box with 1 CPU (no customize section).

[jrb]

This appears to be caused by a combination of an unattended-upgraded package policy set by the Debian/Ubuntu preseed, and a change in how unattended upgrades are handled.

In past versions of Ubuntu, unattended package upgrades were handled by creating a cron job in /etc/cron.daily responsible for running apt. With 16.04 and the shift to systemd, unattended upgrades are handled by a systemd unit. The new systemd unit has OnCalendar set to 6am and 6pm, along with Persistent set to true. The result is that a full apt-get update && apt-get ugrade will run the first time the box boots, in order to "catch-up" with missed runs.

The idea that packages will be upgraded automatically twice a day in my development VMs is a little scary, so I propose disabling the automatic apt upgrades entirely. #612

[hrak]

Since #612 is merged now, can the affected boxes please be updated so they include this fix?

Btw: i'm not sure that #612 fixes the entire issue. There is also a service and a timer in systemd on Ubuntu 16.04 for updating at boot time that may need to be disabled:

sudo systemctl disable apt-daily.service # disable run when system boot
sudo systemctl disable apt-daily.timer   # disable timer run

Kudo's to this page for that

[rmoriz]

@hrak can you check which package provides the unit files?

UPDATE:

apt-daily.service calls /usr/lib/apt/apt.systemd.daily from the apt package and has the following "kill switch":

# check if the user really wants to do something
AutoAptEnable=1  # default is yes
eval $(apt-config shell AutoAptEnable APT::Periodic::Enable)

if [ $AutoAptEnable -eq 0 ]; then
    exit 0
fi

The variable is not set (=default, enable):

# This file understands the following apt configuration variables:
# Values here are the default.
# Create /etc/apt/apt.conf.d/10periodic file to set your preference.
#
#…
#
#  APT::Periodic::Enable "1";
#  - Enable the update/upgrade script (0=disable)

So I guess we'll need /etc/apt/apt.conf.d/10periodic with APT::Periodic::Enable "0"; in it?

[jrb]

Whoops, I guess #612 only disabled the upgrade portion, not the accompanying update, speeding it up just enough to pass in my tests.

It looks like @rmoriz has the right fix -- seems to do the job locally.