Systemd fails to restart DHCP & DNS services when creating Cobbler resources

[thijsschnitger]

When I use Cobbler to manage DHCP and DNS, even with as few as 7 systems systemd fails to restart these services repeatedly and as a result does not keep them running:

systemd: start request repeated too quickly for dhcpd.service
systemd: Failed to start DHCPv4 Server Daemon.
systemd: Unit dhcpd.service entered failed state.
systemd: dhcpd.service failed.
(...)
systemd: start request repeated too quickly for named-setup-rndc.service
systemd: Failed to start Generate rndc key for BIND (DNS).
systemd: Unit named-setup-rndc.service entered failed state.
systemd: named-setup-rndc.service failed.
systemd: start request repeated too quickly for named.service
systemd: Failed to start Berkeley Internet Name Domain (DNS).
systemd: Unit named.service entered failed state.
systemd: named.service failed.

Terraform apply fails with the error:

Cobbler System: Error syncing system: error: "<class 'cobbler.cexceptions.CX'>:'cobbler trigger failed: cobbler.modules.sync_post_restart_services'" code: 1

Terraform Version

Terraform v0.6.15

Affected Resource(s)

cobbler_system

Terraform Configuration Files

provider "cobbler" {
  url      = "http://1.2.3.4/cobbler_api"
  username = "cobbler"
  password = "cobbler"
}
resource "cobbler_system" "server" {
  count           = "9"
  profile         = "centos7-x86_64"
  name            = "server-${count.index}"
  hostname        = "server-${count.index}.mydomain.com"
  gateway         = "1.2.3.1"
  kickstart       = "/var/lib/cobbler/kickstarts/sample_end.ks"
  name_servers    = ["8.8.8.8", "1.2.3.4"]
  netboot_enabled = "1"
  interface {
    name        = "eth0"
    mac_address = "11:22:33:44:55:0${count.index}"
    ip_address  = "1.2.4.${count.index}"
    netmask     = "255.255.255.0"
    dns_name     = "server-${count.index}.mydomain.com"
    static      = true
  }
}

Debug Output

Panic Output

Expected Behavior

Resources created successfully

Actual Behavior

Error applying plan:

2 error(s) occurred:

• cobbler_system.server.2: Cobbler System: Error syncing system: error: "<class 'cobbler.cexceptions.CX'>:'cobbler trigger failed: cobbler.modules.sync_post_restart_services'" code: 1
• cobbler_system.server.1: Cobbler System: Error syncing system: error: "<class 'cobbler.cexceptions.CX'>:'cobbler trigger failed: cobbler.modules.sync_post_restart_services'" code: 1

Steps to Reproduce

1. terraform apply

Important Factoids

References

• Cobbler Provider #5969

[jtopjian]

@thijsschnitger Thanks for the report!

It looks like we've got a sticky situation with three different culprits:

1. systemd is regulating the number of restarts
2. Terraform doesn't have a clean way of issuing a single, final post-apply step.
3. Cobbler doesn't account for a large number of sync requests.

I can think of a few different solutions:

1. We implement the channel method from your original code.
2. Modify the provider to optionally disable a sync upon each system save and push that onto a provisioning step or a meta-resource ilke cobbler_system_sync.
3. Push the resolution onto systemd since it seems to be artificially preventing this. sysv handles it just fine.

They all feel a bit hacky in their own way, accounting for something another piece should be handling. The first option might be the best way to go.

@phinze Thoughts?

This issue has been automatically migrated to hashicorp/terraform-provider-cobbler#27 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to hashicorp/terraform-provider-cobbler#27.

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.