DLPX-73229 iSCSI initiator name should be unique on each engine

[don-brady]

Problem:

On Ubuntu, the Initiator IQN is generated when the open-iscsi package is installed. However, we want a unique IQN name to be generated on first boot. We also want the name generation to not be dependent on performing any iSCSI initiator setup (e.g. the unique IQN should be ready before discovery phase).

Solution:

Added a new service, iscsi-name-init.service, that runs at boot time to generate a unique IQN. This service runs a new script, unique-name-check, that will generate a unique initiator IQN.

This new service is based on the Fedora, First-time Service Setup model.

Testing

ab-pre-push:
http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-orchestrator-pre-push/5270/
pre_checkin:
http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/blackbox-self-service/91842/consoleFull

Manual testing to confirmed that:

• after first boot a new engine UUID based IQN is generated.
• after subsequent boots the name is no longer changing
• cloning an engine with an existing engine UUID based IQN will generate a new IQN that matches the new engine UUID

Here is expected log output for the new service

delphix@djb-master-uniq-iqn:~$ sudo journalctl -u iscsi-name-init
-- Logs begin at Thu 2021-05-13 19:33:19 UTC, end at Thu 2021-05-13 21:18:59 UTC. --
May 13 19:33:41 localhost systemd[1]: Starting One time IQN configuration for iscsid.service...
May 13 19:33:53 djb-master-uniq-iqn.dcol1 unique-name-check.sh[1836]: Generating unique iSCSI name for engine 564d7744-7deb-3c35-9882-fb2fd4d01a63
May 13 19:33:53 djb-master-uniq-iqn.dcol1 systemd[1]: Started One time IQN configuration for iscsid.service.

Implementation:

Note that the open-iscsi service and iscsid service don't run until iSCSI is configured (i.e. a target is discovered), so we can't use them to run our unique-name-check script.

The initiator IQN is based on the unique engine UUID. This engine UUID is unique after first boot and after an existing engine is cloned.

An example IQN looks like: iqn.2008-07.com.delphix:564d7744-7deb-3c35-9882-fb2fd4d01a63

NOTE: since the /etc/engine-UUID file is written during the startup of delphix-mgmt service and the iscsi-name-init service occurs earlier, we use the bos_mgmt get_system_uuid command to obtain the engine id sooner.

[sebroy]

Instead of introducing a dependent service, could this have been done by adding an ExecStartPre to the open-iscsi.service unit via override?

[don-brady]

I tried that initially but that service doesn't run on first boot.

[sebroy]

This adds an implicit dependency on delphix-virtualization, which seems wrong given that delphix-virtualization depends on delphix-platform. Does this indicate that perhaps the UUID logic should live in delphix-platform rather than delphix-virtualization?

[don-brady]

Great point. The unique-name-check was designed to only run if an engine id is supplied.

Given that the IQN is now tied to the engine UUID, does it make sense to change to a delphix-iscsi-init service and move it into dlpx-app-gate?

[pzakha]

I think if we want to tie the IQN to the engine UUID, then that would make sense to me.

Have you considered putting this logic into cloud-init and have it tied to the instance?

[don-brady]

From what I observed in cloud init, there isn't an instance-id when running on ESX

I could also decouple this from the engine-uuid and directly use something like dmidecode -s system-uuid

[pzakha]

We have added some code in cloud-init to the NoCloud DataSource (used by ESX) that checks for the system-uuid to determine if the "instance" has changed.

[sebroy]

It feels like this bit of code could live in its own script delivered by delphix-platform. Then, the duplicate code in dlpx-app-gate (from where this was copied) could be removed and replaced with an invocation of this script.

[don-brady]

good point. I'll add the script and then file a JIRA to track moving app-gate to switch over.