* build RL9.3 container images with systemd

* remove old workaround message

There was a race conditions between slurmctld starting up and slurmd.
This adds a few retries to make it more robust.

The ReturnToService parameter is not rendered. A newline between that and the jinja conditional above fixes it

Fix ReturnToService slurm.conf templating

* fix duplicated nodename

* Nodes may appear in more than one group

---------

Co-authored-by: jianguo-ukaea <jianguo.rao@ukaea.uk>

* output slurm config as facts

* address linter errors

* fix facts parsing 'SlurmctldParameters     = enable_configlessReturnToService=2'

* remove out of date test7

* skip broken mariadb on RL8

* Fix slurm.conf jinja templating

trim_blocks on by default in Ansible so extra newline required at end of template tag

* Update slurm.conf.j2

* Fix openhpc_configless when set to false

Need to pass this through the bool filter so that string values are
interpreted as booleans.

* Remove conf-server command line option if configless mode is disabled

* nodegroups using nodesets - doesn't handle empty nodegroups

* cope with empty nodegroups/partitions

* make gres work again

* make node/partition parameters more greppable

* use features to simplify nodeset configuration

* add nodegroup.features

* add validation

* document nodegroup.features to README

* add better examples in README

* tidy up README

* fix validate task path

* fix lint error

* default partitions to nodegroups to make CI easier

* update molecule tests for openhpc_nodegroups

* remove checks from runtime now validation defined

* fix NodeName= lines missing newlines between them when multiple hostlists within a node group

* remove tests for extra_nodes

* allow missing inventory groups (as per docs) when validating nodegroups

* only run validation once

* remove test14 from CI - extra_nodes feature removed

* update complex test for new group/partition variables

* rename  openhpc_partitions.groups -> openhpc_partitions.nodegroups  for clarity

* output NodeName hostlists on single line to improve large BM scheduler perf

This reverts commit 0dec9d8.

* add support for upgrading database

* fix db upgrade check for initial deploy

* add mysql tool to support checking db status

* make slurm db service consistent

* make backup optional

* fix ansible-lint whinging

* fix upgrade code for mariadb in CI

* fix mariadb connections

* fix upgrade logic

* Fix readme typo

Co-authored-by: Matt Crees <matthew.crees1@gmail.com>

* fix upgrade logic

* explain upgrade logic

---------

Co-authored-by: Matt Crees <matthew.crees1@gmail.com>

* re-add "Separate node and partition configuration (#183)"

This reverts commit ed717da.

* readme tweak for clarity

* re-add "Separate node and partition configuration (#183)"

This reverts commit ed717da.

* readme tweak for clarity

* update podman molecule driver tmpfs definition

* nodegroups using nodesets - doesn't handle empty nodegroups

* cope with empty nodegroups/partitions

* make gres work again

* template slurm.conf parameters from combined variables

* simplify omit description

* remove dead code

* fixup SlurmctldParameters for configless mode

* fix slurmctld when not defined

* Make omit clearer

Co-authored-by: Matt Anson <matta@stackhpc.com>

---------

Co-authored-by: Matt Anson <matta@stackhpc.com>

* Relax gres configuration validation to allow multiple list items

* Add all allowable properties on nodegroup

* Allow additional properties in nodegroup schema validation