Merge pull request ceph#1776 from batrick/yaml-preprocessor

teuthology: add pre/post fragment merge hooks

docs/fragment_merging.rst

@@ -0,0 +1,318 @@
+.. _fragment_merging:
+
+================
+Fragment Merging
+================
+
+Once the matrix of YAML fragments is contructed by teuthology, the fragments
+must be merged together and processed. Up until 2022, this merging process was
+static: all of the YAML fragments were joined together in lexicographical order
+with duplicate fragment members *deep merged* together (e.g. the "tasks"
+array). Now, fragments and entire job specifications can be dynamically changed
+or dropped according to Lua scripts embedded in the fragment.
+
+premerge Scripts
+================
+
+The first phase of script execution takes place in the *premerge* step.  Each
+fragment may have its own premerge script which is run before the fragment is
+merged. The script is defined as follows::
+
+    teuthology:
+      premerge: |
+                if yaml.os_type == 'ubuntu' then reject() end
+
+Again, this script will run prior to the YAML fragment merging into the
+complete YAML specification for a job. The script has access to the YAML job
+description (the ``yaml`` variable) generated so far from the fragments merged
+prior to this one (remember: fragments are ordered lexicographically).  In the
+above case, the ``os_type`` is checked such that the fragment is dropped
+(rejected) if the job is configured to run on Ubuntu. Note: this does not
+account for a jobs' default os_type which is not yet known; only the
+``os_type`` specified by the YAML fragments is usable in these scripts.
+
+When run in the premerge step, the ``reject`` function causes the fragment to be
+dropped from the job: none of its YAML will be merged into the job. The
+``accept`` function causes the fragment to be merged. The default action is to
+accept the fragment.
+
+postmerge Scripts
+=================
+
+The second phase of script execution is the *postmerge* step run after all
+fragments have been merged. At this point, the YAML specification for the job
+is all but complete. Scripts can now make final modifications to the YAML or
+reject the job completely causing it to be removed from the list of jobs to be
+scheduled.  An example postmerge script::
+
+    teuthology:
+      postmerge:
+        - if yaml.os_type == "ubuntu" then reject() end
+
+This script is the same but has a different effect: after combining all the
+YAML fragments for a job, if the os_type is "ubuntu" then the entire job is
+dropped (filtered out / rejected). postmerge scripts are also specified as a
+list of strings in the ``teuthology.postmerge`` array which may span multiple
+fragments. During the postmerge step, all of these strings are concatenated and
+then executed as a single script. You may use this to define variables,
+functions, or anything else you need.
+
+Scripts have access to the entire yaml object and may use it to do advanced
+checks. It is also possible to programatically change the YAML definition::
+
+    teuthology:
+      postmerge:
+        - |
+          -- use the lupa "attrgetter" to fetch attrs not items via Lua's indexing
+          local attr = py_attrgetter
+          local tasks = py_list()
+          for i = 1, 3 do
+            local task = py_dict()
+            task.exec = py_dict()
+            task.exec["mon.a"] = py_list()
+            attr(task.exec["mon.a"]).append("echo "..i)
+            attr(tasks).append(task)
+          end
+          deep_merge(yaml.tasks, tasks)
+
+
+This will be as if the YAML fragment contained::
+
+    tasks:
+      - exec:
+          mon.a:
+            - echo 1
+      - exec:
+          mon.a:
+            - echo 2
+      - exec:
+          mon.a:
+            - echo 3
+
+Except the tasks are appended to the end after all fragments have been loaded.
+This is opposed to the normal mode of the tasks appending when the fragment is
+merged (in lexicographic order).
+
+API
+===
+
+Scripts are well sandboxed with access to a small selection of the Lua builtin
+libraries. There is also access to some Python/Lupa specific functions which
+are prefixed with ``py_``. No I/O or other system functions permitted.
+
+The Lua builtins available include::
+
+    assert
+    error
+    ipairs
+    pairs
+    tonumber
+    tostring
+
+Additionally, the Python functions exposed via Lupa include::
+
+    py_attrgetter = python.as_attrgetter
+    py_dict = python.builtins.dict
+    py_list = python.builtins.list
+    py_tuple = python.builtins.tuple
+    py_enumerate = python.enumerate
+    py_iterex = python.iterex
+    py_itemgetter = python.as_itemgetter
+
+These are all prefixed with ``py_``. See the `Lupa documentation
+<https://pypi.org/project/lupa/>`__ for more information.
+
+Finally, teuthology exposes the following functions for scripts:
+
+::
+
+	accept()
+
+The ``accept`` function stops script execution and causes the fragment to be
+merged (premerge script) or the job to be accepted for scheduling (postmerge
+script). The default action of a script is to accept.
+
+::
+
+	reject()
+
+The ``reject`` function stops script execution and causes the fragment to be
+dropped (premerge script) or the job to be rejected for scheduling (postmerge
+script).
+
+
+::
+
+	deep_merge(a, b)
+
+The ``deep_merge`` function comes from the teuthology code base. It's used to
+merge YAML structures. It's provided for convenience to ease a common operation
+on Python (yaml) objects. The function merges ``b`` into ``a``.
+
+
+::
+
+	log
+
+The ``log`` Python class (object) allows Lua to leave debugging in the
+``teuthology-suite`` log.
+
+::
+
+	yaml_load(str)
+
+This function loads the YAML string and returns it as a Python structure (of
+dicts, lists, etc.).
+
+
+Concrete Example
+================
+
+The
+`fs:upgrade:mds_upgrade_sequence <https://github.com/ceph/ceph/tree/edd4e553efd3934292c768b39d9ca1ff8d920ef1/qa/suites/fs/upgrade/mds_upgrade_sequence>`__
+sub-suite tests that the `upgrade sequence for CephFS <https://docs.ceph.com/en/quincy/cephfs/upgrading/>`__
+is followed when the cluster is managed by cephadm. The most interesting set of YAML in this suite is in ``tasks/``::
+
+    %
+    0-from/
+      pacific.yaml
+      v16.2.4.yaml
+    1-volume/
+      0-create.yaml
+      1-ranks/
+        1.yaml
+        2.yaml
+      2-allow_standby_replay/
+        yes.yaml
+        no.yaml
+      3-inline
+        yes.yaml
+        no.yaml
+      4-verify.yaml
+    2-client.yaml
+    3-upgrade-with-workload.yaml
+    4-verify.yaml
+
+Basically: upgrade the cluster from one of two versions of pacific, create a
+volume (fs), possibly turn some knobs in the MDSMap, and verify the upgrade
+completes correctly. This works well and is an excellent example of effective
+matrix construction for testing.
+
+The feature we want to test is a `new upgrade procedure
+<https://tracker.ceph.com/issues/55715>`__ for the MDS. It only requires
+"failing" the file systems which removes all running MDS from the MDSMap and
+prevents any MDS from "joining" the file system (becoming active).  The upgrade
+procedure then upgrades the packages, restarts the MDS, then sets the file
+system to allow MDS to join (become active). Ideally, we could modify the
+matrix this way::
+
+    %
+    fail_fs/
+      yes.yaml
+      no.yaml
+    tasks/
+      %
+      0-from/
+        pacific.yaml
+        v16.2.4.yaml
+      1-volume/
+        0-create.yaml
+        1-ranks/
+          1.yaml
+          2.yaml
+        2-allow_standby_replay/
+          yes.yaml
+          no.yaml
+        3-inline
+          yes.yaml
+          no.yaml
+        4-verify.yaml
+      2-client.yaml
+      3-upgrade-with-workload.yaml
+      4-verify.yaml
+
+So we just change (or don't change) a single config option in ``fail_fs``
+which turns on that upgrade path::
+
+    overrides:
+      ceph:
+        conf:
+          mgr:
+            mgr/orchestrator/fail_fs: true
+
+The complication however is that this new ``fail_fs`` config option is only
+understood by the newest mgr (the ``main`` branch or possibly the latest
+pacific or quincy)... and the mons won't let you set a config unknown to exist.
+So, we must do a staggered upgrade to test this new upgrade path: the mgr must
+be upgraded, a config option set to change how MDS upgrades are performed, and
+then the cluster may continue upgrading.
+
+**Here's the problem**: the mgr only knows how to do a staggered upgrade
+beginning with v16.2.10. So, we can't even upgrade from v16.2.4 to test this
+new upgrade path.
+
+(One might be tempted to remove v16.2.4 as an upgrade path in
+QA but we must continue testing this due to major (breaking) changes in the
+MDSMap across v16.2.4 and v16.2.5. It would not be acceptable to remove it.)
+
+To get around this awkward problem, we can use the new scripting of fragment
+merging to control whether this ``mgr/orchestrator/fail_fs`` config option is
+set. If we are upgrading from v16.2.4, then drop any jobs in the matrix that
+also want to test this new MDS upgrade procedure. So we modify the yaml
+fragments as::
+
+  fail_fs/no.yaml:
+    teuthology:
+      variables:
+        fail_fs: false
+    overrides:
+      ceph:
+        conf:
+          mgr:
+            mgr/orchestrator/fail_fs: false
+
+  fail_fs/yes.yaml:
+    teuthology:
+      variables:
+        fail_fs: true
+    overrides:
+      ceph:
+        conf:
+          mgr:
+            mgr/orchestrator/fail_fs: true
+
+  tasks/0-from/v16.2.4.yaml:
+    teuthology:
+      postmerge:
+        - if yaml.teuthology.variables.fail_fs then reject() end
+    ...
+
+
+We have set a variable (for ease of programming) in a
+``teuthology['variables']`` dictionary which indicates whether the merged YAML
+includes the ``fail_fs`` feature or not. Then, if we're upgrading from v16.2.4
+and that variable is true, drop that set of jobs in the matrix. This
+effectively prevents any testing of this upgrade procedure when the cluster is
+upgraded from v16.2.4.
+
+Note: the final merged QA code also includes a YAML fragment to perform a
+staggered upgrade of the ``ceph-mgr``. This YAML fragment is dropped using a
+premerge script if we're not testing ``fail_fs``; there is no reason to do a
+staggered upgrade if we don't need to. See the code if you'd like to see how
+that works!
+
+
+Why Lua
+=======
+
+Lua is a small, extensible, and easily sandboxed scripting environment. Python
+is difficult to sandbox correctly and its restrictions make it difficult to
+embed in YAML (like indentation for code blocks).
+
+
+Python-Lua
+==========
+
+`Lupa <https://pypi.org/project/lupa/>`__ is the most recent derivative of the
+"lunatic" python project. It allows for trivial cross-talk between Python and
+Lua worlds.

docs/index.rst

@@ -6,6 +6,7 @@ Content Index
 
    README.rst
    intro_testers.rst
+   fragment_merging.rst
    siteconfig.rst
    detailed_test_config.rst
    openstack_backend.rst

scripts/schedule.py

@@ -12,6 +12,7 @@
 
 positional arguments:
   <conf_file>                          Config file to read
+                                       "-" indicates read stdin.
 
 optional arguments:
   -h, --help                           Show this help message and exit

setup.cfg

@@ -43,6 +43,7 @@ install_requires =
     gevent
     httplib2
     humanfriendly
+    lupa
     ndg-httpsclient
     netaddr
     paramiko
@@ -119,6 +120,8 @@ teuthology.openstack =
     openstack-user-data.txt
     openstack.yaml
     setup-openstack.sh
+teuthology.suite =
+    fragment-merge.lua
 teuthology.task.install =
     adjust-ulimits
     daemon-helper