Fix worlddump log collection

All credit for figuring this out goes to frickler (and that was the hard
bit so thank you!). The worlddump files were not being collected because
they weren't in our log collection list. Add worlddump to this list
so that we collect these files.

One thing that makes this slightly complicated is the worlddump files
are named with a timestamp and we can't have globs in our collection
list. To address this we create a copy of the file with a -latest.txt
suffix. This gives us a deterministic file name for log collection
without using globs.

Note we do not use a symlink here because some jobs gzip their log files
(breaking symlinks) and others do not. This makes it painful to always
have a valid link. Not having a valid link can break log collection.

Hardlinks may be another option but simply making a copy is easier to
manage as you don't have to worry about links preexisting and the
dumpfiles are not that large.

Change-Id: I96ae5f5290546ad25ca434c1106c01354d2d053c

Author: jharbott

.zuul.yaml

@@ -233,6 +233,7 @@
         '{{ devstack_log_dir }}/devstacklog.txt': logs
         '{{ devstack_log_dir }}/devstacklog.txt.summary': logs
         '{{ devstack_log_dir }}/tcpdump.pcap': logs
+        '{{ devstack_log_dir }}/worlddump-latest.txt': logs
         '{{ devstack_full_log}}': logs
         '{{ stage_dir }}/verify_tempest_conf.log': logs
         '{{ stage_dir }}/apache': logs

tools/worlddump.py

@@ -25,6 +25,7 @@
 import fnmatch
 import os
 import os.path
+import shutil
 import subprocess
 import sys
 
@@ -248,6 +249,14 @@ def main():
         compute_consoles()
         guru_meditation_reports()
         var_core()
+    # Singular name for ease of log retrieval
+    copyname = os.path.join(opts.dir, 'worlddump')
+    if opts.name:
+        copyname += '-' + opts.name
+    copyname += '-latest.txt'
+    # We make a full copy to deal with jobs that may or may not
+    # gzip logs breaking symlinks.
+    shutil.copyfile(fname, copyname)
 
 
 if __name__ == '__main__':