Bug fix to correct job log returining and other edge cases (#688)

* Update zos_job_submit to handle some edge cases and return the job log nearly always

Signed-off-by: ddimatos <dimatos@gmail.com>

* Linting correction

Signed-off-by: ddimatos <dimatos@gmail.com>

* Add changelog fragement for pr 683

Signed-off-by: ddimatos <dimatos@gmail.com>

* Clean up comments in code

Signed-off-by: ddimatos <dimatos@gmail.com>

* Update restructured text for modules and changelog fragment

Signed-off-by: ddimatos <dimatos@gmail.com>

* Add support to check for security exception to job submit module

Signed-off-by: ddimatos <dimatos@gmail.com>

* Add additonal logic for jobs that use typerun=scan

Signed-off-by: ddimatos <dimatos@gmail.com>

* Update chnagelog fragment

Signed-off-by: ddimatos <dimatos@gmail.com>

* Correct run on sentence

Signed-off-by: ddimatos <dimatos@gmail.com>

---------

Signed-off-by: ddimatos <dimatos@gmail.com>

Author: ddimatos

changelogs/fragments/683-zos_job_submit-bugs.yml

@@ -0,0 +1,35 @@
+bugfixes:
+- zos_job_submit - Fixes the issue when invalid JCL syntax is submitted that a
+  stack trace would result in the response, issue 623.
+  (https://github.com/ansible-collections/ibm_zos_core/pull/683)
+- zos_job_submit - Fixes the issue when a job is purged by the system that a
+  stack trace would result in the response, issue 681.
+  (https://github.com/ansible-collections/ibm_zos_core/pull/683)
+- zos_job_submit - Fixes the issue where the response did not include the
+  job log when a non-zero return code would occur, issue 655.
+  (https://github.com/ansible-collections/ibm_zos_core/pull/683)
+- zos_job_submit - Fixes the issue when resources (data sets) identified in JCL
+  did not exist such that a stack trace would result in the response, issue 624.
+  (https://github.com/ansible-collections/ibm_zos_core/pull/683)
+- zos_job_submit - Fixes the issue when `wait_time_s` was set to 0 that would
+  result in a `type` error that a stack trace would result in the response,
+  issue 670. (https://github.com/ansible-collections/ibm_zos_core/pull/683)
+- zos_job_submit - Fixes the issue when a job encounters a security exception no
+  job log would would result in the response, issue 684.
+  (https://github.com/ansible-collections/ibm_zos_core/pull/683)
+- zos_job_submit - Fixes the issue when a job is configured for a syntax check
+  using TYPRUN=SCAN that it would wait the full duration set by `wait_time_s`
+  to return a response, issue 685.
+  (https://github.com/ansible-collections/ibm_zos_core/pull/683)
+- zos_job_submit - Fixes the issue when a job is configured for a syntax check
+  using TYPRUN=SCAN that no job log would result in the response, issue 685.
+  (https://github.com/ansible-collections/ibm_zos_core/pull/683)
+trivial:
+- zos_job_submit - Update documentation to for deprecated `wait` option and
+  expand on the `wait_time_s` description, issue 670.
+  (https://github.com/ansible-collections/ibm_zos_core/pull/683)
+- zos_job_submit - Update documentation to describing the significance of '?'
+  for the 'ret_code' properties 'msg_text', 'msg_code' and 'msg', issue 685.
+  (https://github.com/ansible-collections/ibm_zos_core/pull/683)
+- zos_operator - Update restructured text to include the updated examples.
+  (https://github.com/ansible-collections/ibm_zos_core/pull/683)

docs/source/modules/zos_job_submit.rst

@@ -16,9 +16,9 @@ zos_job_submit -- Submit JCL
 
 Synopsis
 --------
-- Submit JCL from DATA_SET , USS, or LOCAL location.
-- Submit a job and optionally monitor for its execution.
-- Optionally wait a designated time until the job finishes.
+- Submit JCL from a data set, USS, or from the controller.
+- Submit a job and optionally monitor for completion.
+- Optionally, wait a designated time until the job finishes.
 - For an uncataloged dataset, specify the volume serial number.
 
 
@@ -32,7 +32,7 @@ Parameters
 src
   The source file or data set containing the JCL to submit.
 
-  It could be physical sequential data set or a partitioned data set qualified by a member or a path. (e.g "USER.TEST","USER.JCL(TEST)")
+  It could be a physical sequential data set, a partitioned data set qualified by a member or a path. (e.g "USER.TEST","USER.JCL(TEST)")
 
   Or a USS file. (e.g "/u/tester/demo/sample.jcl")
 
@@ -58,20 +58,20 @@ location
 
 
 wait
-  Configuring wait used by the `zos_job_submit <./zos_job_submit.html>`_ module has been deprecated and will be removed in ibm.ibm_zos_core collection.
+  Setting this option will yield no change, it is deprecated. There is no no need to set *wait*; setting *wait_times_s* is the correct way to configure the amount of tme to wait for a job to execute.
 
-  Setting this option will yield no change, it is deprecated.
+  Configuring wait used by the `zos_job_submit <./zos_job_submit.html>`_ module has been deprecated and will be removed in ibm.ibm_zos_core collection.
 
-  See option ``wait_time_s``.
+  See option *wait_time_s*.
 
   | **required**: False
   | **type**: bool
 
 
 wait_time_s
-  When *wait* is true, the module will wait for the number of seconds for Job completion.
+  Option *wait_time_s* is the total time that module `zos_job_submit <./zos_job_submit.html>`_ will wait for a submitted job to complete. The time begins when the module is executed on the managed node.
 
-  User can set the wait time manually with this option.
+  *wait_time_s* is measured in seconds and must be a value greater than 0 and less than 86400.
 
   | **required**: False
   | **type**: int
@@ -100,7 +100,7 @@ volume
 
   When configured, the `zos_job_submit <./zos_job_submit.html>`_ will try to catalog the data set for the volume serial. If it is not able to, the module will fail.
 
-  Ignored for USS and LOCAL.
+  Ignored for *location=USS* and *location=LOCAL*.
 
   | **required**: False
   | **type**: str
@@ -548,18 +548,18 @@ jobs
           }
 
     msg
-      Return code resulting from the job submission.
+      Return code resulting from the job submission. Jobs that take longer to assign a value can have a value of '?'.
 
       | **type**: str
       | **sample**: CC 0000
 
     msg_code
-      Return code extracted from the `msg` so that it can be evaluated as a string.
+      Return code extracted from the `msg` so that it can be evaluated as a string. Jobs that take longer to assign a value can have a value of '?'.
 
       | **type**: str
 
     msg_txt
-      Returns additional information related to the job.
+      Returns additional information related to the job. Jobs that take longer to assign a value can have a value of '?'.
 
       | **type**: str
       | **sample**: The job completion code (CC) was not available in the job output, please review the job log."

docs/source/modules/zos_operator.rst

@@ -59,10 +59,12 @@ wait_time_s
 
 
 wait
-  Configuring wait used by the `zos_operator <./zos_operator.html>`_ module has been deprecated and will be removed in ibm.ibm_zos_core collection.
+  Configuring wait used by the `zos_operator <./zos_operator.html>`_ module has been deprecated and will be removed in a future ibm.ibm_zos_core collection.
 
   Setting this option will yield no change, it is deprecated.
 
+  Review option *wait_time_s* to instruct operator commands to wait.
+
   | **required**: False
   | **type**: bool
   | **default**: True
@@ -76,13 +78,13 @@ Examples
 .. code-block:: yaml+jinja
 
 
-   - name: Execute an operator command to show active jobs
+   - name: Execute an operator command to show device status and allocation
      zos_operator:
-       cmd: 'd u,all'
+       cmd: 'd u'
 
-   - name: Execute an operator command to show active jobs with verbose information
+   - name: Execute an operator command to show device status and allocation with verbose information
      zos_operator:
-       cmd: 'd u,all'
+       cmd: 'd u'
        verbose: true
 
    - name: Execute an operator command to purge all job logs (requires escaping)
@@ -91,12 +93,12 @@ Examples
 
    - name: Execute operator command to show jobs, waiting up to 5 seconds for response
      zos_operator:
-       cmd: 'd u,all'
+       cmd: 'd a,all'
        wait_time_s: 5
 
    - name: Execute operator command to show jobs, always waiting 7 seconds for response
      zos_operator:
-       cmd: 'd u,all'
+       cmd: 'd a,all'
        wait_time_s: 7
 
 

plugins/action/zos_job_submit.py

@@ -135,4 +135,20 @@ def run(self, tmp=None, task_vars=None):
                 )
             )
 
+        def delete_dict_entries(entries, dictionary):
+            """ Deletes entries from a dictionary when provided key and dictionary.
+
+                Arguments:
+                    entries    (tuple)  - entries to delete from dictionary
+                    dictionary (dic)    - dictionary to remove entries
+            """
+            for key in entries:
+                if key in dictionary:
+                    del dictionary[key]
+
+        # Currently the direction is undecided if we should continue to use the
+        # community action plugins or transition to SFTP, so this code
+        # can remain should we want to clean up unrelated response values.
+        # entries = ('checksum', 'dest', 'gid', 'group', 'md5sum', 'mode', 'owner', 'size', 'src', 'state', 'uid')
+        # delete_dict_entries(entries, result)
         return result

plugins/module_utils/job.py

@@ -35,10 +35,13 @@ def job_output(job_id=None, owner=None, job_name=None, dd_name=None, duration=0,
     """Get the output from a z/OS job based on various search criteria.
 
     Keyword Arguments:
-        job_id {str} -- The job ID to search for (default: {None})
-        owner {str} -- The owner of the job (default: {None})
-        job_name {str} -- The job name search for (default: {None})
-        dd_name {str} -- The data definition to retrieve (default: {None})
+        job_id (str) -- The job ID to search for (default: {None})
+        owner (str) -- The owner of the job (default: {None})
+        job_name (str) -- The job name search for (default: {None})
+        dd_name (str) -- The data definition to retrieve (default: {None})
+        duration (int) -- The time the submitted job ran for
+        timeout (int) - how long to wait in seconds for a job to complete
+        start_time (int) - time the JCL started its submission
 
     Returns:
         list[dict] -- The output information for a list of jobs matching specified criteria.
@@ -220,7 +223,8 @@ def _get_job_status(job_id="*", owner="*", job_name="*", dd_name=None, duration=
             job["ret_code"] = {}
             job["ret_code"]["msg"] = entry.status + " " + entry.rc
             job["ret_code"]["msg_code"] = entry.rc
-            job["ret_code"]["code"] = ""
+            # Why was this set to an empty string?
+            job["ret_code"]["code"] = None
             if len(entry.rc) > 0:
                 if entry.rc.isdigit():
                     job["ret_code"]["code"] = int(entry.rc)
@@ -312,7 +316,7 @@ def _get_job_status(job_id="*", owner="*", job_name="*", dd_name=None, duration=
                         job["ret_code"]["msg"] = tmptext.strip()
                         job["ret_code"]["msg_code"] = None
                         job["ret_code"]["code"] = None
-            if len(list_of_dds) > 1:
+            if len(list_of_dds) > 0:
                 # The duration should really only be returned for job submit but the code
                 # is used job_output as well, for now we can ignore this point unless
                 # we want to offer a wait_time_s for job output which might be reasonable.