[ML] Audit Messages are logged deleting a missing job

[davidkyle]

The screen shot above shows an error message 'Error deleting job' in this case there had been no attempt to delete the job. The problem is that deleting an unknown job causes the auditor to log a message here before returning a 404.

This issue was spotted running the ml sample scripts downloadable from https://www.elastic.co/guide/en/elastic-stack-overview/6.5/ml-gs-data.html#ml-gs-sampledata which always delete the job before creating it. If the job does not exist then the error message is audited with that job id and when the job is created it now has the error associated with it. It's common practice in test scripts to clean the environment first by deleting jobs.

Reproduce

Delete a job
DELETE _xpack/ml/anomaly_detectors/not-a-job
Then search the .ml-notifications index
GET .ml-notifications/_search { "query": { "match_all": {} } }

And 2 messages have been indexed:

"hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".ml-notifications",
        "_type" : "audit_message",
        "_id" : "E0Fe7mYBhHmym0_kaqEW",
        "_score" : 1.0,
        "_source" : {
          "job_id" : "farequote",
          "message" : "Error deleting job: No known job with id 'not-a-job'",
          "level" : "error",
          "timestamp" : 1541597456916,
          "node_name" : "node-0"
        }
      },
      {
        "_index" : ".ml-notifications",
        "_type" : "audit_message",
        "_id" : "FEFe7mYBhHmym0_kaqHY",
        "_score" : 1.0,
        "_source" : {
          "job_id" : "farequote",
          "message" : "Deleting job by task with id 'JSEDHs3IQ-6lWgFssejABw:20'",
          "level" : "info",
          "timestamp" : 1541597456803,
          "node_name" : "node-0"
        }
      }
    ]
  }

The solution is not to audit messages for jobs that do not exist

[elasticmachine]

Pinging @elastic/ml-core