[CI] Failure in "yaml=painless/50_update_scripts/Test transform scripts are updated on execution"

[cbuescher]

This happened on CI master with https://scans.gradle.com/s/x4mlwdvr5qyb4/tests (https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/7/console) as well as on a recent PR run of mine (https://scans.gradle.com/s/j5adddfjkjby4/tests). Adding the details from the CI run from 2019-07-03 here:

Doesn't reproduce locally so far:

./gradlew :x-pack:qa:smoke-test-watcher:integTestRunner --tests "org.elasticsearch.smoketest.WatcherRestIT.test {yaml=painless/50_update_scripts/Test transform scripts are updated on execution}" \
  -Dtests.seed=BD820A2993117365 \
  -Dtests.security.manager=true \
  -Dtests.locale=nb-SJ \
  -Dtests.timezone=America/Argentina/Cordoba \
  -Dcompiler.java=12 \
  -Druntime.java=11

java.lang.AssertionError
: 
Failure at [painless/50_update_scripts:57]: field [watch_record.result.transform.payload.email] is null
Open stacktrace
Caused by: 
java.lang.AssertionError
: 
field [watch_record.result.transform.payload.email] is null
Open stacktrace
[2019-07-03T05:30:29,452][INFO ][o.e.s.WatcherRestIT      ] [test] [yaml=painless/50_update_scripts/Test transform scripts are updated on execution] before test
[2019-07-03T05:31:26,200][INFO ][o.e.s.WatcherRestIT      ] [test] Stash dump on test failure [{
  "stash" : {
    "body" : {
      "_id" : "my_watch_1989c09f-09de-443e-86dd-f1b1750cd0bc-2019-07-03T08:30:59.894613Z",
      "watch_record" : {
        "watch_id" : "my_watch",
        "node" : "ZuKEXWpqSoaiL9JB6GlUGg",
        "state" : "not_executed_already_queued",
        "trigger_event" : {
          "type" : "manual",
          "triggered_time" : "2019-07-03T08:30:59.894Z",
          "manual" : {
            "schedule" : {
              "scheduled_time" : "2019-07-03T08:30:59.894Z"
            }
          }
        },
        "messages" : [
          "Watch is already queued in thread pool"
        ]
      }
    }
  }
}]
[2019-07-03T05:31:26,363][INFO ][o.e.s.WatcherRestIT      ] [test] [yaml=painless/50_update_scripts/Test transform scripts are updated on execution] after test

[elasticmachine]

Pinging @elastic/es-core-features

[mayya-sharipova]

More recent failures: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/49/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/56/console

doesn't reproduce for me either

[ebadyano]

Another failure on master: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/367/consoleFull

[markharwood]

Had this on a PR build - didn't reproduce:

./gradlew :x-pack:qa:smoke-test-watcher:integTestRunner --tests "org.elasticsearch.smoketest.WatcherRestIT.test {yaml=painless/50_update_scripts/Test transform scripts are updated on execution}" -Dtests.seed=628EF7B4397841DD -Dtests.security.manager=true -Dtests.locale=vun -Dtests.timezone=Pacific/Pitcairn -Dcompiler.java=12 -Druntime.java=11

Stash dump:

   1> [2019-07-22T09:04:33,881][INFO ][o.e.s.WatcherRestIT      ] [test] Stash dump on test failure [{
   1>   "stash" : {
   1>     "body" : {
   1>       "_id" : "my_watch_776128ec-0a1d-42ab-8070-cce016cc46f7-2019-07-22T17:04:07.813203Z",
   1>       "watch_record" : {
   1>         "watch_id" : "my_watch",
   1>         "node" : "56HPI9UNQnqsvDDkcFuz_w",
   1>         "state" : "not_executed_already_queued",
   1>         "trigger_event" : {
   1>           "type" : "manual",
   1>           "triggered_time" : "2019-07-22T17:04:07.813Z",
   1>           "manual" : {
   1>             "schedule" : {
   1>               "scheduled_time" : "2019-07-22T17:04:07.813Z"
   1>             }
   1>           }
   1>         },
   1>         "messages" : [
   1>           "Watch is already queued in thread pool"
   1>         ]
   1>       }
   1>     }
   1>   }
   1> }]
   1> [2019-07-22T09:04:34,082][INFO ][o.e.s.WatcherRestIT      ] [test] [yaml=painless/50_update_scripts/Test transform scripts are updated on execution] after test

[jkakavas]

This have been failing quite often: https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-90d,mode:quick,to:now))&_a=(columns:!(_source),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'%22Test%20transform%20scripts%20are%20updated%20on%20execution%22'),sort:!(time,desc)), will mute on master

[mark-vieira]

The test {yaml=painless/50_update_scripts/Test condition scripts are updated on execution} test has been failing often as well.

https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now%2Fw,mode:quick,to:now%2Fw))&_a=(columns:!(_source),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:kuery,query:'test:%20%22test%20%7Byaml%3Dpainless%2F50_update_scripts%2FTest%20condition%20scripts%20are%20updated%20on%20execution%7D%22'),sort:!(time,desc))