NPE when creating a set_security_user processor

[Mpdreamz]

Elasticsearch version: 7.7.0-SNAPSHOT
Possibly relates to: #52032

Our integration tests started failing when setting up the set_security_user ingest processor with the following NPE.

We use basic auth during these API calls.

PUT: /_ingest/pipeline/pipeline-1?pretty=true&error_trace=true

{
	"description": "My test pipeline",
	"processors": [ {
		"set_security_user": {
			"field": "name"
		}
	}]
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "java.lang.NullPointerException: security context must be provided",
        "processor_type" : "set_security_user",
        "stack_trace" : "ElasticsearchException[java.lang.NullPointerException: security context must be provided]; nested: NullPointerException[security context must be provided];\n\tat org.elasticsearch.ExceptionsHelper.convertToElastic(ExceptionsHelper.java:66)\n\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:322)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:441)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:398)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:336)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:74)\n\tat org.elasticsearch.ingest.IngestService.validatePipeline(IngestService.java:320)\n\tat org.elasticsearch.ingest.IngestService.putPipeline(IngestService.java:226)\n\tat org.elasticsearch.action.ingest.PutPipelineTransportAction.lambda$masterOperation$0(PutPipelineTransportAction.java:83)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1116)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1194)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1174)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:754)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:830)\nCaused by: java.lang.NullPointerException: security context must be provided\n\tat java.base/java.util.Objects.requireNonNull(Objects.java:247)\n\tat org.elasticsearch.xpack.security.ingest.SetSecurityUserProcessor.<init>(SetSecurityUserProcessor.java:44)\n\tat org.elasticsearch.xpack.security.ingest.SetSecurityUserProcessor$Factory.create(SetSecurityUserProcessor.java:183)\n\tat org.elasticsearch.xpack.security.ingest.SetSecurityUserProcessor$Factory.create(SetSecurityUserProcessor.java:161)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:428)\n\t... 27 more\n"
      }
    ],
    "type" : "exception",
    "reason" : "java.lang.NullPointerException: security context must be provided",
    "processor_type" : "set_security_user",
    "caused_by" : {
      "type" : "null_pointer_exception",
      "reason" : "security context must be provided",
      "stack_trace" : "java.lang.NullPointerException: security context must be provided\n\tat java.base/java.util.Objects.requireNonNull(Objects.java:247)\n\tat org.elasticsearch.xpack.security.ingest.SetSecurityUserProcessor.<init>(SetSecurityUserProcessor.java:44)\n\tat org.elasticsearch.xpack.security.ingest.SetSecurityUserProcessor$Factory.create(SetSecurityUserProcessor.java:183)\n\tat org.elasticsearch.xpack.security.ingest.SetSecurityUserProcessor$Factory.create(SetSecurityUserProcessor.java:161)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:428)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:398)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:336)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:74)\n\tat org.elasticsearch.ingest.IngestService.validatePipeline(IngestService.java:320)\n\tat org.elasticsearch.ingest.IngestService.putPipeline(IngestService.java:226)\n\tat org.elasticsearch.action.ingest.PutPipelineTransportAction.lambda$masterOperation$0(PutPipelineTransportAction.java:83)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1116)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1194)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1174)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:754)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:830)\n"
    },
    "stack_trace" : "ElasticsearchException[java.lang.NullPointerException: security context must be provided]; nested: NullPointerException[security context must be provided];\n\tat org.elasticsearch.ExceptionsHelper.convertToElastic(ExceptionsHelper.java:66)\n\tat org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:322)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:441)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:398)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:336)\n\tat org.elasticsearch.ingest.Pipeline.create(Pipeline.java:74)\n\tat org.elasticsearch.ingest.IngestService.validatePipeline(IngestService.java:320)\n\tat org.elasticsearch.ingest.IngestService.putPipeline(IngestService.java:226)\n\tat org.elasticsearch.action.ingest.PutPipelineTransportAction.lambda$masterOperation$0(PutPipelineTransportAction.java:83)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1116)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1194)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1174)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:754)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:830)\nCaused by: java.lang.NullPointerException: security context must be provided\n\tat java.base/java.util.Objects.requireNonNull(Objects.java:247)\n\tat org.elasticsearch.xpack.security.ingest.SetSecurityUserProcessor.<init>(SetSecurityUserProcessor.java:44)\n\tat org.elasticsearch.xpack.security.ingest.SetSecurityUserProcessor$Factory.create(SetSecurityUserProcessor.java:183)\n\tat org.elasticsearch.xpack.security.ingest.SetSecurityUserProcessor$Factory.create(SetSecurityUserProcessor.java:161)\n\tat org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:428)\n\t... 27 more\n"
  },
  "status" : 500
}

[elasticmachine]

Pinging @elastic/es-security (:Security/Authentication)

[tvernum]

@Mpdreamz I can only reproduce this if security is disabled on the cluster.
Is it possible that this is the case for your cluster?

This is a regression, and I'll fix it.
The original behaviour was we would allow you define a processor which would then be guaranteed to fail if it was used. I can revert to that behaviour (at least for 7.x)

[russcam]

I can only reproduce this if security is disabled on the cluster.
Is it possible that this is the case for your cluster?

@tvernum it looks like this is possible in the integration test, as it runs against a cluster that does not specify security enabled. I'll update the test now

[russcam]

Running the integration test in question against a cluster with security enabled does not exhibit NPE.