[Ingest Pipelines] Preserve unknown fields in processors #91146

jloleysens · 2021-02-11T15:20:56Z

Summary

After elastic/elasticsearch#65314 was merged we discovered that when editing a processor unknown fields are not returned from the forms in ingest pipelines editor.

This contribution adds logic to pick out the unknown fields and keep them, even after updating a processor.

Also fix #91135

How to test

Start Elastic and Kibana on basic
Send the following request to ES to create a pipeline called nginx

cURL request

curl --request PUT \
  --url 'http://localhost:9200/_ingest/pipeline/nginx?=' \
  --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
  --header 'Content-Type: application/json' \
  --data '{
  "description": "Pipeline for parsing Nginx ingress controller access logs. Requires the geoip and user_agent plugins.",
  "processors": [
    {
			"set": {
				"field": "test",
				"value": "ok",
				"media_type": "application/json"
			},
      "grok": {
				"description": "test",
        "field": "message",
        "patterns": [
          "(%{NGINX_HOST} )?\"?(?:%{NGINX_ADDRESS_LIST:nginx.ingress_controller.remote_ip_list}|%{NOTSPACE:source.address}) - (-|%{DATA:user.name}) \\[%{HTTPDATE:nginx.ingress_controller.time}\\] \"%{DATA:nginx.ingress_controller.info}\" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} \"(-|%{DATA:http.request.referrer})\" \"(-|%{DATA:user_agent.original})\" %{NUMBER:nginx.ingress_controller.http.request.length:long} %{NUMBER:nginx.ingress_controller.http.request.time:double} \\[%{DATA:nginx.ingress_controller.upstream.name}\\] \\[%{DATA:nginx.ingress_controller.upstream.alternative_name}\\] (%{UPSTREAM_ADDRESS}|-) (%{NUMBER:nginx.ingress_controller.upstream.response.length:long}|-) (%{NUMBER:nginx.ingress_controller.upstream.response.time:double}|-) (%{NUMBER:nginx.ingress_controller.upstream.response.status_code:long}|-) %{GREEDYDATA:nginx.ingress_controller.http.request.id}"
        ],
        "pattern_definitions": {
          "NGINX_HOST": "(?:%{IP:destination.ip}|%{NGINX_NOTSEPARATOR:destination.domain})(:%{NUMBER:destination.port})?",
          "NGINX_NOTSEPARATOR": "[^\t ,:]+",
          "NGINX_ADDRESS_LIST": "(?:%{IP}|%{WORD})(\"?,?\\s*(?:%{IP}|%{WORD}))*",
          "UPSTREAM_ADDRESS": "%{IP:nginx.ingress_controller.upstream.ip}(:%{NUMBER:nginx.ingress_controller.upstream.port})?"
        },
        "ignore_missing": true
      }
    },
    {
      "grok": {
        "field": "nginx.ingress_controller.info",
        "patterns": [
          "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}",
          ""
        ],
        "ignore_missing": true
      }
    },
    {
      "remove": {
        "field": "nginx.ingress_controller.info"
      }
    },
    {
      "split": {
        "field": "nginx.ingress_controller.remote_ip_list",
        "separator": "\"?,?\\s+",
        "ignore_missing": true
      }
    },
    {
      "split": {
        "field": "nginx.ingress_controller.origin",
        "separator": "\"?,?\\s+",
        "ignore_missing": true
      }
    },
    {
      "set": {
        "field": "source.address",
        "if": "ctx.source?.address == null",
        "value": ""
      }
    },
    {
      "script": {
        "if": "ctx.nginx?.access?.remote_ip_list != null && ctx.nginx.ingress_controller.remote_ip_list.length > 0",
        "lang": "painless",
        "source": "boolean isPrivate(def dot, def ip) {\n  try {\n    StringTokenizer tok = new StringTokenizer(ip, dot);\n    int firstByte = Integer.parseInt(tok.nextToken());\n    int secondByte = Integer.parseInt(tok.nextToken());\n    if (firstByte == 10) {\n      return true;\n    }\n    if (firstByte == 192 && secondByte == 168) {\n      return true;\n    }\n    if (firstByte == 172 && secondByte >= 16 && secondByte <= 31) {\n      return true;\n    }\n    if (firstByte == 127) {\n      return true;\n    }\n    return false;\n  }\n  catch (Exception e) {\n    return false;\n  }\n} try {\n  ctx.source.address = null;\n  if (ctx.nginx.ingress_controller.remote_ip_list == null) {\n    return;\n  }\n  def found = false;\n  for (def item : ctx.nginx.ingress_controller.remote_ip_list) {\n    if (!isPrivate(params.dot, item)) {\n      ctx.source.address = item;\n      found = true;\n      break;\n    }\n  }\n  if (!found) {\n    ctx.source.address = ctx.nginx.ingress_controller.remote_ip_list[0];\n  }\n} catch (Exception e) {\n  ctx.source.address = null;\n}",
        "params": {
          "dot": "."
        }
      }
    },
    {
      "remove": {
        "field": "source.address",
        "if": "ctx.source.address == null"
      }
    },
    {
      "grok": {
        "field": "source.address",
        "patterns": [
          "^%{IP:source.ip}$"
        ],
        "ignore_failure": true
      }
    },
    {
      "remove": {
        "field": "message"
      }
    },
    {
      "rename": {
        "field": "@timestamp",
        "target_field": "event.created"
      }
    },
    {
      "date": {
        "field": "nginx.ingress_controller.time",
        "target_field": "@timestamp",
        "formats": [
          "dd/MMM/yyyy:H:m:s Z"
        ],
        "on_failure": [
          {
            "append": {
              "field": "error.message",
              "value": "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    },
    {
      "remove": {
        "field": "nginx.ingress_controller.time"
      }
    },
    {
      "user_agent": {
        "field": "user_agent.original",
        "ignore_missing": true
      }
    },
    {
      "geoip": {
        "field": "source.ip",
        "target_field": "source.geo",
        "ignore_missing": true
      }
    },
    {
      "geoip": {
        "database_file": "GeoLite2-ASN.mmdb",
        "field": "source.ip",
        "target_field": "source.as",
        "properties": [
          "asn",
          "organization_name"
        ],
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "source.as.asn",
        "target_field": "source.as.number",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "source.as.organization_name",
        "target_field": "source.as.organization.name",
        "ignore_missing": true
      }
    },
    {
      "set": {
        "field": "event.kind",
        "value": "event"
      }
    },
    {
      "append": {
        "field": "event.category",
        "value": "web"
      }
    },
    {
      "append": {
        "field": "event.type",
        "value": "info"
      }
    },
    {
      "set": {
        "field": "event.outcome",
        "value": "success",
        "if": "ctx?.http?.response?.status_code != null && ctx.http.response.status_code < 400"
      }
    },
    {
      "set": {
        "field": "event.outcome",
        "value": "failure",
        "if": "ctx?.http?.response?.status_code != null && ctx.http.response.status_code >= 400"
      }
    },
    {
      "lowercase": {
        "field": "http.request.method",
        "ignore_missing": true
      }
    },
    {
      "append": {
        "field": "related.ip",
        "value": "{{source.ip}}",
        "if": "ctx?.source?.ip != null"
      }
    },
    {
      "append": {
        "field": "related.ip",
        "value": "{{destination.ip}}",
        "if": "ctx?.destination?.ip != null"
      }
    },
    {
      "append": {
        "field": "related.user",
        "value": "{{user.name}}",
        "if": "ctx?.user?.name != null"
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "error.message",
        "value": "{{ _ingest.on_failure_message }}"
      }
    }
  ]
}
'

Open the ingest pipelines UI in the management section and manage the nginx pipeline
Edit the first processor (which should be a set processor). Change the field value to anything, e.g., "foobar"
Save the processor
Click on the "Show request" link at the bottom of the UI, the first processor should still have the "media_type" field
Save the pipeline
Manage it again and see that the media_type value is still present on the set processor

Notes

These changes ensure that unknown values in processor options are preserved. We only check for unknown entries at the top-level of the options object; the assumption is that options configuration with nested values do not exist.

That is:

{
  processors: [
    { processor_1: { checked: 'test' } },
    { processor_2: { checked: 'test', checked_2: { not_checked: 'test' } } },
  ]
}

Release note

Fixes a small form validation bug that required an input for the set processor value field and gsub processor replacement field. Both fields now accept an empty value to either set a value to an empty string in the case of the set processor and remove any matched text in the case of the gsub processor.

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2021-02-11T15:20:59Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

alisonelizabeth

Changes LGTM. Verified locally. Thanks for the quick turnaround on this!

alisonelizabeth · 2021-02-12T01:06:57Z

@elasticmachine merge upstream

…ds-in-processors

jloleysens · 2021-02-15T08:47:56Z

@elasticmachine merge upstream

…ds-in-processors

jloleysens · 2021-02-15T11:08:32Z

@alisonelizabeth , I've realised that this implementation is based on the incorrect premise that fields without values willl still be included in the processor form output - by default the form lib will strip them. The misunderstanding snuck in for me with the undefinedIfValue serializer that exists on many forms outputting field_name: undefined.

I will update this implementation to be more robust.

…"value" for set and gsub

jloleysens · 2021-02-15T13:07:13Z

...pplication/components/pipeline_editor/components/processor_form/processor_form.container.tsx

+   * This is important for allowing configuration of empty text fields in certain processors that
+   * remove values from their inputs.
+   */
+  stripEmptyFields: false,


This is quite an important change for this fix, it also means we need to add the appropriate serializer for all fields. I tested all forms locally and the LGTM. But it would be good to check them for any unexpected resulting "foo": "" entries.

alisonelizabeth

@jloleysens I went through and retested all the processors. I found two potential issues -

I think the serializer needs to be fixed for the regex_file option in the user agent processor. If I do not provide a value for the field, the request looks like:

    {
      "user_agent": {
        "field": "sdf",
        "regex_file": ""
      }
    }

I also noticed for the foreach processor, I am able to save the form without specifying a processor and the request retains the {} as the value, which results in an error from ES.

     {
      "foreach": {
        "field": "asdf",
        "processor": {}
      }
    }

alisonelizabeth · 2021-02-16T13:55:19Z

.../public/application/components/pipeline_editor/components/processor_form/processors/gsub.tsx

@@ -50,15 +51,6 @@ const fieldsConfig: FieldsConfig = {
      'xpack.ingestPipelines.pipelineEditor.gsubForm.replacementFieldHelpText',
      { defaultMessage: 'Replacement text for matches.' }
    ),
-    validations: [


I wonder if we should add some help text indicating that leaving the input blank will default to an empty string value.

alisonelizabeth · 2021-02-16T14:00:59Z

...ublic/application/components/pipeline_editor/components/processor_form/processors/script.tsx

@@ -81,7 +81,7 @@ const fieldsConfig: FieldsConfig = {
  lang: {
    type: FIELD_TYPES.TEXT,
    deserializer: String,
-    serializer: from.undefinedIfValue('painless'),
+    serializer: (v: unknown) => (v === 'painless' || v === '' ? undefined : v),


[nit] I know this wasn't introduced in this PR - but I noticed that if I set the lang to "painless" via the ES API, it does retain the value. I think it could be potentially confusing to a user if they add painless to the text field in the UI and then it is stripped from the request.

alisonelizabeth · 2021-02-16T14:01:36Z

...s/public/application/components/pipeline_editor/components/processor_form/processors/set.tsx

@@ -35,15 +28,6 @@ const fieldsConfig: FieldsConfig = {
    helpText: i18n.translate('xpack.ingestPipelines.pipelineEditor.setForm.valueFieldHelpText', {
      defaultMessage: 'Value for the field.',
    }),
-    validations: [


Same comment here about possibly adding some help text

jloleysens · 2021-02-16T16:53:34Z

@elasticmachine merge upstream

…ds-in-processors

jloleysens · 2021-02-16T17:57:41Z

Thanks for the review @alisonelizabeth ! I think I have addressed all of your feedback, would you mind taking another look?

I also noticed for the foreach processor, I am able to save the form without specifying a processor and the request retains the {} as the value, which results in an error from ES.

I'm going to hold off on this change for now since it is pre-exsting behaviour, I think the error response from ES is also very helpful in guiding users. Let me know what you think!

alisonelizabeth

Latest code LGTM. Did not test again. I'm fine with holding off on fixing the foreach behavior.

jloleysens · 2021-02-17T08:28:49Z

@elasticmachine merge upstream

…ds-in-processors

* keep known and unknown options in processor config * added test for preserving unknown fields * refactor form for NOT stripping empty field values, also allow empty "value" for set and gsub * remove unused i18n * fix user agent form serialization, update field help text * remove out of date translation Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-02-17T14:08:32Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`ingestPipelines`	722.7KB	723.5KB	+788.0B

History

💚 Build #107194 succeeded 2bb54ef
💔 Build #106936 failed 7e11f44
💚 Build #106916 succeeded 41b289c
💚 Build #106586 succeeded eb062f2
💔 Build #106574 failed 431dbc5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…1630) * keep known and unknown options in processor config * added test for preserving unknown fields * refactor form for NOT stripping empty field values, also allow empty "value" for set and gsub * remove unused i18n * fix user agent form serialization, update field help text * remove out of date translation Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

jloleysens added 2 commits February 11, 2021 15:23

keep known and unknown options in processor config

f660b16

added test for preserving unknown fields

7ae7390

jloleysens added v8.0.0 Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more release_note:skip Skip the PR/issue when compiling release notes Feature:Ingest Node Pipelines Ingest node pipelines management v7.12.0 labels Feb 11, 2021

jloleysens requested a review from alisonelizabeth February 11, 2021 15:20

jloleysens requested a review from a team as a code owner February 11, 2021 15:20

jloleysens changed the title ~~[Ingest Pipeliens] Preserve unknown fields in processors~~ [Ingest Pipelines] Preserve unknown fields in processors Feb 11, 2021

alisonelizabeth approved these changes Feb 12, 2021

View reviewed changes

Merge branch 'master' into ingest-pipelines/fix-preserve-unknown-fiel…

c6a1392

…ds-in-processors

Merge branch 'master' into ingest-pipelines/fix-preserve-unknown-fiel…

df12eb7

…ds-in-processors

jloleysens added 2 commits February 15, 2021 13:37

refactor form for NOT stripping empty field values, also allow empty …

431dbc5

…"value" for set and gsub

remove unused i18n

eb062f2

jloleysens requested a review from alisonelizabeth February 15, 2021 13:05

jloleysens commented Feb 15, 2021

View reviewed changes

alisonelizabeth reviewed Feb 16, 2021

View reviewed changes

kibanamachine and others added 2 commits February 16, 2021 11:53

Merge branch 'master' into ingest-pipelines/fix-preserve-unknown-fiel…

41b289c

…ds-in-processors

fix user agent form serialization, update field help text

7e11f44

alisonelizabeth approved these changes Feb 16, 2021

View reviewed changes

remove out of date translation

a810b19

Merge branch 'master' into ingest-pipelines/fix-preserve-unknown-fiel…

2bb54ef

…ds-in-processors

jloleysens merged commit edc11d9 into elastic:master Feb 17, 2021

jloleysens added release_note:fix and removed release_note:skip Skip the PR/issue when compiling release notes labels Feb 17, 2021

jloleysens deleted the ingest-pipelines/fix-preserve-unknown-fields-in-processors branch February 17, 2021 11:59

jloleysens mentioned this pull request Feb 17, 2021

[7.x] [Ingest Pipelines] Preserve unknown fields in processors (#91146) #91630

Merged

This was referenced Jul 16, 2021

[Ingest pipelines] add community id processor #103863

Merged

[Ingest Pipelines] Allow to update an existing processor and change its type #105765

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Ingest Pipelines] Preserve unknown fields in processors #91146

[Ingest Pipelines] Preserve unknown fields in processors #91146

Uh oh!

jloleysens commented Feb 11, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Feb 11, 2021

Uh oh!

alisonelizabeth left a comment

Uh oh!

alisonelizabeth commented Feb 12, 2021

Uh oh!

jloleysens commented Feb 15, 2021

Uh oh!

jloleysens commented Feb 15, 2021 •

edited

Loading

Uh oh!

jloleysens Feb 15, 2021

Uh oh!

alisonelizabeth left a comment

Uh oh!

alisonelizabeth Feb 16, 2021

Uh oh!

alisonelizabeth Feb 16, 2021

Uh oh!

alisonelizabeth Feb 16, 2021

Uh oh!

jloleysens commented Feb 16, 2021

Uh oh!

jloleysens commented Feb 16, 2021

Uh oh!

alisonelizabeth left a comment

Uh oh!

jloleysens commented Feb 17, 2021

Uh oh!

kibanamachine commented Feb 17, 2021

Uh oh!

Uh oh!

[Ingest Pipelines] Preserve unknown fields in processors #91146

[Ingest Pipelines] Preserve unknown fields in processors #91146

Uh oh!

Conversation

jloleysens commented Feb 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test

Notes

Release note

Checklist

Uh oh!

elasticmachine commented Feb 11, 2021

Uh oh!

alisonelizabeth left a comment

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth commented Feb 12, 2021

Uh oh!

jloleysens commented Feb 15, 2021

Uh oh!

jloleysens commented Feb 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jloleysens Feb 15, 2021

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth left a comment

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Feb 16, 2021

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Feb 16, 2021

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Feb 16, 2021

Choose a reason for hiding this comment

Uh oh!

jloleysens commented Feb 16, 2021

Uh oh!

jloleysens commented Feb 16, 2021

Uh oh!

alisonelizabeth left a comment

Choose a reason for hiding this comment

Uh oh!

jloleysens commented Feb 17, 2021

Uh oh!

kibanamachine commented Feb 17, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

History

Uh oh!

Uh oh!

jloleysens commented Feb 11, 2021 •

edited

Loading

jloleysens commented Feb 15, 2021 •

edited

Loading