Skip to content

Conversation

@cbuescher
Copy link
Member

Currently runtime fields from search requests don't appear in the output of the
field capabilities API, but some consumer of runtime fields would like to see
runtime section just like they are defined in search requests reflected and
merged into the field capabilities output.
This change adds parsing of a "runtime_mappings" section equivallent to the one
on search requests to the _field_caps endpoint, passes this section down to
the shard level where any runtime fields defined here overwrite the mapping of
the targetet indices.

Closes #68117

Currently runtime fields from search requests don't appear in the output of the
field capabilities API, but some consumer of runtime fields would like to see
runtime section just like they are defined in search requests reflected and
merged into the field capabilities output.
This change adds parsing of a "runtime_mappings" section equivallent to the one
on search requests to the `_field_caps` endpoint, passes this section down to
the shard level where any runtime fields defined here overwrite the mapping of
the targetet indices.

Closes elastic#68117
@cbuescher cbuescher added >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types v8.0.0 v7.12.0 labels Feb 11, 2021
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Feb 11, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@cbuescher
Copy link
Member Author

One caveat of this PR: I found that we do not propagate back exceptions thrown in the shard operation in TransportFieldCapabilitiesIndexAction to the calling nodes in a good way currently. If there are errors in the runtime_mappins section, e.g.

{
  "runtime_mappings": {
    "day_of_week": {
      "type": "some_bad_type",
      "script": {
        "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
      }
    }
  }
}

The resulting MapperParsingException in that is thrown when parsing the runtime fields upon creating the SearchExecutionContext on the shard is dropped somewhere. Fixing this is a bit more involved so I would prefer doing this in a follow up.

Copy link
Contributor

@jimczi jimczi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@javanna javanna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me, error handling looks like wasn't part of the initial design of field_caps and it now becomes more important as scripts may not compile etc. Although field_caps should not be used to validate scripts (and possibly we should not even need to compile them for field_caps), returning an empty response whenever there is an error does not seem like a good idea.

I'd say merge this one and let's discuss and work on the error handling as a follow-up?

request -> request.indexFilter(request.indexFilter() != null ? request.indexFilter().boost(2) : QueryBuilders.matchAllQuery())
);
mutators.add(request -> request.runtimeFields(Collections.singletonMap("other_key", "other_value")));

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make sense to add a test for the toXContent output of the request if we don't test it yet?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, will do

Christoph Büscher and others added 2 commits February 11, 2021 15:06
Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
@cbuescher cbuescher merged commit 3f267ad into elastic:master Feb 11, 2021
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this pull request Feb 11, 2021
Currently runtime fields from search requests don't appear in the output of the
field capabilities API, but some consumer of runtime fields would like to see
runtime section just like they are defined in search requests reflected and
merged into the field capabilities output.
This change adds parsing of a "runtime_mappings" section equivallent to the one
on search requests to the `_field_caps` endpoint, passes this section down to
the shard level where any runtime fields defined here overwrite the mapping of
the targetet indices.

Closes elastic#68117
cbuescher pushed a commit that referenced this pull request Feb 11, 2021
Currently runtime fields from search requests don't appear in the output of the
field capabilities API, but some consumer of runtime fields would like to see
runtime section just like they are defined in search requests reflected and
merged into the field capabilities output.
This change adds parsing of a "runtime_mappings" section equivallent to the one
on search requests to the `_field_caps` endpoint, passes this section down to
the shard level where any runtime fields defined here overwrite the mapping of
the targetet indices.

Closes #68117
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this pull request Mar 1, 2021
Typically we silently ignore nodes below a certain version that don't support a
specific feature when serializing requests between nodes. With runtime fields we
chose to throw errors instead when one of the nodes a search request is sent to
has a version that does not support the runtime fields feature. This change adds
the same behaviour for support of the "runtime_mappings" section introduced in elastic#68904
in the "_field_caps" API.

Relates to elastic#68904
cbuescher pushed a commit that referenced this pull request Mar 1, 2021
Typically we silently ignore nodes below a certain version that don't support a
specific feature when serializing requests between nodes. With runtime fields we
chose to throw errors instead when one of the nodes a search request is sent to
has a version that does not support the runtime fields feature. This change adds
the same behaviour for support of the "runtime_mappings" section introduced in #68904
in the "_field_caps" API.

Relates to #68904
cbuescher pushed a commit that referenced this pull request Mar 1, 2021
Typically we silently ignore nodes below a certain version that don't support a
specific feature when serializing requests between nodes. With runtime fields we
chose to throw errors instead when one of the nodes a search request is sent to
has a version that does not support the runtime fields feature. This change adds
the same behaviour for support of the "runtime_mappings" section introduced in #68904
in the "_field_caps" API.

Relates to #68904
cbuescher pushed a commit that referenced this pull request Mar 2, 2021
Typically we silently ignore nodes below a certain version that don't support a
specific feature when serializing requests between nodes. With runtime fields we
chose to throw errors instead when one of the nodes a search request is sent to
has a version that does not support the runtime fields feature. This change adds
the same behaviour for support of the "runtime_mappings" section introduced in #68904
in the "_field_caps" API.

Relates to #68904
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

>feature :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Meta label for search team v7.12.0 v8.0.0-alpha1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a runtime field section to the Field Capabilities API

5 participants