Implementation tracking for 7.0 types deprecation.

[jtibshirani]

Tracks the details of the 'In 7.0' part of this comment: #15613 (comment)

Plan for 7.0

• For requests with a type in the URL or as a leaf field, we will accept both typed + typeless versions of the API. We’ll emit a deprecation warning to tell users they need to move to the typeless endpoints before 8.0. Responses will still contain a _type field, but we will return the dummy name _doc regardless of the underlying type name.
• For APIs whose request/ response structure changes with the deprecation (create index, get mapping, etc.), we’ll have a request parameter include_type_name that should be set to false to omit types in requests + responses. It will default to true in 6.7 with a warning that it needs to be explicitly specified (to either true or false), default to false in 7.0 with a warning to stop specifying it, and finally be removed in 8.0.

More information can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

To-do List

Phase One: Add Typeless APIs. These items are critical for 6.7/ 7.0, and should be in before feature freeze.

• Remove the include_type_name parameter from the bulk, delete, get, index, update, and search APIs. @jtibshirani Remove include_type_name from the relevant APIs. #35192
• Make sure include_type_name is added to APIs that we missed, and has the right default values on both 6.7 and 7.0. @jtibshirani Support include_type_name in the field mapping and index template APIs. #37210 Update the default for include_type_name to false. #37285
• Add the ability to ignore type-related warnings in REST tests. @jtibshirani Ignore warnings related to types deprecation in REST tests. #35395
• Emit deprecation warnings when typed APIs are used. As part of this work, we should deprecate the appropriate methods in the Java HLRC. For some APIs, we may also need to create a duplicate REST test without types. The following APIs should be updated:
  • search @jpountz Deprecate filtering on _type. #29468
  • count, msearch @jtibshirani Deprecate types in count and msearch. #35421
  • explain @jtibshirani Deprecate types in explain requests. #35611
  • search_template, msearch_template @jtibshirani Deprecate types in search and multi search templates. #35669
  • termvectors, mtermvectors @jtibshirani Deprecate types in termvector and mtermvector requests. #36182
  • get, exists, mget @jtibshirani Deprecate types in get, exists, and multi get. #35930
  • delete @jtibshirani Deprecate types in document delete requests. #36087
  • bulk @markharwood Types removal - deprecate types in _bulk apis #36549
  • index (note that the Java HLRC deprecations may be tricky) @mayya-sharipova Deprecate types in index API #36575
  • update @jtibshirani Deprecate types in update requests. #36181
  • get_source, exists_source @cbuescher Add typless endpoints for get_source and exist_source #36426
  • document _create @jtibshirani Deprecate the document create endpoint. #36863
  • reindex @cbuescher Deprecate use of type in reindex request body #36823
  • delete_by_query, update_by_query @mayya-sharipova Deprecate types in update_by_query and delete_by_query #36365
  • indices.validate_query @jtibshirani Deprecate types in validate query requests. #35575
  • indices.create @jtibshirani Deprecate types in create index requests. #37134
  • indices.put_mapping @jtibshirani Deprecate types in the put mapping API. #37280
  • indices.get @cbuescher Support 'include_type_name' in RestGetIndicesAction #37149
  • indices.get_mapping @jtibshirani Support both typed and typeless 'get mapping' requests in the HLRC. #37796
  • indices.get_field_mapping @mayya-sharipova Deprecate types in get field mapping API #37667
  • indices.get_template, indices.put_template @markharwood Types removal - deprecate include_type_name with index templates #37484
  • indices.rollover @mayya-sharipova Deprecate types in rollover index API #38039
• Ensure that if an index’s type is named something other than _doc, then typeless API calls still work. Currently in this situation, using certain typeless APIs will produce an error due to a mismatched type name. @jpountz Make typeless APIs usable with indices whose type name is different from _doc #35790
• Allow for typeless 'lookup' queries (MoreLikeThis, ids query, terms query, GeoShapeQuery) @mayya-sharipova @jtibshirani Deprecate reference to _type in lookup queries #37016
• Follow-up fixes from the above work.
  • Backport the relevant parts of Make typeless APIs usable with indices whose type name is different from _doc #35790 (allow typeless APIs when an index has a custom type) to 6.7. @jpountz Add an include_type_name option to 6.x. (#29453) #37147
  • Typeless index call can fail against an index with a custom type (Typeless index call can fail against an index with a custom type. #36811). @jtibshirani Make sure to use the resolved type in DocumentMapperService#extractMappings. #37451
• Deprecate types in watches (search templates and index actions). @jakelandis deprecate types for watcher #37594

Phase Two: Important Clean-up. These tasks should be in by 6.7/ 7.0, but can go in after feature freeze.

• Update removal_of_types.asciidoc to reflect the new plan. @jtibshirani Update 'removal of types' docs to reflect the new plan for 7.0. #38003
• Switch to typeless index creation in REST tests. @colings86 Removes typed calls from YAML REST tests #37611 Default include_type_name to false in the yml test harness. #38058
• Switch to typeless index creation (remove include_type_name=true) in documentation. @cbuescher Remove include_type_name in asciidoc where possible #37568 Remove more include_type_name and types from docs #37601 Remove remaining occurances of "include_type_name=true" in docs #37646
• Solidify the approach to allowing typeless requests on an index with a custom type (Solidify the approach to allowing typeless requests on an index with a custom type. #37450).

Phase Three: Additional Deprecations. These are good to have by 6.7/ 7.0, but could be pushed into 7.1 if strictly necessary.

• Deprecate references to _type in the search request body.
  • uses of _type as a field name (in match queries, etc.) @mayya-sharipova Deprecate uses of _type as a field name in queries #36503 Avoid duplicate types deprecation messages in search-related APIs. #36802
  • aggregations @jtibshirani Deprecate use of the _type field in aggregations. #37131
  • retrieving fields Loading _type as a field doesn't actually work, and doesn't make much sense because each search hit already contains a _type component.
• Emit a deprecation warning when a search template containing a type is triggered. Note this is different from emitting a warning when a typed template is originally added. Already covered by other search-related PRs.
• Emit a deprecation warning when an index template containing a type is triggered. Note this is different from emitting a warning when a typed template is originally added. We support making typeless calls against an index with a custom type, so it is not really harmful to have a template with a custom type (and in fact several internal templates around monitoring use typed templates).
• Deprecate references to _type in scripts. @jdconrad Deprecate _type from LeafDocLookup #37491 Add types deprecation to script contexts #37554
• Deprecate types in simulate pipeline requests (Deprecate types in simulate pipeline requests. #37731). @gwbrown Deprecate _type in simulate pipeline requests #37949
• Deprecate types in graph explore requests. (Deprecate types in _graph/explore calls. #40466)

Items we’re still following-up on

• For responses that contain the type as a leaf field, should we always return _doc regardless of the underlying type even when the old typed APIs are used?
• Types may be present in saved search requests, including search templates and watches. We should think through the upgrade plan here. Implementation tracking for 7.0 types deprecation. #35190 (comment)
• When an index template is stored, the mappings are nested under the type name. We also need to consider how these will be accessed and upgraded.

[elasticmachine]

Pinging @elastic/es-search-aggs

[jtibshirani]

CC @elastic/es-clients

[Mpdreamz]

One last thing I want to raise which might be super contentious:

Right now the default for include_type_name when not specified is true in 7.0. There are two downsides with this:

• New installations will get deprecation warnings OOTB unless they specify include_type_name=false which they need to remove in 8.0.
• Someone who wants to go through all the type related changes in one major version can not do so.

Can we remove some of the confusion of include_type_name defaulting to true by simply having it
default to false in 7.0 already?

• New installations will return the desired responses by default.
• Someone who wants to go through all the changes in one major version can do so.

Upgrades will have to account for the type removal whether the include_type_name defaults to true or false.

Are we actually doing our users a disservice by holding on to a deprecation period?

[markharwood]

Types may be present in saved search requests, including search templates and watches. We should think through the upgrade plan here.

The proposal is in 7.0 we will introduce deprecation warnings when saved queries with types are executed, but not do anything to proactively detect and upgrade them automatically. Some of the easier type removals would be in the structured metadata held around a saved search whereas other changes may be buried in clauses embedded in the saved search's choice of query DSL.

[jpountz]

@Mpdreamz I'm thinking that this could be an issue for mixed version clusters that have 6.latest and 7.0 nodes as the response format would depend on the version of the node that you are querying?