Skip to content

Commit 6672f53

Browse files
jtibshiranijtibshirani
authored
Add support for field aliases to 6.x. (#32184)
* Add basic support for field aliases in index mappings. (#31287) * Allow for aliases when fetching stored fields. (#31411) * Add tests around accessing field aliases in scripts. (#31417) * Return both concrete fields and aliases in DocumentFieldMappers#getMapper. (#31671) * Add documentation around field aliases. (#31538) * Add validation for field alias mappings. (#31518) * Make sure that field-level security is enforced when using field aliases. (#31807) * Add more comprehensive tests for field aliases in queries + aggregations. (#31565) * Remove the deprecated method DocumentFieldMappers#getFieldMapper. (#32148) * Ensure that field aliases cannot be used in multi-fields. (#32219) * Make sure that field aliases count towards the total fields limit. (#32222) * Fix a test bug around nested aggregations and field aliases. (#32287) * Make sure the _uid field is correctly loaded in scripts. * Fix the failing test case FieldLevelSecurityTests#testParentChild_parentField. * Enforce that field aliases can only be specified on indexes with a single type.
1 parent 0e96e97 commit 6672f53

File tree

128 files changed

+4425
-1147
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

128 files changed

+4425
-1147
lines changed

‎docs/reference/indices/clearcache.asciidoc

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ explicitly by setting `query`, `fielddata` or `request`.
1616

1717
All caches relating to a specific field(s) can also be cleared by
1818
specifying `fields` parameter with a comma delimited list of the
19-
relevant fields.
19+
relevant fields. Note that the provided names must refer to concrete
20+
fields -- objects and field aliases are not supported.
2021

2122
[float]
2223
=== Multi Index

‎docs/reference/mapping.asciidoc

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,8 +120,10 @@ fields to an existing index with the <<indices-put-mapping,PUT mapping API>>.
120120

121121
Other than where documented, *existing field mappings cannot be
122122
updated*. Changing the mapping would mean invalidating already indexed
123-
documents. Instead, you should create a new index with the correct mappings
124-
and <<docs-reindex,reindex>> your data into that index.
123+
documents. Instead, you should create a new index with the correct mappings
124+
and <<docs-reindex,reindex>> your data into that index. If you only wish
125+
to rename a field and not change its mappings, it may make sense to introduce
126+
an <<alias, `alias`>> field.
125127

126128
[float]
127129
== Example mapping

‎docs/reference/mapping/types.asciidoc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,8 @@ string:: <<text,`text`>> and <<keyword,`keyword`>>
4040

4141
<<parent-join>>:: Defines parent/child relation for documents within the same index
4242

43+
<<alias>>:: Defines an alias to an existing field.
44+
4345
[float]
4446
=== Multi-fields
4547

@@ -54,6 +56,8 @@ the <<analysis-standard-analyzer,`standard` analyzer>>, the
5456
This is the purpose of _multi-fields_. Most datatypes support multi-fields
5557
via the <<multi-fields>> parameter.
5658

59+
include::types/alias.asciidoc[]
60+
5761
include::types/array.asciidoc[]
5862

5963
include::types/binary.asciidoc[]

‎docs/reference/mapping/types/alias.asciidoc

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
[[alias]]
2+
=== Alias datatype
3+
4+
NOTE: Field aliases can only be specified on indexes with a single mapping type. To add a field
5+
alias, the index must therefore have been created in 6.0 or later, or be an older index with
6+
the setting `index.mapping.single_type: true`. Please see <<removal-of-types>> for more information.
7+
8+
An `alias` mapping defines an alternate name for a field in the index.
9+
The alias can be used in place of the target field in <<search, search>> requests,
10+
and selected other APIs like <<search-field-caps, field capabilities>>.
11+
12+
[source,js]
13+
--------------------------------
14+
PUT trips
15+
{
16+
"mappings": {
17+
"_doc": {
18+
"properties": {
19+
"distance": {
20+
"type": "long"
21+
},
22+
"route_length_miles": {
23+
"type": "alias",
24+
"path": "distance" // <1>
25+
},
26+
"transit_mode": {
27+
"type": "keyword"
28+
}
29+
}
30+
}
31+
}
32+
}
33+
34+
GET _search
35+
{
36+
"query": {
37+
"range" : {
38+
"route_length_miles" : {
39+
"gte" : 39
40+
}
41+
}
42+
}
43+
}
44+
--------------------------------
45+
// CONSOLE
46+
47+
<1> The path to the target field. Note that this must be the full path, including any parent
48+
objects (e.g. `object1.object2.field`).
49+
50+
Almost all components of the search request accept field aliases. In particular, aliases can be
51+
used in queries, aggregations, and sort fields, as well as when requesting `docvalue_fields`,
52+
`stored_fields`, suggestions, and highlights. Scripts also support aliases when accessing
53+
field values. Please see the section on <<unsupported-apis, unsupported APIs>> for exceptions.
54+
55+
In some parts of the search request and when requesting field capabilities, field wildcard patterns can be
56+
provided. In these cases, the wildcard pattern will match field aliases in addition to concrete fields:
57+
58+
[source,js]
59+
--------------------------------
60+
GET trips/_field_caps?fields=route_*,transit_mode
61+
--------------------------------
62+
// CONSOLE
63+
// TEST[continued]
64+
65+
[[alias-targets]]
66+
==== Alias targets
67+
68+
There are a few restrictions on the target of an alias:
69+
70+
* The target must be a concrete field, and not an object or another field alias.
71+
* The target field must exist at the time the alias is created.
72+
* If nested objects are defined, a field alias must have the same nested scope as its target.
73+
74+
Additionally, a field alias can only have one target. This means that it is not possible to use a
75+
field alias to query over multiple target fields in a single clause.
76+
77+
[[unsupported-apis]]
78+
==== Unsupported APIs
79+
80+
Writes to field aliases are not supported: attempting to use an alias in an index or update request
81+
will result in a failure. Likewise, aliases cannot be used as the target of `copy_to` or in multi-fields.
82+
83+
Because alias names are not present in the document source, aliases cannot be used when performing
84+
source filtering. For example, the following request will return an empty result for `_source`:
85+
86+
[source,js]
87+
--------------------------------
88+
GET /_search
89+
{
90+
"query" : {
91+
"match_all": {}
92+
},
93+
"_source": "route_length_miles"
94+
}
95+
--------------------------------
96+
// CONSOLE
97+
// TEST[continued]
98+
99+
Currently only the search and field capabilities APIs will accept and resolve field aliases.
100+
Other APIs that accept field names, such as <<docs-termvectors, term vectors>>, cannot be used
101+
with field aliases.
102+
103+
Finally, some queries, such as `terms`, `geo_shape`, and `more_like_this`, allow for fetching query
104+
information from an indexed document. Because field aliases aren't supported when fetching documents,
105+
the part of the query that specifies the lookup path cannot refer to a field by its alias.

‎modules/lang-expression/src/test/java/org/elasticsearch/script/expression/ExpressionTests.java

Lines changed: 54 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -20,27 +20,52 @@
2020
package org.elasticsearch.script.expression;
2121

2222
import org.elasticsearch.common.settings.Settings;
23-
import org.elasticsearch.index.IndexService;
24-
import org.elasticsearch.index.query.QueryShardContext;
23+
import org.elasticsearch.index.fielddata.AtomicNumericFieldData;
24+
import org.elasticsearch.index.fielddata.IndexNumericFieldData;
25+
import org.elasticsearch.index.fielddata.SortedNumericDoubleValues;
26+
import org.elasticsearch.index.mapper.MapperService;
27+
import org.elasticsearch.index.mapper.NumberFieldMapper.NumberFieldType;
28+
import org.elasticsearch.index.mapper.NumberFieldMapper.NumberType;
2529
import org.elasticsearch.script.ScriptException;
2630
import org.elasticsearch.script.SearchScript;
2731
import org.elasticsearch.search.lookup.SearchLookup;
28-
import org.elasticsearch.test.ESSingleNodeTestCase;
32+
import org.elasticsearch.test.ESTestCase;
2933

34+
import java.io.IOException;
3035
import java.text.ParseException;
3136
import java.util.Collections;
3237

33-
public class ExpressionTests extends ESSingleNodeTestCase {
34-
ExpressionScriptEngine service;
35-
SearchLookup lookup;
38+
import static org.mockito.Matchers.anyInt;
39+
import static org.mockito.Matchers.anyObject;
40+
import static org.mockito.Mockito.mock;
41+
import static org.mockito.Mockito.when;
42+
43+
public class ExpressionTests extends ESTestCase {
44+
private ExpressionScriptEngine service;
45+
private SearchLookup lookup;
3646

3747
@Override
3848
public void setUp() throws Exception {
3949
super.setUp();
40-
IndexService index = createIndex("test", Settings.EMPTY, "type", "d", "type=double");
50+
51+
NumberFieldType fieldType = new NumberFieldType(NumberType.DOUBLE);
52+
MapperService mapperService = mock(MapperService.class);
53+
when(mapperService.fullName("field")).thenReturn(fieldType);
54+
when(mapperService.fullName("alias")).thenReturn(fieldType);
55+
56+
SortedNumericDoubleValues doubleValues = mock(SortedNumericDoubleValues.class);
57+
when(doubleValues.advanceExact(anyInt())).thenReturn(true);
58+
when(doubleValues.nextValue()).thenReturn(2.718);
59+
60+
AtomicNumericFieldData atomicFieldData = mock(AtomicNumericFieldData.class);
61+
when(atomicFieldData.getDoubleValues()).thenReturn(doubleValues);
62+
63+
IndexNumericFieldData fieldData = mock(IndexNumericFieldData.class);
64+
when(fieldData.getFieldName()).thenReturn("field");
65+
when(fieldData.load(anyObject())).thenReturn(atomicFieldData);
66+
4167
service = new ExpressionScriptEngine(Settings.EMPTY);
42-
QueryShardContext shardContext = index.newQueryShardContext(0, null, () -> 0, null);
43-
lookup = new SearchLookup(index.mapperService(), shardContext::getForField, null);
68+
lookup = new SearchLookup(mapperService, ignored -> fieldData, null);
4469
}
4570

4671
private SearchScript.LeafFactory compile(String expression) {
@@ -50,22 +75,38 @@ private SearchScript.LeafFactory compile(String expression) {
5075

5176
public void testNeedsScores() {
5277
assertFalse(compile("1.2").needs_score());
53-
assertFalse(compile("doc['d'].value").needs_score());
78+
assertFalse(compile("doc['field'].value").needs_score());
5479
assertTrue(compile("1/_score").needs_score());
55-
assertTrue(compile("doc['d'].value * _score").needs_score());
80+
assertTrue(compile("doc['field'].value * _score").needs_score());
5681
}
5782

5883
public void testCompileError() {
5984
ScriptException e = expectThrows(ScriptException.class, () -> {
60-
compile("doc['d'].value * *@#)(@$*@#$ + 4");
85+
compile("doc['field'].value * *@#)(@$*@#$ + 4");
6186
});
6287
assertTrue(e.getCause() instanceof ParseException);
6388
}
6489

6590
public void testLinkError() {
6691
ScriptException e = expectThrows(ScriptException.class, () -> {
67-
compile("doc['e'].value * 5");
92+
compile("doc['nonexistent'].value * 5");
6893
});
6994
assertTrue(e.getCause() instanceof ParseException);
7095
}
96+
97+
public void testFieldAccess() throws IOException {
98+
SearchScript script = compile("doc['field'].value").newInstance(null);
99+
script.setDocument(1);
100+
101+
double result = script.runAsDouble();
102+
assertEquals(2.718, result, 0.0);
103+
}
104+
105+
public void testFieldAccessWithFieldAlias() throws IOException {
106+
SearchScript script = compile("doc['alias'].value").newInstance(null);
107+
script.setDocument(1);
108+
109+
double result = script.runAsDouble();
110+
assertEquals(2.718, result, 0.0);
111+
}
71112
}

‎modules/percolator/src/main/java/org/elasticsearch/percolator/PercolateQuery.java

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,10 @@ Query getCandidateMatchesQuery() {
194194
return candidateMatchesQuery;
195195
}
196196

197+
Query getVerifiedMatchesQuery() {
198+
return verifiedMatchesQuery;
199+
}
200+
197201
// Comparing identity here to avoid being cached
198202
// Note that in theory if the same instance gets used multiple times it could still get cached,
199203
// however since we create a new query instance each time we this query this shouldn't happen and thus

‎modules/percolator/src/main/java/org/elasticsearch/percolator/PercolateQueryBuilder.java

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -634,13 +634,13 @@ protected Analyzer getWrappedAnalyzer(String fieldName) {
634634
docSearcher.setQueryCache(null);
635635
}
636636

637-
PercolatorFieldMapper percolatorFieldMapper = (PercolatorFieldMapper) docMapper.mappers().getMapper(field);
638-
boolean mapUnmappedFieldsAsString = percolatorFieldMapper.isMapUnmappedFieldAsText();
637+
PercolatorFieldMapper.FieldType pft = (PercolatorFieldMapper.FieldType) fieldType;
638+
String name = this.name != null ? this.name : pft.name();
639639
QueryShardContext percolateShardContext = wrap(context);
640+
PercolateQuery.QueryStore queryStore = createStore(pft.queryBuilderField,
641+
percolateShardContext,
642+
pft.mapUnmappedFieldsAsText);
640643

641-
String name = this.name != null ? this.name : field;
642-
PercolatorFieldMapper.FieldType pft = (PercolatorFieldMapper.FieldType) fieldType;
643-
PercolateQuery.QueryStore queryStore = createStore(pft.queryBuilderField, percolateShardContext, mapUnmappedFieldsAsString);
644644
return pft.percolateQuery(name, queryStore, documents, docSearcher, context.indexVersionCreated());
645645
}
646646

‎modules/percolator/src/main/java/org/elasticsearch/percolator/PercolatorFieldMapper.java

Lines changed: 23 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -142,13 +142,32 @@ public PercolatorFieldMapper build(BuilderContext context) {
142142
fieldType.rangeField = rangeFieldMapper.fieldType();
143143
NumberFieldMapper minimumShouldMatchFieldMapper = createMinimumShouldMatchField(context);
144144
fieldType.minimumShouldMatchField = minimumShouldMatchFieldMapper.fieldType();
145+
fieldType.mapUnmappedFieldsAsText = getMapUnmappedFieldAsText(context.indexSettings());
146+
145147
context.path().remove();
146148
setupFieldType(context);
147149
return new PercolatorFieldMapper(name(), fieldType, defaultFieldType, context.indexSettings(),
148150
multiFieldsBuilder.build(this, context), copyTo, queryShardContext, extractedTermsField,
149151
extractionResultField, queryBuilderField, rangeFieldMapper, minimumShouldMatchFieldMapper);
150152
}
151153

154+
private static boolean getMapUnmappedFieldAsText(Settings indexSettings) {
155+
if (INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.exists(indexSettings) &&
156+
INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.exists(indexSettings)) {
157+
throw new IllegalArgumentException("Either specify [" + INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.getKey() +
158+
"] or [" + INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.getKey() + "] setting, not both");
159+
}
160+
161+
if (INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.exists(indexSettings)) {
162+
DEPRECATION_LOGGER.deprecatedAndMaybeLog(INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.getKey(),
163+
"The [" + INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.getKey() +
164+
"] setting is deprecated in favour for the [" + INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.getKey() + "] setting");
165+
return INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.get(indexSettings);
166+
} else {
167+
return INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.get(indexSettings);
168+
}
169+
}
170+
152171
static KeywordFieldMapper createExtractQueryFieldBuilder(String name, BuilderContext context) {
153172
KeywordFieldMapper.Builder queryMetaDataFieldBuilder = new KeywordFieldMapper.Builder(name);
154173
queryMetaDataFieldBuilder.docValues(false);
@@ -201,6 +220,7 @@ static class FieldType extends MappedFieldType {
201220
MappedFieldType minimumShouldMatchField;
202221

203222
RangeFieldMapper.RangeFieldType rangeField;
223+
boolean mapUnmappedFieldsAsText;
204224

205225
FieldType() {
206226
setIndexOptions(IndexOptions.NONE);
@@ -215,6 +235,7 @@ static class FieldType extends MappedFieldType {
215235
queryBuilderField = ref.queryBuilderField;
216236
rangeField = ref.rangeField;
217237
minimumShouldMatchField = ref.minimumShouldMatchField;
238+
mapUnmappedFieldsAsText = ref.mapUnmappedFieldsAsText;
218239
}
219240

220241
@Override
@@ -333,7 +354,6 @@ Tuple<List<BytesRef>, Map<String, List<byte[]>>> extractTermsAndRanges(IndexRead
333354

334355
}
335356

336-
private final boolean mapUnmappedFieldAsText;
337357
private final Supplier<QueryShardContext> queryShardContext;
338358
private KeywordFieldMapper queryTermsField;
339359
private KeywordFieldMapper extractionResultField;
@@ -354,27 +374,9 @@ Tuple<List<BytesRef>, Map<String, List<byte[]>>> extractTermsAndRanges(IndexRead
354374
this.extractionResultField = extractionResultField;
355375
this.queryBuilderField = queryBuilderField;
356376
this.minimumShouldMatchFieldMapper = minimumShouldMatchFieldMapper;
357-
this.mapUnmappedFieldAsText = getMapUnmappedFieldAsText(indexSettings);
358377
this.rangeFieldMapper = rangeFieldMapper;
359378
}
360379

361-
private static boolean getMapUnmappedFieldAsText(Settings indexSettings) {
362-
if (INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.exists(indexSettings) &&
363-
INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.exists(indexSettings)) {
364-
throw new IllegalArgumentException("Either specify [" + INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.getKey() +
365-
"] or [" + INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.getKey() + "] setting, not both");
366-
}
367-
368-
if (INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.exists(indexSettings)) {
369-
DEPRECATION_LOGGER.deprecatedAndMaybeLog(INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.getKey(),
370-
"The [" + INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.getKey() +
371-
"] setting is deprecated in favour for the [" + INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.getKey() + "] setting");
372-
return INDEX_MAP_UNMAPPED_FIELDS_AS_STRING_SETTING.get(indexSettings);
373-
} else {
374-
return INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.get(indexSettings);
375-
}
376-
}
377-
378380
@Override
379381
public FieldMapper updateFieldType(Map<String, MappedFieldType> fullNameToFieldType) {
380382
PercolatorFieldMapper updated = (PercolatorFieldMapper) super.updateFieldType(fullNameToFieldType);
@@ -421,7 +423,7 @@ public Mapper parse(ParseContext context) throws IOException {
421423

422424
Version indexVersion = context.mapperService().getIndexSettings().getIndexVersionCreated();
423425
createQueryBuilderField(indexVersion, queryBuilderField, queryBuilder, context);
424-
Query query = toQuery(queryShardContext, mapUnmappedFieldAsText, queryBuilder);
426+
Query query = toQuery(queryShardContext, isMapUnmappedFieldAsText(), queryBuilder);
425427
processQuery(query, context);
426428
return null;
427429
}
@@ -541,7 +543,7 @@ protected String contentType() {
541543
}
542544

543545
boolean isMapUnmappedFieldAsText() {
544-
return mapUnmappedFieldAsText;
546+
return ((FieldType) fieldType).mapUnmappedFieldsAsText;
545547
}
546548

547549
/**

0 commit comments

Comments
 (0)