Word_delimiter and Word_delimiter_graph not working with numbers and spaces

[xDouglasx]

Describe the feature:

Elasticsearch version (bin/elasticsearch --version):6.4.0

Plugins installed: []

JVM version (java -version): 1.8.0_181"

Windows, Docker 18.06.1

Description of the problem including expected versus actual behavior:

The following index-setting and mapping script works in versions 5.6.6.X but fails when run against V 6.4.0:

Hi, im upgrading ElasticSearch from 5.6.6 to 6..4.0
I have an image running on docker.

When im ingesting data to elastic search
im having problem with word_delimiter.

according with this ticket
#28474

Changing the word_delimiter to word_delimiter_graph should fix the problem.
but i changed and the problem remains.

Analyzer:

"nameAnalyzer": {
"type": "custom",
"filter": ["lowercase",
"trim",
"word_delimiter_graph",
"name_stopwords",
"my_ascii_folding"],
"tokenizer": "nameNGram"
},

Data:

{"dateOfBirth":"1988-12-21","yearOfBirth":"1988","monthOfBirth":"12","dayOfBirth":"21","lastName":"Juan Humara","firstName":"Romeo","middleName":"H","suffix":""}

Logs:

{
"took": 14,
"errors": true,
"items": [
{
"index": {
"_index": "my-index",
"_type": "my_files",
"_id": "9PyI12UB7zhqwOMdQng1",
"status": 400,
"error": {
"type": "illegal_argument_exception",
"reason": "startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=4,endOffset=7,lastStartOffset=5 for field 'lastName'"
}
}
}
]
}

It happens because i have two names as last names "Juan Humara" and space between then.
If i Remove the space and put "JHumara" it works perfectly.

[elasticmachine]

Pinging @elastic/es-search-aggs

[romseygeek]

Hi @xDouglasx, can you post the complete mapping, including the settings for word_delimiter_graph, name_stopwords and my_ascii_folding so that I can try and reproduce this?

[romseygeek]

I have a feeling that the tokenizer may be part of the problem as well.

[xDouglasx]

Hi,

Here is my mapping

{
"mappings": {
"my_files": {
"_all": {
"enabled": false
},
"_source": {
"excludes": [
"CFC"
]
},
"properties": {
"lastName": {
"type": "text",
"analyzer": "nameAnalyzer"
},
"firstName": {
"type": "text",
"analyzer": "nameAnalyzer"
},
"middleName": {
"type": "text",
"analyzer": "nameAnalyzer"
}
}
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"analysis": {
"char_filter": {
"whitespaceCharFilter": {
"type": "pattern_replace",
"pattern": " ",
"replacement": ""
},
"numberify": {
"type": "pattern_replace",
"pattern": "[^\\d]+",
"replacement": ""
},
"dashCharFilter": {
"type": "mapping",
"mappings": [
"-=>"
]
}
},
"filter": {
"whitespace_remove": {
"type": "pattern_replace",
"pattern": " ",
"replacement": ""
},
"stateNGram": {
"type": "nGram",
"min_gram": "3",
"max_gram": "5"
},
"english_stemmer": {
"type": "stemmer",
"language": "english"
},
"cityNGram": {
"type": "nGram",
"min_gram": "5",
"max_gram": "5"
},
"snowball": {
"type": "snowball",
"language": "English"
},
"zip_edge_ngrams": {
"type": "edge_ngram",
"min_gram": "5",
"max_gram": "9"
},
"zipCodeNGram": {
"type": "nGram",
"min_gram": "5",
"max_gram": "9"
},
"my_ascii_folding": {
"type": "asciifolding",
"preserve_original": "true"
},
"truncate": {
"type": "truncate",
"length": "4"
},
"english_possessive_stemmer": {
"type": "stemmer",
"language": "possessive_english"
},
"worddelimiter": {
"type": "word_delimiter_graph"
},
"state_abbreviations": {
"type": "synonym",
"synonyms_path": "state_abbreviations.txt"
},
"address_abbreviations": {
"type": "synonym",
"synonyms_path": "street_abbreviations.txt"
},
"suffix_abbreviations": {
"type": "synonym",
"synonyms_path": "suffix_abbreviations.txt"
},
"addressLineNGram": {
"type": "nGram",
"min_gram": "3",
"max_gram": "5"
},
"stopwords": {
"type": "stop",
"stopwords": [
"industries",
"industry"
]
},
"name_stopwords": {
"type": "stop",
"stopwords": [
"nothing"
]
},
"address_stopwords": {
"type": "stop",
"stopwords": [
"alley",
"aly",
"anx",
"arc",
"ave",
"byu",
"bch"
]
},
"name_abbreviations": {
"type": "synonym",
"synonyms_path": "name_abbreviations.txt"
},
"dbl_metaphone": {
"type": "phonetic",
"encoder": "metaphone",
"replace": "true"
}
},
"analyzer": {
"addressLineAnalyzer": {
"type": "custom",
"filter": [
"lowercase",
"unique",
"address_abbreviations",
"address_stopwords"
],
"tokenizer": "standard"
},
"addressLineRawAnalyzer": {
"type": "custom",
"filter": [
"lowercase",
"unique",
"address_abbreviations",
"address_stopwords"
],
"tokenizer": "standard"
},
"nameAnalyzer": {
"type": "custom",
"filter": [
"lowercase",
"trim",
"word_delimiter_graph",
"name_stopwords",
"my_ascii_folding"
],
"tokenizer": "nameNGram"
},
"nameRawAnalyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"trim",
"name_stopwords",
"my_ascii_folding"
]
},
"nameSynAnalyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"trim",
"my_ascii_folding",
"name_stopwords",
"name_abbreviations"
]
},
"nameSndxAnalyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"trim",
"my_ascii_folding",
"name_stopwords",
"dbl_metaphone"
]
},
"suffixAnalyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"trim",
"my_ascii_folding",
"suffix_abbreviations"
]
},
"allNameAnalyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"trim",
"my_ascii_folding",
"name_stopwords",
"unique"
]
}
},
"tokenizer": {
"csvTokenizer": {
"type": "pattern",
"pattern": ","
},
"nameNGram": {
"type": "nGram",
"min_gram": "3",
"max_gram": "4"
},
"phoneNGram": {
"type": "nGram",
"min_gram": "4",
"max_gram": "4"
}
}
},
"number_of_shards": "25"
}
}
}

and here is my data:

{"index":{"_index":"my-index","_type":"my_files"}}
{"dateOfBirth": "1988-12-21", "yearOfBirth": "1988","monthOfBirth": "12", "dayOfBirth": "21", "lastName": "Juan Humara", "firstName": "Romeo", "middleName": "H", "suffix": "", "addressline": "","streetNumber": "6","streetName": "Side st","streetType": "","preDirection": "", "postDirection": "","unitType": "", "unitNumber": "", "city": "Pigeon forge","state": "TN", "zip": "98987", "zip4": "", "phone": "", "addressType": ""}

thanks in advance

[romseygeek]

This is a combination of ngram tokenizer and word delimiter filter that's producing bad offsets, and is a bug somewhere in lucene.

What are you trying to achieve with the nameAnalyzer? Ngrams and word_delimiter together are going to produce a very strange tokenstream.

[romseygeek]

Scratch that, it appears that it's actually the combination of trim and word_delimiter_graph that's causing the problem, and ngram is only contributing by emitting tokens that contain whitespace.

[xDouglasx]

there is a way to make they work? it will be fixed?

[romseygeek]

I'll try and get a fix into the next lucene release, but for now you'll need to work around it. What are you trying to achieve with this analysis chain? Will it still do what you need if you remove the trim filter?

[xDouglasx]

Thanks, it seems we found a work around here for now, but still not tested, as soon that i test this, i will share with you what we have done to avoid this.

[romseygeek]

I opened a lucene issue here: https://issues.apache.org/jira/browse/LUCENE-8509

[xDouglasx]

Thanks!