normalizers to support pattern_replace char filter

[synhershko]

PUT test-index
{
  "settings": {
    "analysis": {
      "char_filter": {
        "whitespace_remove": {
          "type": "pattern_replace",
          "pattern": "\\s+",
          "replacement": " "
        }
      },
      "normalizer": {
        "my_normalizer": {
          "type": "custom",
          "char_filter": ["whitespace_remove"],
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "type": {
      "properties": {
        "foo": {
          "type": "keyword",
          "normalizer": "my_normalizer"
        }
      }
    }
  }
}

Returns with :

{
   "error": {
      "root_cause": [
         {
            "type": "illegal_argument_exception",
            "reason": "Custom normalizer [my_normalizer] may not use char filter [whitespace_remove]"
         }
      ],
      "type": "illegal_argument_exception",
      "reason": "Custom normalizer [my_normalizer] may not use char filter [whitespace_remove]"
   },
   "status": 400
}

Note the use case - normalizing whitespace of keyword fields, to overcome differences in user queries (for exact boosting) or aggregations on user based content

[jpountz]

We accept the same set of analysis components for normalizers and analyzing prefix/wildcard queries, and I think pattern_replace filters should be applied. I opened https://issues.apache.org/jira/browse/LUCENE-7692 to discuss it.

[pmusa]

I am getting the same error trying to use synonyms. Is it also going to be supported by the change @jpountz?

[jpountz]

The particular issue that @synhershko reported will be fixed when we upgrade to Lucene 6.5 (which is expected to be the case for Elasticsearch 5.4). However synonyms are more complicated since the set of synonym rules is currently opaque to the consumers of the synonym filter, so they have no way to know whether the synonym filter introduces new tokens or whether it only replaces some tokens.

[pmusa]

What I wanted was a dictionary. Like synonyms with just the keyword tokenizer, but that is not guaranteed for all users :/ Guess fielddata it is :)

[jpountz]

Can't you do something with ingest?

[pmusa]

I need a dictionary lookup: "elastic search" => "elasticsearch", "es" => "elasticsearch", which is not natively implemented (and I don't think it will). I could do it with scripts defining the dictionary (it is small) in the pipeline itself. (I will try both and check the performance impact). Thanks

[maciejjarzynski]

Is there any update on that issue?
I want to be able to sort on the text field which has pattern_replace char filter applied to analyzer.
With that issue working, I'd put a keyword multifield with appropriate normalizer.
Is it possible in any way without enabling fielddata on that text field?

[Lackoftactics]

Any progress on 6.2? Need this feature

[jpountz]

Added to 5.4.0 via #23385.