Suggester throws error when it has 30 or more contexts

[leibale]

When indexing a document and there is a completion field with 30 or more contexts it throws an error Illegal value [\u001] UTF-16 codepoint [0x1d] at position 0 is a reserved character.

Elasticsearch version: 6.3.2

Steps to reproduce:
Use this node.js script to reproduce the bug

(async () => {
    const elasticSearch = require('elasticsearch'),
        client = new elasticSearch.Client({
            host: 'host'
        }),
        index = 'test-index';

    async function _createIndex(numOfContexts) {
        const numOfContextsArray = new Array(numOfContexts).fill(null);

        await client.indices.delete({
            index,
            ignore: 404
        });

        await client.indices.create({
            index,
            body: {
                mappings: {
                    _doc: {
                        properties: {
                            ...numOfContextsArray.reduce((branchMappings, value, i) => {
                                branchMappings[`property${i}`] = {
                                    type: 'keyword'
                                };
                                return branchMappings;
                            }, {}),
                            suggest: {
                                type: 'completion',
                                contexts: numOfContextsArray.map((value, i) => ({
                                    name: `context${i}`,
                                    type: 'category',
                                    path: `property${i}`
                                }))
                            }
                        }
                    }
                }
            }
        });

        await client.index({
            index,
            type: '_doc',
            body: {
                ...numOfContextsArray.reduce((branchMappings, value, i) => {
                    branchMappings[`property${i}`] = 'value';
                    return branchMappings;
                }, {}),
                suggest: ['test']
            }
        });
    }

    await _createIndex(29);
    console.log('29 contexts is working');

    try {
        await _createIndex(30); // throwing an error
    } catch (err) {
        console.error(err);
    }
})();

[jimczi]

The context suggester uses an id to differentiate the contexts that can clash with a reserved character. 0x1d is 29 in decimal so you can use up to 29 contexts without hitting this bug(we start the id at 0) . This is clearly a bug but 30 is already too much. We don't limit anything here but more than 2, 3 contexts should be discouraged so I am inclined to use this bug to limit the number of allowed context mapping.

[elasticmachine]

Pinging @elastic/es-search-aggs

[leibale]

So, what do you suggest me to do?

I have a store index with products, each product has names array and branches object (with id as the key and a branch object as the value). Each branch object contains data of the product in the specific branch (like isActive, price, etc.).

How can I perform a suggest on the names when the product is active in a specific branch (like branches.${branchId}.isActive) without adding a context for each branch (each product has 40-50 branches)?

Product example:

{
  "id": 1,
  "names": ["Product Name", "Another Product Name"],
  "branches": {
    "1": {
      "isActive": true,
      "price": 10
    },
    "2": {
      "isActive": false
    }
  }
}

[jimczi]

If you have a lot of contexts with different types (numbers, booleans, texts) I'd suggest to use a text field for names with a keyword tokenizer and an edge_ngram filter to index all prefixes in the index_analyzer and a simple keyword tokenizer for the search_analyzer. This way you can use the query DSL to search your names field and use the other fields in your mapping as filters. Using too many contexts in the completion suggester is a sign that you need a real index with multi-fields support, the cost of recreating and querying this index in the completion suggester defeats the purpose of this special field.

[jimczi]

How can I perform a suggest on the names when the product is active in a specific branch (like branches.${branchId}.isActive) without adding a context for each branch (each product has 40-50 branches)?

Sorry I misread your comment, if you only need to know if the suggestion is active in a branch you can use a single context mapping, branch for instance, and use the ids of the active branches as context values. So for instance if product 1 is active in branches 1, 4 and 45 you can index it as:

{
    "suggest": {
        "input": "product 1",
        "contexts": {
            "place_type": ["1", "4", "45"] 
        }
    }
}

I opened #32741 to address the >=30 contexts failures
so I hope you don't mind if I close this issue.