Multi-word tokens in non-English languages

[danielhers]

Feature request: include tokenization to syntactic words in default models for non-English languages.
This is necessary for parsing Universal Dependencies: http://universaldependencies.org/format.html#words-tokens-and-empty-nodes, http://universaldependencies.org/conll17/evaluation.html

As expected, spaCy correctly splits English multi-word tokens (which are admittedly already pre-split in the UD data...):

nlp = spacy.load("en")
list(nlp("don't know"))  # -> [do, n't, know]
list(nlp("cannot"))  # -> [can, not]

However, for German and French, for example, the default models do not split multi-word tokens:

nlp = spacy.load("de_core_news_md")
list(nlp("zur Schule"))  # -> [zur, Schule]; expected: [zu, der, Schule]
list(nlp("Jugendtrainer beim Münchner TSV"))  # -> [Jugendtrainer, beim, Münchner, TSV]; expected: [Jugendtrainer, bei, dem, Münchner, TSV]

nlp = spacy.load("fr_depvec_web_lg")
list(fr("Gâteau au chocolat"))  # -> [Gâteau, au, chocolat]; expected: [Gâteau, à, le, chocolat]

Info about spaCy

• Python version: 3.5.2+
• spaCy version: 1.9.0
• Installed models: en_core_web_md, en, de_core_news_md, de, fr_depvec_web_lg
• Platform: Linux-4.8.4-aufs-1-x86_64-with-debian-stretch-sid

[ines]

I think the main problem here is that spaCy doesn't modify the token text – the following always needs to hold up:

assert ''.join([token.text_with_ws for token in doc]) == doc.text

However, using tokenizer exceptions, you can enforce the splitting behaviour, and set the full token text as the NORM attribute – for example 'zur': [{ORTH: 'zu'}, {ORTH: 'r', NORM: 'der'}]. In spaCy v2.0, the NORM attribute is also used as a feature in the model – this also allows you to normalise tokens to ensure they receive similar representations, even if one of them is much less frequent in the training data (see here for examples).

I guess the reason this wasn't implemented in spaCy by default for German is that the result is pretty confusing an unintuitive to the user. A tokenizer exception rule like "zur": [{ORTH: 'zu'}, {ORTH: 'r', NORM: 'der'}] would result in "zur Schule" → ['zu', 'r', 'Schule'].

[danielhers]

Thanks, that's very helpful. Perhaps this could be an optional feature, so that users won't have to implement the tokenizer exceptions themselves? From a universal perspective, I'd expect to get zur -> zu + r (der) and au -> a (à) + u (le) just as much as I'd expect to get don't -> do + n't (not) and I'm -> I + 'm (am).

[ines]

@danielhers Yeah, this definitely makes sense – the only question is, how we would present this additional feature, without making it confusing.

I think this may actually be a good case for a spaCy extension package? With the new version coming up, we'd love to encourage more third-party spaCy plugins – especially now that it's so much easier to extend the pipeline and Doc, Token and Span attributes. Another advantage is that users can try out new features and publish them as separate packages first – without having to worry how something might fit into the core library. Of course, integrating extensions into the core library is always an option – but it's much easier if the extension already exists.

In this case, the extension would be slightly different from the pipeline component extensions, since it hooks in before the pipeline. But it could still work in a similar way, and probably add other features useful for working with UD:

nlp = spacy.load('de')
nlp.tokenizer = UniversalDependencify(nlp.tokenizer)

What do you think?

[danielhers]

Sounds good, I think. I need to learn more about extension packages, but I'm happy with the API you suggested.

[ines]

@danielhers Nice!

The new extension packages are mostly for pipeline components, i.e. everything that happens after a Doc object is created. Since the UD extension needs to hook in earlier and modify the tokenizer, you could just write a function that takes the nlp object and takes care of modifying the tokenizer etc. For example, something like this (untested and maybe not the most elegant way, but should work):

# lookup table of UD-specific tokenization patterns, keyed by language.
# (the special case rules can also set the LEMMA and TAG attribute btw!)
special_case_rules = { 
    'de': {'zur': [{ORTH: 'zu'}, {ORTH: 'r', NORM: 'der'}]}
}

def universal_dependencify(nlp):
    if nlp.lang not in special_case_rules:
        raise ValueError("No data available for %s" % nlp.lang)
    for string, rule in special_case_rules[nlp.lang].items():
        nlp.tokenizer.add_special_case(string, rule)
    return nlp

Since the nlp object is modified in place, you don't even have to return it. But it might be nice to allow usage like:

nlp =  universal_dependencify(spacy.load('de'))

[ines]

Merging this with #2011!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.