Update wikipedia tutorial

[tmylk]

@mukulmalik18 had an idea to re-do the wikipedia tutorial with wikipedia-extractor package and then spacy/pattern parser.

It will be a nice new tutorial. Leaving to @mukulmalik18 to add some starting code and expand.

[mukulmalik18]

Hey!
So I use word vectors a lot. One main issue i faced was with compound words.
Example:
-> Normally
most_similar( [ 'iron' , 'maiden' ] ) is 'copper'

-> Whereas if we use compound nouns
most_similar( 'iron_maiden' ) is 'judas_preists'

Now by default people use "make_wiki.py" ( 'gensim/scripts/' ) to clean the data.
The issues with that are as follow:

1. It cleans all the alpha-numerics like '6s' which can be pretty usefull at times.
2. It removes all the grammar (english grammar), which can be parsed and different variants of word2vecs can be created (Like the one I created for compound nouns).

What I propose:

1. 'bwbaugh' has created a nice modules called 'wikipedia-extractor' which just removes the XML and other styles, leaving behind sentences with their grammar intact. Link : https://github.com/bwbaugh/wikipedia-extractor
2. Provide modules with compound words etc.

[mukulmalik18]

I had the opportunity to show @tmylk one such code. I have pasted that same code with its output below.

#Libraries
from spacy.en import English
parser_spacy = English()
import re
from pattern.en import parse as parser_pattern

#Cleaning function
def compound(text):
    results=''
    parsed = parser_spacy(text.decode('utf-8'))
    sents = [str(x) for x in list(parsed.sents)]
    for sent in sents:
        sent=sent.decode('utf-8').replace('-','_')
        tokens = [x.split('/')[:2] if x.split('/')[1].startswith('NN') else " "+x.split('/')[0]+" " for x in parser_pattern(sent).split(' ')]
        for i,token in enumerate(tokens):
                if token[1].startswith('NN'):
                    if tokens[i][1] == tokens[i-1][1]:
                        results+="_"+token[0]
                    else:
                        results+=" $"+token[0]
                else:
                    results+=" "+token
    return re.sub('\s+',' ',results.strip())

#Styling-free text
text = "The NVIDIA TITAN X, featuring the NVIDIA Pascal architecture, is the ultimate graphics card."

compound(text)

Output

u'The $NVIDIA_TITAN_X , featuring the $NVIDIA_Pascal $architecture , is the ultimate $graphics $card .'

Here words starting with '$' are nouns and compound nouns are combined using '_'

This is just one of the many experiments we can be done with Word2Vec.

[tmylk]

Thanks for sharing the code. It will be most useful in a context of a jupyter notebook where the compound words word2vec model is trained.

[menshikh-iv]

@mukulmalik18 it's relevant now?

[mukulmalik18]

@menshikh-iv No, you can easily do the same with Spacy using noun_chunks functionality.

[menshikh-iv]

Thanks @mukulmalik18