How to download spacy NLP models using rye?

[saona-raimundo]

SpaCy is a Python NLP library.
Since models are big, they use a special pipeline to download them.
A basic example is

> python -m spacy download en_core_web_trf

// main.py
import spacy
nlp = spacy.load("en_core_web_trf")
import en_core_web_trf
nlp = en_core_web_trf.load()
doc = nlp("This is a sentence.")
print([(w.text, w.pos_) for w in doc])

from https://spacy.io/models

I have trouble running the special download pipeline of SpaCy through rye.
How would you make it work?

[goschtl]

Hi,

it's easy you can just use this one

rye run python -m spacy download en_core_web_trf

you have to make sure that pip is also installed via

rye add pip

HTH
Christian

[limarta]

It seems that after pip installs a model, rye is not informed and thus the pyproject.toml doesn't have the downloaded model. Indeed rye proceeds to uninstall en_core_web_trf after a rye sync or rye add x. Is it possible to prevent rye from removing the downloaded model, say, like adding it to the pyproject.toml?

[math625f]

@limarta Did you ever manage to solve this?

[cdolek]

I just asked here: #1387

[drothermel]

According to the Spacy docs:

To download a trained pipeline directly using pip, point pip install to the URL or local path of the wheel file or archive. Installing the wheel is usually more efficient.

# With external URL
pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0-py3-none-any.whl
pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0.tar.gz

# Using spacy info to get the external URL
pip install $(spacy info en_core_web_sm --url)

# With local file
pip install /Users/you/en_core_web_sm-3.0.0-py3-none-any.whl
pip install /Users/you/en_core_web_sm-3.0.0.tar.gz

So I tried the following which seems to work:
rye add en_core_web_sm --url $(spacy info en_core_web_sm --url)

[cdolek]

@drothermel you are the best, thank you!