What's wrong
test-unstructured2graph fails 22 tests with:
LookupError:
**********************************************************************
Resource 'averaged_perceptron_tagger_eng' not found.
Please use the NLTK Downloader to obtain the resource:
>>> import nltk
>>> nltk.download('averaged_perceptron_tagger_eng')
Attempted to load 'taggers/averaged_perceptron_tagger_eng/'
Searched in:
- '/home/runner/nltk_data'
- '/home/runner/work/ai-toolkit/ai-toolkit/unstructured2graph/.venv/nltk_data'
- '/home/runner/work/ai-toolkit/ai-toolkit/unstructured2graph/.venv/share/nltk_data'
- '/home/runner/work/ai-toolkit/ai-toolkit/unstructured2graph/.venv/lib/nltk_data'
- '/usr/share/nltk_data'
- '/usr/local/share/nltk_data'
- '/usr/lib/nltk_data'
- '/usr/local/lib/nltk_data'
**********************************************************************
Example failure (test_from_texts_only_chunks_creates_real_chunk_nodes), full call chain:
tests/test_e2e.py: from_texts(["Alice works on the graph engine.", ...], memgraph, only_chunks=True)
-> src/unstructured2graph/loaders.py:309 (from_texts) -> parse_text(text)
-> src/unstructured2graph/loaders.py:99 -> unstructured.partition.text.partition_text
-> unstructured/partition/text.py -> element_from_text -> is_possible_narrative_text
-> unstructured/partition/text_type.py:186 (contains_verb) -> nltk pos_tag
-> nltk/tag/__init__.py -> PerceptronTagger() -> nltk/data.py:find() -> LookupError
unstructured.nlp.tokenize actually has a built-in safety net for exactly this
(.venv/lib/python3.10/site-packages/unstructured/nlp/tokenize.py):
# auto download nltk packages if the environment variable is set
if os.getenv("AUTO_DOWNLOAD_NLTK", "True").lower() == "true":
download_nltk_packages()
download_nltk_packages() checks for averaged_perceptron_tagger_eng (taggers) and
punkt_tab (tokenizers) and calls nltk.download(..., quiet=True) for whichever is
missing -- this runs automatically at import time and defaults to on. It's evidently
not succeeding in the CI runner: quiet=True suppresses nltk.download()'s own
diagnostic output, so a failed download (network/CDN blip, egress restriction to
NLTK's data host, timeout, etc.) fails silently and this LookupError is the first
visible symptom, later, wherever pos_tag/contains_verb actually gets called. This
would also explain the "52 passed, 22 failed" split in one run: only the tests whose
code path reaches pos_tag fail, and since the download attempt only happens once per
process (module-level, first import), every subsequent call in that same test run
keeps failing once it's failed once.
Not something introduced by #256 -- that PR only touched unstructured2graph's
README/example, no dependency or code changes. Unclear whether this is newly broken
(e.g. a CDN/host change, or an interaction with the nltk<3.10 pin from #251) or a
latent gap that just hadn't been hit before; worth checking both.
Acceptance criteria
Related
Found while investigating CI failures on #256 (unrelated to that PR's own changes).
What's wrong
test-unstructured2graphfails 22 tests with:Example failure (
test_from_texts_only_chunks_creates_real_chunk_nodes), full call chain:unstructured.nlp.tokenizeactually has a built-in safety net for exactly this(
.venv/lib/python3.10/site-packages/unstructured/nlp/tokenize.py):download_nltk_packages()checks foraveraged_perceptron_tagger_eng(taggers) andpunkt_tab(tokenizers) and callsnltk.download(..., quiet=True)for whichever ismissing -- this runs automatically at import time and defaults to on. It's evidently
not succeeding in the CI runner:
quiet=Truesuppressesnltk.download()'s owndiagnostic output, so a failed download (network/CDN blip, egress restriction to
NLTK's data host, timeout, etc.) fails silently and this LookupError is the first
visible symptom, later, wherever
pos_tag/contains_verbactually gets called. Thiswould also explain the "52 passed, 22 failed" split in one run: only the tests whose
code path reaches
pos_tagfail, and since the download attempt only happens once perprocess (module-level, first import), every subsequent call in that same test run
keeps failing once it's failed once.
Not something introduced by #256 -- that PR only touched
unstructured2graph'sREADME/example, no dependency or code changes. Unclear whether this is newly broken
(e.g. a CDN/host change, or an interaction with the
nltk<3.10pin from #251) or alatent gap that just hadn't been hit before; worth checking both.
Acceptance criteria
download_nltk_packages()'s auto-download isn't succeeding in thetest-unstructured2graphCI job (network egress? rate limiting? interactionwith the
nltk<3.10pin?)explicit
python -m nltk.downloader averaged_perceptron_tagger_eng punkt_tabstep in
.github/workflows/tests.yaml'stest-unstructured2graphjob (withvisible output, not
quiet=True), so a real failure is loud instead ofsurfacing as 22 unrelated-looking test failures
test-unstructured2graph's full suite passes cleanly in CIRelated
Found while investigating CI failures on #256 (unrelated to that PR's own changes).