Skip to content

Conversation

JackyHe398
Copy link
Contributor

@JackyHe398 JackyHe398 commented Sep 15, 2025

fix of issue #19. The model now should normally through an error when the model corrupted and fallback to low memory model when memory not enough.

test.py:

import logging
logging.basicConfig(level=logging.DEBUG, format='%(levelname)s:%(name)s:%(message)s')

from src import fast_langdetect
fast_langdetect.infer._default_detector = fast_langdetect.infer.LangDetector(fast_langdetect.infer.LangDetectConfig(cache_dir="/tmp/fasttext-langdetect"))

import os, sys, resource
limit = <limit in MiB>* 1024 * 1024
resource.setrlimit(resource.RLIMIT_AS, (limit, limit))

print("Testing different behavior between MemoryError and model corruption:")
result = fast_langdetect.detect('Hello world', low_memory=False)
print('Result:', result)

result when model corrupted:

(GPTSoVITS) hekh@ljz-MS-7D76:/tmp/fasttext-langdetect$ /home/hekh/miniconda3/envs/GPTSoVITS/bin/python /home/hekh/code/fast-langdetect/test.py
Testing different behavior between MemoryError and model corruption:
Traceback (most recent call last):
  File "/home/ljz/code/fast-langdetect/src/fast_langdetect/infer.py", line 192, in _load_unix
    return fasttext.load_model(str(model_path))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hekh/miniconda3/envs/GPTSoVITS/lib/python3.11/site-packages/fasttext/FastText.py", line 82, in load_model
    return _FastText(model_path=path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hekh/miniconda3/envs/GPTSoVITS/lib/python3.11/site-packages/fasttext/FastText.py", line 29, in __init__
    self.f.loadModel(model_path)
ValueError: /tmp/fasttext-langdetect/lid.176.bin has wrong file format!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hekh/code/fast-langdetect/test.py", line 12, in <module>
    result = fast_langdetect.detect('Hello world', low_memory=False)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljz/code/fast-langdetect/src/fast_langdetect/infer.py", line 444, in detect
    return _default_detector.detect(text, low_memory=low_memory)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljz/code/fast-langdetect/src/fast_langdetect/infer.py", line 343, in detect
    model = self._get_model(low_memory)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljz/code/fast-langdetect/src/fast_langdetect/infer.py", line 317, in _get_model
    model = self._model_loader.load_with_download(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljz/code/fast-langdetect/src/fast_langdetect/infer.py", line 130, in load_with_download
    return self.load_local(model_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljz/code/fast-langdetect/src/fast_langdetect/infer.py", line 124, in load_local
    return self._load_unix(model_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljz/code/fast-langdetect/src/fast_langdetect/infer.py", line 197, in _load_unix
    raise DetectError(f"fast-langdetect: Failed to load model: {e}")
src.fast_langdetect.infer.DetectError: fast-langdetect: Failed to load model: /tmp/fasttext-langdetect/lid.176.bin has wrong file format!

result when memory not enough but model normal:

(GPTSoVITS) hekh@ljz-MS-7D76:/tmp/fasttext-langdetect$ /home/hekh/miniconda3/envs/GPTSoVITS/bin/python /home/hekh/code/fast-langdetect/test.py
Testing different behavior between MemoryError and model corruption:
INFO:src.fast_langdetect.infer:fast-langdetect: Falling back to low-memory model...
Result: {'lang': 'en', 'score': 0.1682594120502472}

@sudoskys
Copy link
Member

Thank you for your contribution! It means a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants