import ctranslate2 eagerly imports torch (via converters + specs), even for inference-only use

[vannhph53095]

Summary

Importing the top-level package —

import ctranslate2

— unconditionally imports PyTorch into the process if torch is installed in the environment, even when the user only does inference (Translator / Generator / Whisper) and never touches the converters. torch is only actually needed by the model-conversion code paths, not by inference, so pulling it in at package-import time is unnecessary for a large class of users.

Why this matters (real-world impact)

Inference-only users who happen to have torch in the same environment (very common — e.g. an app that also runs a torch-based component such as demucs, in the same Python process) pay for it in two ways:

1. Startup cost. Importing torch (2.8.0 + CUDA) adds several seconds to import ctranslate2 (see reproducer: ~4 s total, dominated by torch/CUDA init) even though inference never calls torch.

2. Cross-runtime interference. In our application, the same process also runs onnxruntime. We measured that merely having torch imported in the process makes onnxruntime inference ~4× slower on GPU (and measurably slower on CPU) — a known interaction (DLL / OpenMP / CUDA allocator) between torch and onnxruntime. Because import ctranslate2 (used only for faster-whisper inference) drags torch in, we were forced to move faster-whisper into a separate subprocess purely to keep the CTranslate2 side torch-free. A lazy import in CTranslate2 would remove the need for that workaround for anyone mixing CT2 inference with onnxruntime (or any other native runtime that dislikes sharing a process with torch).

The core, provable point is narrow and framework-agnostic: import ctranslate2 imports torch when it doesn't need to for inference. The onnxruntime slowdown is just our motivation for noticing it.

Root cause (verified on master, and on 4.8.0 / 4.8.1)

python/ctranslate2/__init__.py eagerly imports the converters and specs submodules:

from ctranslate2 import converters, models, specs
from ctranslate2.version import __version__

Both of those submodule trees do a module-level import torch:

• converters/__init__.py imports transformers.py, whose top of file has (lines 7-10 on master):

  try:
      import huggingface_hub
      import torch
      import transformers
  except ImportError:
      pass

• specs/__init__.py imports model_spec.py, whose top of file has (lines 17-22 on master):

  try:
      import torch
  
      torch_is_available = True
  except ImportError:
      torch_is_available = False

Note that most other torch imports in the converters are already lazy (function-level) — e.g. converters/fairseq.py, converters/opennmt_py.py, converters/utils.py, converters/eole_ct2.py. So there is already precedent in the codebase for importing torch only inside the code path that uses it. The two module-level sites above are the remaining eager ones.

Reproducer

import sys, time
t0 = time.perf_counter()
import ctranslate2
t1 = time.perf_counter()
print("ctranslate2 version :", ctranslate2.__version__)
print("import time         : %.2fs" % (t1 - t0))
print("torch in sys.modules:", "torch" in sys.modules)
print("transformers loaded :", "transformers" in sys.modules)

Output (Windows, Python 3.11, ctranslate2 4.8.0, torch 2.8.0+cu128):

ctranslate2 version : 4.8.0
import time         : 4.01s
torch in sys.modules: True
transformers loaded : False

torch is imported, while transformers is not (it isn't installed here) — showing the converter dependency tree is incomplete anyway, yet torch still gets loaded. This is pure overhead for an inference-only import.

Proposed fix (lazy import)

Make torch load only when a converter / spec code path that actually needs it runs. Two low-risk options, ideally combined:

1. Lazy-load the converters submodule in __init__.py via PEP 562 __getattr__, so an inference-only import ctranslate2 never pulls the converter dependency tree (transformers, huggingface_hub, torch):

   from ctranslate2 import models, specs
   from ctranslate2.version import __version__
   
   def __getattr__(name):
       if name == "converters":
           import ctranslate2.converters as converters
           return converters
       raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

   ctranslate2.converters.XxxConverter keeps working (imported on first access); the ct2-*-converter CLIs are unaffected.

2. Defer torch in specs/model_spec.py. Replace the module-level try: import torch with a lazy helper (import inside the function(s) that build torch tensors), and compute torch_is_available on demand. This removes the eager torch import from the specs path, which is loaded even in option 1.

Both keep behavior identical for converter users, and make inference-only import ctranslate2 torch-free.

Environment

• CTranslate2: 4.8.0 (root cause also present verbatim on master)
• Python: 3.11, Windows
• torch: 2.8.0+cu128 (present in env because another, unrelated component uses it)

---

I did a quick search of existing issues and didn't find this reported; apologies + please close as duplicate if I missed one. Happy to open a PR implementing the lazy-import approach above if you're open to it.

[Takemicom]

FWIW this same eager-init pattern exists one layer up too. faster-whisper (a common consumer of ctranslate2 for Whisper inference) has faster_whisper/__init__.py unconditionally do from faster_whisper.transcribe import BatchedInferencePipeline, WhisperModel, which imports ctranslate2 — so even code that only wants faster_whisper.vad.get_vad_model() (Silero VAD, no ctranslate2/Whisper involved at all) pays the full cost regardless of whether this fix lands in a release, unless faster-whisper also lazy-loads .transcribe the same way.

Measured on faster-whisper 1.2.1 / ctranslate2 4.8.1 (before this PR is released): importing faster_whisper.vad alone costs 7.31s, 6.49s of which is exactly this converters->transformers chain.

Opened SYSTRAN/faster-whisper#1479 with the same repro + a suggested __getattr__-based fix mirroring this PR's approach, in case it's useful there too.