fix(layout): fall back to the uncompiled model when torch.compile fails (#3956)

[Anai-Guo]

Related to #3956 (and complementary to #3962, which changes the default rather than the failure mode).

The problem

torch.compile() is lazy. It returns an OptimizedModule immediately and the backend only builds a kernel on the first forward pass. In TransformersObjectDetectionEngine the compile call sits inside initialize()'s try/except, but the first forward happens in predict_batch():

with torch.inference_mode():
    outputs = self._model(**inputs)

Nothing catches it there, so on a machine without a working C++ compiler the conversion dies with torch._dynamo.exc.BackendCompilerFailed wrapping InvalidCxxCompiler, surfaced to the user as a ConversionError.

Since #3914 made LayoutObjectDetectionOptions the default and compile_torch_models defaults to True, this sits on the default PDF path — a plain DocumentConverter().convert(...) raises. It is not Windows-specific: it reproduces the same way on a slim Linux image with no g++ (reported against python:3.12-slim in neuml/txtai#1176, where it also broke that project's Windows CI).

The change

Compilation is an optimization, not a correctness requirement, so keep a handle on the uncompiled module and degrade to it with a warning instead of failing the conversion:

• initialize() records self._uncompiled_model before handing the model to torch.compile().
• A small _forward() helper wraps the call. On torch._dynamo.exc.TorchDynamoException it logs a warning naming the env var, swaps _model back to the uncompiled module, and retries once.
• The swap is permanent, so subsequent batches do not pay the failed-compile cost again.

Deliberately narrow: if no uncompiled handle was recorded (compile_model=False, so nothing was compiled), the error propagates unchanged; anything that is not a TorchDynamoException propagates unchanged, so a genuine model error is never masked as a compile problem.

Verification

torch.compile() really does defer the failure to the first forward — confirmed on torch 2.5.1, Python 3.12, Windows with no cl/g++ on PATH:

compile() returned OK: OptimizedModule
FORWARD FAILED at first call -> torch._dynamo.exc.BackendCompilerFailed
MRO: [BackendCompilerFailed, TorchDynamoException, RuntimeError, Exception, ...]

Against a genuinely failing compile on that machine:

	before	after
first forward	raises BackendCompilerFailed → ConversionError	warns, falls back, returns output
genuine model error	propagates	propagates (unchanged)
nothing to fall back to	raises	raises (unchanged)

tests/test_object_detection_compile_fallback.py covers those three rows and needs no model download. It fails 3/3 against the code before this change and passes 3/3 after.

ruff==0.15.12 check and format --check clean with the repo config.

Notes

• The sibling image_classification and vlm transformers engines share the same shape. I left them alone to keep this focused on the default layout path that refactor(layout): run all layout inference through the object-detection factory path #3914 turned on; happy to extend if you'd prefer one pass.
• This is orthogonal to fix(settings): disable torch compilation by default on Windows #3962: that one decides whether to compile, this one decides what happens when compiling fails anyway — including on Linux, and on Windows machines that do have MSVC but hit some other backend failure.

🤖 Generated with Claude Code

[github-actions]

✅ DCO Check Passed

Thanks @Anai-Guo, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ce_engines/object_detection/transformers_engine.py	91.66%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!