[Bug] DFine-X fails to train with custom input size

[olkham]

🧩 Issue type

Bug 🐞

📄 Description

Changing input size to 124x124, 800x800 or 1024x1024 it fails to train.

📋 Steps to reproduce (if applicable)

1. Select advanced settings when training a DFine model
2. Increase input size to 124x124, 800x800 or 1024x1024 (only tried testing these)
3. Leave all others default
4. hit Start

💻 Expected behavior

Model should train without any issue using a larger input size

🖼️ Screenshots, video recordings or logs

No response

📦 Geti version

2.12.1

Hardware

i9-14900KS, 128GB, RTX 4500 24GB

🖥️ Operating System

Ubuntu 24.04

Browser (if applicable)

No response

Python Version (if relevant)

No response

➕ Additional context

No response

[jcchr]

The error in logs from the training is as follows:

        size mismatch for model.decoder.anchors: copying a param with shape torch.Size([1, 336, 4]) from checkpoint, the shape in current model is torch.Size([1, 21504, 4]).
        size mismatch for model.decoder.valid_mask: copying a param with shape torch.Size([1, 336, 1]) from checkpoint, the shape in current model is torch.Size([1, 21504, 1]).
Runnning training trial with bs = 8 ...
Traceback (most recent call last):
  File "/interactive_ai/workflows/train/trainer/scripts/run.py", line 75, in <module>
    execute(work_dir=work_dir)
  File "/interactive_ai/workflows/train/trainer/scripts/utils.py", line 38, in _wrapped
    outputs = func(*args, **kwargs)
  File "/interactive_ai/workflows/train/trainer/scripts/run.py", line 34, in execute
    train(
  File "/interactive_ai/workflows/train/trainer/scripts/utils.py", line 38, in _wrapped
    outputs = func(*args, **kwargs)
  File "/interactive_ai/workflows/train/trainer/scripts/train.py", line 50, in train
    engine.train(**train_kwargs)
  File "/interactive_ai/workflows/train/trainer/.venv/lib/python3.10/site-packages/otx/backend/native/engine.py", line 236, in train
    adapt_batch_size(engine=self, **locals(), not_increase=(adaptive_bs != "Full"))
  File "/interactive_ai/workflows/train/trainer/.venv/lib/python3.10/site-packages/otx/backend/native/tools/adaptive_bs/runner.py", line 63, in adapt_batch_size
    new_batch_size = bs_search_algo.auto_decrease_batch_size()
  File "/interactive_ai/workflows/train/trainer/.venv/lib/python3.10/site-packages/otx/backend/native/tools/adaptive_bs/algorithm.py", line 101, in auto_decrease_batch_size
    oom, max_memory_reserved = self._try_batch_size(current_bs)
  File "/interactive_ai/workflows/train/trainer/.venv/lib/python3.10/site-packages/otx/backend/native/tools/adaptive_bs/algorithm.py", line 68, in _try_batch_size
    raise RuntimeError(msg)
RuntimeError: There is no output from the trial for adaptive batch size.```

Please check what is its reason and how (and where) it should be resolved.

[olkham]

side question, can the --install-telemetry-stack flag be used when upgrading to install it if it wasn't installed initially?

[piotrgrubicki]

@olkham, yes, it can.