Adding breakpoint in random_images function crashes pdb

[cgebbe]

🐛 Bug

I'd like to debug the random_images function taken from the first example. However, when adding a breakpoint() line in that function, python crashes.

Environment

• PyTorch Version (e.g., 1.0): 2.3.0
• OS (e.g., Linux): ubuntu 20.04
• How you installed PyTorch (conda, pip, source): uv pip
• Build command you used (if compiling from source): None
• Python version: 3.11
• CUDA/cuDNN version: cuda 12.1

Additional context

Traceback

Workers are ready ! Starting data processing...
                                                                                      > /path/to/repo/subdir/convert_zipfiles_to_litdata.py(43)random_images()
-> "index": index,  # int data type
(Pdb) 
Traceback (most recent call last):
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/data_processor.py", line 626, in _handle_data_chunk_recipe
    item_data_or_generator = self.data_recipe.prepare_item(current_item)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/functions.py", line 148, in _prepare_item
    return self._fn(item_metadata)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/repo/subdir/convert_zipfiles_to_litdata.py", line 43, in random_images
    "index": index,  # int data type
             ^^^^^
  File "/usr/lib/python3.11/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/bdb.py", line 115, in dispatch_line
    if self.quitting: raise BdbQuit
                      ^^^^^^^^^^^^^
bdb.BdbQuit

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/data_processor.py", line 423, in run
    self._loop()
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/data_processor.py", line 472, in _loop
    self._handle_data_chunk_recipe(index)
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/data_processor.py", line 638, in _handle_data_chunk_recipe
    raise RuntimeError(f"Failed processing {self.items[index]}") from e
RuntimeError: Failed processing 0

Worker 0 is done.
Traceback (most recent call last):
  File "/path/to/repo/subdir/convert_zipfiles_to_litdata.py", line 107, in <module>
    main()
  File "/path/to/repo/subdir/convert_zipfiles_to_litdata.py", line 31, in main
    convert_zipfile_to_litdata(s3_filepath=VAL_PATH, outdir=Path.home() / "tmp/tica")
  File "/path/to/repo/subdir/convert_zipfiles_to_litdata.py", line 67, in convert_zipfile_to_litdata
    litdata.optimize(
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/functions.py", line 358, in optimize
    data_processor.run(
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/data_processor.py", line 981, in run
    self._exit_on_error(error)
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/data_processor.py", line 1040, in _exit_on_error
    raise RuntimeError(f"We found the following error {error}.")
RuntimeError: We found the following error Traceback (most recent call last):
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/data_processor.py", line 626, in _handle_data_chunk_recipe
    item_data_or_generator = self.data_recipe.prepare_item(current_item)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/functions.py", line 148, in _prepare_item
    return self._fn(item_metadata)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/repo/subdir/convert_zipfiles_to_litdata.py", line 43, in random_images
    "index": index,  # int data type
             ^^^^^
  File "/usr/lib/python3.11/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/bdb.py", line 115, in dispatch_line
    if self.quitting: raise BdbQuit
                      ^^^^^^^^^^^^^
bdb.BdbQuit

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/data_processor.py", line 423, in run
    self._loop()
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/data_processor.py", line 472, in _loop
    self._handle_data_chunk_recipe(index)
  File "/path/to/repo/.venv/lib/python3.11/site-packages/litdata/processing/data_processor.py", line 638, in _handle_data_chunk_recipe
    raise RuntimeError(f"Failed processing {self.items[index]}") from e
RuntimeError: Failed processing 0

[github-actions]

Hi! thanks for your contribution!, great first issue!

[tchaton]

Hey @cgebbe Maybe we could try the breakpoint from lightning

https://github.com/Lightning-AI/pytorch-lightning/blob/896c2a656ad2db3278ec11520aed04e378f4462b/src/lightning/app/pdb/pdb.py#L29

[cgebbe]

@tchaton : Thanks, I'll give it a try next time, wasn't aware of that!

[tchaton]

@cgebbe If you try it, can you let me know. Maybe worth to copy the code from lightning to litdata if it works.