File saving error due to parallel file opening.

[jyhong836]

Bug description

When running benchmarks.hotpot_qa.adal_exp.train_agent_rag, the script may fail to save to checkpoint. I guess this is because of the use of multiple parallel threads.

What version are you seeing the problem on?

Github commit version: b43a866

How to reproduce the bug

python -m benchmarks.hotpot_qa.adal_exp.train_agent_rag

Error messages and logs

Loading Data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1319.52it/s]
Predicting: step(9): 0.0 across 3 samples, Max potential: 0.0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 316.29it/s]
2025-06-05 09:33:07 - [trainer.py:2230:_text_grad_constraint_propose_step] - Fail minibatch check, try next proposal: True, 0.0 <= 0.6666666666666666                                 | 0/3 [00:00<?, ?it/s]
Proposing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [02:41<00:00, 32.24s/it]
No proposal can improve the subset and full set, and val set██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [02:41<00:00, 18.09s/it]
Saving checkpoint to chkpts/hotpot_qa/adal_exp/constrained_max_steps_12_b36c8_run_1.json
Training Step: 9:  32%|███████████████████████████████████████████████                                                                                                    | 8/25 [37:57<1:20:40, 284.74s/it]
Epoch:   0%|                                                                                                                                                                          | 0/1 [37:57<?, ?it/s]
Traceback (most recent call last):
  File "/ssd2/junyuan/AdalFlow/adalflow/adalflow/utils/file_io.py", line 26, in save_json
OSError: [Errno 24] Too many open files: 'chkpts/hotpot_qa/adal_exp/constrained_max_steps_12_b36c8_run_1.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/ssd2/junyuan/AdalFlow/benchmarks/hotpot_qa/adal_exp/train_agent_rag.py", line 225, in <module>
  File "/ssd2/junyuan/AdalFlow/benchmarks/hotpot_qa/adal_exp/train_agent_rag.py", line 171, in train
  File "/ssd2/junyuan/AdalFlow/adalflow/adalflow/optim/trainer/trainer.py", line 668, in fit
  File "/ssd2/junyuan/AdalFlow/adalflow/adalflow/optim/trainer/trainer.py", line 644, in run_text_optimizers
  File "/ssd2/junyuan/AdalFlow/adalflow/adalflow/optim/trainer/trainer.py", line 2385, in _fit_text_grad_constraint
  File "/ssd2/junyuan/AdalFlow/adalflow/adalflow/optim/trainer/trainer.py", line 2324, in _text_grad_constraint_propose_step
  File "/ssd2/junyuan/AdalFlow/adalflow/adalflow/utils/file_io.py", line 30, in save_json
OSError: Error saving object to JSON file chkpts/hotpot_qa/adal_exp/constrained_max_steps_12_b36c8_run_1.json: [Errno 24] Too many open files: 'chkpts/hotpot_qa/adal_exp/constrained_max_steps_12_b36c8_run_1.json'

Environment

• OS: Linux (Ubuntu)

More info

No response

[jyhong836]

I think this is a general issue for the whole package. Adalflow does not handle any parallel file writing.

1. How parallel call happens? For example, log_call will be called when generator finishes a generation. log_call will call append_to_jsonl. And in training and prediction, prallel execution are performed, resulting in such glitch.
2. Another serious issue is that append_to_jsonl or any jsonl file writer does not lock the file, resulting in dead-lock in file-reading.

[riyaz-maker]

Solution Proposal: Making File Operations Safe for Parallel Runs

Hi,
After looking into the traceback and the code, I agree that the crashes during parallel runs are due to how we're handling file writes. Here’s a proposal to fix it.

The Problem in Simple Terms

Right now, when Adalflow runs tasks in parallel (like during training or logging), multiple processes can try to write to the same file (like a checkpoint or a log) at the exact same time.
The best way to solve this is to introduce a locking mechanism.
If another process already has the lock, any other processes will simply wait for their turn.
Release the Lock: As soon as the first process is done writing, it releases the lock.
This prevents crashes and ensuring log files and checkpoints are never corrupted. The rest of the application remains fully parallel and fast; only the brief moment of writing to a single shared file is serialized.

Implementation Plan

I've added the standard filelock library to the project's dependencies to handle the locking.

All relevant functions in adalflow/utils/file_io.py have been updated with a with FileLock(...) statement. This change is encapsulated within the utility functions, so no other part of the codebase needs to be modified.

The test confirms that the final file is complete and uncorrupted, proving the solution is effective.

Could you please check if it is working on your end:
https://github.com/riyaz-maker/AdalFlow/tree/handle-parallel-file-writing

[riyaz-maker]

let me know what you think @liyin2015

[devdattatalele]

Hi @riyaz-maker and @liyin2015, thanks for the solid analysis and FileLock-based solution. I have built on your approach and submitted PR #440 with architectural enhancements and maintainability while keeping full backward compatibility. Please review the PR here: #440.