Inference Engine async writes #574
Conversation
| original_filepath = Path(output_filepath) | ||
| return str(original_filepath.parent / "scratch" / original_filepath.name) | ||
|
|
||
| async def _save_conversation_async( |
There was a problem hiding this comment.
Is there a lock on the file writer? If so, at what level is it defined?
There was a problem hiding this comment.
Since this comment I've removed the _save_conversation_async method as I found it's actually slower for writes (mostly benefiting reads).
That said, with my new code we get strong guarantees about no concurrent access from asyncio itself. In asyncio code, a context switch can only happen when you hit an await block. As I'm now doing my write outside of an await, there can only be one concurrent write at a time.
This is distinct from threads, where context switches are far less predictable.
| # Write what we have so far to our scratch directory. | ||
| self._save_conversation( | ||
| new_conversation, | ||
| self._get_scratch_filepath(generation_config.output_filepath), |
There was a problem hiding this comment.
this is saving conversations into scratch directory, not to the originally configured dir. Wouldn't it be a problem ?
There was a problem hiding this comment.
This is a fair point. I've updated my approach now to only save to a scratch directory for async code. Sync paths will simply append to the target file. Updated!
This PR is based on #558
This PR updates how writes are done for inference.
/scratchdirectory containing a file with the same name. There is no guarantee of order on the values written to this file (this truly only matters for the InferenceEngines that leverage parallelism).Non-parallel engines will write to disk in-line (blocking). I've benchmarked this for medium sized files (100s of MB): appending a line of text to these files takes on average 1.788e-05 seconds