Feature/parallel doc loading

[ahmed-ashraff]

Feature/Parallel Document Loading

Description

This feature introduces parallel document loading to improve performance. By utilizing concurrency, the document loading process is now significantly faster.

Key Changes

• Implemented parallel processing for document loading.
• Refactored the loader logic to handle concurrency safely.
• Ensured synchronization to prevent race conditions.

Performance Improvement

• Achieved a 5.01x speedup compared to the previous sequential implementation on my D partition.

Testing

• Added unit tests to verify correct document loading behavior.
• Conducted performance benchmarks to validate the speedup.

Impact

This enhancement greatly improves efficiency, especially for large data directories.