1BRC in Julia

[vnegi10]

My strategy so far:

• Mmap file
• Split into chunks
• Read each chunk in parallel (12 threads) into a DataFrame
• Use DataFrame groupby and combine results

Note that I have used external dependencies as well, so it's not completely in line with the rules of the original challenge.

Best time is 89.459 s on a desktop equipped with Ryzen 9 5900X and 32 GB RAM. Code and benchmark results can be found here.

I am pretty sure there's a lot more room for improvement.

[pddshk]

It would be easier to make it via CSV as CSV.File has an option

ntasks::Integer=Threads.nthreads(): [not applicable to CSV.Rows] for multithreaded parsed files, this controls the number of tasks spawned to read a file in concurrent chunks; defaults to the # of threads Julia was started with (i.e. JULIA_NUM_THREADS environment variable or julia -t N); setting ntasks=1 will avoid any calls to Threads.@spawn and just read the file serially on the main thread; a single thread will also be used for smaller files by default (< 5_000 cells)