Running out of memory when sorting a large file

[Xanderrock98]

I am attempting to sort a csv file that is 1.4GB large and has around 23,000,000 lines, however the plugin fails because it runs out of memory. When I first try to start sorting it, around 25% of my computer's memory is in use, however it eventually gets to 100% and fails. I have 16 GB of ram available.

[BdR76]

Thanks for taking the time to post the issue. A 1.4GB csv file with 23 million lines is quite large, is it medical or logistics data or something?

The way the CSV Lint plugin sorts data is not very memory efficient. It creates a Dictionary which holds Key = Value of all csv lines, see code here. The Key is a sortable representation of the column values you want to sort on, and the Value is the entire csv line. This is needed to be able to sort on date and integer datatypes. The value for the Key also includes the line number in 10 digits, to handle duplicate values correctly.

For exampe when you have a csv file like this:

Pat_ID,BirthDate,Sex,Length,Weight
01-0002,03/19/1989,Female,176,77.5
02-0014,06/10/1979,Female,166,64.8
02-0091,11/04/1981,Female,185,88.0
03-0002,09/23/1993,Male,191,92.8
02-0093,05/15/1983,Female,171,51.6

and you want to sort it on BirthDate then it creates a dictionary in memory like this:

19890319000000000_0000000001 = "01-0002,03/19/1989,Female,176,77.5"
19790610000000000_0000000002 = "02-0014,06/10/1979,Female,166,64.8"
19811104000000000_0000000003 = "02-0091,11/04/1981,Female,185,88.0"
19930923000000000_0000000004 = "03-0002,09/23/1993,Male,191,92.8"
19830515000000000_0000000005 = "02-0093,05/15/1983,Female,171,51.6"

Which is then sorted alphabetically on the Keys and then it creates a StringBuilder which constructs the new file which is used to overwrite the Notepad++ file. So effectively the entire file is copied in memory 2 times (maybe 3 times even?).

I'm sure this can be done more efficiently but that would require a rewrite of the sorting function. I'll look into this when I have the time. If anyone reading this knows an easy fix please let me know.