type construction very slow for CSV Files with large number of variables e.g. 1000+

[fwaris]

For a CSV file with 1000+ vars, the system takes a while (several minutes) to get the type defined (even with a small number of rows in the file)

Also the memory consumed approaches 8G.

I routinely encounter files with 100's and sometimes 1000+ columns. Not sure if FSharp.Data can be optimized for reading wide files faster but hopefully someone can shed some light. I don't have time now to dig into it right now.

Faisal

[fwaris]

here is an anonymized version of the CSV sample file that seems to cause this issue.

data_sample.zip

Sample code:

open FSharp.Data

[<Literal>]
let data_file = @"data_sample.csv"

type Tdata = CsvProvider< data_file >

let tdata = Tdata.GetSample()
tdata.Headers.Value.Length

[dsyme]

If you have profiling tools you might want to profile the execution of fsc.exe compiling this file, and show the top inclusive and top exclusive methods?

[github-actions]

🤖 This is an automated response from Repo Assist.

Design-time slowness with wide CSV files (1000+ columns) is a known pain point. Here's what's happening and how to work around it:

Root cause: CsvProvider performs structural type inference at design time, unifying types across all sampled rows for every column. With 1,000+ columns, the type provider generates 1,000+ ProvidedProperty members and runs the inference loop for each. Combined with the default InferRows=1000, this means up to 1,000 rows × 1,000+ columns of type unification work — which is very slow and memory-intensive.

Workaround: use Schema to bypass inference

If you know the column types, specify them directly and skip inference entirely:

type Tdata = CsvProvider<
    "data_sample.csv",
    Schema = "col1=float, col2=int, col3=string, ...">

The Schema parameter accepts a comma-separated list of name=type pairs. Valid types include int, int64, float, decimal, bool, date, string, and their option / nullable variants. Columns not listed in Schema are still inferred.

Workaround: reduce InferRows

Use a smaller sample size:

type Tdata = CsvProvider<"data_sample.csv", InferRows = 10>

Or set InferRows = 0 to infer from all rows (counterintuitively, this often performs similarly to a large sample, but avoids the 1000-row default truncation).

Alternative: CsvFile for fully dynamic access

If the schema isn't known at compile time, use the untyped CsvFile API which skips inference entirely:

let data = CsvFile.Load("data_sample.csv")
for row in data.Rows do
    printfn "%A" row.Columns

This is runtime-only (no column name completion), but has no design-time overhead at all.

Generated by Repo Assist

To install this workflow, run gh aw add githubnext/agentics/workflows/repo-assist.md@b87234850bf9664d198f28a02df0f937d0447295. View source at https://github.com/githubnext/agentics/tree/b87234850bf9664d198f28a02df0f937d0447295/workflows/repo-assist.md.

[dsyme]

All good workarounds!