csv_reader with limited number of columns should should completely disregard the unused fields

[cordeiro]

xref #6710

I have a CSV whose lines may have 11 or 18 fields. I only need to read the first 6 fields, so I use "usecols=range(6)". Even with the limited number of columns, I get the exception:

ValueError: Expected 11 fields in line 776483, saw 18

The csv_reader should completely disregard the unused fields.

Small test case:

csv = '19,29,39\n'*2 + '10,20,30,40\n'
df = pd.read_csv(io.StringIO(csv), engine='python', header=None, usecols=list(range(3)))

It also affects the C engine.

Discussed at the users mailing list at https://groups.google.com/d/topic/pydata/vjhFpHtgnvw/discussion

[jreback]

this is like #6710. Usecols will only select from the valid columns. It is inferring that you have 3, so it is a bit contradictory here.

Soln is to use names (or names=range(3)) to just get those first 3

In [21]: pd.read_csv(io.StringIO(csv), header=None, names=range(4))
Out[21]: 
    0   1   2   3
0  19  29  39 NaN
1  19  29  39 NaN
2  10  20  30  40

not sure if this is a bug or not; i'll mark it same as the other one.

[cordeiro]

I don't agree that they are the same bug.

Bug #6710 is about the ability to infer whether the next lines will have more fields or not.

In this case, all rows have all fields of interest ( [0:6] ). The remaining of the line will not be used anyway and should not be considered at all.

[jreback]

@cordeiro I said they are alike, that's why its a separate issue :)

[cordeiro]

Oups. :)

[jreback]

np. if you do have interested in looking at this would be appreciated.

[dxe4]

sounds easy ill try to make a pr today for this