Unable to read first n rows from parquet file.

[san089]

df = pd.read_parquet(path= 'filepath', nrows = 10)

Problem description

I have a parquet file and I want to read first n rows from the file into a pandas data frame. I did not find any way to do this in the documentation. I tried the 'nrows' and 'skiprows' parameter, but it did not work with the read_parquet() method. Do let me know if there is any way to achieve it that is not mentioned in the documentation.

Alternatively, I can read the complete parquet file and filter the first n rows, but that will require more computations which I want to avoid.

[TomAugspurger]

Does pyarrow or fastparquet allow reading a subset of the file?

[san089]

Unfortunately, it does not.

[TomAugspurger]

Perhaps raise issues upstream to see if there's interest in support this feature. Once the engines support it we'll be able to add a keyword.

…

On Mon, Dec 31, 2018 at 12:24 PM san089 ***@***.***> wrote: Unfortunately, it does not. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#24511 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIjpvUFpJzn3r8Bl7jxElDlmMR0x8ks5u-lZkgaJpZM4ZlKRb> .

[jreback]

closing as out of scope; we would need the engines to do this. Note that you might be able to simply filter by row groups.

[AlJohri]

Is there any chance we can define a flag like num_chunks? It takes N num_chunks from each column and limits to the column with the least amount of rows. Since I think 1 chunk from each column may result in a different number of rows depending on the data type.

Basically, it would be great to have a way to read just a small portion of a parquet file for local development.

[TomAugspurger]

You would need to raise that with fastparquet or pyarrow. I don't whether or not the parquet format supports that.

…

On Tue, Aug 27, 2019 at 9:14 AM Al Johri ***@***.***> wrote: Is there any chance we can define a flag like num_chunks? It takes N num_chunks from each column and limits to the column with the least amount of rows. Since I think 1 chunk from each column may result in a different number of rows depending on the data type. Basically, it would be great to have a way to read just a small portion of a parquet file for local development. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#24511?email_source=notifications&email_token=AAKAOIWAXK3MLDNVRFFH753QGUZE7A5CNFSM4GMUURN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5H4ENQ#issuecomment-525320758>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIU62NHQ7NO6A7U22ODQGUZE7ANCNFSM4GMUURNQ> .

[fny]

I want to reopen this issue. PyArrow now supports many of the required operations to make nrows possible and even usecols.