[WIP] Initial commit of bulk reading into numpy arrays

[snorfalorpagus]

This is a first attempt at providing a bulk load of data into NumPy arrays, as suggested in #469.

It's a work in progress. I'm sure it's not perfectly optimized and there are some edge cases that aren't handle yet (e.g. NULL values in int columns), date fields.

Does this provide the right kind of interface for geopandas @jorisvandenbossche? Do you have any opinions @sgillies?

[coveralls]

Coverage decreased (-0.3%) to 84.065% when pulling dc4ab3d on snorfalorpagus:vectorized into 1938c44 on Toblerity:master.

[coveralls]

Coverage decreased (-0.2%) to 83.389% when pulling 37400b7 on snorfalorpagus:vectorized into 8867328 on Toblerity:master.

[jorisvandenbossche]

@snorfalorpagus Thanks a lot for exploring this!
To say whether this would output something useful for geopandas, I would first need to take a closer look (but in principle returning numpy arrays is fine). You now create one array per dtype? That would be perfect for geopandas, although from a more general user point of view, a 1D array per field might make more sense.

With regard to the TODO "best way to return geometries for shapely?", I would say either WKB or WKT, depending on which is the fastest/cheapest to create and convert afterwards (which I assume will be WKB?)

[snorfalorpagus]

• Support WKT or WKB
• Support for all data types available in Fiona already
• Support for ignoring fields
• Improved iterating over features

[snorfalorpagus]

It looks like GDAL 1.x is having problem with the datetime field? It looks like we don't actually test this in the regular reader/writer.

[snowman2]

Thoughts on using a structured array for data_properties?

[brendan-ward]

For those finding this issue later and wanting to know how support for vectorized reading from OGR/GDAL evolved, see pyogrio.

Thanks again @snorfalorpagus for the initial work on this!