Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW] from_geopandas converts GeoPandas GeoSeries objects into cuspatial GeoArrow form. #300

Merged
merged 111 commits into from
May 21, 2021
Merged
Show file tree
Hide file tree
Changes from 108 commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
3f67569
First new geopandas io files.
thomcom Sep 11, 2020
31b6b6f
Pass one test!
thomcom Sep 16, 2020
848cefa
Pass tests for multipoint and linestring, working on multilinestring.
thomcom Sep 16, 2020
c39aa19
Add multilinestring too!
thomcom Sep 16, 2020
a292717
Added polygon and multipolygon.
thomcom Sep 17, 2020
8319c88
Refactoring. Broke tests. Create GeoSeries class.
thomcom Sep 18, 2020
04c06ec
Refactoring to handle much more complicated geometries.
thomcom Sep 22, 2020
cece7e6
Properly serializing the offsets of mixed dataframes. Can't remember…
thomcom Sep 23, 2020
de85fc0
Woo, passing all tests. Still need to validate against shapely reads.
thomcom Sep 23, 2020
ea29e53
Starting to write to_geopandas inverter.
thomcom Sep 24, 2020
c248dbd
Merge branch 'branch-0.16' of github.com:rapidsai/cuspatial into fea-…
thomcom Sep 24, 2020
42902c1
Refactoring.
thomcom Oct 5, 2020
532b6da
Merge branch 'branch-0.16' into fea-from-geopandas
thomcom Oct 5, 2020
6075ccf
Pass tests and clean up formatting.
thomcom Oct 5, 2020
07f4c21
CHANGELOG
thomcom Oct 5, 2020
25d1945
Improve documentation step 1
thomcom Oct 5, 2020
aaeca09
Resolve various formatting issues.
thomcom Oct 5, 2020
6e76e6c
Major performance increases in the reading of GeoSeries.
thomcom Oct 5, 2020
4d8d8a8
Remove unneeded enumerator.
thomcom Oct 5, 2020
73dafc0
Formatting
thomcom Oct 5, 2020
e23b9d0
Fix crucial bug in rings production.
thomcom Oct 5, 2020
3ce0e76
Lots of indexing code that needs revision
thomcom Oct 8, 2020
b3c489e
Get all tests passing except cuMultiPolygon
thomcom Oct 13, 2020
de8ff02
Passing all tests.
thomcom Oct 15, 2020
7806df9
Pass all tests!
thomcom Oct 17, 2020
c337b32
Update CHANGELOG.md
raydouglass Oct 21, 2020
9aeebc8
Update CHANGELOG.md
raydouglass Dec 10, 2020
7f13b5e
Update python/cuspatial/cuspatial/geometry/geoseries.py
thomcom Jan 14, 2021
c8f232a
Merge in 0.18
thomcom Jan 15, 2021
2ca8926
ix an offset computation
thomcom Jan 15, 2021
6ea1be5
Merge branch 'fea-from-geopandas' of github.com:thomcom/cuspatial int…
thomcom Jan 15, 2021
80d043f
Merge remote-tracking branch 'upstream/branch-0.18' into fea-from-geo…
thomcom Jan 19, 2021
5cd2a57
Resolve deprecated language and args about interleaving
thomcom Jan 19, 2021
121e29d
Add NotImplemented de_interleave fn
thomcom Jan 19, 2021
cfecdd5
Handle more review comments.
thomcom Jan 19, 2021
f711ef4
Create new test test_mixed_dataframe, fixing reviewer comment.
thomcom Jan 22, 2021
9f47f2c
Update CHANGELOG.md
thomcom Jan 26, 2021
a0180c4
Improve copying of GeoSeries
thomcom Jan 26, 2021
69eb4bc
Merge branch 'fea-from-geopandas' of github.com:thomcom/cuspatial int…
thomcom Jan 26, 2021
ecbd79a
Create geodataframe and refactor to support it.
thomcom Jan 26, 2021
e811a5d
Improve GeoDataFrame support and test passing./
thomcom Jan 27, 2021
c39d6e4
Fix important bug involving double speed iteration.
thomcom Jan 27, 2021
cced5e7
Pass all tests.
thomcom Jan 28, 2021
83bb2fb
Resolve style issues.
thomcom Feb 2, 2021
1d250ed
Also do black reformatting duh.
thomcom Feb 2, 2021
bf547de
I have a dependency difference that breaks inverts two test successes.
thomcom Feb 2, 2021
55882f8
Merge remote-tracking branch 'upstream/branch-0.16' into fea-from-geo…
thomcom Feb 2, 2021
ae16f81
Merge remote-tracking branch 'upstream/branch-0.17' into fea-from-geo…
thomcom Feb 2, 2021
b1c0723
Missing one dependency in conda recipe.
thomcom Feb 2, 2021
5d3a2bd
Change np arrays to cp arrays.
thomcom Feb 3, 2021
a486ce0
Add a list comprehension to geodataframe.py
thomcom Feb 4, 2021
38c4ae5
Making more list comprehensions for @kkraus
thomcom Feb 4, 2021
e0796b5
Update python/cuspatial/cuspatial/tests/test_from_geopandas.py
thomcom Feb 11, 2021
ca361a3
Create new x and y properties, resolve a few questions in code review.
thomcom Feb 22, 2021
bebf0db
Merge branch 'fea-from-geopandas' of github.com:thomcom/cuspatial int…
thomcom Feb 22, 2021
dfdb9e7
Remove de_interleave function and replace with .x and .y acessors. Cl…
thomcom Feb 23, 2021
fe017d7
Eliminate use of pd.Series for basic iteration. Add docstrings for gp…
thomcom Feb 25, 2021
f01aef4
Improve docstrings, change all repr to fstring,
thomcom Feb 25, 2021
0dd8c5d
formatting
thomcom Feb 25, 2021
ff7f4ff
Modified some properties to be protected and got rid of has_z
thomcom Feb 25, 2021
9c4d822
Tons of docs updates.
thomcom Mar 1, 2021
2d64750
Remove a breakpoint
thomcom Mar 1, 2021
332ca94
Black
thomcom Mar 1, 2021
33112a3
Refactor to use GeoSeries and GeoColumn.
thomcom Mar 8, 2021
0b8f83a
Get rid of gpu buffers during conversion. Improve GeoSeries/GeoColumn…
thomcom Mar 9, 2021
ec88ba0
Formatting.
thomcom Mar 9, 2021
1ae066c
Updating to branch-0.19
thomcom Mar 10, 2021
8353a10
Meta.yaml conflict.
thomcom Mar 10, 2021
e1bc8ef
Move GeoColumn to its own file.
thomcom Mar 15, 2021
f63168c
Formatting.
thomcom Mar 15, 2021
e3b138a
Mostly done.
thomcom Mar 15, 2021
2103b39
Big refactor to separate GeoArrowBuffers from GeoColumn. Still need t…
thomcom Mar 23, 2021
bd46b87
Clean up init.py and dataframe, add docs to geopandas_adapter.
thomcom Mar 24, 2021
1832937
Docs updates mostly.
thomcom Mar 25, 2021
10c6a84
Formatting
thomcom Mar 25, 2021
31cb04b
Handle arrow buffers metadata and create GeoArrowBuffers tests for po…
thomcom Mar 29, 2021
268da6d
Add test for GeoArrowBuffers lines and multilines.
thomcom Mar 29, 2021
40f612e
Full testing for GeoArrowBuffers created GeoSeries
thomcom Mar 30, 2021
40e5498
Merge branch 'branch-0.19' into fea-from-geopandas
thomcom Mar 31, 2021
2e20e87
Remove use of .all and update geopandas dependency in conda envs.
thomcom Apr 1, 2021
8a7fc20
Move gpu data to host before serialization.
thomcom Apr 2, 2021
01600e9
Updating docs.
thomcom Apr 5, 2021
e3d4c95
Various docs cleanup.
thomcom Apr 8, 2021
cb38b0c
Correct use of data_location with _serialize
thomcom Apr 9, 2021
43490d6
Refactoring cleanup.
thomcom Apr 12, 2021
d38ec10
Eliminate hasattr.
thomcom Apr 12, 2021
18a19d7
Responding to comments.
thomcom Apr 13, 2021
077d697
Speed up list extension and privatize geocolum. Nothing left to do bu…
thomcom Apr 14, 2021
3f087c9
Formatting.
thomcom Apr 14, 2021
0e6a249
Detailed description of GeoArrow format.
thomcom Apr 14, 2021
5bd60ad
conda env conflict resolved.
thomcom Apr 14, 2021
9328d09
Final updates to documentation?
thomcom Apr 15, 2021
bb61d2e
Final review comment addressed.
thomcom Apr 15, 2021
17d955a
Black one more file.
thomcom Apr 15, 2021
7c7a998
Remove cached_property since it isn't supported in python 3.7
thomcom Apr 16, 2021
ee8e3b9
A few review comments resolved.
thomcom Apr 27, 2021
cca09b1
Merge branch 'branch-0.20' into fea-from-geopandas
thomcom Apr 27, 2021
70c9cd4
Handle many more Keith Kraus review comments.
thomcom Apr 28, 2021
d97f83e
remove print hurr
thomcom Apr 29, 2021
2f63818
Black formatting.
thomcom Apr 29, 2021
4c6c5b7
Rename GpuClasses to ShapelySerializer classes.
thomcom May 10, 2021
4479f44
Style
thomcom May 11, 2021
37048a1
Move nasty multiline/multipolygon iterations to gpu.
thomcom May 11, 2021
ccc05ef
Update python/cuspatial/cuspatial/geometry/geodataframe.py
thomcom May 11, 2021
ee8ed03
Use np as a like data source.
thomcom May 12, 2021
9b3a9d1
Correct a committed typo.
thomcom May 12, 2021
ba954f6
Use values for inline mlines length calculation to avoid unneeded ind…
thomcom May 12, 2021
909abe9
Merge branch 'branch-0.20' into fea-from-geopandas
May 13, 2021
d33395b
Merge branch 'branch-21.06' into fea-from-geopandas
thomcom May 21, 2021
ee13fc9
Do Keith's final suggestion.
thomcom May 21, 2021
9f9048c
Merge branch 'fea-from-geopandas' of github.com:thomcom/cuspatial int…
thomcom May 21, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ Please see https://github.com/rapidsai/cuspatial/releases/tag/v0.20.0a for the l
- PR #320 Fix quadtree construction bug: zero out `device_uvector` before `scatter`
- PR #328 Fix point in polygon test for cudf::gather breaking change

# cuSpatial 0.16.0 (Date TBD)
# cuSpatial 0.16.0 (21 Oct 2020)

## New Features
- PR #290 Add Java bindings and a cuSpatial JAR package for Java users.
Expand Down
2 changes: 1 addition & 1 deletion conda/environments/cuspatial_dev_cuda11.0.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dependencies:
- cudf=0.20.*
- cudatoolkit=11.0
- gdal>=3.0.2
- geopandas=0.7.0
- geopandas>=0.9.0
ajschmidt8 marked this conversation as resolved.
Show resolved Hide resolved
thomcom marked this conversation as resolved.
Show resolved Hide resolved
- cmake>=3.18
- cython>=0.29,<0.30
- gtest
Expand Down
2 changes: 1 addition & 1 deletion conda/environments/cuspatial_dev_cuda11.1.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dependencies:
- cudf=0.20.*
- cudatoolkit=11.1
- gdal>=3.0.2
- geopandas=0.7.0
- geopandas>=0.9.0
- cmake>=3.14
- cython>=0.29,<0.30
- gtest
Expand Down
2 changes: 1 addition & 1 deletion conda/environments/cuspatial_dev_cuda11.2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dependencies:
- cudf=0.20.*
- cudatoolkit=11.2
- gdal>=3.0.2
- geopandas=0.7.0
- geopandas>=0.9.0
- cmake>=3.14
- cython>=0.29,<0.30
- gtest
Expand Down
1 change: 1 addition & 0 deletions conda/recipes/cuspatial/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ requirements:
- cudf {{ minor_version }}.*
- rmm {{ minor_version }}.*
- gdal >=3.2.0,<3.3.0a0
- geopandas >=0.9.0,<0.10.0a0

test:
commands:
Expand Down
84 changes: 53 additions & 31 deletions docs/source/api.rst
Original file line number Diff line number Diff line change
@@ -1,62 +1,84 @@
~~~~~~~~~~~~~~~~~~~~~~~
cuSpatial API Reference
~~~~~~~~~~~~~~~~~~~~~~~
GeoPandas Compatibility
-----------------------

We support any geometry format supported by `GeoPandas`. Load geometry information from a `GeoPandas.GeoSeries` or `GeoPandas.GeoDataFrame`.

GIS
---
.. currentmodule:: cuspatial
>>> gpdf = geopandas.read_file('arbitrary.txt')
cugpdf = cuspatial.from_geopandas(gpdf)

.. autofunction:: cuspatial.directed_hausdorff_distance
.. autofunction:: cuspatial.haversine_distance
.. autofunction:: cuspatial.lonlat_to_cartesian
.. autofunction:: cuspatial.point_in_polygon
.. autofunction:: cuspatial.polygon_bounding_boxes
.. autofunction:: cuspatial.polyline_bounding_boxes
or

>>> cugpdf = cuspatial.GeoDataFrame(gpdf)

Indexing
--------
.. currentmodule:: cuspatial

.. autofunction:: cuspatial.quadtree_on_points
.. autoclass:: cuspatial.GeoDataFrame
:members:
:show-inheritance:
.. autoclass:: cuspatial.GeoSeries
:members:
:show-inheritance:
.. autoclass:: cuspatial.geometry.geocolumn.GeoColumn
:members:
:show-inheritance:
.. autoclass:: cuspatial.GeoArrowBuffers
:members:
:show-inheritance:


Spatial Indexing
--------

Spatial indexing functions provide blisteringly-fast on-GPU point-in-polygon
operations.

Interpolation
-------------
.. currentmodule:: cuspatial

.. autoclass:: CubicSpline
.. automethod:: CubicSpline.__init__
.. automethod:: CubicSpline.__call__

.. autofunction:: cuspatial.quadtree_point_in_polygon
.. autofunction:: cuspatial.quadtree_point_to_nearest_polyline
.. autofunction:: cuspatial.point_in_polygon
.. autofunction:: cuspatial.polygon_bounding_boxes
.. autofunction:: cuspatial.polyline_bounding_boxes
.. autofunction:: cuspatial.quadtree_on_points
.. autofunction:: cuspatial.join_quadtree_and_bounding_boxes
.. autofunction:: cuspatial.points_in_spatial_window

Spatial Querying
----------------
.. currentmodule:: cuspatial

.. autofunction:: cuspatial.points_in_spatial_window
GIS
---

Two GIS functions make it easier to compute distances with geographic coordinates.

Spatial Joining
----------------
.. currentmodule:: cuspatial

.. autofunction:: cuspatial.join_quadtree_and_bounding_boxes
.. autofunction:: cuspatial.quadtree_point_in_polygon
.. autofunction:: cuspatial.quadtree_point_to_nearest_polyline
.. autofunction:: cuspatial.haversine_distance
.. autofunction:: cuspatial.lonlat_to_cartesian


Trajectory
----------

Trajectory functions make it easy to identify and group trajectories from point data.

.. currentmodule:: cuspatial

.. autofunction:: cuspatial.derive_trajectories
.. autofunction:: cuspatial.trajectory_bounding_boxes
.. autofunction:: cuspatial.trajectory_distances_and_speeds
.. autofunction:: cuspatial.directed_hausdorff_distance
.. autofunction:: cuspatial.trajectory_bounding_boxes
.. autoclass:: CubicSpline
.. automethod:: CubicSpline.__init__
.. automethod:: CubicSpline.__call__


IO
--

cuSpatial offers native GPU-accelerated shapefile reading. In addition, any host-side GeoPandas DataFrame can be copied into GPU memory for use with cuSpatial
algorithms.

.. currentmodule:: cuspatial

.. autofunction:: cuspatial.read_polygon_shapefile
.. autofunction:: cuspatial.from_geopandas

3 changes: 2 additions & 1 deletion docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
#
import os
import sys
import cudf

sys.path.insert(0, os.path.abspath("../.."))

Expand All @@ -35,6 +34,7 @@
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
"sphinx.ext.autosectionlabel",
"sphinx.ext.intersphinx",
"sphinx.ext.autodoc",
"sphinx.ext.autosummary",
Expand Down Expand Up @@ -197,5 +197,6 @@
numpydoc_class_members_toctree = False
nbsphinx_allow_errors = True


def setup(app):
app.add_stylesheet("params.css")
26 changes: 25 additions & 1 deletion docs/source/index.rst
Original file line number Diff line number Diff line change
@@ -1,12 +1,36 @@
Welcome to cuSpatial's documentation!
=====================================

cuSpatial provides functionality for fast GPU-based spatial index and join
functions via point-in-polygon, a pathing library for trajectory identification
and reconstruction, and accelerated GIS functions like haversine distance and
grid projection. The icing-on-the-cake of cuSpatial is that we integrate neatly
with `GeoPandas` and RAPIDS `cudf`.

This enables you to take advantage of all of the features of `GeoPandas`, plus
blisteringly fast GPU acceleration for pandas-like functionality in `cudf` and
spatial functions that perform quickly on millions and tens of millions of
geometries.

GeoArrow
--------

cuSpatial proposes a new GeoArrow format from the fruit of discussions with the GeoPandas team. GeoArrow is a packed columnar data format for the six fundamental geometry types: Point, MultiPoint, Lines, MultiLines, Polygons, and MultiPolygons. MultiGeometry is a possibility that may be implemented in the future. GeoArrow uses packed coordinate and offset columns to define objects, which enables very-fast copy between CPU, GPU, and NIC.

Any data source that is loaded into cuSpatial via :func:`cuspatial.from_geopandas` can then take advantage of `cudf`'s GPU-accelerated Arrow I/O routines.

Read more about GeoArrow format in :func:`GeoArrowBuffers<cuspatial.GeoArrowBuffers>`

cuSpatial API Reference
~~~~~~~~~~~~~~~~~~~~~~~

.. toctree::
:maxdepth: 2
:caption: Contents:

api.rst


~~~~~~~~~~~~~~~~~~~~~~~

Indices and tables
==================
Expand Down
4 changes: 4 additions & 0 deletions python/cuspatial/cuspatial/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,11 @@
trajectory_bounding_boxes,
trajectory_distances_and_speeds,
)
from .geometry.geoseries import GeoSeries
from .geometry.geodataframe import GeoDataFrame
from .geometry.geoarrowbuffers import GeoArrowBuffers
from .io.shapefile import read_polygon_shapefile
from .io.geopandas import from_geopandas

__version__ = get_versions()["version"]
del get_versions
Empty file.
Loading