feat: Geospatial Data Type and GIS Function Support for milvus #37417

tasty-gumi · 2024-11-04T10:48:23Z

Main Goals

Create and describe collections with geospatial fields, enabling both client and server to recognize and process geo fields.
Insert geospatial data as payload values in the insert binlog, and print the values for verification.
Load segments containing geospatial data into memory.
Ensure query outputs can display geospatial data.
Support filtering on GIS functions for geospatial columns.

Solution

Add Type: Modify the Milvus core by adding a Geospatial type in both the C++ and Go code layers, defining the Geospatial data structure and the corresponding interfaces.
Dependency Libraries: Introduce necessary geospatial data processing libraries. In the C++ source code, use Conan package management to include the GDAL library. In the Go source code, add the go-geom library to the go.mod file.
Protocol Interface: Revise the Milvus protocol to provide mechanisms for Geospatial message serialization and deserialization.
Data Pipeline: Facilitate interaction between the client and proxy using the WKT format for geospatial data. The proxy will convert all data into WKB format for downstream processing, providing column data interfaces, segment encapsulation, segment loading, payload writing, and cache block management.
Query Operators: Implement simple display and support for filter queries. Initially, focus on filtering based on spatial relationships for a single column of geospatial literal values, providing parsing and execution for query expressions.
Index Construction: Consider building an H3 index, utilizing the C interface provided by the H3 system.
Client Modification: Enable the client to handle user input for geospatial data and facilitate end-to-end testing.Check the modification in pymilvus.

delete incomplete H3 Index development and useless generated files.
fix conanfiles in milvus conan repo so that local can fetch the packages to build libraries

sre-ci-robot · 2024-11-04T10:48:34Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tasty-gumi
To complete the pull request process, please assign czs007 after the PR has been reviewed.
You can assign the PR to them by writing /assign @czs007 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2024-11-04T10:56:00Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T10:58:06Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-04T11:35:49Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-04T11:36:31Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

czs007 · 2024-11-04T12:12:13Z

rerun go-sdk

czs007 · 2024-11-04T12:12:24Z

rerun cpp-unit-test

mergify · 2024-11-04T12:52:00Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T13:10:21Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

czs007 · 2024-11-04T13:15:43Z

/run-cpu-e2e

mergify · 2024-11-04T13:19:53Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-04T13:21:14Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T14:09:21Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T14:43:21Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

internal/core/unittest/test_utils/DataGen.h

mergify · 2024-11-05T08:36:01Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-05T08:57:51Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-05T09:11:29Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-05T12:02:43Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-07T08:10:22Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-08T10:32:12Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-08T11:28:31Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

tasty-gumi · 2024-11-08T11:34:42Z

/run-cpu-e2e

mergify · 2024-11-08T12:29:49Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-08T13:51:04Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-08T14:26:42Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

czs007 · 2024-11-09T01:59:01Z

@tasty-gumi
pytest : test] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=create auto index on type:JSON is not supported)>

[pytest : test] (api_request.py:45)

[pytest : test] [2024-11-08 14:24:30 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=create auto index on type:JSON is not supported)> (api_request.py:46)

[pytest : test] ---------- generated html file: file:///tmp/ci_logs/test/report.html -----------

[pytest : test] =========================== short test summary info ============================

[pytest : test] FAILED testcases/test_index.py::TestIndexInvalid::test_create_index_json

[pytest : test] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!

mergify · 2024-11-11T11:32:09Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-12T10:13:53Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-12T12:38:32Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

czs007 · 2024-11-13T07:21:21Z

@tasty-gumi Intergration Test failed.

2024-11-12T13:06:50.0930073Z [2024/11/12 13:02:43.107 +00:00] [INFO] [querynodev2/services.go:483] ["start to load segments in parallel"] [collectionID=453880325253365774] [segmentType=Sealed] [requestSegments="[453880325253771188]"] [preparedSegments="[453880325253771188]"] [segmentNum=1] [concurrencyLevel=1]
2024-11-12T13:06:50.0931913Z [2024/11/12 13:02:43.107 +00:00] [WARN] [querynodev2/services.go:461] ["worker failed to load segments"] [collectionID=453880325253365774] [channel=by-dev-rootcoord-dml_0_453880325253365774v0] [replicaID=453880325350359041] [workID=1] [segments="[453880325253769583]"] [error="At LoadSegment: => unsupported data type at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp:442\n"]
2024-11-12T13:06:50.0933010Z [2024/11/12 13:02:43.108 +00:00] [INFO] [funcutil/parallel.go:86] ["load segment..."] [collectionID=453880325253365774] [segmentType=Sealed] [requestSegments="[453880325253771188]"] [preparedSegments="[453880325253771188]"] [partitionID=453880325253365775] [segmentID=453880325253771188] [segmentType=L1]
2024-11-12T13:06:50.0935035Z [2024/11/12 13:02:43.108 +00:00] [WARN] [querynode/service.go:301] ["delegator failed to load segments"] [collectionID=453880325253365774] [partitionID=453880325253365775] [shard=by-dev-rootcoord-dml_0_453880325253365774v0] [segmentID=453880325253769583] [level=L1] [currentNodeID=1] [dstNodeID=1] [error="At LoadSegment: => unsupported data type at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp:442\n"]
2024-11-12T13:06:50.0936237Z [2024/11/12 13:02:43.108 +00:00] [INFO] [segments/segment_loader.go:334] ["start loading segment files"] [collectionID=453880325253365774] [partitionID=453880325253365775] [shard=by-dev-rootcoord-dml_0_453880325253365774v0] [segmentID=453880325253771188] [rowNum=3000] [segmentType=Sealed]
2024-11-12T13:06:50.0938012Z [2024/11/12 13:02:43.108 +00:00] [WARN] [task/executor.go:160] ["failed to load segment"] [taskID=1731416037011] [collectionID=453880325253365774] [replicaID=453880325350359041] [segmentID=453880325253769583] [node=1] [source=segment_checker] [shardLeader=1] [error="At LoadSegment: => unsupported data type at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp:442\n"]
2024-11-12T13:06:50.0939242Z [2024/11/12 13:02:43.109 +00:00] [INFO] [task/executor.go:142] ["execute ac

mergify · 2024-11-13T08:41:45Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

tasty-gumi · 2024-11-13T08:48:15Z

/run-cpu-e2e

mergify · 2024-11-13T09:00:33Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-13T09:25:13Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-13T09:44:41Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

add geospatial interface in src common change type define and add segcore support add storage & chunkdata support feature: go package storage & proxy & typeutil support geospatial type in internal and typeutil in pkg Signed-off-by: tasty-gumi <1021989072@qq.com> add geospatial interface in src common change type define and add segcore support change: use wkb only in core Signed-off-by: tasty-gumi <1021989072@qq.com> fix:the geospatial only use std::string as FieldDataImpl template paramters && add geospatial data generation && pass chunk ,growing , sealed test fix : merge confilcts after rebase ,test nullable not pass due to upstream feat:basic GIS Function expr and visitor impl and GIS proto support && add:storage test of geo data Signed-off-by: tasty-gumi <1021989072@qq.com> feat:add proxy validate (pass httpserver test) && plan parser of geospatialfunction fix:sealedseg && go tidy fix:go mod feat:can produce wkt result for pymilvus client feat: add parser and query operator for geos filed && print geos binlog as wkt fix:fielddataimpl interface Signed-off-by: tasty-gumi <1021989072@qq.com> fix: some format of code && segmentfault debug for rebase Signed-off-by: tasty-gumi <1021989072@qq.com> add: import util test for parquet and mix compaction test Signed-off-by: tasty-gumi <1021989072@qq.com> fix: delete useless file and fix error for rebase Signed-off-by: tasty-gumi <1021989072@qq.com> fix: git rebase for custom function feat Signed-off-by: tasty-gumi <1021989072@qq.com> fix:rename geospatial field && update proto && rewrite Geometry class with smart pointer Signed-off-by: tasty-gumi <1021989072@qq.com> add:last commit miss add files Signed-off-by: tasty-gumi <1021989072@qq.com> fix: geospatial name replace in test files && fix geomertry and parser fix:remove some file change for dev Signed-off-by: tasty-gumi <1021989072@qq.com> fix:remove size in if && add destory in ~Geometry() Signed-off-by: tasty-gumi <1021989072@qq.com> add:conan file gdal rep Signed-off-by: tasty-gumi <1021989072@qq.com> remove:gdal fPIC Signed-off-by: tasty-gumi <1021989072@qq.com> fix: for rebase Signed-off-by: tasty-gumi <1021989072@qq.com> remove:log_warn Signed-off-by: tasty-gumi <1021989072@qq.com> remove:gdal shared Signed-off-by: tasty-gumi <1021989072@qq.com> remove:tbbproxy Signed-off-by: tasty-gumi <1021989072@qq.com> fix:add gdal option && update go mod Signed-off-by: tasty-gumi <1021989072@qq.com> dev:change some scripts Signed-off-by: tasty-gumi <1021989072@qq.com> remove: dev scripts Signed-off-by: tasty-gumi <1021989072@qq.com> add:conan files dependency of gdal Signed-off-by: tasty-gumi <1021989072@qq.com> fix:fmt cpp code Signed-off-by: tasty-gumi <1021989072@qq.com> add:delete geos-config in cmake_bulid/bin which may cause permission deny Signed-off-by: tasty-gumi <1021989072@qq.com> fix: add go client geometry interface && fix group by test Signed-off-by: tasty-gumi <1021989072@qq.com> fix: mod tidy for tests go client Signed-off-by: tasty-gumi <1021989072@qq.com> fix:memory leak in test and go fmt Signed-off-by: tasty-gumi <1021989072@qq.com> fix: datagen function remove pkoffset Signed-off-by: tasty-gumi <1021989072@qq.com> fix: go-client test add entity.geometry Signed-off-by: tasty-gumi <1021989072@qq.com> fix: fix test args and add some annotations Signed-off-by: tasty-gumi <1021989072@qq.com> fix:name and remove wkt marshl MaxDecimalDigits limit Signed-off-by: tasty-gumi <1021989072@qq.com> fix:misspell Signed-off-by: tasty-gumi <1021989072@qq.com> fix:go client test Signed-off-by: tasty-gumi <1021989072@qq.com> fix:listA size Signed-off-by: tasty-gumi <1021989072@qq.com> add:field data in schema_test Signed-off-by: tasty-gumi <1021989072@qq.com> test:add mergefield data Signed-off-by: tasty-gumi <1021989072@qq.com> fix:test err code modify Signed-off-by: tasty-gumi <1021989072@qq.com> fmt code Signed-off-by: tasty-gumi <1021989072@qq.com> fix:add geo type in client Signed-off-by: tasty-gumi <1021989072@qq.com> fix:add type in chunksegment sealdimpl Signed-off-by: tasty-gumi <1021989072@qq.com> fix:add chunk writer for geometry Signed-off-by: tasty-gumi <1021989072@qq.com>

Signed-off-by: tasty-gumi <1021989072@qq.com>

mergify · 2024-11-13T11:33:47Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-13T11:38:36Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

sre-ci-robot requested review from bigsheeper, congqixia and czs007 November 4, 2024 10:48

sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. area/dependency Pull requests that update a dependency file area/test sig/testing test/integration integration test labels Nov 4, 2024

mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Nov 4, 2024

MrPresent-Han reviewed Nov 5, 2024

View reviewed changes

internal/core/unittest/test_utils/DataGen.h Show resolved Hide resolved

tasty-gumi force-pushed the geo_pr0905 branch from 3b943e5 to ebf2daa Compare November 5, 2024 08:04

tasty-gumi force-pushed the geo_pr0905 branch from 3289ea0 to 9fba036 Compare November 8, 2024 09:32

tasty-gumi force-pushed the geo_pr0905 branch from 34ea50d to 4753bbe Compare November 9, 2024 08:19

mergify bot added the ci-passed label Nov 9, 2024

tasty-gumi force-pushed the geo_pr0905 branch from 4753bbe to 0173a25 Compare November 11, 2024 10:38

mergify bot removed the ci-passed label Nov 11, 2024

tasty-gumi force-pushed the geo_pr0905 branch from 0173a25 to 51ceb9a Compare November 12, 2024 09:23

tasty-gumi force-pushed the geo_pr0905 branch from bf84749 to 01c7024 Compare November 12, 2024 11:55

tasty-gumi force-pushed the geo_pr0905 branch from 01c7024 to 95b17a4 Compare November 13, 2024 08:28

tasty-gumi added 2 commits November 13, 2024 17:49

fix:go mod change

66bdeb8

Signed-off-by: tasty-gumi <1021989072@qq.com>

tasty-gumi force-pushed the geo_pr0905 branch from 95b17a4 to 66bdeb8 Compare November 13, 2024 10:16

feat: Geospatial Data Type and GIS Function Support for milvus #37417

Are you sure you want to change the base?

feat: Geospatial Data Type and GIS Function Support for milvus #37417

Conversation

tasty-gumi commented Nov 4, 2024

Main Goals

Solution

sre-ci-robot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

czs007 commented Nov 4, 2024

czs007 commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

czs007 commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 8, 2024

mergify bot commented Nov 8, 2024

tasty-gumi commented Nov 8, 2024

mergify bot commented Nov 8, 2024

mergify bot commented Nov 8, 2024

mergify bot commented Nov 8, 2024

czs007 commented Nov 9, 2024

mergify bot commented Nov 11, 2024

mergify bot commented Nov 12, 2024

mergify bot commented Nov 12, 2024

czs007 commented Nov 13, 2024 • edited Loading

mergify bot commented Nov 13, 2024

tasty-gumi commented Nov 13, 2024

mergify bot commented Nov 13, 2024

mergify bot commented Nov 13, 2024

mergify bot commented Nov 13, 2024

mergify bot commented Nov 13, 2024

mergify bot commented Nov 13, 2024

czs007 commented Nov 13, 2024 •

edited

Loading