Support json contains feature #25384

xiaocai2333 · 2023-07-06T12:37:14Z

Redefine the json_contains expression so that it no longer depends on the term expression.
support json_contains(json_array, element), element can be an array.
support json_contains_all(json_array, element_array), all element in element_array must be in the json_array.
support json_contains_any(json_array, element_array), any element in element_array is in the json_array.

codecov · 2023-07-07T02:44:33Z

Codecov Report

Merging #25384 (6cbd644) into master (b6fcbb0) will increase coverage by 0.02%.
Report is 1 commits behind head on master.
The diff coverage is 87.00%.

@@            Coverage Diff             @@
##           master   #25384      +/-   ##
==========================================
+ Coverage   82.62%   82.65%   +0.02%     
==========================================
  Files         812      812              
  Lines      109153   109522     +369     
==========================================
+ Hits        90191    90528     +337     
- Misses      15828    15863      +35     
+ Partials     3134     3131       -3

Files Changed	Coverage Δ
internal/core/src/common/Types.h	`3.12% <ø> (ø)`
internal/core/src/query/PlanProto.h	`7.14% <ø> (ø)`
...nternal/core/src/query/generated/ExecExprVisitor.h	`100.00% <ø> (ø)`
.../core/src/query/generated/ExtractInfoExprVisitor.h	`100.00% <ø> (ø)`
...nternal/core/src/query/generated/ShowExprVisitor.h	`0.00% <ø> (ø)`
...ternal/core/src/query/visitors/ShowExprVisitor.cpp	`0.00% <0.00%> (ø)`
...rnal/core/src/query/visitors/VerifyExprVisitor.cpp	`0.00% <0.00%> (ø)`
...ternal/core/src/query/visitors/ExecExprVisitor.cpp	`78.29% <81.57%> (+0.67%)`	⬆️
internal/core/src/query/PlanProto.cpp	`82.75% <85.10%> (+0.40%)`	⬆️
internal/core/src/query/Expr.h	`85.71% <100.00%> (+1.09%)`	⬆️
... and 4 more

... and 10 files with indirect coverage changes

xiaocai2333 · 2023-07-28T09:00:06Z

test example:

def test():
    connections.connect()
    int64_field = FieldSchema(name="int64", dtype=DataType.INT64, is_primary=True)
    json_field = FieldSchema(name="json", dtype=DataType.JSON)
    float_vector = FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=128)

    schema = CollectionSchema(fields=[int64_field, json_field, float_vector], enable_dynamic_field=True)
    hello_milvus = Collection("hello_milvus", schema=schema)

    import numpy as np
    rng = np.random.default_rng(seed=19530)
    rows = []
    rows.append({"int64": 1, "float_vector": rng.random((1, 128))[0], "json": {"array": [1,2,3,4,5,6]}, "array": [[1,2,3,4,5,6]]})
    rows.append({"int64": 2, "float_vector": rng.random((1, 128))[0], "json": {"array": ["a","b","c"]}, "array": [["a","b","c"]]})
    rows.append({"int64": 3, "float_vector": rng.random((1, 128))[0], "json": {"array": [1,"a","d"]}, "array": [[1,"a","d"]]})
    rows.append({"int64": 4, "float_vector": rng.random((1, 128))[0], "json": {"array": [1,2,3,4,5,6]}, "array": [[1,2,3,4,5,6]]})

    hello_milvus.insert(rows)
    index_type = "IVF_FLAT"
    index_params = {"nlist": 128}
    hello_milvus.create_index("float_vector",
                              index_params={"index_type": index_type, "params": index_params, "metric_type": "L2"})
    hello_milvus.load()

    expr = r'json_contains(json["array"], 3)'
    res = hello_milvus.search([rng.random((1, 128))[0]], "float_vector", {"metric_type": "L2"}, limit=6, expr=expr, output_fields=["$meta"])
    print(expr)
    print(res)

    expr = r'json_contains(json["array"], "a")'
    res = hello_milvus.search([rng.random((1, 128))[0]], "float_vector", {"metric_type": "L2"}, limit=6, expr=expr, output_fields=["$meta"])
    print(expr)
    print(res)

    expr = r'json_contains(array, [1,2,3,4,5,6])'
    res = hello_milvus.search([rng.random((1, 128))[0]], "float_vector", {"metric_type": "L2"}, limit=6, expr=expr,
                              output_fields=["$meta"])
    print(expr)
    print(res)

    expr = r'json_contains_all(json["array"], [1,2,3])'
    res = hello_milvus.search([rng.random((1, 128))[0]], "float_vector", {"metric_type": "L2"}, limit=6, expr=expr, output_fields=["json"])
    print(expr)
    print(res)

    expr = r'json_contains_all(json["array"], [1,"a"])'
    res = hello_milvus.search([rng.random((1, 128))[0]], "float_vector", {"metric_type": "L2"}, limit=6, expr=expr, output_fields=["y"])
    print(expr)
    print(res)

    expr = r'json_contains_all(json["array"], [1,"a", 2])'
    res = hello_milvus.search([rng.random((1, 128))[0]], "float_vector", {"metric_type": "L2"}, limit=6, expr=expr, output_fields=["$meta"])
    print(expr)
    print(res)

    expr = r'json_contains_any(json["array"], [1,"a", 2])'
    res = hello_milvus.search([rng.random((1, 128))[0]], "float_vector", {"metric_type": "L2"}, limit=6, expr=expr, output_fields=["json"])
    print(expr)
    print(res)

    expr = r'json_contains_any(json["array"], [1,7])'
    res = hello_milvus.search([rng.random((1, 128))[0]], "float_vector", {"metric_type": "L2"}, limit=6, expr=expr, output_fields=["$meta"])
    print(expr)
    print(res)

    expr = r'json_contains_any(json["array"], [7,8])'
    res = hello_milvus.search([rng.random((1, 128))[0]], "float_vector", {"metric_type": "L2"}, limit=6, expr=expr,
                              output_fields=["$meta"])
    print(expr)
    print(res)

    expr = r'json_contains_any(json["array"], [1,"a"])'
    res = hello_milvus.search([rng.random((1, 128))[0]], "float_vector", {"metric_type": "L2"}, limit=6, expr=expr,
                              output_fields=["$meta"])
    print(expr)
    print(res)
    hello_milvus.drop()

mergify · 2023-07-28T09:33:38Z

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-07-31T06:38:39Z

@xiaocai2333 ut workflow job failed, comment rerun ut can trigger the job again.

mergify · 2023-07-31T06:55:29Z

@xiaocai2333 Thanks for your contribution. Please submit with DCO, see the contributing guide https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md#developer-certificate-of-origin-dco.

mergify · 2023-07-31T07:30:41Z

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

xiaocai2333 · 2023-07-31T07:35:12Z

/run-cpu-e2e

mergify · 2023-08-02T08:29:54Z

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

xiaocai2333 · 2023-08-02T08:31:09Z

/run-cpu-e2e

yah01

Some copying could be avoided, and the contains time complexity could be optimized, others lgtm

internal/core/src/query/visitors/ExecExprVisitor.cpp

czs007 · 2023-08-10T03:09:13Z

/hold

longjiquan

Please add ut for the changes of segcore.

internal/core/src/query/visitors/ExecExprVisitor.cpp

internal/parser/planparserv2/parser_visitor.go

mergify · 2023-08-10T11:18:47Z

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

czs007 · 2023-08-11T02:27:53Z

/approve

sre-ci-robot · 2023-08-11T02:28:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: czs007, xiaocai2333

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

czs007 · 2023-08-11T02:28:08Z

/unhold

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

mergify · 2023-08-11T07:17:53Z

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

xiaocai2333 · 2023-08-11T07:19:12Z

/run-cpu-e2e

mergify · 2023-08-11T08:00:03Z

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

xiaocai2333 · 2023-08-11T08:01:44Z

/run-cpu-e2e

longjiquan · 2023-08-11T09:08:55Z

/lgtm

sre-ci-robot requested review from bigsheeper and xige-16 July 6, 2023 12:37

sre-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines. label Jul 6, 2023

mergify bot added the dco-passed DCO check passed. label Jul 6, 2023

xiaocai2333 force-pushed the json_contains_feature-4 branch from e3f8640 to 16e210c Compare July 7, 2023 01:56

xiaocai2333 force-pushed the json_contains_feature-4 branch from 16e210c to 9c1bace Compare July 28, 2023 08:59

xiaocai2333 force-pushed the json_contains_feature-4 branch from 9c1bace to 31d2476 Compare July 31, 2023 06:24

sre-ci-robot added area/test sig/testing test/integration integration test labels Jul 31, 2023

xiaocai2333 force-pushed the json_contains_feature-4 branch from 31d2476 to bffeb8d Compare July 31, 2023 06:55

mergify bot added needs-dco DCO is missing in this pull request. and removed dco-passed DCO check passed. labels Jul 31, 2023

mergify bot added dco-passed DCO check passed. and removed needs-dco DCO is missing in this pull request. labels Jul 31, 2023

xiaocai2333 force-pushed the json_contains_feature-4 branch from bffeb8d to 1d41cd6 Compare August 2, 2023 07:56

xiaocai2333 requested a review from czs007 August 9, 2023 09:11

xiaocai2333 assigned czs007 Aug 9, 2023

yah01 suggested changes Aug 10, 2023

View reviewed changes

sre-ci-robot added the do-not-merge/hold label Aug 10, 2023

longjiquan suggested changes Aug 10, 2023

View reviewed changes

internal/core/src/query/visitors/ExecExprVisitor.cpp Outdated Show resolved Hide resolved

internal/parser/planparserv2/parser_visitor.go Show resolved Hide resolved

xiaocai2333 force-pushed the json_contains_feature-4 branch from 1d41cd6 to 15a67a7 Compare August 10, 2023 10:33

xiaocai2333 force-pushed the json_contains_feature-4 branch from 15a67a7 to f9abc11 Compare August 11, 2023 00:35

sre-ci-robot added the approved label Aug 11, 2023

sre-ci-robot removed the do-not-merge/hold label Aug 11, 2023

xiaocai2333 force-pushed the json_contains_feature-4 branch from f9abc11 to be994bd Compare August 11, 2023 03:34

czs007 added this to the 2.3 milestone Aug 11, 2023

Support json contains feature

6cbd644

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

xiaocai2333 force-pushed the json_contains_feature-4 branch from be994bd to 6cbd644 Compare August 11, 2023 07:02

mergify bot added the ci-passed label Aug 11, 2023

sre-ci-robot assigned longjiquan Aug 11, 2023

sre-ci-robot added the lgtm label Aug 11, 2023

sre-ci-robot merged commit a0198ce into milvus-io:master Aug 11, 2023

xiaocai2333 deleted the json_contains_feature-4 branch November 1, 2023 12:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support json contains feature #25384

Support json contains feature #25384

xiaocai2333 commented Jul 6, 2023 •

edited

Loading

codecov bot commented Jul 7, 2023 •

edited

Loading

xiaocai2333 commented Jul 28, 2023 •

edited

Loading

mergify bot commented Jul 28, 2023

mergify bot commented Jul 31, 2023

mergify bot commented Jul 31, 2023

mergify bot commented Jul 31, 2023

xiaocai2333 commented Jul 31, 2023

mergify bot commented Aug 2, 2023

xiaocai2333 commented Aug 2, 2023

yah01 left a comment

czs007 commented Aug 10, 2023

longjiquan left a comment

mergify bot commented Aug 10, 2023

czs007 commented Aug 11, 2023

sre-ci-robot commented Aug 11, 2023

czs007 commented Aug 11, 2023

mergify bot commented Aug 11, 2023

xiaocai2333 commented Aug 11, 2023

mergify bot commented Aug 11, 2023

xiaocai2333 commented Aug 11, 2023

longjiquan commented Aug 11, 2023

Support json contains feature #25384

Support json contains feature #25384

Conversation

xiaocai2333 commented Jul 6, 2023 • edited Loading

codecov bot commented Jul 7, 2023 • edited Loading

Codecov Report

xiaocai2333 commented Jul 28, 2023 • edited Loading

mergify bot commented Jul 28, 2023

mergify bot commented Jul 31, 2023

mergify bot commented Jul 31, 2023

mergify bot commented Jul 31, 2023

xiaocai2333 commented Jul 31, 2023

mergify bot commented Aug 2, 2023

xiaocai2333 commented Aug 2, 2023

yah01 left a comment

Choose a reason for hiding this comment

czs007 commented Aug 10, 2023

longjiquan left a comment

Choose a reason for hiding this comment

mergify bot commented Aug 10, 2023

czs007 commented Aug 11, 2023

sre-ci-robot commented Aug 11, 2023

czs007 commented Aug 11, 2023

mergify bot commented Aug 11, 2023

xiaocai2333 commented Aug 11, 2023

mergify bot commented Aug 11, 2023

xiaocai2333 commented Aug 11, 2023

longjiquan commented Aug 11, 2023

xiaocai2333 commented Jul 6, 2023 •

edited

Loading

codecov bot commented Jul 7, 2023 •

edited

Loading

xiaocai2333 commented Jul 28, 2023 •

edited

Loading