[Bug]: Build BITMAP index on array fields on loaded collection not raises error

[wangting0128]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-20240701-87bccb1a-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq   
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server:

NAME                                                              READY   STATUS                        RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
wt-test-etcd-0                                                    1/1     Running                       0               2d      10.104.33.247   4am-node36   <none>           <none>
wt-test-milvus-standalone-65987544cf-b696c                        1/1     Running                       0               32h     10.104.16.191   4am-node21   <none>           <none>
wt-test-minio-7b8f7b7444-bzx5d                                    1/1     Running                       0               2d      10.104.33.246   4am-node36   <none>           <none>

go sdk log:

2024/07/03 19:27:54 main_test.go:542: parse addr=10.104.16.191:19530
=== RUN   TestQueryArrayFieldExpr
2024/07/03 19:27:54 milvus_client.go:14: (ApiRequest): func [NewDefaultGrpcClient], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m59.999939239s]) 10.104.16.191:19530]
2024/07/03 19:27:54 milvus_client.go:21: (ApiResponse): func [NewDefaultGrpcClient], results: [0xc00007c980]
2024/07/03 19:27:54 milvus_client.go:14: (ApiRequest): func [CreateCollection], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m59.958860294s]) CJtd 0xc0000c4dc0 2 [0x16d1ea0]]
2024/07/03 19:27:54 milvus_client.go:21: (ApiResponse): func [CreateCollection], results: []
2024/07/03 19:27:54 milvus_client.go:14: (ApiRequest): func [InsertRows], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m59.776459134s]) CJtd  3000]
2024/07/03 19:28:01 milvus_client.go:21: (ApiResponse): func [InsertRows], results: [0xc01f2c8390]
2024/07/03 19:28:01 milvus_client.go:14: (ApiRequest): func [Flush], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m52.959462023s]) CJtd false]
2024/07/03 19:28:03 milvus_client.go:21: (ApiResponse): func [Flush], results: []
2024/07/03 19:28:03 milvus_client.go:14: (ApiRequest): func [CreateIndex], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m50.520137605s]) CJtd floatVec false 0xc00007c9c0 []]
2024/07/03 19:28:06 milvus_client.go:21: (ApiResponse): func [CreateIndex], results: []
2024/07/03 19:28:06 milvus_client.go:14: (ApiRequest): func [CreateIndex], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m48.235574645s]) CJtd binaryVec false 0xc01f388990 []]
2024/07/03 19:28:09 milvus_client.go:21: (ApiResponse): func [CreateIndex], results: []
2024/07/03 19:28:09 milvus_client.go:14: (ApiRequest): func [CreateIndex], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m45.344453586s]) CJtd fp16Vec false 0xc00007c9c0 []]
2024/07/03 19:28:12 milvus_client.go:21: (ApiResponse): func [CreateIndex], results: []
2024/07/03 19:28:12 milvus_client.go:14: (ApiRequest): func [CreateIndex], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m42.3276343s]) CJtd bf16Vec false 0xc00007c9c0 []]
2024/07/03 19:28:15 milvus_client.go:21: (ApiResponse): func [CreateIndex], results: []
2024/07/03 19:28:15 milvus_client.go:14: (ApiRequest): func [LoadCollection], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m39.256333532s]) CJtd []]
2024/07/03 19:28:16 milvus_client.go:21: (ApiResponse): func [LoadCollection], results: []
2024/07/03 19:28:16 query_test.go:830: boolArray[0] == false
2024/07/03 19:28:16 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m37.787007381s]) CJtd [] boolArray[0] == false [count(*)] []]
2024/07/03 19:28:16 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f371230]]
2024/07/03 19:28:16 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:1500 > > 
2024/07/03 19:28:16 query_test.go:830: int64Array[0] > 0
2024/07/03 19:28:16 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m37.637099126s]) CJtd [] int64Array[0] > 0 [count(*)] []]
2024/07/03 19:28:16 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f3717a0]]
2024/07/03 19:28:16 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:2999 > > 
2024/07/03 19:28:16 query_test.go:830: int8Array[0] > 0
2024/07/03 19:28:16 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m37.437873284s]) CJtd [] int8Array[0] > 0 [count(*)] []]
2024/07/03 19:28:17 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f41d560]]
2024/07/03 19:28:17 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:1524 > > 
2024/07/03 19:28:17 query_test.go:830: json_contains (int16Array, 100)
2024/07/03 19:28:17 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m37.238502317s]) CJtd [] json_contains (int16Array, 100) [count(*)] []]
2024/07/03 19:28:17 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f41da70]]
2024/07/03 19:28:17 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:100 > > 
2024/07/03 19:28:17 query_test.go:830: array_contains (int16Array, 100)
2024/07/03 19:28:17 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m37.037952312s]) CJtd [] array_contains (int16Array, 100) [count(*)] []]
2024/07/03 19:28:17 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f41df50]]
2024/07/03 19:28:17 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:100 > > 
2024/07/03 19:28:17 query_test.go:830: array_contains (int32Array, 1)
2024/07/03 19:28:17 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m36.838517442s]) CJtd [] array_contains (int32Array, 1) [count(*)] []]
2024/07/03 19:28:17 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f430450]]
2024/07/03 19:28:17 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:2 > > 
2024/07/03 19:28:17 query_test.go:830: json_contains (int32Array, 1)
2024/07/03 19:28:17 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m36.639033739s]) CJtd [] json_contains (int32Array, 1) [count(*)] []]
2024/07/03 19:28:17 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f430930]]
2024/07/03 19:28:17 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:2 > > 
2024/07/03 19:28:17 query_test.go:830: array_contains (int32Array, 1000000)
2024/07/03 19:28:17 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m36.437701727s]) CJtd [] array_contains (int32Array, 1000000) [count(*)] []]
2024/07/03 19:28:18 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f430e10]]
2024/07/03 19:28:18 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:0 > > 
2024/07/03 19:28:18 query_test.go:830: json_contains_all (int64Array, [90, 91])
2024/07/03 19:28:18 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m36.237433298s]) CJtd [] json_contains_all (int64Array, [90, 91]) [count(*)] []]
2024/07/03 19:28:18 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f4312f0]]
2024/07/03 19:28:18 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:91 > > 
2024/07/03 19:28:18 query_test.go:830: array_contains_all (int64Array, [1, 2])
2024/07/03 19:28:18 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m36.039881296s]) CJtd [] array_contains_all (int64Array, [1, 2]) [count(*)] []]
2024/07/03 19:28:18 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f371b00]]
2024/07/03 19:28:18 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:2 > > 
2024/07/03 19:28:18 query_test.go:830: array_contains_any (floatArray, [0, 100, 10000])
2024/07/03 19:28:18 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m35.838581092s]) CJtd [] array_contains_any (floatArray, [0, 100, 10000]) [count(*)] []]
2024/07/03 19:28:18 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f37e000]]
2024/07/03 19:28:18 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:101 > > 
2024/07/03 19:28:18 query_test.go:830: json_contains_any (floatArray, [0, 100, 10])
2024/07/03 19:28:18 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m35.63462068s]) CJtd [] json_contains_any (floatArray, [0, 100, 10]) [count(*)] []]
2024/07/03 19:28:18 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f37e4e0]]
2024/07/03 19:28:18 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:101 > > 
2024/07/03 19:28:18 query_test.go:830: doubleArray == [0, 1]
2024/07/03 19:28:18 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m35.440034504s]) CJtd [] doubleArray == [0, 1] [count(*)] []]
2024/07/03 19:28:19 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f37e9c0]]
2024/07/03 19:28:19 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:0 > > 
2024/07/03 19:28:19 query_test.go:830: array_length(varcharArray) == 10
2024/07/03 19:28:19 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m35.234378562s]) CJtd [] array_length(varcharArray) == 10 [count(*)] []]
2024/07/03 19:28:19 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f4317d0]]
2024/07/03 19:28:19 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:0 > > 
2024/07/03 19:28:19 query_test.go:830: array_length(doubleArray) == 100
2024/07/03 19:28:19 milvus_client.go:14: (ApiRequest): func [Query], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m35.039117911s]) CJtd [] array_length(doubleArray) == 100 [count(*)] []]
2024/07/03 19:28:19 milvus_client.go:21: (ApiResponse): func [Query], results: [[0xc01f431cb0]]
2024/07/03 19:28:19 query_test.go:833: type:Int64 field_name:"count(*)" scalars:<long_data:<data:3000 > > 
2024/07/03 19:28:19 milvus_client.go:14: (ApiRequest): func [ReleaseCollection], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m34.836432128s]) collName]
2024/07/03 19:28:19 milvus_client.go:19: (ApiResponse): func [ReleaseCollection], error: collection collName does not exist
2024/07/03 19:28:19 milvus_client.go:14: (ApiRequest): func [DescribeCollection], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m34.823459254s]) CJtd]
2024/07/03 19:28:19 milvus_client.go:21: (ApiResponse): func [DescribeCollection], results: [0xc00035c0e0]
2024/07/03 19:28:19 milvus_client.go:14: (ApiRequest): func [CreateIndex], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m34.811400567s]) CJtd boolArray false 0xc01f4116e0 [0x16d1e20]]
2024/07/03 19:28:22 milvus_client.go:21: (ApiResponse): func [CreateIndex], results: []
2024/07/03 19:28:22 milvus_client.go:14: (ApiRequest): func [CreateIndex], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m32.332976999s]) CJtd int8Array false 0xc01f4116e0 [0x16d1e20]]
2024/07/03 19:28:25 milvus_client.go:21: (ApiResponse): func [CreateIndex], results: []
2024/07/03 19:28:25 milvus_client.go:14: (ApiRequest): func [LoadCollection], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m29.248128641s]) CJtd []]
2024/07/03 19:28:25 milvus_client.go:19: (ApiResponse): func [LoadCollection], error: can't change the index for loaded collection: invalid parameter[expected=map[109:450859145800553228 110:450859145800553266 111:450859145800553290 112:450859145800553245]][actual=map[109:450859145800553228 110:450859145800553266 111:450859145800553290 112:450859145800553245 113:450859145800553346 114:450859145800553370]]
    response_check.go:19: 
                Error Trace:    /Users/wt/work/code/milvus-io/milvus-sdk-go/test/testcases/response_check.go:19
                                                        /Users/wt/work/code/milvus-io/milvus-sdk-go/test/testcases/query_test.go:857
                Error:          Received unexpected error:
                                can't change the index for loaded collection: invalid parameter[expected=map[109:450859145800553228 110:450859145800553266 111:450859145800553290 112:450859145800553245]][actual=map[109:450859145800553228 110:450859145800553266 111:450859145800553290 112:450859145800553245 113:450859145800553346 114:450859145800553370]]
                                (1) attached stack trace
                                  -- stack trace:
                                  | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus
                                  |     /Users/wt/work/code/milvus-io/milvus-sdk-go/client/collection.go:39
                                  | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).LoadCollection
                                  |     /Users/wt/work/code/milvus-io/milvus-sdk-go/client/collection.go:445
                                  | github.com/milvus-io/milvus-sdk-go/v2/test/base.(*MilvusClient).LoadCollection
                                  |     /Users/wt/work/code/milvus-io/milvus-sdk-go/test/base/milvus_client.go:194
                                  | github.com/milvus-io/milvus-sdk-go/v2/test/testcases.TestQueryArrayFieldExpr
                                  |     /Users/wt/work/code/milvus-io/milvus-sdk-go/test/testcases/query_test.go:856
                                  | testing.tRunner
                                  |     /usr/local/go/src/testing/testing.go:1439
                                  | runtime.goexit
                                  |     /usr/local/go/src/runtime/asm_amd64.s:1571
                                Wraps: (2) can't change the index for loaded collection: invalid parameter[expected=map[109:450859145800553228 110:450859145800553266 111:450859145800553290 112:450859145800553245]][actual=map[109:450859145800553228 110:450859145800553266 111:450859145800553290 112:450859145800553245 113:450859145800553346 114:450859145800553370]]
                                Error types: (1) *withstack.withStack (2) *errutil.leafError
                Test:           TestQueryArrayFieldExpr
2024/07/03 19:28:25 milvus_client.go:14: (ApiRequest): func [DropCollection], args: [context.Background.WithDeadline(2024-07-03 19:29:54.416047 +0800 CST m=+120.006104895 [1m29.215239867s]) CJtd]
2024/07/03 19:28:25 milvus_client.go:21: (ApiResponse): func [DropCollection], results: []
2024/07/03 19:28:25 milvus_client.go:14: (ApiRequest): func [Close], args: []
2024/07/03 19:28:25 milvus_client.go:21: (ApiResponse): func [Close], results: []
--- FAIL: TestQueryArrayFieldExpr (30.81s)
FAIL
2024/07/03 19:28:25 main_test.go:29: Start to tear down all
2024/07/03 19:28:25 milvus_client.go:14: (ApiRequest): func [NewDefaultGrpcClient], args: [context.Background.WithDeadline(2024-07-03 19:30:25.228554 +0800 CST m=+150.818482195 [1m59.999986462s]) 10.104.16.191:19530]
2024/07/03 19:28:25 milvus_client.go:21: (ApiResponse): func [NewDefaultGrpcClient], results: [0xc01f51f0c0]
2024/07/03 19:28:25 milvus_client.go:14: (ApiRequest): func [ListDatabases], args: [context.Background.WithDeadline(2024-07-03 19:30:25.228554 +0800 CST m=+150.818482195 [1m59.961292065s])]
2024/07/03 19:28:25 milvus_client.go:21: (ApiResponse): func [ListDatabases], results: [[{default []}]]
2024/07/03 19:28:25 milvus_client.go:14: (ApiRequest): func [Close], args: []
2024/07/03 19:28:25 milvus_client.go:21: (ApiResponse): func [Close], results: []

test case: https://github.com/milvus-io/milvus-sdk-go/blob/6f53a285e0a3dc33a94a48f227cc48b6faa36d11/test/testcases/index_test.go#L221

Expected Behavior

No response

Steps To Reproduce

Go sdk:
1. create a collection with all field types, collection name=CJtd, collection ID=450859145800553155
2. insert 3k data
3. flush collection
4. build index on vector fields
5. load collection and query
8. build BITMAP index on boolArray and int8Array fields
9. load collection <- raises error

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

I thought build index need to check on that, not load

[yanliang567]

agree, as we discussed, as long as the collection was loaded, any requests to build index/alter index/drop index should be forbidden

[zhagnlu]

for now, could create index when loaded, why not forbid to create index , any comment ? @xiaocai2333

[xiaocai2333]

I thought build index need to check on that, not load

@xiaofan-luan @yanliang567

Previously discussed, our constraint is:

1. Cannot load without building a vector index.
2. Must release collection before drop index.
3. Index can be built after loading. Based on the two constrains above, the index that can be built must be either a scalar index or the same vector index previously built (Idempotency).

So the expected behavior here should be that after building the scalar index, it should still be possible to load successfully.

[xiaofan-luan]

I don't think we can support it for now?

index build has to be happened on collection release state.

[xiaocai2333]

I don't think we can support it for now?

index build has to be happened on collection release state.

I don't quite understand with this point. Creating a new index is the same behavior as the segment handoff event from no index to having an index. As long as we ensure that now different index is created.