[FEATURE]Enable gpu cache to accelerate SQ8H index #9

wxyucs · 2019-10-16T03:21:25Z

Is your feature request related to a problem? Please describe.
Default gpu_cache_capacity is 0，it makes SQ8H index work slow.

Describe the solution you'd like
Set default gpu_cache_capacity in server_config.template.

#9 change default gpu_cache_capacity to 4

cydrain · 2019-10-16T09:41:41Z

done

Crud

* Refine add segment logic for bulk load Also restructured `addNewSegment`, `addNormalSegment` and `addFlushedSegment` into a single `addSegment` to avoid duplicate code block and to see more clearly how adding these segments are different. /kind improvement Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Add garbage collection function for segments produced by partially failed imports (#6) Signed-off-by: xingzhao <xing.zhao@zilliz.com> Co-authored-by: xingzhao <xing.zhao@zilliz.com> * Enhance error handle in bulk load (#5) Signed-off-by: wayblink <anyang.wang@zilliz.com> * Refine expiration/retention policy for import tasks (milvus-io#9) * Persist import task info to meta before update in memory (milvus-io#12) Signed-off-by: wayblink <anyang.wang@zilliz.com> * Add start position for import segment (milvus-io#7) Signed-off-by: wayblink <anyang.wang@zilliz.com> * Refine bulkload proto (milvus-io#11) Signed-off-by: groot <yihua.mo@zilliz.com> * Remove queryable/indexed logic and some other refinement (milvus-io#14) Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Resolve conflicts Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Some minor tweaks Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Implement CompleteBulkLoad feature (milvus-io#19) * Update interface and fix bug Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Fix dml start position mistake in AddSegmentReq (milvus-io#21) Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: wayblink <anyang.wang@zilliz.com> * Fix getDmlChannelPositionByBroadcast (milvus-io#23) * Fix null start position issue Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Fix merge conflicts Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Fix conflicts Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Bulkload enhancement (milvus-io#24) Signed-off-by: yhmo <yihua.mo@zilliz.com> * Add some unit tests Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: groot <yihua.mo@zilliz.com> Signed-off-by: yhmo <yihua.mo@zilliz.com> Co-authored-by: xing.zhao <34002927+EricStarer@users.noreply.github.com> Co-authored-by: xingzhao <xing.zhao@zilliz.com> Co-authored-by: wayblink <anyang.wang@zilliz.com> Co-authored-by: groot <yihua.mo@zilliz.com>

* Refine add segment logic for bulk load Also restructured `addNewSegment`, `addNormalSegment` and `addFlushedSegment` into a single `addSegment` to avoid duplicate code block and to see more clearly how adding these segments are different. /kind improvement Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Add garbage collection function for segments produced by partially failed imports (#6) Signed-off-by: xingzhao <xing.zhao@zilliz.com> Co-authored-by: xingzhao <xing.zhao@zilliz.com> * Enhance error handle in bulk load (#5) Signed-off-by: wayblink <anyang.wang@zilliz.com> * Refine expiration/retention policy for import tasks (#9) * Persist import task info to meta before update in memory (#12) Signed-off-by: wayblink <anyang.wang@zilliz.com> * Add start position for import segment (#7) Signed-off-by: wayblink <anyang.wang@zilliz.com> * Refine bulkload proto (#11) Signed-off-by: groot <yihua.mo@zilliz.com> * Remove queryable/indexed logic and some other refinement (#14) Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Resolve conflicts Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Some minor tweaks Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Implement CompleteBulkLoad feature (#19) * Update interface and fix bug Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Fix dml start position mistake in AddSegmentReq (#21) Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: wayblink <anyang.wang@zilliz.com> * Fix getDmlChannelPositionByBroadcast (#23) * Fix null start position issue Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Fix merge conflicts Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Fix conflicts Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Bulkload enhancement (#24) Signed-off-by: yhmo <yihua.mo@zilliz.com> * Add some unit tests Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: groot <yihua.mo@zilliz.com> Signed-off-by: yhmo <yihua.mo@zilliz.com> Co-authored-by: xing.zhao <34002927+EricStarer@users.noreply.github.com> Co-authored-by: xingzhao <xing.zhao@zilliz.com> Co-authored-by: wayblink <anyang.wang@zilliz.com> Co-authored-by: groot <yihua.mo@zilliz.com>

* Refine add segment logic for bulk load Also restructured `addNewSegment`, `addNormalSegment` and `addFlushedSegment` into a single `addSegment` to avoid duplicate code block and to see more clearly how adding these segments are different. /kind improvement Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Add garbage collection function for segments produced by partially failed imports (#6) Signed-off-by: xingzhao <xing.zhao@zilliz.com> Co-authored-by: xingzhao <xing.zhao@zilliz.com> * Enhance error handle in bulk load (#5) Signed-off-by: wayblink <anyang.wang@zilliz.com> * Refine expiration/retention policy for import tasks (milvus-io#9) * Persist import task info to meta before update in memory (milvus-io#12) Signed-off-by: wayblink <anyang.wang@zilliz.com> * Add start position for import segment (milvus-io#7) Signed-off-by: wayblink <anyang.wang@zilliz.com> * Refine bulkload proto (milvus-io#11) Signed-off-by: groot <yihua.mo@zilliz.com> * Remove queryable/indexed logic and some other refinement (milvus-io#14) Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Resolve conflicts Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Some minor tweaks Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Implement CompleteBulkLoad feature (milvus-io#19) * Update interface and fix bug Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Fix dml start position mistake in AddSegmentReq (milvus-io#21) Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: wayblink <anyang.wang@zilliz.com> * Fix getDmlChannelPositionByBroadcast (milvus-io#23) * Fix null start position issue Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Fix merge conflicts Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Fix conflicts Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> * Bulkload enhancement (milvus-io#24) Signed-off-by: yhmo <yihua.mo@zilliz.com> * Add some unit tests Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com> Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: groot <yihua.mo@zilliz.com> Signed-off-by: yhmo <yihua.mo@zilliz.com> Co-authored-by: xing.zhao <34002927+EricStarer@users.noreply.github.com> Co-authored-by: xingzhao <xing.zhao@zilliz.com> Co-authored-by: wayblink <anyang.wang@zilliz.com> Co-authored-by: groot <yihua.mo@zilliz.com>

Add more getters in SearchResponse & add normalize method in unittest

Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Update the pb go files

[ISSUE-milvus-io#8]: add new api

This was referenced Oct 16, 2019

change default gpu_cache_capacity to 4 #10

Merged

#9 change default gpu_cache_capacity to 4 #17

Merged

JinHai-CN added a commit that referenced this issue Oct 16, 2019

Merge pull request #17 from cydrain/0.5.0

6c6508e

#9 change default gpu_cache_capacity to 4

JinHai-CN closed this as completed Oct 16, 2019

JinHai-CN reopened this Oct 16, 2019

JinHai-CN closed this as completed Oct 16, 2019

JinHai-CN assigned cydrain Oct 16, 2019

cydrain mentioned this issue Dec 14, 2019

Milvus crash when searching and building index simultaneously using SQ8H #715

Closed

lale314 mentioned this issue Jan 12, 2020

Server crashed at startup in CentOS 7.6 #981

Closed

yhmo pushed a commit that referenced this issue Jan 15, 2020

Merge pull request #9 from milvus-io/crud

0f66ae5

Crud

op-hunter mentioned this issue Apr 11, 2020

Search failed with flat index #1901

Closed

jhd900518 mentioned this issue Dec 8, 2020

Multi-threads to call load_collection() and search() cause Milvus hang #4378

Closed

yanliang567 mentioned this issue Jun 25, 2021

Error msg is meaningless for indexcoord unavailable #6105

Closed

yanliang567 mentioned this issue Jul 3, 2021

[Chaos] Create collection and search get stuck during datacoord is killed #5985

Closed

3 tasks

xuqinkun mentioned this issue Jul 21, 2021

Run rootcoord error #6685

Closed

yanliang567 mentioned this issue Jul 21, 2021

[CI] <BaseException: (code=1, message=syncTimestamp Failed:Connect to rootcoord failed with error #6695

Closed

SuoSiFire mentioned this issue Nov 11, 2021

[Bug]: MilvusV1.1.1, search failed. #11644

Closed

1 task

xige-16 mentioned this issue Nov 18, 2021

[Bug]: Load failed reporting "<BaseException: (code=1, message=rpc error: code = Unknown desc = collection has not been loaded to memory or load failed)" #11969

Closed

1 task

bigsheeper mentioned this issue Nov 29, 2021

[Bug]: [benchmark][cluster] 1 billion dataset, query、load、search error #12323

Closed

1 task

qi49125 mentioned this issue Feb 18, 2022

[Bug]: The cluster cannot release the loaded collection or query #15623

Closed

1 task

yanliang567 mentioned this issue Mar 1, 2022

[Bug]: Searching and deleting vectors is not stable #15744

Closed

1 task

lijonnas mentioned this issue Mar 26, 2022

[Bug]: [cluster] Nightly test case hangs for panic in querynode and datacoord #15283

Closed

1 task

zbhuang666 mentioned this issue Apr 21, 2022

[Bug]: milvus2.0 cluster: collection load or release Causing cluster nodes to hang up and search timeout #16563

Closed

1 task

yhmo mentioned this issue Apr 21, 2022

[Bug]: when collection loadding or release Causing cluster nodes to hang up #16566

Closed

1 task

xiaofan-luan mentioned this issue May 6, 2022

[Bug]: Flush stuck #16819

Closed

1 task

WangErXiao mentioned this issue May 13, 2022

[Bug]:Chaos Test; When I kill proxy, I found dataCoord panic... #16963

Closed

1 task

qi49125 mentioned this issue May 24, 2022

[Bug]: QueryNode Restarts Due to Query Operations #17187

Closed

1 task

angelapytao mentioned this issue May 26, 2022

[Bug]: data service DropVirtualChannel failed #17233

Closed

1 task

qi49125 mentioned this issue Jun 6, 2022

[Bug]: query node restart when search #17397

Closed

1 task

Roger-Z mentioned this issue Oct 3, 2022

[Bug]: fail to load segment, error: NoSuchKey #19610

Closed

1 task

SamrudhiRRao mentioned this issue Nov 9, 2022

[Bug]: "StatusCode.UNAVAILABLE, internal: Milvus Proxy is not ready yet" #20433

Closed

1 task

jaime0815 referenced this issue in jaime0815/milvus Nov 18, 2022

Merge pull request #9 from youny626/branch-0.5.0

59758d8

Add more getters in SearchResponse & add normalize method in unittest

JiehangXie mentioned this issue Nov 29, 2022

[Bug]: Upgrading to 2.2.0 breaks S3 access #20823

Closed

1 task

smallcai03 mentioned this issue Dec 15, 2022

[Bug]: MultiThreadedRendezvous: StatusCode.UNAVAILABLE, internal: Milvus Proxy is not ready yet. please wait #21256

Closed

1 task

wayblink mentioned this issue Mar 2, 2023

[Bug]: [benchmark]Milvus restarted 2 times and load failed #22457

Closed

1 task

WishHe mentioned this issue Mar 3, 2023

[Bug]: use docker-compose start Milvus failed #22553

Closed

1 task

fall-nebula mentioned this issue Apr 6, 2023

[Bug]: service exited with a WARN [datanode/flush_manager.go:853] ["failed to SaveBinlogPaths"] #23240

Closed

1 task

longjiquan added a commit to longjiquan/milvus that referenced this issue Apr 29, 2023

Compitable logic of default database (milvus-io#9)

67b0304

Signed-off-by: longjiquan <jiquan.long@zilliz.com>

bedivere2018 mentioned this issue Jun 1, 2023

[Bug]: datanode channel_meta.go:224 "failed to init bloom filter" #24584

Closed

1 task

cyrusvahidi mentioned this issue Jun 13, 2023

[Bug]: v2.2.8 S3 docker config: "The Access Key Id you provided does not exist in our records." #24864

Closed

1 task

NicoYuan1986 mentioned this issue Jul 10, 2023

[Bug]: Indexnode will crash and auto restart later after create index #25438

Closed

1 task

iytprince2 mentioned this issue Jul 17, 2023

insert data too quickly causing datanode crash #25680

Closed

1 task

yandechao mentioned this issue Jul 20, 2023

[Bug]: data service save bin log path failed, reason = role datacoord[nodeID: 3] is not serving, reason: sate code: Abnormal #25800

Closed

1 task

zhaoxinmao mentioned this issue Jul 24, 2023

[Bug]: querycoord[ERROR] [meta/coordinator_broker.go:155] ["get recovery info failed"] #25861

Closed

1 task

NicoYuan1986 mentioned this issue Jul 28, 2023

[Bug]: [Nightly]DataNode crash reporting error syncTimestamp Failed : find no available rootcoord #25976

Closed

1 task

bingbb47 mentioned this issue Sep 13, 2023

[Bug]: Fail to load when create GPU_IVF_FLAT index #27057

Closed

1 task

yiwangdr mentioned this issue Sep 15, 2023

[Bug]: Standalone restarted once during test #27145

Closed

1 task

Richard-lrg mentioned this issue Nov 17, 2023

[Bug]: [datanode]flush pack with error, DataNode quit now. panic! #28549

Closed

1 task

zhaowenZhou mentioned this issue Nov 23, 2023

[Bug]: when I try to run connections.connect, it raised grpc.FutureTimeoutError #28697

Closed

1 task

SimFG pushed a commit to SimFG/milvus that referenced this issue Dec 14, 2023

Merge pull request milvus-io#9 from SimFG/kdxf-2.2.16-fixflush

c35c88f

Update the pb go files

charleskangzq mentioned this issue Mar 20, 2024

[Bug]: failed to load one of collections #31442

Closed

1 task

TonyAnn mentioned this issue Apr 28, 2024

[Bug]: datanode throw failed to init bloom filter , NoSuchKey(key=file/stats_log/449033619691139641/449033619691139642/449033619691141724/100/449033619691141732) #32658

Closed

1 task

MrPresent-Han pushed a commit to MrPresent-Han/milvus that referenced this issue Jun 4, 2024

Merge pull request milvus-io#9 from XuanYang-cn/master

94eb786

[ISSUE-milvus-io#8]: add new api

zhagnlu mentioned this issue Jul 23, 2024

[Bug]: [benchmark] test standalone crash because of build index core dump #34933

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE]Enable gpu cache to accelerate SQ8H index #9

[FEATURE]Enable gpu cache to accelerate SQ8H index #9

wxyucs commented Oct 16, 2019

cydrain commented Oct 16, 2019 •

edited

Loading

[FEATURE]Enable gpu cache to accelerate SQ8H index #9

[FEATURE]Enable gpu cache to accelerate SQ8H index #9

Comments

wxyucs commented Oct 16, 2019

cydrain commented Oct 16, 2019 • edited Loading

cydrain commented Oct 16, 2019 •

edited

Loading