[Bug]:[most urgent]segment can not loaded beacuse of index_null_offset not existed.

[melon1705]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.5.4
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):kafka
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.4
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Step1. Using Milvus 2.5.0, a collection was created with 500 partitions. This collection includes an IVF_FLAT type index. The collection schema is as follows:
`schema = MilvusClient.create_schema(
auto_id=False,
enable_dynamic_field=True,
)

schema.add_field(field_name="image_id", datatype=DataType.VARCHAR, is_primary=True, max_length=50, auto_id=False)
schema.add_field(field_name="image_vector", datatype=DataType.FLOAT_VECTOR, dim=1024)
schema.add_field(field_name="image_desc", datatype=DataType.VARCHAR, max_length=150, enable_analyzer=True, enable_match=True, analyzer_params={"type": "chinese"})
schema.add_field(field_name="index", datatype=DataType.VARCHAR, max_length=50)

index_params = client.prepare_index_params()

index_params.add_index(
field_name="image_vector",
index_type="IVF_FLAT",
index_name="image_vector_index",
metric_type="COSINE"
)`
Step2.The program started normally, but after running for a month, it failed to load the partition, which caused the vector search to be unavailable.the reason likes as follows:
2025/02/14 06:44:40.286 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]

Expected Behavior

The issue persists even after exporting the data from version 2.5.0 to the directory of version 2.5.4.I tried deleting the index and recreating it, but the issue was not resolved.

Steps To Reproduce

Created a collection in v2.5.0 or lower using the schema above and then upgrade Milvus to v2.5.4. Check if the collection can be loaded.

Milvus Log

I20250214 06:44:39.216392 646 InvertedIndexTantivy.cpp:113] [SERVER][~InvertedIndexTantivy][CGO_LOAD][]inverted index remove path:
[2025/02/14 06:44:39.248 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]
[2025/02/14 06:44:39.248 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]
[2025/02/14 06:44:39.289 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]
I20250214 06:44:40.195726 646 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 100 with num_rows 1
I20250214 06:44:40.195737 675 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 101 with num_rows 1
I20250214 06:44:40.195974 646 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 100 task to thread pool
I20250214 06:44:40.195771 643 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 104 with num_rows 816
I20250214 06:44:40.195722 548 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 104 with num_rows 1
I20250214 06:44:40.195744 587 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 104 with num_rows 3
I20250214 06:44:40.196064 548 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 104 task to thread pool
I20250214 06:44:40.196076 587 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 104 task to thread pool
I20250214 06:44:40.195799 244 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 100 with num_rows 3
I20250214 06:44:40.196004 675 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 101 task to thread pool
I20250214 06:44:40.195820 586 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 100 with num_rows 816
I20250214 06:44:40.195833 553 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 103 with num_rows 1
I20250214 06:44:40.196048 643 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 104 task to thread pool
I20250214 06:44:40.196403 586 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 100 task to thread pool
I20250214 06:44:40.196481 244 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 100 task to thread pool
I20250214 06:44:40.196491 553 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 103 task to thread pool
I20250214 06:44:40.196635 646 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 100 mmap false done
I20250214 06:44:40.196679 675 IndexConfigGenerator.cpp:71] [SERVER][VecIndexConfig][CGO_LOAD][]VecIndexConfig: origin_index_type=IVF_FLAT, index_type=IVF_FLAT_CC, metric_type=COSINE, config={"metric_type":"COSINE","nlist":"128","ssize":"48"}
I20250214 06:44:40.196704 675 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 101 mmap false done
I20250214 06:44:40.196782 587 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 104 mmap false done
I20250214 06:44:40.196796 675 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 0 with num_rows 1
I20250214 06:44:40.196877 646 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 102 with num_rows 816
I20250214 06:44:40.197628 675 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 0 task to thread pool
I20250214 06:44:40.196887 548 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 104 mmap false done
I20250214 06:44:40.197664 646 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 102 task to thread pool
I20250214 06:44:40.197796 548 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 0 with num_rows 816
I20250214 06:44:40.197830 548 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 0 task to thread pool
I20250214 06:44:40.197836 675 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 0 mmap false done
I20250214 06:44:40.197031 244 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 100 mmap false done
I20250214 06:44:40.197440 586 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 100 mmap false done
I20250214 06:44:40.197953 587 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 101 with num_rows 3
I20250214 06:44:40.197986 587 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 101 task to thread pool
I20250214 06:44:40.197553 553 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 103 mmap false done
I20250214 06:44:40.197990 586 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 1 with num_rows 1
I20250214 06:44:40.198027 586 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 1 task to thread pool
I20250214 06:44:40.198040 244 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 0 with num_rows 3
I20250214 06:44:40.198071 244 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 0 task to thread pool
I20250214 06:44:40.198122 553 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 1 with num_rows 816
I20250214 06:44:40.198141 553 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 1 task to thread pool
I20250214 06:44:40.198184 643 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 104 mmap false done
I20250214 06:44:40.198313 675 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 103 with num_rows 3
I20250214 06:44:40.198355 675 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 103 task to thread pool
I20250214 06:44:40.198370 244 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 0 mmap false done
I20250214 06:44:40.198444 643 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 103 with num_rows 816
I20250214 06:44:40.198463 587 IndexConfigGenerator.cpp:71] [SERVER][VecIndexConfig][CGO_LOAD][]VecIndexConfig: origin_index_type=IVF_FLAT, index_type=IVF_FLAT_CC, metric_type=COSINE, config={"metric_type":"COSINE","nlist":"128","ssize":"48"}
I20250214 06:44:40.198508 587 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 101 mmap false done
I20250214 06:44:40.198534 586 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 1 mmap false done
I20250214 06:44:40.198565 244 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 1 with num_rows 3
I20250214 06:44:40.198590 643 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 103 task to thread pool
I20250214 06:44:40.198640 244 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 1 task to thread pool
I20250214 06:44:40.198699 553 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 1 mmap false done
I20250214 06:44:40.198450 548 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 0 mmap false done
I20250214 06:44:40.198683 586 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 102 with num_rows 1
I20250214 06:44:40.198815 586 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 102 task to thread pool
I20250214 06:44:40.198827 675 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 103 mmap false done
I20250214 06:44:40.198916 675 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 101 with num_rows 816
I20250214 06:44:40.198941 675 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 101 task to thread pool
I20250214 06:44:40.198982 244 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 1 mmap false done
I20250214 06:44:40.199023 646 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 102 mmap false done
I20250214 06:44:40.199025 548 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 102 with num_rows 3
I20250214 06:44:40.199105 548 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 102 task to thread pool
I20250214 06:44:40.199169 586 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 102 mmap false done
I20250214 06:44:40.199206 643 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 103 mmap false done
=> invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51

I20250214 06:44:40.199466 643 InvertedIndexTantivy.cpp:113] [SERVER][~InvertedIndexTantivy][CGO_LOAD][]inverted index remove path:
I20250214 06:44:40.199766 548 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 102 mmap false done
=> invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51

I20250214 06:44:40.203465 548 InvertedIndexTantivy.cpp:113] [SERVER][~InvertedIndexTantivy][CGO_LOAD][]inverted index remove path:
I20250214 06:44:40.206964 675 IndexConfigGenerator.cpp:71] [SERVER][VecIndexConfig][CGO_LOAD][]VecIndexConfig: origin_index_type=IVF_FLAT, index_type=IVF_FLAT_CC, metric_type=COSINE, config={"metric_type":"COSINE","nlist":"128","ssize":"48"}
I20250214 06:44:40.207391 675 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 101 mmap false done
=> invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51

I20250214 06:44:40.214983 675 InvertedIndexTantivy.cpp:113] [SERVER][~InvertedIndexTantivy][CGO_LOAD][]inverted index remove path:
[2025/02/14 06:44:40.246 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]
[2025/02/14 06:44:40.286 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]
[2025/02/14 06:44:40.286 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]

Anything else?

Are there any other ways to temporarily avoid this problem? The inability to load data is causing the database to be unable to perform query operations, which has a significant impact.

[yanliang567]

@melon0803 please attach the milvus logs for investigation.
If you install Milvus with k8s, please refer this doc to export the whole Milvus logs.
If you install Milvus with docker-compose, please use docker-compose logs > milvus.log to export the logs.

/assign @melon0803

[melon1705]

logs.tar.gzThis is the k8s pod log I exported using export-milvus-log.sh.

[joenzx]

I have similar problems, waiting for solutions

[smellthemoon]

From this log, we can see all the index files were missed. Text index will be build if you set **enable_match = true**, which can not be rebuild now. Are you rebuilding on a vector column? @melon0803

[smellthemoon]

It related with #39635. It try to load on dropped segment, which has gced index binlog.

[smellthemoon]

you can try to release it and load collection again. @melon0803

[smellthemoon]

/assign