Skip to content

Error with index rebuilding #3867

@ExtReMLapin

Description

@ExtReMLapin

Hello,

In production on data I cannot share we're metting difficulties with stability of arcadedb.

The database is quite heavy, but again that's just our opinion, we adde inside it a lot of data. (Folder is 9.8 Gb)

While some of the issues are starting to be a little mitigated by performances patches, there are others issues like crash , OOM crashes (ubunting killing it).

One of those issues are for example this error during graph rebuilding :

2026-04-16 00:15:06.657 INFO  [LSMVectorIndex] Graph build building: 35833/36080 (vector accesses=35841, heap=18371,9/32768,0MB, offheap=0,6MB, files=9,8MB [idx=0,5, graph=9,3, pq=0,0, compacted=0,0])
2026-04-16 00:15:11.662 INFO  [LSMVectorIndex] Graph build building: 35996/36080 (vector accesses=36004, heap=25043,9/32768,0MB, offheap=0,6MB, files=9,8MB [idx=0,5, graph=9,3, pq=0,0, compacted=0,0])
2026-04-16 00:15:14.065 INFO  [LSMVectorIndex] Graph build building: 36080/36080 (vector accesses=36088, heap=28259,9/32768,0MB, offheap=0,6MB, files=9,8MB [idx=0,5, graph=9,3, pq=0,0, compacted=0,0])
2026-04-16 00:15:25.091 INFO  [LSMVectorIndex] JVector graph index built successfully
2026-04-16 00:15:25.093 INFO  [LSMVectorIndex] Graph build persisting: 0/744824 (vector accesses=0, heap=29799,1/32768,0MB, offheap=0,6MB, files=187,8MB [idx=6,0, graph=181,8, pq=0,0, compacted=0,0])
2026-04-16 00:15:25.094 INFO  [LSMVectorIndexGraphFile] Starting graph write (sequential) with chunking: 744824 nodes, 50MB chunk size
2026-04-16 00:15:25.094 INFO  [LSMVectorIndexGraphFile] Writing graph WITHOUT inline vectors - topology only (vectors fetched from documents on-demand)
2026-04-16 00:15:26.546 INFO  [LSMVectorIndex] Graph persistence chunk complete: 50,0MB written
2026-04-16 00:15:27.817 INFO  [LSMVectorIndex] Graph persistence chunk complete: 50,0MB written
2026-04-16 00:15:27.818 SEVER [LSMVectorIndexGraphFile] Error writing graph to pages: Transaction error on commitcom.arcadedb.exception.TransactionException: Transaction error on commit
        at com.arcadedb.database.TransactionContext.commit1stPhase(TransactionContext.java:718)
        at com.arcadedb.database.TransactionContext.commit(TransactionContext.java:133)
        at com.arcadedb.database.LocalDatabase.lambda$commit$5(LocalDatabase.java:499)
        at com.arcadedb.database.LocalDatabase.executeInReadLock(LocalDatabase.java:1602)
        at com.arcadedb.database.LocalDatabase.commit(LocalDatabase.java:493)
        at com.arcadedb.index.vector.LSMVectorIndex.lambda$buildGraphFromScratchWithRetry$11(LSMVectorIndex.java:1569)
        at com.arcadedb.index.vector.ContiguousPageWriter.write(ContiguousPageWriter.java:211)
        at com.arcadedb.index.vector.ContiguousPageWriter.writeInt(ContiguousPageWriter.java:133)
        at io.github.jbellis.jvector.graph.disk.OnDiskSequentialGraphIndexWriter.write(OnDiskSequentialGraphIndexWriter.java:145)
        at com.arcadedb.index.vector.LSMVectorIndexGraphFile.writeGraph(LSMVectorIndexGraphFile.java:232)
        at com.arcadedb.index.vector.LSMVectorIndex.buildGraphFromScratchWithRetry(LSMVectorIndex.java:1577)
        at com.arcadedb.index.vector.LSMVectorIndex.buildGraphFromScratch(LSMVectorIndex.java:1027)
        at com.arcadedb.index.vector.LSMVectorIndex.buildGraphFromScratch(LSMVectorIndex.java:1016)
        at com.arcadedb.index.vector.LSMVectorIndex.lambda$startAsyncGraphRebuild$13(LSMVectorIndex.java:1956)
        at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: java.lang.IllegalArgumentException: Cannot resize the buffer (autoResizable=false)
        at com.arcadedb.database.Binary.checkForAllocation(Binary.java:677)
        at com.arcadedb.database.Binary.size(Binary.java:609)
        at com.arcadedb.engine.CachedPage.loadMetadata(CachedPage.java:63)
        at com.arcadedb.engine.PageManager.loadPage(PageManager.java:426)
        at com.arcadedb.engine.PageManager.getMostRecentVersionOfPage(PageManager.java:180)
        at com.arcadedb.engine.PageManager.checkPageVersion(PageManager.java:217)
        at com.arcadedb.database.TransactionContext.commit1stPhase(TransactionContext.java:696)
        ... 14 more
2026-04-16 00:15:27.819 SEVER [LSMVectorIndex] PERSIST: Failed to persist graph for CHUNK_EMBEDDING_0_593681324616448 (nodes=744824, storeVectorsInGraph=false, txStatus=INACTIVE): IndexException - Error writing graph to pages
2026-04-16 00:15:27.819 INFO  [LSMVectorIndex] Built graph for index: CHUNK_EMBEDDING_0_593681324616448
2026-04-16 00:15:27.819 INFO  [LSMVectorIndex] Async graph rebuild completed for index: CHUNK_EMBEDDING_0_593681324616448
2026-04-16 00:15:54.107 INFO  [LSMVectorIndex] JVector graph index built successfully

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No fields configured for Bug.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions