Skip to content

[Enhancement] When observable data is too big, use hash #2288

Closed
@To-om

Description

@To-om

Request Type

Enhancement

Feature Description

Fields which contains more that 32k of data cannot be indexed, and breaks index engine. The aim of this issue is to store huge observable data in a dedicated unindexed field (named fullData) and store the hash in indexed field (instead of the real value). This change must be implemented in the observable creation and in the observable search (in properties).

Existing data must also be processed but the schema evolution cannot be used because the index may be broken. The processing can use the immense term processing of Scalligraph TheHive-Project/ScalliGraph#17

In order to fix existing data, the following configuration must be set:

db.janusgraph {
  immenseTermProcessing: {
    data: observableHashToIndex
  }
}

This make the next startup slower because the whole database must be crawled.
IMPORTANT This configuration should be present only for one startup to fix the data. It should be removed as soon as the process if finished.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions