Brought back the elastic plugin.

[scudette]

The reason it has a huge footprint is due to inefficiently written
client library. Forked client library and made it more efficient (only
export the bulk upload API).

Also added a server side event monitoring artifact which uploads to
elastic the results of any completed flows. This allows the results of
collected artifacts to be examined/visualized in Kibana.

Additionally VQL now supports SELECT *, column FROM plugin(). This is a good way of just adding extra columns without needing to list all the columns explicitly.

[j-mie]

Is there anyway to generate some kind of Elastic schema or template? Or do you rely on Elastic's dynamic mapping?

[scudette]

The index we use is derived from the artifact name so you would need to create the index and the proper mapping by external means before you can push data to it. If not then the index will be automatically created with default mappings the first item that is pushed to it based on the type of the first row.

For example you can use curl to initialize the index with the correct mappings:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#mappings

and because the index name is derived from the name of the artifact then all data from the same artifact will go there.

For Velociraptor it is difficult to figure out the mappings automatically because the types and fields emitted really depend on the VQL query and therefore each artifact is different. We pretty much also guess the column types by the heuristics in a similar way to elastic so I did not see the point of having a way of overriding the elastic heuristics (other than setting the mappings manually on the elastic side).

BTW I am still playing with this feature so it is considered experimental - I am trying to see if we can make up some useful dashboards and visualizations in kibana and how that would translate to the index structure.