Nginx and node.js logs, new Elasticsearch alternative

[anutator]

1. There is new Elasticsearch + Kibana alternative (from Elasticsearch 7.10.2 & Kibana 7.10.2). Amazon will develop this project because of changes in Elasticsearch license.
   https://opensearch.org/
   https://docs-beta.opensearch.org/
   I think I can pass logs to OpenSearch as for Elasticsearch.

2. I'm newbie and I can't understand how to pass Nginx access and error logs using vector agent. Should I change access logs format to json in Nginx or leave it as it is by default and parse using vector config? Can you give an example how to parse Nginx access and error logs? Can I filter access logs to collect everything except HTTP 200 messages (I'd like to collect messages with anomalistic HTTP codes).

3. I've read that Timber.io can collect data from node.js application? Is it possible to collect data to Elasticsearch/Opensearch using on premise vector installation (agent)? We use pm2 and keep logs in .pm2/logsdirectory. Example:

{"level":50,"time":1621270199982,"pid":18971,"hostname":"main-infra-backup1","name":"GraphQL Error","requestId":"ad73eb20-f803-4e99-986b-05ac82a7a2bc","msg":"Headers:","message":"ER_NO_REFERENCED_ROW_2: Cannot add or update a child row: a foreign key constraint fails (cimp_staging.schedule_plan_stages_has_stages, CONSTRAINT FK_d54ff796a859acb62dd1f4d88b0 FOREIGN KEY (childrenId) REFERENCES schedule_plan_stages (schedulePlanStageId) ON DELETE CAS)","locations":[{"line":2,"column":3}],"path":["updateSchedulePlanStage"], ...

where can I find example how to collect nginx and node.js logs and sent them to Elasticsearch?

4. Would vector collect data unless i press Ctrl+C using command in manual:
   vector --config /etc/vector/name_of_file.toml

Should I use systemctl start vector to make it work permanently?

5. Can I use the same name in config files on different servers? For example [sources.nginx_metrics]

[jszwedko]

Hi @anutator !

Thanks for checking out Vector! Some answers:

1. I haven't tested OpenSearch, but I imagine it'll work based on its claim of API compatibility: https://opensearch.org/faq/#q1.8
2. You could approach this multiple ways. You could change your nginx logs to JSON, as you noted, or you can use their default text format and use the parse_nginx_logs) function in VRL. You can use VRL to transform (parse, modify, etc.) using the remap transform..
3. Yes, you can ingest logs from files using the file source. You would just point it at the directory containing the log files.
4. Correct, vector will only collect data while it is running. You can use systemd, as you mentioned, to run it as a service. By default it will try to read /etc/vector/vector.toml
5. Yes, you can reuse component names across servers. They must just be unique within a given vector instance.

I also recommend checking out https://vector.dev/docs/setup/quickstart/ for some introduction to vector including ingesting and transforming logs.

Let us know if you have any other questions! Feel free to drop into the discord server too.