forked from vectordotdev/vector
-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.md.erb
146 lines (103 loc) · 8.11 KB
/
README.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
<p align="center">
<strong>
<a href="<%= metadata.links.fetch("urls.vector_docs") %>">Docs<a/> • <a href="<%= metadata.links.fetch("urls.vector_community") %>">Chat<a/> • <a href="<%= metadata.links.fetch("urls.vector_twitter") %>">@vectordotdev<a/> • <a href="<%= metadata.links.fetch("urls.vector_download") %>">Download v<%= metadata.latest_version %><a/>
</strong>
</p>
---
<p align="center">
<img src="./website/static/img/readme_diagram.svg" alt="Vector">
</p>
<%- if metadata.new_post -%>
---
<p align="center">
<strong>
New post! <a href="<%= metadata.new_post.permalink %>"><%= metadata.new_post.title %></a>
</strong>
</p>
---
<%- end -%>
Vector is a _highly reliable_ observability data router built for demanding
production environments. Vector is designed on the following principles:
* **High reliability** - Built in [Rust][urls.rust], Vector is [memory safe][urls.rust_memory_safety], [correct][pages.index#correctness], and [performant][pages.index#performance].
* **Operator safety** - Vector is pragmatic and hard to break. It [avoids the common pitfalls](#the-little-details-make-all-of-the-difference) in similar tools.
* **All data** - [Logs][docs.data-model.log], [metrics][docs.data-model.metric], and traces (coming soon). A [sophisticated data model][docs.data-model] enables _correct_ interoperability.
* **One tool** - Deploys as an [agent][docs.roles.agent] or [service][docs.roles.service]. One tool gets your data from A to B.
Vector is **deployed over 100,000 times per day**, and is trusted by Fortune 500
companies and engineering teams trying to tame observability pipelines.
## [Documentation](https://vector.dev/docs/)
#### About
* [**Concepts**][docs.concepts]
* [**Data model (Event)**][docs.data_model] - [log event][docs.data-model.log], [metric event][docs.data-model.metric]
* [**Guarantees**][docs.guarantees]
#### Setup
* [**Installation**][docs.installation] - [containers][docs.containers], [operating systems][docs.operating_systems], [package managers][docs.package_managers], [from archives][docs.from-archives], [from source][docs.from-source]
* [**Configuration**][docs.configuration]
* [**Deployment**][docs.deployment] - [topologies][docs.topologies], [roles][docs.roles]
* [**Guides**][docs.guides] - [getting started][docs.guides.getting_started], [unit testing][docs.guides.unit-testing]
#### Reference
* [**Sources**][docs.sources] - <%= common_component_links(:source) %>
* [**Transforms**][docs.transforms] - <%= common_component_links(:transform) %>
* [**Sinks**][docs.sinks] - <%= common_component_links(:sink) %>
#### Administration
* [**Process management**][docs.process-management]
* [**Monitoring**][docs.monitoring]
* [**Updating**][docs.updating]
* [**Validating**][docs.validating]
#### Resources
* [**Community**][urls.vector_community] - [chat][urls.vector_chat], [@vectordotdev][urls.vector_twitter], [mailing list][urls.mailing_list]
* [**Releases**][urls.vector_releases] - [v<%= metadata.latest_version %> (latest)][urls.v<%= metadata.latest_version %>]
* [**Roadmap**][urls.vector_roadmap] - [vote on new features][urls.vote_feature]
## Performance
| Test | Vector | Filebeat | FluentBit | FluentD | Logstash | SplunkUF | SplunkHF |
| ---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| [TCP to Blackhole](https://github.com/timberio/vector-test-harness/tree/master/cases/tcp_to_blackhole_performance) | _**86mib/s**_ | n/a | 64.4mib/s | 27.7mib/s | 40.6mib/s | n/a | n/a |
| [File to TCP](https://github.com/timberio/vector-test-harness/tree/master/cases/file_to_tcp_performance) | _**76.7mib/s**_ | 7.8mib/s | 35mib/s | 26.1mib/s | 3.1mib/s | 40.1mib/s | 39mib/s |
| [Regex Parsing](https://github.com/timberio/vector-test-harness/tree/master/cases/regex_parsing_performance) | 13.2mib/s | n/a | _**20.5mib/s**_ | 2.6mib/s | 4.6mib/s | n/a | 7.8mib/s |
| [TCP to HTTP](https://github.com/timberio/vector-test-harness/tree/master/cases/tcp_to_http_performance) | _**26.7mib/s**_ | n/a | 19.6mib/s | <1mib/s | 2.7mib/s | n/a | n/a |
| [TCP to TCP](https://github.com/timberio/vector-test-harness/tree/master/cases/tcp_to_tcp_performance) | 69.9mib/s | 5mib/s | 67.1mib/s | 3.9mib/s | 10mib/s | _**70.4mib/s**_ | 7.6mib/s |
To learn more about our performance tests, please see the [Vector test harness][urls.vector_test_harness].
## Correctness
The following correctness tests are not exhaustive, but they demonstrate
fundamental differences in quality and design:
| Test | Vector | Filebeat | FluentBit | FluentD | Logstash | Splunk UF | Splunk HF |
| ---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| [Disk Buffer Persistence](https://github.com/timberio/vector-test-harness/tree/master/cases/disk_buffer_persistence_correctness) | ✅ | ✅ | ❌ | ❌ | ⚠️ | ✅ | ✅ |
| [File Rotate (create)](https://github.com/timberio/vector-test-harness/tree/master/cases/file_rotate_create_correctness) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [File Rotate (copytruncate)](https://github.com/timberio/vector-test-harness/tree/master/cases/file_rotate_truncate_correctness) | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| [File Truncation](https://github.com/timberio/vector-test-harness/tree/master/cases/file_truncate_correctness) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Process (SIGHUP)](https://github.com/timberio/vector-test-harness/tree/master/cases/sighup_correctness) | ✅ | ❌ | ❌ | ❌ | ⚠️ | ✅ | ✅ |
| [JSON (wrapped)](https://github.com/timberio/vector-test-harness/tree/master/cases/wrapped_json_correctness) | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |
To learn more about our correctness tests, please see the [Vector test harness][urls.vector_test_harness].
## The Little Details Make All Of The Difference
* **Zero delay start** - Starts and restarts without a delay.
* **Hot reload** - [Reload configuration on the fly][docs.process-management#reloading] without disrupting data flow.
* **Configurable concurrency** - All CPU cores (service) or just one (agent) via the [`--threads` flag][docs.process-management#starting].
* **Programmable transforms** - [Lua][docs.transforms.lua], [Javascript (coming soon)][urls.pr_721], and [WASM (coming soon)][urls.issue_1802].
* **Templating** - [Use event fields to create dynamic values][docs.reference.templating], enabling dynamic partitioning and more.
* **Sink healthchecks** - Protection against downstream disruptions make Vector a good citizen.
* **Customizable schema** - A schema is not forced on you. [Change Vector's schema][docs.global-options#log_schema] to anything you like.
* **Decoupled buffer design** - Buffers are coupled with sinks; a bad sink won't bring the entire pipeline to a halt.
* **Thoughtful disk buffering** - Transfers data to operating system memory to reduce the risk of data loss.
* **Merge split logs** - Easily merge multi-line logs into one event.
* **Custom DNS** - [Custom DNS][docs.global-options#dns_servers] makes service discovery possible.
* **Rate-limited logging** - Vector's internal logging is rate-limited and noise-free.
* **Optionally fully static binary** - Optional MUSL static binaries mean zero required dependencies.
* **Thoughtful docs** - [Quality documentation][docs.what-is-vector] that respects your time and reduces communication overhead.
* **Clear Guarantees** - A [guarantee support matrix][docs.guarantees] helps you make the appropriate tradeoffs.
* **Config unit tests** - [Develop Vector config files like code][docs.guides.unit-testing]. Avoid the frustrating dev style required by other tools.
* **Config linting** - [Quickly lint][docs.administration.validating] Vector config files to spot errors and prevent bad configs in CI.
## Installation
Run the following in your terminal, then follow the on-screen instructions.
```bash
<%= install_command %>
```
Or use your own [preferred method][docs.installation].
## Latest Posts & Announcements
<%- metadata.posts.reverse[0..4].each do |post| -%>
* [<%= post.title %>](<%= post.permalink %>)
<%- end -%>
[view all...][urls.vector_blog]
---
<p align="center">
Developed with ❤️ by <strong><a href="https://timber.io">Timber.io</a></strong>
</p>