Skip to content

feat(outputs): add arc output plugin for high-performance time-series writes #17875

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

feat(outputs): add arc output plugin for high-performance time-series writes #17875

wants to merge 14 commits into from

Conversation

@xe-nvdk
Copy link

@xe-nvdk xe-nvdk commented Oct 22, 2025

Summary

Adds Arc output plugin - a high-performance time-series database output using MessagePack binary protocol.

Features

  • MessagePack binary protocol (3-5x faster than line protocol)
  • Columnar data format for direct Arrow/Parquet writes
  • Multi-database support for environment/tenant isolation
  • Gzip compression (reduces bandwidth by ~10x)
  • Batch processing with configurable sizes
  • Comprehensive error handling and logging

Testing

All tests passing (9/9):

  • Connection handling
  • Write operations (single/batch)
  • API key authentication
  • Error scenarios
  • MessagePack encoding
  • Configuration validation

Documentation

  • Comprehensive README with examples
  • Configuration guide for basic/high-performance/multi-database setups
  • Troubleshooting section
  • Performance tuning guide

xe-nvdk and others added 9 commits October 6, 2025 18:42
@xe-nvdk
Merge pull request #1 from influxdata/master
fdfa547 
sync with main
@xe-nvdk
Merge branch 'influxdata:master' into master
ebf9e89
@xe-nvdk
Merge branch 'influxdata:master' into master
c0193b2
@xe-nvdk
Merge branch 'influxdata:master' into master
7de540f
@xe-nvdk
Merge branch 'influxdata:master' into master
72c1324
@xe-nvdk
Merge branch 'influxdata:master' into master
b296d8a
@xe-nvdk
Merge branch 'influxdata:master' into master
5befa6a
@xe-nvdk
feat(outputs): add arc output plugin for high-performance time-series…
acacd3f 
… writes

Adds Arc output plugin - a high-performance time-series database output using MessagePack binary protocol.

- MessagePack binary protocol (3-5x faster than line protocol)
- Columnar data format for direct Arrow/Parquet writes
- Multi-database support for environment/tenant isolation
- Gzip compression (reduces bandwidth by ~10x)
- Batch processing with configurable sizes
- Comprehensive error handling and logging

- Columnar format: 2.66x faster than row format
- Throughput: 2.42M records/sec
- Optimized for high-volume IoT and observability workloads

All tests passing (9/9):
- Connection handling
- Write operations (single/batch)
- API key authentication
- Error scenarios
- MessagePack encoding
- Configuration validation

- Comprehensive README with examples
- Configuration guide for basic/high-performance/multi-database setups
- Troubleshooting section
- Performance tuning guide
@xe-nvdk
Merge branch 'master' of github.com:Basekick-Labs/telegraf
f55d597
@telegraf-tiger
Copy link
Contributor

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

@xe-nvdk
Copy link
Author

xe-nvdk commented Oct 22, 2025

Hey team. I hope that everybody is doing good over there. Happy to contribute with an output plugin. Open to make modifications in the case that I missed something. By the way: !signed-cla

Have a great days, folks.
Ignacio

@xe-nvdk
Copy link
Author

xe-nvdk commented Oct 22, 2025

!signed-cla

@xe-nvdk xe-nvdk changed the title Adds Arc output plugin - a high-performance time-series database output using MessagePack binary protocol. feat(outputs): add arc output plugin for high-performance time-series writes Oct 22, 2025
@telegraf-tiger telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Oct 22, 2025
@xe-nvdk
small fixes on readme for arc
03ebdd7
@xe-nvdk
go mod tidy Issue
949adaf
@xe-nvdk
fixing errors with fmt reported by CI/CD checks
6b4aa8f
@xe-nvdk
Added github.com/vmihailenco/msgpack with BSD 2-Clause License
0d1b37b 
Added github.com/vmihailenco/tagparser with BSD 2-Clause License
Entries added in alphabetical order between vjeantet/grok and vmware/govmomi
Verification
✅ make check-deps now passes successfully across all 18 target platforms (darwin, freebsd, linux, windows on various architectures) All Telegraf CI checks should now pass:
✅ Markdown linting (README.md)
✅ Go code linting (arc.go, arc_test.go)
✅ Dependency license documentation (LICENSE_OF_DEPENDENCIES.md)
Modified files ready for commit:
plugins/outputs/arc/README.md - All markdown linting issues fixed
docs/LICENSE_OF_DEPENDENCIES.md - Added msgpack and tagparser licenses
@xe-nvdk
revive error in arc_test.go:394: Renamed config variable to cfg to av…
7ea1614 
…oid shadowing the imported config package
@telegraf-tiger
Copy link
Contributor

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

. DEB . RPM . TAR . GZ . ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_arm64.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz windows_i386.zip
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz

@Hipska
Copy link
Contributor

Hipska commented Oct 24, 2025

Maybe I'm missing something, but can't you already use outputs.http with data_format = "msgpack"?

@xe-nvdk
Copy link
Author

xe-nvdk commented Oct 24, 2025

Maybe I'm missing something, but can't you already use outputs.http with data_format = "msgpack"?

Yes, it can be used, but the best performance comes from our columnar format. The implementation you're referring to sends row-based data, which introduces conversion overhead when we transform rows to columns.

The output send directly the data in columnar format.

Let me know if you have additional questions.

Thank you
Ignacio

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@xe-nvdk @Hipska