Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New azure_monitor_logs sink #1808

Closed
nazar554 opened this issue Feb 16, 2020 · 10 comments
Closed

New azure_monitor_logs sink #1808

nazar554 opened this issue Feb 16, 2020 · 10 comments
Labels
domain: logs Anything related to Vector's log events domain: sinks Anything related to the Vector's sinks have: should We should have this feature, but is not required. It is medium priority. meta: good first issue Anything that is good for new contributors. needs: outside help Needs help outside of the Vector core team provider: azure Anything `azure` service provider related type: feature A value-adding code addition that introduce new functionality.

Comments

@nazar554
Copy link
Contributor

I'd really like to have support for Azure Log Analytics.
https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api
@binarylogic binarylogic added domain: sinks Anything related to the Vector's sinks type: new feature labels Feb 16, 2020
@binarylogic binarylogic changed the title Support for Azure log analytics? New azure_log_analytics sink Feb 16, 2020
@binarylogic binarylogic changed the title New azure_log_analytics sink New azure_monitor_logs sink Feb 16, 2020
@binarylogic binarylogic added meta: good first issue Anything that is good for new contributors. needs: outside help Needs help outside of the Vector core team labels Feb 16, 2020
@binarylogic
Copy link
Contributor

Thanks @nazar554, Azure Monitor looks powerful. We're hoping to wrap up support for the core GCP services and will likely start on Azure next. We'd love help from anyone willing to build this as well.

@zoni
Copy link

zoni commented Mar 24, 2020

I'd be very interested in support for this in vector as well. I'm currently picking up Rust, but don't feel proficient enough yet to offer implementation work on this.

That being said, if work on this was done, I'd be perfectly happy to test out any PRs and report issues on that and/or help out with documentation work if that is helpful.

@Hoverbear
Copy link
Contributor

@zoni Thanks! If you feel up for a project before we get to this, let me know, we can always offer some mentoring. :)

@nazar554
Copy link
Contributor Author

I have a working setup based on stdout & my forwarder, but I am not sure how to port the request batching logic to vector. I am supposed to use HttpSink?

Anyway, here is the repo.

@Hoverbear
Copy link
Contributor

@LucioFranco think this is a good job for good old HttpSink?

@LucioFranco
Copy link
Contributor

So I'll preface this with I don't know much about azure_monitor_logs but if its like a lot of other logging services. It looks like you need first someway to encode events into some sort of message encoding then a way to build requests. This is basically what HttpSink exposes without the user needing to deal with any async code. So I think from what I can tell the answer is yes.

@lukasmrtvy
Copy link

Any progress on this ?

@binarylogic binarylogic added have: should We should have this feature, but is not required. It is medium priority. provider: azure Anything `azure` service provider related labels Apr 20, 2020
@bigredmachine
Copy link

I'm interested in this Azure Monitor Logs sink, as well as the Azure Event Hubs sink (#2434). And think there is some overlap in functionality from my intended use case, so sharing my thoughts.

I'm looking to scrape an applications prometheus metrics endpoint, convert it into Application Insights JSON metrics format (something like this http://apmtips.com/blog/2017/10/27/send-metric-to-application-insights/), and then send them in compressed batches to an Azure Event hub (preferred, or to Log Analytics direct as per this Sink).
After it gets into Azure Event hub, then ends up in Log Analytics via another service, along with any other desired destinations.

Either way the metrics need to be converted to the same Application Insights JSON metrics format. So suggesting if that part could be re-usable/separate, it would help with my desired use case.

Let me know if I can help, e.g. QA this out.
Or help out with the metrics format conversion? Where would that live? I've not used vector or rust before. But looks promising that can achieve what I need if this conversion and sinks can be added.

@nazar554 nazar554 mentioned this issue Jun 13, 2020
Closed
@binarylogic binarylogic added type: feature A value-adding code addition that introduce new functionality. and removed type: new feature labels Jun 16, 2020
@binarylogic binarylogic added the domain: logs Anything related to Vector's log events label Aug 7, 2020
bors bot pushed a commit that referenced this issue Oct 4, 2020
@nazar554 @lukesteensen @jamtur01
feat(new sink): Initial azure_monitor_logs sink (#2811)
f77faa5 
Solves #1808 
Will add validations for configuration and unit tests

Co-authored-by: Luke Steensen <luke.steensen@gmail.com>
Co-authored-by: James Turnbull <james@lovedthanlost.net>
@nazar554
Copy link
Contributor Author

Should this be closed, or #4464 needs to be done first?

@jamtur01
Copy link
Contributor

This can be closed! Thanks.

mengesb pushed a commit to jacobbraaten/vector that referenced this issue Dec 9, 2020
@nazar554 @lukesteensen @jamtur01
feat(new sink): Initial azure_monitor_logs sink (vectordotdev#2811)
110a83e 
Solves vectordotdev#1808
Will add validations for configuration and unit tests

Co-authored-by: Luke Steensen <luke.steensen@gmail.com>
Co-authored-by: James Turnbull <james@lovedthanlost.net>
Signed-off-by: Brian Menges <brian.menges@anaplan.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
domain: logs Anything related to Vector's log events domain: sinks Anything related to the Vector's sinks have: should We should have this feature, but is not required. It is medium priority. meta: good first issue Anything that is good for new contributors. needs: outside help Needs help outside of the Vector core team provider: azure Anything `azure` service provider related type: feature A value-adding code addition that introduce new functionality.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@jamtur01 @Hoverbear @zoni @lukasmrtvy @binarylogic @LucioFranco @bigredmachine @nazar554