Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom sink for handling large audit events #15

Merged
merged 37 commits into from
Nov 3, 2020
Merged

Custom sink for handling large audit events #15

merged 37 commits into from
Nov 3, 2020

Conversation

smahapatra86
Copy link
Contributor

  • As part of handling large audit events, we have created a custom sink which will check the message size.

  • If the message size is greater than 256 KB it will first push it to a azure blob, get the blob url and push message to event hub with the blob url.

  • If message size is less than 256 KB it will be usual flow. It will directly push to event hub.

  • We have created a client to use this custom sink. End users will be using this client.

@thecorrado
Copy link
Contributor

You have to update appveyor in order to fix build problems, see https://ci.appveyor.com/project/kmdlogic/kmd-logic-audit-client/builds/35867841

@thecorrado thecorrado self-requested a review October 21, 2020 07:44
@smahapatra86
Copy link
Contributor Author

smahapatra86 commented Oct 21, 2020 •
edited
Loading

You have to update appveyor in order to fix build problems, see https://ci.appveyor.com/project/kmdlogic/kmd-logic-audit-client/builds/35867841

We need to manually install the dotnet sdk 3.1.100 by script and referencing it in the yml file. Please see this link appveyor/ci#3234
Ideally appveyor CI should support this. But also I think we need to upgrade the image from VS 2017 to VS 2019 as appveyor build image VS 2017 supports only till .net sdk 2.2. After that we need to use VS 2019. Please this - https://www.appveyor.com/docs/windows-images-software/

thecorrado
thecorrado suggested changes Oct 22, 2020
View reviewed changes
...it.Client.SerilogLargeAuditEvents/SerilogLargeAuditEventClientBlobConfigurationExtensions.cs Outdated Show resolved Hide resolved
...it.Client.SerilogLargeAuditEvents/SerilogLargeAuditEventClientBlobConfigurationExtensions.cs Outdated Show resolved Hide resolved
src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AuditEventPayload.cs Outdated Show resolved Hide resolved
src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AuditEventPayload.cs Show resolved Hide resolved
src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobOrEventHubSink.cs Outdated Show resolved Hide resolved
...udit.Client.AzureBlobOrEventHubSink/AzureEventhubServiceHelper/AzureEventhubServiceHelper.cs Outdated Show resolved Hide resolved
...c.Audit.Client.AzureBlobOrEventHubSink/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink.csproj Outdated Show resolved Hide resolved
...gic.Audit.Client.AzureBlobOrEventHubSink/LoggerConfigurationAzureBlobOrEventHubExtensions.cs Outdated Show resolved Hide resolved
...c.Audit.Client.SerilogLargeAuditEvents/Kmd.Logic.Audit.Client.SerilogLargeAuditEvents.csproj Outdated Show resolved Hide resolved
....Audit.Client.SerilogLargeAuditEvents/SerilogLargeAuditEventClientConfigurationExtensions.cs Outdated Show resolved Hide resolved
@thecorrado thecorrado self-requested a review October 23, 2020 07:52
thecorrado
thecorrado suggested changes Oct 23, 2020
View reviewed changes
Copy link
Contributor

@thecorrado thecorrado left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why I have to review my changes in this PR? Something goes wrong in terms of merging. Please prepare this PR with changes that are made by you only.

src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobOrEventHubCustomSink.cs Outdated Show resolved Hide resolved
src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobOrEventHubCustomSink.cs Outdated Show resolved Hide resolved
src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobOrEventHubCustomSink.cs Outdated Show resolved Hide resolved
src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobOrEventHubCustomSink.cs Outdated Show resolved Hide resolved
src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobOrEventHubCustomSink.cs Outdated Show resolved Hide resolved
...ic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobServiceProvider/AzureBlobServiceProvider.cs Outdated Show resolved Hide resolved
...ic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobServiceProvider/AzureBlobServiceProvider.cs Outdated Show resolved Hide resolved
...ic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobServiceProvider/AzureBlobServiceProvider.cs Outdated Show resolved Hide resolved
...ic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobServiceProvider/AzureBlobServiceProvider.cs Outdated Show resolved Hide resolved
...c.Audit.Client.SerilogLargeAuditEvents/Kmd.Logic.Audit.Client.SerilogLargeAuditEvents.csproj Outdated Show resolved Hide resolved
@smahapatra86
Copy link
Contributor Author

Why I have to review my changes in this PR? Something goes wrong in terms of merging. Please prepare this PR with changes that are made by you only.

This PR is based on the merge from a forked repository i.e. avin101/kmd-logic-audit-client master branch to another branch custom-sink-large-messages in source repository kmdlogic. If the PR was to merge to master branch of kmdlogic repository then you would not have seen your changes. Once the PR is completed, the changes will be available in custom-sink-large-messages branch in kmdlogic from where we will create nuget packages which will be dev version containing the custom sink.

@smahapatra86
Copy link
Contributor Author

smahapatra86 commented Oct 30, 2020 •
edited
Loading

Why I have to review my changes in this PR? Something goes wrong in terms of merging. Please prepare this PR with changes that are made by you only.

This PR is based on the merge from a forked repository i.e. avin101/kmd-logic-audit-client master branch to another branch custom-sink-large-messages in source repository kmdlogic. If the PR was to merge to master branch of kmdlogic repository then you would not have seen your changes. Once the PR is completed, the changes will be available in custom-sink-large-messages branch in kmdlogic from where we will create nuget packages which will be dev version containing the custom sink.

@thecorrado We have merged the branch custom-sink-large-messages from master. Now your changes will not reflect in the PR.

@avin101
Copy link
Member

avin101 commented Nov 2, 2020

LGTM

@thecorrado thecorrado self-requested a review November 2, 2020 12:04
thecorrado
thecorrado suggested changes Nov 2, 2020
View reviewed changes
Copy link
Contributor

@thecorrado thecorrado left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some minor comments

src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobOrEventHubCustomSink.cs Outdated Show resolved Hide resolved
src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobOrEventHubCustomSink.cs Outdated Show resolved Hide resolved
src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AzureBlobOrEventHubCustomSink.cs Show resolved Hide resolved
src/Kmd.Logic.Audit.Client.AzureBlobOrEventHubSink/AzureEventHubServiceHelper.cs Outdated Show resolved Hide resolved
src/Kmd.Logic.Audit.Client.SerilogLargeAuditEvents/SerilogLargeAuditEventClient.cs Show resolved Hide resolved
thecorrado
thecorrado approved these changes Nov 3, 2020
View reviewed changes
Copy link
Contributor

@thecorrado thecorrado left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@smahapatra86 smahapatra86 merged commit 7dff721 into kmdlogic:custom-sink-large-messages Nov 3, 2020
smahapatra86 added a commit that referenced this pull request Nov 5, 2020
@smahapatra86 @satyajitmahapatra
Custom sink to handle large messages (#17)
6783eb4 
* Custom sink for handling large audit events (#15)

* added structure for the additional serilog sink for blob and eventhub

* added addition blob storage serilog sink. now audit events would go to both eventhub and blob destinations

* Separate extension logger methods created for blob and eventhub for the purpose of conditional logging

* Adding custom sink

* adding blob upload

* upload blob with name as event id

* Renaming project

* restructured the folders

* adding event hub upload

* Adding message check

* upgrade image to chcek if build is passing

* reverting the image upgrade as build failed

* configurable event size limit and renaming variables

* removing unwanted files

* changing variable name

* audit client

* pr comments

* pr comments

* pr comments

* pr comments

* library updated

* fixing appveyor build issues

* code fix for build failure

* pr review comments

* removing blob name from class variable

* pr comments

* renaming blob name

* removing unused package reference

* remove unwanted interfaces

* refactoring

* pr comments

* renaming variable

* added extension method accepting blob client

* pr comments

* pr comments

Co-authored-by: Avin Mathew (UYW) <UYW@kmd.dk>
Co-authored-by: Satyajit <YYW@kmd.dk>

* Upgrading version

* version number change as per norms. This is a minor release hence changing minor version number.

* version details added

Co-authored-by: smahapatra86 <YYW@kmd.dk>
Co-authored-by: Avin Mathew (UYW) <UYW@kmd.dk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@manikraja manikraja manikraja left review comments

@avin101 avin101 avin101 left review comments

@thecorrado thecorrado thecorrado approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@smahapatra86 @thecorrado @avin101 @manikraja @satyajitmahapatra