Skip to content

Duplicated logs ingested into Sentinel with OCI (Azure Functions) Data Connector #10863

Closed

Description

Describe the bug
Hello team,

When I was going to model and create some analytics rules for a client, I noticed that there were multiple logs with the same "id_g" field populated into the OCI_Logs_CL table.

This "id_g" field (in OCI logs it's the eventID) is the unique identifier for which every alert receives, so there shouldn't be multiple logs with the same ID in Sentinel.
Source: https://docs.oracle.com/en-us/iaas/Content/Audit/Reference/logeventreference.htm

I have installed the default Azure Functions with the ARM template, while I have crosschecked multiple eventIDs in the OCI audit logs, I'm not sure why the same log is being populated 2,3,5, 30 times or more.

See the attached screenshots to understand the issue better.

P.S. I suspected something was odd with the connector because the amount of logs that's being ingested because the amount of logs that are produced in OCI is much less than the client's AWS environment.

To Reproduce
Steps to reproduce the behavior:

  1. Install the data connector via ARM: Oracle Cloud Infrastructure (using Azure Functions)
  2. Create all the required backend logging on the OCI side
  3. Insert all the client details in the template and in "CursorType" instead of "group", insert "partition"
  4. Wait for the logs to populate into the table
  5. In the Sentinel Workspace, query the following table "OCI_Logs_CL" and look for the eventIDs in the "id_g" column

Expected behavior
The expected behavior is to ingest each eventID once and not multiple times.

Screenshots
image
image
image
image
image

Additional context

To install and configure the data connector, I used the following resources:

  1. https://docs.oracle.com/en/learn/stream-oci-logs-to-azure-sentinel/index.html#related-links
  2. https://learn.microsoft.com/en-us/azure/sentinel/data-connectors/oracle-cloud-infrastructure

While the log is unique within the Audit Log service, I strongly believe there is something wrong with the Function App or the Python script where the cursor isn't being set up correctly as it pages through the logs.

I believe the problem is somewhere near this line in the main.py file:

def process_events(client: oci.streaming.StreamClient, stream_id, initial_cursor, limit, sentinel: AzureSentinelConnector, start_ts):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Labels

ConnectorConnector specialty review needed

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions