Description
openedon Jul 23, 2024
Describe the bug
Hello team,
When I was going to model and create some analytics rules for a client, I noticed that there were multiple logs with the same "id_g" field populated into the OCI_Logs_CL table.
This "id_g" field (in OCI logs it's the eventID) is the unique identifier for which every alert receives, so there shouldn't be multiple logs with the same ID in Sentinel.
Source: https://docs.oracle.com/en-us/iaas/Content/Audit/Reference/logeventreference.htm
I have installed the default Azure Functions with the ARM template, while I have crosschecked multiple eventIDs in the OCI audit logs, I'm not sure why the same log is being populated 2,3,5, 30 times or more.
See the attached screenshots to understand the issue better.
P.S. I suspected something was odd with the connector because the amount of logs that's being ingested because the amount of logs that are produced in OCI is much less than the client's AWS environment.
To Reproduce
Steps to reproduce the behavior:
- Install the data connector via ARM: Oracle Cloud Infrastructure (using Azure Functions)
- Create all the required backend logging on the OCI side
- Insert all the client details in the template and in "CursorType" instead of "group", insert "partition"
- Wait for the logs to populate into the table
- In the Sentinel Workspace, query the following table "OCI_Logs_CL" and look for the eventIDs in the "id_g" column
Expected behavior
The expected behavior is to ingest each eventID once and not multiple times.
Additional context
To install and configure the data connector, I used the following resources:
- https://docs.oracle.com/en/learn/stream-oci-logs-to-azure-sentinel/index.html#related-links
- https://learn.microsoft.com/en-us/azure/sentinel/data-connectors/oracle-cloud-infrastructure
While the log is unique within the Audit Log service, I strongly believe there is something wrong with the Function App or the Python script where the cursor isn't being set up correctly as it pages through the logs.
I believe the problem is somewhere near this line in the main.py file: