Add pagination and retries based on load, support queries - PrismaCloudDriver

[raj-axe]

Summary - Pagination and query handling changes

• Implemented pagination for Prisma Cloud API requests to handle large datasets efficiently.
• Added retry logic to automatically retry API requests based on load conditions.
• Improved error handling and logging.

Changes

• Updated _fetch_prisma_data method to include retry logic.
• Modified _paginate_prisma_search to handle pagination properly.
• Refactored query functions to use pagination.

Summary Entity driven hunting additions

Introduced support queries that can be used as detection triggers in the playbook or for investigating purposes as well.The entire concept of these queries involve combining risk data with the threat data and perform entity driven hunting

Dynamic Query Definitions:

Query definitions for Prisma Cloud are stored in YAML files. The metadata in these files includes properties such as:

, name, and description: Identify the query set.
data_environments and data_families: Specify the environment (e.g., Prismacloud) and categorization for the queries.
tags and aliases: Help classify and reference queries (e.g., asset, event, config_resource queries).
defaults and sources: Provide default parameters (e.g., time unit, amount, start, end) and define dynamic query templates via macros.

Working with Query Provider:

The driver integrates with MSTICPy’s QueryProvider framework. With methods like list_queries() and browse_queries(), users can easily explore the available queries organized by family and environment. This makes it simpler to identify the appropriate query before executing it.

Query Types Implemented:

Asset Queries:
Methods like prisma_search_assets() and specialized queries such as search_asset_relative and search_asset_relative_with_prismafindings enable retrieval of asset data. These queries leverage dynamic query macros (using aliases like *asset_query_relative) to allow filtering by cloud service or asset type. This is essential for threat hunting across diverse cloud assets and analyze assets with findings.
This helps us to analyze cloud assets in an multicloud environment.

Event Queries:
Event queries are defined via aliases such as *event_query_relative and *event_query_absolute. They are designed to retrieve security event data by either using a relative time range (e.g., last N minutes) or absolute timestamps. The driver method prisma_search_events() handles these queries, ensuring that time parameters and result limits are applied correctly for effective event monitoring.
Event queries are similar to audit logs except the fact it is multicloud normalized.

Configuration Resource Queries:
Queries like prisma_search_config_resource() retrieve configuration or compliance data from Prisma Cloud. These queries are particularly useful for assessing configurations in cloud resources (e.g., Azure Key Vault) for security misconfigurations. The YAML metadata includes examples such as config_resource_firewall_disabled_azurekeyvault and config_resource_rbac_disabled_azurekeyvault, which help detect vulnerabilities in resource settings.

Testing

• Tested with various query types and confirmed pagination is working.
• Verified that retry mechanism triggers correctly when API fails intermittently.
• Verified query provider working
• verified support query working

Notes

• This improvement optimizes API requests and reduces failures due to timeouts or rate limits.
• Also enables the user with queryprovider way of handling prismacloud

[ianhelle]

There is a pylint and a few mypy errors. Looks like the mypy errors are mypy not tracking the typed dict.
If you're confident that non-int-convertible values are not going to sneak in to the kwargs, it's prob ok to #type: ignore them.

[raj-axe]

There is a pylint and a few mypy errors. Looks like the mypy errors are mypy not tracking the typed dict. If you're confident that non-int-convertible values are not going to sneak in to the kwargs, it's prob ok to #type: ignore them.

Thank you Ian for the feedback! i have addressed the comment :)

[ianhelle]

Annoying - still one mypy issue. Could be that you can derive DriverConfig from a Protocol.
Although I think it's probably OK to suppress this with a type: ignore since we know that DriverConfig does support this. Kind of weird that Mypy can't work this out from the declaration

[raj-axe]

There is a pylint and a few mypy errors. Looks like the mypy errors are mypy not tracking the typed dict. If you're confident that non-int-convertible values are not going to sneak in to the kwargs, it's prob ok to #type: ignore them.

There is a pylint and a few mypy errors. Looks like the mypy errors are mypy not tracking the typed dict. If you're confident that non-int-convertible values are not going to sneak in to the kwargs, it's prob ok to #type: ignore them.

@ianhelle sorry for the delay took some time in understanding query store and how reading from queries folder works, introduced support queries ,driver base inheritence, fixed all the errors , i have documented in the top of all the changes

[raj-axe]

There is a pylint and a few mypy errors. Looks like the mypy errors are mypy not tracking the typed dict. If you're confident that non-int-convertible values are not going to sneak in to the kwargs, it's prob ok to #type: ignore them.

There is a pylint and a few mypy errors. Looks like the mypy errors are mypy not tracking the typed dict. If you're confident that non-int-convertible values are not going to sneak in to the kwargs, it's prob ok to #type: ignore them.

@ianhelle have sorry for the delay took some understanding query store and reading from queries, introduced support queries ,driver base inheritence, fixed all the errors , i have documented in the top of all the changes

thank you 👍

[ianhelle]

I made a quick fix that was causing a bit of obscure unit test to fail.
Also added a bit more output in the unit test to provide a bit of explanation beyond "query is not valid"

[ianhelle]

Some comments about use of logging vs logger

[ianhelle]

At least it is now building without errors! Just fix the logging stuff and we should be good to go.

[raj-axe]

I made a quick fix that was causing a bit of obscure unit test to fail. Also added a bit more output in the unit test to provide a bit of explanation beyond "query is not valid"

Thank you @ianhelle