Support a `filter_by` option in Gaurd

[ketgo]

I am not sure if the following fits with the future vision of the package, but wanted to see if it can help resolve the issue -- all policy retrieval from Storage for Regex and Rule based checker.

Provide a filter_by option in Guard initialization, or the Guard.is_allowed method. This value can be then used by the Storage to filter policies on DB. In fact, if MongoEngine back-end is used to implement the MongoStorage, the structure of filter_by can be the same as that used in MongoEngine package. The other option is to just let it be a MongoDB query json.

Since the filter_by is DB dependent, a unified interface to create these filters that can be used for all storage types might be useful and part of a future feature. Some indexing strategy will also be useful.

Overall the approach has the following pros and cons:

Pros:
DB level filtering of policies before evaluation resulting in faster performance.

Cons:
It is possible for a user to use a filter that may result in some policies to be missed for evaluation.

I don't think the con is a big issue since the filter_by option can be left for advance usage with a user warning in the docs.

[kolotaev]

Probably, I'm missing the point, but what is the benefit of using a separate filter_by option that user can pass to Guard and down to a particular storage?

My view of the situation is the following:
Requirements:

• We need to be able to filter on storage level for better performance.
• We need to support storage-level filtering for String (regex)-based policies.
• We need to support storage-level filtering for Rule-based policies.
• We need to support ability to use custom Rules and thus Rule-based policies defined with them.

What is needed to be done:

• Within private filter methods for any given particular Storage we can figure out the resulting filter that later be used in find_for_inquiry. Here, based on the Inquiry and Checker we can define how to construct a filter.
• Any particular Storage must adjust/implement write part of its interface to store policies in such a way that will be appropriate and convenient for the later querying with the query constructed with the aforementioned filter method (if needed, of course). Thus, it's very Storage-specific. Based on the Policy information it can be achieved both for String- and Rule-based policies.
• In order to support custom Rules we need a way to pass some kind of a mapper that will give information or hint for a Storage on how to store and/or query policies with custom rules.
• If we modify existing Storages, we need to add a migration that will adjust saved policies and add indexes where appropriate.

@ketgo, what do you think?

[ketgo]

I agree with the laid out requirements and what is needed to be done.

Regarding the filter_by option, I had not thought through it completely earlier so was unclear. The idea here is to provide policy grouping. This grouping can be by a key or ID. This way the storage can then retrieve only those policies which belong to the same group during evaluation. Thus the group ID will need to be passed during inquiry by the Guard, or it can be part of the Inquiry. There are a couple of use-cases that I can think of where this might be useful:

1. Segregation of policies based on customers, accounts, etc. This way evaluation of polices for account 'A' or customer A will not require evaluation of those for account B or customer B.
2. Provisioning of policy creation limits, e.g. AWS IAM limits.

As can be seen, a good grouping strategy will result in better policy evaluation performance.

[kolotaev]

Yes, absolutely agree with you. Grouping will let us shard the data-set for a small query scope which will result in much better performance.
I've already thought about it, but in a little bit different context: I was thinking of creating an AuthZ server that uses vakt underneath and exposes a convenient API (REST, gRPC, etc.) for managing policies and checking policy enforcement. So I thought about this "sharding" of policies for various users, tenants, etc.
So, it's a good idea to think in terms of adding this group functionality in core vakt. Let's keep this in mind.