Add key_storage, isolation_level and lock_manager configuration to cache policies

[desertaxle]

This PR adds the ability to specify a storage location for cache entries via a CachePolicy. The isolation level for the cache and locking mechanism are also configurable via a CachePolicy.

Example

Cache task results based on the input and source code a task. Store those cache entries at ~/.prefect/cache. Allow only one actor to access a cache entry at a time and store lock files to enforce that at ~.prefect/locks.

from prefect import task
from prefect.cache_policies import INPUTS, TASK_SOURCE
from prefect.transactions import IsolationLevel


@task(
    cache_policy=(INPUTS + TASK_SOURCE).configure(
        key_storage="~/.prefect/cache",
        lock_manager="~/.prefect/locks",
        isolation_level=IsolationLevel.SERIALIZABLE,
    )
)
def my_task(x):
    return x


if __name__ == "__main__":
    print(my_task(10))

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• If this pull request adds new functionality, it includes unit tests that cover the changes
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[codspeed-hq]

CodSpeed Performance Report

Merging #15382 will not alter performance

_{Comparing cache-policy-enhancements (8f18646) with main (387cbe3)}

Summary

✅ 3 untouched benchmarks

[cicdw]

One note on locks as a kwarg name, otherwise this is great

[cicdw]

note for future: I think we should refactor many of these errors into special prefect.exceptions.ConfigurationError exceptions for clarity and consistency

[cicdw]

this looks awesome -- left one small comment about the deepcopy thing, and one question that isn't super important but I wanted to give it some airtime just in case. cc: @zzstoatzz in case you have thoughts about the kwarg name storage

[cicdw]

yea it's longer but I do like this more - very clear

[cicdw]

Only because now is the time to ask these questions - are there any other name options we have for this one, just to make crystal clear it's not the same thing as result storage?

[desertaxle]

Another possibility could be cache_storage. It'd be a tad redundant, but it'd be explicit.

[cicdw]

A few other ideas just to explore our options:

• key_storage
• metadata_storage (mirrors the internal API)
• key_files
• metadata_files
• cache_files
• cache_data
• metadata

[desertaxle]

key_storage jumps out as a good option to me. I think I'll change to that.