Implementing a more robust caching mechanism #814
Description
We need to have a more robust function/kernel caching mechanism. An example could be something like that has been implemented in numba.
https://numba.readthedocs.io/en/stable/developer/caching.html
https://github.com/numba/numba/blob/main/numba/core/caching.py
However the numba implementation uses files to save compiled (pickled) kernels. Here are some ideas:
- Instead of using files, we can keep using the memory (dictionary)
- Implement a queuing mechanism that prioritizes the compiled functions/kernels based on cache hit.
- Optimize the file access: we will start saving in file only when the size of the dictionary reaches a certain limit, etc.
The caching will be used here:
https://github.com/IntelPython/numba-dpex/blob/main/numba_dpex/compiler.py