Enhance the cache ability

[SimFG]

Is your feature request related to a problem? Please describe.

Amazing project!!! And it really appeals to me. After carefully reading the readme and browsing the code structure, I find the concept of cache already exists, which's key is composed of parameters and llm information.

I'm considering enhancing the cache ability using semantic cache, but due to my limited knowledge of the project and lack of understanding of its core concept, I'm unsure if this approach is suitable for the current scenario.

If it is, I would be happy to contribute by submitting a pull request.

Describe the solution you'd like

GPTCache is a semantic cache for LLMS.

[slundberg]

Hey! I think semantic cache support would be great. Right now there is a cache property that is on the LLM object it is used like a key/value store. You could directly replace that with GPTCache, but the key would not be in a nice format for semantic lookup.

I think the best path forward would be to define a nice simple API for the cache object, and then let people replace that with any caching solution they want. A first proposal for the API would be:

class Cache:
    def __getitem__(self, key):
        ''' get an item from the cache or throw key error '''
        pass

    def __setitem__(self, key, value):
        ''' set an item in the cache'''
        pass

    def __contains__(self, key):
        ''' see if we can return a cached value for the passed key '''

    def create_key(self, llm, **kwargs):
        ''' Define a lookup key for a call to the given llm with the given kwargs.
        One of the keyword args could be `cache_key` in which case this function should respect that
        and use it.
        '''

If we define an API like above, then the current disk cache solution could be the default, and GPTCache can be a really simple implementation of that API that could be swapped in, as could any other caching solution people come up with.

Thoughts on what this is missing?

[SimFG]

Thank you very much for your reply, I will provide a pr to enhance the ability of the cache in the next few days.

[TomNeyland]

@SimFG @slundberg I was just chewing this over when I figured I better check to see if anyone else was thinking about this. In my case am looking to switch out the disk cache with redis/memcache maybe even a db.

In line with what @slundberg suggested I was thinking

import json
import hashlib
from typing import Any, Dict
from abc import ABC, abstractmethod

class Cache(ABC):
    @abstractmethod
    def __getitem__(self, key: str) -> str:
        ''' get an item from the cache or throw key error '''
        pass

    @abstractmethod
    def __setitem__(self, key: str, value: str) -> None:
        ''' set an item in the cache'''
        pass

    @abstractmethod
    def __contains__(self, key: str) -> bool:
        ''' see if we can return a cached value for the passed key '''
        pass

    def create_key(self, llm: str, **kwargs: Dict[str, Any]) -> str:
        ''' Define a lookup key for a call to the given llm with the given kwargs.
        One of the keyword args could be `cache_key` in which case this function should respect that
        and use it.
        '''
        if 'cache_key' in kwargs:
            return kwargs['cache_key']

        hasher = hashlib.md5()
        options_str = json.dumps(kwargs, sort_keys=True)

        combined = "{}{}".format(llm, options_str).encode()

        hasher.update(combined)
        return hasher.hexdigest()


class RedisCache(Cache):
    def __init__(self, host: str = 'localhost', port: int = 6379, db: int = 0):
        try:
            import redis
            self._redis = redis.Redis(host=host, port=port, db=db)
        except ImportError:
            raise Exception("Please install the 'redis' library to use RedisCache.")

    def __getitem__(self, key: str) -> str:
        result = self._redis.get(key)
        if result is None:
            raise KeyError("Key not found in RedisCache: " + key)
        return result.decode('utf-8')

    def __setitem__(self, key: str, value: str) -> None:
        self._redis.set(key, value)

    def __contains__(self, key: str) -> bool:
        return self._redis.exists(key) == 1


class Memcache(Cache):
    def __init__(self, server: str = 'localhost', port: int = 11211):
        try:
            from pymemcache.client import base
            self._client = base.Client((server, port))
        except ImportError:
            raise Exception("Please install the 'pymemcache' library to use Memcache.")

    def __getitem__(self, key: str) -> str:
        result = self._client.get(key)
        if result is None:
            raise KeyError("Key not found in Memcache: " + key)
        return result.decode('utf-8')

    def __setitem__(self, key: str, value: str) -> None:
        self._client.set(key, value)

    def __contains__(self, key: str) -> bool:
        return self._client.get(key) is not None

Note: The connection params and key generation are just roughed in here, they need more thought