File caching for the oauth2client >= 4.0.0

[xmedeko]

@tmatsuo proposed in #110 to cache discovery.build(), however, #294 has stopped using cache outside of the GAE (appengine) environment (see also #299). E.g. the discovery.build() is not cached in the CGE environment. Is it possible to bring the file caching back, or provide some example of file cache (subclass of googleapiclient.discovery_cache.Cache)?

[theacodes]

Sure, mostly what needs to happen is that the current file cache needs to be updated to not use oauth2client's locked file.

…

On Sun, Jan 8, 2017, 2:14 PM Ondrej Medek ***@***.***> wrote: @tmatsuo <https://github.com/tmatsuo> proposed in #110 <#110> to cache discovery.build(), however, #294 <#294> has stopped using cache outside of the GAE (appengine) environment (see also #299 <#299>). E.g. the discovery.build() is not cached in the CGE environment. Is it possible to bring the file caching back, or provide some example of file cache (subclass of googleapiclient.discovery_cache.Cache)? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#325>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc2WtGDCjrz4H1QhftQxypDuOidnpks5rQV-6gaJpZM4Ld0fj> .

[Schweigi]

Is there a known workaround in the meantime?

[Schweigi]

I created a small workaround by using a in memory cache (see example below). Does anyone know if there should be a regular cache invalidation? Currently the memory cache is cleared on each deployment, I hope that's ok for the time being.

In memory cache:

class MemoryCache(Cache):
    _CACHE = {}

    def get(self, url):
        return MemoryCache._CACHE.get(url)

    def set(self, url, content):
        MemoryCache._CACHE[url] = content

Usage:

http = ...
service = discovery.build('calendar', 'v3', http=http, cache=MemoryCache())

[xmedeko]

AFAIK the Google production API should not change (the same version of API), so the you do not need to invalidate the cache. However, the discovery_cache/__init__.py defines the limit DISCOVERY_DOC_MAX_AGE as one day.

+1 for a neat, simple MemoryCache. IMO it's good when you have a single, long running process.

If you want to share the cache among processes, (like running scripts by the cron,) you may copy the old good oauth2client locked_file.py and then to plug it somehow with discovery_cache/file_cache.py. Beware, the locked_file.py is considered not 100% save, so you take the risk. (But works well in practice.) By "plugin somehow" I mean either copy & change the discovery_cache/file_cache.py to take your locked_file.py or to trick the Python to think your locked_file.py is from the oauth2client.locked_file package.

[xmedeko]

@jonparrott what about to change file lock to the multiprocessing.RLock() in the original locked_file.py? Would be it sufficient solution to accept back to the repo?

[theacodes]

I think it's likely possible to do this without any sort of locking.

…

On Wed, May 17, 2017, 12:02 AM Ondrej Medek ***@***.***> wrote: @jonparrott <https://github.com/jonparrott> what about to change file lock to the multiprocessing.RLock() in the original locked_file.py? Would be it sufficient solution to accept back to the repo? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#325 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc-ne-ZS8p4rfQbV1qs3RtjXgc7Opks5r6puOgaJpZM4Ld0fj> .

[xmedeko]

You mean a file cache, shared among multiple processes, without any locking? How to prevent data corruption, when 2 process write to the same file?

[theacodes]

Likely using a folder containing a file for each cached discovery doc. No locking needed. Worst case scenario two processes write the same file twice.

…

On Wed, May 17, 2017, 12:08 AM Ondrej Medek ***@***.***> wrote: You mean a file cache, shared among multiple processes, without any locking? How to prevent data corruption, when 2 process write to the same file? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#325 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc9PWEHEBR_dWKW4KmNhmgV4lH8qiks5r6pzbgaJpZM4Ld0fj> .