[Prototype] Client-side caching option for Epidata.covidcast Python

docs/api/covid_hosp.md

@@ -24,7 +24,9 @@ General topics not specific to any particular data source are discussed in the
 ## Metadata
 
 This data source provides various measures of COVID-19 burden on patients and healthcare in the US.
-- Data source: [US Department of Health & Human Services](https://healthdata.gov/dataset/covid-19-reported-patient-impact-and-hospital-capacity-state-timeseries) (HHS)
+- Data source: US Department of Health & Human Services (HHS) [COVID-19 Reported Patient Impact and
+Hospital Capacity by State Timeseries](https://healthdata.gov/Hospital/COVID-19-Reported-Patient-Impact-and-Hospital-Capa/g62h-syeh) 
+  and [COVID-19 Reported Patient Impact and Hospital Capacity by State](https://healthdata.gov/dataset/COVID-19-Reported-Patient-Impact-and-Hospital-Capa/6xf2-c3ie)
 - Temporal Resolution: Daily, starting 2020-01-01
 - Spatial Resolution: US States plus DC, PR, and VI
 - Open access via [Open Data Commons Open Database License (ODbL)](https://opendatacommons.org/licenses/odbl/1.0/)

docs/api/covid_hosp_facility.md

@@ -20,7 +20,7 @@ General topics not specific to any particular data source are discussed in the
 ## Metadata
 
 This data source provides various measures of COVID-19 burden on patients and healthcare in the US.
-- Data source: [US Department of Health & Human Services](https://healthdata.gov/dataset/covid-19-reported-patient-impact-and-hospital-capacity-facility) (HHS)
+- Data source: [US Department of Health & Human Services](https://healthdata.gov/Hospital/COVID-19-Reported-Patient-Impact-and-Hospital-Capa/anag-cw7u) (HHS)
 - Geographic resolution: healthcare facility (address, city, zip, fips)
 - Temporal resolution: weekly (Friday -- Thursday)
 - First week: 2020-07-31

docs/api/covid_hosp_facility_lookup.md

@@ -24,7 +24,7 @@ General topics not specific to any particular data source are discussed in the
 ## Metadata
 
 This data source provides metadata about healthcare facilities in the US.
-- Data source: [US Department of Health & Human Services](https://healthdata.gov/dataset/covid-19-reported-patient-impact-and-hospital-capacity-facility) (HHS)
+- Data source: [US Department of Health & Human Services](https://healthdata.gov/Hospital/COVID-19-Reported-Patient-Impact-and-Hospital-Capa/anag-cw7u) (HHS)
 - Total number of facilities: 4922
 - Open access via [Open Data Commons Open Database License (ODbL)](https://opendatacommons.org/licenses/odbl/1.0/)
 

docs/api/covidcast-signals/hospital-admissions.md

@@ -22,6 +22,9 @@ hospital admissions, provided to us by health system partners. We use this
 inpatient data to estimate the percentage of new hospital admissions with a
 COVID-associated diagnosis code in a given location, on a given day.
 
+See also our [Health & Human Services](hhs.md) data source for official COVID
+hospitalization reporting from the Department of Health & Human Services.
+
 | Signal | Description |
 | --- | --- |
 | `smoothed_covid19_from_claims` | Estimated percentage of new hospital admissions with COVID-associated diagnoses, based on claims data from health system partners, smoothed in time using a Gaussian linear smoother <br/> **Earliest date available:** 2020-02-01 |

docs/api/covidcast-signals/safegraph.md

@@ -84,4 +84,4 @@ COVIDcast API.
 SafeGraph's Social Distancing Metrics and Weekly Patterns are based on mobile devices that are members of SafeGraph panels, which is not necessarily the same thing as measuring the general public. These counts do not represent absolute counts, and only count visits by members of the panel in that region. This can result in several biases:
 
 * **Geographic bias.** If some regions have a greater density of SafeGraph panel members as a percentage of the population than other regions, comparisons of metrics between regions may be biased. Regions with more SafeGraph panel members will appear to have more visits counted, even if the rate of visits in the general population is the same.
-* **Demographic bias.** SafeGraph panels may not be representative of the local population as a whole. For example, [some research suggests](https://arxiv.org/abs/2011.07194) that "older and non-white voters are less likely to be captured by mobility data", so this data will not accurately reflect behavior in those populations. Since population demographics vary across the United States, this can also contribute to geographic biases.
+* **Demographic bias.** SafeGraph panels may not be representative of the local population as a whole. For example, [some research suggests](https://doi.org/10.1145/3442188.3445881) that "older and non-white voters are less likely to be captured by mobility data", so this data will not accurately reflect behavior in those populations. Since population demographics vary across the United States, this can also contribute to geographic biases.

integrations/client/test_delphi_epidata.py

@@ -284,21 +284,28 @@ def test_covidcast(self):
       # check result
       self.assertEqual(response_1, {'message': 'no results', 'result': -2})
 
-  @patch('requests.post')
-  @patch('requests.get')
-  def test_request_method(self, get, post):
+  @patch('requests_cache.CachedSession')
+  @patch('requests.Session')
+  def test_request_method(self, _Session, _CachedSession):
     """Test that a GET request is default and POST is used if a 414 is returned."""
-    with self.subTest(name='get request'):
+    with self.subTest(name='get request, no cache'):
+      Session = MagicMock()
+      _Session.return_value = Session
       Epidata.covidcast('src', 'sig', 'day', 'county', 20200414, '01234')
-      get.assert_called_once()
-      post.assert_not_called()
+      Session.request.assert_called_once()
+    with self.subTest(name='get request, cache'):
+      CachedSession = MagicMock()
+      _CachedSession.return_value = CachedSession
+      Epidata.covidcast('src', 'sig', 'day', 'county', 20200414, '01234', cache_timeout=5)
+      CachedSession.request.assert_called_once()
     with self.subTest(name='post request'):
       mock_response = MagicMock()
       mock_response.status_code = 414
-      get.return_value = mock_response
+      Session = MagicMock()
+      Session.request.return_value = mock_response
+      _Session.return_value = Session
       Epidata.covidcast('src', 'sig', 'day', 'county', 20200414, '01234')
-      self.assertEqual(get.call_count, 2)  # one from post test and one from get test
-      post.assert_called_once()
+      assert Session.request.call_count == 2
 
   def test_geo_value(self):
     """test different variants of geo types: single, *, multi."""

src/client/delphi_epidata.py

@@ -9,13 +9,17 @@
 """
 
 # External modules
-import requests
+from typing import Union, Optional
+from datetime import timedelta, datetime
+from requests import Session
+from requests_cache import CachedSession
 import asyncio
-import warnings
 
 from aiohttp import ClientSession, TCPConnector
 from pkg_resources import get_distribution, DistributionNotFound
 
+CacheTime = Union[int, datetime, timedelta]
+
 # Obtain package version for the user-agent. Uses the installed version by
 # preference, even if you've installed it and then use this script independently
 # by accident.
@@ -54,7 +58,7 @@ def _list(values):
 
   # Helper function to request and parse epidata
   @staticmethod
-  def _request(params):
+  def _request(params, cache_timeout: Optional[CacheTime] = None):
     """Request and parse epidata.
 
     We default to GET since it has better caching and logging
@@ -63,9 +67,14 @@ def _request(params):
     """
     try:
       # API call
-      req = requests.get(Epidata.BASE_URL, params, headers=_HEADERS)
+      session = Session() if cache_timeout is None else CachedSession(
+        'covidcast_cache', expire_after=cache_timeout
+      )
+      req = session.request('get', Epidata.BASE_URL, params, headers=_HEADERS)
+      # Fallback to requests if we have to use POST
       if req.status_code == 414:
-        req = requests.post(Epidata.BASE_URL, params, headers=_HEADERS)
+        req = session.request('post', Epidata.BASE_URL, params, headers=_HEADERS)
+      session.close()
       return req.json()
     except Exception as e:
       # Something broke
@@ -601,8 +610,13 @@ def covidcast(
     if 'format' in kwargs:
       params['format'] = kwargs['format']
 
+    if 'cache_timeout' in kwargs:
+      cache_timeout = kwargs['cache_timeout']
+    else:
+      cache_timeout = None
+
     # Make the API call
-    return Epidata._request(params)
+    return Epidata._request(params, cache_timeout)
 
   # Fetch Delphi's COVID-19 Surveillance Streams metadata
   @staticmethod

tests/client/test_delphi_epidata.py

@@ -2,10 +2,15 @@
 
 # standard library
 import unittest
+import time
+from datetime import date
+
+import pandas as pd
 
 # py3tester coverage target
 __test_target__ = 'delphi.epidata.client.delphi_epidata'
 
+from delphi.epidata.client.delphi_epidata import Epidata
 
 class UnitTests(unittest.TestCase):
   """Basic unit tests."""