geomap refactor refactor

[dshemetov]

Some outstanding fixes for the geomap utility refactor #137

• documentation
• remove full flag
• cross files
• • zip -> county : population weighted
• • zip -> fips : population weighted
• • zip -> hrr : unweighted
• • county -> state : unweighted
• • county -> msa : unweighted
• • county -> megacounty
• • fips -> hrr
• • zip -> state
• • zip -> msa
• • state code -> hhs region number
• • state code -> us national
• rethink nan handling. In many to many mappings (like zip to fips), we zero-fill nans in the aggregation step. This conflicts with smoothing, where it would best to distinguish between true zeros and zero filled nans.
• • Update: no better solution found than to zero-fill when aggregating.

[dshemetov]

How do we know that our geo codes are the same as those used by our data sources? For instance, what authority does Google use for their coding for msa and hrr? Our matching for msa and hrr appears to come from a variety of tables (I'm still trying to figure out from exactly where) and I believe that it's a many to many mapping (though I may be wrong).

This seems like an important point to get right. Our work in finding the most up to date codes might be wasted if the sources don't match.

[krivard]

Our data sources typically are keyed to a single fine-grained geo level. For example, the surveys and quidel tests come in with zip code, and hospital-admissions are tagged with a hospital identifier. GHT is an exception, since they presumably have IP addresses on their end but they aggregate to designated market areas (dma) before sending to us. The GHT team at Google no longer exists, so it’s very likely their geo mappings are out of date. I wouldn’t spend too much time worrying about it just now; Google is working on a new search trends tool as we speak. Once it’s released, we can revisit.

…

On Wed, Aug 12, 2020 at 11:37 PM Dmitry Shemetov ***@***.***> wrote: How do we know that we are using the same geo codes as our data sources? For instance, what authority does Google use for their coding <https://github.com/cmu-delphi/delphi-epidata/blob/e7cd1012d2046980571a3ded6db6803f64e4506b/docs/api/covidcast-signals/ght.md> for msa and hrr? Our matching for msa and hrr appears to come from a variety of tables. This seems like an important point to get right. Our work in finding the most up to date codes might be wasted if the sources don't match. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#215 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI24CSTRUOHTP2OG5XSR63SANNV3ANCNFSM4P5FVFXQ> .

[krivard]

Caching mapping files for each pair of region types src/dest as binary pickle files is fine; it's just a new set of static files and we already do that.

Region type additions:

• HHS: https://www.hhs.gov/about/agencies/iea/regional-offices/index.html
• National (USA)

[dshemetov]

DMA regions were also requested in another thread. Are these the ones by Nielsen?

[krivard]

yes -- they're needed for GHT.

[dshemetov]

Got it, I'm looking for a ZIP to DMA mapping.

[dshemetov]

Can I get further specifications for the National code? Is that more than just an aggregation across states?

[krivard]

No, you have it right — just an aggregation across states. There’s a sample implementation in the fb-package branch (in R) if you’re looking for something to match.

…

On Tue, Sep 8, 2020 at 6:20 PM Dmitry Shemetov ***@***.***> wrote: Can I get further specifications for the National code? Is that more than just an aggregation across states? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#215 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI24CRI372B2UZ5NRYRH3LSE2U23ANCNFSM4P5FVFXQ> .

[dshemetov]

Sorry, I can't find this implementation. It's in this repo somewhere?

[capnrefsmmat]

https://github.com/cmu-delphi/covidcast-indicators/blob/fb-package/facebook/delphiFacebook/R/geo.R