JHU changed Puerto Rico death reporting, causing us to fail to report it

[capnrefsmmat]

There is no Puerto Rico cases or deaths data available in the API since July 17:

> covidcast_signal("jhu-csse", "deaths_incidence_num", geo_type="state", geo_values="pr", start_day="2020-07-15")
A `covidcast_signal` data frame with 3 rows and 10 columns.

signals     : jhu-csse:deaths_incidence_num
geo_type    : state

  geo_value time_value direction      issue lag value stderr sample_size
1        pr 2020-07-15        NA 2020-07-18   3     2     NA          NA
2        pr 2020-07-16        NA 2020-07-18   2     1     NA          NA
3        pr 2020-07-17        NA 2020-07-18   1     5     NA          NA

> covidcast_signal("jhu-csse", "confirmed_incidence_num", geo_type="state", geo_values="pr", start_day="2020-07-15")
A `covidcast_signal` data frame with 3 rows and 10 columns.

signals     : jhu-csse:confirmed_incidence_num
geo_type    : state

  geo_value time_value direction      issue lag value stderr sample_size
1        pr 2020-07-15        NA 2020-07-18   3   256     NA          NA
2        pr 2020-07-16        NA 2020-07-18   2   195     NA          NA
3        pr 2020-07-17        NA 2020-07-18   1   546     NA          NA

The JHU time series of deaths seems to support this, showing 0 deaths for all time in every county in Puerto Rico -- but that's because the deaths are listed under "Unassigned, Puerto Rico". We should be ingesting these deaths.

Meanwhile, their time series of confirmed cases shows plenty of cases, but for some reason we are not reporting them.

This is preventing forecasting from issuing death forecasts for Puerto Rico, and will block case forecasts as well.

[capnrefsmmat]

See also CSSEGISandData/COVID-19#2889

[dshemetov]

I'm looking into tackling this with #215. I'm thinking of splitting the deaths across the FIPS codes based on population data, so that we don't mix state level data with FIPS level data. Would this cause issues down the pipeline? It would help making the geocoding consistent. We can reaggregate the deaths back into commonwealth level and serve only that at the API.

[ajgreen93]

@krivard , @capnrefsmmat asked me to ping you. The issue with "Unassigned" counts for JHU data seems to be affecting states like Wyoming and Rhode Island (and presumably all other states as well). As a result, it is affecting our state-level forecasts for all states.

(This also seems related to this closed issue.)

[krivard]

In theory, a behavior where we map unassigned cases/deaths to a megacounty that then gets aggregated into the state figures was added in commit 5ff04c0. This commit is present in the commit log for the version of the JHU indicator in production, so if it's not doing the right thing now, then it's likely this was later overridden when we switched to the new geo aggregator. @dshemetov, investigate? it may be worth boosting the priority on merging #215; what would we need to get that done in the next two weeks?

In the meantime, @ajgreen93 iirc we added USAFacts to avoid this exact issue, so you might try switching indicators.

[dshemetov]

Just looked: yes, in deploy-jhu UIDs 840900XX are being mapped to 900XX without being converted to XX000 (the mega-county fix). This is already handled in #217.

[dshemetov]

@krivard Two week merging of #215 is likely doable. What's left are HHS, National, and DMA level geocodes. DMA may take some work to track down the crosswalks.

[dshemetov]

Just looked into the Puerto Rico issues. Like capnrefsmmat reports, we still don't have Puerto Rico cases information in the API after July 17th, despite the data being present in the JHU time_series we pull. Here is the strange thing: the Puerto Rico cases after July 17th are showing up in the receiving folder on the deploy-jhu branch. So I think the issue must be on the ingestion step after we pull.

The Puerto Rico deaths issue has been fixed with the megaFIPS fix.

[krivard]

We are successfully ingesting data from Puerto Rico for the following combinations: (state, county) X (cases, deaths) X (incidence, cumulative) X (num):

$ zgrep -il -e "^72" -e "^pr" /common/covidcast/archive/successful/jhu-csse/20200908*
/common/covidcast/archive/successful/jhu-csse/20200908_county_confirmed_7dav_cumulative_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_county_confirmed_7dav_incidence_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_county_confirmed_cumulative_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_county_confirmed_incidence_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_county_deaths_7dav_cumulative_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_county_deaths_7dav_incidence_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_county_deaths_cumulative_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_county_deaths_incidence_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_state_confirmed_7dav_cumulative_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_state_confirmed_7dav_incidence_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_state_confirmed_cumulative_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_state_confirmed_incidence_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_state_deaths_7dav_cumulative_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_state_deaths_7dav_incidence_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_state_deaths_cumulative_num.csv.gz
/common/covidcast/archive/successful/jhu-csse/20200908_state_deaths_incidence_num.csv.gz

For (state) X (cases, deaths) X (incidence, cumulative) X (prop), the deploy-jhu pipeline is generating PR data, but it puts inf in the value column, which is not permitted:

$ grep -i "^pr" bad-jhu/20200908*
bad-jhu/20200908_state_confirmed_7dav_cumulative_prop.csv:PR,inf,NA,NA
bad-jhu/20200908_state_confirmed_7dav_incidence_prop.csv:PR,inf,NA,NA
bad-jhu/20200908_state_confirmed_cumulative_prop.csv:PR,inf,NA,NA
bad-jhu/20200908_state_confirmed_incidence_prop.csv:PR,inf,NA,NA
bad-jhu/20200908_state_deaths_7dav_cumulative_prop.csv:PR,inf,NA,NA
bad-jhu/20200908_state_deaths_cumulative_prop.csv:PR,inf,NA,NA
bad-jhu/20200908_state_deaths_incidence_prop.csv:PR,inf,NA,NA

(this is probably related to #227)

Since this would cause the ingestion mechanism to reject the whole file, while we're waiting for fixes on #227 and #254 I have a cron job running to pick up the JHU files from receiving on the server, and strip out lines with illegal values and geo identifiers before they reach ingestion. This will probably wreak havoc with the diff-based archive utility once we have fixes in place, but it was a better option than having state cases and deaths ratios be completely unavailable.

For (county) X (cases, deaths) X (incidence, cumulative) X (prop), the deploy-jhu pipeline does not appear to be generating any entries for Puerto Rico counties at all.

[dshemetov]

@krivard can you check the same thing but for 20200826? For some reason, our API provides Puerto Rico cases since 20200827, but not before.

>cc_df = covidcast.signal("jhu-csse", "confirmed_incidence_num",
                        date(2020, 7, 14), date(2020, 9, 11),
                        geo_type="county")
>cc_df[cc_df["geo_value"].isin([str(x) for x in range(72001, 72999)])]["time_value"].min()
Timestamp('2020-08-27 00:00:00')

It is likely a population divide by zero issue, but not sure why it's day-dependent?

[krivard]

There are no mentions of `72XXX` counties before 31 August in the success files. We only keep backups of the most recently submitted csv for each day, so any day with PR for an earlier issue (before the inf bug) will have been overwritten by a more recent issue (after the inf bug) which has had all the invalid PR lines filtered out. Recall however that the issue definition for JHU has changed multiple times, including: 1. all days going back to 2 February 2. only the last 7 days of raw data and the last 1 day of 7dav data 3. only the new or updated lines in any new or updated file going back to 2 February so any appearance of day-dependence of a zero population effect may instead be related to which days of data were in the issue when the population data first went awry.

…

On Fri, Sep 11, 2020 at 12:33 PM Dmitry Shemetov ***@***.***> wrote: @krivard <https://github.com/krivard> can you check the same thing but for 20200826? For some reason, our API provides Puerto Rico cases since 20200827, but not before. >cc_df = covidcast.signal("jhu-csse", "confirmed_incidence_num", date(2020, 7, 14), date(2020, 9, 11), geo_type="county") >cc_df[cc_df["geo_value"].isin([str(x) for x in range(72001, 72999)])]["time_value"].min() Timestamp('2020-08-27 00:00:00') It is likely a population divide by zero issue, but I'm not sure why we have a day-dependent bug. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#179 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI24CXJ3L7R5O3HEWLCPZLSFJGMJANCNFSM4PTMBBRQ> .

[krivard]

Here's a plot of the number of days in each issue from 2 July to 10 Sept where county data for 72000 (the PR megacounty) or 72001 are available:

df <- suppressMessages(covidcast_signal("jhu-csse","confirmed_incidence_num","2020-07-01","2020-09-09",
                       "county",c("72000","72001"),issues=c("2020-07-02","2020-09-10")))
dfn <- group_by(df,geo_value,issue) %>% summarise(n=n())
ggplot(dfn, aes(x=issue, y=n, group=geo_value, color=geo_value)) + geom_line() + ggtitle("number of dates of available data") 

So it looks like we supported PR mega counties through mid-July, then picked up on individual county data on 27 August.

In theory the diff-based issue generator should reissue PR county data back to 2 Feb as soon as it becomes available and valid; in practice we may have to babysit it a bit.