functionality to retrieve data from updated NHSN hospital respiratory data stream

[vpnagraj]

as of November 2024, the NHSN weekly hospital respiratory data (including flu admissions) are being distributed at via a new CDC API:

https://data.cdc.gov/Public-Health-Surveillance/Weekly-Hospital-Respiratory-Data-HRD-Metrics-by-Ju/ua7e-t2fy/about_data

this data is aggregated to the weekly resolution. we need to add a function to retrieve and prep this data.

[vpnagraj]

this needs to be more fully thought through / tested / documented ...

but see below for an initial sketch of the function:

get_nhsn_weekly <- function(endpoint = "https://data.cdc.gov/api/views/ua7e-t2fy/rows.csv",
                            adjust_partial = TRUE,
                            statesonly=TRUE,
                            min_per_week=1,
                            augment = FALSE,
                            augment_stop = "2020-10-18") {
  
  dat <-
    readr::read_csv(endpoint) %>%
    dplyr::select(week_end = `Week Ending Date`, abbreviation = `Geographic aggregation`, `Total Influenza Admissions`, `Percent Hospitals Reporting Influenza Admissions`)
  
  if(adjust_partial) {
    dat <- 
      dat %>%
      dplyr::group_by(abbreviation) %>%
      dplyr::mutate(max_reporting = max(`Percent Hospitals Reporting Influenza Admissions`, na.rm = TRUE)) %>%
      dplyr::ungroup() %>%
      dplyr::mutate(flu.admits = round((max_reporting/`Percent Hospitals Reporting Influenza Admissions`)*`Total Influenza Admissions`)) 
  } else {
    dat <- 
      dat %>%
      dplyr::mutate(flu.admits = round(`Total Influenza Admissions`))
  }
  
  ## if the augment option is set then prepend with the imputed data
  if(augment) {
    dat <-
      nhsn_imputed %>%
      dplyr::select(abbreviation = location, week_end = date, flu.admits = mean_flu_admits) %>%
      ## NOTE: augmentation includes data that extends beyond the date we have used to trim HHS data previously
      ## this line (in combination with filter in the next line) ...
      ## ... will ensure that the data pulled from healthdata.gov is used for all dates after augment_stop date
      dplyr::filter(week_end <= as.Date(augment_stop)) %>%
      dplyr::bind_rows(dplyr::filter(dat, week_end > as.Date(augment_stop)),.) %>%
      dplyr::arrange(abbreviation, week_end)
  }
  
  
  dat <-
    dat %>%
    dplyr::select(abbreviation, flu.admits, week_end) %>%
    dplyr::left_join(locations) %>%
    dplyr::mutate(epiyear=lubridate::epiyear(week_end), epiweek=lubridate::epiweek(week_end)) %>%
    dplyr::mutate(week_start = mmwr_week_to_date(epiyear, epiweek, 1)) %>%
    dplyr::mutate(monday = mmwr_week_to_date(epiyear, epiweek, 2)) %>%
    dplyr::left_join(historical_severity, by="epiweek")
    
    # Trim to US+states+DC only
    if (statesonly) {
      message("Filtering to US+DC+States only")
      dat <-
        dat %>%
        dplyr::filter(location %in% c("US", stringr::str_pad(1:56, width=2, pad="0")))
    }
  
  dat
}

ill refine that ^ in the coming-soon commits to https://github.com/signaturescience/fiphde/tree/225-functionality-to-retrieve-data-from-updated-nhsn-hospital-respiratory-data-stream

[vpnagraj]

this functionality is drafted and now on the dev branch. the retrieval script works for both the preliminary (https://data.cdc.gov/Public-Health-Surveillance/Weekly-Hospital-Respiratory-Data-HRD-Metrics-by-Ju/mpgq-jmmr/about_data) and final (https://data.cdc.gov/Public-Health-Surveillance/Weekly-Hospital-Respiratory-Data-HRD-Metrics-by-Ju/ua7e-t2fy/about_data) APIs.