v5.2.0

What's Changed

Add susspellid to apc data

This field allows you to link to all of the other episodes in the spell. Will make the capacity conversion work simpler in the future. (adds susspellid to apc data by @tomjemmett in #258)

Changes to definition of maternity patients

In inpatients, we now capture more patients in the maternity definition by looking at the treatment function, as well as the admission method. We also ensure that the maternity group is applied to valid patients (only females between age of 13 and 55) (alters logic for group in inpatients by @tomjemmett in #259)

Changes to demographic modelling

Introduced a new column demog_type which is either "demographics" or "births". This makes it simpler to figure out which type of demographic modelling to apply in the model. It also has allowed us to choose the births adjustment for appropriate outpatient appointments.

• adds demog_type column by @tomjemmett in #261
• adds demog_type column to op/a&e aggregated data layer by @tomjemmett in #267
• fix demog type nulls by @tomjemmett in #268

Improved synthetic data generation

Working towards removing any disclosure risks from the synthetic data. Note that this has not yet been validated, so the synthetic data should still be treated as sensitive.

• Improve synth data by @tomjemmett in #246
• removes susspellid from synth data by @tomjemmett in #266
• removes the dataset column from saved ip file by @tomjemmett in #265

Other fixes/changes

• use updated policy for compute by @tomjemmett in #262
• Update extract by @tomjemmett in #263
• fix issue where merge fails because source and target have different columns by @tomjemmett in #264

Full Changelog: v5.0.0...v5.2.0

v5.0.0

What's Changed

Switch to UDAL

We've move away from our own infrastructure to use UDAL ( Switch over to UDAL by @tomjemmett in #237).

This has allowed us to add 2024/25 data to the pipeline (updates pipeline to extract 2024/25 data by @tomjemmett in #242).

We've also increased the maximum year supported to model to to 2047 (updates maximum year for demographics to 2047 by @tomjemmett in #249).

Other changes

• fixes issue with numerical rounding in frailty scores by @tomjemmett in #235
• fixes typo in docstrings by @tomjemmett in #241
• fixes issues with data extraction by @tomjemmett in #245
• properly counts op/a&e activity for HSA GAMs by @tomjemmett in #248
• removes databricks deploy workflow by @tomjemmett in #253
• Fix default view owners by @tomjemmett in #254
• removes custom population projections by @tomjemmett in #252

Full Changelog: v4.4.0...v5.0.0

v4.4.0

What's Changed

Added new population catchments #227

We now use the same logic for creating catchment populations for both the inputs rates data and the model demographics module. The logic has been slightly refined to be closer to OHID's earlier work to create hospital catchments.

Abstract inputs geography type by #230

We now have inputs data available both at provider level, and local authority (LAD23CD) level.

We also now include all providers in the inputs data, not just acute providers (#233)

Removes mental health providers from tables by #231

By including all providers in the inputs data, we introduced a new issue where mean lengths of stay for inpatients were being skewed by Mental Health providers, where patients can have significantly greater lengths of stays than in acute providers. We remove these providers from all of our data extracts.

We do this slightly later for ECDS data though, as some providers which are classed as Mental Health do run walk in centres/minor injury units (#234).

Other PR's

• fix typing issues by @tomjemmett in #228
• removes deprecated function by @tomjemmett in #229
• fix typing issues for ty 0.0.12 by @tomjemmett in #232

Full Changelog: v4.3.0...v4.4.0

v4.3.0

What's Changed

• Abstract table names by @tomjemmett in #225
• adds filter to remove long los (data quality) by @tomjemmett in #226

Full Changelog: v4.2.1...v4.3.0

v4.2.1

What's Changed

• adds script to generate synthetic data by @tomjemmett in #217
• uses correct specialty code by @tomjemmett in #219

Full Changelog: v4.2.0...v4.2.1

v4.2.0

What's Changed

Refactored to be a python package

The code is now a python package that can be built and deployed to databricks.

Switch to asset bundle deployments

All of the workflows are deployed via GitHub actions now on push to main.

Implemented inequalities changes

Inequalities is now calculated by both provider and ICB.

Data changes

• we were previously dropping some inpatients rows when they had a missing hsagrp. We now introduce an "unknown" group, so a very small amount of additional rows included
• we move the filtering of speldur to earlier in the pipeline. This only has a small impact on some of the mitigators and inputs data
• we have updated the day procedures code list to use the same filtering on procedure codes as the has_procedures column in inpatients, and based on 2023/24 rather than 2019/20.

PR's

• Update asset bundle deployment by @tomjemmett in #185
• add script to generate day procedures code list by @tomjemmett in #172
• updates readme by @tomjemmett in #192
• fix typing on evidence based interventions by @tomjemmett in #197
• fix typing is not null by @tomjemmett in #196
• fixes return types of functions by @tomjemmett in #198
• fix typing issues in ods trusts by @tomjemmett in #194
• fix typing issues by @tomjemmett in #199
• adds ty: ignore for monkey-patched methods by @tomjemmett in #201
• switches to databricks 16.4 runtime by @tomjemmett in #200
• add icb to inequalities by @yiwen-h in #195
• adds lint/typing actions by @tomjemmett in #203
• Add ability to support ICB model runs by @tomjemmett in #202
• fixes issue introduced in previous commit with null checks by @tomjemmett in #206
• fix wrong param in pop_by_imd_decile by @yiwen-h in #207
• add inequalities to pyproject.toml entrypoints by @yiwen-h in #208
• removes waiting list/covid adjustment inputs steps by @tomjemmett in #209
• correct fyear format and save path for inequalities data extract by @yiwen-h in #210
• fix day procedure mitigator by @tomjemmett in #212
• adds script to clean up spark files by @tomjemmett in #213
• adds can manage permissions to the service principal by @tomjemmett in #214
• fix issue with logic (prior bad regex replace) by @tomjemmett in #215
• add missing providers to inequalities extract by @yiwen-h in #216

Full Changelog: v4.1.0...v4.2.0

v4.1.0

What's Changed

• ensures all datasets have age filter by @tomjemmett in #167
• move steps from nhp_model into nhp_data by @tomjemmett in #166
• fixes typo in function call by @tomjemmett in #168
• ensures age column is created before using it by @tomjemmett in #169
• fixes typo by @tomjemmett in #170
• adds icb to ecds/op aggregated data by @tomjemmett in #171
• removes maternity 'mitigators' from inputs data by @tomjemmett in #174
• adds script to get the rtt incompletes data by @tomjemmett in #175
• adds script to generate the trust types parquet file by @tomjemmett in #176
• add ICB to population by imd decile table by @yiwen-h in #179
• fix duplicate age_group col in pop_by_imd_decile script by @yiwen-h in #180

Full Changelog: v4.0.0...v4.1.0

v4.0.0

What's Changed

Add 2022 pop projections #146

2022-based Subnational Population Projections have been released by ONS. We now use these for the demographic adjustment in the model. This PR introduced this new data into the code base. The following PR's are fixes to ensure that this actually worked.

• update provider catchment methodology by @tomjemmett in #151
• changes which schema the catchments table lives in by @tomjemmett in #153
• fix demographic extract by @tomjemmett in #154
• remove 2019 and 2022 from gams scripts, change principal_proj to migration_category by @yiwen-h in #155
• fixes birth extract logic by @tomjemmett in #158
• switches the file we use for the migration_category variant by @tomjemmett in #157
• adds local authority successor mappings by @tomjemmett in #156
• adds missing var_proj_zero_net_migration to snpp step by @tomjemmett in #159
• fix error with partitioning in custom birth factors by @yiwen-h in #160
• ensures we union in the base birth adjustment data with custom projections by @tomjemmett in #163

Alter tretspef cols #141

Previously, we had a bit of a bodge with the tretspef (treatment specialty function) column. When we extracted the data, we grouped these specialties to only keep the specialties used in RTT reporting. We later added in the ungrouped tretspef column, but in order not to break other code we added this as tretspef_raw.

We now handle this a bit more cleanly by having the actual values in tretspef, introducing a new column called tretspef_grouped. This has reduced the need for the same bit of code to be present in multiple locations, as we now handle the grouping early in the processing.

General tidying up

• adds scripts to recreate the default layer views by @tomjemmett in #143

• renames files to match the folder structure by @tomjemmett in #142

• fixes the extract workflow by @tomjemmett in #144

• alter workflows by @tomjemmett in #164

Full Changelog: v3.6.0...v4.0.0

v3.6.0

What's Changed

Custom Population Projections

• Add custom_projection for R0A to extracted demographic factors data by @yiwen-h in #97
• add custom RD8 projections to model data extract by @yiwen-h in #121
• fix custom birth factors by @tomjemmett in #127
• fixes issue with birth factors by @tomjemmett in #133

Data Changes

• switch to using both ccg of residence and responsibility by @tomjemmett in #108
• uses correct columns for ecds ccgs by @tomjemmett in #117
• switch EBI mitigators to age field by @tomjemmett in #110
• adds the ethnic category (ethnos) to the raw_data schema tables by @tomjemmett in #115
• adds the fyear/provider partition columns to mitigators table by @tomjemmett in #116
• force maternity spells to be Other (Medical) treatment function by @tomjemmett in #107

Refactors

• refactor virtual wards by @tomjemmett in #106
• change to SparkSession by @tomjemmett in #120
• importing from incorrect module by @tomjemmett in #124
• Split model data extract into separate files by @yiwen-h in #125
• updates data extract workflow for new file structure by @yiwen-h in #129
• update codeowners by @tomjemmett in #130
• fixes issue with search/replace by @tomjemmett in #131
• fix inputs pipeline by @tomjemmett in #132

Full Changelog: v3.5.0...v3.6.0

v3.5.0

What's Changed

Population Data

Previously, we had been using an older code base (nhp_demogr_module_inputs).
This code base was not in any automated processes and hasn't been re-run in a long time.

This process has now been moved across to run in databricks with the rest of our data pipelines.
Also, they are now ready for switching to newer population projections when ONS releases them.

• add population projections by @tomjemmett in #98
• fixes issue with variant projections by @tomjemmett in #105

Health Status Adjustment GAMs

A bug was discovered in the way we generate the activity rates which are used by the GAMs for Health Status Adjustment.
When we work out the amount of activity by provider, we join to the catchment population for that provider.
However, the join was incorrect and was not including the provider, so we were dividing by all of the populations (for every provider).

• fix gams by @tomjemmett in #103
• Fix GAMs part deux by @tomjemmett in #104

Full Changelog: v3.4.0...v3.5.0