Commit for 2023-24 publication code

NHSDigital · Dec 4, 2024 · 0cd098c · 0cd098c
1 parent a27896d
commit 0cd098c
Show file tree

Hide file tree

Showing 44 changed files with 20,029 additions and 1 deletion.
diff --git a/LICENSE b/LICENSE
@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2024 NHS Digital
+Copyright (c) 2023 NHS Digital
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

diff --git a/README.md b/README.md
@@ -0,0 +1,55 @@
+# Mental Health Act Annual Statistics
+
+**_Warning - this repository is a snapshot of a repository internal to NHS England. This means that links to videos and some URLs may not work._**
+
+**Repository owner:** Analytical Services: Community and Mental Health
+
+**Email:** england.mentalhealthanalysis@nhs.net
+
+To contact us raise an issue on Github or via email and we will respond promptly.
+
+## Introduction
+
+This codebase is used in the creation of the Mental Health Act Annual Statistics publication. This repository includes all of the code used to create the CSV and the associated outputs. The publication uses the Mental Health Services Dataset (MHSDS), further information about the dataset can be found at https://digital.nhs.uk/data-and-information/data-collections-and-data-sets/data-sets/mental-health-services-data-set.
+
+The full publication series can be found at https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-act-statistics-annual-figures. 
+
+Other Mental Health related publications and dashboards can be found at the Mental Health Data Hub: https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/mental-health-data-hub 
+
+## Installation and running
+Please note that the code included in this project is designed to be run on Databricks within the NHS England systems. As such some of the code included here may not run on other MHSDS assets. The logic and methods used to produce the metrics included in this codebase remain the same though. 
+
+## Understanding the Mental Health Services Dataset
+
+MHSDS is collected on a monthly basis from providers of secondary mental health services. On average around 210 million rows of data flow into the dataset on a monthly basis. More information on the data quality of the dataset, including the numbers of providers submitting data and the volumes of data flowing to each table can be found in the Data Quality Dashboard: https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/mental-health-data-hub/data-quality/mental-health-services-dataset---data-quality-dashboard 
+
+The MHSDS tables and fields used within the code are all documented within the MHSDS tools and guidance. This guidance can be found here: https://digital.nhs.uk/data-and-information/data-collections-and-data-sets/data-sets/mental-health-services-data-set/tools-and-guidance
+
+Within the guidance are three key documents:
+
+1) MHSDS Technical Output Specification - This provides technical details of all of the fields and tables contained within the dataset. It also contains details of the validations applied to specific tables and fields. The specification also includes details of derivations and how they are constructed.
+2) MHSDS Data Model - This details all of the tables and fields within the dataset and how they relate to each other.
+3) MHSDS User Guidance - This document provides details of all of the tables and fields within the dataset and gives examples of how a user might populate the data in certain scenarios.
+
+Additionally, users might want to consult the Data Dictionary for specific fields within the dataset: https://www.datadictionary.nhs.uk/ 
+
+## Appendix and Notes
+
+In places the notebooks above use some acronyms. The main ones used are as follows:
+
+- MHSDS: Mental Health Services Dataset
+- CCG: Clinical Commissioning Group. These were replaced by Sub Integrated Care Boards (ICBs) in July 2022.
+- ICB: Integrated Care Board. These came into effect on July 1st 2022. Further information can be found at https://www.kingsfund.org.uk/publications/integrated-care-systems-explained#development.
+- Provider: The organisation is providing care. This is also the submitter of MHSDS data
+-LA: Local Authority
+
+## Support
+If you have any questions or issues regarding the constructions or code within this repository please contact mh.analysis@nhs.net
+
+## Authors and acknowledgment
+Community and Mental Health Team, NHS England
+mh.analysis@nhs.net
+
+## License
+The menh_bbrb codebase is released under the MIT License.
+The documentation is © Crown copyright and available under the terms of the [Open Government 3.0] (https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/) licence.
diff --git a/mha-annual-main/README.md b/mha-annual-main/README.md
@@ -0,0 +1,55 @@
+# Mental Health Act Annual Statistics
+
+**_Warning - this repository is a snapshot of a repository internal to NHS England. This means that links to videos and some URLs may not work._**
+
+**Repository owner:** Analytical Services: Community and Mental Health
+
+**Email:** mh.analysis@nhs.net
+
+To contact us raise an issue on Github or via email and we will respond promptly.
+
+## Introduction
+
+This codebase is used in the creation of the Mental Health Act Annual Statistics publication. This repository includes all of the code used to create the CSV and the associated outputs. The publication uses the Mental Health Services Dataset (MHSDS), further information about the dataset can be found at https://digital.nhs.uk/data-and-information/data-collections-and-data-sets/data-sets/mental-health-services-data-set.
+
+The full publication series can be found at https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-act-statistics-annual-figures. 
+
+Other Mental Health related publications and dashboards can be found at the Mental Health Data Hub: https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/mental-health-data-hub 
+
+## Installation and running
+Please note that the code included in this project is designed to be run on Databricks within the NHS England systems. As such some of the code included here may not run on other MHSDS assets. The logic and methods used to produce the metrics included in this codebase remain the same though. 
+
+## Understanding the Mental Health Services Dataset
+
+MHSDS is collected on a monthly basis from providers of secondary mental health services. On average around 210 million rows of data flow into the dataset on a monthly basis. More information on the data quality of the dataset, including the numbers of providers submitting data and the volumes of data flowing to each table can be found in the Data Quality Dashboard: https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/mental-health-data-hub/data-quality/mental-health-services-dataset---data-quality-dashboard 
+
+The MHSDS tables and fields used within the code are all documented within the MHSDS tools and guidance. This guidance can be found here: https://digital.nhs.uk/data-and-information/data-collections-and-data-sets/data-sets/mental-health-services-data-set/tools-and-guidance
+
+Within the guidance are three key documents:
+
+1) MHSDS Technical Output Specification - This provides technical details of all of the fields and tables contained within the dataset. It also contains details of the validations applied to specific tables and fields. The specification also includes details of derivations and how they are constructed.
+2) MHSDS Data Model - This details all of the tables and fields within the dataset and how they relate to each other.
+3) MHSDS User Guidance - This document provides details of all of the tables and fields within the dataset and gives examples of how a user might populate the data in certain scenarios.
+
+Additionally, users might want to consult the Data Dictionary for specific fields within the dataset: https://www.datadictionary.nhs.uk/ 
+
+## Appendix and Notes
+
+In places the notebooks above use some acronyms. The main ones used are as follows:
+
+- MHSDS: Mental Health Services Dataset
+- CCG: Clinical Commissioning Group. These were replaced by Sub Integrated Care Boards (ICBs) in July 2022.
+- ICB: Integrated Care Board. These came into effect on July 1st 2022. Further information can be found at https://www.kingsfund.org.uk/publications/integrated-care-systems-explained#development.
+- Provider: The organisation is providing care. This is also the submitter of MHSDS data
+-LA: Local Authority
+
+## Support
+If you have any questions or issues regarding the constructions or code within this repository please contact mh.analysis@nhs.net
+
+## Authors and acknowledgment
+Community and Mental Health Team, NHS England
+mh.analysis@nhs.net
+
+## License
+The menh_bbrb codebase is released under the MIT License.
+The documentation is © Crown copyright and available under the terms of the [Open Government 3.0] (https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/) licence.
diff --git a/mha-annual-main/mha_code/Agg/MHA Annual - Discharges following detention.sql b/mha-annual-main/mha_code/Agg/MHA Annual - Discharges following detention.sql
@@ -0,0 +1,259 @@
+-- Databricks notebook source
+-- # %py
+-- # db_source  = dbutils.widgets.get("db_source")
+-- # rp_enddate = dbutils.widgets.get("rp_enddate")
+-- # rp_startdate = dbutils.widgets.get("rp_startdate")
+-- # year = dbutils.widgets.get("year")
+-- # personal_db = dbutils.widgets.get("personal_db")
+
+-- COMMAND ----------
+
+INSERT INTO $db_output.mha_unformatted
+
+SELECT 
+'$year' AS YEAR
+,'Discharges following detention' as Measure
+,'All' as MeasureSubcategory
+,'All' as Demographic
+,'All' as DemographicBreakdown
+,'All submissions' AS OrganisationBreakdown
+,'' as DataSource
+,'All prov' as OrgID
+,'All providers' as OrgName
+,COUNT(DISTINCT UniqHospProvSpellID)
+FROM
+$db_output.detentions
+WHERE
+DischDateHospProvSpell BETWEEN '$rp_startdate' AND '$rp_enddate'
+AND EndDateMHActLegalStatusClass <= DischDateHospProvSpell
+AND EndDateMHActLegalStatusClass >= StartDateHospProvSpell
+AND (MHA_LOGIC_CAT_FULL IN ('A','B','C','D','P'))
+
+-- COMMAND ----------
+
+INSERT INTO $db_output.mha_unformatted
+
+SELECT 
+'$year' AS YEAR
+,'Discharges following detention' as Measure
+,'All' as MeasureSubcategory
+,'All' as Demographic
+,'All' as DemographicBreakdown
+,ORG_TYPE_CODE AS OrganisationBreakdown
+,'' as DataSource
+,'All prov' as OrgID
+,'All providers' as OrgName
+,COUNT(DISTINCT UniqHospProvSpellID)
+FROM
+$db_output.detentions
+WHERE
+DischDateHospProvSpell BETWEEN '$rp_startdate' AND '$rp_enddate'
+AND EndDateMHActLegalStatusClass <= DischDateHospProvSpell
+AND EndDateMHActLegalStatusClass >= StartDateHospProvSpell
+AND (MHA_LOGIC_CAT_FULL IN ('A','B','C','D','P'))
+GROUP BY
+ORG_TYPE_CODE
+
+-- COMMAND ----------
+
+INSERT INTO $db_output.mha_unformatted
+
+SELECT 
+'$year' AS YEAR
+,'Discharges following detention' as Measure
+,'All' as MeasureSubcategory
+,'All' as Demographic
+,'All' as DemographicBreakdown
+,ORG_TYPE_CODE AS OrganisationBreakdown
+,'' as DataSource
+,OrgIDProv as OrgID
+,Name as OrgName
+,COUNT(DISTINCT UniqHospProvSpellID)
+FROM
+$db_output.detentions
+WHERE
+DischDateHospProvSpell BETWEEN '$rp_startdate' AND '$rp_enddate'
+AND EndDateMHActLegalStatusClass <= DischDateHospProvSpell
+AND EndDateMHActLegalStatusClass >= StartDateHospProvSpell
+AND (MHA_LOGIC_CAT_FULL IN ('A','B','C','D','P'))
+GROUP BY
+ORG_TYPE_CODE
+,ORGIDPROV
+,NAME
+
+-- COMMAND ----------
+
+INSERT INTO $db_output.mha_unformatted
+
+SELECT 
+'$year' AS YEAR
+,'Discharges following detention' as Measure
+,'All' as MeasureSubcategory
+,'Gender' as Demographic
+,Der_Gender as DemographicBreakdown
+,'All submissions' AS OrganisationBreakdown
+,'' as DataSource
+,'All prov' as OrgID
+,'All providers' as OrgName
+,COUNT(DISTINCT UniqHospProvSpellID)
+FROM
+$db_output.detentions
+WHERE
+DischDateHospProvSpell BETWEEN '$rp_startdate' AND '$rp_enddate'
+AND EndDateMHActLegalStatusClass <= DischDateHospProvSpell
+AND EndDateMHActLegalStatusClass >= StartDateHospProvSpell
+AND (MHA_LOGIC_CAT_FULL IN ('A','B','C','D','P'))
+-- AND Gender IN ('1','01','2','02')
+GROUP BY Der_Gender
+
+-- COMMAND ----------
+
+INSERT INTO $db_output.mha_unformatted
+
+SELECT 
+'$year' AS YEAR
+,'Discharges following detention' as Measure
+,'All' as MeasureSubcategory
+,'Age' as Demographic
+,CASE 
+WHEN AgeRepPeriodEnd between 0 and 15 then '15 and under'
+WHEN AgeRepPeriodEnd between 16 and 17 then '16 to 17'
+WHEN AgeRepPeriodEnd between 18 and 34 then '18 to 34'
+WHEN AgeRepPeriodEnd between 35 and 49 then '35 to 49'
+WHEN AgeRepPeriodEnd between 50 and 64 then '50 to 64'
+WHEN AgeRepPeriodEnd >= 65 then '65 and over' 
+ELSE 'Unknown'
+END as DemographicBreakdown
+,'All submissions' AS OrganisationBreakdown
+,'' as DataSource
+,'All prov' as OrgID
+,'All providers' as OrgName
+,COUNT(DISTINCT UniqHospProvSpellID)
+FROM
+$db_output.detentions
+WHERE
+DischDateHospProvSpell BETWEEN '$rp_startdate' AND '$rp_enddate'
+AND EndDateMHActLegalStatusClass <= DischDateHospProvSpell
+AND EndDateMHActLegalStatusClass >= StartDateHospProvSpell
+AND (MHA_LOGIC_CAT_FULL IN ('A','B','C','D','P'))
+-- AND AgeRepPeriodEnd is not null
+GROUP BY 
+CASE 
+WHEN AgeRepPeriodEnd between 0 and 15 then '15 and under'
+WHEN AgeRepPeriodEnd between 16 and 17 then '16 to 17'
+WHEN AgeRepPeriodEnd between 18 and 34 then '18 to 34'
+WHEN AgeRepPeriodEnd between 35 and 49 then '35 to 49'
+WHEN AgeRepPeriodEnd between 50 and 64 then '50 to 64'
+WHEN AgeRepPeriodEnd >= 65 then '65 and over' 
+ELSE 'Unknown'
+END
+
+-- COMMAND ----------
+
+INSERT INTO $db_output.mha_unformatted
+
+SELECT 
+'$year' AS YEAR
+,'Discharges following detention' as Measure
+,'All' as MeasureSubcategory
+,'Ethnicity' as Demographic
+,CASE
+WHEN EthnicCategory IN ('A','B','C') THEN 'White'
+WHEN EthnicCategory IN ('D','E','F','G') THEN 'Mixed'
+WHEN EthnicCategory IN ('H','J','K','L') THEN 'Asian or Asian British'
+WHEN EthnicCategory IN ('M','N','P') THEN 'Black or Black British'
+WHEN EthnicCategory IN ('R','S') THEN 'Other Ethnic Groups'
+ELSE 'Unknown'
+END as DemographicBreakdown
+,'All submissions' AS OrganisationBreakdown
+,'' as DataSource
+,'All prov' as OrgID
+,'All providers' as OrgName
+,COUNT(DISTINCT UniqHospProvSpellID)
+FROM
+$db_output.detentions
+WHERE
+DischDateHospProvSpell BETWEEN '$rp_startdate' AND '$rp_enddate'
+AND EndDateMHActLegalStatusClass <= DischDateHospProvSpell
+AND EndDateMHActLegalStatusClass >= StartDateHospProvSpell
+AND (MHA_LOGIC_CAT_FULL IN ('A','B','C','D','P'))
+and EthnicCategory is not null
+AND AgeRepPeriodEnd is not null
+AND Der_Gender in ('1','2')
+GROUP BY
+CASE
+WHEN EthnicCategory IN ('A','B','C') THEN 'White'
+WHEN EthnicCategory IN ('D','E','F','G') THEN 'Mixed'
+WHEN EthnicCategory IN ('H','J','K','L') THEN 'Asian or Asian British'
+WHEN EthnicCategory IN ('M','N','P') THEN 'Black or Black British'
+WHEN EthnicCategory IN ('R','S') THEN 'Other Ethnic Groups'
+ELSE 'Unknown'
+END
+
+-- COMMAND ----------
+
+INSERT INTO $db_output.mha_unformatted
+
+SELECT 
+'$year' AS YEAR
+,'Discharges following detention' as Measure
+,'All' as MeasureSubcategory
+,'Ethnicity' as Demographic
+,CASE 
+    WHEN EthnicCategory = 'A' THEN 'British'
+    WHEN EthnicCategory = 'B' THEN 'Irish'
+    WHEN EthnicCategory = 'C' THEN 'Any Other White Background'
+    WHEN EthnicCategory = 'D' THEN 'White and Black Caribbean' 
+    WHEN EthnicCategory = 'E' THEN 'White and Black African'
+    WHEN EthnicCategory = 'F' THEN 'White and Asian'
+    WHEN EthnicCategory = 'G' THEN 'Any Other Mixed Background'
+    WHEN EthnicCategory = 'H' THEN 'Indian'
+    WHEN EthnicCategory = 'J' THEN 'Pakistani'
+    WHEN EthnicCategory = 'K' THEN 'Bangladeshi'
+    WHEN EthnicCategory = 'L' THEN 'Any Other Asian Background'
+    WHEN EthnicCategory = 'M' THEN 'Caribbean'
+    WHEN EthnicCategory = 'N' THEN 'African'
+    WHEN EthnicCategory = 'P' THEN 'Any Other Black Background'
+    WHEN EthnicCategory = 'R' THEN 'Chinese'
+    WHEN EthnicCategory = 'S' THEN 'Any other ethnic group'
+    WHEN EthnicCategory = 'Z' THEN 'Not Stated'
+    WHEN EthnicCategory = '99' THEN 'Not Known'
+    ELSE 'Unknown'
+    END as DemographicBreakdown
+,'All submissions' AS OrganisationBreakdown
+,'' as DataSource
+,'All prov' as OrgID
+,'All providers' as OrgName
+,COUNT(DISTINCT UniqHospProvSpellID)
+FROM
+$db_output.detentions
+WHERE
+DischDateHospProvSpell BETWEEN '$rp_startdate' AND '$rp_enddate'
+AND EndDateMHActLegalStatusClass <= DischDateHospProvSpell
+AND EndDateMHActLegalStatusClass >= StartDateHospProvSpell
+AND (MHA_LOGIC_CAT_FULL IN ('A','B','C','D','P'))
+AND EthnicCategory is not null
+AND AgeRepPeriodEnd is not null
+AND Der_Gender in ('1','2')
+GROUP BY 
+CASE 
+    WHEN EthnicCategory = 'A' THEN 'British'
+    WHEN EthnicCategory = 'B' THEN 'Irish'
+    WHEN EthnicCategory = 'C' THEN 'Any Other White Background'
+    WHEN EthnicCategory = 'D' THEN 'White and Black Caribbean' 
+    WHEN EthnicCategory = 'E' THEN 'White and Black African'
+    WHEN EthnicCategory = 'F' THEN 'White and Asian'
+    WHEN EthnicCategory = 'G' THEN 'Any Other Mixed Background'
+    WHEN EthnicCategory = 'H' THEN 'Indian'
+    WHEN EthnicCategory = 'J' THEN 'Pakistani'
+    WHEN EthnicCategory = 'K' THEN 'Bangladeshi'
+    WHEN EthnicCategory = 'L' THEN 'Any Other Asian Background'
+    WHEN EthnicCategory = 'M' THEN 'Caribbean'
+    WHEN EthnicCategory = 'N' THEN 'African'
+    WHEN EthnicCategory = 'P' THEN 'Any Other Black Background'
+    WHEN EthnicCategory = 'R' THEN 'Chinese'
+    WHEN EthnicCategory = 'S' THEN 'Any other ethnic group'
+    WHEN EthnicCategory = 'Z' THEN 'Not Stated'
+    WHEN EthnicCategory = '99' THEN 'Not Known'
+    ELSE 'Unknown'
+    END