Skip to content

demographics file #1202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Jennit07 merged 27 commits into from
Nov 18, 2025
Merged

demographics file #1202

Jennit07 merged 27 commits into from
Nov 18, 2025

Conversation

@lizihao-anu
Copy link
Collaborator

@lizihao-anu lizihao-anu commented Oct 31, 2025

A few changes to social care datasets:

  • The demographics lookup is sliced into financial year. This enhances stability and reproducibility. People may move places from year to year, so it keeps postcode stable.
  • Remove care home postcode. It mainly aims at scenarios where people received SDS, HC, AT at home, then moved to a care home permnantly. The postcode used for depravation analysis should be home postcode. However, in this case, the postcode submitted or produced in the final yearly demographics lookup is care home postcode.
    • To avoid extreme uncommon cases where people live in a postcode which is a care home postcode but not living in a care home, we extract care home episodes. Basically, only if there is a care home record in that year, then we check if postcode in demographic lookup is a care home postcode. There is a strange case that, someone started to live in a care home in fy2022, but the demographics info submitted for fy2021 in fy2022 may be care home postcode. However, the correct postcode should be the home postcode. We have allowed for this case in the code by doing nearest join.
  • Join demographics file by year with all at, ch, hc, sds

Jennit07 and others added 16 commits September 17, 2025 16:41
@Jennit07
Change order - aligns with sc methods
1346f70
@Jennit07
Move postcode code to own section
b882590
@Jennit07
Add latest record flag notes from sc team
30eb049
@Jennit07 @github-actions
Style code
9e2c124
@Jennit07
Move arrange to make it more clear this is done
7c3b103
@Jennit07
Use SC methodology to select latest record flag
887befd
@Jennit07
remove redundant code
84813a3
@OluwatobiOni
Merge branch 'dec25-update' into sc_latest_record_flag
8e416f8
@lizihao-anu
sc: demog lookup by fy and align with sc
693ad34 
slice demog graphics file by financial year and remove care home postcode, enforce 1chi to multi scid
@lizihao-anu
sc: replace scid with latest grouped by anon_chi
47d9110
@lizihao-anu
sc: to adapt at with demog lookup
0428f3c
@lizihao-anu
sc: to adapt ch with demog lookup
c406c8b
@lizihao-anu
sc: to adapt hc with demog lookup
dd54988
@lizihao-anu
sc: to adapt sds with demog lookup
3c21a08
@lizihao-anu
sc: to adapt join_sc_client with demog lookup
cf2773c 
use linking_id to link ep with sc_client information
@lizihao-anu @github-actions
Style code
3a549d0
@Jennit07 Jennit07 mentioned this pull request Nov 3, 2025
Closed
Jennit07
Jennit07 reviewed Nov 3, 2025
View reviewed changes
Jennit07
Jennit07 reviewed Nov 4, 2025
View reviewed changes
@lizihao-anu lizihao-anu requested a review from Jennit07 November 6, 2025 15:23
@lizihao-anu
Copy link
Collaborator Author

lizihao-anu commented Nov 6, 2025

Good to review again.

I wonder if it is worth to run the whole updating process now to check bugs.

@Jennit07
Copy link
Collaborator

Jennit07 commented Nov 10, 2025

sc_demog_data_extract = read_lookup_sc_demographics()

this function takes a long time to run - potentially need to look at this to see if we can speed up

Jennit07
Jennit07 reviewed Nov 10, 2025
View reviewed changes
Jennit07
Jennit07 reviewed Nov 10, 2025
View reviewed changes
Jennit07
Jennit07 reviewed Nov 10, 2025
View reviewed changes
Jennit07
Jennit07 reviewed Nov 10, 2025
View reviewed changes
Jennit07
Jennit07 reviewed Nov 10, 2025
View reviewed changes
Jennit07
Jennit07 reviewed Nov 10, 2025
View reviewed changes
OluwatobiOni
OluwatobiOni reviewed Nov 12, 2025
View reviewed changes
Jennit07
Jennit07 reviewed Nov 12, 2025
View reviewed changes
Jennit07
Jennit07 reviewed Nov 12, 2025
View reviewed changes
Jennit07
Jennit07 reviewed Nov 12, 2025
View reviewed changes
Jennit07
Jennit07 reviewed Nov 12, 2025
View reviewed changes
Jennit07
Jennit07 reviewed Nov 12, 2025
View reviewed changes
OluwatobiOni
OluwatobiOni reviewed Nov 13, 2025
View reviewed changes
@lizihao-anu
Copy link
Collaborator Author

lizihao-anu commented Nov 14, 2025

read for review again

Jennit07
Jennit07 reviewed Nov 17, 2025
View reviewed changes
Jennit07
Jennit07 approved these changes Nov 17, 2025
View reviewed changes
Copy link
Collaborator

@Jennit07 Jennit07 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @lizihao-anu Ive ran through the demographics file and happy to approve the PR. Can we please leave this open and discuss at the team meeting tomorrow then we can merge into the dec25 branch? I only have one further question about joining the client file by linking id as i dont see process_lookup_sc_client modified here

Hi @OluwatobiOni Could you please check again and add your review when you have a moment? Thanks

@lizihao-anu lizihao-anu requested a review from Jennit07 November 18, 2025 09:42
OluwatobiOni
OluwatobiOni approved these changes Nov 18, 2025
View reviewed changes
Copy link
Collaborator

@OluwatobiOni OluwatobiOni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lizihao-anu, I've reviewed the changes and it looks good and ready to merge.

@Jennit07
Copy link
Collaborator

Jennit07 commented Nov 18, 2025

@lizihao-anu, I've reviewed the changes and it looks good and ready to merge.

Thanks @OluwatobiOni I'll merge this into the Dec25 branch

@Jennit07 Jennit07 merged commit a1b6500 into dec25-update Nov 18, 2025
7 checks passed
@Jennit07 Jennit07 deleted the ch_pc branch November 18, 2025 14:23
github-merge-queue bot pushed a commit that referenced this pull request Dec 12, 2025
@Jennit07 @OluwatobiOni @lizihao-anu
Dec25 update (#1212)
8e372f0 
* Extract tests - adding cost uplift for PLICS inflation

* Style code

* Update documentation

* pull request review

* Update documentation

* Update document.yaml

Although the checks passed, I noticed the document.yaml workflow was failing on the smr-cost-uplift branch. It looks like this issue is also present in the master and dec25-update branches when I checked the document workflow in the actions tab for those branches too.

I inspected the error and it was a git push rejection. This happened because the workflow's automated git-auto-commit-action tried to push documentation updates, but the remote branch on GitHub already had newer commits. 

To fix this, I've committed a change to the document.yaml workflow. The solution was to add a step to git pull the latest changes from the remote branch before the auto-commit action runs. This ensures the workflow is always working on the most up-to-date version of the branch and can push its changes without conflict.

The workflow should now run and pass correctly. Let me know if you have any questions.

* Update document.yaml

Reverting changes as it failed because of an overall changed workflow

* adding functions to _pkgdown.yml references

* sc extract tests count fix

* updating count of missing_dob and n_missing_postcode

* Style code

* remove quality 19, 20 as they do not make sense (#1205)

* export console output to txt file

* Style code

* demographics file (#1202)

* Change order - aligns with sc methods

* Move postcode code to own section

* Add latest record flag notes from sc team

* Style code

* Move arrange to make it more clear this is done

* Use SC methodology to select latest record flag

* remove redundant code

* sc: demog lookup by fy and align with sc
slice demog graphics file by financial year and remove care home postcode, enforce 1chi to multi scid

* sc: replace scid with latest grouped by anon_chi

* sc: to adapt at with demog lookup

* sc: to adapt ch with demog lookup

* sc: to adapt hc with demog lookup

* sc: to adapt sds with demog lookup

* sc: to adapt join_sc_client with demog lookup
use linking_id to link ep with sc_client information

* Style code

* Update documentation

* bug fix for process_lookup_sc_demographics

* update targets and join_sc_client()

* update process_lookup_sc_demographics.R

* Update documentation

* remove latest_record_flag

---------

Co-authored-by: jennit07 <Jennifer.Thom@phs.scot>
Co-authored-by: Jennit07 <67372904+Jennit07@users.noreply.github.com>
Co-authored-by: Oluwatobi Oni <Oluwatobi.Oni@phs.scot>

* Update homelessness completeness flag (#1208)

* Update new HL1 file path

* Update HL1 workbook to include new cols

* Style code

* pull request review

* create write_console_output.R function

* Update documentation

* Update Refs for Dec25 update

* Style code

* Update Refs - no sc data for Q1/2

* Style code

* update check_year_valid

* remove latest_flag from process_lookup_sc_client

* Update documentation

* fix target error - remove extract date

* fix all_sds target

* update sc_client_lookup in _targets.R

* update process_tests_sc_demographics

* fixing bugs in target

* Style code

* update cost_uplift.R to allow for missing months

* Style code

* update mpat in add_smr for recid == "02b"

* Update documentation

* adding check_year_valid to process_tests

* adding check_year_valid as guard

* fix

* fix ch postcodes directory connection

* Linking id dropped from sc - add this back in

* failed checks - issue with _pkgdown.yml

* bug fix

* bug fix

* fix tests for add_smrtype

* remove examples of write_console_output

* fix no visible binding issue

* remove any non standard character

* Update documentation

* Update copy to hscdiip

---------

Co-authored-by: Oluwatobi <oluwatobi.eoni@gmail.com>
Co-authored-by: OluwatobiOni <108776112+OluwatobiOni@users.noreply.github.com>
Co-authored-by: Oluwatobi Oni <oluwatobilobaebenezer@gmail.com>
Co-authored-by: Oluwatobi Oni <Oluwatobi.Oni@phs.scot>
Co-authored-by: Zihao Li <lizihao_anu@outlook.com>
Co-authored-by: Zihao Li <zihao.li@phs.scot>
Co-authored-by: lizihao-anu <97514488+lizihao-anu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@OluwatobiOni OluwatobiOni OluwatobiOni approved these changes

@Jennit07 Jennit07 Awaiting requested review from Jennit07

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@lizihao-anu @Jennit07 @OluwatobiOni