adds filter to remove long los (data quality) - #226
Merged
Conversation
spells with over a 2 year length of stay are incredibly rare, outside of mental health. this filter aims to remove any long length of stay that is likely caused by a data quality issue
tomjemmett
force-pushed
the
filter_out_long_los
branch
from
November 20, 2025 11:42
3b33491 to
31a2089
Compare
tomjemmett
marked this pull request as ready for review
November 20, 2025 11:58
Member
Author
|
checked counts on databricks, is removing an expected number of rows |
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds a data quality filter to remove hospital spells with excessively long lengths of stay (over 2 years) that are likely caused by data quality issues. The filter excludes long stays except for mental health cases or cases with 10 or more episodes.
- Filters out spells with duration > 2 years that are likely data quality errors
- Preserves mental health specialty spells (specialty codes starting with "7")
- Preserves complex cases with many episodes (10 or more)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
yiwen-h
approved these changes
Nov 20, 2025
yiwen-h
left a comment
yiwen-h
left a comment
Member
There was a problem hiding this comment.
Thank you! Note to self: add to project info as well
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
spells with over a 2 year length of stay are incredibly rare, outside of mental health. this filter aims to remove any long length of stay that is likely caused by a data quality issue