Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

postgreql safe data_quality__eligibility_death_flag #640

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

BuzzCutNorman
Copy link

Describe your changes

In the model the_tuva_project.data_quality.dqi.intermediate.atomic_checks.claims.eligibility.data_quality__eligibility_death_flag I changed the case statement to utilize in ('1', '0') which is exceptable in postgresql. I believe it also to be allowed as a equivilant to the orginal in (1,0) in the officially supported data warehouses.

Closes #553

How has this been tested?

I created a branch with the chagnes locally. Pointed my project dbt packages.yml to install tuva from that local repositry and ran dbt deps then did a dbt run --select the_tuva_project.data_quality.dqi.intermediate.atomic_checks.claims.eligibility.data_quality__eligibility_death_flag which completed sucessfully. I then ran 'dbt run' which completed beyond the data_quality__eligibility_death_flag model.

This will defenately need to be test against all supported datawarehouses. I didn't have access to the to test but did utilize documentation to if this code change would be equivilant to the orginal code.

Reviewer focus

in data_quality__eligibility_death_flag.sql please check the slight change to case statement in the select clause.

    ,case
        when m.death_flag in ('1','0') then 'valid'
        when m.death_flag is null then 'null'
        else 'invalid'
        end as bucket_name

Checklist before requesting a review

  • I have added at least one Github label to this PR (bug, enhancement, breaking change,...)
  • My code follows style guidelines
  • (New models) YAML files are categorized by sub folder and models listed in alphabetical order
  • (New models) I have added a config to each new model to enable it for claims and/or clinical data
  • (New models) I have added the variable tuva_last_run to the final output
  • (Optional) I have recorded a Loom to explain this PR

Package release checklist

  • I have updated dbt docs
  • I have updated the version number in the dbt_project.yml

(Optional) Gif of how this PR makes you feel

Loom link

@aneiderhiser aneiderhiser added the community Label for issues created by community members label Nov 9, 2024
Copy link
Member

@sarah-tuva sarah-tuva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @BuzzCutNorman, thanks for submitting this bug and fix! I have one suggested change to make this work in all of our supported data warehouses.

@@ -11,7 +11,7 @@ SELECT DISTINCT
,'ELIGIBILITY' AS claim_type
,'DEATH_FLAG' AS field_name
,case
when m.death_flag in (1,0) then 'valid'
when m.death_flag in ('1','0') then 'valid'
Copy link
Member

@sarah-tuva sarah-tuva Nov 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This syntax does not work in all of our supported data warehouses. Casting the death_flag as a string first and then evaluating the values as strings does work.

Suggested change
when m.death_flag in ('1','0') then 'valid'
when cast(m.death_flag as {{ dbt.type_string() }}) in ('1','0') then 'valid'

Copy link

Workflow has finished with the following statuses:

  • BigQuery Clinical and Claims: failure
  • BigQuery Claims: skipped
  • BigQuery Clinical: skipped
  • Fabric Clinical and Claims: success
  • Fabric Claims: success
  • Fabric Clinical: success
  • Redshift Clinical and Claims: success
  • Redshift Claims: success
  • Redshift Clinical: success
  • Snowflake Clinical and Claims: success
  • Snowflake Claims: success
  • Snowflake Clinical: success

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community Label for issues created by community members
Projects
Status: Ready for Review
Development

Successfully merging this pull request may close these issues.

The data_quality__eligibility_death_flag check causes a comparison error when run on PostgreSQL
3 participants