Allow exclusion of priority data

[SteadyGiant]

What change(s) does this PR introduce?

My team doesn't use Priority in Jira, the API returns nothing from our Priority endpoint, and so source jira.priority never gets loaded into the warehouse. Instead, we use custom fields. We still need downstream models like jira__issue_enhanced, which depend on priority data. This PR allows exclusion of priority data, so jira__issue_enhanced can get built even if a team lacks priority data.

Makes priority data optional. Allows new env var jira_using_priorities. Models jira__issue_enhanced and int_jira__issue_join won't require source jira.priority or contain priority-related columns if jira_using_priorities: false.

Did you update the CHANGELOG?

• Yes

Does this PR introduce a breaking change?

• Yes (please provide breaking change details below.)
• No (please provide an explanation as to how the change is non-breaking below.)

Priority data is still used by default (env var jira_using_priorities: true in dbt_project.yml).

Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)

• Yes

Project versions bumped in both files.

Is this PR in response to a previously created Bug or Feature Request

• Yes, Issue/Feature [link bug/feature number here]
• No

How did you test the PR changes?

• CircleCi
• Local (please provide additional testing details below)

Tested manually with dbt CLI (dbt run --select jira_source jira) from my local machine (Windows 10, PowerShell) to a Redshift warehouse using dbt-core 1.0.4 and dbt-redshift 1.0.0. All models defined in this repo are built successfully. All models in fivetran/jira_source package except stg_jira__priority_tmp and stg_jira__priority are built successfully; stg_jira__priority_tmp throws an error and stg_jira__priority gets skipped, but this is intentional.

Output:

> dbt run --select jira_source jira
17:38:22  Running with dbt=1.0.4
17:38:25  Found 130 models, 117 tests, 0 snapshots, 0 analyses, 586 macros, 0 operations, 0 seed files, 37 sources, 0 exposures, 0 metrics
17:38:25
17:38:29  Concurrency: 1 threads (target='dev')
17:38:29
17:38:29  1 of 46 START view model stg_jira.stg_jira__comment_tmp......................... [RUN]
17:38:29  1 of 46 OK created view model stg_jira.stg_jira__comment_tmp.................... [CREATE VIEW in 0.60s]
17:38:29  2 of 46 START view model stg_jira.stg_jira__component_tmp....................... [RUN]
17:38:30  2 of 46 OK created view model stg_jira.stg_jira__component_tmp.................. [CREATE VIEW in 0.59s]
17:38:30  3 of 46 START view model stg_jira.stg_jira__field_option_tmp.................... [RUN]
17:38:30  3 of 46 OK created view model stg_jira.stg_jira__field_option_tmp............... [CREATE VIEW in 0.56s]
17:38:30  4 of 46 START view model stg_jira.stg_jira__field_tmp........................... [RUN]
17:38:31  4 of 46 OK created view model stg_jira.stg_jira__field_tmp...................... [CREATE VIEW in 0.60s]
17:38:31  5 of 46 START view model stg_jira.stg_jira__issue_field_history_tmp............. [RUN]
17:38:32  5 of 46 OK created view model stg_jira.stg_jira__issue_field_history_tmp........ [CREATE VIEW in 0.59s]
17:38:32  6 of 46 START view model stg_jira.stg_jira__issue_link_tmp...................... [RUN]
17:38:32  6 of 46 OK created view model stg_jira.stg_jira__issue_link_tmp................. [CREATE VIEW in 0.60s]
17:38:32  7 of 46 START view model stg_jira.stg_jira__issue_multiselect_history_tmp....... [RUN]
17:38:33  7 of 46 OK created view model stg_jira.stg_jira__issue_multiselect_history_tmp.. [CREATE VIEW in 0.60s]
17:38:33  8 of 46 START view model stg_jira.stg_jira__issue_tmp........................... [RUN]
17:38:33  8 of 46 OK created view model stg_jira.stg_jira__issue_tmp...................... [CREATE VIEW in 0.60s]
17:38:33  9 of 46 START view model stg_jira.stg_jira__issue_type_tmp...................... [RUN]
17:38:34  9 of 46 OK created view model stg_jira.stg_jira__issue_type_tmp................. [CREATE VIEW in 0.58s]
17:38:34  10 of 46 START view model stg_jira.stg_jira__priority_tmp....................... [RUN]
17:38:34  10 of 46 ERROR creating view model stg_jira.stg_jira__priority_tmp.............. [ERROR in 0.32s]
17:38:34  11 of 46 START view model stg_jira.stg_jira__project_tmp........................ [RUN]
17:38:35  11 of 46 OK created view model stg_jira.stg_jira__project_tmp................... [CREATE VIEW in 0.57s]
17:38:35  12 of 46 START view model stg_jira.stg_jira__resolution_tmp..................... [RUN]
17:38:36  12 of 46 OK created view model stg_jira.stg_jira__resolution_tmp................ [CREATE VIEW in 0.58s]
17:38:36  13 of 46 START view model stg_jira.stg_jira__sprint_tmp......................... [RUN]
17:38:36  13 of 46 OK created view model stg_jira.stg_jira__sprint_tmp.................... [CREATE VIEW in 0.57s]
17:38:36  14 of 46 START view model stg_jira.stg_jira__status_category_tmp................ [RUN]
17:38:37  14 of 46 OK created view model stg_jira.stg_jira__status_category_tmp........... [CREATE VIEW in 0.54s]
17:38:37  15 of 46 START view model stg_jira.stg_jira__status_tmp......................... [RUN]
17:38:37  15 of 46 OK created view model stg_jira.stg_jira__status_tmp.................... [CREATE VIEW in 0.57s]
17:38:37  16 of 46 START view model stg_jira.stg_jira__user_tmp........................... [RUN]
17:38:38  16 of 46 OK created view model stg_jira.stg_jira__user_tmp...................... [CREATE VIEW in 0.56s]
17:38:38  17 of 46 START view model stg_jira.stg_jira__version_tmp........................ [RUN]
17:38:38  17 of 46 OK created view model stg_jira.stg_jira__version_tmp................... [CREATE VIEW in 0.57s]
17:38:38  18 of 46 START table model stg_jira.stg_jira__comment........................... [RUN]
17:38:41  18 of 46 OK created table model stg_jira.stg_jira__comment...................... [SELECT in 2.11s]
17:38:41  19 of 46 START table model stg_jira.stg_jira__component......................... [RUN]
17:38:41  19 of 46 OK created table model stg_jira.stg_jira__component.................... [SELECT in 0.84s]
17:38:41  20 of 46 START table model stg_jira.stg_jira__field_option...................... [RUN]
17:38:42  20 of 46 OK created table model stg_jira.stg_jira__field_option................. [SELECT in 0.85s]
17:38:42  21 of 46 START table model stg_jira.stg_jira__field............................. [RUN]
17:38:43  21 of 46 OK created table model stg_jira.stg_jira__field........................ [SELECT in 0.84s]
17:38:43  22 of 46 START table model stg_jira.stg_jira__issue_field_history............... [RUN]
17:38:52  22 of 46 OK created table model stg_jira.stg_jira__issue_field_history.......... [SELECT in 9.19s]
17:38:52  23 of 46 START table model stg_jira.stg_jira__issue_link........................ [RUN]
17:38:53  23 of 46 OK created table model stg_jira.stg_jira__issue_link................... [SELECT in 0.84s]
17:38:53  24 of 46 START table model stg_jira.stg_jira__issue_multiselect_history......... [RUN]
17:38:56  24 of 46 OK created table model stg_jira.stg_jira__issue_multiselect_history.... [SELECT in 3.05s]
17:38:56  25 of 46 START table model stg_jira.stg_jira__issue............................. [RUN]
17:38:58  25 of 46 OK created table model stg_jira.stg_jira__issue........................ [SELECT in 1.88s]
17:38:58  26 of 46 START table model stg_jira.stg_jira__issue_type........................ [RUN]
17:38:59  26 of 46 OK created table model stg_jira.stg_jira__issue_type................... [SELECT in 0.89s]
17:38:59  27 of 46 SKIP relation stg_jira.stg_jira__priority.............................. [SKIP]
17:38:59  28 of 46 START table model stg_jira.stg_jira__project........................... [RUN]
17:39:00  28 of 46 OK created table model stg_jira.stg_jira__project...................... [SELECT in 0.96s]
17:39:00  29 of 46 START table model stg_jira.stg_jira__resolution........................ [RUN]
17:39:01  29 of 46 OK created table model stg_jira.stg_jira__resolution................... [SELECT in 0.88s]
17:39:01  30 of 46 START table model stg_jira.stg_jira__sprint............................ [RUN]
17:39:02  30 of 46 OK created table model stg_jira.stg_jira__sprint....................... [SELECT in 0.98s]
17:39:02  31 of 46 START table model stg_jira.stg_jira__status_category................... [RUN]
17:39:03  31 of 46 OK created table model stg_jira.stg_jira__status_category.............. [SELECT in 0.92s]
17:39:03  32 of 46 START table model stg_jira.stg_jira__status............................ [RUN]
17:39:04  32 of 46 OK created table model stg_jira.stg_jira__status....................... [SELECT in 0.94s]
17:39:04  33 of 46 START table model stg_jira.stg_jira__user.............................. [RUN]
17:39:05  33 of 46 OK created table model stg_jira.stg_jira__user......................... [SELECT in 0.90s]
17:39:05  34 of 46 START table model stg_jira.stg_jira__version........................... [RUN]
17:39:06  34 of 46 OK created table model stg_jira.stg_jira__version...................... [SELECT in 0.93s]
17:39:06  35 of 46 START incremental model test.int_jira__issue_calendar_spine............ [RUN]
17:40:50  35 of 46 OK created incremental model test.int_jira__issue_calendar_spine....... [SELECT in 104.21s]
17:40:50  36 of 46 START incremental model test.int_jira__agg_multiselect_history......... [RUN]
17:40:59  36 of 46 OK created incremental model test.int_jira__agg_multiselect_history.... [SELECT in 8.69s]
17:40:59  37 of 46 START table model test.int_jira__issue_type_parents.................... [RUN]
17:41:02  37 of 46 OK created table model test.int_jira__issue_type_parents............... [SELECT in 3.11s]
17:41:02  38 of 46 START incremental model test.int_jira__combine_field_histories......... [RUN]
17:41:56  38 of 46 OK created incremental model test.int_jira__combine_field_histories.... [SELECT in 53.70s]
17:41:56  39 of 46 START incremental model test.int_jira__daily_field_history............. [RUN]
17:42:01  39 of 46 OK created incremental model test.int_jira__daily_field_history........ [SELECT in 4.83s]
17:42:01  40 of 46 START table model test.int_jira__issue_join............................ [RUN]
17:42:07  40 of 46 OK created table model test.int_jira__issue_join....................... [SELECT in 6.12s]
17:42:07  41 of 46 START incremental model test.int_jira__pivot_daily_field_history....... [RUN]
17:42:09  41 of 46 OK created incremental model test.int_jira__pivot_daily_field_history.. [SELECT in 1.23s]
17:42:09  42 of 46 START table model test.int_jira__field_history_scd..................... [RUN]
17:42:10  42 of 46 OK created table model test.int_jira__field_history_scd................ [SELECT in 1.61s]
17:42:10  43 of 46 START incremental model test.jira__daily_issue_field_history........... [RUN]
17:44:59  43 of 46 OK created incremental model test.jira__daily_issue_field_history...... [SELECT in 168.66s]
17:44:59  44 of 46 START table model test.jira__issue_enhanced............................ [RUN]
17:45:25  44 of 46 OK created table model test.jira__issue_enhanced....................... [SELECT in 25.88s]
17:45:25  45 of 46 START table model test.jira__project_enhanced.......................... [RUN]
17:45:27  45 of 46 OK created table model test.jira__project_enhanced..................... [SELECT in 1.93s]
17:45:27  46 of 46 START table model test.jira__user_enhanced............................. [RUN]
17:45:28  46 of 46 OK created table model test.jira__user_enhanced........................ [SELECT in 1.22s]
17:45:28
17:45:28  Finished running 17 view models, 23 table models, 6 incremental models in 423.05s.
17:45:28
17:45:28  Completed with 1 error and 0 warnings:
17:45:28
17:45:28  Database Error in model stg_jira__priority_tmp (models\tmp\stg_jira__priority_tmp.sql)
17:45:28    relation "jira.priority" does not exist
17:45:28    compiled SQL at target\run\jira_source\models\tmp\stg_jira__priority_tmp.sql
17:45:28
17:45:28  Done. PASS=44 WARN=0 ERROR=1 SKIP=1 TOTAL=46

Select which warehouse(s) were used to test the PR

• BigQuery
• Redshift
• Snowflake
• Postgres
• Databricks
• Other (provide details below)

[fivetran-joemarkiewicz]

@Everettttt thanks for opening this PR! Taking a look at the changes you have proposed, I feel this would be a valuable addition to the package.

There is one adjustment that will need to be made in order for the package to properly succeed with this variable. We will need to add the jira_using_priorities config logic to our source models:

• stg_jira__priority_tmp
• stg_jira__priority

Once the config logic is added to these models to be disabled with the jira_using_priorities: false then we will be good to go! Let me know if you would be open to creating a PR on our source package. If not, I am happy to tackle that update 😄

[SteadyGiant]

Let me know if you would be open to creating a PR on our source package. If not, I am happy to tackle that update

I'll do it.

[fivetran-joemarkiewicz]

Thanks so much @Everettttt! Feel free to tag me in the PR and I will be sure to review it as soon as possible once it is open 😄

[SteadyGiant]

Just realized I'll have to return to this in 2 weeks, though. If this needs doing before then, feel free, but otherwise I'm willing to do it.

[fivetran-joemarkiewicz]

Thanks for the update @Everettttt! 2 weeks will be perfect. There haven't been any other requests at the moment for this to be addressed asap. So that timeline is great. Thanks again 😄

[SteadyGiant]

There is one adjustment that will need to be made in order for the package to properly succeed with this variable. We will need to add the jira_using_priorities config logic to our source models:

Submitted a PR to fivetran/dbt_jira_source.

I'll resolve the conflicts in the present PR tomorrow.

[SteadyGiant]

Conflicts resolved, I hope. Added myself as a contributor to the changelog.

[fivetran-joemarkiewicz]

@Everettttt thanks again for putting these PRs together and contributing back to the package 🙌

I made a few minor changes in your branch, but everything looks good to go on my end! I will merge and plan to cut the official releases tomorrow.