BUG - new field values not being reflected in `jira__issue_enhanced` on incremental runs

[fivetran-jamie]

Are you a current Fivetran customer?

nope but this came from https://getdbt.slack.com/archives/C01D1R2JLLA/p1635355824026800

Describe the bug

on incremental runs of the package, if you have removed (or changed ? -- asking Jacob this...) a field value, the new value is not being reflected in the issue enhanced model

Steps to reproduce

1. create a ticket and add label
2. sync fivetran
3. execute incremental run
4. confirm the label value is there in jira__issue_enhanced
5. remove the label in jira
6. resync
7. execute incremental run
8. if you still see the old label value, run a full refresh and see if it works now

Expected behavior

the model should pick up the new changes on an incremental run, full refreshes or materializing as tables should not be required

Project variables configuration

vars:
  jira_database: raw
  jira_schema: jira_fivetran
  jira:
    # Field options can be found with this query:
    # select name from RAW.JIRA_FIVETRAN.FIELD;
    issue_field_history_columns: ['Labels', 'Key', 'Epic Name', 'Severity', 'Project']

  'dbt_date:time_zone': 'America/New_York'
  dbt_artifacts:
    dbt_artifacts_database: analytics
    dbt_artifacts_schema: dbt_artifacts
    dbt_artifacts_table: artifacts

Package Version

copy packages.yml here

Warehouse

• BigQuery
• Redshift
• Snowflake
• Postgres
• Databricks
• Other (provide details below)

Additional context

this line may be an issue -- we're subtracting 1 day from the wrong side i think (so like we should do this to max(date_day) from {{ this }} instead of when the field was updated_at . i think this would cause the package to ignore the most recent day of data on incremental runs.

otherwise, the problem may live in the pivot model or the final daily issue field history model

Screenshots

Please indicate the level of urgency

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
• No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[fivetran-jamie]

taking a look at this next week btw!

[fivetran-joemarkiewicz]

Hey @jcbmllgn I just wanted to post back here and provide an update on this task. @fivetran-jamie was able to help identify what the fix for this issue will be. They are currently only vacation and I will be taking this ticket on moving forward.

I will communicate back here next week once I have a working branch that you may test out. Thank you for your patience 😃

[jcbmllgn]

Thanks for the update @fivetran-joemarkiewicz! I'll keep an eye out for an update here.

[fivetran-joemarkiewicz]

Hey @jcbmllgn I was able to replicate the issue you mentioned in the dbt Slack message linked to this ticket! Since replicating the issue, I was able to take the advice made by @fivetran-jamie regarding modifying the int_jira__agg_multiselect_history incremental logic. Once I updated the incremental logic I was able to see the changing labels were accurately reflected in the jira__issue_enhanced model!

Would you be able to test the below version and let me know if the issue you noted still persists? If not, we can integrate this into the next release of the package 😄

packages:
    - git: https://github.com/fivetran/dbt_jira.git
      revision: bugfix/new-field-values
      warn-unpinned: false

Thanks again for your patience on this issue.

[jcbmllgn]

Joe, thanks for the follow up! I tested this out locally and it works for me. Let me know when this is live and I'll confirm again that it works in production.

[fivetran-joemarkiewicz]

That's great to hear! I'll confirm here once the release is live 😄

[fivetran-joemarkiewicz]

Hi @jcbmllgn I wanted to let you know that the latest version of the dbt_jira package v0.6.0 is now live on the dbt hub!

Be sure to read the Release Notes to be aware of the changes that are included in the latest release along with the change from this issue, and to run a dbt run --full-refresh after upgrading in order to capture changes in the incremental models within this package.

Thanks again for helping raise this issue and test out the working branch before release! We are extremely grateful for your contributions to this package and for helping make this package better for all it's users. 🎉 ❤️ Please feel free to open a new issue if you experience any other bugs or a feature request if you think of a new enhancement to that the package would benefit from integrating.