dbt 1.6 - ERROR X validation errors for ManifestV10

[dlawin]

Describe the bug
Now that dbt 1.6 is fully released, we are seeing validation errors for manifests from that version. The validation is using a beta version of the manifest schema.

Users will see:
ERROR X validation errors for ManifestV10

Need to update dbt-artifacts-parser to support the official V10 manifest schema. Issue here: yu-iskw/dbt-artifacts-parser#54

Other context for the future:

• We're using dbt-artifacts-parser to access dbt manifests as python objects vs. dicts
  • this isn't totally necessary and may be causing more trouble than it's worth

[sungchun12]

Is it worth forking dbt-artifacts-parser and owning that dependency ourselves?

[dlawin]

Is it worth forking dbt-artifacts-parser and owning that dependency ourselves?

I think if we're considering that we should just drop the package entirely. It's nice to use objects vs. dictionary.get() everywhere but not needed

[leoebfolsom]

Without diving into the details, is it viable to say "1.6 is not supported," which would buy us some time? I'm assuming most dbt users are not actually on 1.6.
Sorry if i'm misunderstanding the problem / possible solution.
cc @sungchun12

[sungchun12]

@dlawin I recommend we drop it because this becomes a single point of failure outside our control each time dbt upgrades in version. We've already had dozens of conversations around this since 1.6 released.

@leoebfolsom "1.6 is not supported" feels sloppy on our part. We should say we're actively working on it and fix it as a default, especially since this tool is so closely tied to dbt usage.

[leoebfolsom]

@sungchun12 sorry I didn't mean the blunt "is not supported" statement. 100% agree that "we're working on it" and "will be supported" -- I just wanted to get a message out there, and explicit version dependency requirement, to reduce the immediate urgency and user pain point (at least we're telling them it doesn't work) given other priorities being juggled.

[sungchun12]

@leoebfolsom I recommend we let people reach out to us for now given your point, most people do not migrate to new dbt versions right away as you said!

Already marked here: https://getdbt.slack.com/archives/C03D25A92UU/p1692206342810729

[dlawin]

PR here yu-iskw/dbt-artifacts-parser#55

[dlawin]

this becomes a single point of failure outside our control each time dbt upgrades in version. We've already had dozens of conversations around this since 1.6 released.

@sungchun12 While this is true the point of failure stems from dbt itself

The package protects us from dbt by throwing validation errors when there are unexpected manifest schemas, with that said, the fields we depend on in the manifest seem pretty stable.

[sungchun12]

@dlawin I understand that. I guess the more fundamental question to ask is that are the tradeoffs better in using this dependency for those validation errors already built for us and having to micromanage this dependency each time dbt upgrades the manifest in a new version each quarter/patch?

My intuition goes towards the tradeoffs aren't worth it, but Dan you're more intimate with this than me, so can help me understand those tradeoffs in detail?