Insufficent permission to catalog='system' when using dbt-databricks=1.8.7

[VippsMe]

Describe the bug

Issue when using dbt-databricks==1.8.7

• We just upgraded from dbt-databricks 1.5 to 1.8.7. This upgrade resulted in an error in the ci job. Downgraded to vs 1.7.17 and then it worked again.
• Error message in github action ci job:

[INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User does not have USE CATALOG on Catalog 'system'. SQLSTATE: 42501

Steps To Reproduce

See above. Run dbt compile and it will give you the error message. Target is set to a different catalog in databricks with permissions for the user. Somehow, the dbt-databricks adapter is trying to access catalog 'system'. This catalog is not part of our internal code at all.

Expected behavior

See above.

Screenshots and log output

System information

The output of dbt --version:

<output goes here>

The operating system you're using:

The output of python --version:3.10.12

Additional context

Add any other context about the problem here.

[benc-db]

This looks like your admin has removed your permission to the system catalog, as every user should have access to this table by default. We can investigate a flag to allow you to reference UC information_schemas scoped by catalog, but we moved to using system catalog as the catalog-scoped information_schema views break if you rename a catalog, so it's kind of a no win situation.

[d474b3r9]

Hello, I encounter the exact same issue even if in my profiles.yml.
It works in 1.7.17 but not in 1.8.7 and >

I work on a managed databricks workspace and my company will not allow me rights on the system_catalog.

Do you have a workaround ?
Something like set use_system_catalog: false should be great ! ;)

[VippsMe]

a

Its not because of our admin. Access to catalog 'system' has been the exact same all the time.
So it is something about dbt-databricks adapter 1.8.XX that is trying to access catalog 'system' (and not our internal repo/code)

[benc-db]

@VippsMe yes, we changed to using system, but we changed because a.) it is more reliable than the individual catalog info_schemas because they are just views of system that can break if the catalog is renamed, and b.) everyone has access to system.information_schema by default. So the reason I'm suggesting that your admin at some point removed your permission, is because you should have access, and if you don't, it's something that an admin can change.

[rheanes]

It works in 1.7.17 but not in 1.8.7 and >

I work on a managed databricks workspace and my company will not allow me rights on the system_catalog.

Did you ever figure out a work around for this?

[d474b3r9]

It works in 1.7.17 but not in 1.8.7 and >
I work on a managed databricks workspace and my company will not allow me rights on the system_catalog.

Did you ever figure out a work around for this?

You need those rights for your user, you have no other choices.

[joaofonsecapromptly]

Could we at least have more documentation on this and why is it necessary? For example, within the system schema what tables does dbt actually need?

[benc-db]

We require select on system.information_schema.*.