[CT-2854] [Feature] `.dbtignore` doesn't ignore files in dbt_packages

[rlh1994]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Adding reference to any files in the dbt_packages folder to the .dbtignore file does not lead to these being ignored. This may be needed as you wish to ignore certain files from the package (e.g. a sources or other yaml file that clashes, or docs. Models/seeds could be disabled via the project yaml).

Expected Behavior

These files would be ignored on a compile or a run.

Steps To Reproduce

1. Initialise a new project
2. Install a package that contains some models or seeds, e.g. :

   packages:
     - package: snowplow/snowplow_web
       version: [">=0.15.0"]
   

3. Create and ddd the whole package folder to the .dbtignore file:

   dbt_packages/snowplow_web/*
   

4. Run dbt run and notice that all models from the package is still included in the run.

Edit with a slightly easier example

2. Install the dbt_utils package

   packages:
     - package: dbt-labs/dbt_utils
       version: 1.1.1
   

3. Create and add the whole package folder to the .dbtignore file:

   dbt_packages
   

4. Create a model that uses a dbt_utils macro, e.g. date_spine:

   {{ dbt_utils.date_spine(
       datepart="day",
       start_date="cast('2019-01-01' as date)",
       end_date="cast('2020-01-01' as date)"
      )
   }}
   

5. Do a dbt run and this model passes, when it should fail (which you can verify by removing the package and doing a dbt clean, which should be the equivalent state if the package was being ignored).

End Edit

Relevant log output

No response

Environment

- OS: Mac OS
- Python: 3.9.13
- dbt:
Core:
  - installed: 1.5.1
  - latest:    1.5.3 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.5.4 - Update available!
  - bigquery:   1.5.1 - Update available!
  - snowflake:  1.5.1 - Update available!
  - redshift:   1.5.4 - Update available!
  - postgres:   1.5.1 - Update available!
  - spark:      1.5.0 - Up to date!

Which database adapter are you using with dbt?

Postgres

Additional Context

This PR introduced the feature, but I am not sure where in here these would be being not included in the ignore...
#5897

[dbeatty10]

Thanks for reporting this @rlh1994 !

Putting some debugging breakpoints around here, it looks to me like each package can include its own .dbtignore file, but it's not currently possible to refer to installed packages within the root .dbtignore file.

I'm assuming that you already tried disabling via dbt_projects.yml like the following, but it wasn't applicable (or didn't work)?

dbt_projects.yml

...

models:
  snowplow_web:
    +enabled: false

seeds:
  snowplow_web:
    +enabled: false

snapshots:
  snowplow_web:
    +enabled: false

tests:
  snowplow_web:
    +enabled: false

sources:
  snowplow_web:
    +enabled: false

metrics:
  snowplow_web:
    +enabled: false

exposures:
  snowplow_web:
    +enabled: false

Could you share an example of a sources or other yaml file that clashes (and the net affect it has on you)?

[rlh1994]

@dbeatty10 yeah I was able to disable the seeds (in my case) themselves in the package via the dbt_project.yml, which works exactly as expected.

However the "issue" comes when I then build a model in the root project with the same name as the seed from the package (basically I need to overwrite a seed with a model but use the same name so it gets used in refs), it leads to a warning on compile that the model is under the seeds heading in the yaml file. It still compiles and runs, but it would be nice to be able to avoid the warnings by somehow disabling the associated yaml for the seeds, and the .dbtignore was the only way I thought I might be able to.

I'll throw together a more thorough reprex over the weekend if needed.

[rlh1994]

Reprex of the warning issue:

# dbt_project.yml
...

models:
  snowplow_web:
    +enabled: false

seeds:
  snowplow_web:
    +enabled: false

snapshots:
  snowplow_web:
    +enabled: false

tests:
  snowplow_web:
    +enabled: false

sources:
  snowplow_web:
    +enabled: false

metrics:
  snowplow_web:
    +enabled: false

# packages.yml
packages:
  - package: snowplow/snowplow_web
    version: 0.15.0

{# models/dim_ga4_source_categories.sql #}
select 1 as a

(not relevant as we have confirmed it does not work, but ideally this would ignore these files thus removing the issue)

# .dbtignore
dbt_packages/

Commands:

dbt clean
dbt deps
dbt compile

Output:

09:09:05  Running with dbt=1.5.1
09:09:05  Unable to do partial parsing because saved manifest not found. Starting full parse.
09:09:07  [WARNING]: 'dim_ga4_source_categories' is a model node, but it is specified in the seeds
section of seeds/seeds.yml. To fix this error, place the
`dim_ga4_source_categories` specification under the models key instead.
09:09:07  Found 1 model, 0 tests, 0 snapshots, 0 analyses, 611 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups
09:09:07  
09:09:08  Concurrency: 1 threads (target='postgres')
09:09:08  

[aayushr7]

The issue also occurs for model properties file. Adding the following in .dbtignore

dbt_packages/**.yml

did not exclude the conflicting schema files, and resulted in the following error:

Compilation Error
  dbt found two schema.yml entries for the same resource named test_model. Resources and their associated columns may only be described a single time. To fix this, remove the resource entry for test_model in one of these files:
   - models/common/test_model.yml
  models/mart/test_model.yml

[graciegoheen]

Hey! Hoping into this thread. In our recent 1.6 release, we introduced 2-argument refs to allow for duplicate resource names across multiple packages/projects:

Model names can be duplicated across different namespaces (projects/packages), so long as they are unique within each project/package. We strongly encourage using two-argument ref when referencing a model from a different package/project.

Would you be able to try out updating your refs to use 2 arguments and see if that resolves this issue?

select * from {{ ref('project_or_package', 'model_name') }}