Different configs for microbatch models to use when the model is performing an increment or a full-refresh

[amauryfouville]

I didn't find a related topic.

We are using microbatch models in our dbt project, and one behavior hasn't been optimal so far.

The ideal scenario I have in mind would be to:

• using different batch_size and lookback based on if the model to run is to refresh fully (pre-existing or not) or to increment. For example, if the model is to refresh fully -> yearly batches, no lookback / if to increment -> daily batches, lookback of 1

One asset of dbt is that it does not execute anything against the Warehouse during the parse, and the parse reads from the models configs. By order of operations, I can't use the Warehouse information for configuring my model, and so this behavior can not easily be achieved.

We've got 3 cases, with not-so-clean approaches to achieve my ideal scenario:

1. full-refresh with the flag --full-refresh
2. regular run but model doesn't preexist
3. regular run and the model preexist (only the increment)

For 1., I can use the flag flags.FULL_REFRESH, and adapt the config, but still I would have a discrepancy between my manifest and the actual run with the flag (different batch_size and lookback).
Note: this interacts badly with what is recommended here: Also, https://docs.getdbt.com/docs/build/incremental-microbatch#full-refresh. --> the batch becomes a year for a single dayto reload
For 3., nothing to do really, this can the default behavior
For 2., I can not do anything really to split the behavior from 3.

And if we can find ways of achieving it, unless it's a super clean approach I missed, it doesn't feel appropriate.

And so the idea would be to define upfront in the configs different batch sizes and look-backs. One would be used when is_incremental() renders false and the other when it would render yes. This way all information would be available in the manifest whatever the state of the Warehouse is, and we would achieve more optimal microbatch runs.

The main impact I could anticipate would be on all products/use-cases consuming the dbt artifacts, and using the lookback and batch_size information.