Airflow 3.0 deprecations

[dhruvmanila]

Tracking issue for the AIR3xx rules that implements Airflow 3.0 deprecations.

For details, refer to the Airflow tracking issue: apache/airflow#44556

Help wanted

The following is a "help wanted" task list:

• Add infrastructure to provide an auto-fix for the migration rules. Look at NumPy 2.0 implementation for reference
• Add a guard specifically for Airflow imports and attributes where a try .. except block could include both the new and deprecated symbols for backwards compatibility. Look at NumPy 2.0 implementation for reference

Ref: apache/airflow#41641

[Lee-W]

There's already a short proposal #14616 (comment). Putting it here might make it easier to find 🙂

---

I'm thinking of separating "removal" and "rename" at least. It would be much easier for the users to know whether there's anything they could do to fix it.

[dhruvmanila]

Additionally, the NumPy 2.0 migration rule is a good reference whose implementation is at: https://github.com/astral-sh/ruff/blob/6f1cf5b686e67fb74383641fcbbe98e33481a271/crates/ruff_linter/src/rules/numpy/rules/numpy_2_0_deprecation.rs

[Lee-W]

Hi, we have grouped our deprecations into several categories and would appreciate it if the team could take a look and let us know what you think about it 🙂

• already merged
  • AIR301: Avoid implicit DAG schedule [airflow] Avoid implicit DAG schedule (AIR301) #14581
• newly proposed error number
  • AIR302: removal or no direct mapping
    • following [airflow] Avoid deprecated values (AIR302) #14582
  • AIR303: rename
    • It would be much easier for the users to know whether they could do anything to fix it.
  • AIR304: moved to airflow providers
    • for these, they're similar to "rename" but require the user to install an additional package (e.g., apache-airflow-providers-fab)

Here's the categorised list we have now
apache/airflow#41641 (comment)
(there's another group we temporarily mark as AIR305, but we'll need some time to figure out what we can do for those. Will likely to be merged into 302 or 303)

[MichaReiser]

I like what's proposed here. Thanks for going through all the deprecations and grouping them by possible rules. We can also do some fine-tuning as we go.

[Lee-W]

Later discuss with @uranusjr, we think it would be better for us to merge rename and removal into 302. So the current idea would be

• already merged
  • AIR301: Avoid implicit DAG schedule ([airflow] Avoid implicit DAG schedule (AIR301) #14581)
  • AIR302: removal ([airflow] Avoid deprecated values (AIR302) #14582)
• new rule number
  • AIR303: moved to airflow providers
    • for these, they're similar to "rename" but require the user to install an additional package (e.g., apache-airflow-providers-fab)

We now use apache/airflow#44556 to track the detail of this task instead of the original one as some of the migration might not be able to be done in ruff

[MichaReiser]

Sorry for pinging you here, but could someone more familiar with Airflow volunteer looking at the changes in #14631? We just want to make sure that the change is correct.

[Lee-W]

Sorry for pinging you here, but could someone more familiar with Airflow volunteer looking at the changes in #14631? We just want to make sure that the change is correct.

Looks like a valid one to me

[MichaReiser]

Thanks for taking a look!

[dhruvmanila]

I think it'll also be really useful to provide an auto-fix for these rules. Refer to how NumPy 2.0 deprecation rule does it:

ruff/crates/ruff_linter/src/rules/numpy/rules/numpy_2_0_deprecation.rs

Lines 683 to 708 in 1bd8fbb

	match replacement.details {
	Details::AutoImport {
	path,
	name,
	compatibility,
	} => {
	diagnostic.try_set_fix(|| {
	let (import_edit, binding) = checker.importer().get_or_import_symbol(
	&ImportRequest::import_from(path, name),
	expr.start(),
	checker.semantic(),
	)?;
	let replacement_edit = Edit::range_replacement(binding, expr.range());
	Ok(match compatibility {
	Compatibility::BackwardsCompatible => {
	Fix::safe_edits(import_edit, [replacement_edit])
	}
	Compatibility::Breaking => Fix::unsafe_edits(import_edit, [replacement_edit]),
	})
	});
	}
	Details::AutoPurePython { python_expr } => diagnostic.set_fix(Fix::safe_edit(
	Edit::range_replacement(python_expr.to_string(), expr.range()),
	)),
	Details::Manual { guideline: _ } => {}
	};

And, I think we'll also need to add this guard check specific to Airflow imports and attributes which allows for having both the new and deprecated symbols to exists in a try .. except .. block:

ruff/crates/ruff_linter/src/rules/numpy/rules/numpy_2_0_deprecation.rs

Lines 671 to 673 in 1bd8fbb

	if is_guarded_by_try_except(expr, &replacement, semantic) {
	return;
	}

[uranusjr]

I agree. We’re on a tight schedule and would like to prioritise adding all the rules first, but contribution is always welcomed on these.