Remove the ability to set parallelism to infinity in Airflow V3

[o-nikolas]

Description

See this issue and this subsequent PR for context. But in short: There was a undocumented/untested/buggy feature to allow setting parallelism to infinity (by setting it to zero) which was dropped during the implementation of Multiple Executor Config. This raised a discussion of if this feature should really be supported. I propose we drop this feature in Airflow 3 because:

1. It adds unnecessary complexity to the code
2. This behaviour can be achieved by setting parallelism to a sufficiently high number
3. (most importantly) I think it's actually important for the user to have to do 2), it forces them to actually think "hmm how parallel should I actually run tasks? Is infinity appropriated? Will infinity actually cause degraded performance?". I think allowing 0 gives an easy way for folks to set and forget without weighing the implications.
4. scheduler.max_tis_per_query is a very important config for performance and depends on core.parallelism if it is set to 0 (which means to track the value of parallelism) then we may have infinite query sizes which would drastically impact performance. This is an easy trap for users to fall into.

Related issues

#41055
#41107

Code of Conduct

• I agree to follow this project's Code of Conduct