Airflow : filedescriptor out of range in select

[shubhampatel94]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

Airflow version 2.3.4

I was getting dag_bag timeout error. Since one of the dag has around 350 tasks, loading and processing were causing issues there.
While debugging for the issue i made following config changes.
parallelism = 1800
max_active_tasks_per_dag = 200
dagbag_import_timeout = 1200.0
dagbag_import_error_traceback_depth = 30
dag_file_processor_timeout = 1000
default_task_retry_delay = 3600
min_serialized_dag_fetch_interval = 20
sql_alchemy_pool_recycle = 600
max_db_retries = 10

From the above dag_bag-related changes are the only one that is helpful. But that i realized after figuring out the issue.

But other config changes created side-effect and I am started getting it once in a while. Need to understand what config changes might have been causing this.

  File "/home/alpha/.local/lib/python3.8/site-packages/airflow/providers/ssh/operators/ssh.py", line 158, in execute
    result = self.run_ssh_client_command(ssh_client, self.command)
  File "/home/alpha/.local/lib/python3.8/site-packages/airflow/providers/ssh/operators/ssh.py", line 143, in run_ssh_client_command
    exit_status, agg_stdout, agg_stderr = self.ssh_hook.exec_ssh_client_command(
  File "/home/alpha/.local/lib/python3.8/site-packages/airflow/providers/ssh/hooks/ssh.py", line 494, in exec_ssh_client_command
    readq, _, _ = select([channel], [], [], timeout)
ValueError: filedescriptor out of range in select()

What you think should happen instead

Airflow jobs should not fail with above error. It's look like we are hitting ssh file descriptor limit here. But I don't clearly understand why that is the case and where all of the active file descriptor are holding up.

How to reproduce

Please set the above mention config.
And run jobs with large amount of stages.

Operating System

NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal

Versions of Apache Airflow Providers

[03:37 AM]alpha@devops18:airflow$ pip freeze | grep "apache-airflow-provider"
/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
from cryptography.utils import int_from_bytes
/usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
from cryptography.utils import int_from_bytes
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.3.0

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[hussein-awala]

When using LocalExecutor, the parallelism conf is considered as worker concurrency, in your case the scheduler may create 1800 tasks running in sub-processes (and there are already other processes created to process dag files, handle SLA, process callbacks, ...).

Your system has a limit for the maximum number of open file descriptors, so when you reach this limit, any python script will try to open a file, will fail with this error ValueError: filedescriptor out of range in select(). (same for all the processes and not only python)

You can check the limit in your system using the command:

$ ulimit -n
1024

You can try to increase the limit and check if Airflow tasks will still fail or not:

$ ulimit -n 8192

[garywhiteford]

Cross-link: #30927

[giosdas]

Hello,
I have found the same issue, but in my case the problem appear in the same long running task, which make a lot of ssh connection in sequential order.
The problem seems related to the paramiko ssh connection which is not closed, to resolve I change my code from this:

    ssh_hook = SSHHook(ssh_conn_id)
    (exit_code, stdout, stderr) = ssh_hook.exec_ssh_client_command(
        ssh_hook.get_conn(),
        command
    )

to this:

    ssh_hook = SSHHook(ssh_conn_id)
    ssh_conn = ssh_hook.get_conn()
    (exit_code, stdout, stderr) = ssh_hook.exec_ssh_client_command(
        ssh_conn,
        command
    )
    ssh_conn.close()

Regards

[rahidzeynal]

We ran into the same ValueError: filedescriptor out of range in select() issue with our Airflow 3 SSH operator. Setting get_pty=True on the operator resolved it for us:

SSHOperator(
    task_id='my_task',
    ssh_conn_id='my_ssh_conn',
    command='my_command',
    get_pty=True,
)

This works because get_pty=True makes paramiko use a PTY channel that reads output via recv() in a loop instead of relying on select(), which fails when the file descriptor number exceeds the system limit of 1024.

Hope this helps someone.