Bug: Dependencies in requirements.txt are not correctly updated on environment restart

[ivan-toriya-precis]

Description

When updating a Python package version in requirements.txt of local environment, the composer-local-dev tool does not appear to install the updated version when the environment is restarted. This leads to "dependency conflicts" and runtime errors if other packages rely on the newer versions.

I encountered this issue when trying to resolve a TypeError: SSHHook.__init__() got an unexpected keyword argument 'host_proxy_cmd' between apache-airflow-providers-sftp and its dependency, apache-airflow-providers-ssh. The SFTP provider required a newer version of the SSH provider. Even after specifying the newer version in requirements.txt and rebuilding the environment, the old version of the SSH provider remained, causing my DAG to fail.

A diagnostic DAG confirmed that the Airflow worker was still using the old library version, not the one specified in the updated requirements.txt.

Environment Details

• composer-local-dev version: v0.9.4
• Cloud Composer image version: composer-2.10.1-airflow-2.10.2
• Operating System: Linux, Debian 13
• Docker version: 28.3.0

Steps to Reproduce

1. Start with a clean environment

2. Start the local environment:

   composer-dev start ...

   Wait for the environment to build and become healthy. At this point, apache-airflow-providers-ssh==3.14.0 is installed. This is a constrain from composer version

3. Update the requirements file to specify a newer version. Overwrite requirements.txt with:

   apache-airflow-providers-ssh>=4.0.0
   apache-airflow-providers-sftp>=5.1.0
   

4. Rebuild the environment to apply the changes:

   composer-dev restart ...

5. Check the version inside the image

$ airflow providers get apache-airflow-providers-ssh
Provider                     | Version
=============================+========
apache-airflow-providers-ssh | 4.1.0

6. Add a diagnostic DAG to check the installed package's method signature. Create a new file dags/version_checker.py:

import datetime as dt
import inspect
import sys

from airflow.decorators import dag, task
from airflow.providers.sftp.hooks.sftp import SFTPHook
from airflow.providers.ssh.hooks.ssh import SSHHook


@dag(
    dag_id="provider_version_checker",
    start_date=dt.datetime(2023, 1, 1),
    schedule=None,
    catchup=False,
    tags=["debug"],
)
def provider_version_checker_dag():
    @task
    def check_versions():
        """
        Prints the versions of key packages and inspects the SSHHook.__init__ signature.
        """
        print(f"Using Python version: {sys.version}")

        print(f"Using SSHHook: {inspect.getmodule(SSHHook)}")
        print(f"Using SFTPHook: {inspect.getmodule(SFTPHook)}")

        with open("/opt/python3.11/lib/python3.11/site-packages/airflow/providers/ssh/__init__.py", "r") as f:
            print(f.read())

    check_versions()


provider_version_checker_dag()

7. Run the provider_version_checker DAG and inspect the task logs.

Expected Behavior

After rebuilding the environment in step 4, the process should recognize the change in requirements.txt and install apache-airflow-providers-ssh>=4.0.0.

The provider_version_checker DAG should succeed, and its logs should show the version newer than 4.0.0

Actual Behavior

The environment rebuild does not update the package. The old version (3.14.0) of apache-airflow-providers-ssh persists.

The task logs show the following, proving the old version is still active:

[2025-07-03, 15:51:16 UTC] {logging_mixin.py:190} INFO - Using Python version: 3.11.8 (main, Mar 12 2024, 15:38:35) [GCC 9.4.0]
[2025-07-03, 15:51:16 UTC] {logging_mixin.py:190} INFO - Using SSHHook: <module 'airflow.providers.ssh.hooks.ssh' from '/opt/python3.11/lib/python3.11/site-packages/airflow/providers/ssh/hooks/ssh.py'>
[2025-07-03, 15:51:16 UTC] {logging_mixin.py:190} INFO - Using SFTPHook: <module 'airflow.providers.sftp.hooks.sftp' from '/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/sftp/hooks/sftp.py'>
[2025-07-03, 15:51:16 UTC] {logging_mixin.py:190} INFO - # Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#
# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
# OVERWRITTEN WHEN PREPARING DOCUMENTATION FOR THE PACKAGES.
#
# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
# `PROVIDER__INIT__PY_TEMPLATE.py.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
#
from __future__ import annotations
import packaging.version
from airflow import __version__ as airflow_version
__all__ = ["__version__"]
__version__ = "3.14.0"
if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
    "2.8.0"
):
    raise RuntimeError(
        f"The package `apache-airflow-providers-ssh:{__version__}` needs Apache Airflow 2.8.0+"
    )

The logs also reveal:

1. The SSHHook is loaded from the system site-packages (/opt/python3.11/lib/...), which contains the provider versions baked into the base Composer image.
2. The SFTPHook is loaded from the user site-packages (/home/airflow/.local/lib/...), which is where packages from requirements.txt are installed.

[ahidalgob]

The version 0.10.2 containing this fix will be released soon. Thank you for your contribution