Skip to content

Conversation

@anishgirianish
Copy link
Contributor

@anishgirianish anishgirianish commented Jan 18, 2026


Summary

Adds SSH/SFTP protocol support for Airflow's ObjectStoragePath
abstraction, enabling file operations over SFTP using existing Airflow
connections.

  • Implements get_fs() function using the sshfs library
    (fsspec-compatible)
  • Supports sftp:// and ssh:// URI schemes
  • Integrates with existing SFTP connection configuration (host, port,
    credentials, key files)

Usage

from airflow.sdk import ObjectStoragePath                              
                                                                       
# Read/write files over SFTP                                           
path = ObjectStoragePath("sftp://my_sftp_conn@server/path/to/file.txt")
content = path.read_text()                                             
                                                                       
# Copy between backends                                                
path.copy(ObjectStoragePath("s3://aws_conn@bucket/file.txt"))          
                                                                                                                               

Changes

  • providers/sftp/src/airflow/providers/sftp/fs/sftp.py - Core
    implementation
  • providers/sftp/tests/unit/sftp/fs/test_sftp.py - Unit tests
  • providers/sftp/provider.yaml - Register filesystem
  • providers/sftp/pyproject.toml - Add sshfs optional dependency
  • providers/sftp/docs/filesystems/sftp.rst - Documentation

Installation

pip install apache-airflow-providers-sftp[sshfs]

closes: #60558

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@anishgirianish anishgirianish mentioned this pull request Jan 18, 2026
2 tasks
@anishgirianish anishgirianish force-pushed the feature/60558-ssh-sftp-object-store branch from 2517925 to b092dbc Compare January 18, 2026 22:12
@anishgirianish anishgirianish force-pushed the feature/60558-ssh-sftp-object-store branch from b092dbc to 71e7d1e Compare January 18, 2026 23:51
Copy link
Contributor

@fritz-astronomer fritz-astronomer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.
I see lots of tests for connection parsing, and I was going to comment about having an integration test - but I suppose as long as the connection is passed as expected, sshfs is going to handle all that.

Were you able to do a full test with an SSH/SFTP system?

@anishgirianish
Copy link
Contributor Author

lgtm. I see lots of tests for connection parsing, and I was going to comment about having an integration test - but I suppose as long as the connection is passed as expected, sshfs is going to handle all that.

Were you able to do a full test with an SSH/SFTP system?

Thanks for the review, @fritz-astronomer! Yes, I spun up a local SFTP server with Docker and ran through the
basic operations - reads, writes, listing, deletes all worked fine.

@anishgirianish
Copy link
Contributor Author

Hi @bolkedebruin, hope you're doing well. I would like to request your review on this PR when you have a moment. Please take your time, and I truly appreciate any feedback you might have. Thank you so much - it means a lot! 🙏

@bolkedebruin bolkedebruin merged commit 8e372e1 into apache:main Jan 25, 2026
100 checks passed
suii2210 pushed a commit to suii2210/airflow that referenced this pull request Jan 26, 2026
* 60558 (feature) : added ssh sftp object store

* fix depreciation warning

* documentation clean ups

* clean ups
shreyas-dev pushed a commit to shreyas-dev/airflow that referenced this pull request Jan 29, 2026
* 60558 (feature) : added ssh sftp object store

* fix depreciation warning

* documentation clean ups

* clean ups
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SSH/SFTP Object Store

3 participants