-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Add SSH/SFTP support for ObjectStoragePath #60757
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add SSH/SFTP support for ObjectStoragePath #60757
Conversation
2517925 to
b092dbc
Compare
b092dbc to
71e7d1e
Compare
fritz-astronomer
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm.
I see lots of tests for connection parsing, and I was going to comment about having an integration test - but I suppose as long as the connection is passed as expected, sshfs is going to handle all that.
Were you able to do a full test with an SSH/SFTP system?
Thanks for the review, @fritz-astronomer! Yes, I spun up a local SFTP server with Docker and ran through the |
|
Hi @bolkedebruin, hope you're doing well. I would like to request your review on this PR when you have a moment. Please take your time, and I truly appreciate any feedback you might have. Thank you so much - it means a lot! 🙏 |
* 60558 (feature) : added ssh sftp object store * fix depreciation warning * documentation clean ups * clean ups
* 60558 (feature) : added ssh sftp object store * fix depreciation warning * documentation clean ups * clean ups
Summary
Adds SSH/SFTP protocol support for Airflow's
ObjectStoragePathabstraction, enabling file operations over SFTP using existing Airflow
connections.
get_fs()function using thesshfslibrary(fsspec-compatible)
sftp://andssh://URI schemescredentials, key files)
Usage
Changes
implementation
Installation
pip install apache-airflow-providers-sftp[sshfs]
closes: #60558
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.