Features/sftp to wasb: Transfer to scan files from sftp source path and upload them to Azure Blob Storage #14254
Conversation
boring-cyborg
bot
added
area:dev-tools
kind:documentation
provider:microsoft-azure
|
The Workflow run is cancelling this PR. Building image for the PR has been cancelled |
|
Nice @wolvery. One more suggestion, can you add a system test for this change? Here is an example system test: https://github.com/apache/airflow/blob/master/tests/providers/microsoft/azure/transfers/test_file_to_wasb_system.py |
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*. |
tests/providers/microsoft/azure/transfers/test_sftp_to_wasb_system.py
Outdated
Show resolved
Hide resolved
|
LGTM, just a nit..Can you use pure assert in tests(Because of this PR #12951) |
airflow/providers/microsoft/azure/example_dags/example_sftp_to_wasb.py
Outdated
Show resolved
Hide resolved
airflow/providers/microsoft/azure/example_dags/example_sftp_to_wasb.py
Outdated
Show resolved
Hide resolved
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*. |
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*. |
|
The Workflow run is cancelling this PR. Building image for the PR has been cancelled |
|
Can you rebase again? |
wolvery
requested review from
ashb,
dimberman,
potiuk,
ryanahamilton and
XD-DENG
as code owners
wolvery
force-pushed
the
features/sftp_to_wasb
branch
2 times, most recently
from
5a19657 to
bda3c1a
Compare
sftp_transfer provides mechanism to extract file from a sftp path or files using a wildcard.
Changes test_get_sftp_files_map to uses a proper name
…_wasb.py Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
…_wasb.py Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
wolvery
force-pushed
the
features/sftp_to_wasb
branch
from
bda3c1a to
b3bd462
Compare
Sure |
I have performed rebase and saw the execution of these tests running ok at local breeze environment. Is there something that I need to repair? |
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
github-actions
bot
added
the
full tests needed
github-actions
bot
removed
the
full tests needed
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
|
@ashb you can take a look once more |
|
Hi @ashb, I hope you can check this PR, again, please |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
Description
Transfer to scan files from sftp source path and upload them to Azure Blob Storage
Use case / motivation
Scan a source path and upload files using a wildcard or the complete source path of a file at SFTP and upload these files to a Azure Blob Storage.
PR Created
Process to upload sftp files to blob storage #9683
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.