-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix : SFTP Sensor fails to locate file when file_pattern is provided #25705
Conversation
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
|
It seems like the tests need some work to fix temp file usasges. |
Yep. Fix would be nice here. |
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
Background : Pull request # 24084 added the ability to provide
fnamtch
file pattern to SFTP sensor. But sensor fails to locate the file when the file pattern is provided.Issue : Code fails to get modified time once file matching to the given pattern is found. Line # 77 from sftp.py file fails with no such file error.
mod_time = self.hook.get_mod_time(actual_file_to_check)
Root cause: Code assumes that the
get_file_by_pattern
method returns a complete path for a file matching the givenfnamtch
expression. While we are only getting file name in return.get_file_by_pattern
internally relies on Paramiko SFTP clients listdir() method to retrieve files contained in given path folder.listdir()
method only returns file names and not the complete file path.Related code, sftp.py file line # 68.
file_from_pattern = self.hook.get_file_by_pattern(self.path, self.file_pattern)
Fix : Prepending the path to
actual_file_to_check
variable fixes the issue.actual_file_to_check = os.path.join(self.path, file_from_pattern)
Related : 24084