Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix : SFTP Sensor fails to locate file when file_pattern is provided #25705

Closed
wants to merge 3 commits into from
Closed

Conversation

ajinkya-xyz
Copy link

Background : Pull request # 24084 added the ability to provide fnamtch file pattern to SFTP sensor. But sensor fails to locate the file when the file pattern is provided.

Issue : Code fails to get modified time once file matching to the given pattern is found. Line # 77 from sftp.py file fails with no such file error.

mod_time = self.hook.get_mod_time(actual_file_to_check)

Root cause: Code assumes that the get_file_by_pattern method returns a complete path for a file matching the given fnamtch expression. While we are only getting file name in return. get_file_by_pattern internally relies on Paramiko SFTP clients listdir() method to retrieve files contained in given path folder. listdir() method only returns file names and not the complete file path.

Related code, sftp.py file line # 68.
file_from_pattern = self.hook.get_file_by_pattern(self.path, self.file_pattern)

Fix : Prepending the path to actual_file_to_check variable fixes the issue.

actual_file_to_check = os.path.join(self.path, file_from_pattern)

Related : 24084

@boring-cyborg
Copy link

boring-cyborg bot commented Aug 13, 2022

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@ajinkya-xyz ajinkya-xyz changed the title Fix : SFTP Sensor not able to locate file when file_pattern is provided Fix : SFTP Sensor fails to locate file when file_pattern is provided Aug 13, 2022
@uranusjr
Copy link
Member

It seems like the tests need some work to fix temp file usasges.

@potiuk
Copy link
Member

potiuk commented Aug 23, 2022

Yep. Fix would be nice here.

@ajinkya-xyz
Copy link
Author

@uranusjr @potiuk Facing issue in doing local setup on windows machine. Will try to spend some time on weekend.

@github-actions
Copy link

github-actions bot commented Oct 8, 2022

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Oct 8, 2022
@github-actions github-actions bot closed this Oct 13, 2022
@eladkal eladkal mentioned this pull request Oct 31, 2022
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers stale Stale PRs per the .github/workflows/stale.yml policy file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants