-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implemented MSGraphSensor as a deferrable sensor #39304
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…en JSON decode error occurs
boring-cyborg
bot
added
area:providers
provider:microsoft-azure
Azure-related issues
labels
Apr 29, 2024
…raphAsyncOperator
FYI: I've seen accross different PR that following test is sometimes randomly failing:
|
…yload could not have a response key if call isn't done
…he status key present
…he status key present
…alized by MSGraphTrigger
dabla
changed the title
Implement default response handler method and suppress JSONDecodeError
Implement default response handler and use deferable mechanism in MSGraphSensor
Apr 30, 2024
dabla
changed the title
Implement default response handler and use deferable mechanism in MSGraphSensor
Implemented MSGraphSensor as a deferrable sensor
Apr 30, 2024
eladkal
approved these changes
May 5, 2024
RodrigoGanancia
pushed a commit
to RodrigoGanancia/airflow
that referenced
this pull request
May 10, 2024
* refactor: Implement default response handler method and added test when JSON decode error occurs * refactor: Reformatted some code to comply to static checks * refactor: Changed debugging level to debug for printing response in operator * docs: Added example on how to refresh a PowerBI dataset using the MSGraphAsyncOperator * refactor: Changed some info logging statements to debug * refactor: Changed some info logging statements to debug * fix: Fixed mock_json_response * refactor: Return content if response is not a JSON * refactor: Make sure the operator passes the response_handler to the triggerer * refactor: Should use get instead of directly _getitem_ brackets as payload could not have a response key if call isn't done * refactor: If event has status failure then the sensor should stop the async poke * refactor: Changed default_event_processor as not all responses have the status key present * refactor: Changed default_event_processor as not all responses have the status key present * refactor: Removed response_handler parameter as lambda cannot be serialized by MSGraphTrigger * refactor: Changed some logging statements * refactor: Updated PowerBI dataset refresh example * refactor: Fixed 2 static check errors * refactor: Refactored MSGraphSensor as a real async sensor * refactor: Changed logging level of sensor statements back to debug * refactor: Fixed 2 static checks * refactor: Changed docstring hook * refactor: Put docstring on one line --------- Co-authored-by: David Blain <david.blain@infrabel.be>
66 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implemented a default response handler which suppresses JSONDecodeError when response is empty or isn't of type json but the default response type json was specified, otherwise will return the contents and if that's empty the response headers. This can for example occur when triggering PowerBI dataset refreshes. Also added a test in TestResponseHandler and added example on how to use the operator to refresh a PowerBI dataset. This PR also removed the response_handler param as lambda expressions can't be serialized and triggers always get serialized to the database when deferred before getting executed. The most import change in thi PR is that the MSGraphSensor now also uses the deferable task mechanism like the MSGraphAsyncOperator, so it doesn't rely on the classic blocking poke method but defers the tasks in a 2 step. First it will execute the call in a deferred way and check the event, if the condition is met the sensor will stop. If not then the sensor will first defer a TimeDeltaTrigger with the specified retry_delay (default is 60 for MSGraphSensor ) and once the task completed it will redefer the poll task, that way the sensor doesn't block the workers.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.