Adding asyncio to tap to allow for concurrent polling of reports #13
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The long pole (pardon the pun) in this tap appears to be waiting for each account's reports to be generated by the Bing Ads API, and then polling for each of them one-at-a-time in serial.
This code change adds an event loop at the entry point of the tap in the
main
method and converts several underlying functions into coroutines.There are two places in the code where concurrency is introduced. In
do_sync_all_accounts
andsync_reports
the underlying tasks to sync account data and report data, respectively, are run concurrently and awaited.Finally, in
poll_report
a call totime.sleep()
has been replaced with an awaited call toasyncio.sleep()
. This is where concurrency is actually introduced and where another task will be scheduled on the event loop if the report is not yet ready to download.I have tested this code locally using my own developer key and my company's data, but would be glad to share the non-sensitive parts of my (admittedly brittle) integration tests.
This is my first PR for a Singer project and I'd love feedback about how I could make it better.