Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding asyncio to tap to allow for concurrent polling of reports #13

Merged
merged 1 commit into from
Feb 21, 2018

Conversation

agaman
Copy link
Contributor

@agaman agaman commented Feb 20, 2018

The long pole (pardon the pun) in this tap appears to be waiting for each account's reports to be generated by the Bing Ads API, and then polling for each of them one-at-a-time in serial.

This code change adds an event loop at the entry point of the tap in the main method and converts several underlying functions into coroutines.

There are two places in the code where concurrency is introduced. In do_sync_all_accounts and sync_reports the underlying tasks to sync account data and report data, respectively, are run concurrently and awaited.

Finally, in poll_report a call to time.sleep() has been replaced with an awaited call to asyncio.sleep(). This is where concurrency is actually introduced and where another task will be scheduled on the event loop if the report is not yet ready to download.

I have tested this code locally using my own developer key and my company's data, but would be glad to share the non-sensitive parts of my (admittedly brittle) integration tests.

This is my first PR for a Singer project and I'd love feedback about how I could make it better.

@cmerrick
Copy link
Contributor

Hi @agaman, thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes.

@cmerrick
Copy link
Contributor

You did it @agaman!

Thank you for signing the Singer Contribution License Agreement.

@KAllan357
Copy link
Contributor

👍

@KAllan357 KAllan357 merged commit 6bc30fa into singer-io:master Feb 21, 2018
prijendev pushed a commit that referenced this pull request Oct 7, 2021
Adding asyncio to tap to allow for concurrent polling of reports
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants