Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Very slow to pull charges for the last year. #2284

Closed
benwmills opened this issue Feb 11, 2021 · 12 comments
Closed

Very slow to pull charges for the last year. #2284

benwmills opened this issue Feb 11, 2021 · 12 comments
Assignees
@remi-stripe

Comments

@benwmills
Copy link

We're trying to make a simple call to get all our charges for the last year. We're using stripe.net version 37.6.0 to make this request from the .NET Framework 4.7. The code is really simple:

        var options = new ChargeListOptions
        {
            Created = new DateRangeOptions
            {
                GreaterThanOrEqual = DateTime.Today.AddDays(1).AddYears(-1),
                LessThanOrEqual = DateTime.Today.AddDays(1)
            },
            Limit = 100
        };

        var charges = _stripeChargeService.ListAutoPaging(options).ToList();

For some reason this runs extremely slow. I let it run for 30 minutes before I had to stop the call. We're only pulling 3,700 records, so it doesn't seem like the call should take this long.

When we download these records in the Stripe UI, it only takes a minute or 2 to build the CSV download.
@remi-stripe remi-stripe self-assigned this Feb 11, 2021
@remi-stripe
Copy link
Contributor

@benwmills Thanks for reaching out. Github issues should be limited to bugs with the library itself. Your question here is more about the API itself and overall latency and it would be better suited for our support team. They can look into your account, your logs and better understand what's causing this. You can contact them here: https://support.stripe.com/contact

Overall, listing charges (or most resources) can be quite slow as you need to render many objects. The Charge API especially is quite a large object to render and paginate through. When listing 3700 charges sequentially you need to do 37 separate calls which can each take up to 30 seconds to complete. Overall it shouldn't be 30 minutes but it will definitely not take a few seconds (or even a minute or two). When you need to do bulk data exports, you should prefer alternatives such as the Dashboard exports, Sigma queries in the Dashboard or the API or using the Reporting API such as https://stripe.com/docs/reports/report-types#schema-balance-change-from-activity-itemized-1 instead.

@benwmills
Copy link
Author

I do still think this might be a bug in the library.

I was using the ListAutoPaging and it never completed. I just made a change to do manual paging using the standard List method (ListAsync actually) and each query of 100 is taking just a few seconds (like you said). I wonder if the ListAutoPaging is broken and stuck in a loop.

@remi-stripe remi-stripe reopened this Feb 12, 2021
@remi-stripe
Copy link
Contributor

@benwmills Would you be able to add some logs to the pagination logic to track where it gets stuck and provide maybe a repro?

@benwmills
Copy link
Author

In my calling code? It's just the call to ListAutoPaging.

Or do you mean in the code in this GitHub repo? I took a peek at the code and followed the trail from that method. I suspect it may be an issue here where it calls a sync or async method depending on the version of .NET:

stripe-dotnet/src/Stripe.net/Services/_base/Service.cs

Line 260 in 5f97b09

protected IEnumerable<T> ListRequestAutoPaging<T>(

I haven't tested this yet, but I also suspect that the ListAutoPagingAsync method will work just fine. I'll report back.

@remi-stripe
Copy link
Contributor

@benwmills Sorry yes I meant logs in your own code. Basically does it suddenly stops returning data and there's no new log, or is the code just running slower over time and making it appear as if it's not doing anything?

Also what is your environment like (exact .NET version and stripe-dotnet version)?

It's definitely possible that there's a bug here though this code has been here for a while. Would you be able to also provide an example id from your account such as as Test request log id from https://dashboard.stripe.com/test/logs That would help me lookup the logs quickly on our end to see whether the requests are stopping abruptly (After being fast for a while) or if they are just slowing down/spaced out

@benwmills
Copy link
Author

If you look at the code in the original post, you'll see it's just a single call to ListAutoPaging. The code works fine until that point and then that line hangs forever. I've let it run for 40 minutes before. There's not really anything to log.

The versions number are in the original post too.

We turned this code off at some point in the last year (it's a non essential report) and I was asked to revisit. I know it worked at some point. It always took several minutes to run, but nothing like this. It's hard to pinpoint when it got this slow.

At this point, I have a workaround doing the paging myself. In my case, I have about 3700 records to pull, so I have to pull 37 pages of data, but each page only takes a few seconds, so the entire loop is maybe a minute or 2. I don't know why ListAutoPaging is still running after 40 minutes, but I have to think that it's pretty broken. Like I said, I have a feeling there's some off sync/async mix up going on in there.

@remi-stripe
Copy link
Contributor

If you look at the code in the original post, you'll see it's just a single call to ListAutoPaging. The code works fine until that point and then that line hangs forever. I've let it run for 40 minutes before. There's not really anything to log.

The code will run sequential calls to the List Charges API though and return a page of 100 objects at a time while you paginate through the list. It doesn't automatically go and ingest every single charge in the account. This means that you can log explicitly for each page to figure out whether the code is becoming slower over time (for example after 10 pages it takes 1 second instead of 500ms, and after 100 pages it takes 7 seconds instead of 500ms, etc.) or if it is stable and just suddenly hangs.

For example you could try something like this:

var options = new ChargeListOptions
{
  Created = new DateRangeOptions
  {
    GreaterThanOrEqual = DateTime.Today.AddDays(1).AddYears(-1),
    LessThanOrEqual = DateTime.Today.AddDays(1)
  },
  Limit = 100
};

var service = new ChargeService();
// Synchronously paginate
long index = 1;
foreach (var charge in service.ListAutoPaging(options)) {
  // Log the first and last element on the page
  if(index % options.Limit == 1 )
  {
    Console.WriteLine(DateTime.Now.ToString() + " - Start of a page - #" + index + " - " + charge.Id); 
  }
  else if (index % options.Limit == 0 )
  {
    Console.WriteLine(DateTime.Now.ToString() + " - End of a page - #" + index + " - " + charge.Id); 
    Console.WriteLine("Fetching next page...");
  }
  index++;
}

Console.WriteLine("Done paginating");

This is my current output when running it for a few a bit:

2/14/2021 12:07:36 PM - Start of a page - #1 -  ch_xxx
2/14/2021 12:07:36 PM - End of a page - #100 -  ch_xxx
Fetching next page...
2/14/2021 12:07:39 PM - Start of a page - #101 -  ch_xxx
2/14/2021 12:07:39 PM - End of a page - #200 -  ch_xxx
Fetching next page...
2/14/2021 12:07:43 PM - Start of a page - #201 -  ch_xxx
2/14/2021 12:07:43 PM - End of a page - #300 -  ch_xxx
Fetching next page...
2/14/2021 12:07:47 PM - Start of a page - #301 -  ch_xxx
2/14/2021 12:07:47 PM - End of a page - #400 -  ch_xxx
Fetching next page...
2/14/2021 12:07:51 PM - Start of a page - #401 -  ch_xxx
2/14/2021 12:07:51 PM - End of a page - #500 -  ch_xxx
Fetching next page...
2/14/2021 12:07:56 PM - Start of a page - #501 -  ch_xxx
2/14/2021 12:07:56 PM - End of a page - #600 -  ch_xxx
Fetching next page...
2/14/2021 12:08:00 PM - Start of a page - #601 -  ch_xxx
2/14/2021 12:08:00 PM - End of a page - #700 -  ch_xxx
Fetching next page...
2/14/2021 12:08:04 PM - Start of a page - #701 -  ch_xxx
2/14/2021 12:08:04 PM - End of a page - #800 -  ch_xxx
Fetching next page...
2/14/2021 12:08:08 PM - Start of a page - #801 -  ch_xxx
2/14/2021 12:08:08 PM - End of a page - #900 -  ch_xxx
Fetching next page...
2/14/2021 12:08:12 PM - Start of a page - #901 -  ch_xxx
2/14/2021 12:08:12 PM - End of a page - #1000 -  ch_xxx
Fetching next page...
2/14/2021 12:08:17 PM - Start of a page - #1001 -  ch_xxx
2/14/2021 12:08:17 PM - End of a page - #1100 -  ch_xxx
Fetching next page...
2/14/2021 12:08:21 PM - Start of a page - #1101 -  ch_xxx
2/14/2021 12:08:21 PM - End of a page - #1200 -  ch_xxx
Fetching next page...
2/14/2021 12:08:25 PM - Start of a page - #1201 -  ch_xxx
2/14/2021 12:08:25 PM - End of a page - #1300 -  ch_xxx
Fetching next page...
2/14/2021 12:08:30 PM - Start of a page - #1301 -  ch_xxx
2/14/2021 12:08:30 PM - End of a page - #1400 -  ch_xxx
Fetching next page...
2/14/2021 12:08:34 PM - Start of a page - #1401 -  ch_xxx
2/14/2021 12:08:34 PM - End of a page - #1500 -  ch_xxx
Fetching next page...

@clement911
Copy link
Contributor

Could this be that you're on a lib version before this fix was released? #1982
#1946 (comment)

@remi-stripe
Copy link
Contributor

@clement911 Yeah I thought about that one but it was released in 35.12.1 but the original message said they are on 37.6.0. Could still be related though (version mismatch or another subtle bug of similar shape).

@benwmills
Copy link
Author

@remi-stripe, I added your logging code and I never get in to the foreach loop.

@clement911, that issue sounds very similar to what I'm seeing. My action method is async (another developer has been actively swapping them over to async), but we're making the regular ListAutoPaging call. I tried to switch to ListAutoPagingAsync, but then I'm getting a different build error:

'IAsyncEnumerable<Charge>' does not contain a definition for 'GetAwaiter' and no accessible extension method 'GetAwaiter' accepting a first argument of type 'IAsyncEnumerable<Charge>' could be found (are you missing a using directive or an assembly reference?)

At this point, I'm happy enough to manually page. ListAsync works just fine.

@clement911
Copy link
Contributor

I wonder if it could have something to do with the c# version you're using...
https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/configure-language-version

@remi-stripe
Copy link
Contributor

Going to close this out for inactivity as we haven't heard any other similar reports but let us know if you are still experiencing this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@remi-stripe remi-stripe

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@benwmills @clement911 @remi-stripe