Unable to scan library (I have a lot of books)

[YOZZOPANOZZO]

I tried multiple times to tun a library scan but i always get this error:

could this be caused by the high number of books i have, around 10000 ?
Would it be possibile to import them in batches so it doesn't crash?

at AudibleApi.RestMessageValidator.ThrowStrongExceptionsIfInvalid(String message, Uri requestUri)
at AudibleApi.ApiMessageHandler.ProcessResponse(HttpResponseMessage response, CancellationToken cancellationToken)
at System.Net.Http.MessageProcessingHandler.<>c.b__5_0(Task1 task, Object state) --- End of stack trace from previous location --- at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at AudibleApi.Api.AdHocAuthenticatedGetAsync(HttpRequestMessage request, IHttpClientActions client) at AudibleApi.Api.AdHocAuthenticatedGetAsync(String requestUri, IHttpClientActions client) at AudibleApi.Api.getLibraryAsync(String parameters) at AudibleApi.Api.GetLibraryAsync(LibraryOptions libraryOptions) at AudibleApi.Api.getAllLibraryItemsAsync(LibraryOptions libraryOptions) at AudibleApi.Api.GetAllLibraryItemsAsync(ResponseGroupOptions responseGroups) at AudibleUtilities.ApiExtended.getItemsAsync(ResponseGroupOptions responseGroups, Boolean importEpisodes) in C:\Dropbox\DinahsFolder\coding\_NET\Visual Studio 2022\Libation\AudibleUtilities\ApiExtended.cs:line 134 at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates1 shouldRetryResultPredicates, Func5 onRetryAsync, Int32 permittedRetryCount, IEnumerable1 sleepDurationsEnumerable, Func4 sleepDurationProvider, Boolean continueOnCapturedContext)
at Polly.AsyncPolicy.ExecuteAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext) at ApplicationServices.LibraryCommands.scanAccountAsync(ApiExtended apiExtended, Account account, ResponseGroupOptions libraryResponseGroups) in C:\Dropbox\DinahsFolder\coding\_NET\Visual Studio 2022\Libation\ApplicationServices\LibraryCommands.cs:line 159 at ApplicationServices.LibraryCommands.scanAccountsAsync(Func2 apiExtendedfunc, Account[] accounts, ResponseGroupOptions libraryResponseGroups) in C:\Dropbox\DinahsFolder\coding_NET\Visual Studio 2022\Libation\ApplicationServices\LibraryCommands.cs:line 143
at ApplicationServices.LibraryCommands.ImportAccountAsync(Func`2 apiExtendedfunc, Account[] accounts) in C:\Dropbox\DinahsFolder\coding_NET\Visual Studio 2022\Libation\ApplicationServices\LibraryCommands.cs:line 86
at LibationWinForms.Dialogs.IndexLibraryDialog.IndexLibraryDialog_Shown(Object sender, EventArgs e) in C:\Dropbox\DinahsFolder\coding_NET\Visual Studio 2022\Libation\LibationWinForms\Dialogs\IndexLibraryDialog.cs:line 34

Thanks

[rmcrackan]

That is indeed a lot of books. It shouldn't be a problem though; I pull them from audible in batches of a few hundred. Can you attach your log file(s)? You can find it in Settings > "Open log folder"

[YOZZOPANOZZO]

Log202203.log

[YOZZOPANOZZO]

with debug on:
logs.txt

[rmcrackan]

Thanks for the logs. Your 2nd log isn't working but that's probably due to the outage github had this morning.

Oof. Yup, it's a size issue. In theory audible supports getting info for up to 40,000 titles: max 1,000 titles per page, max 40 pages. In practice, trying to get the info for batches of 1,000 times out. I set batch size to 250 since that seems to work for nearly all users. 40 * 250 = 10,000, so it has issues after 10,000.

1: I'll add better error handling so 10k+ doesn't throw errors. That will be in the next version

2: I'm not sure what to do for your case. Do you want extreme measures for the books older than 10k? Or are you ok with those?

[YOZZOPANOZZO]

I currently have around 10800 books so I am a little bit over 10K. Is there anything I could change ( maybe the batch size of 250) to get it to work? Otherwise, I could just start getting rid of some books …. 😅

Thanks

[rmcrackan]

Thanks again for your help. Bug fixed in v6.7.1

• Get the latest
• Then get this and unzip: AudibleApi.zip
• In the folder with Libation's exe and dll.s, copy/paste these new unzipped files to overwrite the existing one

If I did everything right and if you don't time out, then you should now be able to read a library of up to 12k titles. When the next Libation updates, the limit will go back to 10k. That won't matter though -- you'll already have those oldest ones and they won't be deleted.

This limitation only affects scanning your library. ie: getting the list of all of your books and their details. Every other thing Libation does will process each book individually.

[YOZZOPANOZZO]

Thanks! I'll try it out.

[YOZZOPANOZZO]

Unfortunaly i'm still getting the error. I did a clean install. I noticed you changed the batch size to 300. I wonder why it's not working still.... could it be amazon's fault?
_Log202203.log

[rmcrackan]

Interesting. It's not a 40 page limit after all. It's a 10k title limit. Earlier it broke at 250 books per page * 40 pages = 10,000. Now it returns the "Implied library size is unsupported" error on page 34: 300 * 34 = 10,200. Looks like they put a hard limit on me. Well damn -- I don't see a way around this. However, I will fix the bug that only stops after 40 pages rather than 10k titles.

[YOZZOPANOZZO]

No biggie, 10k it's way more then enough. Should I get my library size down to exactly 10K? or will it import 10K even though I have more?
Thank you again for taking the time and effort in fixing this.

[rmcrackan]

Thank you for your understanding. v6.7.2 has the fix that should at least stop the errors. Do not use with the zipped dll.s I previously put in this issue -- the new version of that is where the fix lives.

The new version should pull in the first 10,000 then stop trying to fetch more. That is: it should act as though your library were 10k books long. Podcasts are different. For this calculation, a whole podcast counts as 1 title, then the episodes are pulled separately.

The frustrating thing is I can handle any number of books. Libation handles them 1 at a time. In a since, during download and decrypt, every library is only 1 book long. Then it goes to the next library of 1 book. The problem is that audible won't let me pull the basic info about those books that you own.

[YOZZOPANOZZO]

That did it! it imported exactly 10000 books. Thanks!

Quick question, is it normal for a book ( around 200mb) to take up to 20/25 minutes to decrypt?

[rmcrackan]

I would call that unusual. Most of mine take no more than a minute or 3. Assuming my ISP is behaving and audible isn't having issues. Audible US was having issues earlier today but that seems to be cleared up. Oh, unless you're doing mp3s -- they take significantly longer because that's a whole extra step to convert the entire book from m4b => mp3.

[wtanksleyjr]

I can't prove some books take a long time, because I really can't be bothered to do the work, but sometimes I'll find what looks like the same book being loaded over a long period of time. Sometimes I ignore it (and everything finishes), while sometimes I ctrl-C it and restart. I don't think any processing step should take as long as you're describing. I can, however, report that ctrl-C and try again works.

…

-Wm

On Fri, Mar 18, 2022 at 1:26 PM rmcrackan ***@***.***> wrote: I would call that unusual. Most of mine take no more than a minute or 3. Assuming my ISP is behaving and audible isn't having issues. Audible US was having issues earlier today but that seems to be cleared up. Oh, unless you're doing mp3s -- they take significantly longer because that's a whole extra step to convert the entire book from m4b => mp3. — Reply to this email directly, view it on GitHub <#210 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7H6PWKJSUMVUT7AVSQ2LVATRI5ANCNFSM5Q6LHBHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mkb79]

@rmcrackan

The frustrating thing is I can handle any number of books. Libation handles them 1 at a time. In a since, during download and decrypt, every library is only 1 book long. Then it goes to the next library of 1 book. The problem is that audible won't let me pull the basic info about those books that you own.

Tomorrow I found out, that you can get the numbers of items in the library. If you request the library, the response headers will have a total-count key. This contains the number on books!

Edit: I mean the morning instead of Tomorrow!

[rmcrackan]

@mkb79 That's awesome! I never noticed that. Now to figure out how to discover which titles are beyond the 10,000 limit

[mkb79]

@rmcrackan
Yeah. I searched a solution for the same issue. I had to request every bunch one after one. Now I can implement a function to retrieve the pages asynchronous. I will implement a library request without any response_groups and num_results=0 to get a quick response with the number of items!

I also found out yesterday, that there is sometimes a continuation-token in the response headers of a library request and they can put in the params of the next library request. So you don’t have to handle with pages. But this is still synchronously 🙁! So I would prefer the method above.

[rmcrackan]

without any response_groups and num_results=0 to get a quick response with the number of items!

That's a great trick.

I wish there were an easy way to embed your library in C#. Yours is fantastic. I'd much rather use yours rather than maintain my own. Sadly I've spend A LOT of time researching ways to interop python and C# and there's no viable way to do this for non-trivial code.

sometimes a continuation-token in the response headers of a library request and they can put in the params of the next library request

Can you elaborate on how to get this token? I wonder if this token would get around the 10,000 title limit problem

[mkb79]

An example. You request this page: https://api.audible.de/1.0/library?response_groups=product_desc,product_attrs,sku,media,contributors,is_finished,origin_asin,order_details,product_details,category_ladders,percent_complete,is_playable,is_removable,is_visible,badge_types,relationships,customer_rights,is_archived&image_sizes=1215,408,360,882,315,570,252,558,900&include_pending=true&marketplace=AN7V1F1VY261K. There is no page key in the param query, so you got only 100 items from your library. In the response headers you got a Continuation-Token like eyJz….

Your next request contains the token like this https://api.audible.de/1.0/library?continuation_token=eyJz…&marketplace=AN7V1F1VY261K&response_groups=product_desc,product_attrs,sku,media,contributors,is_finished,origin_asin,order_details,product_details,category_ladders,percent_complete,is_playable,is_removable,is_visible,badge_types,relationships,customer_rights,is_archived&include_pending=true&image_sizes=1215,408,360,882,315,570,252,558,900. This gives me the rest of my library, because I don’t have more than 200 items in my library. But if you have more, it would give you a new Continuation-Token in the response headers, which you can use in your next request.

This way you don’t have to use pages and the 40 pages limit.

[YOZZOPANOZZO]

After switching to m4b you suggested @rmcrackan now it takes around 10 seconds per book..Thanks!

While downloading I sometimes get this message, as soon as I click continue the backup resumes just fine though so it's not a bid deal.

[rmcrackan]

now it takes around 10 seconds per book

That sounds more like it. Lossless decrypt is very fast; mp3 conversion takes quite a while. That error message concerns me though. Can you attach the log file?

[YOZZOPANOZZO]

It'has been running for a while but it's seems the error went away. I will post the logs if it shows up again.

Any update on adding more than 10k? If I understood correctly it's technically feasible (?)

[rmcrackan]

Any update on adding more than 10k? If I understood correctly it's technically feasible (?)

I have created a feature request to explore this possibility: #214

[YOZZOPANOZZO]

I got the error I mentioned above. Here's the logs https://pastebin.com/UADe6JSQ

[rmcrackan]

Please attach the log file. It reports errors deeper than the errors that make it to the screen.

It looks like something though which might be fixed by restarting Libation and retrying your previous action.

[Mbucari]

I've added some error handling for that socket exception. I didn't realize that since the Downloader runs on an anonymous thread, there are no try-catch blocks to handle any exceptions. I'll push it a little later.