MarkdownDB fails with more than 500 files in a top directory

[luccasmmg]

Steps to reproduce

• Copy a directory with more than 500 markdown files inside to your content folder eg: https://github.com/openspending/community.openspending.org/tree/gh-pages/resources

Try indexing this in markdowndb, this is the outcome

I've also tried indexing this directory on a fresh flowershow app, following this guide https://flowershow.app/docs/publish-tutorial and on the export section it will silently fail, but it will build again if i remove that dir

Possible problem

I'm pretty sure the problem lies somewhere here, from what i could gather in stackoverflow, sqlite has a limit on the batch insert of 500, so a possible fix would be to slice the filesToInsert variable in batches of 499 and then insert each batch separatedly.
https://github.com/datopian/markdowndb/blob/main/src/lib/markdowndb.ts#LL181C4-L181C4

[rufuspollock]

@luccasmmg good find. do you think you could look at doing this patch? Maybe with some pointers from @olayway

[olayway]

@luccasmmg that's a good catch! This definitely needs to be solved for the first indexing run but this reminded me that we could also probably use some way to not re-index the whole content folder each time the server is restarted, although I'm not yet sure how this could be done. Anyway, I'll take care of it. Thanks for the proposed solution!

[rufuspollock]

@olayway this is probably pretty urgent as it will break all flowershow sites over 500 files right ...?

[dstengle]

It looks like this fix got lost somewhere along the way. Possible to bring it back? I can't quite figure out where it went.

[dstengle]

Found it: f6fe5b0

Looks like things got moved around here and now there is both duplicate and missing code?

[rufuspollock]

@mohamedsalem401 can you look at this asap ... is this now broken or is it fixed.

[mohamedsalem401]

@dstengle @rufuspollock
You are right, the fix needs to be re-added.

[mohamedsalem401]

This is fixed