Skip to content

Conversation

@Phu2
Copy link
Contributor

@Phu2 Phu2 commented May 8, 2025

@Phu2 Phu2 requested review from dr0i and katauber and removed request for katauber May 8, 2025 12:41
@Phu2
Copy link
Contributor Author

Phu2 commented May 8, 2025

Unfortunately 4bf31e7 does not prevent the behavior described in #159 (comment)

This big JSON file was created by processing a specific OAI endpoint with >50k records. I can reproduce this behavior with this same endpoint. Lots of requests (per resumptionToken) are being made while the result file is still empty. At some point in time the requests come to a halt and the result file is written very fast.

@TobiasNx
Copy link
Contributor

TobiasNx commented May 9, 2025

Unfortunately 4bf31e7 does not prevent the behavior described in #159 (comment)

This big JSON file was created by processing a specific OAI endpoint with >50k records. I can reproduce this behavior with this same endpoint. Lots of requests (per resumptionToken) are being made while the result file is still empty. At some point in time the requests come to a halt and the result file is written very fast.

I assume that the oaipmh opener does not stream data through but buffer until all data is fetched, @dr0i can you confirm this?

@dr0i dr0i assigned Phu2 May 9, 2025
@dr0i dr0i added this to Metafacture May 9, 2025
@dr0i dr0i moved this to Working in Metafacture May 9, 2025
@Phu2 Phu2 closed this May 9, 2025
@github-project-automation github-project-automation bot moved this from Working to Done in Metafacture May 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants