Skip to content

File access rework #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 31, 2018
Merged

File access rework #1

merged 4 commits into from
Aug 31, 2018

Conversation

Querz
Copy link
Owner

@Querz Querz commented Aug 31, 2018

This is a complete rework of the pipeline that loads files, processes them and then saves the result to disk with an aim for abstraction so this can be used for everything that loads, processes and saves files.

Explanation

There is a "MCAFilePipe" class now that runs 3 ThreadPoolExecutors:

  • One is for loading files in order, because loading files in order is faster than loading a lot of files at the same time (at least for HDDs).
  • The second one is for processing the loaded data on as many Threads as Runtime.getRuntime().availableProcessors() returns.
  • The third one is for saving the resulting data. This usually happens in 1-4 Threads.

A custom workflow for loading, processing and saving can be implemented by extending LoadDataJob, ProcessDataJob and SaveDataJob, see for example RegionImageGenerator.

@Querz Querz merged commit cc41bee into master Aug 31, 2018
@Querz Querz deleted the file-access-rework branch April 6, 2021 11:44
HoldYourWaffle added a commit to HoldYourWaffle/mcaselector that referenced this pull request Jul 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant