Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce AirConcurrentMap as possible backing HashMap #308

Closed
pjeli opened this issue Apr 6, 2022 · 1 comment · Fixed by #309
Closed

Introduce AirConcurrentMap as possible backing HashMap #308

pjeli opened this issue Apr 6, 2022 · 1 comment · Fixed by #309
Assignees
Labels
enhancement New feature or request

Comments

@pjeli
Copy link
Collaborator

pjeli commented Apr 6, 2022

Ever since #230 and #293, I have been interested in trying to find optimizations to the backing HashMap that separates files and directories of the LightweightGSet.

I have always allowed folks to try their own hand at providing a faster implementation by the configuration property nna.inode.collection.impl.

Well at long last I think I finally have one. The early results look promising: https://github.com/boilerbay/airconcurrentmap

From the ValidQueryChecker benchmark and the GSetFilteringBenchmark underneath src/test/java I have seen an 18->13 second reduction for ValidQuery and a 36->30 second reduction for GSetFiltering. These speed-ups have been consistent as I tested with 1M files, 2M files, and 4M files.

I think the results are good enough that I will soon try a cut of this on a much larger system.

@pjeli pjeli self-assigned this Apr 6, 2022
@pjeli pjeli added the enhancement New feature or request label Apr 6, 2022
pjeli pushed a commit to pjeli/NNAnalytics that referenced this issue Apr 6, 2022
pjeli pushed a commit to pjeli/NNAnalytics that referenced this issue Apr 6, 2022
@pjeli pjeli closed this as completed in #309 Apr 6, 2022
@pjeli
Copy link
Collaborator Author

pjeli commented Apr 6, 2022

Unfortunately this did not end up having the performance increase I was expecting (20 second SuggestionEngine runs became 2 minutes). I think it may be due to the way Streams and AirConcurrentMap interact. Unfortunately I can't make much more progress since Air is closed source. So debugging is a pain. I will leave this alone for now since it does work though. If anyone wants to take a crack at it go ahead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant