Replaces DashMap with HashMap in secondary indexes

[brooksprumo]

Problem

This is a copy of the original PR to upstream solana: solana-labs#34955. Please read the description and the discussion afterwards.

Summary of Changes

• Corrects test names
• Replaces DashMap with HashMap as the secondary index entry type

Results

Anecdotally I have heard from a few RPC providers that have run this patch. I'll ask them to reply here as well, and did want to share the summary that results have been positive. RAM usage is significantly reduced, and performance was not impacted. One noted that servicing getProgramAccounts actually was a bit faster with this PR!

A quote:

I’m quite happy that we can finally run 3 indexes on a 512GB baremetal

[brooksprumo]

@vovkman Can you share your experiences?

[brooksprumo]

@stakeconomy Can you comment on the results you've seen too?

[brooksprumo]

@dnut Have you been running this patch on any production RPC nodes? If yes, how has your experience been?

[vovkman]

I've been running tests on a portion of prod traffic and not seeing any noticeable performance difference. Can confirm the account index size shrinks to about 230GB so allows running on smaller ram servers.

[brooksprumo]

I've been running tests on a portion of prod traffic and not seeing any noticeable performance difference.

👍

Can confirm the account index size shrinks to about 230GB so allows running on smaller ram servers.

For posterity and comparison, how much memory is used without this patch?

[stakeconomy]

We have this patch running on all our RPC nodes for 2-3 months without any problems

[stakeconomy]

We have seen over 1TB memory usage on 64 core machines. Somehow more cores = more memory usage with dashmap

[vovkman]

I've been running tests on a portion of prod traffic and not seeing any noticeable performance difference.

👍

Can confirm the account index size shrinks to about 230GB so allows running on smaller ram servers.

For posterity and comparison, how much memory is used without this patch?

On the same machine, 745gb without this patch (32c, 1tb)

[brooksprumo]

@carllin @jeffwashington Requesting your review here. Is there any additional testing you feel is necessary?

@t-nelson Tagging you here so you know about it, and can opt to champion a backport 😉

[jeffwashington]

looks right to me. I don't have a lot of experience with rpc.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.1%. Comparing base (9403ca6) to head (098be28).
Report is 5 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##           master    #1137     +/-   ##
=========================================
- Coverage    82.1%    82.1%   -0.1%     
=========================================
  Files         880      880             
  Lines      235665   235634     -31     
=========================================
- Hits       193716   193634     -82     
- Misses      41949    42000     +51     

[ultd]

@dnut Have you been running this patch on any production RPC nodes? If yes, how has your experience been?

I'll step in here for @dnut. Yes, we've been running this patch in prod across 100+ boxes for past 4 months without issues. Reduced RAM requirements significantly.

[t-nelson]

We have seen over 1TB memory usage on 64 core machines. Somehow more cores = more memory usage with dashmap

DashMap has some static allocations per-core

[t-nelson]

@t-nelson Tagging you here so you know about it, and can opt to champion a backport 😉

done!

[mergify]

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.

[msetodev]

Hey there. What's the best branch to clone to merge this in? I have tried 1.17.16 and I got build failures.