Issues when switching from Hoomd ver 2 to 4.2

[ShubhamsPhD]

Hi all, I am currently updating my hoomd 2.9.7 code to 4.2. The script runs but it takes about almost 10 times the time longer to do identical tasks my ver 2 code did. I would love to hear some insights as to what or why this might be happening. A small snippet of the NPT step from both my codes are below:

The version 2.9.7 one (takes about 8 mins to run):

steps = 1e6
npt_int = hoomd.md.integrate.npt(group=all, kT=305/units['T'], tau=1000/units['t'],
P=1.0/units['P'], tauP=10000/units['t'], couple='xyz')
hoomd.run_upto(current+steps, limit_multiple=1e3)
npt_int.disable()
current = current + steps

The version 4.2.1 one (takes 81 mins to run):

npt = hoomd.md.methods.ConstantPressure(
filter=all_filter,
thermostat=hoomd.md.methods.thermostats.MTTK(kT=305/units['T'], tau=1000/units['t']),
S=1.0/units['P'],
tauS=10000/units['t'],
couple='xyz')
integrator.methods.append(npt)
sim.run(1000000)
integrator.methods.remove(npt)
hoomd.write.GSD.write(state=sim.state, mode='xb', filename='endNPTshort.gsd')

I would be glad to provide more context if required. And any help would be greatly appreciated.

[joaander]

Are you possibly running the v2 script on the GPU and the v4 on the CPU? Also, 4.2 is 18 months old at this point. You should upgrade to 5.1.1.

In any case, you certainly did not provide enough context. There are no known performance bugs with the NPT integration methods in general. You didn't mention what forces you apply to particles -- in most simulations, the neighbor list and pair force take most of the time.

If you can provide a full (and minimal!) example script that demonstrates what you think is slow performance on 5.1.1, I can run it with a profiler and see what I find, or perhaps the reason will be obvious when I see the full script. It is also possible that as you remove parts of your full script to create a minimal example, you discover the part that slows the simulation yourself.

[ShubhamsPhD]

Thanks for responding so quickly. I have attached parts of our overall script. I too suspect it might do with how we import in the forces. It's almost a direct translation of a skin lipid self-assembly code we used in 2.9.7, which I tried not to change too much in order to reproduce our previous results with this new version.

Could you please take a look if possible and suggest any recommended changes, which I suspect that might be a few. The input gsd file is created using mbuild ver. 0.10.5.

Thanks again
Stu
cgRun.txt

[joaander]

I see no obvious cases that would cause slow performance in your script.

As I said above, you are going to need to try removing operations and forces until you find which one is causing the slow down. Then you can prepare a minimal script (one that requires no input files and is relatively short) that demonstrates the problem.

[ShubhamsPhD]

Hi Josh

Thank you for your reply.. I’ve done a more detailed performance comparison between HOOMD v2.9.7 and v4.2.1/v5.1.1, and I’ve observed a significant slowdown across multiple stages of the simulation, not just the NPT step. Both scripts were made to be identical in the operations they performed including the number of timesteps (= 1000 just for this test) and the properties of each NVE/NPT step. Below is the breakdown of each step:

1. Input file setup/loading: Slightly faster in v5 (0.14s vs. 0.27s).

2. Context & simulation initialization: Takes longer in v5 (0.38s vs. 0.27s).

3. Pair potentials setup: Faster in v5 (0.09s vs. 0.41s)

4. NVE Run 1 (105 K): Much slower in v5 (11.21s vs. 1.54s)

5. NVE Run 2 (305 K): Much slower in v5 (5.36s vs. 0.82s)

6. Short NPT equilibration: Slower in v5 (4.13s vs. 0.63s)

7. Shape annealing: Slower in v5 (3.90s vs. 0.60s)

8. NPT temperature annealing: Slower in v5 (4.51s vs. 0.62s)

9. NPT relaxation: Slower in v5(4.51s vs. 0.64s)

10. Production NPT run: Slower in v5 (4.55s vs. 0.66s)

I am also running both versions on the same GPU. Before I create a minimal example, I wanted to check whether you think any of these steps in the newer versions might be expected to take longer due to changes in implementation (e.g., if theres a different handling of forces, neighbor lists, or pressure coupling).

Do you have any recommendations on specific profiling tools/settings in HOOMD that could help pinpoint the bottleneck? I’ll proceed with creating a minimal script by removing operations too, but I’d appreciate any guidance on where I should focus first.

Thanks again for all your help.

[chrisjonesBSU]

Just as a quick update as I was chatting with @ShubhamsPhD over slack about this. I suspect performance differences are likely arising from Hoomd 2.9 being compiled with single-precision enabled, and Hoomd 4 using a mix of single and double (per the defaults). The simulations are being ran on an A40, which I think is especially bad with double precision compared to single.

[joaander]

Yes, that would explain it. A40 has 1:64 DP:SP compute (https://www.techpowerup.com/gpu-specs/a40-pcie.c3700).