[BUG] DP compress+DPLR+GPU results in bug in building neighbor list in LAMMPS

[Liu-RX]

Bug summary

I was running DPMD of a system with 1023 water plus one OH- ion with DPLR model, when I found the bug. The DP energy model of the DPLR is compressed. I found for certain atomic geometries, the LAMMPS would fail on the neighbor-list-building step, with an error saying Out of range atoms - cannot compute PPPM (src/KSPACE/pppm.cpp:1839). However, I do not know why these geometries could result in the bug, while others do not.
For example, the following geometry file would result in the bug above.
lammps-input.zip

DeePMD-kit Version

2.2.7

TensorFlow Version

2.9.0

How did you download the software?

Offline packages

Input Files, Running Commands, Error Log, etc.

lammps-input.zip

Steps to Reproduce

run lammps with the input files.

Further Information, Files, and Links

No response

[wanghan-iapcm]

It seems that some atoms has large force predictions (result from ill-posed configuraiton) thus the simulation crashes.
you may check the quality of the model by model deviation or reducing the dt to avoid ill-posed configurations.

[Liu-RX]

I reran the simulation with dt=0.05 fs, and it could successfully continue, but dt=0.1 fs does not work. The uncompressed model with dt = 0.5 fs works. The compressed model on the CPU also works.
I did not train other models, so I cannot do the model deviation test. However, I have tested the model on some 64-water systems as a test set, and the atomic force deviation is ~0.8 eV/Angstrom. I thought the force deviation is already good enough.

[njzjz]

You didn't update the neighbor list every step, which is unsafe when an atom has high velocity.

[njzjz]

Do you mean you cannot get the same forces from uncompressed/compressed model on CPUs/GPUs?

[wanghan-iapcm]

Do you mean you cannot get the same forces from uncompressed/compressed model on CPUs/GPUs?

This happens when distance between atoms falls out of the tabulation range, i.e. out of the range of training data.

[Liu-RX]

You didn't update the neighbor list every step, which is unsafe when an atom has high velocity.

The simulation temperature is 330 K. I think the atomic velocities won't become too high.

[Liu-RX]

Do you mean you cannot get the same forces from uncompressed/compressed model on CPUs/GPUs?

Sorry I did not explain the situation clearly enough. I just checked out some of the atomic forces from the compressed/uncompressed model on CPU/GPU.
I can get the same forces from the uncompressed model on CPU/GPU. I can also get the same forces from the compressed model on CPU/GPU. The atomic force difference on x/y/z on each atom between the compressed and uncompressed model is less than 0.01 eV/A.
The problem is that, if I set the number of neighbor list update steps to N (with neigh_modify every N delay 0 check no exclude type 1 3 command), the compressed model on GPU will fail on the Nth step. For example, if I set the neighbor list update steps to 2, it will fail on the 2nd step. if I set the neighbor list update steps to 5, it will fail on the 5th step, etc. This failure does not come up for the compressed model on CPU. It also does not come up for the uncompressed model on either GPU or CPU. So I wonder if it is a bug.

[wanghan-iapcm]

You didn't update the neighbor list every step, which is unsafe when an atom has high velocity.

The simulation temperature is 330 K. I think the atomic velocities won't become too high.

depends on your training data. If they were generated at 100K, the 330K simulation would be definitely too high.
it would be helpful if you report how did you generate your training data.

[Liu-RX]

The training data is generated by DP-GEN, and the MD simulation temperature in DP-GEN is also 330 K. The only difference is that the training data is 64-water system, while this DPMD system is a 1024-water system. I have tried the compressed model on 64-water and 256-water DPMD (at 330 K), and they both work well.

[njzjz]

Can you obtain the same energy/forces with different every N settings?

[Liu-RX]

Yes, I obtained the same results with the compressed model with different N. I just tried out N=2, 5, 8, and the energy/atomic forces are the same for both CPU and GPU.

[wanghan-iapcm]

The training data is generated by DP-GEN, and the MD simulation temperature in DP-GEN is also 330 K. The only difference is that the training data is 64-water system, while this DPMD system is a 1024-water system. I have tried the compressed model on 64-water and 256-water DPMD (at 330 K), and they both work well.

In 1024 water system it seems that you have encountered local configurations that are never sample from the 64 water system.
I would recommend doing dpgen in a boarder range of thermodynamic conditions to increase the data coverage.

The reason why un-compressed dp model works is that it usually has the the ability to "extrapolate" to some extend, but it is not true for the compressed model.

[Liu-RX]

Thank you so much. Now I could understand the problem. I further ran some 4096-water system DPMD, and found out that the uncompressed model could also give the same error. I still have one more problem, though. Does 'a boarder range of thermodynamic conditions' mean sampling at higher temperatures or sampling at higher temperatures as well as in a wider density range?

[njzjz]

Could you share the model? I could take a look in this month.