How to correctly copy particle data from GPU to CPU in AMReX?

[Afraid31]

Hello everyone,

I’m currently developing a custom solver based on the AmrLevelAdv tutorial example. As part of this work, I need to extract particle data from the GPU to the CPU for post-processing and coupling with other modules.

I followed a suggestion from a previous discussion and tried the following approach:

for (MyParIter pti(*this, level); pti.isValid(); ++pti) {
    const Long np = pti.numParticles();

    // Option 1
    using HostTile = typename ContainerLike<std::allocator>::ParticleTileType;

    // Option 2
    // using HostTile = typename ImmersedBoundaryPoints::ParticleTileType;

    HostTile host_tile;
    host_tile.resize(np);

    auto& gpu_tile = DefineAndReturnParticleTile(level, pti);
    amrex::copyParticles(host_tile, gpu_tile);

    auto& host_particles = host_tile.GetArrayOfStructs();
    for (int i = 0; i < np; ++i) {
        auto& p = host_particles[i];
        Real fx = -p.rdata(Pibm::ufx) * F_c;
        ......
    }
}

However, I encounter different errors depending on which option I use:

Option 1:

I get a runtime error during the copyParticles call:
amrex::Abort::0::CUDA error 700 in file .../AMReX_GpuDevice.cpp line 689: an illegal memory access was encountered !!!

Option 2:

The code compiles and runs past the copy stage, but it crashes at:

Real fx = -p.rdata(Pibm::ufx) * F_c;
with a segmentation fault.

Some additional context:

1. ImmersedBoundaryPoints is a custom class derived from:

class ImmersedBoundaryPoints
    : public amrex::AmrParticleContainer<Pibm::nattribs, Pibm_i::nattribs, 0>
{
public:
    using Base = amrex::AmrParticleContainer<Pibm::nattribs, Pibm_i::nattribs, 0>;
    using ParticleType = typename Base::ParticleType;
    ......
};

2. The same code works perfectly when compiled in CPU mode. The errors occur only when CUDA is enabled.

My main question is:

What is the correct and recommended way to copy particle data from GPU memory to CPU memory so that I can safely access it on the host?

If there are specific examples or idiomatic patterns for doing this in AMReX, I would greatly appreciate any pointers or code snippets.

Thank you very much!

[AlexanderSinn]

I think there was a mistake in the answer to the first discussion. std::allocator is allocated and accessible on the CPU; however, it is not accessible by the GPU. Instead, amrex::PinnedArenaAllocator should be used, which is also allocated in CPU memory but is accessible by both the CPU and GPU.

using HostTile = typename ContainerLike<amrex::PinnedArenaAllocator>::ParticleTileType;

This should work; however, from a performance standpoint, I would expect it to be better to not copy the particles to the CPU and to instead copy the solver output to the GPU and use ParallelFor to update the particles.

[Afraid31]

Thank you, your answer is very helpful. Copying the solver data directly from CPU to GPU is indeed the optimal solution. However, I am currently facing an issue: the correspondence between particles and solid boundary points is stored in the particle's AoS. Therefore, I need to copy the particles from GPU to CPU to determine which solid boundary point each particle corresponds to.

I wonder if there is an alternative approach to maintain the particle–boundary point correspondence without having to copy the particles back to the CPU?

[atmyers]

What kind of data structure are you using to store the solver data? Could the look up be ported to GPU as well?

[Afraid31]

Thanks a lot! You’re right — this is definitely the best solution.