Skip to content

illegal memory access was encountered #52

Description

@koparasy

Hello, I am compiling gpusph with "make" then I execute " make ProblemExample".
And then I execute ./GPUSPH and I get the following error:

Device 0 thread 140735808663952 iteration 0 last command: 7. Exception: src/cuda/forces.cu(516) : in unbind_textures() @ thread 0x140735808663952 : cudaSafeCall() runtime API error 77 : an illegal memory access was encountered

The same error rises also when executing different problems.

My system information is the following:
g++ : (GCC) 6.4.0
nvcc : release 9.1, V9.1.85
GPU devices: 4 x Tesla V100-SXM2.

Activity

  1. Oblomov commented on Mar 23, 2019

    @Oblomov
    Contributor

    Hello @koparasy

    which GPUSPH version (or branch) does this happen with?

  2. koparasy commented on Mar 25, 2019

    @koparasy
    Author

    Hello @Oblomov

    I am working on the main branch.

  3. Oblomov commented on Mar 26, 2019

    @Oblomov
    Contributor

    Hello @koparasy

    can you please try the next branch and see if it's fixed there already?

  4. koparasy commented on Mar 26, 2019

    @koparasy
    Author

    I checkout to the next branch and the error persists.

  5. Oblomov commented on Mar 27, 2019

    @Oblomov
    Contributor

    I'm afraid I'm unable to reproduce the issue locally. Can you see if running GPUSPH under cuda-memcheck gives some indiciation of where the illegal access is coming from?

  6. Oblomov commented on Mar 27, 2019

    @Oblomov
    Contributor

    Also: are you running single- or multi-GPU?

  7. koparasy commented on Mar 28, 2019

    @koparasy
    Author

    The error appears on single and multi GPU runs. I attach you the output of a run with a single gpu without mpi support without hdf5 support.

    I attach the output of the cuda-memcheck.
    cudaMemCheck.txt

  8. Oblomov commented on Mar 28, 2019

    @Oblomov
    Contributor

    Thanks for the report. From the log, it would seem that the issue happens when the forcesDevice kernel tries to fetch the neighbors position, but the array where it's trying to read from should be valid. I do not have any GPU with Compute Capability 7.0 and cannot reproduce the error on my machine, so I'm afraid debugging will be a bit slow and you'll have to be my hands and eyes 8-)

    For starters, I would recommend to updated to the latest next that I just pushed, which includes a small fix for neighbors traversal. I don't think it's directly relevant to the case, but we never know.

    if the latest next (currently at commit add5af0) doesn't fix the issue, I would ask you to try the following change: in src/cuda/textures.cuh, replace the line:

    #if __COMPUTE__ >= 20 && __COMPUTE__/10 != 3
    

    with

    #if 1
    

    and see if you can replicate the error, and then again replacing it with

    #if 0
    

    and see if you can replicate the error. This should help us pinpoint a bit better the possible source of error.

  9. anthropoy commented on Jul 26, 2019

    @anthropoy

    Had the same issue on a 6.1 CUDA device. I am actually working on the wsl branch but I did merge that fix from the 'next' branch. I also tried the fix in textures.cuh, none worked.

  10. Oblomov commented on Jul 26, 2019

    @Oblomov
    Contributor

    Can you please provide the output of make show? It should be available as info/show.txt ready for export if you're on a recent enough branch.

  11. anthropoy commented on Jul 29, 2019

    @anthropoy

    show.txt

    As attached, thanks.

  12. Oblomov commented on Jul 29, 2019

    @Oblomov
    Contributor

    The Microsoft compiler suffers from this bug, which affects GPUSPH. A large part of the changes introduced in the wsl branch are specifically to work around this, but it seems you're hitting a case we missed. We'll look into it.

  13. anthropoy commented on Jul 29, 2019

    @anthropoy

    I see, thanks for looking into this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions