Skip to content

profiler failure on multi-gpu (works on single GPU #9052

Closed
@panyx0718

Description

@panyx0718

reproduced in PR: #9051

this line:
"PADDLE_ENFORCE(cudaEventRecord(event_, stream));"

terminate called after throwing an instance of 'paddle::platform::EnforceNotMet'
what(): invalid resource handle at [/paddle/dev/my/Paddle3/paddle/fluid/platform/profiler.cc:73]
PaddlePaddle Call Stacks:
0 0x7f21a215138cp paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 572
1 0x7f21a220e66dp paddle::platform::Event::Event(paddle::platform::EventKind, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, unsigned int, paddle::platform::DeviceContext const*) + 317
2 0x7f21a220f316p paddle::platform::PopEvent(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, paddle::platform::DeviceContext const*) + 246
3 0x7f21a220f943p paddle::platform::RecordEvent::~RecordEvent() + 211

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions