Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Erroneous arithmetic operation is detected by operating system #4333

Closed
Vaishal16 opened this issue Oct 14, 2021 · 3 comments
Closed

Erroneous arithmetic operation is detected by operating system #4333

Vaishal16 opened this issue Oct 14, 2021 · 3 comments
Assignees

Comments

@Vaishal16
Copy link

Vaishal16 commented Oct 14, 2021

Using PP-OCR english recognition model for training purpose.
Model name : en_number_mobile_v2.0_rec

I am getting the following error while training on the above pretrained model:

C++ Traceback (most recent call last):
0 paddle::imperative::Tracer::TraceOp(std::string const&, paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap, std::map<std::string, std::string, std::less<std::string >, std::allocator<std::pair<std::string const, std::string > > > const&)
1 paddle::imperative::Tracer::TraceOp(std::string const&, paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap, paddle::platform::Place const&, bool, std::map<std::string, std::string, std::less<std::string >, std::allocator<std::pair<std::string const, std::string > > > const&)
2 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap const&)
3 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::WarpCTCKernel<paddle::platform::CUDADeviceContext, float>, paddle::operators::WarpCTCKernel<paddle::platform::CUDADeviceContext, double> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1 (closed)}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&)
4 paddle::operators::WarpCTCKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const
5 paddle::operators::math::UnpaddingLoDTensorFunctor<paddle::platform::CUDADeviceContext, int>::operator()(paddle::platform::CUDADeviceContext const&, paddle::framework::LoDTensor const&, paddle::framework::LoDTensor*, int, int, bool, paddle::operators::math::PadLayout)
6 paddle::framework::SignalHandle(char const*, int)
7 paddle::platform::GetCurrentTraceBackStringabi:cxx11

Error Message Summary:
FatalError: Erroneous arithmetic operation is detected by the operating system.
[TimeInfo: *** Aborted at 1634193209 (unix time) try "date -d @1634193209" if you are using GNU date ***]
[SignalInfo: *** SIGFPE (@0x7fc3b6a3a120) received by PID 32157 (TID 0x7fc3ff43d100) from PID 18446744072478761248 ***]

Please help me out with the issue.

@Vaishal16 Vaishal16 changed the title Erroneous arithmetic operation by operating system Erroneous arithmetic operation is detected by operating system Oct 14, 2021
@tink2123
Copy link
Collaborator

Could you provide the following details:

System information

  • PaddlePaddle version (eg.1.1)or CommitID
  • CPU: including CPUMKL/OpenBlas/MKLDNN version
  • GPU: including CUDA/CUDNN version
  • OS Platform (eg.Mac OS 10.14)

To Reproduce
Describe your current behavior
Code to reproduce the issue

According to the current error message, it may be that the CUDA environment is installed incorrectly. It is recommended to use it in docker.

You can refer to installation

@Vaishal16
Copy link
Author

CommitID : 5071292 (branch : release/2.3)
paddlepaddle-gpu==2.0.0
CUDNN version 10.0
OS : Ubuntu 18.04

To reprodce:

Code used if from branch : release/2.3.
Model and config files used are of English Recognition Model from here

Meanwhile trying out docker installation and setting up CUDA environment again.

@Vaishal16
Copy link
Author

Worked with proper CUDA version installation in Docker.

Closing the issue now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants