Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Serving 调用时Out of Memory #664

Closed
omtbreak opened this issue Jun 9, 2020 · 8 comments
Closed

Serving 调用时Out of Memory #664

omtbreak opened this issue Jun 9, 2020 · 8 comments
Assignees
@MRXLT @bjjwwang
Labels
question Further information is requested

Comments

@omtbreak
Copy link

omtbreak commented Jun 9, 2020

  1. 在服务器上使用0号显卡或者1号显卡开启目标检测模型，并且服务器显卡显存充足
    截屏2020-06-0915 51 22

  2. 使用RPC方式调用时，服务器端报错out of memory
    截屏2020-06-0915 52 28

并且按照指示设置

export FLAGS_fraction_of_gpu_memory_to_use=0.9

同样出现问题，没有解决
@MRXLT
Copy link
Contributor

MRXLT commented Jun 9, 2020

你好，可以在启动server服务时，添加--mem_optim True参数开启显存优化

@omtbreak
Copy link
Author

omtbreak commented Jun 9, 2020

--mem_optim True

解决了谢谢

@omtbreak omtbreak closed this as completed Jun 9, 2020
@omtbreak omtbreak reopened this Jun 10, 2020
@omtbreak
Copy link
Author

你好，可以在启动server服务时，添加--mem_optim True参数开启显存优化

您好，在持续调用多张图片的情况下，添加--mem_optim True参数后，仍出现这个问题
截屏2020-06-1010 33 09

@MRXLT
Copy link
Contributor

MRXLT commented Jun 10, 2020

你好，请问使用的模型是什么样的，启动服务时显卡上的可用显存是多少

@omtbreak
Copy link
Author

你好，请问使用的模型是什么样的，启动服务时显卡上的可用显存是多少

启动的模型时cascade rcnn，启动服务时显卡情况就如截屏2020-06-0915 51 22所示

@omtbreak
Copy link
Author

你好，请问使用的模型是什么样的，启动服务时显卡上的可用显存是多少

有可能是，服务器端再完成一次图片的检测后，并不释放当前使用的显存，而当下一次调用来时，又去重新申请新的显存。

@bjjwwang
Copy link
Collaborator

现在暂时是把thread最小限制压缩到2，可以编译一下再试试看显存消耗问题

@TeslaZhao TeslaZhao added the wontfix This will not be worked on label Aug 31, 2020
@TeslaZhao TeslaZhao added question Further information is requested and removed wontfix This will not be worked on labels Aug 31, 2020
@sevenold
Copy link 

export FLAGS_fraction_of_gpu_memory_to_use=0.9

python -m paddle_serving_server_gpu.serve --model server_model/server_rec  --port 9294 --gpu_id 2 --mem_optim --thread 2 --max_body_size 1073741824

image

模型是paddleOCR

  • ch_det_r50_vd_db
  • ch_rec_r34_vd_crnn
  • 显存11G

一张类似PDF的图片就直接OOM!!

@wangjiawei04

@paddle-bot paddle-bot bot closed this as completed Mar 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MRXLT MRXLT

@bjjwwang bjjwwang

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@TeslaZhao @omtbreak @MRXLT @bjjwwang @sevenold