Utilize StreamSafeCUDAAllocator to support fast GC in new executor #37642

From00 · 2021-11-28T05:54:39Z

PR types

New features

PR changes

Others

Describe

We have impletemented a StreamSafeCUDAAllocator in #37290, yet in this PR we do the following things to support fast GC in new executor:

Add a memory::InSameStream function, which is used in mutil-stream Tensor::mutable_data function to avoid repeatedly alloc memory of the same stream
Implement a RecordStreamForGC function in Interpretercore, which synchronizes the vars for a cross-stream OP, i.e., calls RecordStream before deleting CUDA tensor
Apply fast GC using the above implemented functions when FLAGS_fast_eager_deletion_mode and FLAGS_use_stream_safe_cuda_allocator are both set to true

paddle-bot-old · 2021-11-28T05:54:43Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

wanghuancoder

已线下讨论H2D的一种场景，需要添加逻辑支持。

paddle/fluid/framework/tensor.cc

Aurelius84

Great work~~

Aurelius84 · 2021-11-29T07:46:17Z

paddle/fluid/framework/new_executor/new_executor_defs.cc

@@ -693,6 +693,29 @@ const std::vector<size_t>& Instruction::GCCheckVars() const {
  return gc_check_var_list_;
 }

+void Instruction::InferNeedStreamSyncVars() {
+  auto ParseIndexFromMap = [](
+      std::map<std::string, std::vector<int>>& index_map,


Suggested change

std::map<std::string, std::vector<int>>& index_map,

const std::map<std::string, std::vector<int>>& index_map,

Thanks. This code has been deleted, please check the new commit.

paddle/fluid/framework/new_executor/new_executor_defs.cc

paddle/fluid/memory/allocation/stream_safe_cuda_allocator.cc

paddle/fluid/framework/new_executor/interpretercore.cc

paddle/fluid/framework/new_executor/new_executor_defs.cc

paddle/fluid/framework/new_executor/new_executor_defs.h

paddle/fluid/framework/new_executor/stream_analyzer.cc

From00 · 2021-11-30T03:20:52Z

已线下讨论H2D的一种场景，需要添加逻辑支持。

代码已修改，新的设计如下：

OP跑在哪个stream，其申请的显存就在哪个stream。对应到新执行器，目前H2D和D2H申请的显存都在单独的stream，其它计算OP申请的显存默认在主stream。
OP执行完，CheckGC之前，gc_check_var_list_里的变量都会做recored。如果是同stream的record，allocator中会做判断后直接返回，开销不会很大。

Aurelius84

LGTM

… utilize-stream-safe-cuda-allocator-in-new-executor-gc

Shixiaowei02 · 2021-12-22T02:45:45Z

paddle/fluid/framework/tensor.cc

@@ -235,5 +237,13 @@ void Tensor::ResetHolderWithType(std::shared_ptr<memory::Allocation> holder,

 void Tensor::set_type(const proto::VarType::Type& type) { type_ = type; }

+#if defined(PADDLE_WITH_CUDA) || defined(PADDLE_WITH_HIP)


在不同设备共用的核心数据结构中，原则上禁止用宏定义控制 class / struct 中成员的类型或个数。确有需要的，需讨论后再合入。上个提交已经加入的，正在商量修改方法去除

macros used in Tensor/LoDTensor declaration will severely block work progresses on Pten, please modify.

Done, thanks.

Shixiaowei02 · 2021-12-22T02:45:57Z

paddle/fluid/framework/tensor.h

@@ -290,6 +290,10 @@ class Tensor {
    return *inplace_version_counter_;
  }

+#if defined(PADDLE_WITH_CUDA) || defined(PADDLE_WITH_HIP)


同上，需要修改

代码已经修改，在Tensor类中去除了宏控制。

Shixiaowei02 · 2021-12-22T02:52:22Z

paddle/fluid/memory/allocation/allocator_facade.cc

@@ -151,11 +151,12 @@ class AllocatorFacadePrivate {
        }


#37978
如果可能，这个PR里，请改用 unique_ptr<Allocation> 替代 shared_ptr<Allocation>

可以，后续新写一个PR对base_ptr_test.cu单测中的shared_ptr替换成unique_ptr

jim19930609 · 2021-12-22T03:00:00Z

paddle/fluid/framework/tensor.cc

@@ -235,5 +237,13 @@ void Tensor::ResetHolderWithType(std::shared_ptr<memory::Allocation> holder,

 void Tensor::set_type(const proto::VarType::Type& type) { type_ = type; }

+#if defined(PADDLE_WITH_CUDA) || defined(PADDLE_WITH_HIP)
+void Tensor::RecordStream(const gpuStream_t& stream) {


Adding backend/device specific interfaces to backend-agnostic data structures such as Tensor is strongly unrecommended, and will block works related to Pten and Dygraph Refactor.

Taking "Tensor::RecordStream" as an example. When adding this interface to Tensor, the principle is that the interface has to "exist" whatsoever (regardless of the macro, the backend, the way you compile...), but you may use macros inside a source file to control how this interface is implemented towards different backends.

The macros have been deleted, please check the new commit.

jim19930609

macros used in Tensor/LoDTensor declaration will severely block work progresses on Pten, please modify.

…e/Paddle into utilize-stream-safe-cuda-allocator-in-new-executor-gc

… utilize-stream-safe-cuda-allocator-in-new-executor-gc

jim19930609

LGTM

… utilize-stream-safe-cuda-allocator-in-new-executor-gc

paddle/fluid/memory/allocation/allocator_facade.cc

zhiqiu

LGTM

wanghuancoder reviewed Nov 29, 2021

View reviewed changes

paddle/fluid/framework/tensor.cc Outdated Show resolved Hide resolved

Aurelius84 reviewed Nov 29, 2021

View reviewed changes

zhiqiu reviewed Nov 29, 2021

View reviewed changes

Aurelius84 previously approved these changes Nov 30, 2021

View reviewed changes

From00 dismissed Aurelius84’s stale review via efd363b November 30, 2021 04:48

chenwhql and others added 12 commits December 1, 2021 12:04

fix reshape move storage error

47e5ae1

remove needless set type

b913475

alloc tensor by shared storage

76099a7

Utilize StreamSafeCUDAAllocator to support fast GC in new executor

febdcba

Fix compile error for Windows and ROCm

f64b0b9

Fix compile error for Windows

e0a4ab8

Modify UT stream_safe_cuda_alloc_test

f94b7d0

Modify UT stream_safe_cuda_alloc_test

7643f6e

Rewrite fast GC

3fa9045

Rewrite fast GC

a361e00

Fix compile error for BOOST_GET_CONST

4366e6e

Fix compile error for BOOST_GET_CONST

f3737e6

From00 added 4 commits December 15, 2021 17:46

Fix CI error

b556656

Add UT for fast GC

350e189

Fix CI error

caa1b6e

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

e6e3024

… utilize-stream-safe-cuda-allocator-in-new-executor-gc

Shixiaowei02 requested changes Dec 22, 2021

View reviewed changes

Shixiaowei02 reviewed Dec 22, 2021

View reviewed changes

jim19930609 reviewed Dec 22, 2021

View reviewed changes

jim19930609 requested changes Dec 22, 2021

View reviewed changes

zhiqiu and others added 7 commits December 23, 2021 15:57

add device-agnostic stream class

b52d1bb

add stream.h

46df30c

fix ut

8995d23

fix cpu compile

21d023f

Use RWLock in GetAllocator

0b5130c

Merge commit 'refs/pull/38391/head' of https://github.com/PaddlePaddl…

bad5118

…e/Paddle into utilize-stream-safe-cuda-allocator-in-new-executor-gc

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

d99133f

… utilize-stream-safe-cuda-allocator-in-new-executor-gc

From00 requested review from Shixiaowei02, jim19930609 and zhiqiu December 25, 2021 04:50

jim19930609 previously approved these changes Dec 27, 2021

View reviewed changes

Fix CI error

642b52c

From00 dismissed jim19930609’s stale review via 642b52c December 27, 2021 04:35

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

c2b638b

… utilize-stream-safe-cuda-allocator-in-new-executor-gc

Shixiaowei02 reviewed Dec 28, 2021

View reviewed changes

paddle/fluid/memory/allocation/allocator_facade.cc Show resolved Hide resolved

Shixiaowei02 approved these changes Dec 28, 2021

View reviewed changes

zhiqiu approved these changes Dec 28, 2021

View reviewed changes

jim19930609 approved these changes Dec 28, 2021

View reviewed changes

chenwhql approved these changes Dec 28, 2021

View reviewed changes

zhiqiu merged commit 0c7153a into PaddlePaddle:develop Dec 28, 2021

From00 deleted the utilize-stream-safe-cuda-allocator-in-new-executor-gc branch December 30, 2021 06:14

	std::map<std::string, std::vector<int>>& index_map,
	const std::map<std::string, std::vector<int>>& index_map,

		@@ -235,5 +237,13 @@ void Tensor::ResetHolderWithType(std::shared_ptr<memory::Allocation> holder,

		void Tensor::set_type(const proto::VarType::Type& type) { type_ = type; }

		#if defined(PADDLE_WITH_CUDA) \|\| defined(PADDLE_WITH_HIP)

Utilize StreamSafeCUDAAllocator to support fast GC in new executor #37642

Utilize StreamSafeCUDAAllocator to support fast GC in new executor #37642

Uh oh!

Conversation

From00 commented Nov 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR types

PR changes

Describe

Uh oh!

paddle-bot-old bot commented Nov 28, 2021

Uh oh!

wanghuancoder left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Aurelius84 left a comment

Choose a reason for hiding this comment

Uh oh!

Aurelius84 Nov 29, 2021

Choose a reason for hiding this comment

Uh oh!

From00 Nov 30, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

From00 commented Nov 30, 2021

Uh oh!

Aurelius84 left a comment

Choose a reason for hiding this comment

Uh oh!

Shixiaowei02 Dec 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

From00 Dec 25, 2021

Choose a reason for hiding this comment

Uh oh!

Shixiaowei02 Dec 22, 2021

Choose a reason for hiding this comment

Uh oh!

From00 Dec 25, 2021

Choose a reason for hiding this comment

Uh oh!

Shixiaowei02 Dec 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

From00 Dec 25, 2021

Choose a reason for hiding this comment

Uh oh!

jim19930609 Dec 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

From00 Dec 25, 2021

Choose a reason for hiding this comment

Uh oh!

jim19930609 left a comment

Choose a reason for hiding this comment

Uh oh!

jim19930609 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhiqiu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

From00 commented Nov 28, 2021 •

edited

Loading

Shixiaowei02 Dec 22, 2021 •

edited

Loading

Shixiaowei02 Dec 22, 2021 •

edited

Loading

jim19930609 Dec 22, 2021 •

edited

Loading