Skip to content

Commit 91a810d

Browse files
committed
Translate lines 3 to 28 in intermediate_source/optimizer_step_in_backward_tutorial.py to Korean
1 parent 54ae0c4 commit 91a810d

File tree

1 file changed

+24
-24
lines changed

1 file changed

+24
-24
lines changed

โ€Žintermediate_source/optimizer_step_in_backward_tutorial.pyโ€Ž

Lines changed: 24 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,31 @@
11
"""
22
3-
How to save memory by fusing the optimizer step into the backward pass
3+
์˜ตํ‹ฐ๋งˆ์ด์ € ๋‹จ๊ณ„๋ฅผ ์—ญ์ „ํŒŒ ๊ณผ์ •์— ํ•ฉ์ณ์„œ ๋ฉ”๋ชจ๋ฆฌ ์ ˆ์•ฝํ•˜๊ธฐ
44
======================================================================
55
6-
Hello there! This tutorial aims to showcase one way of reducing the
7-
memory footprint of a training loop by reducing the memory taken by
8-
the *gradients*. Say you have a model and you're interested in ways to
9-
optimize memory to avoid ``Out of Memory`` (OOM) errors or simply to ooze
10-
more out of your GPU. Well, you _might_ be in luck (if gradients take up
11-
a portion of your memory and you do not need to do gradient accumulation).
12-
We will explore the following:
13-
14-
1. What takes up memory during your training or finetuning loop,
15-
2. How to capture and visualize memory snapshots to determine the bottleneck,
16-
3. The new ``Tensor.register_post_accumulate_grad_hook(hook)`` API, and finally,
17-
4. How everything fits together in 10 lines to achieve memory savings.
18-
19-
To run this tutorial, you will need:
20-
21-
* PyTorch 2.1.0 or newer with ``torchvision``
22-
* 1 CUDA GPU if you'd like to run the memory visualizations locally.
23-
Otherwise, this technique would benefit similarly on any device.
24-
25-
Let us start by importing the required modules and models. We will use a
26-
vision transformer model from torchvision, but feel free to substitute
27-
with your own model. We will also use ``torch.optim.Adam`` as our optimizer,
28-
but, again, feel free to substitute with your own optimizer.
6+
์•ˆ๋…•ํ•˜์„ธ์š”! ์ด ํŠœํ† ๋ฆฌ์–ผ์—์„œ๋Š” *๋ณ€ํ™”๋„(gradient)*๊ฐ€ ์ฐจ์ง€ํ•˜๋Š” ๋ฉ”๋ชจ๋ฆฌ๋ฅผ ์ค„์ž„์œผ๋กœ์จ
7+
ํ•™์Šต ๋‹จ๊ณ„(training loop)์—์„œ์˜ ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰์„ ์ค„์ด๋Š” ํ•œ ๊ฐ€์ง€ ๋ฐฉ๋ฒ•์„ ์†Œ๊ฐœํ•ฉ๋‹ˆ๋‹ค.
8+
๋ชจ๋ธ์„ ๊ฐ–๊ณ  ์žˆ๋Š” ์ƒํ™ฉ์—์„œ ๋ฉ”๋ชจ๋ฆฌ ๋ถ€์กฑ(Out of Memory, OOM) ์˜ค๋ฅ˜๋ฅผ ๋ฐฉ์ง€ํ•˜๊ณ  ์‹ถ๊ฑฐ๋‚˜,
9+
GPU์˜ ์„ฑ๋Šฅ์„ ์ตœ๋Œ€ํ•œ ํ™œ์šฉํ•˜๊ณ  ์‹ถ์€ ๊ฒฝ์šฐ ์ด ๋ฐฉ๋ฒ•์ด ๋„์›€์ด ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
10+
(๋ณ€ํ™”๋„๊ฐ€ ๋ฉ”๋ชจ๋ฆฌ์˜ ์ผ๋ถ€๋ฅผ ์ฐจ์ง€ํ•˜๊ณ  ์žˆ์œผ๋ฉฐ, ๋ณ€ํ™”๋„ ๋ˆ„์ ์ด ํ•„์š”ํ•˜์ง€ ์•Š์€ ๊ฒฝ์šฐ๋ผ๋ฉด ๋ง์ž…๋‹ˆ๋‹ค.)
11+
12+
์•„๋ž˜ ๋‚ด์šฉ์„ ๋‹ค๋ฃน๋‹ˆ๋‹ค:
13+
14+
1. ํ•™์Šต ๋˜๋Š” ๋ฏธ์„ธ์กฐ์ •(finetuning) ๋‹จ๊ณ„ ์ค‘ ๋ฉ”๋ชจ๋ฆฌ๋ฅผ ์ฐจ์ง€ํ•˜๋Š” ์š”์†Œ๋“ค,
15+
2. ๋ฉ”๋ชจ๋ฆฌ ์Šค๋ƒ…์ƒท์„ ์บก์ฒ˜ํ•˜๊ณ  ์‹œ๊ฐํ™”ํ•˜์—ฌ ๋ณ‘๋ชฉ ํ˜„์ƒ์„ ํŒŒ์•…ํ•˜๋Š” ๋ฐฉ๋ฒ•,
16+
3. ์ƒˆ๋กœ์šด ``Tensor.register_post_accumulate_grad_hook(hook)`` API, ๊ทธ๋ฆฌ๊ณ 
17+
4. ๋‹จ 10์ค„์˜ ์ฝ”๋“œ๋กœ ๋ฉ”๋ชจ๋ฆฌ ์ ˆ์•ฝํ•˜๋Š” ๋ฒ•.
18+
19+
์ด ํŠœํ† ๋ฆฌ์–ผ์„ ์‹คํ–‰ํ•˜๋ ค๋ฉด ๋‹ค์Œ์ด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค:
20+
21+
* ``torchvision``์ด ํฌํ•จ๋œ PyTorch 2.1.0 ํ˜น์€ ๊ทธ ์ด์ƒ์˜ ๋ฒ„์ „
22+
* ๋ฉ”๋ชจ๋ฆฌ ์‹œ๊ฐํ™”๋ฅผ ๋กœ์ปฌ์—์„œ ์‹คํ–‰ํ•˜๋ ค๋ฉด CUDA GPU 1๊ฐœ
23+
๋ฉ”๋ชจ๋ฆฌ ์‹œ๊ฐํ™”๋ฅผ ์ œ์™ธํ•˜๋ฉด ์ด ๊ธฐ๋ฒ•์€ ๋ชจ๋“  ์žฅ์น˜์—์„œ ์œ ์‚ฌํ•œ ์ด์ ์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.
24+
25+
๋จผ์ € ํ•„์š”ํ•œ ๋ชจ๋“ˆ๊ณผ ๋ชจ๋ธ์„ importํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค.
26+
์˜ˆ์‹œ์—์„œ๋Š” torchvision์˜ ๋น„์ „ ํŠธ๋žœ์Šคํฌ๋จธ ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•˜์ง€๋งŒ, ์›ํ•˜๋Š” ๋ชจ๋ธ๋กœ ๋Œ€์ฒดํ•ด๋„ ์ข‹์Šต๋‹ˆ๋‹ค.
27+
๋˜ ์˜ˆ์‹œ์—์„œ๋Š” ์˜ตํ‹ฐ๋งˆ์ด์ €๋กœ ``torch.optim.Adam``์„ ์‚ฌ์šฉํ•˜์ง€๋งŒ, ๋งˆ์ฐฌ๊ฐ€์ง€๋กœ ์›ํ•˜๋Š” ์˜ตํ‹ฐ๋งˆ์ด์ €๋กœ ๋Œ€์ฒดํ•ด๋„ ๋ฉ๋‹ˆ๋‹ค.
28+
2929
3030
"""
3131

0 commit comments

Comments
ย (0)