Update tensors and autograd #1185

subramen · 2020-10-12T20:10:57Z

Refactor content of tensors and autograd in 60 Minute Blitz

netlify · 2020-10-12T20:13:52Z

Deploy preview for pytorch-tutorials-preview ready!

Built with commit 9568cd2

https://deploy-preview-1185--pytorch-tutorials-preview.netlify.app

subramen · 2020-10-14T02:20:08Z

Add "What is PyTorch" context to blitz homepage

albanD

I only had time to look at the tensor section, will do the autograd later.

beginner_source/blitz/tensor_tutorial.py

albanD · 2020-10-14T14:16:15Z

beginner_source/blitz/tensor_tutorial.py

+#
+# **Docs Issues** - https://pytorch.org/docs/stable/tensor_attributes.html
+# is not comprehensive (missing data, grad, grad_fn, shape). Contains
+# ``memory_format`` which is not an attribute


Does that mean that we have no way to know the memory format of a given Tensor?

Looks like it, unless I'm mistaken

In [52]: a = a.to(memory_format=torch.channels_last) In [53]: a.memory_format --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-53-a1dabca59cd9> in <module> ----> 1 a.memory_format AttributeError: 'Tensor' object has no attribute 'memory_format'

beginner_source/blitz/tensor_tutorial.py

beginner_source/blitz/autograd_tutorial.py

ranman · 2020-10-31T00:38:18Z

beginner_source/blitz/autograd_tutorial.py

-on Tensors. It is a define-by-run framework, which means that your backprop is
-defined by how your code is run, and that every single iteration can be
-different.
+model = torchvision.models.resnet18(pretrained=True)


I think you need a bit more text explanation here - either in text or in code comments.

very roughly something like:
"Let's start by looking at the a single training step. Here we load an existing pretrained resnet18 model, create random input data tensor of shape 3x64x64, and some random output labels."

...

Then we might want to pause again and say "here we push the data forward through the model, calculate the loss, and then call .backward() to collect the gradients for each parameter in the model. <more detail here on what backward does (i.e. where are the gradients stored?>

....

Then we say: "Finally we load in an optimizer, in this case SGD, with a learning rate and momentum of ..., and call .step() to use our gradients and step backwards"

That's a basic model, below we dive a bit deeper into what is going on when we call .backward()

I agree with Randall here

ranman · 2020-11-02T21:40:57Z

beginner_source/blitz/autograd_tutorial.py

+# The ``requires_grad`` flag lets autograd know
+# if we need gradients w.r.t. these tensors. If it is ``True``, autograd


This doesn't appear to render correctly as text in the preview: https://deploy-preview-1185--pytorch-tutorials-preview.netlify.app/beginner/blitz/autograd_tutorial.html#differentiation-in-autograd

ranman · 2020-11-02T21:42:05Z

beginner_source/blitz/autograd_tutorial.py

+# tensors. By tracing this graph from roots to leaves, you can
+# automatically compute the gradients using the chain rule.
+#
+# In a forward pass, autograd does two things simultaneously: \* run the


are the * here supposed to render as bullet points? they don't appear to in the preview.

ranman · 2020-11-02T21:42:22Z

beginner_source/blitz/autograd_tutorial.py

+# root. Autograd then \* computes the gradients from each ``.grad_fn``, \*
+# accumulates them in the respective tensor’s ``.grad`` attribute, and \*
+# using the chain rule, propagates all the way to the leaf tensors.
+#


Are there any pictures or graphics that would work to show this?

Added an image of the DAG. Would you say the description is accurate and informative? cc: @soumith

ranman · 2020-11-02T21:43:30Z

beginner_source/blitz/autograd_tutorial.py

+print(a.requires_grad==True)
+b = x + z
+print("Does `b` require gradients?")
+print(b.requires_grad==True)


would it be helpful to show the output here?

Also would it be sufficient to simply say print(a.requires_grad) ?

ranman · 2020-11-02T21:45:43Z

beginner_source/blitz/tensor_tutorial.py


-Tensors
-^^^^^^^
+Tensors are similar to NumPy’s ndarrays, except that tensors can run on


Might be worth adding "on GPUs or other specialized hardware"

ranman · 2020-11-02T21:48:02Z

beginner_source/blitz/tensor_tutorial.py

-###############################################################
-# Construct a matrix filled zeros and of dtype long:
+tnsr_from_data = torch.tensor(data)
+tnsr_from_np = torch.from_numpy(np_array)


Is it worth calling out that you can also go back to numpy? I'm not sure it is but just throwing it out there for consideration. Maybe just link to the "bridge back to numpy" section below.

ranman · 2020-11-02T21:49:04Z

beginner_source/blitz/tensor_tutorial.py


-###############################################################
-# Addition: syntax 2
+shape = (2,3,)


I think you should add a very brief explanation of shape here. It will be obvious to most but for some it a one sentence explanation will provide a lot of value.

ranman · 2020-11-02T21:50:38Z

beginner_source/blitz/tensor_tutorial.py

 #
-# You can use standard NumPy-like indexing with all bells and whistles!
+# Tensor attributes describe their shape data type and where they live.


need commas after shape and data type - "where they live" makes sense to me but I worry it might not translate well... maybe "on which device the tensor is stored" or something similar.

soumith

see comments inline

soumith · 2020-11-03T05:08:38Z

beginner_source/blitz/tensor_tutorial.py

+# Tensor Initialization
+# ~~~~~~~~~~~~~~~~~~~~~
+#
+# Tensors can be initialized in various ways. Take a look at the following examples


each introduction of a concept followed by a code snippet. one at a time.
Cleanly separating introducing concepts is way more important than minimizing code.

Tensors can be initialized in various ways. Take a look at the following examples

Directly from data

data = [[1, 2],[3, 4]] x_data = torch.tensor(data) print(x_data)

From a NumPy array:

x_np = np.array(data) x_data = torch.from_numpy(x_np) print(x_data)

soumith · 2020-11-03T05:09:03Z

beginner_source/blitz/tensor_tutorial.py


-###############################################################
-# Construct a matrix filled zeros and of dtype long:
+tnsr_from_data = torch.tensor(data)


the use of the variable name tnsr all over the tutorial is just weird. Either call it tensor or call it x

soumith · 2020-11-03T05:10:43Z

beginner_source/blitz/tensor_tutorial.py

-# or create a tensor based on an existing tensor. These methods
-# will reuse properties of the input tensor, e.g. dtype, unless
-# new values are provided by user
+try:


if you are trying to make a point that typecasting will throw a runtime error, please do so in a Note.

Actually showcasing that behavior is not needed, and the try/catch with a verbose RuntimeError logic is more intimidating than introducing.

Agreed, showcasing typecasting here is not needed; the point here is about overriding properties during tensor init. I am removing the highlight on typecasting

soumith · 2020-11-03T05:11:44Z

beginner_source/blitz/tensor_tutorial.py

+rand_tnsr = torch.rand(shape)
+ones_tnsr = torch.ones(shape)
+zeros_tnsr = torch.zeros(shape)
+print(f"Random Tensor:\n{rand_tnsr} \n\n \


three separate print statments, one in each line are cleaner and better.

soumith · 2020-11-03T05:17:44Z

beginner_source/blitz/tensor_tutorial.py

+print(tnsr)
+
+######################################################################
+# **Both of these are joining ops, but they are subtly different.**


Introduce one joining op in words, and in code, provide a reference to the other. The tutorial needs to introduce just-enough to a new user.

You can join two tensors with torch.cat, which concatenates them in a given dimension.

t1 = torch.cat([tnsr, tnsr], dim=1) print(t1)

Also see torch.stack, another subtly different tensor joining operation.

soumith · 2020-11-03T05:30:50Z

beginner_source/deep_learning_60min_blitz.rst

+PyTorch is a Python-based scientific computing package for two broad purposes:
+
+-  A replacement for NumPy to use the power of GPUs.
+-  A deep learning research platform that provides maximum flexibility and speed


An automatic differentiation library that is often useful to implement neural networks

soumith · 2020-11-03T05:34:06Z

beginner_source/blitz/autograd_tutorial.py

+A Gentle Introduction to Autograd
+---------------------------------
+
+Autograd is PyTorch’s automatic differentiation engine that powers


Fix: The torch.autograd package is PyTorch's automatic differentiation engine

Context for Suraj: this is the real autograd package: https://github.com/HIPS/autograd , we used the name, somewhat accidentally. So, it's important to introduce torch.autograd or as the autograd pacakge, not just Autograd. Please adjust references in this tutorial to Autograd as needed after this context.

soumith · 2020-11-03T05:39:14Z

beginner_source/blitz/autograd_tutorial.py

-on Tensors. It is a define-by-run framework, which means that your backprop is
-defined by how your code is run, and that every single iteration can be
-different.
+model = torchvision.models.resnet18(pretrained=True)


I agree with Randall here

soumith · 2020-11-03T05:42:48Z

beginner_source/blitz/autograd_tutorial.py

+# .. math::
+#
+#
+#    \frac{\partial Q}{\partial a} = 9a^2


first list the original equation, for clarity, i.e. Q = 3a^3 - b^2

soumith · 2020-11-03T05:49:23Z

beginner_source/blitz/autograd_tutorial.py

+    param.requires_grad = False
+# Replace the last fully-connected layer
+# Parameters of nn.Module instances have requires_grad=True by default
+model.fc = nn.Linear(512, 100)


after this line, it's worth stating what happened

Now, all parameters in the model, except the parameters of model.fc do not compute gradients. All these parameters that don't compute gradients are usually called frozen parameters. The only parameters that compute gradients are the weights and bias of model.fc. model.fc being the last layer of this network is often called as the classifier.

Adds a DAG figure for autograd, fixes formatting issues, improve readability

* Update tensor_tutorial.py * Update autograd_tutorial.py * Fix import * Fix import * Fix import * Fixes issues in tensor, updates autograd * Adds "what is pytorch" to homepage * Fixes formatting * Fix typo * Fixes typo * fix failing test * Make suggested changes Adds a DAG figure for autograd, fixes formatting issues, improve readability Co-authored-by: Brian Johnson <brianjo@fb.com>

subramen added 2 commits September 28, 2020 23:21

Update tensor_tutorial.py

e62ee7c

Update autograd_tutorial.py

0e867c7

brianjo and others added 4 commits October 13, 2020 08:11

Merge branch 'master' into 60mb_setup

e874967

Fix import

edf2495

Fix import

6d094b7

Fix import

cd6f97a

subramen requested a review from soumith October 14, 2020 02:12

subramen marked this pull request as ready for review October 14, 2020 02:13

subramen marked this pull request as draft October 14, 2020 02:19

subramen removed the request for review from soumith October 14, 2020 02:19

albanD reviewed Oct 14, 2020

View reviewed changes

subramen added 4 commits October 20, 2020 11:24

Fixes issues in tensor, updates autograd

e0b21e3

Adds "what is pytorch" to homepage

bc50302

Fixes formatting

0d16b7b

Fix typo

6d91b98

subramen requested a review from jspisak October 20, 2020 16:46

subramen added 2 commits October 20, 2020 14:35

Fixes typo

75efe8b

fix failing test

adee684

subramen marked this pull request as ready for review October 22, 2020 15:00

subramen added 2 commits October 26, 2020 08:27

Merge branch 'master' into 60mb_setup

d7b0648

Merge branch 'master' into 60mb_setup

422296e

subramen requested review from soumith and ranman October 30, 2020 14:50

facebook-github-bot added the cla signed label Oct 30, 2020

ranman suggested changes Nov 2, 2020

View reviewed changes

soumith requested changes Nov 3, 2020

View reviewed changes

subramen and others added 2 commits November 11, 2020 09:57

Make suggested changes

9dc3c79

Adds a DAG figure for autograd, fixes formatting issues, improve readability

Merge branch 'master' into 60mb_setup

f56befe

Merge branch 'master' into 60mb_setup

9568cd2

subramen merged commit b3dd09e into pytorch:master Dec 30, 2020

holly1238 mentioned this pull request Jun 15, 2021

Adding explanation for parameter to backward in autograd tutorial #729

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update tensors and autograd #1185

Update tensors and autograd #1185

subramen commented Oct 12, 2020

netlify bot commented Oct 12, 2020 •

edited

Loading

subramen commented Oct 14, 2020 •

edited

Loading

albanD left a comment

albanD Oct 14, 2020

subramen Oct 20, 2020

ranman Oct 31, 2020

soumith Nov 3, 2020

ranman Nov 2, 2020

ranman Nov 2, 2020

ranman Nov 2, 2020

subramen Nov 11, 2020

ranman Nov 2, 2020

ranman Nov 2, 2020

ranman Nov 2, 2020

ranman Nov 2, 2020

ranman Nov 2, 2020

soumith left a comment

soumith Nov 3, 2020

soumith Nov 3, 2020

soumith Nov 3, 2020

subramen Nov 9, 2020

soumith Nov 3, 2020

soumith Nov 3, 2020

soumith Nov 3, 2020

soumith Nov 3, 2020

soumith Nov 3, 2020

soumith Nov 3, 2020

soumith Nov 3, 2020

		# The ``requires_grad`` flag lets autograd know
		# if we need gradients w.r.t. these tensors. If it is ``True``, autograd

Update tensors and autograd #1185

Update tensors and autograd #1185

Conversation

subramen commented Oct 12, 2020

netlify bot commented Oct 12, 2020 • edited Loading

subramen commented Oct 14, 2020 • edited Loading

albanD left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soumith left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netlify bot commented Oct 12, 2020 •

edited

Loading

subramen commented Oct 14, 2020 •

edited

Loading