Implement tensor handling for ToTensor()

[syedsalman137]

🚀 The feature

The ToTensor() function of torchvision.transforms must be able to handle torch Tensors. If it gets a tensor, it must return the same tensor without modification

Motivation, pitch

The function ToTensor can take a NumPy array as input. But if a torch tensor is passed to it, the function raises an error. It will be better if the function can take PyTorch tensors too without raising error.

Alternatives

No response

Additional context

No response

cc @vfdev-5 @datumbox

[pmeier]

Hey @SalmanHabeeb. That is a good proposal and we actually already implemented this in our prototype namespace:

vision/torchvision/prototype/transforms/functional/_type_conversion.py

Lines 33 to 34 in 149edda

	else: # isinstance(inpt, torch.Tensor):
	output = image

However, ToTensor is soft-deprecated for a while now and we will actually deprecate it with the new API:

vision/torchvision/prototype/transforms/_deprecated.py

Lines 21 to 24 in 149edda

	warnings.warn(
	"The transform `ToTensor()` is deprecated and will be removed in a future release. "
	"Instead, please use `transforms.Compose([transforms.ToImageTensor(), transforms.ConvertImageDtype()])`."
	)

Thus, I don't want to introduce any new features on the current stable transform.

While the prototype transform also takes numpy images, unfortunately there is no stable transform except for ToTensor that does. Meaning, if you have tensor and numpy images, you can only use it and wait for us the finish the new API.

That being said, if you have PIL image and tensors, there is the stable tranform

vision/torchvision/transforms/transforms.py

Line 141 in 149edda

class PILToTensor:

I would be happy to review a PR that makes this a no-op for tensors.

[syedsalman137]

@pmeier I have implemented your suggestions in a fork at
https://github.com/SalmanHabeeb/vision/blob/16dd5d5e48e2ed9ad7fc9589a66a5540eb6d8c52/torchvision/transforms/transforms.py#L163

Yet to make pull request for the same. Do I need to make any new unit tests? Or is this enough?

[datumbox]

@SalmanHabeeb as Philip said, at this point we don't want to update the ToTensor any further. This is because the implementation does too many things, for example rescaling the input as opposed to just converting them to tensors. This is why it's an idiom we try to phase out on the new API.

I am going to close the issue as the proposal doesn't align with our plans for the new Transforms API. If you would like to contribute to the project, we have several potential issues we would love the help of the community. See here:
help wanted

[pmeier]

@datumbox As suggested by me, the patch was made to PILToTensor, which will not be deprecated:

vision/torchvision/prototype/transforms/_type_conversion.py

Line 40 in 0610b13

class PILToTensor(Transform):

Given that we have the same tensor no-op behavior in the new ToImageTensor transform, I would be ok to also have it here.

I acknowledge that going by the name, passing tensors to PILToTensor is little odd. On the other hand, users currently don't have a universal conversion transform in the stable API except for ToTensor. Meaning, no combination of transforms can currently replace ToTensor. Allowing tensor pass-through in PILToTensor still doesn't support numpy images, but it is a start.

@SalmanHabeeb Please wait with further patches until we have aligned if we want this change or not. As for your question: yes, we also need tests for the new behavior if we decide we want it. Plus, your current change is wrong, since it would return nothing in case we encounter a tensor. Instead of pass it should be return pic.

[datumbox]

@pmeier Thanks for the comments.

This is quite nuanced. I'll write my concerns here in more detail (mentioning things you already know) so that my position is clearer. The PILToTensor's behaviour to be a no-op for Tensors is the result of _transformed_types = (PIL.Image.Image,). This is where every new Transform Class has the ability to define dispatchable types. On the other hand, kernels such as pil_to_tensor always assume they are invoked with the correct types.

The operation quoted at to_image_tensor is not a no-op in-case of pure tensor. We just don't need to do anything else other than wrapping it to an Image object.

My main concern about extending ToTensor is that:

1. it's deprecated on the new API and we should discourage people from using it.
2. receiving a tensor should still do a rescaling which is quite counter intuitive. A tensor will be given in one scale and then the same tensor is casted to a different scale despite the naming of the transform not hinting anything about it.

For the above reasons, my recommendation is not to add any further magic features in ToTensor, document clearly on the new API our decision to move away from it and offer better alternatives to our users.