`unhashable type: non-nested SymInt`

[bhack]

With pytorch and autoquant nighties:

    model = autoquant(
        model, 
    )

  File "/opt/conda/lib/python3.11/site-packages/torchao/quantization/autoquant.py", line 1140, in autoquant_prehook
    real_model.forward(*args, **kwargs)
    ....
    features = self.backbone(inputs)
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
...
    x = blk(x)
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/workspace/modeling/backbone/vit.py", line 362, in forward
    x = self.attn(x)
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
....
    qkv = self.qkv(x).reshape(B, H * W, 3, self.num_heads, self.head_dim).permute(0, 3, 2, 1, 4).contiguous()

[bhack]

pytorch/pytorch#135099

[bhack]

@bdhirsh @jerryzh168 Do you think it is a duplicate of pytorch/pytorch#136287?

[jerryzh168]

@bhack can you try with pytorch nightly as well

pip3 install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu124

[bhack]

It was already on pytorch nightly and ao nightly.

[jerryzh168]

I see, in that case can you give us a minimal repro for the issue

[bhack]

It is a quite large model that I cannot share. Can I give you any debug element?

[jerryzh168]

I see, maybe @bdhirsh can provide some pointers, looks like this is related to torch.compile

[bhack]

Ok, in the mean time I was seeing some example in SAM2 server for ao + AOTI. As in this case I am trying to use both what is in general the best practice to use ao with AOTI?

[jerryzh168]

some example in https://gist.github.com/sayakpaul/de0eeeb6d08ba30a37dcf0bc9dacc5c5 (ao quant + AOTI) and https://github.com/sayakpaul/diffusers-torchao/blob/main/inference/aot_serialization.md (general AOTI)

[bhack]

Thanks I see the example isn't with autoquant. Isn't currently going to play well autoquant with AOTI?

[bhack]

If this could help I have this error with

autoquant(model, qtensor_class_list=DEFAULT_FLOAT_AUTOQUANT_CLASS_LIST,min_sqnr=40)

Instead with

quantize_(model, int8_dynamic_activation_int8_weight())
model = unwrap_tensor_subclass(model)

I got

cannot mutate tensors with frozen storage

While executing %add_ : [num_users=1] = call_function[target=torch.ops.aten.add_.Tensor](args = (%to_2, %backbone_blocks_0_attn_qkv_bias), kwargs = {})
.....
 qkv = self.qkv(x).reshape(B, H * W, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4)

[bhack]

For the 2nd case where we had cannot mutate tensors with frozen storage I've moved the error removing the .to from the input example allocated on GPU directly.

But now we have the same error cannot mutate tensors with frozen storage from official timm

While executing %add__24 : [num_users=1] = call_function[target=torch.ops.aten.add_.Tensor](args = (%to_26, %backbone_blocks_0_mlp_fc2_bias), kwargs = {})
Original traceback:
....
  File "/opt/conda/lib/python3.11/site-packages/timm/layers/mlp.py", line 46, in forward
    x = self.fc2(x)

[bhack]

Let me know if I can debug this more both quant and autoquant errors.

[bdhirsh]

I'll let Jerry comment on the autoquant + AOTI support. for that error cannot mutate tensors with frozen storage error, I think the export team still wants to fix this, although if you end up hitting it you can usually work around by sprinkling in a clone() at the right place (usually right before the mutation referenced in the stack trace, e.g. that aten.add_).