[spike] prototype blog post examples to identify most impactful graph breaks from there

[Titus-von-Koeller]

most impactful graph breaks

• Params4bit.t used in bnb.matmul_4bit

[Titus-von-Koeller]

graphbreak 1 (43 occurrences) bnb.matmul_4bit(x, self.weight.t(), bias=bias, quant_state=self.weight.quant_state).to(inp_dtype):

Graph break in user code at /home/ubuntu/src/bnb/bitsandbytes/nn/modules.py:483
Reason: Unsupported: call_method UserDefinedObjectVariable(Params4bit) t [] {}
User code traceback:
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 579, in torch_dynamo_resume_in_forward_at_549
    layer_outputs = decoder_layer(
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 260, in forward
    hidden_states, self_attn_weights = self.self_attn(
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 162, in forward
    query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/peft/tuners/lora/bnb.py", line 494, in forward
    result = self.base_layer(x, *args, **kwargs)
  File "/home/ubuntu/src/bnb/bitsandbytes/nn/modules.py", line 483, in forward
    return bnb.matmul_4bit(x, self.weight.t(), bias=bias, quant_state=self.weight.quant_state).to(inp_dtype)

Traceback (most recent call last):
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 659, in wrapper
    return inner_fn(self, inst)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1736, in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 897, in call_function
    self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/nn_module.py", line 914, in call_function
    return variables.UserFunctionVariable(fn, source=source).call_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 317, in call_function
    return super().call_function(tx, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 118, in call_function
    return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 903, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 3072, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 3198, in inline_call_
    tracer.run()
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
    while self.step():
          ^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 659, in wrapper
    return inner_fn(self, inst)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1736, in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 897, in call_function
    self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/lazy.py", line 170, in realize_and_forward
    return getattr(self.realize(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/nn_module.py", line 914, in call_function
    return variables.UserFunctionVariable(fn, source=source).call_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 317, in call_function
    return super().call_function(tx, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 118, in call_function
    return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 903, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 3072, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 3198, in inline_call_
    tracer.run()
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
    while self.step():
          ^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 659, in wrapper
    return inner_fn(self, inst)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2341, in CALL
    self._call(inst)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2335, in _call
    self.call_function(fn, args, kwargs)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 897, in call_function
    self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/lazy.py", line 170, in realize_and_forward
    return getattr(self.realize(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/nn_module.py", line 914, in call_function
    return variables.UserFunctionVariable(fn, source=source).call_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 317, in call_function
    return super().call_function(tx, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 118, in call_function
    return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 903, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 3072, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 3198, in inline_call_
    tracer.run()
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
    while self.step():
          ^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 659, in wrapper
    return inner_fn(self, inst)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1736, in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 897, in call_function
    self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/lazy.py", line 170, in realize_and_forward
    return getattr(self.realize(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/nn_module.py", line 914, in call_function
    return variables.UserFunctionVariable(fn, source=source).call_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 317, in call_function
    return super().call_function(tx, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/functions.py", line 118, in call_function
    return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 903, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 3072, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 3198, in inline_call_
    tracer.run()
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
    while self.step():
          ^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 659, in wrapper
    return inner_fn(self, inst)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2341, in CALL
    self._call(inst)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2335, in _call
    self.call_function(fn, args, kwargs)
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 897, in call_function
    self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/misc.py", line 1022, in call_function
    return self.obj.call_method(tx, self.name, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/user_defined.py", line 823, in call_method
    return super().call_method(tx, name, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/variables/base.py", line 414, in call_method
    unimplemented(f"call_method {self} {name} {args} {kwargs}")
  File "/home/ubuntu/.condax/mamba/envs/bnb/lib/python3.11/site-packages/torch/_dynamo/exc.py", line 317, in unimplemented
    raise Unsupported(msg, case_name=case_name)
torch._dynamo.exc.Unsupported: call_method UserDefinedObjectVariable(Params4bit) t [] {}

[Titus-von-Koeller]

graph break 2 (1 occurrance) Linear4bit.forward(self, x: torch.Tensor):

[Titus-von-Koeller]

torch trace file, can be visualized in browser with tlparse