ML Program matmul not handling 1D input correctly

[skottmckay]

🐞Describing the bug

• Make sure to only create an issue here for bugs in the coremltools Python package. If this is a bug with the Core ML Framework or Xcode, please submit your bug here: https://developer.apple.com/bug-reporting/
• Provide a clear and consise description of the bug.
  The spec says handling of 1D inputs should have numpy behavior.

The output shape appears to be calculated incorrectly and the predict crashes. Not sure if the crash is an issue as the Core ML Framework level though.

Stack Trace

• If applicable, please paste the complete stack trace.

To Reproduce

• Please add a minimal code example that can reproduce the error when running it.

import numpy as np
import coremltools as ct
from coremltools.converters.mil import Builder as mb

target = ct.target.iOS15
x_shape = (4,)
y_shape = (10, 4, 3)
# expected output shape according to spec
# z_shape = (10, 3)

@mb.program(input_specs=[mb.TensorSpec(shape=x_shape),
                         mb.TensorSpec(shape=y_shape)],
            opset_version=target)
def prog(x, y):
    return mb.matmul(x=x, y=y)

print(prog)

m = ct.convert(prog, minimum_deployment_target=target)

x = np.random.rand(*x_shape)
y = np.random.rand(*y_shape)
print(m.predict({'x': x, 'y': y}))

This is the output from print(prog). The expected output shape is {10,3} according to the spec, but appears to be {1, 3}.

main[CoreML7](%x: (4,fp32)(Tensor),
              %y: (10, 4, 3, fp32)(Tensor)) {
  block0() {
    %matmul_0: (1, 3, fp32)(Tensor) = matmul(x=%x, y=%y, transpose_x=False, transpose_y=False, name="matmul_0")
  } -> (%matmul_0)
}

Output from predict

Running MIL frontend_milinternal pipeline: 0 passes [00:00, ? passes/s]
Running MIL default pipeline: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 71/71 [00:00<00:00, 8710.78 passes/s]Running MIL backend_mlprogram pipeline: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 31655.12 passes/s]Traceback (most recent call last):
  File "matmul_test.py", line 35, in <module>
    print(m.predict({'x': x, 'y': y}))
  File "/Users/me/Library/Python/3.9/lib/python/site-packages/coremltools/models/model.py", line 596, in predict
    return MLModel._get_predictions(self.__proxy__, verify_and_convert_input_dict, data)
  File "/Users/me/Library/Python/3.9/lib/python/site-packages/coremltools/models/model.py", line 648, in _get_predictions
    return proxy.predict(data)
RuntimeError: Caught an unknown exception!

System environment (please complete the following information):

• coremltools version: 7.1
• OS (e.g. MacOS version or Linux type): MacOS 14.5

Additional context

• Add anything else about the problem here that you want to share.

[skottmckay]

There are also a number of places that seem to assume both x and y are 2D or more.

e.g. x_shape[-2] and y_shape[-2]

coremltools/coremltools/converters/mil/mil/ops/defs/iOS15/linear.py

Lines 196 to 208 in 0e292a0

	if self.transpose_x.val:
	x_shape = list(x_shape)
	x_shape[-1], x_shape[-2] = x_shape[-2], x_shape[-1]
	x_shape = tuple(x_shape)
	if self.transpose_y.val:
	y_shape = list(y_shape)
	y_shape[-1], y_shape[-2] = y_shape[-2], y_shape[-1]
	y_shape = tuple(y_shape)
	if not (
	x_shape[-1] == y_shape[-2]
	or is_symbolic(x_shape[-1])
	or is_symbolic(y_shape[-2])
	):

Which results in errors like this for 1D y input

File "/Users/me/Library/Python/3.9/lib/python/site-packages/coremltools/converters/mil/mil/builder.py", line 184, in _add_op
    new_op.type_value_inference()
  File "/Users/me/Library/Python/3.9/lib/python/site-packages/coremltools/converters/mil/mil/operation.py", line 257, in type_value_inference
    output_types = self.type_inference()
  File "/Users/me/Library/Python/3.9/lib/python/site-packages/coremltools/converters/mil/mil/ops/defs/iOS15/linear.py", line 205, in type_inference
    x_shape[-1] == y_shape[-2]
IndexError: list index out of range

[TobyRoseman]

I can reproduce this issue with coremltools version 8.0b1. The output should indeed have shape (10, 3).

This look like a bug in coremltools, not the Framework. Specifically, it looks like an issue in the type_inference method for matmul.