Add test breaking applyMultiQubitOp and then fix it.

[vincentmr]

Before submitting

Please complete the following checklist when submitting a PR:

• All new features must include a unit test.
  If you've fixed a bug or added code that should be tested, add a test to the
  tests directory!

• All new functions and code must be clearly commented and documented.
  If you do make documentation changes, make sure that the docs build and
  render correctly by running make docs.

• Ensure that the test suite passes, by running make test.

• Add a new entry to the .github/CHANGELOG.md file, summarizing the
  change, and including a link back to the PR.

• Ensure that code is properly formatted by running make format.

When all the above are checked, delete everything above the dashed
line and fill in the pull request template.

---

Context:
The multiQubitOpFunctor functor attributes indices and coeffs_in are used as local variables, causing issues in parallel for loops. Managed view (standard) declaration is considered a host function, and hence simply declaring local variables solves the parallel host implementation, but not the parallel device implementation.

Description of the Change:
Add breaking test in Test_StateVectorKokkos_NonParam.cpp. Fix it using a TeamPolicy instead of a RangePolicy as a parallelization policy for multiQubitOpFunctor. Use the TeamPolicy scratch space as a temporary variable. Add a second level of parallelism (indices & matrix product) in passing. Instead of fixing the same bug in getExpectationValueMultiQubitOpFunctor, we simply change the implementation to call multiQubitOpFunctor and then take the inner product.

Benefits:
Bug fix, more parallelism.

Possible Drawbacks:
Possibly slower in certain cases.

Related GitHub Issues:
PR 25

[multiphaseCFD]

Thanks @vincentmr ! It looks like it still needs some updates to let macos py tests pass.

[vincentmr]

Thanks @vincentmr ! It looks like it still needs some updates to let macos py tests pass.

Yes, strange I don't think I did anything to break them ...

[vincentmr]

This PR is breaking the Lightning CI.

[multiphaseCFD]

Thanks @vincentmr ! I'm curious about the performance of the new implementation.

[vincentmr]

Thanks @vincentmr ! I'm curious about the performance of the new implementation.

I quick check with the script below (Kokkos+SERIAL+OPENMP+CUDA_AMPERE80) yields on a A100 card (Perlmutter)

• main Time: 1.248e-01 (s)
• bugfix/multiqubitop Time: 9.571e-02 (s)

# %% Import PennyLane
import pennylane as qml
import pennylane.numpy as np

n_wires = 28
n_uni = 3
n_layers = 10
obs = [qml.PauliZ(i) for i in range(n_wires)]
obs = qml.operation.Tensor(*obs)
dev = qml.device('lightning.kokkos', wires=n_wires)
    
U = np.random.rand(2**n_uni, 2**n_uni) + 1j * np.random.rand(2**n_uni, 2**n_uni)
@qml.qnode(dev)
def example_circuit():
    qml.QubitUnitary(U, wires=range(n_uni))
    return qml.expval(qml.PauliZ(0))

# %%
import time
repeat = 100

t0 = time.time()
for _ in range(repeat):
    example_circuit()
print(f"Time: {(time.time()-t0)/repeat:1.3e} (s)")

[AmintorDusko]

I left some comments to keep track of what we need to change back before merging.
Let me know when a bugfix comes on Pennylane so I will give a final review here.

[AmintorDusko]

Nothing more to add. Good job on this!

[multiphaseCFD]

Thanks @vincentmr for the nice work!

[mlxd]

Thanks @vincentmr
Feel free to merge when ready