BUILD: No PyTorch dep

[makslevental]

$ rm -rf venv
$ python -m venv venv
$ source venv/bin/activate
(venv) $ python test/python/smoketest.py
Traceback (most recent call last):
  File "test/python/smoketest.py", line 3, in <module>
    import torch_mlir.ir
ModuleNotFoundError: No module named 'torch_mlir'
$ pip install -r build-requirements.txt
$ TORCH_MLIR_ENABLE_ONLY_MLIR_PYTHON_BINDINGS=1 CMAKE_GENERATOR=Ninja python setup.py bdist_wheel -d wheelhouse
<STUFF HAPPENS>
(venv) $ pip install wheelhouse/torch-mlir.whl
<STUFF HAPPENS>
(venv) $ python test/python/smoketest.py
module {
  %none = torch.constant.none
}
(venv) $ pip freeze | grep torch
torch-mlir==0.0.1

Note the absense of torch==2.0.0.dev20230205.

[powderluv]

We should also try a "Oneshot nightly build"

[silvasean]

Hey folks, we really need to think this through a bit more. The current torch_mlir python module is very fundamentally tied to PyTorch. I think if we want to do this properly, we should basically just expose the vanilla torch_mlir python module which is basically just a wrapper around our MLIR infra. This would mostly be cmake work, such as bypassing things like

torch-mlir/python/CMakeLists.txt

Line 48 in cc819e7

declare_mlir_python_sources(TorchMLIRPythonSources.TopLevel

We might want to go even further -- having two python modules -- one which is just the vanilla MLIR infra, and then a wrapper around those which provides the integration into PyTorch.

For context, the torch_mlir Python module is currently the MLIR wrapper one (things like torch_mlir.ir.Module, torch_mlir.passmanager, etc.) and then we decorate some stuff inside it (all of which depends on PyTorch). We can either use cmake to eliminate all the additional stuff we put inside it, or pivot to having two separate Python modules. I think using cmake here is probably the right choice. The flag would be called TORCH_MLIR_ENABLE_ONLY_MLIR_PYTHON_BINDINGS which implies TORCH_MLIR_ENABLE_JIT_IR_IMPORTER=NO and DTORCH_MLIR_ENABLE_LTC=NO and some other stuff.

[powderluv]

I like having two. python modules / whl packages. This way we can actually pass auditwheel and publish to pypi

[makslevental]

I like having two. python modules / whl packages. This way we can actually pass auditwheel and publish to pypi

I'm not 100% sure what it means to "pass auditwheel" but after building the wheel with TORCH_MLIR_ENABLE_ONLY_MLIR_PYTHON_BINDINGS=1 I can do this:

(venv) $ auditwheel repair --plat manylinux_2_35_x86_64 torch_mlir-0.0.1-cp310-cp310-linux_x86_64.whl 
INFO:auditwheel.main_repair:Repairing torch_mlir-0.0.1-cp310-cp310-linux_x86_64.whl
INFO:auditwheel.wheeltools:Previous filename tags: linux_x86_64
INFO:auditwheel.wheeltools:New filename tags: manylinux_2_35_x86_64
INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp310-cp310-linux_x86_64
INFO:auditwheel.wheeltools:New WHEEL info tags: cp310-cp310-manylinux_2_35_x86_64
INFO:auditwheel.main_repair:
Fixed-up wheel written to /home/mlevental/dev_projects/torch-mlir/wheelhouse/wheelhouse/torch_mlir-0.0.1-cp310-cp310-manylinux_2_35_x86_64.whl

[makslevental]

I believe this looks reasonable (but I've been wrong before 😅). Adjusting the actual build scripts should be a separate PR as they are ...dense...

[silvasean]

Do we want to frontload splitting the modules? Or do that later? This seems reasonable to me if it works. @powderluv ?

[makslevental]

Do we want to frontload splitting the modules? Or do that later? This seems reasonable to me if it works. @powderluv ?

@silvasean

I believe this meets all of our current needs completely so I'm in support of deferring the split until there's a particular need.

[powderluv]

Agreed. This is good to land and iterate.

[powderluv]

Please change commit message to remove WIP

[powderluv]

Also FYI for @ashay since he is cleaning up the whl and pytorch requirements etc.