Use symlinks for external plugins to fix TRITON_PLUGIN_DIRS

[mgorny]

Disable the new backend installing logic for external plugins, since package_dir does not accept absolute paths. Instead, use a hybrid approach where in-tree backends are installed using the new logic and external backends are symlinked. This implies that source distributions cannot be created when using external plugins.

Fixes #6612

---

New contributor declaration

• I am not making a trivial change, such as fixing a typo in a comment.

• I have written a PR description following these
  rules.

• I have run pre-commit run --from-ref origin/main --to-ref HEAD.

• Select one of the following.

  • I have added tests.
    • /test for lit tests
    • /unittest for C++ tests
    • /python/test for end-to-end tests
  • This PR does not need a test because it touches build system only.

• Select one of the following.

  • I have not added any lit tests.
  • The lit tests I have added follow these best practices,
    including the "tests should be minimal" section. (Usually running Python code
    and using the instructions it generates is not minimal.)

[mgorny]

One note: I've noticed a potential bug in the logic (that was there before my first change):

creating symlink: /home/mgorny/git/triton/python/triton/backends/triton_shared -> /tmp/triton_shared/backend
creating symlink: /home/mgorny/git/triton/python/triton/tools/extra/triton-shared-opt -> /tmp/triton_shared/tools/triton-shared-opt
creating symlink: /home/mgorny/git/triton/python/triton/tools/extra/RegisterTritonSharedDialects.h -> /tmp/triton_shared/tools/RegisterTritonSharedDialects.h
creating symlink: /home/mgorny/git/triton/python/triton/tools/extra/CMakeLists.txt -> /tmp/triton_shared/tools/CMakeLists.txt

Note that files like RegisterTritonSharedDialects.h and CMakeLists.txt ended up being symlinked into tools/extra/. Perhaps we should consider only directories there?

[BabakkGraphcore]

Seeing the same error as before with this change. I think because even though the symlinks are setup for the external plugins their package_dir path is still TRITON_PLUGIN_DIRS and so absolute.

[mgorny]

They shouldn't, unless I really messed something up:

triton/setup.py

Lines 599 to 601 in f5c252c

	# we use symlinks for external plugins
	if backend.is_external:
	continue

Did you do a full clean of the build tree? Ideally git clean -dfx, as setuptools metadata tends to persist a lot.

[BabakkGraphcore]

Ah, sorry - you're right. Successful build after that 🎉 However non .py files from the backend didn't get installed. Does the manifest need to be tweaked ?

[mgorny]

Sorry — just made a dummy backend and reproduced the problem. I think the problem is that find_packages() is run prior to creating the symlinks. I'll investigate — perhaps the simplest solution will be to create external plugin symlinks in global scope.

[mgorny]

@BabakkGraphcore, could you try now?

[BabakkGraphcore]

@BabakkGraphcore, could you try now?

No change. Tested with...

 git clean -dfx
 pip -v install  --force-reinstall  . 

The specific file that isn't being copied is a .cpp if that helps with your testing setup at all.

[mgorny]

In backend directory? Let me sigh a bit, then try to fix it. Data files with setuptools are a nightmare.

[mgorny]

Ah, sorry, now I understand what you meant by the manifest. I forgot that it affects installing data files.

[BabakkGraphcore]

thanks @mgorny that looks to be working now, build + tests passing 🎉

going to try one more test and get it a passing CI workflow with it

[BabakkGraphcore]

going to try one more test and get it a passing CI workflow with it

Took a while 😅 but all good, thanks for the fix