Directly use tree sitter C API

[pascalkuthe]

I am not convinced the tree sitter rust crate is worth using . The api is really subotimal. The C API is stable and the rust api is fairly easy to replace since its just bindings.

Things to improve (add more as we find more)

• Use regex-cursor for queries instead of buffering (we allocate for every query, I found out)

  • using the cursor trait instead of their text provider api (which is only used for regex captures) it would be possible to fix crashed like Helix crashes when editing files with Russian text #7273 which are caused by captures within unicode chars. Regex-cursor fully supports invalid utf8 and won't call rope.slice but manage the iteration itself. I could remiplement this osmewhat with the current api but it would be reimplementing some nont quite tirival logic. I don't want to do that for code I don't expect to keep

• fix undefined behaviour in the query api

• remvoe code we don't neeed to speedup builds

we can bundle this crate together with our highlighter so other people don't need to vendor our code. I want to seperate our TS code from config and other code into a seperate crate anyway. They really shouldn't be coupled

[pppKin]

I did some quick search and it seems that zed and lapce still use the tree-sitter crate. Of course that doesn't mean we should not explore other options. Maybe we can make something that'll benefit everyone?

[pascalkuthe]

That what I meant with

I want to seperate our TS code from config and other code into a seperate crate anyway.

We want to publish the highlighter on crates.io which would cover that. I don't think zed would want to use that (the guy that maintains tree sitter works on zed) but Laplace already uses a vendored copy of our highlighter (which is pretty old and is missing lots of bug fixes and performance improvements I amde since)