Consider rewriting the parsing back-end as a clang plugin instead of using libclang.

[emilio]

This would be sort of a pain, but it's the only way we see us generating more correct code for all the partial template specialization madness we're going through with m-c. I already discussed this in #41 and similar IIRC.

We would need to comunicate with the front-end of course, we could do that via json or similar.

I might introduce an experimental back-end for this if I have the time and we keep breaking stylo.

[emilio]

I've started experimenting with this in https://github.com/emilio/rust-bindgen/tree/clang-plugin.

@fitzgen, would you be interested in reviewing and getting this in-tree once it's minimally working? I don't plan to do this the default (not just yet, at least), but does it seem sensible to you?

The idea is that this might simplify a lot of stuff, including the parent/child relationships, and of course template stuff, which we can recognise in a better way that going through Unexposed stuff in libclang.

[fitzgen]

Our frontend libclang -> ir phase is the part of the code base I am least happy with right now, and I believe you've made comments to a similar effect before. After some reflection about why I am unhappy with it, which was harder to pinpoint than I anticipated, I concluded that it is because we don't really know what we're dealing with, and because of that are forced to resort to various special cases and hacks. We don't know the concrete structure of the AST, and we can only generically traverse it, and attempt to infer the real structure. Yes, a big part of this is also the Unexposed stuff, and guessing what we're dealing with based on the shapes of ASTs we've been given in the past. It feels very shaky, not something we'd like to use as our foundation.

So, with all that said, I am relatively eager to replace our frontend with something better. I've never written a clang plugin either, so my question is: does using a clang plugin actually give us something better? A better API with a more concrete picture of the AST?

Additionally, how would this change the ergonomics for bindgen's users? Will we still be able to leverage libbindgen in build.rs?

[emilio]

So, with all that said, I am relatively eager to replace our frontend with something better. I've never written a clang plugin either, so my question is: does using a clang plugin actually give us something better? A better API with a more concrete picture of the AST?

Yes, it gives us basically the whole thing (access to all the info clang has).

Additionally, how would this change the ergonomics for bindgen's users? Will we still be able to leverage libbindgen in build.rs?

Hopefully! I need to think about this a bit, and I think clang headers are far more common that libclang itself, so I believe I can make it work with the cmake package without too much trouble.

[rpjohnst]

An additional benefit to making this into a clang plugin is that it would work with a wider variety of existing clang installs.

For example, I have a clang toolchain that targets a closed platform. It is built without a clang.dll, so bindgen can't use it. Mainline clang chokes on many of its header files, so I haven't been able to use it as a workaround either.

[emilio]

I think this would still be nice to do (plus will make bindgen saner / faster in general, I'd think). I don't think I'll have the time for it near-term though.

[rinon]

I've run into a few issues with class layouts that would be trivial to solve with deeper clang integration and an absolute pain to solve without it. Because of this, I decided I'd take a crack at replacing the libclang frontend with direct integration with the C++ API. I've got it almost back to feature parity with the libclang version so far, so this is looking promising.

My WIP branch for this is at https://github.com/immunant/rust-bindgen/tree/clang_tooling. Just wanted to ping this issue with an update and check if this might be something we can upstream.

[emilio]

Potentially, yeah! That looks exciting.

Do you know what's the story for using it (clang version requirements, etc.)?

[rinon]

I’m developing it against the latest clang, but this uses basically the same build system as c2rust which is compatible with clang back to at least 6 (and probably further if I fix things). Most distro clang-dev type packages include either static or dynamic libs for all the clang modules, which we link this against. Should be pretty viable for deployment, as we’ve gotten c2rust building against distro packages on all the major Linux distros and homebrew with OS X. Not sure about Windows, but I’m sure we can get that working too.