Improve Parser: Use the Clang API

[Arcnor]

I'm in the process of doing this right now. Currently, the following issues exists with the approach I'm taking:

• Preprocessor directives are not supported: This is fixable, because Clang can be made aware of those, but it doesn't by default. I'll leave this for later.
• Normal comments cannot be read: Unless I've missed some Clang flag, non-doc comments are completely ignored. I'm not sure this is a big deal, but nonetheless, it's different from the current behavior.

[saudet]

BTW, we probably want to use the C++ API of Clang for this. It is not currently mapped by the presets, so as initial work, we would either have to:

1. Code the new parser temporarily in C++, or
2. Create the presets for the C++ API of Clang, using the current Parser.

Either way is fine with me. Thanks for your interest in this project and let me know how I can help!

[Arcnor]

Sorry, I haven't had the time to work on this lately.

My final changes allowed me to parse a lot, but some missing things from the C++ API prevented me finishing it IIRC, so I think option 2 should help us get there (or, as you said, code it in C++, but it's non-trivial :D)

[Arcnor]

I've continued a bit on this, and for now, I've decided to write the Clang bindings manually, as I think the surface we need from Clang is not that big (I might be wrong, though).

Once we hace a working parser, we can generate proper bindings for Clang itself and use them in the generator, closing the circle ;)

I have some doubts about how to implement some of the bindings, though, so I'll hit the forums in a few days with my questions :)

[saudet]

@Arcnor Any progress with this?

libclang seems to be getting pretty good for that sort of thing, for example:
https://github.com/rust-lang-nursery/rust-bindgen
https://rust-lang-nursery.github.io/rust-bindgen/
So maybe we won't need to use the C++ API after all...

[Arcnor]

Hi Samuel,

No, unfortunately I haven't had the time to continue, not enough incentive for me to do so right now (the project(s) that were using JavaCPP all stopped for one reason or another, mostly priorities).

So libclang is getting better, eh? That's great news! I've had a quick look at the API again, and it seems to contain some goodies I don't remember from 2 years ago, so yeah, maybe now it's enough for our purposes.

If they kept the same names on the AST I might even be able to reuse some of the code I made years ago that parsed the unstable AST output of CLang (https://github.com/Arcnor/objc2robovm/blob/master/src/main/java/com/arcnor/objcclang/parser/CLangHandler.java for example).

Anyway, unless somebody else is working on this, I'll try to give it another look if it doesn't look too complex to interact with it, as time is limited :).

[saudet]

Looks good! No one else is looking into this AFAIK, so please do continue to check it out! The C API is pretty stable BTW. Thanks

[Arcnor]

I'm going to need some help to generate the bindings it seems. I'll put my question(s) here as they are related, but if you need me to use the forum I'll go there instead:

The bindings have the following code:

typedef struct CXVirtualFileOverlayImpl *CXVirtualFileOverlay;

CXVirtualFileOverlay clang_VirtualFileOverlay_create(unsigned options);
...

How can I rename CXVirtualFileOverlayImpl to CXVirtualFileOverlay? I've tried with javaNames but that's obviously not for that.

[Arcnor]

So besides that small problem, the whole API seems to work (well, at least compile) with very few manual mappings, which is cool.

I'll take more time tomorrow to actually figure out if the stuff I couldn't do ~2 years ago is now possible :).

[saudet]

Sounds good! BTW, the bindings for libclang are already available here:
https://github.com/bytedeco/javacpp-presets/blob/master/llvm/src/main/java/org/bytedeco/javacpp/clang.java
AFAIK, we just need to use those.

If there's anything to fix about those though, please send pull requests against the presets config:
https://github.com/bytedeco/javacpp-presets/blob/master/llvm/src/main/java/org/bytedeco/javacpp/presets/clang.java
Thanks!!

[Arcnor]

Ahh, I didn't realize that. Thanks, I'll start using those. The only change I'd like is generating them against 5.0.0 if they're not already, and making some functions return String instead of BytePointer, I'll open a PR later when I get the time.

…

On Dec 7, 2017 17:24, "Samuel Audet" ***@***.***> wrote: Sounds good! BTW, the bindings for libclang are already available here: https://github.com/bytedeco/javacpp-presets/blob/master/ llvm/src/main/java/org/bytedeco/javacpp/clang.java AFAIK, we just need to use those. If there's anything to fix about those though, please send pull requests against the presets config: https://github.com/bytedeco/javacpp-presets/blob/master/ llvm/src/main/java/org/bytedeco/javacpp/presets/clang.java Thanks!! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABvyIQQvMfdHkhow8JHNefQ7-6VZEZkks5s94SUgaJpZM4GuXQL> .