Reading compilation database using rapidjson.

[Hylen]

Hi,

This is a draft for the compilation database caching using rapidjson. I know rapidjson wasn't your first choice, but since I didn't understand exactly what requirements you have on the JSON reader and how rapidjson failed them I didn't know what else to look for. It should be simple enough to change it in the code though, if you decide we should go with something else. I'm also open to writing our own methods for reading the compilation database, if you prefer.

Let me know what you think,
Karl

[Hylen]

Also, I should say that this lacks support on Windows for now. The only thing that isn't portable AFAIK is the method for getting the time of modification of a compilation database. That is probably really easy to fix, I don't have a Windows computer at hand though.

[Sarcasm]

Interesting...
I'm in vacation, ping me back in 2 weeks! :)

[Hylen]

Alright, then I will probably get started on the guessing algorithm in the meantime. Have a nice vacation!

[Hylen]

I also have a prototype for the compilation database guessing by now. If you'd like, I could add it to this pull request. Otherwise I'll just open another request once we're done reviewing/reworking this one.

[Sarcasm]

Please leave the guessing out of this PR.

I will give feedback during the week on this PR, thank you.

[Sarcasm]

So I was able to give it a quick try (on Linux).
It doesn't seem to work.
I open main.cpp. It loads the compile commands fine.
Then I open Irony.cpp, I get the message "I: Reloading database.".
Same after I open Command.cpp, I get "I: Reloading database.".

I have the feeling that the caching does not work.

Timestamp based caching is trivial to implement. The most difficult will be to get it work in a cross-platform way.

This leave me wondering about this PR.
What should go in, what should go out?
Do you want me to include this PR in irony-mode or is it just a PoC, a base for smart-guessing?

I have been thinking about server guessing, and I'm wondering if a compilation database for header files should be generated by a python script. The python script would process the compilation database and generate a new one for header files. Python has all the necessary stuff to deal with filesystem, parse JSON, ... This could prove simple and efficient. If we validate that a compilation database for headers make sense, then we could implement what is required in C++/Elisp. Actually the langage doesn't matter much to me, I'm more interested in the idea of a compilation database for headers. If we have such database, the only guessing left will be to find the compile options for files not in any of the compilation database, in this case we could just take the same flags as a sibling file.

[Hylen]

Timestamp based caching is trivial to implement. The most difficult will be to get it work in a cross-platform way.

It should be easy to do. All we need is a call to a Windows routine to get the file modification time.

So I was able to give it a quick try (on Linux).
It doesn't seem to work.
I open main.cpp. It loads the compile commands fine.
Then I open Irony.cpp, I get the message "I: Reloading database.".
Same after I open Command.cpp, I get "I: Reloading database.".

Your right, I see it as well. I will investigate when I have time.

This leave me wondering about this PR.
What should go in, what should go out?
Do you want me to include this PR in irony-mode or is it just a PoC, a base for smart-guessing?

Since it isn't cross platform, I saw this as a proof of concept. Now even more so since the caching doesn't work.

I have been thinking about server guessing, and I'm wondering if a compilation database for header files should be generated by a python script. The python script would process the compilation database and generate a new one for header files. Python has all the necessary stuff to deal with filesystem, parse JSON, ... This could prove simple and efficient. If we validate that a compilation database for headers make sense, then we could implement what is required in C++/Elisp. Actually the langage doesn't matter much to me, I'm more interested in the idea of a compilation database for headers. If we have such database, the only guessing left will be to find the compile options for files not in any of the compilation database, in this case we could just take the same flags as a sibling file.

I agree that a more complex solution would be nice looking forward. However, I think that the guessing can be fast and accurate enough for now and that we should consider something similar to what I have been working on as a first step.

Do you agree? If you do not want this patch, I won't put anymore time into it. I'm already using this with my C++ guessing patch daily, and I'm content with the way it works.

/Karl

[Sarcasm]

It should be easy to do. All we need is a call to a Windows routine to get the file modification time.

This could be a simple function in support/.

Since it isn't cross platform, I saw this as a proof of concept. Now even more so since the caching doesn't work.

ok

I agree that a more complex solution would be nice looking forward. However, I think that the guessing can be fast and accurate enough for now and that we should consider something similar to what I have been working on as a first step.

Do you agree? If you do not want this patch, I won't put anymore time into it. I'm already using this with my C++ guessing patch daily, and I'm content with the way it works.

What I'm okay to integrate from this patch is a cross-platform guessing (I can test/do the Windows one if the code is clean enough that I just have to plug the right function call and test), but to integrate it, it should be done based on the current irony-master, we don't need rapidjson and so on for caching, we can use the old infrastructure. That will be a concise feature by itself.

For the rest I need to have a feel for it, could you do a PR with the guessing?

[Hylen]

Opened a separate PR with the guessing on top of this.

[Sarcasm]

Interestingly, I think the latest version of libclang now provides the filename (http://clang.llvm.org/doxygen/group__COMPILATIONDB.html#gaea64448378c6174e7896a81d26ff454a). This may obsolete the need for a custom JSON parser.

[Hylen]

That's really nice, we should absolutely use that instead. The problem with making the path operations portable (boost) still remains though. Do you think we should use boost and enable guessing if it is detected on the system?