RFC: Daemon mode

[vhda]

I've been toying around with this idea in my head, where one could run ctags in a daemon mode. This could prove to be useful specially in development environments, where the file currently under update is constantly changing, but we don't really want to run ctags in the complete repository. In a daemon mode, ctags could quickly replace the tag list in memory of that file with a new one.

I see two different complexity levels for this feature, being possible to develop the second on the top of the first:

1. Simple Unix (or similar) socket that receives a file name and ctags will parse that file, update the tag list in memory and write a new tags file.
2. Use dbus instead of the socket and adding the capability to also query the current tag database.

At this point I just to have your opinions about this idea, as I don't even know if I would have the capability of implementing such a feature!

[masatake]

(I don't know detail but I guess geany's tagmanager may do the same thing though it doesn't have IPC interface. Comparing with our ctags, geany's one has well abstracted I/O. Including daemon mode, if you want to do something interesting with ctags, the I/O subssytem is needed.)

Even using simple unix socket, we can implement query interface. So I think we can forget the type of IPC from this discussion. So my understanding 1 and 2 can be rewriten like

1'. the way updating tags file incrementally
2'. query interface

At least following parts are needed:

A. in-memory data base(data structure)
B. routine loading tags file into data structure
C. routine putting tagsEntryInfo to in-memory data base

As I wrote in #382, functions in xcmd.c can be used for parts B.

As I wrote at the top of this comment, tagmanager/ctags of geany has abstracted output interface.
https://github.com/geany/geany/blob/master/tagmanager/ctags/parse.h#L130
This is related to C.

I have never inspected about A. Internally cork has an array for tagsEntryInfo. You can use it as for testing/verifying your ideas. I guess we can learn many things from tagmanager core.

Though I don't want but before entring this exciting area, I have to ask you about releasing. Do we/you want a releasing a version? I don't want because once we released we have to think about (self)compatibility more seriously. When we can ignore the compatibility, we can enjoy random hacking more:-P But I think releasing is a kind of duty.

[vhda]

I think that at least in a first approach it would be acceptable that ctags parses the selected directory recursively before starting the daemon. That way we don't need the capability to load the tags file.

@masatake since you know ctags internals better than I do, how hard would it be to delete all tags corresponding to a specific file using the currently implemented data structure? Would it be too slow (or slower than reparsing a complete workspace)?

[masatake]

@vhda, My understanding that you are talking about in-memory data base(let's call it mdb here).
There is no mdb in current ctags.

It is not possible to implement based on cork.
Please, look at https://github.com/universal-ctags/ctags/blob/master/docs/output-tag-stream.svg
This figure shows output related code of current ctags.
In the output of parser is corked, an entry passed to makeTagEntry goes a cork queue via queueTagEntry function. When uncorked, the entries in the cork queue are flushed to a file.

The cork queue is part of TagFile global variable. If ctags can have a cork queue per input files,
you can get what you want now.

S1. When ctags starting as a daemon, ctags parses files and store the result to queues. Each queues are associated with each input files.
S2. When ctags get a query from a client, ctags scans the queues for getting an answer.
S3. When ctags detects the update of an input file, ctags clears the queue associated with the input file, parse the input file again and store the tags to the queue.

cork API is introduced for making scope information easier. So its public API is limited.
However, you can extend it for your trial.

For the purpose of S2(querying), cork queues may be slow. It takes O(n * m) here n is number of tags in an input file and m is the number of input files.

Completely different approach is using sqlite. Look at the figure again. There 3 backends(writeEtagsEntry, writeXrefEntry, and writeCtagsEntry) for writeTagEntry. You can add one more, writeSqliteEntry.

[cweagans]

+1 for in-memory sqlite database. That's pretty efficient for this kind of thing.

[vhda]

What do you guys think about the feature itself?
Does it make sense? Or would we just be over-complicating ctags?

[b4n]

@vhda I myself don't see the point.

For me, tagging files is not a daemon job [1].

Also, what is the use case? IDE spawning the daemon and asking it to update files? If that's it, you first need non-file-backed parsing, because and IDE will be interested in realtime changes. Also, IMO for that matter better make ctags a library the IDE uses: it's more flexible, and probably simpler for everyone (unless you also make a client library for users, which just makes it more complex).
For an IDE, not only you want tags, but you want to be able to make something out of them, so it's likely you have some kind of indexing phase on your side (if you want more than simple lookup). E.g. in Geany we keep several optimized lists for various needs (one for each file, a combined one and one containing only type-like tags).
As we also build a per-file tree of the generated tags (based on scope), you could also imagine we wanted to cache this. We don't and just have several optimizations to update it, it it could maybe make sense to cache this also.

My point is that IMO for an IDE what you want is in-memory representation of the tags, and probably in a custom format to meet your own needs. I don't think a daemon mode would achieve this.

Sure, it'd be better than spawning ctags on each update or each file, but I don't think it really is the solution. Again, IMO "just" make CTags a library with a pluggable makeTagEntry() or alike (a bit like currently in Geany's tagmanager) so applications can run the parsers as they wish and get the tags, and then do whatever they see fit with it.

Also, before designing this IMO one should look at what the tools using or embedding CTags do/need. I think Geany, Anjuta, possibly Vim and Emacs, etc.

---

[1] at least not in what "daemon" generally means in UNIX world. If really it's wanted, I guess a stdin-based communication would make more sense, as in a tool spawns ctags and communicates with it to control it -- this tool and this tool only, it's not a system service or something.

[masatake]

...@b4n wrote everything. Whether we will implement the daemon mode or not, what we need is the same: abstraction of I/O routines.

[cweagans]

Personally, I'd prefer a ctags daemon over a tags file for vim editing. The projects I work on are quite large, and getting the data I need from a tags file is very slow.

[vhda]

In the case o Vim, which is the editor I use, I don't think it would be that easy because you would need to modify the code base to link it to libuctags, or use some kind of Python bindings. Even for other editors you would still need to write C code to implement such a support. Let me say, I completely support the library idea, but I feel that its adoption may tend to be slow.

On the other hand, the tags file parsing is already widely supported in most editors! But if we need to update the tags relative to a file then all files need to be reprocessed, which might be slow if we're working in a big project. And this was the basis for this idea: provide a faster way of updating the tags file.

Using a stdin-based communication would be enough, but in (standard) vim it's not so easy to start a background process. Neovim is working on that, but it's still not stable enough for a typical user IMO. And I don't know how easy this is in other editors.

This was the basis for the daemon idea, which would run only for the user (not on a system level) and preferably one process only per project area. This would allow plugin developers (in languages like Lua, VimL, Python, etc) to have a simple way of starting the ctags "daemon" and communicate with it via sockets asking to update the tags relative to a single file. This would also have the advantage that multiple editor processes in the same project could communicate with a single daemon.

And I'm really looking for a very simple implementation, with the least amount of changes to ctags code base, because I agree that on the long run having all editors link to libuctags is the best solution. But such a daemon mode with a simple "Update file X" command support would probably fill in for a missing functionality that can be used almost immediately by all editors supporting tags files.

[masatake]

If you have mergetags command, you can implement what you want in a shell script.

1. ctags makes tags files for each source files initially.
2. If one of source files is updated, a client may let ctags know the name of file.
   Then ctags just updates the associated tag file.
3. mergetags gathers tags files into one tags file and send it to the client.

netcat(nc) command and flock command may be needed.

I'm not sure whether your question is about the project policy/roadmap or a implementation approach.

About the project, the question is meaningless in some aspects. Because even if we will not implement the daemon mode, we will write building blocks needed in the daemon mode like libuctags and mergetags. These things are needed anyway in other applications. The differences are when they are implemented.

[b4n]

@vhda OK so I see that what you want to achieve is something that could be a drop-in replacement of the current ctags usage? If that's it, then okay why not if some people would benefit it.

But then I'd like to suggest the same as @masatake: wouldn't a mergetags command be enough? you actually wouldn't even really need a specific tool, you could just do something like this:

#!/bin/sh

export LANG=C

set -e -x

# remove the existing tags for the file
escaped1="$(echo "$1" | sed 's%/%\\/%g')"
sed -i -r "/[^\t]*\t${escaped1}\t/d" tags

# add new at the end
./ctags -o- "$1" >> tags

# and sort the output (optional)
tmp=$(tempfile --prefix="tags.")
sed '/^!_TAG_/!q' tags > "$tmp" # extract the pseudo-tags
sed '/^!_TAG_/d' tags | sort -u >> "$tmp" # and sort the rest
mv -f "$tmp" tags

[vhda]

Just made a small test with the script.
Ignoring the sort, it takes 6.84s to run. Takes 7.22s with sort.
It takes ctags 0.16s to process the file I randomly chose.
The complete parsing of my files takes 10.33s.

Conclusion: there's not much gain between parsing all files and running the script.

[b4n]

@vhda this is totally naive and might be optimized a little by reducing actual writes. At first glance it'd be possible to at least remove one write by combining the ones of removing existing and adding new:

( sed -r "/[^\t]*\t${escaped1}\t/d" < tags && ./ctags -o- "$1") > tags

I'd be interesting to see what actually takes the most time here -- but I guess if it's so slow it probably is sed removing the lines).

BTW, my escaping of the argument is incomplete, should rather be

escaped1="$(echo "$1" | sed 's/[/.*$(){}]/\\\0/g')"

or similar ot also escape regex control characters. another solution that probably is good enough and might be faster would be be something like grep -vF "\t$1\t". that would require testing.

[b4n]

@vhda what about this

#!/bin/sh

export LANG=C

set -e -x

sort=true
tmp=$(tempfile --prefix="tags.")

(
    # remove the existing tags for the file
    grep -vF "  $1  " tags
    # and add new at the end
    ./ctags -o- "$1"
) > "$tmp"

# and sort the output (optional)
if "$sort"; then
    (
        sed -n '/^!_TAG_/!q;p' "$tmp" # extract the pseudo-tags
        sed '/^!_TAG_/d' "$tmp" | sort -u # and sort the rest
    ) > tags
    rm -f "$tmp"
else
    mv -f "$tmp" tags
fi

It also fixes a bug in the sort part where the first tag after the pseudo tags was duplicated (stupid mistake).