Add extra conf support to Clangd completer

Author: micbou

README.md

@@ -97,14 +97,13 @@ non-semantic.
 
 There are also several semantic engines in YCM. There's a libclang-based
 completer and [clangd][clangd]-based completer that both provide semantic
-completion for C-family languages. The [clangd][clangd]-based completer doesn't
-support extra conf; you must have a compilation database. [clangd][clangd]
-support is currently **experimental** and changes in the near future might break
-backwards compatibility. There's also a Jedi-based completer for semantic
-completion for Python, an OmniSharp-based completer for C#, a
-[Gocode][gocode]-based completer for Go (using [Godef][godef] for jumping to
-definitions), a TSServer-based completer for JavaScript and TypeScript, and a
-[jdt.ls][jdtls]-based server for Java. More will be added with time.
+completion for C-family languages. [clangd][clangd] support is currently
+**experimental** and changes in the near future might break backwards
+compatibility. There's also a Jedi-based completer for semantic completion for
+Python, an OmniSharp-based completer for C#, a [Gocode][gocode]-based completer
+for Go (using [Godef][godef] for jumping to definitions), a TSServer-based
+completer for JavaScript and TypeScript, and a [jdt.ls][jdtls]-based server for
+Java. More will be added with time.
 
 There are also other completion engines, like the filepath completer (part of
 the identifier completer).
@@ -217,8 +216,8 @@ The `.ycm_extra_conf.py` module may define the following functions:
 #### `Settings( **kwargs )`
 
 This function allows users to configure the language completers on a per project
-basis or globally. Currently, it is required by the C-family completer and
-optional for the Python completer. The following arguments can be retrieved from
+basis or globally. Currently, it is required by the libclang-based completer and
+optional for other completers. The following arguments can be retrieved from
 the `kwargs` dictionary and are common to all completers:
 
 - `language`: an identifier of the completer that called the function. Its value
@@ -231,7 +230,7 @@ the `kwargs` dictionary and are common to all completers:
     language = kwargs[ 'language' ]
     if language == 'cfamily':
       return {
-        # Settings for the C-family completer.
+        # Settings for the libclang and clangd-based completer.
       }
     if language == 'python':
       return {
@@ -240,6 +239,8 @@ the `kwargs` dictionary and are common to all completers:
     return {}
   ```
 
+- `filename`: absolute path of the file currently edited.
+
 - `client_data`: any additional data supplied by the client application.
   See the [YouCompleteMe documentation][extra-conf-vim-data-doc] for an
   example.
@@ -248,32 +249,32 @@ The return value is a dictionary whose content depends on the completer.
 
 ##### C-family settings
 
-The `Settings` function is called by the C-family completer to get the compiler
-flags to use when compiling the current file. The absolute path of this file is
-accessible under the `filename` key of the `kwargs` dictionary.
-[clangd][clangd]-based completer doesn't support extra conf files. If you are
-using [clangd][clangd]-based completer, you must have a compilation database in
-your project's root or in one of the parent directories to provide compiler
-flags.
+The `Settings` function is called by the libclang and clangd-based completers to
+get the compiler flags to use when compiling the current file. The absolute path
+of this file is accessible under the `filename` key of the `kwargs` dictionary.
 
-The return value expected by the completer is a dictionary containing the
+The return value expected by both completers is a dictionary containing the
 following items:
 
-- `flags`: (mandatory) a list of compiler flags.
+- `flags`: (mandatory for libclang, optional for clangd) a list of compiler
+  flags.
 
-- `include_paths_relative_to_dir`: (optional) the directory to which the
-  include paths in the list of flags are relative. Defaults to ycmd working
-  directory.
-
-- `override_filename`: (optional) a string indicating the name of the file to
-  parse as the translation unit for the supplied file name. This fairly
-  advanced feature allows for projects that use a 'unity'-style build, or
-  for header files which depend on other includes in other files.
+- `include_paths_relative_to_dir`: (optional) the directory to which the include
+  paths in the list of flags are relative. Defaults to ycmd working directory
+  for the libclang completer and `.ycm_extra_conf.py`'s directory for the
+  clangd completer.
 
 - `do_cache`: (optional) a boolean indicating whether or not the result of
   this call (i.e. the list of flags) should be cached for this file name.
   Defaults to `True`. If unsure, the default is almost always correct.
 
+The libclang-based completer also supports the following items:
+
+- `override_filename`: (optional) a string indicating the name of the file to
+  parse as the translation unit for the supplied file name. This fairly advanced
+  feature allows for projects that use a 'unity'-style build, or for header
+  files which depend on other includes in other files.
+
 - `flags_ready`: (optional) a boolean indicating that the flags should be
   used. Defaults to `True`. If unsure, the default is almost always correct.
 
@@ -393,7 +394,7 @@ License
 -------
 
 This software is licensed under the [GPL v3 license][gpl].
-© 2015-2018 ycmd contributors
+© 2015-2019 ycmd contributors
 
 [ycmd-users]: https://groups.google.com/forum/?hl=en#!forum/ycmd-users
 [ycm]: http://valloric.github.io/YouCompleteMe/

ycmd/completers/cpp/clangd_completer.py

@@ -26,8 +26,10 @@
 import os
 import subprocess
 
-from ycmd import responses
+from ycmd import extra_conf_store, responses
 from ycmd.completers.completer_utils import GetFileLines
+from ycmd.completers.cpp.flags import ( RemoveUnusedFlags,
+                                        ShouldAllowWinStyleFlags )
 from ycmd.completers.language_server import simple_language_server_completer
 from ycmd.completers.language_server import language_server_completer
 from ycmd.completers.language_server import language_server_protocol as lsp
@@ -200,6 +202,27 @@ def ShouldEnableClangdCompleter( user_options ):
   return True
 
 
+def PrependCompilerToFlags( flags, enable_windows_style_flags ):
+  """Removes everything before the first flag and returns the remaining flags
+  prepended with clangd."""
+  for index, flag in enumerate( flags ):
+    if ( flag.startswith( '-' ) or
+         ( enable_windows_style_flags and
+           flag.startswith( '/' ) and
+           not os.path.exists( flag ) ) ):
+      flags = flags[ index: ]
+      break
+  return [ 'clang-tool' ] + flags
+
+
+def BuildCompilationCommand( flags, filepath ):
+  """Returns a compilation command from a list of flags and a file."""
+  enable_windows_style_flags = ShouldAllowWinStyleFlags( flags )
+  flags = PrependCompilerToFlags( flags, enable_windows_style_flags )
+  flags = RemoveUnusedFlags( flags, filepath, enable_windows_style_flags )
+  return flags + [ filepath ]
+
+
 class ClangdCompleter( simple_language_server_completer.SimpleLSPCompleter ):
   """A LSP-based completer for C-family languages, powered by Clangd.
 
@@ -214,6 +237,19 @@ def __init__( self, user_options ):
 
     self._clangd_command = GetClangdCommand( user_options )
     self._use_ycmd_caching = user_options[ 'clangd_uses_ycmd_caching' ]
+    self._flags_for_file = {}
+
+    self.RegisterOnFileReadyToParse( self._SendFlagsFromExtraConf )
+
+
+  def _Reset( self ):
+    with self._server_state_mutex:
+      super( ClangdCompleter, self )._Reset()
+      self._flags_for_file = {}
+
+
+  def GetCompleterName( self ):
+    return 'C-family'
 
 
   def GetServerName( self ):
@@ -224,6 +260,10 @@ def GetCommandLine( self ):
     return self._clangd_command
 
 
+  def Language( self ):
+    return 'cfamily'
+
+
   def SupportedFiletypes( self ):
     return ( 'c', 'cpp', 'objc', 'objcpp', 'cuda' )
 
@@ -361,3 +401,45 @@ def GetDetailedDiagnostic( self, request_data ):
         minimum_distance = distance
 
     return responses.BuildDisplayMessageResponse( message )
+
+
+  def _SendFlagsFromExtraConf( self, request_data ):
+    """Reads the flags from the extra conf of the given request and sends them
+    to Clangd as an entry of a compilation database using the
+    'compilationDatabaseChanges' configuration."""
+    filepath = request_data[ 'filepath' ]
+
+    with self._server_info_mutex:
+      module = extra_conf_store.ModuleForSourceFile( filepath )
+      if not module:
+        return
+
+      settings = self.GetSettings( module, request_data )
+
+      if 'flags' not in settings:
+        # No flags returned. Let Clangd find the flags.
+        return
+
+      if settings.get( 'do_cache', True ) and filepath in self._flags_for_file:
+        # Flags for this file have already been sent to Clangd.
+        return
+
+      flags = settings[ 'flags' ]
+
+      self.GetConnection().SendNotification( lsp.DidChangeConfiguration( {
+        'compilationDatabaseChanges': {
+          filepath: {
+            'compilationCommand': BuildCompilationCommand( flags, filepath ),
+            'workingDirectory': settings.get( 'include_paths_relative_to_dir',
+                                              self._project_directory )
+          }
+        }
+      } ) )
+
+      self._flags_for_file[ filepath ] = flags
+
+
+  def ExtraDebugItems( self, request_data ):
+    return [ responses.DebugInfoItem(
+      'Extra Configuration Flags',
+      self._flags_for_file.get( request_data[ 'filepath' ], False ) ) ]

ycmd/completers/cpp/flags.py

@@ -161,7 +161,7 @@ def _ParseFlagsFromExtraConfOrDatabase( self,
     sanitized_flags = PrepareFlagsForClang( flags,
                                             filename,
                                             add_extra_clang_flags,
-                                            _ShouldAllowWinStyleFlags( flags ) )
+                                            ShouldAllowWinStyleFlags( flags ) )
 
     if results.get( 'do_cache', True ):
       self.flags_for_file[ filename, client_data ] = sanitized_flags, filename
@@ -248,7 +248,7 @@ def _ExtractFlagsList( flags_for_file_output ):
   return [ ToUnicode( x ) for x in flags_for_file_output[ 'flags' ] ]
 
 
-def _ShouldAllowWinStyleFlags( flags ):
+def ShouldAllowWinStyleFlags( flags ):
   if OnWindows():
     # Iterate in reverse because we only care
     # about the last occurrence of --driver-mode flag.
@@ -302,7 +302,7 @@ def PrepareFlagsForClang( flags,
                           enable_windows_style_flags = False ):
   flags = _AddLanguageFlagWhenAppropriate( flags, enable_windows_style_flags )
   flags = _RemoveXclangFlags( flags )
-  flags = _RemoveUnusedFlags( flags, filename, enable_windows_style_flags )
+  flags = RemoveUnusedFlags( flags, filename, enable_windows_style_flags )
   if add_extra_clang_flags:
     # This flag tells libclang where to find the builtin includes.
     flags.append( '-resource-dir=' + CLANG_RESOURCE_DIR )
@@ -407,7 +407,7 @@ def _AddLanguageFlagWhenAppropriate( flags, enable_windows_style_flags ):
   return flags
 
 
-def _RemoveUnusedFlags( flags, filename, enable_windows_style_flags ):
+def RemoveUnusedFlags( flags, filename, enable_windows_style_flags ):
   """Given an iterable object that produces strings (flags for Clang), removes
   the '-c' and '-o' options that Clang does not like to see when it's producing
   completions for a file. Same for '-MD' etc.
@@ -622,7 +622,7 @@ def _MakeRelativePathsInFlagsAbsolute( flags, working_directory ):
   new_flags = []
   make_next_absolute = False
   path_flags = ( PATH_FLAGS + INCLUDE_FLAGS_WIN_STYLE
-                 if _ShouldAllowWinStyleFlags( flags )
+                 if ShouldAllowWinStyleFlags( flags )
                  else PATH_FLAGS )
   for flag in flags:
     new_flag = flag
@@ -685,7 +685,7 @@ def UserIncludePaths( user_flags, filename ):
                       '-isystem':    include_paths,
                       '-F':          framework_paths,
                       '-iframework': framework_paths }
-    if _ShouldAllowWinStyleFlags( user_flags ):
+    if ShouldAllowWinStyleFlags( user_flags ):
       include_flags[ '/I' ] = include_paths
 
     try:

ycmd/completers/language_server/language_server_completer.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2017-2018 ycmd contributors
+# Copyright (C) 2017-2019 ycmd contributors
 #
 # This file is part of ycmd.
 #
@@ -22,6 +22,7 @@
 # Not installing aliases from python-future; it's unreliable and slow.
 from builtins import *  # noqa
 
+from functools import partial
 from future.utils import iteritems, iterkeys
 import abc
 import collections
@@ -741,6 +742,9 @@ def __init__( self, user_options ):
     #    cached query is a prefix of the subsequent queries.
     self._completions_cache = LanguageServerCompletionsCache()
 
+    self._on_file_ready_to_parse_handlers = []
+    self.RegisterOnFileReadyToParse( self._UpdateServerWithFileContents )
+
 
   def ServerReset( self ):
     """Clean up internal state related to the running server instance.
@@ -1087,10 +1091,12 @@ def _DiscoverSubcommandSupport( self, commands ):
     return subcommands_map
 
 
-  def _GetSettings( self, module, client_data ):
+  def GetSettings( self, module, request_data ):
     if hasattr( module, 'Settings' ):
-      settings = module.Settings( language = self.Language(),
-                                  client_data = client_data )
+      settings = module.Settings(
+        language = self.Language(),
+        filename = request_data[ 'filepath' ],
+        client_data = request_data[ 'extra_conf_data' ] )
       if settings is not None:
         return settings
 
@@ -1102,7 +1108,7 @@ def _GetSettings( self, module, client_data ):
   def _GetSettingsFromExtraConf( self, request_data ):
     module = extra_conf_store.ModuleForSourceFile( request_data[ 'filepath' ] )
     if module:
-      settings = self._GetSettings( module, request_data[ 'extra_conf_data' ] )
+      settings = self.GetSettings( module, request_data )
       self._settings = settings.get( 'ls' ) or {}
       # Only return the dir if it was found in the paths; we don't want to use
       # the path of the global extra conf as a project root dir.
@@ -1139,17 +1145,19 @@ def OnFileReadyToParse( self, request_data ):
     if not self.ServerIsHealthy():
       return
 
-    # If we haven't finished initializing yet, we need to queue up a call to
-    # _UpdateServerWithFileContents. This ensures that the server is up to date
-    # as soon as we are able to send more messages. This is important because
-    # server start up can be quite slow and we must not block the user, while we
-    # must keep the server synchronized.
+    # If we haven't finished initializing yet, we need to queue up all functions
+    # registered on the FileReadyToParse event and in particular
+    # _UpdateServerWithFileContents in reverse order of registration. This
+    # ensures that the server is up to date as soon as we are able to send more
+    # messages. This is important because server start up can be quite slow and
+    # we must not block the user, while we must keep the server synchronized.
     if not self._initialize_event.is_set():
-      self._OnInitializeComplete(
-        lambda self: self._UpdateServerWithFileContents( request_data ) )
+      for handler in reversed( self._on_file_ready_to_parse_handlers ):
+        self._OnInitializeComplete( partial( handler, request_data ) )
       return
 
-    self._UpdateServerWithFileContents( request_data )
+    for handler in reversed( self._on_file_ready_to_parse_handlers ):
+      handler( request_data )
 
     # Return the latest diagnostics that we have received.
     #
@@ -1437,8 +1445,8 @@ def OnBufferUnload( self, request_data ):
     # server start up can be quite slow and we must not block the user, while we
     # must keep the server synchronized.
     if not self._initialize_event.is_set():
-      self._OnInitializeComplete(
-        lambda self: self._PurgeFileFromServer( request_data[ 'filepath' ] ) )
+      self._OnInitializeComplete( partial( self._PurgeFileFromServer,
+                                           request_data[ 'filepath' ] ) )
       return
 
     self._PurgeFileFromServer( request_data[ 'filepath' ] )
@@ -1595,7 +1603,7 @@ def _HandleInitializeInPollThread( self, response ):
     # exchange. Typically, this will be calls to _UpdateServerWithFileContents
     # or something that occurred while we were waiting.
     for handler in self._on_initialize_complete_handlers:
-      handler( self )
+      handler()
 
     self._on_initialize_complete_handlers = []
 
@@ -1609,6 +1617,10 @@ def _OnInitializeComplete( self, handler ):
     self._on_initialize_complete_handlers.append( handler )
 
 
+  def RegisterOnFileReadyToParse( self, handler ):
+    self._on_file_ready_to_parse_handlers.append( handler )
+
+
   def GetHoverResponse( self, request_data ):
     """Return the raw LSP response to the hover request for the supplied
     context. Implementations can use this for e.g. GetDoc and GetType requests,