Skip to content

[ClangRepl] Reland Semanic Code Completion #75556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 19, 2023
Merged

Conversation

capfredf
Copy link
Contributor

@capfredf capfredf commented Dec 15, 2023

This patch contains changes from 002d471, in
addition to a bug fix that added a virtual destructor to
CompletionContextHandler

The original changes in the orginal commit piggybacks on clang's
semantic modules to enable semantic completion. In particular, we use
CodeCompletionContext to differentiate two types of code completion. We also
extract the relevant type information from it.

@capfredf
Copy link
Contributor Author

@vgvassilev

@capfredf capfredf marked this pull request as ready for review December 15, 2023 03:20
@llvmbot llvmbot added clang Clang issues not falling into any other category clang:frontend Language frontend issues, e.g. anything involving "Sema" labels Dec 15, 2023
@capfredf capfredf changed the title [ClangRepl] Semanic Code Completion [ClangRepl] Reland Semanic Code Completion Dec 15, 2023
@llvmbot
Copy link
Member

llvmbot commented Dec 15, 2023

@llvm/pr-subscribers-clang

Author: Fred Fu (capfredf)

Changes

Patch is 24.20 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/75556.diff

6 Files Affected:

  • (modified) clang/include/clang/Interpreter/CodeCompletion.h (+22-3)
  • (modified) clang/include/clang/Interpreter/Interpreter.h (+1)
  • (modified) clang/lib/Interpreter/CodeCompletion.cpp (+200-23)
  • (modified) clang/lib/Interpreter/Interpreter.cpp (+4)
  • (modified) clang/tools/clang-repl/ClangRepl.cpp (+9-15)
  • (modified) clang/unittests/Interpreter/CodeCompletionTest.cpp (+209-10)
diff --git a/clang/include/clang/Interpreter/CodeCompletion.h b/clang/include/clang/Interpreter/CodeCompletion.h
index 9adcdf0dc3afac..c64aa899759fd8 100644
--- a/clang/include/clang/Interpreter/CodeCompletion.h
+++ b/clang/include/clang/Interpreter/CodeCompletion.h
@@ -23,8 +23,27 @@ namespace clang {
 class CodeCompletionResult;
 class CompilerInstance;
 
-void codeComplete(CompilerInstance *InterpCI, llvm::StringRef Content,
-                  unsigned Line, unsigned Col, const CompilerInstance *ParentCI,
-                  std::vector<std::string> &CCResults);
+struct ReplCodeCompleter {
+  ReplCodeCompleter() = default;
+  std::string Prefix;
+
+  /// \param InterpCI [in] The compiler instance that is used to trigger code
+  /// completion
+
+  /// \param Content [in] The string where code completion is triggered.
+
+  /// \param Line [in] The line number of the code completion point.
+
+  /// \param Col [in] The column number of the code completion point.
+
+  /// \param ParentCI [in] The running interpreter compiler instance that
+  /// provides ASTContexts.
+
+  /// \param CCResults [out] The completion results.
+  void codeComplete(CompilerInstance *InterpCI, llvm::StringRef Content,
+                    unsigned Line, unsigned Col,
+                    const CompilerInstance *ParentCI,
+                    std::vector<std::string> &CCResults);
+};
 } // namespace clang
 #endif
diff --git a/clang/include/clang/Interpreter/Interpreter.h b/clang/include/clang/Interpreter/Interpreter.h
index 43573fb1a4b891..01858dfcc90ac5 100644
--- a/clang/include/clang/Interpreter/Interpreter.h
+++ b/clang/include/clang/Interpreter/Interpreter.h
@@ -101,6 +101,7 @@ class Interpreter {
   const ASTContext &getASTContext() const;
   ASTContext &getASTContext();
   const CompilerInstance *getCompilerInstance() const;
+  CompilerInstance *getCompilerInstance();
   llvm::Expected<llvm::orc::LLJIT &> getExecutionEngine();
 
   llvm::Expected<PartialTranslationUnit &> Parse(llvm::StringRef Code);
diff --git a/clang/lib/Interpreter/CodeCompletion.cpp b/clang/lib/Interpreter/CodeCompletion.cpp
index c40e11b9d1ece0..a9789355b2c5f3 100644
--- a/clang/lib/Interpreter/CodeCompletion.cpp
+++ b/clang/lib/Interpreter/CodeCompletion.cpp
@@ -12,6 +12,7 @@
 
 #include "clang/Interpreter/CodeCompletion.h"
 #include "clang/AST/ASTImporter.h"
+#include "clang/AST/DeclLookups.h"
 #include "clang/AST/DeclarationName.h"
 #include "clang/AST/ExternalASTSource.h"
 #include "clang/Basic/IdentifierTable.h"
@@ -23,6 +24,8 @@
 #include "clang/Sema/CodeCompleteConsumer.h"
 #include "clang/Sema/CodeCompleteOptions.h"
 #include "clang/Sema/Sema.h"
+#include "llvm/Support/Debug.h"
+#define DEBUG_TYPE "REPLCC"
 
 namespace clang {
 
@@ -39,11 +42,15 @@ clang::CodeCompleteOptions getClangCompleteOpts() {
 
 class ReplCompletionConsumer : public CodeCompleteConsumer {
 public:
-  ReplCompletionConsumer(std::vector<std::string> &Results)
+  ReplCompletionConsumer(std::vector<std::string> &Results,
+                         ReplCodeCompleter &CC)
       : CodeCompleteConsumer(getClangCompleteOpts()),
         CCAllocator(std::make_shared<GlobalCodeCompletionAllocator>()),
-        CCTUInfo(CCAllocator), Results(Results){};
+        CCTUInfo(CCAllocator), Results(Results), CC(CC) {}
 
+  // The entry of handling code completion. When the function is called, we
+  // create a `Context`-based handler (see classes defined below) to handle each
+  // completion result.
   void ProcessCodeCompleteResults(class Sema &S, CodeCompletionContext Context,
                                   CodeCompletionResult *InResults,
                                   unsigned NumResults) final;
@@ -56,26 +63,147 @@ class ReplCompletionConsumer : public CodeCompleteConsumer {
   std::shared_ptr<GlobalCodeCompletionAllocator> CCAllocator;
   CodeCompletionTUInfo CCTUInfo;
   std::vector<std::string> &Results;
+  ReplCodeCompleter &CC;
+};
+
+/// The class CompletionContextHandler contains four interfaces, each of
+/// which handles one type of completion result.
+/// Its derived classes are used to create concrete handlers based on
+/// \c CodeCompletionContext.
+class CompletionContextHandler {
+protected:
+  CodeCompletionContext CCC;
+  std::vector<std::string> &Results;
+
+private:
+  Sema &S;
+
+public:
+  CompletionContextHandler(Sema &S, CodeCompletionContext CCC,
+                           std::vector<std::string> &Results)
+      : CCC(CCC), Results(Results), S(S) {}
+
+  virtual ~CompletionContextHandler() = default;
+  /// Converts a Declaration completion result to a completion string, and then
+  /// stores it in Results.
+  virtual void handleDeclaration(const CodeCompletionResult &Result) {
+    auto PreferredType = CCC.getPreferredType();
+    if (PreferredType.isNull()) {
+      Results.push_back(Result.Declaration->getName().str());
+      return;
+    }
+
+    if (auto *VD = dyn_cast<VarDecl>(Result.Declaration)) {
+      auto ArgumentType = VD->getType();
+      if (PreferredType->isReferenceType()) {
+        QualType RT = PreferredType->castAs<ReferenceType>()->getPointeeType();
+        Sema::ReferenceConversions RefConv;
+        Sema::ReferenceCompareResult RefRelationship =
+            S.CompareReferenceRelationship(SourceLocation(), RT, ArgumentType,
+                                           &RefConv);
+        switch (RefRelationship) {
+        case Sema::Ref_Compatible:
+        case Sema::Ref_Related:
+          Results.push_back(VD->getName().str());
+          break;
+        case Sema::Ref_Incompatible:
+          break;
+        }
+      } else if (S.Context.hasSameType(ArgumentType, PreferredType)) {
+        Results.push_back(VD->getName().str());
+      }
+    }
+  }
+
+  /// Converts a Keyword completion result to a completion string, and then
+  /// stores it in Results.
+  virtual void handleKeyword(const CodeCompletionResult &Result) {
+    auto Prefix = S.getPreprocessor().getCodeCompletionFilter();
+    // Add keyword to the completion results only if we are in a type-aware
+    // situation.
+    if (!CCC.getBaseType().isNull() || !CCC.getPreferredType().isNull())
+      return;
+    if (StringRef(Result.Keyword).startswith(Prefix))
+      Results.push_back(Result.Keyword);
+  }
+
+  /// Converts a Pattern completion result to a completion string, and then
+  /// stores it in Results.
+  virtual void handlePattern(const CodeCompletionResult &Result) {}
+
+  /// Converts a Macro completion result to a completion string, and then stores
+  /// it in Results.
+  virtual void handleMacro(const CodeCompletionResult &Result) {}
+};
+
+class DotMemberAccessHandler : public CompletionContextHandler {
+public:
+  DotMemberAccessHandler(Sema &S, CodeCompletionContext CCC,
+                         std::vector<std::string> &Results)
+      : CompletionContextHandler(S, CCC, Results) {}
+  void handleDeclaration(const CodeCompletionResult &Result) override {
+    auto *ID = Result.Declaration->getIdentifier();
+    if (!ID)
+      return;
+    if (!isa<CXXMethodDecl>(Result.Declaration))
+      return;
+    const auto *Fun = cast<CXXMethodDecl>(Result.Declaration);
+    if (Fun->getParent()->getCanonicalDecl() ==
+        CCC.getBaseType()->getAsCXXRecordDecl()->getCanonicalDecl()) {
+      LLVM_DEBUG(llvm::dbgs() << "[In HandleCodeCompleteDOT] Name : "
+                              << ID->getName() << "\n");
+      Results.push_back(ID->getName().str());
+    }
+  }
+
+  void handleKeyword(const CodeCompletionResult &Result) override {}
 };
 
 void ReplCompletionConsumer::ProcessCodeCompleteResults(
     class Sema &S, CodeCompletionContext Context,
     CodeCompletionResult *InResults, unsigned NumResults) {
-  for (unsigned I = 0; I < NumResults; ++I) {
+
+  auto Prefix = S.getPreprocessor().getCodeCompletionFilter();
+  CC.Prefix = Prefix;
+
+  std::unique_ptr<CompletionContextHandler> CCH;
+
+  // initialize fine-grained code completion handler based on the code
+  // completion context.
+  switch (Context.getKind()) {
+  case CodeCompletionContext::CCC_DotMemberAccess:
+    CCH.reset(new DotMemberAccessHandler(S, Context, this->Results));
+    break;
+  default:
+    CCH.reset(new CompletionContextHandler(S, Context, this->Results));
+  };
+
+  for (unsigned I = 0; I < NumResults; I++) {
     auto &Result = InResults[I];
     switch (Result.Kind) {
     case CodeCompletionResult::RK_Declaration:
-      if (auto *ID = Result.Declaration->getIdentifier()) {
-        Results.push_back(ID->getName().str());
+      if (Result.Hidden) {
+        break;
+      }
+      if (!Result.Declaration->getDeclName().isIdentifier() ||
+          !Result.Declaration->getName().startswith(Prefix)) {
+        break;
       }
+      CCH->handleDeclaration(Result);
       break;
     case CodeCompletionResult::RK_Keyword:
-      Results.push_back(Result.Keyword);
+      CCH->handleKeyword(Result);
       break;
-    default:
+    case CodeCompletionResult::RK_Macro:
+      CCH->handleMacro(Result);
+      break;
+    case CodeCompletionResult::RK_Pattern:
+      CCH->handlePattern(Result);
       break;
     }
   }
+
+  std::sort(Results.begin(), Results.end());
 }
 
 class IncrementalSyntaxOnlyAction : public SyntaxOnlyAction {
@@ -118,6 +246,16 @@ void IncrementalSyntaxOnlyAction::ExecuteAction() {
   CI.getASTContext().getTranslationUnitDecl()->setHasExternalVisibleStorage(
       true);
 
+  // Load all external decls into current context. Under the hood, it calls
+  // ExternalSource::completeVisibleDeclsMap, which make all decls on the redecl
+  // chain visible.
+  //
+  // This is crucial to code completion on dot members, since a bound variable
+  // before "." would be otherwise treated out-of-scope.
+  //
+  // clang-repl> Foo f1;
+  // clang-repl> f1.<tab>
+  CI.getASTContext().getTranslationUnitDecl()->lookups();
   SyntaxOnlyAction::ExecuteAction();
 }
 
@@ -134,6 +272,7 @@ ExternalSource::ExternalSource(ASTContext &ChildASTCtxt, FileManager &ChildFM,
 
 bool ExternalSource::FindExternalVisibleDeclsByName(const DeclContext *DC,
                                                     DeclarationName Name) {
+
   IdentifierTable &ParentIdTable = ParentASTCtxt.Idents;
 
   auto ParentDeclName =
@@ -159,29 +298,67 @@ void ExternalSource::completeVisibleDeclsMap(
   for (auto *DeclCtxt = ParentTUDeclCtxt; DeclCtxt != nullptr;
        DeclCtxt = DeclCtxt->getPreviousDecl()) {
     for (auto &IDeclContext : DeclCtxt->decls()) {
-      if (NamedDecl *Decl = llvm::dyn_cast<NamedDecl>(IDeclContext)) {
-        if (auto DeclOrErr = Importer->Import(Decl)) {
-          if (NamedDecl *importedNamedDecl =
-                  llvm::dyn_cast<NamedDecl>(*DeclOrErr)) {
-            SetExternalVisibleDeclsForName(ChildDeclContext,
-                                           importedNamedDecl->getDeclName(),
-                                           importedNamedDecl);
-          }
-
-        } else {
-          llvm::consumeError(DeclOrErr.takeError());
-        }
+      if (!llvm::isa<NamedDecl>(IDeclContext))
+        continue;
+
+      NamedDecl *Decl = llvm::cast<NamedDecl>(IDeclContext);
+
+      auto DeclOrErr = Importer->Import(Decl);
+      if (!DeclOrErr) {
+        // if an error happens, it usually means the decl has already been
+        // imported or the decl is a result of a failed import.  But in our
+        // case, every import is fresh each time code completion is
+        // triggered. So Import usually doesn't fail. If it does, it just means
+        // the related decl can't be used in code completion and we can safely
+        // drop it.
+        llvm::consumeError(DeclOrErr.takeError());
+        continue;
       }
+
+      if (!llvm::isa<NamedDecl>(*DeclOrErr))
+        continue;
+
+      NamedDecl *importedNamedDecl = llvm::cast<NamedDecl>(*DeclOrErr);
+
+      SetExternalVisibleDeclsForName(ChildDeclContext,
+                                     importedNamedDecl->getDeclName(),
+                                     importedNamedDecl);
+
+      if (!llvm::isa<CXXRecordDecl>(importedNamedDecl))
+        continue;
+
+      auto *Record = llvm::cast<CXXRecordDecl>(importedNamedDecl);
+
+      if (auto Err = Importer->ImportDefinition(Decl)) {
+        // the same as above
+        consumeError(std::move(Err));
+        continue;
+      }
+
+      Record->setHasLoadedFieldsFromExternalStorage(true);
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nCXXRecrod : " << Record->getName() << " size(methods): "
+                 << std::distance(Record->method_begin(), Record->method_end())
+                 << " has def?:  " << Record->hasDefinition()
+                 << " # (methods): "
+                 << std::distance(Record->getDefinition()->method_begin(),
+                                  Record->getDefinition()->method_end())
+                 << "\n");
+      for (auto *Meth : Record->methods())
+        SetExternalVisibleDeclsForName(ChildDeclContext, Meth->getDeclName(),
+                                       Meth);
     }
     ChildDeclContext->setHasExternalLexicalStorage(false);
   }
 }
 
-void codeComplete(CompilerInstance *InterpCI, llvm::StringRef Content,
-                  unsigned Line, unsigned Col, const CompilerInstance *ParentCI,
-                  std::vector<std::string> &CCResults) {
+void ReplCodeCompleter::codeComplete(CompilerInstance *InterpCI,
+                                     llvm::StringRef Content, unsigned Line,
+                                     unsigned Col,
+                                     const CompilerInstance *ParentCI,
+                                     std::vector<std::string> &CCResults) {
   auto DiagOpts = DiagnosticOptions();
-  auto consumer = ReplCompletionConsumer(CCResults);
+  auto consumer = ReplCompletionConsumer(CCResults, *this);
 
   auto diag = InterpCI->getDiagnosticsPtr();
   std::unique_ptr<ASTUnit> AU(ASTUnit::LoadFromCompilerInvocationAction(
diff --git a/clang/lib/Interpreter/Interpreter.cpp b/clang/lib/Interpreter/Interpreter.cpp
index 7968c62cbd3e7b..c9fcef5b5b5af1 100644
--- a/clang/lib/Interpreter/Interpreter.cpp
+++ b/clang/lib/Interpreter/Interpreter.cpp
@@ -319,6 +319,10 @@ const CompilerInstance *Interpreter::getCompilerInstance() const {
   return IncrParser->getCI();
 }
 
+CompilerInstance *Interpreter::getCompilerInstance() {
+  return IncrParser->getCI();
+}
+
 llvm::Expected<llvm::orc::LLJIT &> Interpreter::getExecutionEngine() {
   if (!IncrExecutor) {
     if (auto Err = CreateExecutor())
diff --git a/clang/tools/clang-repl/ClangRepl.cpp b/clang/tools/clang-repl/ClangRepl.cpp
index b9b287127015fd..5bad8145324d06 100644
--- a/clang/tools/clang-repl/ClangRepl.cpp
+++ b/clang/tools/clang-repl/ClangRepl.cpp
@@ -15,6 +15,8 @@
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Interpreter/CodeCompletion.h"
 #include "clang/Interpreter/Interpreter.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Sema/Sema.h"
 
 #include "llvm/ExecutionEngine/Orc/LLJIT.h"
 #include "llvm/LineEditor/LineEditor.h"
@@ -123,22 +125,14 @@ ReplListCompleter::operator()(llvm::StringRef Buffer, size_t Pos,
 
     return {};
   }
-
-  codeComplete(
-      const_cast<clang::CompilerInstance *>((*Interp)->getCompilerInstance()),
-      Buffer, Lines, Pos + 1, MainInterp.getCompilerInstance(), Results);
-
-  size_t space_pos = Buffer.rfind(" ");
-  llvm::StringRef Prefix;
-  if (space_pos == llvm::StringRef::npos) {
-    Prefix = Buffer;
-  } else {
-    Prefix = Buffer.substr(space_pos + 1);
-  }
-
+  auto *MainCI = (*Interp)->getCompilerInstance();
+  auto CC = clang::ReplCodeCompleter();
+  CC.codeComplete(MainCI, Buffer, Lines, Pos + 1,
+                  MainInterp.getCompilerInstance(), Results);
   for (auto c : Results) {
-    if (c.find(Prefix) == 0)
-      Comps.push_back(llvm::LineEditor::Completion(c.substr(Prefix.size()), c));
+    if (c.find(CC.Prefix) == 0)
+      Comps.push_back(
+          llvm::LineEditor::Completion(c.substr(CC.Prefix.size()), c));
   }
   return Comps;
 }
diff --git a/clang/unittests/Interpreter/CodeCompletionTest.cpp b/clang/unittests/Interpreter/CodeCompletionTest.cpp
index 8f5f3545029d08..cd7fdfa588a5d0 100644
--- a/clang/unittests/Interpreter/CodeCompletionTest.cpp
+++ b/clang/unittests/Interpreter/CodeCompletionTest.cpp
@@ -1,7 +1,9 @@
 #include "clang/Interpreter/CodeCompletion.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Interpreter/Interpreter.h"
+#include "clang/Lex/Preprocessor.h"
 #include "clang/Sema/CodeCompleteConsumer.h"
+#include "clang/Sema/Sema.h"
 #include "llvm/LineEditor/LineEditor.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/raw_ostream.h"
@@ -19,7 +21,7 @@ static std::unique_ptr<Interpreter> createInterpreter() {
 }
 
 static std::vector<std::string> runComp(clang::Interpreter &MainInterp,
-                                        llvm::StringRef Prefix,
+                                        llvm::StringRef Input,
                                         llvm::Error &ErrR) {
   auto CI = CB.CreateCpp();
   if (auto Err = CI.takeError()) {
@@ -37,16 +39,14 @@ static std::vector<std::string> runComp(clang::Interpreter &MainInterp,
 
   std::vector<std::string> Results;
   std::vector<std::string> Comps;
-
-  codeComplete(
-      const_cast<clang::CompilerInstance *>((*Interp)->getCompilerInstance()),
-      Prefix, /* Lines */ 1, Prefix.size(), MainInterp.getCompilerInstance(),
-      Results);
+  auto *MainCI = (*Interp)->getCompilerInstance();
+  auto CC = ReplCodeCompleter();
+  CC.codeComplete(MainCI, Input, /* Lines */ 1, Input.size() + 1,
+                  MainInterp.getCompilerInstance(), Results);
 
   for (auto Res : Results)
-    if (Res.find(Prefix) == 0)
+    if (Res.find(CC.Prefix) == 0)
       Comps.push_back(Res);
-
   return Comps;
 }
 
@@ -62,8 +62,9 @@ TEST(CodeCompletionTest, Sanity) {
   }
   auto Err = llvm::Error::success();
   auto comps = runComp(*Interp, "f", Err);
-  EXPECT_EQ((size_t)2, comps.size()); // foo and float
-  EXPECT_EQ(comps[0], std::string("foo"));
+  EXPECT_EQ((size_t)2, comps.size()); // float and foo
+  EXPECT_EQ(comps[0], std::string("float"));
+  EXPECT_EQ(comps[1], std::string("foo"));
   EXPECT_EQ((bool)Err, false);
 }
 
@@ -110,4 +111,202 @@ TEST(CodeCompletionTest, CompFunDeclsNoError) {
   EXPECT_EQ((bool)Err, false);
 }
 
+TEST(CodeCompletionTest, TypedDirected) {
+  auto Interp = createInterpreter();
+  if (auto R = Interp->ParseAndExecute("int application = 12;")) {
+    consumeError(std::move(R));
+    return;
+  }
+  if (auto R = Interp->ParseAndExecute("char apple = '2';")) {
+    consumeError(std::move(R));
+    return;
+  }
+  if (auto R = Interp->ParseAndExecute("void add(int &SomeInt){}")) {
+    consumeError(std::move(R));
+    return;
+  }
+  {
+    auto Err = llvm::Error::success();
+    auto comps = runComp(*Interp, std::string("add("), Err);
+    EXPECT_EQ((size_t)1, comps.size());
+    EXPECT_EQ((bool)Err, false);
+  }
+
+  if (auto R = Interp->ParseAndExecute("int banana = 42;")) {
+    consumeError(std::move(R));
+    return;
+  }
+
+  {
+    auto Err = llvm::Error::success();
+    auto comps = runComp(*Interp, std::string("add("), Err);
+    EXPECT_EQ((size_t)2, comps.size());
+    EXPECT_EQ(comps[0], "application");
+    EXPECT_EQ(comps[1], "banana");
+    EXPECT_EQ((bool)Err, false);
+  }
+
+  {
+    auto Err = llvm::Error::success();
+    auto comps = runComp(*Interp, std::string("add(b"), Err);
+    EXPECT_EQ((size_t)1, comps.size());
+    EXPECT_EQ(comps[0], "banana");
+    EXPECT_EQ((bool)Err, false);
+  }
+}
+
+TEST(CodeCompletionTest, SanityClasses) {
+  auto Interp = createInterpreter();
+  if (auto R = Interp->ParseAndExecute("struct Apple{};")) {
+    consumeError(std::move(R));
+    return;
+  }
+  if (auto R = Interp->ParseAndExecute("void takeApple(Apple &a1){}")) {
+    consumeError(std::move(R));
+    return;
+  }
+  if (auto R = Interp->ParseAndExecute("Apple a1;")) {
+    consumeError(std::move(R));
+    return;
+  }
+  if (auto R = Interp->ParseAndExecute("void takeAppleCopy(Apple a1){}")) {
+    consumeError(std::move(R));
+    return;
+  }
+
+  {
+    auto Err = llvm::Error::success();
+    auto comps = runComp(*Interp, "takeApple(", Err);
+    EXPECT_EQ((size_t)1, comps.size());
+    EXPECT_EQ(comps[0], std::string("a1"));
+    EXPECT_EQ((bool)Err, false);
+  }
+  {
+    auto Err = llvm::Error::success();
+    auto comps = runComp(*Interp, std::string(...
[truncated]

@vgvassilev
Copy link
Contributor

@capfredf, can you include the original commit message in the commit and add a line about what was done to resolve the reason we have reverted it earlier?

@capfredf
Copy link
Contributor Author

@vgvassilev Sure.

@vgvassilev
Copy link
Contributor

@capfredf ping.

This patch contains changes of 002d471, in
addition to a bug fix that added a virtual destructor to
`CompletionContextHandler`

The original changes in the orginal commit piggybacks on clang's
semantic modules to enable semantic completion.  In particular, we use
`CodeCompletionContext` to differentiate two types of code completion. We also
extract the relevant type information from it.
@capfredf
Copy link
Contributor Author

@vgvassilev the commit message in the original commit isn't very useful. To my recollection, I wrote more detailed message than Differential Revision: https://reviews.llvm.org/D159128, but I can't find it now. The original development branch has been deleted. So I rewrote the message.

Copy link
Contributor

@vgvassilev vgvassilev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@vgvassilev vgvassilev merged commit 35b366a into llvm:main Dec 19, 2023
@vvereschaka
Copy link
Contributor

Looks like these changes break theClangReplInterpreterTests unit tests

No available targets are compatible with triple "x86_64-pc-windows-msvc19.37.32825

The builders are ARM/Aarch64 only.

No available targets are compatible with triple "x86_64-pc-windows-msvc19.37.32825"
UNREACHABLE executed at C:\buildbot\temp\llvm-project\llvm\include\llvm/Support/Error.h:759!
Exception Code: 0x80000003
 #0 0x00007ff755428595 setGlobal(int) (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x618595)
 #1 0x00007ff75912bc46 clang::Value::printData(class llvm::raw_ostream &) const (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x431bc46)
 #2 0x00007ff75911d9dc clang::Value::printData(class llvm::raw_ostream &) const (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x430d9dc)
 #3 0x00007ff755398b16 setGlobal(int) (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x588b16)
 #4 0x00007ff754e99976 clang::Value::setUShort(unsigned short) (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x89976)
 #5 0x00007ff754e9c819 setGlobal(int) (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x8c819)
 #6 0x00007ff75546437b setGlobal(int) (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x65437b)
 #7 0x00007ff75548ae64 setGlobal(int) (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x67ae64)
 #8 0x00007ff75548b316 setGlobal(int) (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x67b316)
 #9 0x00007ff75548bd24 setGlobal(int) (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x67bd24)
#10 0x00007ff7554643bb setGlobal(int) (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x6543bb)
#11 0x00007ff75548b668 setGlobal(int) (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x67b668)
#12 0x00007ff7592023c5 clang::Value::printData(class llvm::raw_ostream &) const (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x43f23c5)
#13 0x00007ff759116640 clang::Value::printData(class llvm::raw_ostream &) const (C:\buildbot\temp\build\tools\clang\unittests\Interpreter\ClangReplInterpreterTests.exe+0x4306640)
#14 0x00007ffea7ef7ac4 (C:\WINDOWS\System32\KERNEL32.DLL+0x17ac4)
#15 0x00007ffea855a4e1 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x5a4e1)

Would you take care of it?

@vgvassilev
Copy link
Contributor

@vvereschaka, thanks for the ping. Will look into this shortly. Feel free to revert.

@vgvassilev
Copy link
Contributor

@vvereschaka, would reverting this PR fix the problem? The error seems unrelated to the code changes. IIUC it says that we fail to pass the correct target triple to be able to execute code...

@vvereschaka
Copy link
Contributor

I tried to revert this commit and two next ones with the warning fixes (they are related) locally on the builder. The tests have passed successfully after reverting
These commits have been reverted

@vvereschaka
Copy link
Contributor

I also didn't noticed any related changes with the triple in these commits, but they were the only related with the interpreter in the failed build. I just tried to revert them and run these tests again.

@vgvassilev
Copy link
Contributor

I am testing a fix/workaround. Will send a diff shortly...

@vgvassilev
Copy link
Contributor

Can you try this one:

diff --git a/clang/unittests/Interpreter/CodeCompletionTest.cpp b/clang/unittests/Interpreter/CodeCompletionTest.cpp
index cd7fdfa588a5..873fbda32f05 100644
--- a/clang/unittests/Interpreter/CodeCompletionTest.cpp
+++ b/clang/unittests/Interpreter/CodeCompletionTest.cpp
@@ -50,16 +50,9 @@ static std::vector<std::string> runComp(clang::Interpreter &MainInterp,
   return Comps;
 }
 
-#ifdef _AIX
-TEST(CodeCompletionTest, DISABLED_Sanity) {
-#else
 TEST(CodeCompletionTest, Sanity) {
-#endif
   auto Interp = createInterpreter();
-  if (auto R = Interp->ParseAndExecute("int foo = 12;")) {
-    consumeError(std::move(R));
-    return;
-  }
+  cantFail(Interp->Parse("int foo = 12;"));
   auto Err = llvm::Error::success();
   auto comps = runComp(*Interp, "f", Err);
   EXPECT_EQ((size_t)2, comps.size()); // float and foo
@@ -68,36 +61,19 @@ TEST(CodeCompletionTest, Sanity) {
   EXPECT_EQ((bool)Err, false);
 }
 
-#ifdef _AIX
-TEST(CodeCompletionTest, DISABLED_SanityNoneValid) {
-#else
 TEST(CodeCompletionTest, SanityNoneValid) {
-#endif
   auto Interp = createInterpreter();
-  if (auto R = Interp->ParseAndExecute("int foo = 12;")) {
-    consumeError(std::move(R));
-    return;
-  }
+  cantFail(Interp->Parse("int foo = 12;"));
   auto Err = llvm::Error::success();
   auto comps = runComp(*Interp, "babanana", Err);
   EXPECT_EQ((size_t)0, comps.size()); // foo and float
   EXPECT_EQ((bool)Err, false);
 }
 
-#ifdef _AIX
-TEST(CodeCompletionTest, DISABLED_TwoDecls) {
-#else
 TEST(CodeCompletionTest, TwoDecls) {
-#endif
   auto Interp = createInterpreter();
-  if (auto R = Interp->ParseAndExecute("int application = 12;")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("int apple = 12;")) {
-    consumeError(std::move(R));
-    return;
-  }
+  cantFail(Interp->Parse("int application = 12;"));
+  cantFail(Interp->Parse("int apple = 12;"));
   auto Err = llvm::Error::success();
   auto comps = runComp(*Interp, "app", Err);
   EXPECT_EQ((size_t)2, comps.size());
@@ -113,18 +89,9 @@ TEST(CodeCompletionTest, CompFunDeclsNoError) {
 
 TEST(CodeCompletionTest, TypedDirected) {
   auto Interp = createInterpreter();
-  if (auto R = Interp->ParseAndExecute("int application = 12;")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("char apple = '2';")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("void add(int &SomeInt){}")) {
-    consumeError(std::move(R));
-    return;
-  }
+  cantFail(Interp->Parse("int application = 12;"));
+  cantFail(Interp->Parse("char apple = '2';"));
+  cantFail(Interp->Parse("void add(int &SomeInt){}"));
   {
     auto Err = llvm::Error::success();
     auto comps = runComp(*Interp, std::string("add("), Err);
@@ -132,10 +99,7 @@ TEST(CodeCompletionTest, TypedDirected) {
     EXPECT_EQ((bool)Err, false);
   }
 
-  if (auto R = Interp->ParseAndExecute("int banana = 42;")) {
-    consumeError(std::move(R));
-    return;
-  }
+  cantFail(Interp->Parse("int banana = 42;"));
 
   {
     auto Err = llvm::Error::success();
@@ -157,22 +121,10 @@ TEST(CodeCompletionTest, TypedDirected) {
 
 TEST(CodeCompletionTest, SanityClasses) {
   auto Interp = createInterpreter();
-  if (auto R = Interp->ParseAndExecute("struct Apple{};")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("void takeApple(Apple &a1){}")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("Apple a1;")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("void takeAppleCopy(Apple a1){}")) {
-    consumeError(std::move(R));
-    return;
-  }
+  cantFail(Interp->Parse("struct Apple{};"));
+  cantFail(Interp->Parse("void takeApple(Apple &a1){}"));
+  cantFail(Interp->Parse("Apple a1;"));
+  cantFail(Interp->Parse("void takeAppleCopy(Apple a1){}"));
 
   {
     auto Err = llvm::Error::success();
@@ -192,26 +144,11 @@ TEST(CodeCompletionTest, SanityClasses) {
 
 TEST(CodeCompletionTest, SubClassing) {
   auto Interp = createInterpreter();
-  if (auto R = Interp->ParseAndExecute("struct Fruit {};")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("struct Apple : Fruit{};")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("void takeFruit(Fruit &f){}")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("Apple a1;")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("Fruit f1;")) {
-    consumeError(std::move(R));
-    return;
-  }
+  cantFail(Interp->Parse("struct Fruit {};"));
+  cantFail(Interp->Parse("struct Apple : Fruit{};"));
+  cantFail(Interp->Parse("void takeFruit(Fruit &f){}"));
+  cantFail(Interp->Parse("Apple a1;"));
+  cantFail(Interp->Parse("Fruit f1;"));
   auto Err = llvm::Error::success();
   auto comps = runComp(*Interp, std::string("takeFruit("), Err);
   EXPECT_EQ((size_t)2, comps.size());
@@ -222,18 +159,9 @@ TEST(CodeCompletionTest, SubClassing) {
 
 TEST(CodeCompletionTest, MultipleArguments) {
   auto Interp = createInterpreter();
-  if (auto R = Interp->ParseAndExecute("int foo = 42;")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("char fowl = 'A';")) {
-    consumeError(std::move(R));
-    return;
-  }
-  if (auto R = Interp->ParseAndExecute("void takeTwo(int &a, char b){}")) {
-    consumeError(std::move(R));
-    return;
-  }
+  cantFail(Interp->Parse("int foo = 42;"));
+  cantFail(Interp->Parse("char fowl = 'A';"));
+  cantFail(Interp->Parse("void takeTwo(int &a, char b){}"));
   auto Err = llvm::Error::success();
   auto comps = runComp(*Interp, std::string("takeTwo(foo,  "), Err);
   EXPECT_EQ((size_t)1, comps.size());
@@ -243,9 +171,9 @@ TEST(CodeCompletionTest, MultipleArguments) {
 
 TEST(CodeCompletionTest, Methods) {
   auto Interp = createInterpreter();
-  cantFail(Interp->ParseAndExecute(
+  cantFail(Interp->Parse(
       "struct Foo{int add(int a){return 42;} int par(int b){return 42;}};"));
-  cantFail(Interp->ParseAndExecute("Foo f1;"));
+  cantFail(Interp->Parse("Foo f1;"));
 
   auto Err = llvm::Error::success();
   auto comps = runComp(*Interp, std::string("f1."), Err);
@@ -257,10 +185,10 @@ TEST(CodeCompletionTest, Methods) {
 
 TEST(CodeCompletionTest, MethodsInvocations) {
   auto Interp = createInterpreter();
-  cantFail(Interp->ParseAndExecute(
+  cantFail(Interp->Parse(
       "struct Foo{int add(int a){return 42;} int par(int b){return 42;}};"));
-  cantFail(Interp->ParseAndExecute("Foo f1;"));
-  cantFail(Interp->ParseAndExecute("int a = 84;"));
+  cantFail(Interp->Parse("Foo f1;"));
+  cantFail(Interp->Parse("int a = 84;"));
 
   auto Err = llvm::Error::success();
   auto comps = runComp(*Interp, std::string("f1.add("), Err);
@@ -271,11 +199,11 @@ TEST(CodeCompletionTest, MethodsInvocations) {
 
 TEST(CodeCompletionTest, NestedInvocations) {
   auto Interp = createInterpreter();
-  cantFail(Interp->ParseAndExecute(
+  cantFail(Interp->Parse(
       "struct Foo{int add(int a){return 42;} int par(int b){return 42;}};"));
-  cantFail(Interp->ParseAndExecute("Foo f1;"));
-  cantFail(Interp->ParseAndExecute("int a = 84;"));
-  cantFail(Interp->ParseAndExecute("int plus(int a, int b) { return a + b; }"));
+  cantFail(Interp->Parse("Foo f1;"));
+  cantFail(Interp->Parse("int a = 84;"));
+  cantFail(Interp->Parse("int plus(int a, int b) { return a + b; }"));
 
   auto Err = llvm::Error::success();
   auto comps = runComp(*Interp, std::string("plus(42, f1.add("), Err);
@@ -287,8 +215,8 @@ TEST(CodeCompletionTest, NestedInvocations) {
 TEST(CodeCompletionTest, TemplateFunctions) {
   auto Interp = createInterpreter();
   cantFail(
-      Interp->ParseAndExecute("template <typename T> T id(T a) { return a;} "));
-  cantFail(Interp->ParseAndExecute("int apple = 84;"));
+      Interp->Parse("template <typename T> T id(T a) { return a;} "));
+  cantFail(Interp->Parse("int apple = 84;"));
   {
     auto Err = llvm::Error::success();
     auto comps = runComp(*Interp, std::string("id<int>("), Err);
@@ -297,9 +225,9 @@ TEST(CodeCompletionTest, TemplateFunctions) {
     EXPECT_EQ((bool)Err, false);
   }
 
-  cantFail(Interp->ParseAndExecute(
+  cantFail(Interp->Parse(
       "template <typename T> T pickFirst(T a, T b) { return a;} "));
-  cantFail(Interp->ParseAndExecute("char pear = '4';"));
+  cantFail(Interp->Parse("char pear = '4';"));
   {
     auto Err = llvm::Error::success();
     auto comps = runComp(*Interp, std::string("pickFirst(apple, "), Err);

@vvereschaka
Copy link
Contributor

@vgvassilev thank you. This patch looks promising, the tests have been passed successfully with it

-- Testing: 21 of 19608 tests, 21 workers --
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/8/21 (1 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/3/21 (2 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/5/21 (3 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/1/21 (4 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/7/21 (5 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/4/21 (6 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/0/21 (7 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/6/21 (8 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/12/21 (9 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/2/21 (10 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/9/21 (11 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/11/21 (12 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/10/21 (13 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/17/21 (14 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/18/21 (15 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/16/21 (16 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/15/21 (17 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/19/21 (18 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/20/21 (19 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/14/21 (20 of 21)
PASS: Clang-Unit :: Interpreter/./ClangReplInterpreterTests.exe/13/21 (21 of 21)

@vvereschaka
Copy link
Contributor

I would like to do a full rebuild just in case. I'll let you know about the results.

@vgvassilev
Copy link
Contributor

If all good, would you mind landing it? I need to disappear for couple of hours.

@vvereschaka
Copy link
Contributor

not a problem, I'll land this patch,

@vvereschaka
Copy link
Contributor

Works fine on my builder. Thank you. I'm going to land this patch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
clang:frontend Language frontend issues, e.g. anything involving "Sema" clang Clang issues not falling into any other category
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants