[cxx-interop] Import typedef-ed template instantiations #32950 (#33451)

This is a roll-forward of #32950, with explicit c++17 version removed from tests. This is not needed since C++17 is the default anyway.

--

In this PR we teach `ClangImporter` to import typedef statements with template instantiation as its underlying type.

```c++
template<class T>
struct MagicWrapper {
  T t;
};

struct MagicNumber {};

typedef MagicWrapper<MagicNumber> WrappedMagicNumber;
```

will be made available in Swift as if `WrappedMagicNumber` is a regular struct. 

In C++, multiple distinct typedeffed instantiations resolve to the same canonical type. We implement this by creating a hidden intermediate struct that typedef aliasses.

The struct is named as `__CxxTemplateInst` plus Itanium mangled type of the instantiation. For the example above the name of the hidden struct is `__CxxTemplateInst12MagicWrapperI11MagicNumberE`. Double underscore (denoting a reserved C++ identifier) is used to discourage direct usage. We chose Itanium mangling scheme because it produces valid Swift identifiers and covers all C++ edge cases.

Imported module interface of the example above:

```swift
struct __CxxTemplateInst12MagicWrapperI11MagicNumberE {
  var t: MagicNumber
}
struct MagicNumber {}
typealias WrappedMagicNumber = __CxxTemplateInst12MagicWrapperI11MagicNumberE
```

We modified the `SwiftLookupTable` logic to show hidden structs in `swift_ide_test` for convenience.

Co-authored-by: Rosica Dejanovska <rosica@google.com>
Co-authored-by: Dmitri Gribenko <gribozavr@gmail.com>
Co-authored-by: Robert Widmann <devteam.codafi@gmail.com>

Author: 4 people

docs/ABI/Mangling.rst

@@ -1107,3 +1107,40 @@ nominal type descriptor symbol for ``CxxStruct`` while compiling the ``main`` mo
 .. code::
 
   sSo9CxxStructVMn // -> nominal type descriptor for __C.CxxStruct
+
+Importing C++ class template instantiations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A class template instantiation is imported as a struct named
+``__CxxTemplateInst`` plus Itanium mangled type of the instantiation (see the
+``type`` production in the Itanium specification). Note that Itanium mangling is
+used on all platforms, regardless of the ABI of the C++ toolchain, to ensure
+that the mangled name is a valid Swift type name (this is not the case for MSVC
+mangled names). A prefix with a double underscore (to ensure we have a reserved
+C++ identifier) is added to limit the possibility for conflicts with names of
+user-defined structs. The struct is notionally defined in the ``__C`` module,
+similarly to regular C and C++ structs and classes. Consider the following C++
+module:
+
+.. code-block:: c++
+
+  template<class T>
+  struct MagicWrapper {
+    T t;
+  };
+
+  struct MagicNumber {};
+
+  typedef MagicWrapper<MagicNumber> WrappedMagicNumber;
+
+``WrappedMagicNumber`` is imported as a typealias for struct
+``__CxxTemplateInst12MagicWrapperI11MagicNumberE``. Interface of the imported
+module looks as follows:
+
+.. code-block:: swift
+
+  struct __CxxTemplateInst12MagicWrapperI11MagicNumberE {
+    var t: MagicNumber
+  }
+  struct MagicNumber {}
+  typealias WrappedMagicNumber = __CxxTemplateInst12MagicWrapperI11MagicNumberE

docs/CppInteroperabilityManifesto.md

@@ -67,6 +67,7 @@ Assumptions:
     + [Function templates: calls with generic type parameters](#function-templates-calls-with-generic-type-parameters)
     + [Function templates: importing as real generic functions](#function-templates-importing-as-real-generic-functions)
     + [Class templates](#class-templates)
+    + [Class templates: importing instantiation behind typedef](#class-templates-importing-instantiation-behind-typedef)
     + [Class templates: importing specific specilalizations](#class-templates-importing-specific-specilalizations)
     + [Class templates: using with generic type parameters](#class-templates-using-with-generic-type-parameters)
     + [Class templates: using in generic code through a synthesized protocol](#class-templates-using-in-generic-code-through-a-synthesized-protocol)
@@ -2575,6 +2576,45 @@ We could ignore explicit specializations of function templates, because they
 don't affect the API. Explicit specializations of class templates can
 dramatically change the API of the type.
 
+### Class templates: Importing full class template instantiations
+
+A class template instantiation could be imported as a struct named
+`__CxxTemplateInst` plus Itanium mangled type of the instantiation (see the
+`type` production in the Itanium specification). Note that Itanium mangling is
+used on all platforms, regardless of the ABI of the C++ toolchain, to ensure
+that the mangled name is a valid Swift type name (this is not the case for MSVC
+mangled names). A prefix with a double underscore (to ensure we have a reserved
+C++ identifier) is added to limit the possibility for conflicts with names of
+user-defined structs. The struct is notionally defined in the `__C` module,
+similarly to regular C and C++ structs and classes. Consider the following C++
+module:
+
+```c++
+// C++ header.
+
+template<class T>
+struct MagicWrapper {
+  T t;
+};
+struct MagicNumber {};
+
+typedef MagicWrapper<MagicNumber> WrappedMagicNumber;
+```
+
+`WrappedMagicNumber` will be imported as a typealias for a struct
+`__CxxTemplateInst12MagicWrapperI11MagicNumberE`. Interface of the imported
+module will look as follows:
+
+```swift
+// C++ header imported to Swift.
+
+struct __CxxTemplateInst12MagicWrapperI11MagicNumberE {
+    var t: MagicNumber
+}
+struct MagicNumber {}
+typealias WrappedMagicNumber = __CxxTemplateInst12MagicWrapperI11MagicNumberE
+```
+
 ### Class templates: importing specific specilalizations
 
 Just like with calls to C++ function templates, it is easy to compile a use of a
@@ -2752,7 +2792,7 @@ func useConcrete() {
 
 ### Class templates: importing as real generic structs
 
-If we know the complete set of allowed type arguments to a C++ function
+If we know the complete set of allowed type arguments to a C++ struct
 template, we could import it as an actual Swift generic struct. Every method of
 that struct will perform dynamic dispatch based on type parameters. See
 the section about function templates for more details.

include/swift/Strings.h

@@ -38,6 +38,10 @@ constexpr static const StringLiteral MANGLING_MODULE_OBJC = "__C";
 constexpr static const StringLiteral MANGLING_MODULE_CLANG_IMPORTER =
     "__C_Synthesized";
 
+/// The name prefix for C++ template instantiation imported as a Swift struct.
+constexpr static const StringLiteral CXX_TEMPLATE_INST_PREFIX =
+    "__CxxTemplateInst";
+
 constexpr static const StringLiteral SEMANTICS_PROGRAMTERMINATION_POINT =
     "programtermination_point";
 

lib/AST/ASTMangler.cpp

@@ -39,6 +39,7 @@
 #include "clang/AST/Attr.h"
 #include "clang/AST/Decl.h"
 #include "clang/AST/DeclObjC.h"
+#include "clang/AST/DeclTemplate.h"
 #include "clang/AST/Mangle.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/SmallString.h"
@@ -2114,6 +2115,7 @@ void ASTMangler::appendAnyGenericType(const GenericTypeDecl *decl) {
   // Always use Clang names for imported Clang declarations, unless they don't
   // have one.
   auto tryAppendClangName = [this, decl]() -> bool {
+    auto *nominal = dyn_cast<NominalTypeDecl>(decl);
     auto namedDecl = getClangDeclForMangling(decl);
     if (!namedDecl)
       return false;
@@ -2126,6 +2128,13 @@ void ASTMangler::appendAnyGenericType(const GenericTypeDecl *decl) {
       appendIdentifier(interface->getObjCRuntimeNameAsString());
     } else if (UseObjCRuntimeNames && protocol) {
       appendIdentifier(protocol->getObjCRuntimeNameAsString());
+    } else if (auto ctsd = dyn_cast<clang::ClassTemplateSpecializationDecl>(namedDecl)) {
+      // If this is a `ClassTemplateSpecializationDecl`, it was
+      // imported as a Swift decl with `__CxxTemplateInst...` name.
+      // `ClassTemplateSpecializationDecl`'s name does not include information about
+      // template arguments, and in order to prevent name clashes we use the
+      // name of the Swift decl which does include template arguments.
+      appendIdentifier(nominal->getName().str());
     } else {
       appendIdentifier(namedDecl->getName());
     }

lib/ClangImporter/ImportDecl.cpp

@@ -3477,14 +3477,33 @@ namespace {
 
     Decl *VisitClassTemplateSpecializationDecl(
                  const clang::ClassTemplateSpecializationDecl *decl) {
-      // FIXME: We could import specializations, but perhaps only as unnamed
-      // structural types.
-      return nullptr;
+      // `Sema::isCompleteType` will try to instantiate the class template as a
+      // side-effect and we rely on this here. `decl->getDefinition()` can
+      // return nullptr before the call to sema and return its definition
+      // afterwards.
+      if (!Impl.getClangSema().isCompleteType(
+              decl->getLocation(),
+              Impl.getClangASTContext().getRecordType(decl))) {
+        // If we got nullptr definition now it means the type is not complete.
+        // We don't import incomplete types.
+        return nullptr;
+      }
+      auto def = dyn_cast<clang::ClassTemplateSpecializationDecl>(
+          decl->getDefinition());
+      assert(def && "Class template instantiation didn't have definition");
+      // FIXME: This will instantiate all members of the specialization (and detect
+      // instantiation failures in them), which can be more than is necessary
+      // and is more than what Clang does. As a result we reject some C++
+      // programs that Clang accepts.
+      Impl.getClangSema().InstantiateClassTemplateSpecializationMembers(
+          def->getLocation(), def, clang::TSK_ExplicitInstantiationDefinition);
+
+      return VisitRecordDecl(def);
     }
 
     Decl *VisitClassTemplatePartialSpecializationDecl(
-                 const clang::ClassTemplatePartialSpecializationDecl *decl) {
-      // Note: templates are not imported.
+        const clang::ClassTemplatePartialSpecializationDecl *decl) {
+      // Note: partial template specializations are not imported.
       return nullptr;
     }
 

lib/ClangImporter/ImportName.cpp

@@ -31,8 +31,10 @@
 #include "swift/Basic/StringExtras.h"
 #include "swift/ClangImporter/ClangImporterOptions.h"
 #include "swift/Parse/Parser.h"
+#include "swift/Strings.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/DeclCXX.h"
+#include "clang/AST/Mangle.h"
 #include "clang/Basic/IdentifierTable.h"
 #include "clang/Basic/Module.h"
 #include "clang/Basic/OperatorKinds.h"
@@ -1698,6 +1700,34 @@ ImportedName NameImporter::importNameImpl(const clang::NamedDecl *D,
     }
   }
 
+  if (auto classTemplateSpecDecl =
+          dyn_cast<clang::ClassTemplateSpecializationDecl>(D)) {
+    if (!isa<clang::ClassTemplatePartialSpecializationDecl>(D)) {
+
+      auto &astContext = classTemplateSpecDecl->getASTContext();
+      // Itanium mangler produces valid Swift identifiers, use it to generate a name for
+      // this instantiation.
+      clang::MangleContext *mangler = clang::ItaniumMangleContext::create(
+          astContext, astContext.getDiagnostics());
+      llvm::SmallString<128> storage;
+      llvm::raw_svector_ostream buffer(storage);
+      mangler->mangleTypeName(astContext.getRecordType(classTemplateSpecDecl),
+                              buffer);
+
+      // The Itanium mangler does not provide a way to get the mangled
+      // representation of a type. Instead, we call mangleTypeName() that
+      // returns the name of the RTTI typeinfo symbol, and remove the _ZTS
+      // prefix. Then we prepend __CxxTemplateInst to reduce chances of conflict
+      // with regular C and C++ structs.
+      llvm::SmallString<128> mangledNameStorage;
+      llvm::raw_svector_ostream mangledName(mangledNameStorage);
+      assert(buffer.str().take_front(4) == "_ZTS");
+      mangledName << CXX_TEMPLATE_INST_PREFIX << buffer.str().drop_front(4);
+
+      baseName = swiftCtx.getIdentifier(mangledName.str()).get();
+    }
+  }
+
   // swift_newtype-ed declarations may have common words with the type name
   // stripped.
   if (auto newtypeDecl = findSwiftNewtype(D, clangSema, version)) {

lib/ClangImporter/SwiftLookupTable.cpp

@@ -1881,10 +1881,16 @@ void importer::addEntryToLookupTable(SwiftLookupTable &table,
   // struct names when relevant, not just pointer names. That way we can check
   // both CFDatabase.def and the objc_bridge attribute and cover all our bases.
   if (auto *tagDecl = dyn_cast<clang::TagDecl>(named)) {
-    if (!tagDecl->getDefinition())
+    // We add entries for ClassTemplateSpecializations that don't have
+    // definition. It's possible that the decl will be instantiated by
+    // SwiftDeclConverter later on. We cannot force instantiating
+    // ClassTemplateSPecializations here because we're currently writing the
+    // AST, so we cannot modify it.
+    if (!isa<clang::ClassTemplateSpecializationDecl>(named) &&
+        !tagDecl->getDefinition()) {
       return;
+    }
   }
-
   // If we have a name to import as, add this entry to the table.
   auto currentVersion =
       ImportNameVersion::fromOptions(nameImporter.getLangOpts());
@@ -2077,6 +2083,19 @@ void SwiftLookupTableWriter::populateTableWithDecl(SwiftLookupTable &table,
 
   // Add this entry to the lookup table.
   addEntryToLookupTable(table, named, nameImporter);
+  if (auto typedefDecl = dyn_cast<clang::TypedefNameDecl>(named)) {
+    if (auto typedefType = dyn_cast<clang::TemplateSpecializationType>(
+            typedefDecl->getUnderlyingType())) {
+      if (auto CTSD = dyn_cast<clang::ClassTemplateSpecializationDecl>(
+              typedefType->getAsTagDecl())) {
+        // Adding template instantiation behind typedef as a top-level entry
+        // so the instantiation appears in the API.
+        assert(!isa<clang::ClassTemplatePartialSpecializationDecl>(CTSD) &&
+            "Class template partial specialization cannot appear behind typedef");
+        addEntryToLookupTable(table, CTSD, nameImporter);
+      }
+    }
+  }
 }
 
 void SwiftLookupTableWriter::populateTable(SwiftLookupTable &table,

test/Demangle/Inputs/manglings.txt

@@ -365,3 +365,5 @@ $s0059xxxxxxxxxxxxxxx_ttttttttBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBee -> $s0059xxxx
 $sx1td_t ---> (t: A...)
 $s7example1fyyYF -> example.f() async -> ()
 $s7example1fyyYKF -> example.f() async throws -> ()
+$s4main20receiveInstantiationyySo34__CxxTemplateInst12MagicWrapperIiEVzF ---> main.receiveInstantiation(inout __C.__CxxTemplateInst12MagicWrapperIiE) -> ()
+$s4main19returnInstantiationSo34__CxxTemplateInst12MagicWrapperIiEVyF ---> main.returnInstantiation() -> __C.__CxxTemplateInst12MagicWrapperIiE

test/Interop/Cxx/templates/Inputs/canonical-types.h

@@ -0,0 +1,18 @@
+#ifndef TEST_INTEROP_CXX_TEMPLATES_INPUTS_CANONICAL_TYPES_H
+#define TEST_INTEROP_CXX_TEMPLATES_INPUTS_CANONICAL_TYPES_H
+
+template<class T>
+struct MagicWrapper {
+  T t;
+  int getValuePlusArg(int arg) const { return t.getValue() + arg; }
+};
+
+struct IntWrapper {
+  int value;
+  int getValue() const { return value; }
+};
+
+typedef MagicWrapper<IntWrapper> WrappedMagicNumberA;
+typedef MagicWrapper<IntWrapper> WrappedMagicNumberB;
+
+#endif // TEST_INTEROP_CXX_TEMPLATES_INPUTS_CANONICAL_TYPES_H

test/Interop/Cxx/templates/Inputs/decl-with-definition-including-members.h

@@ -0,0 +1,26 @@
+#ifndef TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITH_DEFINITION_INCLUDING_MEMBERS_H
+#define TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITH_DEFINITION_INCLUDING_MEMBERS_H
+
+template<class T>
+struct MagicWrapper {
+  T t;
+  int getValuePlusArg(int arg) const { return t.getValue() + arg; }
+};
+
+struct IntWrapper {
+  int value;
+  int getValue() const { return value; }
+};
+
+inline int forceInstantiation() {
+  auto t = MagicWrapper<IntWrapper>();
+  return t.getValuePlusArg(14);
+}
+
+// The ClassTemplateSpecializationDecl node for MagicWrapper<IntWrapper> already has a definition
+// because function above forced the instantiation. Its members are fully
+// instantiated, so nothing needs to be explicitly instantiated by the Swift
+// compiler.
+typedef MagicWrapper<IntWrapper> FullyDefinedMagicallyWrappedInt;
+
+#endif // TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITH_DEFINITION_INCLUDING_MEMBERS_H

test/Interop/Cxx/templates/Inputs/decl-with-definition.h

@@ -0,0 +1,24 @@
+#ifndef TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITH_DEFINITION_H
+#define TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITH_DEFINITION_H
+
+template<class T>
+struct MagicWrapper {
+  T t;
+  int getValuePlusArg(int arg) const { return t.getValue() + arg; }
+};
+
+struct IntWrapper {
+  int value;
+  int getValue() const { return value; }
+};
+
+inline MagicWrapper<IntWrapper> forceInstantiation() {
+  return MagicWrapper<IntWrapper>();
+}
+
+// The ClassTemplateSpecializationDecl node for MagicWrapper<IntWrapper> already has a definition
+// because function above forced the instantiation. Its members are not
+// instantiated though, the Swift compiler needs to instantiate them.
+typedef MagicWrapper<IntWrapper> PartiallyDefinedMagicallyWrappedInt;
+
+#endif // TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITH_DEFINITION_H

test/Interop/Cxx/templates/Inputs/decl-with-primitive-argument.h

@@ -0,0 +1,12 @@
+#ifndef TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITH_PRIMITIVE_ARGUMENT_H
+#define TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITH_PRIMITIVE_ARGUMENT_H
+
+template<class T>
+struct MagicWrapper {
+  T t;
+  int getValuePlusArg(int arg) const { return t + arg; }
+};
+
+typedef MagicWrapper<int> WrappedMagicInt;
+
+#endif // TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITH_PRIMITIVE_ARGUMENT_H

test/Interop/Cxx/templates/Inputs/decl-without-definition.h

@@ -0,0 +1,20 @@
+#ifndef TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITHOUT_DEFINITION_H
+#define TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITHOUT_DEFINITION_H
+
+template<class T>
+struct MagicWrapper {
+  T t;
+  int getValuePlusArg(int arg) const { return t.getValue() + arg; }
+};
+
+struct IntWrapper {
+  int value;
+  int getValue() const { return value; }
+};
+
+// The ClassTemplateSpecializationDecl node for MagicWrapper<IntWrapper> doesn't have a
+// definition in Clang because nothing in this header required the
+// instantiation. Therefore, the Swift compiler must trigger instantiation.
+typedef MagicWrapper<IntWrapper> MagicallyWrappedIntWithoutDefinition;
+
+#endif // TEST_INTEROP_CXX_TEMPLATES_INPUTS_DECL_WITHOUT_DEFINITION_H