[5.1] Update with changes from master

Documentation/Development.md

@@ -0,0 +1,124 @@
+# Development
+
+This document contains notes about development and testing of IndexStoreDB.
+
+## Table of Contents
+
+* [Writing Tests](#writing-tests)
+* [Tibs, the "Test Index Build System"](Tibs.md)
+
+## Writing Tests
+
+As much as is practical, all code should be covered by tests. New tests can be added under the `Tests` directory and should use `XCTest`. The rest of this section will describe the additional tools available in the `ISDBTestSupport` module to make it easier to write good and efficient tests.
+
+Most indexer tests follow a pattern:
+
+1. Build and index a test project
+2. Perform index queries
+3. Compare the results against known locations in the test code
+4. (Optional) modify the code and repeat.
+
+### Test Projects (Fixtures)
+
+Index test projects should be put in the `Tests/INPUTS` directory, and use the [Tibs](Tibs.md) build system to define their sources and targets. An example test project might look like:
+
+```
+Tests/
+  INPUTS/
+    MyTestProj/
+      a.swift
+      b.swift
+      c.cpp
+```
+
+Where `project.json` describes the project's targets, for example
+
+```
+{ "sources": ["a.swift", "b.swift", "c.cpp"] }
+```
+
+Tibs supports more advanced project configurations, such as multiple swift modules with dependencies, etc. For much more information about Tibs, including what features it supports and how it works, see [Tibs.md](Tibs.md).
+
+### TibsTestWorkspace
+
+The `TibsTestWorkspace` pulls together the various pieces needed for working with tests, including building the project, creating an IndexStoreDB instance, and keeping it up to date after modifying sources.
+
+To create a `TibsTestWorkspace`, use the `staticTibsTestWorkspace` and `mutableTibsTestWorkspace` methods. Tests that do not mutate sources should use `staticTibsTestWorkspace`.
+
+```swift
+func testFoo() {
+
+  // Loads an immutable workspace for the test project "MyTestProj" and resolves
+  // all of its targets, ready to be indexed.
+  guard let ws = try staticTibsTestWorkspace(name: "MyTestProj") else { return }
+
+  // Perform the build and read the produced index data.
+  try ws.buildAndIndex()
+
+  // See the results
+  ws.index.occurrences(ofUSR: "<some usr>", ...)
+
+  ...
+}
+```
+
+#### Source Locations
+
+It is common to want to refer to specific locations in the source code of a test project. This is supported using inline comment syntax.
+
+```swift
+Test.swift:
+func /*myFuncDef*/myFunc() {
+  /*myFuncCall*/myFunc()
+}
+```
+
+In a test, these locations can be referenced by name. The named location is immediately after the comment.
+
+```swift
+
+let loc = ws.testLoc("myFuncDef")
+// TestLocation(url: ..., line: 1, column: 19)
+```
+
+A common use of a `TestLocation` is to form a `SymbolOccurrence` at that location.
+
+```swift
+let occurrence = Symbol(...).at(ws.testLoc("myFuncDef"), roles: .definition)
+```
+
+#### Comparing SymbolOccurrences
+
+Rather than attempting to compare `SymbolOccurrences` for equality, prefer using the custom XCTest assertion method `checkOccurrences`. This API avoids accidentally comparing details that are undefined (e.g. the order of occurrences), or likely to change frequently (e.g. `SymbolRole`s are checked to be a superset instead of an exact match, as we often add new roles).
+
+```
+checkOccurrences(ws.index.occurrences(ofUSR: sym.usr, roles: .all), expected: [
+  sym.at(ws.testLoc("c"), roles: .definition),
+  sym.at(ws.testLoc("c_call"), roles: .call),
+])
+```
+
+#### Mutating Test Sources
+
+In order to test changes to the index, we create a mutable test workspace. The mutable project takes care of copying the sources to a new location. Afterwards, we can mutate the sources using the `TibsTestWorkspace.edit(rebuild: Bool, block: ...)` method.
+
+```
+func testMutating() {
+  guard let ws = try mutableTibsTestWorkspace(name: "MyTestProj") else { return }
+  try ws.buildAndIndex()
+
+  // Check initial state...
+
+  // Perform modifications to "url"!
+  ws.edit(rebuild: true) { editor, files in
+
+    // Read the original contents and add a new line.
+    let new = try files.get(url).appending("func /*moreCodeDef*/moreCode() {}")
+
+    // Provide the new contents for "url".
+    editor.write(new, to: url)
+  }
+}
+```
+
+After we return from `edit()`, the sources are modified and any changes to stored source locations are reflected. We can now `buildAndIndex()` to update the index, or as a convenience we can pass `rebuild: true` to `edit`.

Documentation/Tibs.md

@@ -0,0 +1,128 @@
+# Tibs
+
+Tibs ("Test Index Build System") is a simple and flexible build system designed for test projects of IndexStoreDB and SourceKit-LSP. Tibs can incrementally build or rebuild the index data and generated module files for a Swift and/or C-family language test project. Tibs can also dump the compiler arguments to a clang-compatible JSON compilation database. It is *not designed to compile to executable code*.
+
+Tibs is implemented using [Ninja](https://ninja-build.org), which introduces a new dependency in IndexStoreDB when running tests.
+
+Tibs projects are described by a `project.json` file containing one or more targets. Typically, a test case will use a project fixture located in the `Tests/INPUTS` directory.
+
+## Project
+
+A Tibs project is described by `project.json` manifest file. The top-level entity is a target, which may depend on other targets.
+
+Example:
+
+```
+{
+  "targets": [
+    {
+      "name": "mytarget",
+      "swift_flags": ["-warnings-as-errors"],
+      "sources": ["a.swift", "b.swift"],
+      "dependencies": ["dep1"],
+    },
+    {
+      "name": "dep1",
+      "sources": ["dep1.swift"],
+    },
+  ]
+}
+```
+
+As a convenience, if the project consists of a single target, it can be written at the top level and the name can be omitted, e.g.
+
+```
+{ "sources": ["main.swift"] }
+```
+
+### Targets
+
+Targets have the following fields:
+
+* "name": The name of the target. If omitted, it will be "main".
+* "sources": Array of source file paths.
+* "bridging_header": Optional path to a Swift bridging header.
+* "dependencies": Optional array of target names that this target depends on. This is useful for Swift module dependencies.
+* "swift_flags": Optional array of extra compiler arguments for Swift.
+* "clang_flags": Optional array of extra compiler arguments for Clang.
+
+The directory containing the `project.json` manifest file is considered the project's root directory and any relative paths are relative to that.
+
+## Building Tibs Projects
+
+Most tests using Tibs will use the APIs provided by the `TibsTestWorkspace` class in the `ISDBTestSupport` module, which provides a high-level API for working with a complete Tibs project. The following information is for anyone needing to work using the lower-level APIs from `ISDBTibs` or the `tibs` command-line tool.
+
+At a high level, both the library and command-line interfaces for Tibs work similarly: given a project and toolchain (containing executables clang, swiftc, ninja, etc.), produce a Ninja build description file and a compilation database.  When built using `ninja`, all of the generated Swift modules are created and the raw index data is produced.
+
+### ISDBTibs Library
+
+The `TibsBuilder` class provides a library interface for building Tibs projects. It is responsible for fully resolving all the dependencies, build outputs, and compiler arguments. It also has APIs for executing the build using the `ninja` command-line tool and outputting a compilation database (`compile_commands.json`).
+
+#### Example
+
+The inputs to the build process are a toolchain, project, and build directory.
+
+```
+import ISDBTibs
+
+let toolchain = TibsToolchain(...)
+let projectDir: URL = ...
+let buildDir: URL = ...
+```
+
+We load the `project.json` manifest from the project directory.
+
+```
+let manifest = try TibsManifest.load(projectRoot: projectDir)
+```
+
+From a manifest, we can create a `TibsBuilder` and examine the fully resolved targets.
+
+```
+let builder = try TibsBuilder(
+  manifest: manifest,
+  sourceRoot: projectDir,
+  buildRoot: buildDir,
+  toolchain: toolchain)
+
+for target in builder.targets {
+  ...
+}
+```
+
+Finally, we can write the build files and build, or incrementally rebuild the project's index data and modules.
+
+```
+try builder.writeBuildFiles()
+try builder.build()
+// edit sources
+try builder.build()
+```
+
+### tibs Command-line Tool
+
+As a convenience, we also provide a command-line interface `tibs`. It uses the `ISDBTibs` library to write out build files, which can then be built using `ninja`.
+
+#### Example
+
+Running `tibs` will write out the necessary Ninja build file and compilation database.
+
+```
+mkdir build
+cd build
+tibs <path to project directory, containing project.json>
+```
+
+To execute the build, use the `ninja` command
+
+```
+ninja
+```
+
+The build directory will be populated with the generated swift modules and index data.
+
+## FAQ
+
+### Why not use the Swift Package Manager?
+
+The primary reason not to use the Swift Package Manager (SwiftPM) for our test projects is that SwiftPM's model is stricter than other build systems we need to support, and stricter than we want for our testing support. For example, we want to be able to test mixed language targets (using bridging headers), and to perform only the module-generation and indexing parts of the build without emitting object code. We need to be able to add features to our test support that would break the clean model that SwiftPM provides.

Package.swift

@@ -11,6 +11,12 @@ let package = Package(
     .library(
       name: "IndexStoreDB_CXX",
       targets: ["IndexStoreDB_Index"]),
+    .library(
+      name: "ISDBTestSupport",
+      targets: ["ISDBTestSupport"]),
+    .executable(
+      name: "tibs",
+      targets: ["tibs"])
   ],
   dependencies: [],
   targets: [
@@ -23,7 +29,28 @@ let package = Package(
 
     .testTarget(
       name: "IndexStoreDBTests",
-      dependencies: ["IndexStoreDB"]),
+      dependencies: ["IndexStoreDB", "ISDBTestSupport"]),
+
+    // MARK: Swift Test Infrastructure
+
+    // The Test Index Build System (tibs) library.
+    .target(
+      name: "ISDBTibs",
+      dependencies: []),
+
+    .testTarget(
+      name: "ISDBTibsTests",
+      dependencies: ["ISDBTibs"]),
+
+    // Commandline tool for working with tibs projects.
+    .target(
+      name: "tibs",
+      dependencies: ["ISDBTibs"]),
+
+    // Test support library, built on top of tibs.
+    .target(
+      name: "ISDBTestSupport",
+      dependencies: ["IndexStoreDB", "ISDBTibs"]),
 
     // MARK: C++ interface
 

README.md

@@ -22,3 +22,7 @@ The C++ code in the index requires `libdispatch`, but unlike Swift code, it cann
 ```sh
 $ swift build -Xcxx -I<path_to_swift_toolchain>/usr/lib/swift -Xcxx -I<path_to_swift_toolchain>/usr/lib/swift/Block
 ```
+
+## Development
+
+For more information about developing IndexStoreDB, see [Development](Documentation/Development.md).

Sources/ISDBTestSupport/SourceFileCache.swift

@@ -0,0 +1,48 @@
+//===----------------------------------------------------------------------===//
+//
+// This source file is part of the Swift.org open source project
+//
+// Copyright (c) 2014 - 2019 Apple Inc. and the Swift project authors
+// Licensed under Apache License v2.0 with Runtime Library Exception
+//
+// See https://swift.org/LICENSE.txt for license information
+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
+//
+//===----------------------------------------------------------------------===//
+
+import Foundation
+
+/// Reads and caches file contents by URL.
+///
+/// Use `cache.get(url)` to read a file, or get its cached contents. The contents can be overridden
+/// or removed from the cache by calling `cache.set(url, to: "new contents")`
+public final class SourceFileCache {
+  var cache: [URL: String] = [:]
+
+  public init(_ cache: [URL: String] = [:]) {
+    self.cache = cache
+  }
+
+  /// Read the contents of `file`, or retrieve them from the cache if available.
+  ///
+  /// * parameter file: The file to read.
+  /// * returns: The file contents as a String.
+  /// * throws: If there are any errors reading the file.
+  public func get(_ file: URL) throws -> String {
+    if let content = cache[file] {
+      return content
+    }
+    let content = try String(contentsOfFile: file.path, encoding: .utf8)
+    cache[file] = content
+    return content
+  }
+
+  /// Set the cached contents of `file` to `content`.
+  ///
+  /// * parameters
+  ///   * file: The file to read.
+  ///   * content: The new file content.
+  public func set(_ file: URL, to content: String?) {
+    cache[file] = content
+  }
+}