Skip to content

Add infrastructure for testing IndexStoreDB #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

benlangmuir
Copy link
Contributor

The goal of this change is to make it convenient to add tests for the
index that test the whole process of producing the raw index data using
the compilers in the toolchain, importing that data, and
producing/updating an IndexStoreDB. The core infrastructure is also
designed to be shared by the SourceKitLSP project. All of this is built
for tests written in Swift.

This change can be broken down into some inter-dependent high-level
components:

  1. Tibs (Test Index Build System)
  2. ISDBTestSupport
  3. Improvements to IndexStoreDB to make it testable
  4. Tests

Tibs

Tibs, or the "Test Index Build System" provides a simple and flexible
build system for test projects. A test case can use a fixture from the
Tests/INPUTS directory containing a project.json describing any
targets of the test. Targets may be Swift modules containing any number
of source files, C translations units, or a mix of both, including mixed
language targets (subject to the capabilities of the platform - Linux
cannot use Objective-C interop, for example). The test can then
programatically build index raw data, make modifications to source
files, and incrementally rebuild.

Tibs is implemented using ninja, which introduces a new dependency in
IndexStoreDB only when running tests.

ISDBTestSupport

This module contains a number of useful types and functions for working
with test fixtures. In particular,

  • TestSources provides a one-stop shop for working with the source files
    inside a test project, including modifying them and scanning them for
    named TestLocations, which can then be used to locate positions within
    source code during a test.

  • TibsTestWorkspace provides a convenient way to work with a tibs test
    project, its TestSources, and the resulting `IndexStoreDB.. Mutable
    projects are created with sources and build products copied into a
    temporary location (and cleared on deinit), and can be edited and
    rebuilt during the test. An immutable (static) test workspace can use
    the sources directly from the INPUTS directory, and its build products
    are stored in the build directory for IndexStoreDB itself, allowing
    faster incremental builds (or null builds) during development.

Improvements to IndexStoreDB

The biggest change here is that the Symbol and SymbolOccurrence
types are now simple struct types rather than wrapping the underlying C
API types. This makes it much more convenient to work with them during
testing, used along with the new checkOccurrences function.

There is also a new API to pollForUnitChangesAndWait, intended only
for testing. This replaces watching for file system changes during
testing by polling and comparing to the last known state of the unit
directory. Watching for file system changes is unnecessarily
asynchronous during testing, and this lets us side step any performance
issues with that, as well as determining when an index update is "done".

Tests

To prove the usefulness of the new testing APIs, several new tests for
indexing functionality, were added. These are the first real tests for
the indexing code. There is lots of room to improve testing coverage
further.

The goal of this change is to make it convenient to add tests for the
index that test the whole process of producing the raw index data using
the compilers in the toolchain, importing that data, and
producing/updating an IndexStoreDB. The core infrastructure is also
designed to be shared by the SourceKitLSP project.  All of this is built
for tests written in Swift.

This change can be broken down into some inter-dependent high-level
components:

1. Tibs (Test Index Build System)
2. ISDBTestSupport
3. Improvements to IndexStoreDB to make it testable
4. Tests

---

Tibs
----

Tibs, or the "Test Index Build System" provides a simple and flexible
build system for test projects. A test case can use a fixture from the
`Tests/INPUTS` directory containing a `project.json` describing any
targets of the test. Targets may be Swift modules containing any number
of source files, C translations units, or a mix of both, including mixed
language targets (subject to the capabilities of the platform - Linux
cannot use Objective-C interop, for example). The test can then
programatically build index raw data, make modifications to source
files, and incrementally rebuild.

Tibs is implemented using `ninja`, which introduces a new dependency in
IndexStoreDB *only when running tests*.

ISDBTestSupport
---------------

This module contains a number of useful types and functions for working
with test fixtures. In particular,

* `TestSources` provides a one-stop shop for working with the source files
inside a test project, including modifying them and scanning them for
named `TestLocation`s, which can then be used to locate positions within
source code during a test.

* `TibsTestWorkspace` provides a convenient way to work with a tibs test
project, its `TestSources`, and the resulting `IndexStoreDB.. Mutable
projects are created with sources and build products copied into a
temporary location (and cleared on deinit), and can be edited and
rebuilt during the test. An immutable (static) test workspace can use
the sources directly from the INPUTS directory, and its build products
are stored in the build directory for IndexStoreDB itself, allowing
faster incremental builds (or null builds) during development.

Improvements to IndexStoreDB
----------------------------

The biggest change here is that the `Symbol` and `SymbolOccurrence`
types are now simple struct types rather than wrapping the underlying C
API types. This makes it much more convenient to work with them during
testing, used along with the new `checkOccurrences` function.

There is also a new API to `pollForUnitChangesAndWait`, intended only
for testing. This replaces watching for file system changes during
testing by polling and comparing to the last known state of the unit
directory. Watching for file system changes is unnecessarily
asynchronous during testing, and this lets us side step any performance
issues with that, as well as determining when an index update is "done".

Tests
-----

To prove the usefulness of the new testing APIs, several new tests for
indexing functionality, were added. These are the first real tests for
the indexing code. There is lots of room to improve testing coverage
further.
@benlangmuir benlangmuir requested a review from akyrtzi as a code owner July 22, 2019 20:46
@benlangmuir
Copy link
Contributor Author

@swift-ci please test

@benlangmuir
Copy link
Contributor Author

libIndexStore.so: cannot open shared object file: No such file or directory

Probably need an update to our build preset in the swift repo.

@benlangmuir
Copy link
Contributor Author

swiftlang/swift#26300

@swift-ci please test

@benlangmuir
Copy link
Contributor Author

swiftlang/swift#26300

@swift-ci please test

@benlangmuir
Copy link
Contributor Author

swiftlang/swift#26300

@swift-ci please test

@benlangmuir benlangmuir merged commit 90cafe9 into swiftlang:master Aug 1, 2019
@benlangmuir benlangmuir deleted the test-infrastructure-rebased branch August 1, 2019 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants