Skip to content

Latest commit

 

History

History
182 lines (135 loc) · 6.89 KB

clang_tool_refactoring.md

File metadata and controls

182 lines (135 loc) · 6.89 KB

Clang Tool Refactoring

[TOC]

Introduction

Clang tools can help with global refactorings of Chromium code. Clang tools can take advantage of clang's AST to perform refactorings that would be impossible with a traditional find-and-replace regexp:

Caveats

An invocation of the clang tool runs on one build config. Code that only compiles on one platform or code that is guarded by a set of compile-time flags can be problematic. Performing a global refactoring typically requires running the tool once in each build config with code that needs to be updated.

Other minor issues:

Prerequisites

A Chromium checkout created with fetch should have everything needed.

For convenience, add third_party/llvm-build/Release+Asserts/bin to $PATH.

Writing the tool

LLVM uses C++11 and CMake. Source code for Chromium clang tools lives in //tools/clang. It is generally easiest to use one of the already-written tools as the base for writing a new tool.

Chromium clang tools generally follow this pattern:

  1. Instantiate a clang::ast_matchers::MatchFinder.
  2. Call addMatcher() to register clang::ast_matchers::MatchFinder::MatchCallback actions to execute when matching the AST.
  3. Create a new clang::tooling::FrontendActionFactory from the MatchFinder.
  4. Run the action across the specified files with clang::tooling::ClangTool::run.
  5. Serialize generated clang::tooling::Replacements to stdout.

Other useful references when writing the tool:

Edit serialization format

==== BEGIN EDITS ====
r:::path/to/file1:::offset1:::length1:::replacement text
r:::path/to/file2:::offset2:::length2:::replacement text

       ...

==== END EDITS ====

The header and footer are required. Each line between the header and footer represents one edit. Fields are separated by :::, and the first field must be r (for replacement). In the future, this may be extended to handle header insertion/removal. A deletion is an edit with no replacement text.

The edits are applied by run_tool.py, which understands certain conventions:

  • The tool should munge newlines in replacement text to \0. The script knows to translate \0 back to newlines when applying edits.
  • When removing an element from a 'list' (e.g. function parameters, initializers), the tool should emit a deletion for just the element. The script understands how to extend the deletion to remove commas, etc. as needed.

TODO: Document more about SourceLocation and how spelling loc differs from expansion loc, etc.

Why not RefactoringTool?

While clang has a clang::tooling::RefactoringTool to automatically apply the generated replacements and save the results, it doesn't work well for Chromium:

  • Clang tools run actions serially, so runtime scales poorly to tens of thousands of files.
  • A parsing error in any file (quite common in NaCl source) prevents any of the generated replacements from being applied.

Building

Synopsis:

tools/clang/scripts/update.py --bootstrap --force-local-build --without-android \
  --tools blink_gc_plugin plugins rewrite_to_chrome_style

Running this command builds the Oilpan plugin, the Chrome style plugin, and the Blink to Chrome style rewriter. Additional arguments to --tools should be the name of subdirectories in //tools/clang. Generally, --tools should always include blink_gc_plugin and plugins: otherwise, Chromium won't build.

It is important to use --bootstrap as there appear to be bugs in the clang library this script produces if you build it with gcc, which is the default.

Running

First, build all Chromium targets to avoid failures due to missing dependencies that are generated as part of the build:

ninja -C out/Debug

Then run the actual tool:

tools/clang/scripts/run_tool.py <toolname> \
  --generate-compdb
  out/Debug <path 1> <path 2> ...

--generate-compdb can be omitted if the compile DB was already generated and the list of build flags and source files has not changed since generation.

<path 1>, <path 2>, etc are optional arguments to filter the files to run the tool across. This is helpful when sharding global refactorings into smaller chunks. For example, the following command will run the empty_string tool across just the files in //base:

tools/clang/scripts/run_tool.py empty_string  \
  --generated-compdb \
  out/Debug base

Debugging

Dumping the AST for a file:

clang++ -cc1 -ast-dump foo.cc

Using clang-query to dynamically test matchers (requires checking out and building clang-tools-extras):

clang-query -p path/to/compdb base/memory/ref_counted.cc

printf debugging:

  clang::Decl* decl = result.Nodes.getNodeAs<clang::Decl>("decl");
  decl->dumpColor();
  clang::Stmt* stmt = result.Nodes.getNodeAs<clang::Stmt>("stmt");
  stmt->dumpColor();

By default, the script hides the output of the tool. The easiest way to change that is to return 1 from the main() function of the clang tool.

Testing

Synposis:

tools/clang/scripts/test_tool.py <tool name>

The name of the tool binary and the subdirectory for the tool in //tools/clang must match. The test runner finds all files that match the pattern //tools/clang/<tool name>/tests/*-original.cc, runs the tool across those files, and compared it to the *-expected.cc version. If there is a mismatch, the result is saved in *-actual.cc.