Skip to content

Commit cdb0af2

Browse files
ahoppenahoppen
committed
Split SwiftSyntaxParser into a separate module
When SwiftSyntax is just being used to generate code and not to parse Swift source files, there is no reason that a compatible `_InternalSwiftSyntaxParser.dylib` needs to be present. We do, however, currently have the requirement of always needing a `_InternalSwiftSyntaxParser.dylib` because `SyntaxParser` lives inside the `SwiftSyntax` module and thus `SwiftSyntax` needs to link against `_InternalSwiftSyntaxParser.dylib`. To lift the requirement, split `SwiftSyntax` into two modules: `SwiftSyntax` has no requirement on `_InternalSwiftSyntaxParser.dylib`. It only offers functionality to programatically generate `SwiftSyntax` trees and walk them. The new `SwiftSyntaxParser` module links against `_InternalSwiftSyntaxParser.dylib` and provides the ability to generate `SwiftSyntax` trees from source files. To efficiently generate `SwiftSyntax` nodes from the parser, `RawSyntax` (and its child types) does depend on the layout of the C nodes (for example, `TokenData` stores `CTriviaPiece`s as trailing trivia and only constructs the trivia’s Swift representation when requested). I don’t think it’s possible to eliminate that behaviour without eager translation and thus decided to include a copy of the C nodes inside the `SwiftSyntax`. When `SwiftSyntax` is used on its own, they are never used and thus their value is irrelevant. When a source file is parsed through `SwiftSyntaxParser`, it first verifies that the C node layouts match those defined in `_InternalSwiftSyntaxParser.dylib`, similar to how we currently compute the node hash. If the layouts are not compatible, we throw an error.
1 parent 8738d98 commit cdb0af2

38 files changed

+468
-144
lines changed

‎Changelog.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ Note: This is in reverse chronological order, so newer entries are added to the
55
## `main`
66

77
* To clarify that the edits passed to `IncrementalParseTransition` are applied concurrently, introduce a new `ConcurrentEdit` type that provides the guarantee and allows translation of sequentially applied edits to the expected concurrent form.
8+
* The `SwiftSyntaxParser` type and a few related types now live in their own module (also named `SwiftSyntaxParser`). This allows using `SwiftSyntax` for code generation purposes without having a compatible `_InternalSwiftSyntaxParser.dylib` around.
9+
10+
`import SwiftSyntaxParser` where necessary.
811

912
## Swift 5.3
1013

‎Package.swift

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,13 @@ let package = Package(
77
name: "SwiftSyntax",
88
targets: [
99
.target(name: "_CSwiftSyntax"),
10-
.testTarget(name: "SwiftSyntaxTest", dependencies: ["SwiftSyntax"], exclude: ["Inputs"]),
10+
.testTarget(name: "SwiftSyntaxTest", dependencies: ["SwiftSyntax"]),
1111
.target(name: "SwiftSyntaxBuilder", dependencies: ["SwiftSyntax"]),
1212
.testTarget(name: "SwiftSyntaxBuilderTest", dependencies: ["SwiftSyntaxBuilder"]),
13-
.target(name: "lit-test-helper", dependencies: ["SwiftSyntax"]),
14-
.testTarget(name: "PerformanceTest", dependencies: ["SwiftSyntax"])
13+
.target(name: "SwiftSyntaxParser", dependencies: ["SwiftSyntax"]),
14+
.testTarget(name: "SwiftSyntaxParserTest", dependencies: ["SwiftSyntaxParser"], exclude: ["Inputs"]),
15+
.target(name: "lit-test-helper", dependencies: ["SwiftSyntax", "SwiftSyntaxParser"]),
16+
.testTarget(name: "PerformanceTest", dependencies: ["SwiftSyntax", "SwiftSyntaxParser"])
1517
// Also see targets added below
1618
]
1719
)
@@ -52,4 +54,5 @@ if ProcessInfo.processInfo.environment["SWIFT_BUILD_SCRIPT_ENVIRONMENT"] != nil
5254
}
5355

5456
package.products.append(.library(name: "SwiftSyntax", type: libraryType, targets: ["SwiftSyntax"]))
57+
package.products.append(.library(name: "SwiftSyntaxParser", type: libraryType, targets: ["SwiftSyntaxParser"]))
5558
package.products.append(.library(name: "SwiftSyntaxBuilder", type: libraryType, targets: ["SwiftSyntaxBuilder"]))

‎Sources/SwiftSyntax/CNodes.swift

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
//===---------------------------- CNodes.swift ----------------------------===//
2+
//
3+
// This source file is part of the Swift.org open source project
4+
//
5+
// Copyright (c) 2014 - 2021 Apple Inc. and the Swift project authors
6+
// Licensed under Apache License v2.0 with Runtime Library Exception
7+
//
8+
// See https://swift.org/LICENSE.txt for license information
9+
// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
10+
//
11+
//===----------------------------------------------------------------------===//
12+
13+
@_implementationOnly import _CSwiftSyntax
14+
15+
public typealias CSyntaxKind = UInt16
16+
public typealias CClientNode = UnsafeMutableRawPointer
17+
18+
typealias CSyntaxNode = swiftparse_syntax_node_t
19+
typealias CSyntaxNodePtr = UnsafePointer<CSyntaxNode>
20+
typealias CTokenKind = swiftparse_token_kind_t
21+
typealias CTriviaKind = swiftparse_trivia_kind_t
22+
typealias CTokenData = swiftparse_token_data_t
23+
typealias CLayoutData = swiftparse_layout_data_t
24+
typealias CTriviaPiecePtr = UnsafePointer<CTriviaPiece>
25+
typealias CTriviaPiece = swiftparse_trivia_piece_t
26+
27+
/// Computes a hash value that describes the layout of all C nodes which are
28+
/// passed as opaque values between `SwiftSyntaxParser` and `SwiftSyntax`.
29+
/// This should match the value returned by the `cNodeLayoutHash` function in
30+
/// the `SwiftSyntaxParser` module.
31+
public func cNodeLayoutHash() -> Int {
32+
var hasher = Hasher()
33+
34+
hasher.combine(MemoryLayout<swiftparse_range_t>.size)
35+
hasher.combine(MemoryLayout<swiftparse_range_t>.offset(of: \.offset))
36+
hasher.combine(MemoryLayout<swiftparse_range_t>.offset(of: \.length))
37+
38+
hasher.combine(MemoryLayout<swiftparse_trivia_kind_t>.size)
39+
hasher.combine(MemoryLayout<swiftparse_token_kind_t>.size)
40+
hasher.combine(MemoryLayout<swiftparse_syntax_kind_t>.size)
41+
42+
hasher.combine(MemoryLayout<swiftparse_client_node_t>.size)
43+
44+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.size)
45+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.offset(of: \.length))
46+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.offset(of: \.kind))
47+
48+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.size)
49+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.leading_trivia))
50+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.trailing_trivia))
51+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.leading_trivia_count))
52+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.trailing_trivia_count))
53+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.kind))
54+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.range))
55+
56+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.size)
57+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.offset(of: \.nodes))
58+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.offset(of: \.nodes_count))
59+
60+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.size)
61+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.offset(of: \.kind))
62+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.offset(of: \.present))
63+
64+
return hasher.finalize()
65+
}

‎Sources/SwiftSyntax/IncrementalParseTransition.swift

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -199,11 +199,11 @@ public struct ConcurrentEdits {
199199

200200
/// Provides a mechanism for the parser to skip regions of an incrementally
201201
/// updated source that was already parsed during a previous parse invocation.
202-
internal struct IncrementalParseLookup {
202+
public struct IncrementalParseLookup {
203203
fileprivate let transition: IncrementalParseTransition
204204
fileprivate var cursor: SyntaxCursor
205205

206-
init(transition: IncrementalParseTransition) {
206+
public init(transition: IncrementalParseTransition) {
207207
self.transition = transition
208208
self.cursor = .init(root: transition.previousTree.data.absoluteRaw)
209209
}
@@ -224,16 +224,16 @@ internal struct IncrementalParseLookup {
224224
///
225225
/// - Parameters:
226226
/// - offset: The byte offset of the source string that is currently parsed.
227-
/// - kind: The `SyntaxKind` that the parser expects at this position.
227+
/// - kind: The `CSyntaxKind` that the parser expects at this position.
228228
/// - Returns: A `SyntaxNode` node from the previous parse invocation,
229229
/// representing the contents of this region, if it is still valid
230230
/// to re-use. `nil` otherwise.
231-
mutating func lookUp(_ newOffset: Int, kind: SyntaxKind) -> SyntaxNode? {
231+
public mutating func lookUp(_ newOffset: Int, kind: CSyntaxKind) -> SyntaxNode? {
232232
guard let prevOffset = translateToPreEditOffset(newOffset) else {
233233
return nil
234234
}
235235
let prevPosition = AbsolutePosition(utf8Offset: prevOffset)
236-
let node = cursorLookup(prevPosition: prevPosition, kind: kind)
236+
let node = cursorLookup(prevPosition: prevPosition, kind: .fromRawValue(kind))
237237
if let delegate = reusedDelegate, let node = node {
238238
delegate.parserReusedNode(
239239
range: ByteSourceRange(offset: newOffset, length: node.byteSize),

‎Sources/SwiftSyntax/Misc.swift.gyb

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,6 @@
1919
//
2020
//===----------------------------------------------------------------------===//
2121

22-
@_implementationOnly import _InternalSwiftSyntaxParser
23-
2422
extension SyntaxNode {
2523
public var isUnknown: Bool { return raw.kind.isUnknown }
2624
public var asUnknown: UnknownSyntax? {
@@ -63,10 +61,3 @@ extension Syntax {
6361
}
6462
}
6563
}
66-
67-
extension SyntaxParser {
68-
static func verifyNodeDeclarationHash() -> Bool {
69-
return String(cString: swiftparse_syntax_structure_versioning_identifier()!) ==
70-
"${calculate_node_hash()}"
71-
}
72-
}

‎Sources/SwiftSyntax/RawSyntax.swift

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1216,12 +1216,12 @@ final class RawSyntax: ManagedBuffer<RawSyntaxBase, RawSyntaxDataElement> {
12161216
}
12171217

12181218
extension RawSyntax {
1219+
static func moveFromOpaque(_ cn: CClientNode) -> RawSyntax {
1220+
return Unmanaged<RawSyntax>.fromOpaque(cn).takeRetainedValue()
1221+
}
1222+
12191223
static func moveFromOpaque(_ cn: CClientNode?) -> RawSyntax? {
1220-
if let subnode = cn {
1221-
return Unmanaged<RawSyntax>.fromOpaque(subnode).takeRetainedValue()
1222-
} else {
1223-
return nil
1224-
}
1224+
return cn.map(moveFromOpaque)
12251225
}
12261226

12271227
static func getFromOpaque(_ cn: CClientNode?) -> RawSyntax? {

‎Sources/SwiftSyntax/_SyntaxParserInterop.swift

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
//===--------------------- _SyntaxParserInterop.swift ---------------------===//
2+
//
3+
// This source file is part of the Swift.org open source project
4+
//
5+
// Copyright (c) 2014 - 2021 Apple Inc. and the Swift project authors
6+
// Licensed under Apache License v2.0 with Runtime Library Exception
7+
//
8+
// See https://swift.org/LICENSE.txt for license information
9+
// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
10+
//
11+
//===----------------------------------------------------------------------===//
12+
13+
/// Namespace for methods needed by the `SwiftSyntaxParser` module to
14+
/// efficiently create `SwiftSyntax` nodes from the C nodes created by the
15+
/// parser.
16+
public enum _SyntaxParserInterop {
17+
private static func getRetainedOpaque(rawSyntax: RawSyntax) -> CClientNode {
18+
return Unmanaged.passRetained(rawSyntax).toOpaque()
19+
}
20+
21+
/// Create a `RawSyntax` node for the given `cnode` and return an opaque
22+
/// pointer to the `RawSyntax` node (a `CClientNode`).
23+
/// After this method finishes, the `RawSyntax` node has a retain count of 1
24+
/// and is owned by whoever manages the returned `CClientNode`. Passing the
25+
/// `CClientNode` to `nodeFromRetainedOpaqueRawSyntax` transfers ownership
26+
/// back to `SwiftSyntax`.
27+
public static func getRetainedOpaqueRawSyntax(
28+
cnode: UnsafeRawPointer, source: String
29+
) -> CClientNode {
30+
let cnode = cnode.assumingMemoryBound(to: CSyntaxNode.self)
31+
// Transfer ownership of the object to the C parser. We get ownership back
32+
// via `moveFromCRawNode()`.
33+
let node = RawSyntax.create(from: cnode, source: source)
34+
return getRetainedOpaque(rawSyntax: node)
35+
}
36+
37+
/// Return an opaque pointer to the given `node`.
38+
/// After this method finishes, the `RawSyntax` node has a retain count of 1
39+
/// and is owned by whoever manages the returned `CClientNode`. Passing the
40+
/// `CClientNode` to `nodeFromRetainedOpaqueRawSyntax` transfers ownership
41+
/// back to `SwiftSyntax`.
42+
public static func getRetainedOpaqueRawSyntax(node: SyntaxNode)
43+
-> CClientNode {
44+
return getRetainedOpaque(rawSyntax: node.raw)
45+
}
46+
47+
/// After an opaque pointer to a `RawSyntax` node has been created using one
48+
/// of the methods above, transfer its ownership back to a `Syntax` node,
49+
/// which is managed by `SwiftSyntax`.
50+
public static func nodeFromRetainedOpaqueRawSyntax(_ cRoot: CClientNode)
51+
-> Syntax {
52+
return Syntax(SyntaxData.forRoot(RawSyntax.moveFromOpaque(cRoot)))
53+
}
54+
}

‎Sources/SwiftSyntax/gyb_generated/Misc.swift

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,6 @@
1212
//
1313
//===----------------------------------------------------------------------===//
1414

15-
@_implementationOnly import _InternalSwiftSyntaxParser
16-
1715
extension SyntaxNode {
1816
public var isUnknown: Bool { return raw.kind.isUnknown }
1917
public var asUnknown: UnknownSyntax? {
@@ -1964,10 +1962,3 @@ extension Syntax {
19641962
}
19651963
}
19661964
}
1967-
1968-
extension SyntaxParser {
1969-
static func verifyNodeDeclarationHash() -> Bool {
1970-
return String(cString: swiftparse_syntax_structure_versioning_identifier()!) ==
1971-
"e9565bceebb81b9c3a69c442a8576b029d7eaf9c"
1972-
}
1973-
}

‎Sources/SwiftSyntaxParser/CNodes.swift

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
//===---------------------------- CNodes.swift ----------------------------===//
2+
//
3+
// This source file is part of the Swift.org open source project
4+
//
5+
// Copyright (c) 2014 - 2021 Apple Inc. and the Swift project authors
6+
// Licensed under Apache License v2.0 with Runtime Library Exception
7+
//
8+
// See https://swift.org/LICENSE.txt for license information
9+
// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
10+
//
11+
//===----------------------------------------------------------------------===//
12+
13+
@_implementationOnly import _InternalSwiftSyntaxParser
14+
15+
typealias CSyntaxNode = swiftparse_syntax_node_t
16+
typealias CTriviaPiece = swiftparse_trivia_piece_t
17+
typealias CSyntaxNodePtr = UnsafePointer<CSyntaxNode>
18+
typealias CTriviaPiecePtr = UnsafePointer<CTriviaPiece>
19+
typealias CSyntaxKind = swiftparse_syntax_kind_t
20+
typealias CTokenKind = swiftparse_token_kind_t
21+
typealias CTriviaKind = swiftparse_trivia_kind_t
22+
typealias CTokenData = swiftparse_token_data_t
23+
typealias CLayoutData = swiftparse_layout_data_t
24+
typealias CParseLookupResult = swiftparse_lookup_result_t
25+
typealias CClientNode = swiftparse_client_node_t
26+
typealias CDiagnostic = swiftparser_diagnostic_t
27+
typealias CFixit = swiftparse_diagnostic_fixit_t
28+
typealias CRange = swiftparse_range_t
29+
30+
/// Computes a hash value that describes the layout of all C nodes which are
31+
/// passed as opaque values between `SwiftSyntaxParser` and `SwiftSyntax`.
32+
/// This should match the value returned by the `cNodeLayoutHash` function in
33+
/// the `SwiftSyntax` module.
34+
public func cNodeLayoutHash() -> Int {
35+
var hasher = Hasher()
36+
37+
hasher.combine(MemoryLayout<swiftparse_range_t>.size)
38+
hasher.combine(MemoryLayout<swiftparse_range_t>.offset(of: \.offset))
39+
hasher.combine(MemoryLayout<swiftparse_range_t>.offset(of: \.length))
40+
41+
hasher.combine(MemoryLayout<swiftparse_trivia_kind_t>.size)
42+
hasher.combine(MemoryLayout<swiftparse_token_kind_t>.size)
43+
hasher.combine(MemoryLayout<swiftparse_syntax_kind_t>.size)
44+
45+
hasher.combine(MemoryLayout<swiftparse_client_node_t>.size)
46+
47+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.size)
48+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.offset(of: \.length))
49+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.offset(of: \.kind))
50+
51+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.size)
52+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.leading_trivia))
53+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.trailing_trivia))
54+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.leading_trivia_count))
55+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.trailing_trivia_count))
56+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.kind))
57+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.range))
58+
59+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.size)
60+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.offset(of: \.nodes))
61+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.offset(of: \.nodes_count))
62+
63+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.size)
64+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.offset(of: \.kind))
65+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.offset(of: \.present))
66+
67+
return hasher.finalize()
68+
}

‎Sources/SwiftSyntax/Diagnostic.swift renamed to ‎Sources/SwiftSyntaxParser/Diagnostic.swift

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@
1212
// This file provides the Diagnostic, Note, and FixIt types.
1313
//===----------------------------------------------------------------------===//
1414

15+
import SwiftSyntax
16+
1517
/// A FixIt represents a change to source code in order to "correct" a
1618
/// diagnostic.
1719
public enum FixIt: Codable, CustomDebugStringConvertible {

‎Sources/SwiftSyntax/DiagnosticConsumer.swift renamed to ‎Sources/SwiftSyntaxParser/DiagnosticConsumer.swift

File renamed without changes.

‎Sources/SwiftSyntax/DiagnosticEngine.swift renamed to ‎Sources/SwiftSyntaxParser/DiagnosticEngine.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
//===----------------------------------------------------------------------===//
1515

1616
import Foundation
17+
import SwiftSyntax
1718

1819
/// The DiagnosticEngine allows Swift tools to emit diagnostics.
1920
public class DiagnosticEngine {

‎Sources/SwiftSyntax/JSONDiagnosticConsumer.swift renamed to ‎Sources/SwiftSyntaxParser/JSONDiagnosticConsumer.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
//===----------------------------------------------------------------------===//
1515

1616
import Foundation
17+
import SwiftSyntax
1718

1819
public final class JSONDiagnosticConsumer: DiagnosticConsumer {
1920
/// Enumerates the possible places this consumer might output diagnostics.

‎Sources/SwiftSyntaxParser/NodeDeclarationHash.swift.gyb

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
%{
2+
# -*- mode: Swift -*-
3+
from gyb_syntax_support import calculate_node_hash
4+
# Ignore the following admonition it applies to the resulting .swift file only
5+
}%
6+
//// Automatically Generated From NodeDeclarationHash.swift.gyb.
7+
//// Do Not Edit Directly!
8+
//===--------------------- NodeDeclarationHash.swift ----------------------===//
9+
//
10+
// This source file is part of the Swift.org open source project
11+
//
12+
// Copyright (c) 2014 - 2021 Apple Inc. and the Swift project authors
13+
// Licensed under Apache License v2.0 with Runtime Library Exception
14+
//
15+
// See https://swift.org/LICENSE.txt for license information
16+
// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
17+
//
18+
//===----------------------------------------------------------------------===//
19+
20+
@_implementationOnly import _InternalSwiftSyntaxParser
21+
22+
extension SyntaxParser {
23+
static func verifyNodeDeclarationHash() -> Bool {
24+
return String(cString: swiftparse_syntax_structure_versioning_identifier()!) ==
25+
"${calculate_node_hash()}"
26+
}
27+
}

‎Sources/SwiftSyntax/PrintingDiagnosticConsumer.swift renamed to ‎Sources/SwiftSyntaxParser/PrintingDiagnosticConsumer.swift

File renamed without changes.

0 commit comments

Comments
 (0)