Remove string-based reference of tokens in CodeGeneration

[ahoppen]

CodeGeneration currently refers to tokens e.g. as IdentifierToken. That string-based reference isn’t very idiomatic Swift. What we should have instead is a Token enum of all the possible tokens in TokenSpec.swift, a computed property on that enum that returns the corresponding TokenSpec of that token. We could then use members of the Token enum to refer to the token kinds, instead of the string-based reference.

[ahoppen]

Tracked in Apple’s issue tracker as rdar://113237461

[Matejkob]

This issue sounds like really amazing improvement. I really like the idea and I can implement it with great pleasure. Just to clarify: we want to replace refers to every token with this new enum, not only SYNTAX_TOKENS?

[ahoppen]

Yes, exactly. For example

https://github.com/apple/swift-syntax/blob/250b431726c7b05309770caaa9989a4166ea6ba7/CodeGeneration/Sources/SyntaxSupport/AvailabilityNodes.swift#L33

should be something like

choices: [.token(tokenKind: .binaryOperator), .token(tokenKind: .identifier)],

And also just to clarify: This is a CodeGeneration-only change. The generated files should not change.