Skip to content

Commit dee42c1

Browse files
committed
Rewrite 'tokens' para...
1 parent 858dfdf commit dee42c1

File tree

1 file changed

+16
-10
lines changed

1 file changed

+16
-10
lines changed

src/macro-expansion.md

Lines changed: 16 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -26,15 +26,21 @@ macro_rules! printer {
2626
}
2727
```
2828

29-
`$mvar` is called a _metavariable_. Unlike normal variables, rather than binding
30-
to a value in a computation, a metavariable binds _at compile time_ to a tree of
31-
_tokens_. A _token_ zero or more symbols that together have some meaning. For
32-
example, in our example definition, `print`, `$mvar`, `=>`, `{` are all tokens
33-
(though that's not an exhaustive list). There are also other special tokens,
34-
such as `EOF`, which indicates that there are no more tokens. The process of
35-
producing a stream of tokens from the raw bytes of the source file is called
36-
_lexing_. For more information about _lexing_, see the [Parsing
37-
chapter][parsing] of this book.
29+
`$mvar` is called a _metavariable_. Unlike normal variables, rather than
30+
binding to a value in a computation, a metavariable binds _at compile time_ to
31+
a tree of _tokens_. A _token_ is a single "unit" of the grammar, such as an
32+
identifier (e.g., `foo`) or punctuation (e.g., `=>`). There are also other
33+
special tokens, such as `EOF`, which indicates that there are no more tokens.
34+
Token trees resulting from paired parentheses-like characters (`(`...`)`,
35+
`[`...`]`, and `{`...`}`) -- they include the open and close and all the tokens
36+
in between (we do require that parentheses-like characters be balanced). Having
37+
macro expansion operate on token streams rather than the raw bytes of a source
38+
file abstracts away a lot of complexity. The macro expander (and much of the
39+
rest of the compiler) doesn't really care that much about the exact line and
40+
column of some syntactic construct in the code; it cares about what constructs
41+
are used in the code. Using tokens allows us to care about _what_ without
42+
worrying about _where_. For more information about tokens, see the
43+
[Parsing][parsing] chapter of this book.
3844

3945
Whenever we refer to the "example _invocation_", we mean the following snippet:
4046

@@ -44,7 +50,7 @@ printer!(print foo); // Assume `foo` is a variable defined somewhere else...
4450

4551
The process of expanding the macro invocation into the syntax tree
4652
`println!("{}", foo)` and then expanding that into a call to `Display::fmt` is
47-
called _macro expansion_, it is the topic of this chapter.
53+
called _macro expansion_, and it is the topic of this chapter.
4854

4955
### The macro parser
5056

0 commit comments

Comments
 (0)