Skip to content

Commit 5d19507

Browse files
jmillikinjmillikin
committed
Document C string literal tokens.
1 parent 8947db0 commit 5d19507

File tree

2 files changed

+90
-8
lines changed

2 files changed

+90
-8
lines changed

‎src/expressions/literal-expr.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88
>    | [BYTE_LITERAL]\
99
>    | [BYTE_STRING_LITERAL]\
1010
>    | [RAW_BYTE_STRING_LITERAL]\
11+
>    | [C_STRING_LITERAL]\
12+
>    | [RAW_C_STRING_LITERAL]\
1113
>    | [INTEGER_LITERAL]\
1214
>    | [FLOAT_LITERAL]\
1315
>    | `true` | `false`
@@ -48,6 +50,12 @@ A string literal expression consists of a single [BYTE_STRING_LITERAL] or [RAW_B
4850

4951
> **Note**: This section is incomplete.
5052
53+
## C string literal expressions
54+
55+
A C string literal expression consists of a single [C_STRING_LITERAL] or [RAW_C_STRING_LITERAL] token.
56+
57+
> **Note**: This section is incomplete.
58+
5159
## Integer literal expressions
5260

5361
An integer literal expression consists of a single [INTEGER_LITERAL] token.
@@ -182,5 +190,7 @@ The expression's type is the primitive [boolean type], and its value is:
182190
[BYTE_LITERAL]: ../tokens.md#byte-literals
183191
[BYTE_STRING_LITERAL]: ../tokens.md#byte-string-literals
184192
[RAW_BYTE_STRING_LITERAL]: ../tokens.md#raw-byte-string-literals
193+
[C_STRING_LITERAL]: ../tokens.md#c-string-literals
194+
[RAW_C_STRING_LITERAL]: ../tokens.md#raw-c-string-literals
185195
[INTEGER_LITERAL]: ../tokens.md#integer-literals
186196
[FLOAT_LITERAL]: ../tokens.md#floating-point-literals

‎src/tokens.md‎

Lines changed: 80 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -24,14 +24,16 @@ Literals are tokens used in [literal expressions].
2424

2525
#### Characters and strings
2626

27-
| | Example | `#` sets\* | Characters | Escapes |
28-
|----------------------------------------------|-----------------|------------|-------------|---------------------|
29-
| [Character](#character-literals) | `'H'` | 0 | All Unicode | [Quote](#quote-escapes) & [ASCII](#ascii-escapes) & [Unicode](#unicode-escapes) |
30-
| [String](#string-literals) | `"hello"` | 0 | All Unicode | [Quote](#quote-escapes) & [ASCII](#ascii-escapes) & [Unicode](#unicode-escapes) |
31-
| [Raw string](#raw-string-literals) | `r#"hello"#` | <256 | All Unicode | `N/A` |
32-
| [Byte](#byte-literals) | `b'H'` | 0 | All ASCII | [Quote](#quote-escapes) & [Byte](#byte-escapes) |
33-
| [Byte string](#byte-string-literals) | `b"hello"` | 0 | All ASCII | [Quote](#quote-escapes) & [Byte](#byte-escapes) |
34-
| [Raw byte string](#raw-byte-string-literals) | `br#"hello"#` | <256 | All ASCII | `N/A` |
27+
| | Example | `#` sets\* | Characters | Escapes |
28+
|----------------------------------------------|-----------------|------------|-----------------|---------------------|
29+
| [Character](#character-literals) | `'H'` | 0 | All Unicode | [Quote](#quote-escapes) & [ASCII](#ascii-escapes) & [Unicode](#unicode-escapes) |
30+
| [String](#string-literals) | `"hello"` | 0 | All Unicode | [Quote](#quote-escapes) & [ASCII](#ascii-escapes) & [Unicode](#unicode-escapes) |
31+
| [Raw string](#raw-string-literals) | `r#"hello"#` | <256 | All Unicode | `N/A` |
32+
| [Byte](#byte-literals) | `b'H'` | 0 | All ASCII | [Quote](#quote-escapes) & [Byte](#byte-escapes) |
33+
| [Byte string](#byte-string-literals) | `b"hello"` | 0 | All ASCII | [Quote](#quote-escapes) & [Byte](#byte-escapes) |
34+
| [Raw byte string](#raw-byte-string-literals) | `br#"hello"#` | <256 | All ASCII | `N/A` |
35+
| [C string](#c-string-literals) | `c"hello"` | 0 | non-`NUL` ASCII | [Quote](#quote-escapes) & [Byte](#byte-escapes) |
36+
| [Raw C string](#raw-c-string-literals) | `cr#"hello"#` | <256 | non-`NUL` ASCII | `N/A` |
3537

3638
\* The number of `#`s on each side of the same literal must be equivalent.
3739

@@ -328,6 +330,76 @@ b"\x52"; b"R"; br"R"; // R
328330
b"\\x52"; br"\x52"; // \x52
329331
```
330332

333+
### C string and raw C string literals
334+
335+
#### C string literals
336+
337+
> **<sup>Lexer</sup>**\
338+
> C_STRING_LITERAL :\
339+
> &nbsp;&nbsp; `c"` ( ASCII_FOR_C_STRING | BYTE_ESCAPE | STRING_CONTINUE )<sup>\*</sup> `"` SUFFIX<sup>?</sup>
340+
>
341+
> ASCII_FOR_C_STRING :\
342+
> &nbsp;&nbsp; _any non-NUL ASCII (i.e 0x01 to 0x7F), except_ `"`, `\` _and IsolatedCR_
343+
344+
A non-raw _C string literal_ is a sequence of ASCII characters and _escapes_,
345+
preceded by the characters `U+0063` (`c`) and `U+0022` (double-quote), and
346+
followed by the character `U+0022`. If the character `U+0022` is present within
347+
the literal, it must be _escaped_ by a preceding `U+005C` (`\`) character.
348+
Alternatively, a C string literal can be a _raw C string literal_, defined
349+
below. The type of a C string literal is `&core::ffi::CStr`.
350+
351+
Some additional _escapes_ are available in either C or non-raw C string
352+
literals. An escape starts with a `U+005C` (`\`) and continues with one of the
353+
following forms:
354+
355+
* A _byte escape_ escape starts with `U+0078` (`x`) and is followed by exactly
356+
two _hex digits_. It denotes the byte equal to the provided hex value. The
357+
byte escape sequence `\x00` is forbidden, as C strings may not contain `NUL`.
358+
* A _whitespace escape_ is one of the characters `U+006E` (`n`), `U+0072`
359+
(`r`), or `U+0074` (`t`), denoting the bytes values `0x0A` (ASCII LF),
360+
`0x0D` (ASCII CR) or `0x09` (ASCII HT) respectively.
361+
* The _backslash escape_ is the character `U+005C` (`\`) which must be
362+
escaped in order to denote its ASCII encoding `0x5C`.
363+
364+
#### Raw C string literals
365+
366+
> **<sup>Lexer</sup>**\
367+
> RAW_C_STRING_LITERAL :\
368+
> &nbsp;&nbsp; `cr` RAW_C_STRING_CONTENT SUFFIX<sup>?</sup>
369+
>
370+
> RAW_C_STRING_CONTENT :\
371+
> &nbsp;&nbsp; &nbsp;&nbsp; `"` ASCII_EXCEPT_NUL<sup>* (non-greedy)</sup> `"`\
372+
> &nbsp;&nbsp; | `#` RAW_C_STRING_CONTENT `#`
373+
>
374+
> ASCII_EXCEPT_NUL :\
375+
> &nbsp;&nbsp; _any non-NUL ASCII (i.e. 0x01 to 0x7F)_
376+
377+
Raw C string literals do not process any escapes. They start with the
378+
character `U+0063` (`c`), followed by `U+0072` (`r`), followed by fewer than 256
379+
of the character `U+0023` (`#`), and a `U+0022` (double-quote) character. The
380+
_raw string body_ can contain any sequence of non-`NUL` ASCII characters and is terminated
381+
only by another `U+0022` (double-quote) character, followed by the same number of
382+
`U+0023` (`#`) characters that preceded the opening `U+0022` (double-quote)
383+
character. A raw C string literal can not contain any non-ASCII byte.
384+
385+
All characters contained in the raw string body represent their ASCII encoding,
386+
the characters `U+0022` (double-quote) (except when followed by at least as
387+
many `U+0023` (`#`) characters as were used to start the raw string literal) or
388+
`U+005C` (`\`) do not have any special meaning.
389+
390+
Examples for C string literals:
391+
392+
```rust
393+
c"foo"; cr"foo"; // foo
394+
c"\"foo\""; cr#""foo""#; // "foo"
395+
396+
c"foo #\"# bar";
397+
cr##"foo #"# bar"##; // foo #"# bar
398+
399+
c"\x52"; c"R"; cr"R"; // R
400+
c"\\x52"; cr"\x52"; // \x52
401+
```
402+
331403
### Number literals
332404

333405
A _number literal_ is either an _integer literal_ or a _floating-point

0 commit comments

Comments
 (0)