Fix Bash parsing that hides commands: continuations, CR, and word # (#243)

* Remove Bash line continuations before reading an expansion

Bash removes a backslash-newline pair before it splits tokens, except in
single quotes, $'...', a comment, and a heredoc body. The lexer read
"$\<newline>(cmd)" as literal text, so cmd ran with no occurrence.

Every lookahead that reads the second character of a multi-character
construct now skips line continuations through one helper
(BashLineContinuation): $, $(, ${, $name, $', operators, here-doc
operators, and descriptor targets. Later readers of authored text
(redirect operators, word-expansion facts, command words, option
spelling) use the same rule. A split $(( marker, a continuation in an
expanding heredoc body or delimiter, and $'...' with a continuation fail
closed.

* Add tests that kill the surviving continuation mutants

* Pin the substitution scan with a quoted inner substitution

* Remove a dead arithmetic-marker guard that a mutant exposed

* Prepare ShellSyntaxTree 0.4.0-beta.22

* Fail closed on a carriage return in Bash code

Bash reads CR as a word character and backslash + CR as an escaped CR.
The lexer read CR as a line end and backslash + CRLF as a continuation,
so 'echo a\<CR><LF>touch x' and 'echo a<CR># ; touch x' hid 'touch'.
A CR outside quotes, comments, and heredoc bodies now makes the source
unparseable, and so does backslash + CR in double quotes. Comments end
only at LF, and a heredoc line with a trailing CR is not its delimiter.

* Take a Bash inline option value from the decoded word

The value argument of '--name=value' came from the authored text after
'=' or, with an escape, from the whole word. 'curl --data=@p\.json' gave
'--data=@p.json' and 'curl --data=\a""' gave '--data=a'. The value is now
the part of the decoded word (or of each proved value) after the option
and its '='. Anything else is Unknown.

* Pin a CR comment in a substitution and computed inline values

* Document the carriage-return and inline-value changes

* Record the beta.22 comparison numbers

* Use LF in a multi-line Bash test source on every platform

* Fix the #243 review: word '#', reserved words, assignment tildes

- A '#' starts a comment only when it begins a new word. After a quoted
  part, $(...), or $((...)) it is word text, as in Bash, so
  'echo "a"# ; touch x' shows 'touch'.
- Reserved words, keywords, and literal checks read the spelling without
  line continuations, so 't\<LF>ime touch x' fails closed like 'time'.
- In an assignment-shaped word (name= or name+=), a '~' after the first
  '=' or after ':' expands from HOME, as Bash does outside POSIX mode.
  '~user', '~+', and a word whose start is not visible give Unknown. A
  here-string keeps the text.

* Add bash-verified tests for the #243 review findings

Includes a generated check: a '#' at every point of 13 seeds must match
the commands that Bash runs (logged, nothing real runs) or fail closed.
The Bash oracle helpers move to a shared BashOracle test class.

* Pin the tilde state after an escape, an expansion, or a continuation

* Pin a continuation before an unprovable assignment tilde

* Read (( and ;; across a line continuation

The ((...)) gate and the ;; case terminator compared raw offsets, so
'(\<LF>(ls))' parsed as two subshells that run 'ls'. Bash removes the
continuation and reads an arithmetic command, which fails closed here.
Found by checking that no continuation variant of an unparseable
source parses.

* Document the #243 review fixes

* Record the final beta.22 comparison numbers

* Expand an assignment-word tilde only in proved non-POSIX Bash

Round 2 of the #243 review. A '~' after '=' or ':' of an
assignment-shaped word now expands only in the submitted source (not a
decoded sh -c or bash -c child), in FreshNonInteractiveNoStartup mode,
with a launch-proved HOME. Everywhere else the value is Unknown. POSIX
mode and dash keep the text, and the other modes can inherit POSIX mode.
A subscripted name (a[0]=~/x) gives Unknown. A real assignment keeps its
leading tilde in every mode.

* Document the assignment-word tilde gate

* Record the round-2 comparison numbers

Author: Aaronontheweb

Directory.Build.props

@@ -8,7 +8,7 @@
     <Nullable>enable</Nullable>
     <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
     <VersionPrefix>0.4.0</VersionPrefix>
-    <VersionSuffix>beta.21</VersionSuffix>
+    <VersionSuffix>beta.22</VersionSuffix>
   </PropertyGroup>
   <PropertyGroup>
     <!-- Target framework matrix -->

IMPLEMENTATION_PLAN.md

@@ -10,11 +10,23 @@ priorities.
 
 ---
 
-## NOW (0.4.0-beta.21 restore the beta.19 verb-slot rule)
+## NOW (0.4.0-beta.22 Bash line continuations, security)
+
+- [x] Remove a Bash line continuation before the lexer reads the next character of an expansion or an operator (#243). `echo "$\⏎(touch /tmp/x)"` gives the `touch` occurrence. Fail closed on a split `$((` marker.
+- [x] Fail closed on a CR in Bash code and on `\` + CR (owner decision). Comments end only at LF.
+- [x] Take a Bash `--name=value` value from the decoded word (owner decision).
+- [x] Review fixes: `#` after a word part is word text; reserved words, `((`, and `;;` read across a continuation; a `~` after `=` or `:` of an assignment-shaped word expands from HOME.
+- [x] Pin each case against real Bash. Prove the general rule: a continuation at any point gives the same facts or fails closed. Run manual mutants and an output comparison against beta.21.
+- [ ] Merge the PR after Linux and Windows CI. Tag `0.4.0-beta.22` and verify NuGet publication.
+- [ ] Move the literal twins of #242 to 0.4.0-beta.23.
+
+## Completed (0.4.0-beta.21 restore the beta.19 verb-slot rule)
 
 - [x] Restore the beta.19 verb-slot rule of `CommandWords` (#240). The verb slot keeps a plain word after an option. Pin `docker --debug run`, `ilspycmd -t X /p/x.dll`, `git --no-pager push`, and `xargs -0 rm`.
 - [x] Prove that the output equals beta.19 over the beta.20 comparison inputs.
-- [ ] Merge the PR after Linux and Windows CI. Tag `0.4.0-beta.21` and verify NuGet publication.
+- [x] Merge PR #241 after Linux and Windows CI. Tag `0.4.0-beta.21` on merge
+      `d197ba35` and verify NuGet publication. Workflow run 37468677260
+      published the package and the GitHub prerelease.
 
 ## Completed (0.4.0-beta.20 option value in the verb slot)
 

README.md

@@ -13,10 +13,10 @@ Hand-rolled, AOT-trim friendly, zero native dependencies. Multi-targets
 
 ```bash
 # Current package
-dotnet add package ShellSyntaxTree --version 0.4.0-beta.21
+dotnet add package ShellSyntaxTree --version 0.4.0-beta.22
 ```
 
-Version `0.4.0-beta.21` includes the restored beta.19 command-word verb-slot rule, Bash ANSI-C quote decoding, per-argument pathname-expansion and field-splitting facts, bounded Bash arithmetic expansion, bounded loop `break` and `continue`, unresolved brace words, effective values and binding command words under the fresh-process mode, bounded Bash `export` and `set --`, unknown variable reads under the fresh-process mode, heredocs in command substitutions, Bash background lists, Bash control-flow statements, wider Bash assignment facts, pathname-expansion facts, launcher-proved environment facts, command-word facts, bounded Bash and PowerShell assignment facts,
+Version `0.4.0-beta.22` includes the Bash line-continuation, carriage-return, and word-`#` security fixes, exact Bash inline option values and assignment-word tildes, the restored beta.19 command-word verb-slot rule, Bash ANSI-C quote decoding, per-argument pathname-expansion and field-splitting facts, bounded Bash arithmetic expansion, bounded loop `break` and `continue`, unresolved brace words, effective values and binding command words under the fresh-process mode, bounded Bash `export` and `set --`, unknown variable reads under the fresh-process mode, heredocs in command substitutions, Bash background lists, Bash control-flow statements, wider Bash assignment facts, pathname-expansion facts, launcher-proved environment facts, command-word facts, bounded Bash and PowerShell assignment facts,
 bounded PowerShell collection facts, shell-neutral filesystem tree-access
 effects, and fail-closed approval diagnostics alongside the typed syntax tree,
 command-occurrence API, bounded authored-value analysis, audited filesystem and

RELEASE_NOTES.md

@@ -1,3 +1,37 @@
+#### 0.4.0-beta.22 2026-10-06 ####
+
+This prerelease fixes security bugs. A Bash line continuation, a carriage return, or a `#` after a quoted part could hide a command from `Parse` (#243). In the examples, `⏎` is a real newline and CR is a carriage return.
+
+## Fixed
+
+- Bash removes a line continuation (a backslash and a newline) before it splits the input into tokens. The lexer did not remove it before it read an expansion. In `echo "$\⏎(touch /tmp/x)"`, Bash runs `touch`, but `Parse` gave only the `echo` occurrence. In `x="$\⏎(touch /tmp/x)"`, `Parse` gave no occurrence. 0.4.0-beta.3 to 0.4.0-beta.21 have this bug.
+- The lexer now skips line continuations before it reads the next character of `$(`, `$((`, `${`, `$name`, `$'`, `$"`, the operators `&&`, `||`, `>>`, `<<`, `<<-`, `<<<`, `&>`, and `&>>`, a numeric descriptor, and a descriptor target (`>&\⏎2`). This applies in unquoted text, in double quotes, and in the substitution boundary scan. `BashLineContinuation` owns the rule.
+- Bash keeps a line continuation in single quotes, in `$'…'`, in a comment, and in a heredoc body with a quoted delimiter. These stay literal.
+- Values: `n=build; rm -rf "$n\⏎dir/"` gave the exact value `builddir/`. Bash reads the variable `ndir`, so it runs `rm -rf /`. The value is now unknown. `"$\⏎n/"` gives `build/`, `"${\⏎x}"` gives the value of `x`, and `x$\⏎'y'` gives `xy`.
+- A carriage return could hide a command. Bash reads CR as a word character and `\` + CR as an escaped CR. The lexer read CR as a line end and `\` + CRLF as a line continuation. `echo a\` + CRLF + `touch /tmp/x` and `echo a` + CR + `# ; touch /tmp/x` run `touch` in Bash, but `Parse` gave no `touch` occurrence. A comment now ends only at LF, and a heredoc line with a trailing CR is not its delimiter, as in Bash.
+- A `#` directly after a quoted part, `$'…'`, `$(…)`, or `$((…))` started a comment. Bash starts a comment only when `#` begins a word, so the `#` is word text. `echo "a"# ; touch x`, `ls 'a'#;touch x`, and `echo $(echo "a"# ; touch x)` run `touch` in Bash, but `Parse` gave no `touch` occurrence (0.4.0-beta.3 to 0.4.0-beta.21). The argument is now `a#`, and the next command is an occurrence.
+- A reserved word or keyword with a line continuation in it is read without the continuation. `t\⏎ime touch x`, `!\⏎ touch x`, `coproc\⏎ touch x`, and `{\⏎ touch x; }` gave the reserved word as the program; they now fail closed like the forms without the continuation. `f\⏎i` and `d\⏎one` are keywords, `(\⏎(` is an arithmetic command (unparseable), and `;\⏎;` ends a case item.
+- A `~` after `=` or `:` of an assignment-shaped word. `make PREFIX=~/x` gave `PREFIX=~/x`, but non-POSIX Bash passes `PREFIX=$HOME/x` (also `a=b:~/y`, `a+=~/x`, `a=~`). POSIX-mode Bash and dash keep the text. The parser now expands it only in the submitted source (not a decoded `sh -c` or `bash -c` child), in `FreshNonInteractiveNoStartup` mode, with a launch-proved HOME. In every other case, and for `~user`, `~+`, a `~` after a quoted part, or a subscripted name (`a[0]=~/x`), the value is Unknown. A quoted prefix, a later `=`, and a here-string keep the text, as in Bash.
+- The value argument of a Bash `--name=value` word is now the part after the first `=` of the decoded word that Bash passes. `curl --data=@p\.json` gave the value `--data=@p.json`, `curl --data=\a""` gave `--data=a`, and `curl --data="a b"` gave `"a b"` with the quotes. Bash passes `@p.json`, `a`, and `a b`. A value that the parser cannot take from the decoded word is Unknown.
+- Readers of authored text use the same rule: the redirect operator analysis (`>\⏎>` is an append), `MayPathnameExpand` and `MayFieldSplit` (`"$\⏎@"`, `{a.\⏎.c}`), `CommandWords`, and option names in the per-verb flag tables (`curl -\⏎o /tmp/out` binds the path). `Raw` keeps the exact source slice.
+
+## Fail closed
+
+- `$(\⏎(` and `$\⏎((`: Bash reads arithmetic, but the grammar reads only an exact `$((` marker. Before, the parser read a subshell that runs `1+2`.
+- As before: a line continuation in an expanding heredoc body or delimiter (Bash joins the body lines before it looks for the delimiter), and in `$'…'`.
+- New: a CR outside quotes, comments, and heredoc bodies, and `\` + CR outside single quotes, comments, and heredoc bodies. This includes CRLF line endings. Bash passes the CR to the program, so a CRLF script does not run as written.
+
+## Changed tests
+
+- Removed the tests that pinned the old CR rules: `command 3\` + CRLF + `>out` and `>&1` (a descriptor across `\` + CRLF), `cat ~\` + CRLF + `/x`, a CRLF heredoc body, `\` + CRLF before a comment in `$( )`, and CRLF as a statement separator. Bash does not read these sources the way the tests claimed. The LF forms stay pinned.
+- Corpus entries 166 and 288: the value argument of `curl --data=…` is now `@$HOME.json`, not the whole word.
+
+## Security and compatibility
+
+- Output comparison against 0.4.0-beta.21: 6,640 inputs, 123,456 records in two launch modes. 4,458 records change, in 213 inputs: 105 with a backslash before LF, 24 with a CR, 47 with a `--name=value` word that has an escape, a quote, or an expansion (only the value of the value argument changes), 9 with a `#` after a quoted part or `)`, and 28 with a `~` after `=` or `:`.
+- Public API: no change.
+- 0.4.0-beta.22 was planned for the literal twins of #242. That change moves to 0.4.0-beta.23.
+
 #### 0.4.0-beta.21 2026-10-06 ####
 
 This prerelease restores the 0.4.0-beta.19 verb-slot rule of `CommandOccurrence.CommandWords` (#240). It reverts the 0.4.0-beta.20 change (#237).