Skip to content

Add literal twins of Bash loop values - #242

Merged
Aaronontheweb merged 5 commits into
devfrom
feat/literal-twin-projection
Oct 6, 2026
Merged

Aaronontheweb merged 5 commits into
devfrom
feat/literal-twin-projection

Conversation

@Aaronontheweb

@Aaronontheweb Aaronontheweb commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Why

Netclaw prompts for this real command each time, and it offers only "Once":

for n in 8250 8244 8223 8208 8187; do gh api -X PATCH repos/akkadotnet/akka.net/issues/$n -f milestone=157 >/dev/null && echo "moved $n"; done

The parser knows every value of $n. But the effective value of repos/akkadotnet/akka.net/issues/$n is Unknown, because an unquoted word can split or glob. So a consumer cannot judge the command as it judges gh api ... repos/akkadotnet/akka.net/issues/8250.

What changed

  • New BashParser.TryProjectLiteralTwins(string, out BashLiteralTwinProjection?) and the records BashLiteralTwinProjection, BashLiteralTwinCommand, BashLiteralTwin, and BashLiteralTwinWord.
  • For each occurrence whose changeable argument words (an expansion or a tilde word) all have a proved finite set of values, the projection writes one twin for each combination of values. A twin text is the text of that one command with each such word written as one literal word. The parser parses it in the exact directory of the command, with the launch facts that are live there, so the twin occurrence has the normal facts of the typed literal. BashLiteralTwin.WorkingDirectory names the directory.
  • Before: the gh api occurrence has Value: Unknown for the issue word. After: the projection gives 5 twins. Each twin has the command words gh api, Kind: Literal, IsPath: true, and Resolved: /work/repos/akkadotnet/akka.net/issues/8250 (and so on), the same facts as the typed literal.
  • for f in a.txt b.txt; do cat "$f"; done: the source has CommandWords: Unknown. Each twin has the command words cat and the path facts of cat a.txt or cat b.txt.
  • An occurrence gets no twins (the caller keeps today's behavior) when:
    • the occurrence is incomplete;
    • a verb word is not static (for example ~/bin/tool "$d");
    • a word has no complete finite value set ($(...), an unknown variable, a glob, "$@", a list over 32 values);
    • an unquoted value can split or glob: the authored value is used only under FreshNonInteractiveNoStartup, and only when each value is nonempty and has no space, tab, newline, *, ?, [, parenthesis (extglob), or backslash;
    • a redirect depends on a value (a here-string, an expanding heredoc, a file target);
    • one word maps to two arguments (--opt=$n), or the word has no span in the source (a bash -c child);
    • the combinations pass 32 for one occurrence, or the twins do not fit in the call budget (128 twin parses and 2,048 + 8 x source length characters of twin text, checked before any build);
    • the source has a backslash-newline, a carriage return, =~, or :~ anywhere (the lexer does not model them as Bash does; this covers a loop list item such as for f in if=~/x);
    • the directory of the occurrence is not exact, it has an assignment prefix, or it has a heredoc;
    • the source has ~ or HOME and no live launch HOME proves the home directory;
    • the twin parse does not match (other commands, incomplete, or the written word is not one literal word with the exact value).
  • Values of different words combine independently. The twin set can hold combinations that never run, so a check of every twin is stricter, not weaker.
  • The rules use general shell facts only. There is no program grammar.
  • Parse output does not change. Public API: additive.
  • Prepare 0.4.0-beta.23 (version, release notes, README, SPEC §2 and §3, consumer guide, plan). The lexer fix ships first as 0.4.0-beta.22. Do not tag. The owner publishes.

Tests

BashLiteralTwinProjectionTests (59 cases) and BashLiteralTwinBashDifferentialTests (16 sources):

  • The real gh api loop: each twin equals the facts of the typed literal command. The echo "moved $n" occurrence gets its own twins.
  • cat "$f" loop: twins give cat command words and the facts of cat a.txt and cat b.txt.
  • No twins: a $(...) list (quoted and unquoted), an unknown variable, a glob list, 33 values, 6×6 combinations, the total budget (the fifth occurrence of 32 twins), a program word that the shell can change, unquoted values with a space, tab, newline, *, ?, [, a backslash, or an empty value, bash "$p" ls (incomplete source, complete twins), cat '' (the twin word is not an exact literal), dependent redirects, --opt=$n, a bash -c child, a literal command.
  • Twins: quoted values with a space, a glob character, or a single quote keep their exact text; nested loops give each combination; git $v origin twins give git push origin and git fetch origin; a value -rf is an option in its twin; an unquoted word needs the fresh-process mode, and a quoted word does not.
  • Review round 2: list items with =~ and :~ (seven inputs) give no twins; the real-Bash test shows that Bash expands them; budget pins: 2,000 commands build exactly 856 characters of twin text, the nested loop builds 728, and a 4 KB command with 32 values builds nothing; an inexact directory and an assignment prefix give no twins.
  • Review round 1: the six line-continuation inputs, carriage returns, =~ and :~, home values with and without a live launch HOME, extglob values, and the parse budget (a failing last value stops at exactly 128 parses; a 2 KB source with 200 occurrences makes 0 twin parses).
  • Bash differential (Linux, GNU Bash at /bin/bash; the theory reports a skip with a reason on other hosts): each source and each twin runs in env-cleared Bash. Each program is a function that prints its argv, and PATH names a missing directory. Every argv of the real run equals the argv of a twin, and each twin run gives the argv of its twin occurrence. With the =~ guard off, the test fails on cp if=~/x "$n".
  • PublicApiSnapshotTests lock the new method and record shapes.

Evidence

  • dotnet test -c Release (rebased on 0.4.0-beta.22, 210adf1): 4930 passed, 0 failed. Build: 0 warnings. Header check passes.
  • twinprobe3 at the rebased head (tgt1 to tgt3 in all ten modes and the cd fuzz set): 13,074 sources and 4,312 twins run in Bash, the same counts as before the rebase. Zero ARGV-MISMATCH and zero CWD-MISMATCH.
  • Timing (reviewer probes, Release, warm; projection time includes its own Parse): 80 KB with 8,000 commands, Parse 5.5 s, projection 5.2 s, 2.3 GB (before: 23 s, 80 GB); 270-character nested loop, Parse 227 ms, projection 176 ms, 57 MB (before: 2.8 s, 4.7 GB).
  • Hand mutants (each built and run against the twin tests): 38 killed in round 2, over the full gate list. Earlier rounds: 40 killed. Round 1 added the carriage-return and continuation guards, each part of the home guard, each part of the =~ and :~ guard, both parentheses, the parse budget, the character budget, and the count of failed parses. The round-0 mutants drop one gate each: incomplete source, the redirect gate, the heredoc and here-string checks, the static-word rule, the program-word rule, the inline pair, the empty set, the authored fallback, the no-change shortcut, the fresh-process check, the empty value, each of the 7 split or glob characters, the combination limit, the total budget, the empty word list, the role rule, and the combined twin kind and value check.
  • Equivalent mutants (survive, because another gate rejects the same input): the redirect IsComplete check and the exact file target (SST already marks such an occurrence incomplete), the heredoc rule (the twin text cannot parse without the body), the span text check (the null-span check rejects a bash -c child), and the single twin self-checks (one command, completeness, directory, kept raw text). The self-checks are invariants. No input that I found fails only one of them.

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review of the literal twin projection. I ran a probe against the PR head (f850f6a) and compared each result with env -i HOME=/home/agent bash --norc --noprofile -c (GNU Bash 5.2.21), with printf '%s\037' as the program.

What held:

  • The rewrite and quoting (Spell) were correct for every value that I tried: $(id), backticks, quotes, \, ;, |, &, #, {, }, ~, =, -rf, empty, non-ASCII, $'...' values, and values inside $(...) and heredoc bodies.
  • Changes to IFS, glob options, variable attributes (declare -i/-u/-n), loop-variable writes, functions, eval, and most parameter operators make the source unparseable in the fresh mode, so they give no twins.
  • Subshell scope (| cat, ( ), &, $( )), skipped loops, break/continue, a program word from a value, --opt=$n, and bash -c children gave no twins or a correct superset.
  • I found no program-specific grammar. Parse does not change.

What did not hold: the twins inherit wrong Parse value facts. The self-check compares SST with SST, so it cannot catch them. See the inline comments. The high finding (line continuation inside an expansion) lets a twin show rm -rf builddir/ while Bash runs rm -rf /.

}

if (!TryGetArgument(occurrence, element, out var argument) ||
!TryGetProvedValues(argument, options, out var values) ||

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: high. Verified with a probe and with env -i bash --norc --noprofile -c (GNU Bash 5.2.21).

The lexer does not remove a line continuation (\ + newline) before it reads an expansion. The value facts are then wrong, and the twin turns the wrong value into a "literal" word.

Inputs (each \ below is followed by a real newline):

Source SST twin Bash runs
for n in build; do rm -rf "$n\⏎dir/"; done rm -rf builddir/ rm -rf / ($ndir is unset)
printf '<%s>' "$\⏎(id -u)" printf '<%s>' '$(id -u)' runs id -u, prints <1000>
cat "$\⏎HOME/.ssh/id_rsa" cat '$HOME/.ssh/id_rsa' (a relative path) cat /home/agent/.ssh/id_rsa
x=/etc/hostname; cat $\⏎x cat '$x' cat /etc/hostname
for n in a; do printf %s "$\⏎{n}"; done '${n}' a
printf %s x$\⏎'y' x$y xy ($'y' is ANSI-C quoting)

In the $(id -u) case, Parse also gives one complete occurrence and no id occurrence. So a hidden command runs.

The root cause is in Parse, not in this file, and it is older than this PR. But the twin makes it worse: the twin has Kind: Literal, an exact value, and a relative path. A consumer that trusts twins as typed literals will approve rm -rf builddir/ while Bash runs rm -rf /. The self-check cannot catch this, because the source parse and the twin parse use the same lexer.

Suggested fix:

  1. In the Bash lexer, remove \+newline outside single quotes (unquoted and inside double quotes) before it reads $, ${, $(, $((, $', and a parameter name. Bash does this in the input layer.
  2. Until then, give no twins when an element Raw has \+newline outside single quotes.
  3. Add a differential test: run the source and each twin in env -i bash with printf '%s\037' as the program, and compare the argv sets.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 153f6b5 with a guard. A source that has a backslash-newline anywhere gets no twins. The check reads the full source, not only the word, because a value can come from another part of the source, such as a loop list. A separate PR fixes the lexer for 0.4.0-beta.22. This PR keeps the guard after that fix and moves to 0.4.0-beta.23. Tests: all six of your inputs, plus a continuation in a loop list, give no twins. A new Linux test runs each source and each twin in GNU Bash (functions that print argv, PATH set to a missing directory) and compares the argv.

// the twin keeps it as authored. A verb word has no analyzed
// argument, so a program word or a verb word that the shell can
// change gives no twins.
if (ShellCommandWordProjection.IsStaticBashWord(element.Raw))

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: medium. Verified with a probe and with env -i bash (5.2.21, default non-POSIX mode).

IsStaticBashWord marks name=~/x as static, because ScanBash sees ~ only at index 0. But Bash also does tilde expansion after = and after : in an argument word that looks like an assignment. The twin keeps the word as authored, and its value is wrong.

Input: for n in out.txt; do dd if=~/.ssh/id_rsa of="$n"; done

  • SST twin: dd if=~/.ssh/id_rsa of=out.txt. The element has Exact { Value = if=~/.ssh/id_rsa }.
  • Bash runs: dd if=/home/agent/.ssh/id_rsa of=out.txt. Bash reports dd: failed to open '/home/agent/.ssh/id_rsa'.

Same result for of=~/.bashrc and PATH=x:~/bin (Bash gives PATH=x:/home/agent/bin). --x=~/y does not expand, because --x is not a name.

This is also a Parse fact bug that is older than this PR. But the doc comment of IsStaticBashWord says "the shell cannot change" the word, and the twin contract says each changeable word is rewritten. Neither is true for this word.

Suggested fix: in ScanBash, when the unquoted prefix matches [A-Za-z_][A-Za-z0-9_]*=, treat an unquoted ~ directly after the = or after an unquoted : as a tilde (shape Expanded). Then fix the Parse value of such a word. Under POSIX mode Bash does not expand it, so the rule depends on the launch mode.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 153f6b5 with a guard. An occurrence with =~ or :~ in any word gets no twins. The check is lexical and stricter than Bash: it also rejects quoted text. I did not change ScanBash or the Parse values, so no comparison record changes. Tests: dd if=~/.ssh/id_rsa of="$n", env PATH=x:~/bin "$n", and a redirect > of=~/x. I also checked that the new Bash differential test fails on cp if=~/x "$n" when the guard is off.

}

var twins = new List<BashLiteralTwin>(count);
if (!TryBuildTwins(parsed, parser, index, words, twins))

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: medium (availability). Verified with a timing probe (Release build, net10.0).

The total budget of 128 counts only twins that succeed. An occurrence that fails the self-check on its last combination costs up to 32 full parses of the whole source, and the budget does not go down. There is no limit on the number of such occurrences.

Input: for n in v0 v1 ... v30 ''; do cat "$n"; cat "$n"; ... done (the '' value fails the twin match for cat).

Occurrences Source size Parse TryProjectLiteralTwins
50 634 bytes 133 ms 15.4 s
200 2.1 KB not measured alone about 290 s (whole probe run)

The success path also costs 128 full parses: the same 50-occurrence source with a good last value takes 2.0 s. A model or a user can send such a command, and the approval path then blocks.

Suggested fix: count each twin parse against one global parse budget (success or failure). Stop the projection when the budget is used. Do the cheap checks for all occurrences first. Also think about a source-length limit, because each parse costs O(source).

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 153f6b5. Every twin parse counts against one budget for the call, also a parse that fails: at most 128 parses and 32,768 source characters in total. The cheap checks run for all occurrences first. An occurrence whose parses do not fit in the rest of the budget gets no twins and costs no parse. A 2 KB source gets at most 16 twin parses. Tests pin the counts through an internal overload: your failing-last-value input stops at exactly 128 parses, and a 2 KB source with 200 occurrences makes 0 twin parses.

var choice = new int[words.Count];
while (true)
{
var source = Render(parsed.Source, words, choice);

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: low. Verified with a probe and with env -i bash.

The Bash lexer treats a bare carriage return (\r) as a command separator. Bash treats it as an ordinary word character. So a twin from a source with CRLF line ends has a value that Bash does not pass.

  • x=/etc + CRLF + printf '%s' "$x": twin value /etc. Bash passes /etc\r.
  • for n in a; do printf '%s' "$n" + \r + id; done: SST gives two occurrences (printf ... a and id). Bash runs one command with the argument a\rid.

The risk is low. The extra text joins an existing word, so it cannot add an option or a command, and SST stays stricter on the command count. But the twin contract says "the facts of the typed literal", and the argv is not the same. A Windows client can send CRLF text.

Suggested fix: make \r an ordinary word character in the Bash lexer, as Bash does. Until then, give no twins when the source has a \r.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 153f6b5 with a guard. A source that has a carriage return anywhere gets no twins. Tests: your two inputs and a CRLF line end give no twins.

domain = argument.AuthoredValue;
}

values = domain switch

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: low. Verified with a probe.

A tilde word gets a twin from HomeDirectory in every launch mode. With InitialStateMode = Unknown, cat ~/x gives the twin cat /home/agent/x. But Bash expands ~ from $HOME, and under FreshNonInteractiveNoStartup SST keeps "$HOME" as Unknown. So the twin treats a caller assumption as a proved value. The twin also removes the ~, so a consumer can no longer see that the value came from HomeDirectory.

The old Resolved path used the same assumption, so this does not add a new path fact. But the twin has Kind: Literal, and that is a stronger claim.

Suggested fix: give a tilde twin only when the launch contract proves HOME. Or state in the SPEC and the consumer guide that a tilde twin uses HomeDirectory as an assumption that the caller must make true.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 153f6b5. When the source has ~ or HOME, an occurrence gets twins only when the launch facts give a live HOME for that occurrence. Without one, the home directory is the HomeDirectory assumption, so the occurrence gets no twins. Test: cat ~/x "$n" and cat "$HOME/x" "$n" give no twins without launch facts, and give /home/agent/... twins with a live launch HOME.

/// when the caller proves the fresh-process contract: that contract removes
/// an inherited <c>IFS</c>, and the parser rejects source that changes
/// <c>IFS</c> or a glob option. Bash then splits only on space, tab, and
/// newline, and it globs only on <c>*</c>, <c>?</c>, and <c>[</c>. A value

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: low. Source read and a check of default Bash options. I did not test a Bash build that has extglob on by default.

The claim "it globs only on *, ?, and [" is true only when extglob is off. With shopt -s extglob, Bash globs an unquoted value such as @(a.txt) or +(a).txt that has none of these characters. I verified this in Bash 5.2.21: the value @(a.txt) became a.txt. SST rejects source that runs shopt -s extglob, so the probe gave no twins, which is correct. But a Bash build with --enable-extended-glob-default has extglob on at startup. The fresh contract names "behavior-changing option flags", not compile-time defaults.

Suggested fix: also reject an unquoted value that has ( directly after @, +, or !. Or state in the fresh contract that Bash must have the default extglob setting. This also applies to the same text in SPEC.md.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 153f6b5. An unquoted value with ( or ) gives no twins. With * and ? already rejected, this covers @(, !(, +(, *(, and ?(. The code comment, the SPEC, and the release notes now name the extglob build default. Tests: @(a), +(a).txt, !(a), a)b, and a(b.

Comment thread docs/CONSUMER_GUIDE.md Outdated
This makes the check stricter, never weaker.
- `twin.Source` is the full source with one occurrence rewritten.
`SourceOccurrenceIndex` is the index of the occurrence in a parse of that
source, so you can run your own analysis on `twin.Source`.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: low (API clarity). Source read.

Some points that a consumer can easily get wrong:

  1. The shell never runs twin.Source. Add a warning: "Do not run twin.Source. Bash runs only the authored source."
  2. twin.Occurrence belongs to a different ParsedCommand. Only the occurrence at SourceOccurrenceIndex has the twin facts. The other occurrences in a parse of twin.Source keep the expansions, and their facts are not the facts of the source run. Say this.
  3. When the method returns false, projection is null, so the caller has no Parsed and must call Parse again. That doubles the cost (see the comment about the budget). Think about always returning the projection, with an empty Commands list when no occurrence has twins.
  4. Say that the twin facts are only as good as the Parse value facts. The self-check compares SST with SST, not with Bash (see the comment about line continuations).

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 153f6b5. The consumer guide now says: do not run twin.Source; only the occurrence at SourceOccurrenceIndex has the twin facts; a false return gives a null projection, so use your own Parse result; and twin facts are only as good as the Parse values, because the self-check compares the parser with itself. The SPEC says the same. I kept the null projection on a false return, as the owner decided.

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2 review of 153f6b5. I used the round 1 probe against this head and compared each result with env -i HOME=/home/agent bash --norc --noprofile -c (GNU Bash 5.2.21). I ran it in the fresh mode with and without a live launch HOME, and in the Unknown and IsolatedNonInteractive modes.

Round 1 fixes:

  • 1 (backslash-newline), 4 (\r), 6 (unquoted (/)), and 7 (consumer guide): closed. All round 1 inputs give no twins.
  • 3 (budget): closed for the round 1 input (15.4 s to 0.43 s). See the inline comment for the cost that is still outside the budget.
  • 5 (HOME): closed. Tilde and $HOME twins need a live launch HOME, and HOME=, unset HOME, export HOME, and for HOME in make the source unparseable.
  • 2 (=~/:~): not fully closed. A for list item still gets the tilde expansion. See the inline comment.

New probes that held (no twins, or a correct superset):

  • Brace expansion in a list item or in the word, $'...' items with \0, \u, \x, \c, and $"...".
  • ${#n}, ${!n}, ${n[0]}, $((n)), $((n=5)), ${n:=x}, let, read, printf -v, mapfile, getopts, export/local/readonly/declare/unset of the loop variable, eval, source.
  • trap ... DEBUG/RETURN/ERR, set -k/-a/-u/-B/-T, set -o posix, shopt options, GLOBIGNORE, enable -n, and aliases.
  • time, !, coproc, exec, command, builtin, and <(...)/>(...).
  • Values that the loop carries between passes, break, continue, break 2, exit, case fall-through, &&/||, pipelines, &, $(...), and while/until around a for.
  • About 50 special and dynamic variables ($_, $LINENO, $PWD, $RANDOM, $@, and others).
  • Values such as ), ', `, ]], esac, ;;, and a newline inside $(...), a heredoc body, case, [[ ]], and arithmetic.
  • 700 random list items and assignment values: 254 sources with twins, and no argv mismatch.

Bash differential test: it is not tautological. It compares the SST argv of each twin with the argv that real Bash passes, for the source and for each twin. The skip is explicit: BashTheoryAttribute gives a reason on a host that is not Linux or that has no /bin/bash. The test checks only occurrences whose program is unique in the source, and checkedCommands > 0 makes sure that one runs. The corpus has no source that takes a value from a list item with =~.

Version and docs: 0.4.0-beta.23 agrees across Directory.Build.props, README, RELEASE_NOTES, SPEC, the consumer guide, and the plan. The release notes say that Parse output equals 0.4.0-beta.22, but dev is still at beta.21 and I did not find an open beta.22 PR. After the beta.22 lexer fix merges, rebase and check the release notes and README again.

for (var index = 0; index < elements.Count; index++)
{
var element = elements[index];
if (HasAssignmentTilde(element.Raw))

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: medium. Verified with a probe and env -i HOME=/home/agent bash --norc --noprofile -c (GNU Bash 5.2.21). The parser has a live launch HOME=/home/agent.

The =~ / :~ guard reads only the words of the occurrence. Bash also expands a tilde after = or : in a for list item that looks like an assignment. The value then comes from the list, not from the occurrence, so this guard does not see it. The parser keeps the list item as plain text, and the twin quotes it as a literal.

Source SST twin value Bash passes
for f in if=~/.ssh/id_rsa; do dd "$f" of=out.txt; done dd 'if=~/.ssh/id_rsa' of=out.txt dd if=/home/agent/.ssh/id_rsa of=out.txt
for n in x=~/y; do printf %s "$n"; done x=~/y x=/home/agent/y
for n in x=~; do ... x=~ x=/home/agent
for n in a=b:~/y; do ... a=b:~/y a=b:/home/agent/y
for n in a b=~/y c; do ... b=~/y b=/home/agent/y
for n in x=~+; do ... x=~+ x= + the working directory
for n in a; do for m in x=~/y; do ... "$n" "$m"; done; done x=~/y x=/home/agent/y

Unquoted $n gives the same result. Without a live launch HOME, the ~ guard stops all of these. So the risk needs a live HOME, and that is the mode in which tilde twins are meant to work.

A consumer that judges the dd twin sees an input path ~/.ssh/id_rsa relative to the work directory. Bash reads the SSH key.

Suggested fix: check the full source, as the backslash-newline and \r guards do. For example, add || HasAssignmentTilde(parsed.Source) to the first check in TryProject. I tried this change locally: the probe then gives no twins for all the inputs above (before: 6 argv mismatches). The cost is that a source with [[ $x =~ re ]] also gets no twins, which is safe. Add a no-twin test for for n in x=~/y; do cat "$n"; done. Then change the text "in a word" to "anywhere in the source" in this remark, in SPEC.md (lines 1550 and 1596), in the consumer guide (line 1040), and in the release note.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in eab57d5. The =~ and :~ check now reads the full source, with the backslash-newline and carriage-return checks. A source with either text anywhere gets no twins, which also stops [[ $x =~ re ]]. Tests: all seven of your inputs and the unquoted for n in x=~/y; do cat $n; done give no twins with a live launch HOME. The real-Bash test now also runs list items with =~ and :~, and checks that Bash passes x=/home/agent/y while the parser keeps x=~/y. The SPEC, the consumer guide, and the release note now say "anywhere in the source".

var characters = 0;
foreach (var candidate in candidates)
{
var sources = RenderAll(parsed.Source, candidate.Words);

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: low (availability). Verified with a timing probe (Release build, net10.0, warm JIT).

The round 1 input is fixed: the 634-byte failing input now takes 0.43 s (before: 15.4 s).

But the budget check runs after RenderAll. RenderAll builds the source of every combination before the check, and the loop continues after the budget is used. So each occurrence that can never fit still costs combinations × source length in string work. With many occurrences, the cost grows with the square of the source size, and no twin parse runs.

Input: for n in v0 v1 ... v31; do cat "$n"; cat "$n"; ...; done with N occurrences.

N Source Parse TryProjectLiteralTwins Twin parses Allocated
2,000 20 KB 1.5 s 1.7 s 0 5.5 GB
4,000 40 KB 3.1 s 5.2 s 0 20.8 GB
8,000 80 KB 5.7 s 23.2 s 0 80.6 GB

The worst case that I found inside the budget: a 270-character source, for a in 0 ... 31; do x=$a; for b in 0 ... 31; do y=$x$b; z=$y$a; cat "$a"; cat "$a"; cat "$a"; cat "$a"; done; done. It makes 96 twin parses in 2.8 s and allocates 4.7 GB. Parse of the same source takes 32 ms. So one call can cost about 90 times Parse and stay inside the budget.

Suggested fix:

  1. Compute the length of each twin before you render it: source.Length plus, for each word, Spell(value).Length minus the span length. Check parseCount + count and the total length before RenderAll.
  2. Stop the loop (break) when the rest of the budget cannot hold one more twin parse.
  3. Think about a smaller parse budget, because a 270-character source reaches 2.8 s.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in eab57d5, with two changes. First, the length of each twin text is computed before any build, and the budget check runs first. A command that does not fit gets no twins and costs no build and no parse. Second, a twin text is now the one command, not the full source. The parser parses it in the exact directory of the command, with the launch facts that are live there, so a twin parse costs about its own length. The budget is 128 twin parses and 2,048 + 8 x (source length) characters of twin text. Your probes, with the projection time including its own Parse: 80 KB (8,000 commands) Parse 5.5 s, projection 5.2 s, 2.3 GB; the 270-character nested loop Parse 227 ms, projection 176 ms, 57 MB. Tests pin the built characters: for 2,000 commands, exactly 4 commands and 856 characters of twin text; for the nested loop, 728 characters; a 4 KB command with 32 values builds nothing. New gates from the slice design: a command with an inexact directory, an assignment prefix, or a heredoc gets no twins. BashLiteralTwin gets a WorkingDirectory property.

@Aaronontheweb

Copy link
Copy Markdown
Owner Author

Verification round fixed in c657124.

  • Unknown initial state: the projection now gives no twins under BashInitialStateMode.Unknown. In that mode Parse reports -o"$n" as the exact value -o$n, but Bash passes -oa. Parse is not changed in this PR. Tests: tool '$n' -o"$n", git -C"$n" status, -xo"$n", -o"${n}", and -o"$n"x. The same words under the fresh-process mode are in the real-Bash comparison and match Bash.
  • Home directory from cd: without a live launch HOME, a cd without an operand goes to the HomeDirectory assumption in the parse, but Bash uses its real HOME. The projection now parses the source once more with another HomeDirectory. A command whose exact directory changes gets no twins. The check does not read the cd forms, so it covers cd, cd -L, cd --, builtin cd, command cd, cd >/dev/null, if cd; then, and a cd in a loop. A directory below the home directory that does not come from HOME (cd /home/agent/repo) keeps its twins. When the launch facts export HOME but it is not live at the command, the command gets no twins. Cost: at most one more parse for each call.
  • The real-Bash comparison now also checks the working directory. With a live launch HOME set to a real directory, each cd case gives the directory that Bash uses. Without it, the parser claims the assumed home, Bash runs elsewhere, and the projection gives no twins.
  • Fixed the stale comment of TryProjectLiteralTwins.
  • twinprobe3 runner (a copy built at this head): tgt1 to tgt3 in all ten modes and the cd fuzz set, 13,074 sources and 4,312 twins run in Bash. Zero ARGV-MISMATCH and zero CWD-MISMATCH. The 15 other findings are harness artifacts: a twin directory that does not exist on disk (/home/agent, /work/nope), and one > ~/out redirect into that missing home.
  • Full suite: 4743 passed. Hand mutants: 43 killed. The new survivor is the exported-HOME check. I found no parseable source in which a launch HOME is not live. The check stays, because without it the second parser would throw.

Add BashParser.TryProjectLiteralTwins. For each occurrence whose
changeable argument words have a proved finite set of values, the
projection writes one literal twin for each combination of values. A
twin is the full source with each such word written as one literal
word. The parser parses it again, so the twin occurrence has the facts
of the typed literal: command words, path facts, and expansion facts.

An occurrence gets no twins when it is incomplete, when a verb word is
not static, when a word has no complete finite value set, when an
unquoted value can split or glob, when a redirect depends on a value,
or when the combinations pass the limit (32 for one occurrence, 128 in
total). An unquoted word uses its authored values only under
FreshNonInteractiveNoStartup, and only when each value is nonempty and
has no space, tab, newline, glob character, or backslash.

The rules use general shell facts only. A twin grants no authority.
Parse output does not change. Prepare 0.4.0-beta.22.
Give no twins for text that the lexer does not model as Bash does:

- a source with a backslash-newline or a carriage return;
- an occurrence with `=~` or `:~` in a word;
- an occurrence in a source with `~` or `HOME` when no live launch
  HOME proves the home directory;
- an unquoted value with a parenthesis, which an extglob pattern needs.

Count every twin parse, also a failed one, against one budget for a
call: 128 parses and 32,768 source characters. An occurrence whose
parses do not fit gets no twins and costs no parse. Run the cheap
checks for all occurrences first.

Add a Linux test that runs each source and each twin in GNU Bash and
compares the argv. Document the consumer warnings. Move the release
to 0.4.0-beta.23, after the beta.22 lexer fix.
Check `=~` and `:~` in the full source, as the backslash-newline and
carriage-return guards do. Bash expands a tilde in a loop list item
such as `for f in if=~/x`, and the parser keeps it as text.

Parse each twin text as one command, not as the full source. The twin
text is the authored text from the first to the last word of the
command. The parser parses it in the exact directory of the command,
with the launch facts that are live there. A twin parse then costs
about its own length. Compute the length of every twin text before any
build, and check the budget first: 128 twin parses and 2,048 + 8 x
(source length) characters. The projection now costs about one Parse.
On an 80 KB source with 8,000 commands it takes about as long as Parse
(before: 23 s and 80 GB).

A command gets no twins when its directory is not exact, when it has an
assignment prefix, or when it has a heredoc. BashLiteralTwin gets a
WorkingDirectory property.
Under BashInitialStateMode.Unknown, the parser can report an exact
value for a word that Bash expands: `-o"$n"` gives `-o$n`, but Bash
passes `-oa`. The projection now gives no twins in that mode. Parse is
not changed here.

A cd without an operand goes to $HOME. Without a live launch HOME, the
parser takes that directory from HomeDirectory, but Bash uses its real
HOME. The projection now parses the source once more with another
HomeDirectory. A command whose exact directory changes gets no twins.
The check does not read the cd forms, so it also covers `cd --`,
`builtin cd`, `command cd`, and `if cd; then`.

Add the inputs as tests and to the real-Bash comparison, which now
also compares the working directory. Fix the stale parser comment.
Add the 0.4.0-beta.23 release notes above 0.4.0-beta.22, and set the
README version line to 0.4.0-beta.23. Move the beta.22 plan section to
Completed.

Since 0.4.0-beta.22, Parse gives Unknown or the Bash value for a loop
list item with `=~` or `:~`, never the text. Update the real-Bash test
to that rule. Keep the line-text guards as defense in depth and say so
in the code, the SPEC, and the consumer guide. Add a carriage return in
single quotes as a guard test, because a carriage return outside quotes
is now unparseable.
@Aaronontheweb
Aaronontheweb force-pushed the feat/literal-twin-projection branch 2 times, most recently from c657124 to 7533b46 Compare October 6, 2026 18:30
@Aaronontheweb
Aaronontheweb marked this pull request as ready for review October 6, 2026 18:34
@Aaronontheweb
Aaronontheweb merged commit aaea78d into dev Oct 6, 2026
2 checks passed
@Aaronontheweb
Aaronontheweb deleted the feat/literal-twin-projection branch October 6, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant