Reproducer: on an interactive shell with the -H option on,

  $ v=foo
  $ print ${#v}

does not print anything (but should print "3").
The 'print' line also is not added to the history.

This bug was exposed by commit 41ee12a which enabled the history
comment character by default, setting it to '#' as on bash. When it
was disabled by default, this bug was rarely exposed.

The problem happens here:

src/cmd/ksh93/edit/hexpand.c
203: if(hc[2] && *cp == hc[2]) /* history comment designator, skip rest of line */
204: {
205: 	stakputc(*cp++);
206: 	stakputs(cp);
207: 	DONE();
208: }

The DONE() macro sets the HIST_ERROR bit flag:

src/cmd/ksh93/edit/hexpand.c
45: #define	DONE()	{flag |= HIST_ERROR; cp = 0; stakseek(0); goto done;}

For the history comment character that is clearly wrong, as no
error has occurred.

There is another problem. The documentation I added for history
expansion states this bit, which is based on bash's behaviour:

    If a word on a command line begins with the history comment
    character #, history expansion is ignored for the rest of that
    line.

With an expansion like ${#v}, the word does not begin with # so
history expansion should not have parsed that as a comment
character. The intention was to act like bash.

src/cmd/ksh93/edit/hexpand.c:
- Split the DONE() macro into DONE and ERROROUT of which only the
  latter sets the HIST_ERROR bit flag. Change usage accordingly.
  Thix fixes the first problem.
- Don't make these new macros function-style macros (with ()) as
  they end in a goto, so that's a bit misleading.
- Add is_wordboundary() which makes a best-effort attempt to
  determine if the character following the specified character is
  considered to start a new word by shell grammar rules. Word
  boundary characters are whitespace and: |&;()`<>
- Only recognise the history comment character if is_wordbounary()
  returns true for the previous character. This fixes the second
  problem.

Thanks to @jghub for the bug report.
Resolves: #513

It worked on macOS, but failed on the github CI runner.

)

Commit 4ca578b accidentally left in half of an if statement that was
originally just a use of the SFMTXBEGIN2 macro (which is a no-op without
vt_threaded). As a result, the head builtin's -s flag broke in one of
the ksh2020 regression tests. Reproducer:
   # Note that head must be enabled in src/cmd/ksh93/data/builtins.c
   builtin head || exit 1
   cat > "/tmp/file1" <<EOF
This is line 1 in file1
This is line 2 in file1
This is line 3 in file1
This is line 4 in file1
This is line 5 in file1
This is line 6 in file1
This is line 7 in file1
This is line 8 in file1
This is line 9 in file1
This is line 10 in file1
This is line 11 in file1
This is line 12 in file1
EOF
   got=$(head -s 5 -c 18 "/tmp/file1")
   exp=$'is line 1 in file1'
   [[ "$got" = "$exp" ]] || print "'head -s' failed" "(expected $(printf %q "$exp"), got $(printf %q "$got"))"

Code change that caused this bug (note that since the if statement
wasn't completely removed, it now guards the for loop despite the
newline):
        if(fw)
-               SFMTXBEGIN2(fw, (Sfoff_t)0);

        for(n_move = 0; n != 0; )

I noticed this since I'm making a separate commit to backport more of
the ksh2020 regression tests.

src/lib/libast/sfio/sfmove.c:
- Remove the incomplete if statement to fix the -s flag.

src/cmd/ksh93/tests/b_head.sh:
- Backport the ksh2020 regression tests for the head builtin.

The dependency rule 'prev libshell.a archive' did not actually
cause shcomp to be rebuilt when something in libast.a changed.
Replace by 'prev ksh' to ensure shcomp is rebuilt/relinked whenever
ksh is.

This reverts commit 42fc5c4.
It somehow caused the following reproducer to fail. Reason unknown.

  typeset -i x
  function x.set { :; }
  x[0]=0
  unset x
  typeset -p x

Expected output: none

Actual output:

  typeset -a -i x=()

The 'x' variable fails to be unset.

With the .get and .getn discipline functions instead of .set, the
above reproducer still fails even after the revert, but that always
has failed, back to at least 93u+ 2012-08-01.

This is yet more evidence that discipline functions are a mess.
But a proper cleanup will require a lot of time and very thorough
testing, so it will have to wait until some future release.

src/cmd/ksh93/tests/variables.sh:
- In addition to the revert, add the above reproducer as a
  regression test for the .set, .unset and .append disciplines.

The referenced commit introduced the NIL (NULL) assignment in:

	stakdelete(slpold->slptr);
	slpold->slptr = NIL(Stak_t*);

First the stack is closed/freed with stakdelete() a.k.a.
stkclose(), then its pointer is reset. Looks correct, right?

Wrong: slpold may itself be in the allocated region that
slpold->slptr points to. That's because we're dealing with a linked
list of stacks, in which a pointer on each stack points to the next
stack. So there are scenarios in which, after the stakdelete()
call, dereferencing slpold is a use after free.

Most systems quietly tolerate this use after free. But, according
to @JohnoKing's testing, this bug was causing 23 crashes in the
regression tests after compiling ksh with AddressSanitizer enabled.

src/cmd/ksh93/sh/parse.c: sh_funstaks():
- Save the value of slpold->slptr and reset that pointer before
  calling stakdelete() a.k.a. stkclose().

Resolves: #517

This macro expansion in lex.c may assign -1 to n if EOF is reached:

1178:	fcgetc(n);

As a result, n may be -1 when this code is reached:

1190:	if(sh_isoption(SH_BRACEEXPAND) && c==LBRACE && !assignment
	&& state[n]!=S_BREAK

'state[n]' is a buffer overflow if n==-1.

src/cmd/ksh93/sh/lex.c: sh_lex(): case S_BRACE:
- Apart from the buffer overflow, if n<=0, none of the code
  following fcget(n) does anything until 'break' on line 1199 is
  reached. So, if fcget(n) yields <=0, just break. This allows some
  code simplification.

Progresses: #518

The ASan crash in basic.sh when sourcing multiple files is caused by
a bug that is similar to the crash fixed in f24040e. This is the
trace for the regression test crash (note that in order to see the
trace, the 2>/dev/null redirect must be disabled):

==1899388==ERROR: AddressSanitizer: heap-use-after-free on address 0x6150000005b0 at pc 0x55a5e3f9432a bp 0x7ffeb91ea110 sp 0x7ffeb91ea100
WRITE of size 8 at 0x6150000005b0 thread T0
    #0 0x55a5e3f94329 in funct /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:967
    #1 0x55a5e3f96f77 in item /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:1349
    #2 0x55a5e3f90c9f in term /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:642
    #3 0x55a5e3f90ac1 in list /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:613
    #4 0x55a5e3f90845 in sh_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:561
    #5 0x55a5e3f909e0 in sh_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:586
    #6 0x55a5e3f8fd5e in sh_parse /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:438
    #7 0x55a5e3fc43c1 in sh_eval /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:635
    #8 0x55a5e4012172 in b_dot_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/bltins/misc.c:318
    #9 0x55a5e3fca3cb in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1254
    #10 0x55a5e3fd01d4 in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1932
    #11 0x55a5e3fc4544 in sh_eval /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:651
    #12 0x55a5e4012172 in b_dot_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/bltins/misc.c:318
    #13 0x55a5e3fca3cb in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1254
    #14 0x55a5e3ecc1cd in exfile /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/main.c:604
    #15 0x55a5e3ec9e7f in sh_main /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/main.c:369
    #16 0x55a5e3ec801d in main /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/pmain.c:41
    #17 0x7f637b4db2cf  (/usr/lib/libc.so.6+0x232cf)
    #18 0x7f637b4db389 in __libc_start_main (/usr/lib/libc.so.6+0x23389)
    #19 0x55a5e3ec7f24 in _start ../sysdeps/x86_64/start.S:115

Code in question:
https://github.com/ksh93/ksh/blob/8d57369b0cb39074437dd82924b604155e30e1e0/src/cmd/ksh93/sh/parse.c#L963-L968

To avoid any more similar crashes, all of the fixes introduced
in 69d37d5 that set slp->slptr to null have been improved with the
fix in f24040e.

The referenced commit made -b/--notify much better at notifying
multiple terminating background jobs, but it still occasionally
misses one if multiple jobs terminate all at once. This race
condition will hopefully be worked out at some point, but it's a
low-priority problem as the notification will appear at the next
prompt in any case, and we still have many worse bugs to fix.

But the associated pty regression test is failing fairly often on
the github runners in the meantime, so this disables it for now.

The commit that backported read -a did not add a case label for it
to read.c. This was under the assumption that AST optget(3) would
always convert -a to -A. However, that was only done for first use.

The cause is the short-form options caching mechanism in optget(3).
On the first run, the pre-caching result would be returned, but the
equivalent option (-a) would be cached as if it is its own option,
so on the second and subsequent runs, optget returned 'a' instead
of 'A'. This only happens if no long-form equivalent is present.

Reproducer:

  $ read -A foo <<< 'foo bar baz'
  $ unset foo
  $ read -a foo <<< 'foo bar baz'
  $ echo ${foo[0]}
  foo bar baz

Expected: foo

src/lib/libast/misc/optget.c,
src/lib/libast/misc/optlib.h:
- [by Martijn Dekker] Implement caching for short-option
  equivalents. If a short-form equivalent is found, instead of
  caching it as a separate option, cache the equivalent in a new
  equiv[] array. Check this when reading the cache and substitute
  the main option for the equivalent if one is cached.

src/lib/libcmd/cp.c:
- Fix cp -r/cp -R symlink handling. The -r and -R options sometimes
  ignored -P, -L and -H.
- The -r and -R options no longer follow symlinks by default.

src/cmd/ksh93/bltins/whence.c,
src/lib/libcmd/*.c:
- Remove case labels that are redundant now that the optget(3)
  caching bug is fixed.

src/cmd/ksh93/tests/libcmd.sh:
- Added. This is the new script for the /opt/ast/bin path-bound
  built-ins from libcmd. Other relevant tests are moved into here.

Co-authored-by: Martijn Dekker <martijn@inlv.org>

If a version of ksh was successfully compiled, then you would
probably want to use it instead of the default system ksh when
invoking 'bin/package use'.

bin/package, src/cmd/INIT/package.sh:
- Remove '32' and '64' arguments to the 'use' subcommand which
  never worked as documented, even in the 93u+ version.
- For the use subcommand, set SHELL to the compiled package's ksh,
  unless it is unavailable or overridden by the user.
- Update and tweak self-documentation.

In vi mode, due to a buffer overflow, <ESC>= as the first input
either produces an invalid completion or crashes the shell. The
easiest fix is to disable <ESC>= as well as <TAB> completion at
the start of the command line, as is already done in emacs mode.

src/cmd/ksh93/edit/vi.c:
- getline(): If cur_virt <= 0 in case '\t', beep and refuse tab
  completion.
- textmod(): Move an 'if' statement checking for INVALID cur_virt
  to include = mode along with * and \ mode.

Co-authored-by: Martijn Dekker <martijn@inlv.org>
Resolves: #520

@GabrielRavier writes:
> When int is the same as int32_t, starting ksh with
> HISTSIZE=2000000000 in the environment (or any number between
> 1073741824 and 2147483647) results in an infinite loop, because
> of this code:
>
>	if(cp=nv_getval(HISTSIZE))
>		maxlines = (unsigned)strtol(cp, (char**)0, 10);
>	else
>		maxlines = HIST_DFLT;
>	for(histmask=16;histmask <= maxlines; histmask <<=1 );
>
> which gets stuck infinitely because histmask <= maxlines will
> always be true (after 26 iterations, histmask == 1073741824,
> after 27 iterations, histmask == -2147483648 and after more than
> 28 iterations it's stuck at 0).

src/cmd/ksh93/edit/history.c: hist_init():
- Add bounds check for maxlines. HIST_MAX is the number of bytes at
  which the history file gets cleaned up, so the theoretical
  maximum number of lines (i.e.: each line being empty, consisting
  only of a terminating 0 byte) should also be HIST_MAX. That value
  is 16384 on a system where sizeof(int)==4.

Resolves: #522

There are still two contexts in which the unhelpful "alias +
builtin + path" completion occurs:

* After the start of an empty line
* After the start of a line, but before the first character

To stop completion from being attempted in these contexts, a loop
is added to vi.c and emacs.c that looks over the previous positions
(current position inclusive) and allows the completion process to
continue once it finds the first non-space character. The existing
placement restrictions remain in case the cursor arrives in a
negative position via a bug.

Co-authored-by: Martijn Dekker <martijn@inlv.org>

In vi mode, using 0 or ^ without moving from the start of the line
places cur_virt in position 0, which is left with a value of NULL.
This causes at least two bugs:

* Entering append mode results in the cursor moving in reverse,
  often with corrupted output.

* The /dev/fd/3 completion (named FIFO on FreeBSD) is reached with
  = and * completion.

Because it is unnecessary to use either 0 or ^ at the start of the
line (whether reached by ESC or otherwise), check that cur_virt is
greater than 0 and abort if the move would be ineffective.

src/cmd/ksh93/edit/vi.c:

- cntlmode(): Remove redundant '0' command handling; this caused
  the '0' command handling in mvcursor() to never be reached. It
  looks like this extra code was a workaround for a bug in ancient
  versions of ksh[*], but since at least 1995, getcount()
  special-cases an initial '0' and will not handle it as a repeat
  count parameter, which made this extra handling unnecessary.
  [*] #524 (comment)

- mvcursor(): For the '0' and '^' commands, fix the bug by returning
  ABORT if the cursor position (cur_virt) is 0 or INVALID.

Related: #520
Co-authored-by: Martijn Dekker <martijn@inlv.org>

This returns job_init() to a version quite close to the original
93u+ version. Upon further code analysis, really everything after
the init_savelist() call is only relevant to interactive shells, so
we can just return at that point for non-interactive shells as 93u+
did. All the script-only job control regression tests still pass.

Resolves: #320

This point release mainly fixes the following:
- A bug in history expansion (set -H) where any use of the history
  comment character caused processing to be aborted as if it were
  an invalid history expansion. Affected e.g. 'echo ${#v}'.
- A bug in command line options processing that caused short-form
  option equivalents on some built-in commands to be ignored after
  one use, e.g., the new read -a equivalent of read -A.
- Ksh freezing or using excessive memory if HISTSIZE is assigned a
  pathologically large value.
- A bug that caused ksh in the vi editor mode to crash or produce
  invalid completions if ESC = was used at the beginning of a line.