Multiple consecutive newlines in heredoc are converted to one

[m-iwanicki]

Bashly Version

Latest Docker Image

Description

Creating heredoc with multiple consecutive newlines (e.g. simulating multiple Enter/default choices) results in those newlines being trimmed to only one newline. This results in possibly hard to debug bugs.

Contents of bashly.yml

Reproduction Steps

bashly init --minimal
cat <<EOT >src/root_command.sh
fdisk "/dev/sdc" <<EOF
n
p
3


w
EOF
EOT
bashly generate

Compare contents of generated download script with src/root_command.sh

Actual Behavior

Only one newline instead of 2.

grep 'fdisk "/dev/sdc" <<EOF' download -B2 -A7

Output:

root_command() {
  # src/root_command.sh
  fdisk "/dev/sdc" <<EOF
n
p
3

w
EOF

Which results in hard to debug failure

Expected Behavior

The same amount of newlines as in script

grep 'fdisk "/dev/sdc" <<EOF' download -B2 -A7

Output:

root_command() {
  # src/root_command.sh
  fdisk "/dev/sdc" <<EOF
n
p
3


w
EOF

[DannyBen]

This is done as post processing. Any number of consecutive newlines are replaced with one, by design. This is to remove the excess newlines that may be generated by the chaining of the view templates.

Source code flow:

Script code string is built here:

https://github.com/DannyBen/bashly/blob/406a86a4fec59bd53f46b22db045e437345be4fb/lib/bashly/script/wrapper.rb#L26

Which calls String#lint, that is removing excess newlines:

https://github.com/DannyBen/bashly/blob/406a86a4fec59bd53f46b22db045e437345be4fb/lib/bashly/extensions/string.rb#L48

I am really beginning to dislike the heredoc feature, it has a lot of overhead.

We had a similar issue with indentation, that required quite a large overhead (new IndentationHelper class) to resolve.

I am not sure that I want to or can change this behavior, but I will let my blood pressure drop back to normal before deciding what to do.

[DannyBen]

So I implemented a fix in #645, which I am not crazy about - mainly due to the overhead it adds for a seemingly simple task of newline linting, but it seems to provide the desired result, and avoid compacting newlines in heredocs. I am still not 100% sure I want to merge it.

[m-iwanicki]

Maybe add visible section in documentation about possible differences in final script that you need to keep in mind e.g. newline trimming (it's also happening in multi-line string). I was quite surprised as I didn't see any mention of this behaviour.
My use case is actually easy to fix e.g.: printf "%s\n" n p 3 "" "" w | sudo fdisk "/dev/sdc" which might be more readable than heredoc, not sure about longer examples.

[DannyBen]

Maybe add visible section in documentation

There are many nuances in how bashly works, documenting everything will make it hard to navigate.
Besides, it is unlikely you would have seen this even if it was documented.

Nevertheless, perhaps a note about newline compaction in the generated script is in order somewhere, just not sure where yet.

My use case is actually easy to fix

I am glad to hear this is the case, as I would love to avoid merging the PR.

[DannyBen]

This can now be mitigated in v1.3.1 by choosing to either disable the formatter, or choosing the external (shfmt) formatter:

# settings.yml
# Choose a post-processor for the generated script:
# formatter: internal     # Use Bashly's internal formatter (compacts newlines)
# formatter: external     # Run the external command `shfmt --case-indent --indent 2`
# formatter: none         # Disable formatting entirely
# formatter: <string>     # Use a custom shell command to format the script.
#                         # The command will receive the script via stdin and
#                         # must output the result to stdout.
#                         # Example: shfmt --minify
formatter: internal

Thanks for the issue.