Console output truncated with an ellipsis causes remaining text to shift to the left when using $PSStyle

[nmbell]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

Console output truncated with an ellipsis causes remaining text to shift to the left when using $PSStyle

I have a .ps1xml format file that defines a view for file and directory objects which uses $PSStyle to highlight parts of the output, e.g.:

<?xml version="1.0" encoding="utf-8"?>
<Configuration>
    <SelectionSets>
        <SelectionSet>
            <Name>FileSystemTypes</Name>
            <Types>
                <TypeName>System.IO.DirectoryInfo</TypeName>
                <TypeName>System.IO.FileInfo</TypeName>
            </Types>
        </SelectionSet>
    </SelectionSets>
    <ViewDefinitions>
        <View>
            <Name>children</Name>
            <ViewSelectedBy>
                <SelectionSetName>FileSystemTypes</SelectionSetName>
            </ViewSelectedBy>
            <TableControl>
                <TableHeaders>
                    <TableColumnHeader> <Label>Name</Label>                <Width>30</Width>  <Alignment>Left</Alignment>  </TableColumnHeader>
                    <TableColumnHeader> <Label>LastWriteTime      </Label> <Width>19</Width>  <Alignment>Left</Alignment>  </TableColumnHeader>
                    <TableColumnHeader> <Label>Size      </Label>          <Width>10</Width>  <Alignment>Right</Alignment> </TableColumnHeader>
                    <TableColumnHeader> <Label>FullName</Label>                               <Alignment>Left</Alignment>  </TableColumnHeader>
                </TableHeaders>
                <TableRowEntries>
                    <TableRowEntry>
                        <TableColumnItems>
                            <TableColumnItem>
                                <ScriptBlock>
                                    "$($PSStyle.Reverse)$($_.Name)$($PSStyle.Reset)"
                                </ScriptBlock>
                            </TableColumnItem>
                            <TableColumnItem>
                                <ScriptBlock>
                                    $_.LastWriteTime.ToString('yyyy-MM-dd HH:mm:ss')
                                </ScriptBlock>
                            </TableColumnItem>
                            <TableColumnItem>
                                <ScriptBlock>
                                    ($_.Length/1MB).ToString('0.000')+' MB'
                                </ScriptBlock>
                            </TableColumnItem>
                            <TableColumnItem>
                                <ScriptBlock>
                                    $_.FullName
                                </ScriptBlock>
                            </TableColumnItem>
                        </TableColumnItems>
                    </TableRowEntry>
                </TableRowEntries>
            </TableControl>
        </View>
    </ViewDefinitions>
</Configuration>

$filePath = 'D:\my.format.ps1xml'
Update-FormatData -PrependPath $filePath
Get-ChildItem -Path $PSHOME -File

When the Name column contains a long file name, the console truncates the name and adds an ellipsis … to indicate that there are truncated characters. When this happens, the remaining characters on the line are shifted 3 spaces to the left:

Removing the references to $PSStyle in the .ps1xml format file gives the expected layout:

The shift effect is present for any value of $PSStyle.OutputRendering.

Expected behavior

Output text on lines with a truncation ellipsis should remain aligned with its column header.

Actual behavior

Output text on lines with a truncation ellipsis is shifted to the left.

Error details

n/a

Environment data

Name                           Value
----                           -----
PSVersion                      7.2.1
PSEdition                      Core
GitCommitId                    7.2.1
OS                             Microsoft Windows 10.0.19043
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

[jhoneill]

Looking at the way the truncation is happening in your two examples, it seems to be mis-counting the number of characters, and cutting the text 3 characters early, which has the knock-on effect of shifting the rest of the line 3 to the left.
Does this happen if you specify a colour instead of reverse ? I may be mis-remembering but I think there was a bug to do with some escape sequences not being recognized, it's almost like it is fixed everywhere except the truncate routine.

[nmbell]

The actual .ps1xml file I use has some logic to set different colors (including reverse) depending on various properties (read only, hidden, etc.), and I see the shifted text any time a color is applied to a truncated name.

[jhoneill]

OK. It's odd that truncate is wrong by 3. You have [esc], [, 0, m, for reset, and [esc], [, 7, m for reverse which is six printable chars and 8 in total. It would have been a neat explanation if the error was it discounted reset and colours but didn't understand reverse.

[jhoneill]

A related problem which isn't the one I referred to in the previous example, but relates to errors when the text is too wide for the column

The screen shot below is showing tasks carried out in different environments, the environments are on the right and have names "Shared services pre-release" and "Shared services dev". Red is used to indicate "All EXCEPT the one(s) shown"
When the there is a sufficient width the end of the first line and the second look like this:
Red Shared services pre-release reset
PanettoneService Red Shared services Dev reset

When there is less width the two lines become three:
Red Shared services
PanettoneService pre-release reset red Shared
services Dev reset
So the columns on default colours become red

[Jaykul]

For the record: the problem here is EXTREMELY ANNOYING and basically left my dream of using color codes in formatting files on hold. It also affects Format-Wide as well.

We can only add escape sequences to text if we're sure the text will not need to be truncated by the formatter.

Here's a simple repro example:

Given a function that outputs objects ...

function Get-DemoObjects {
    [PSCustomObject]@{PSTypeName = "User"; Name = "Bob Saggat"; Tenure = 2; Role = "Developer" }
    [PSCustomObject]@{PSTypeName = "User"; Name = "John Seymour"; Tenure = 6; Role = "Sw Engineer" }
    [PSCustomObject]@{PSTypeName = "User"; Name = "Billy Bob Thorton"; Tenure = 13; Role = "Senior DevOps Engineer" }
}

If you format those objects such that they get truncated (imagine we did this in a format file):

Get-DemoObjects | Format-Table @{Width = 10; Name = "Name"; E = { $_.Name }}, 
                               @{Width = 15; Name = "Role";  E = { $_.Role }},
                               @{Width = 3; Name = "YIR";  E = { $_.Tenure }}

Everything is fine, until you add color, and suddenly things get over-truncated, and start sliding around.

Get-DemoObjects |
    Format-Table @{Width = 10; Name = "Name"; E = { (@("`e[32m", "`e[34m", "`e[33m", "`e[31m", "`e[33m", "`e[34m", "`e[32m")[[array]::BinarySearch(@(3, 5, 8), $_.Tenure)]) + $_.Name + "`e[39m"}},
                 @{Width = 15; Name = "Role";  E = { $_.Role }}, 
                 @{Width = 3; Name = "YIR";  E = { $_.Tenure }}

But it's even worse if you have more escape characters. If you get really unlucky, it can eat the whole text, completely. Other times, you get columns moving around so much they end up under the wrong heading and you can't make heads or tails of it. For my use case last week, imagine we had applied the same color formatting rule to three or four columns. Everything just looked ... random.

Get-DemoObjects |
    Format-Table @{Width = 10; Name = "Name"; E = { (@("`e[32m", "`e[34m", "`e[33m", "`e[31m", "`e[33m", "`e[34m", "`e[32m")[[array]::BinarySearch(@(3, 5, 8), $_.Tenure)]) + $_.Name + "`e[39m"}},
                 @{Width = 15; Name = "Role";  E = {
                                -join $(switch -regex ($_.Role){
                                        "Senior" { "`e[42m" }
                                        "Engineer" { "`e[1;33m" }
                                    }) + $_.Role  + "`e[0m"}}, 
                 @{Width = 3; Name = "YIR";  E = { $_.Tenure }}

Note that Format-Wide does the same thing, but it gets really weird sometimes. These columns overlap! What's even happening here?

(Get-DemoObjects) + (Get-DemoObjects) + 
(Get-DemoObjects) + (Get-DemoObjects) + 
(Get-DemoObjects) + (Get-DemoObjects) | 
    Format-Wide @{E = { (@("`e[32m", "`e[34m", "`e[33m", "`e[31m", "`e[33m", "`e[34m", "`e[32m")[[array]::BinarySearch(@(3, 5, 8), $_.Tenure)]) + $_.Name + "`e[39m"}} -co 2| Out-String -Width 47

Technically, Format-List is not affected by this, but colors don't work there either, because when it wraps, the second line gets no formatting, which is really gross:

Workaround

For table formats, if you force the width of your columns, then you can write truncation in the format file. As long as the format doesn't output text that's too wide, the formatter won't need to truncate it (the miscalculation only affects the truncation, not the length measurement).

[Jaykul]

Also, @jhoneill the reason for the number of characters ... is that you're forgetting that it also removed the escape characters from the end of the string.

If we simplify my example:

Get-DemoObjects | Format-Table @{Width = 10; Name = "Name"; E = { "`e[33m" + $_.Name } }, 
                               @{Width = 15; Name = "Role"; E = { $_.Role }},
                               @{Width = 3;  Name = "YIR";  E = { $_.Tenure }}

Get-DemoObjects | Format-Table @{Width = 10; Name = "Name"; E = { "`e[1;33m" + $_.Name } }, 
                               @{Width = 15; Name = "Role"; E = { $_.Role }},
                               @{Width = 3;  Name = "YIR";  E = { $_.Tenure }}

We can clearly see that it's truncating the number of printable escape characters (i.e. not ```e`` but everything else). When I add 1; at the front, it truncates an extra 2 visible characters from the end. When I add `[39m` on the end, it makes no difference, because the characters I added are the ones that get truncated.

[daxian-dbw]

Technically, Format-List is not affected by this, but colors don't work there either, because when it wraps, the second line gets no formatting, which is really gross:

@Jaykul Can you please also share the script for doing the list view shown in your comment? That would be helpful.

[ninmonkey]

@Jaykul
is the ? here intentional / from your prompt? It doesn't appear to parse as a command or alias, because the color is different. I don't think it changes anything for this ticket

[Jaykul]

Technically, Format-List is not affected by this, but colors don't work there either, because when it wraps, the second line gets no formatting, which is really gross:

@Jaykul Can you please also share the script for doing the list view shown in your comment? That would be helpful.

@daxian-dbw it's basically the same as above, just with | Format-List instead:

Get-DemoObjects |
    Format-List @{Name = "Name"; E = { (@("`e[32m", "`e[34m", "`e[33m", "`e[31m", "`e[33m", "`e[34m", "`e[32m")[[array]::BinarySearch(@(3, 5, 8), $_.Tenure)]) + $_.Name + "`e[39m"}}

Because I didn't want to make up new data, I just made the window very narrow (on Windows 10 you may have to increased the font size) to force the wrapping even though my text was really short 😉

Off topic, and just for the record:
@ninmonkey we talked about this on Discord, but just in case: The "?" in my prompt is due to a totally different bug in PSReadLine. It changes the color of the last character of your prompt as you type whenever the command isn't valid syntax, and if the last character is an extended character, and if your console encoding isn't set to UTF8, it screws up the character because it uses Console.Write (in my case, with OEM 437 encoding, that PowerLine arrow-block character isn't encodable).