terminfo parser cannot handle ncurses 6.4's screen.xterm-256color

[maleadt]

As noticed on Arch Linux, in a screen session:

ERROR: DimensionMismatch: arrays could not be broadcast to a common size; got a dimension with lengths 75 and 76
Stacktrace:
  [1] _bcs1
    @ Base ./broadcast.jl:519 [inlined]
  [2] _bcs
    @ Base ./broadcast.jl:513 [inlined]
  [3] broadcast_shape
    @ Base ./broadcast.jl:507 [inlined]
  [4] combine_axes
    @ Base ./broadcast.jl:488 [inlined]
  [5] instantiate
    @ Base ./broadcast.jl:302 [inlined]
  [6] materialize
    @ Base ./broadcast.jl:867 [inlined]
  [7] extendedterminfo(data::IOStream; NumInt::Type{UInt32})
    @ Base ./terminfo.jl:145
  [8] read(data::IOStream, ::Type{Base.TermInfoRaw})
    @ Base ./terminfo.jl:110
  [9] #436
    @ Base ./io.jl:486 [inlined]
 [10] open(f::Base.var"#436#437"{Base.TermInfoRaw}, args::String; kwargs::@Kwargs{})
    @ Base ./io.jl:396
 [11] open
    @ Base ./io.jl:393 [inlined]
 [12] read
    @ Base ./io.jl:486 [inlined]
 [13] load_terminfo(term::String)
    @ Base ./terminfo.jl:241
 [14] (::Base.var"#1047#1049"{Bool, Symbol, Bool})(REPL::Module)
    @ Base ./client.jl:417
 [15] #invokelatest#2
    @ Base ./essentials.jl:892 [inlined]
 [16] invokelatest
    @ Base ./essentials.jl:889 [inlined]
 [17] run_main_repl(interactive::Bool, quiet::Bool, banner::Symbol, history_file::Bool, color_set::Bool)
    @ Base ./client.jl:415
 [18] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:332
 [19] _start()
    @ Base ./client.jl:552

MWE:

TERM=screen.xterm-256color julia +dev

... or:

read(open("/lib/terminfo/s/screen.xterm-256color"), Base.TermInfoRaw)

screen.xterm-256color.zip

x-ref: #50797 (cc @tecosaur)

[tecosaur]

Ok, I've just had a little look at this. I've done a small investigation.

The short version

I think the terminfo file is malformed, or I've misunderstood something.

The long version

The terminfo file starts as follows.

00000000: 1e02 3d00 2600 0f00 7101 a705 7363 7265  ..=.&...q...scre

Since it starts with the magic value 0x021e (0o01036) we know that numeric values will be unsigned 32-bit integers, not 16. From here, we read the standard terminfo section without incident. We read 38 flags, 15 numbers, and 369 strings (172 of which are non-blank).

At this point, we're at 0x935 and the IO stream still has bytes to read, so we take this as a sign of an extended info section and jump to 0x936 so that we're on an even byte (yes, this is a thing with terminfo).

At 0x936 in the terminfo file, the extended terminfo starts.

00000930: 3225 6473 0000 0200 0000 4a00 9500 b103  2%ds......J.....
00000940: 0101 0000 0700 ffff 1300 2500 2b00 3500  ..........%.+.5.

The next five (UInt32) numbers are:

• 0x0002 (2): the number of flags
• 0x0000 (0): the number of numbers
• 0x004a (74): the number of strings
• 0x0095 (149): the number of table entries
• 0x03b1 (945): the number of bytes in the table

At this point, it's doesn't look like the numbers add up. A table entry is required to label every flag+number+string entry, and also to give the value of each string entry.

In other words,

  table_entries = num_flags + num_numbers + 2*num_strings
            149 = 2 + 0 + 2*74
            149 = 150

This clearly doesn't line up, and we'll see more later.

At this point, we are at 0x940, at which point we read the two flags to be 0x01 and 0x01. There are no numbers to read. At this point, we read the next 149 UInt16s to get the indices of the start of each entry in the table section. We are now at 0xa6e.

00000a60: 6101 6701 6d01 7301 7901 7e01 8301 1b5d  a.g.m.s.y.~....]
00000a70: 3131 3207 001b 5d31 323b 2570 3125 7307  112...]12;%p1%s.

The last index is apparently 0x017e, however note that the next UInt16 also looks like it could be an index (0x0183), and after that extra index we see a common start to a control string (0x5d 0x1b = \e]).

However, since we are told there are 149 table entries (not 150), the next two bytes (0x0183) are read as part of the first table entry "\x83\x01\e]112\a".

The first 74 table entries are read as read as string values. In addition to the first value, the last value also looks odd — more like a label value.

(Base) julia> strings = table_strings[1:string_count]
74-element Vector{String}:
 "\x83\x01\e]112\a"
 "\e]12;%p1%s\a"
 "\e]52;%p1%s;%p2%s\a"
 "\e[2 q"
 ⋮
 "\e[29m"
 "\e[9m"
 "\e[M%?%p4%t%p3%e%{3}%;%' '%+%c%p2%'!'%+%c%p1%'!'%+%c"
 "AX"

Then we end up with 75 labels for 2 flags and 74 strings, and the error you've reported.

So, I'm not sure what's going on with this terminfo file, or how we should handle it.

Immediate steps

This all said, I have noticed one or two touch-ups that should be done to terminfo.jl as a result of this investigation, and will be making a PR shortly.

Other people's thoughts on this particular terminfo file would be appreciated.

[tecosaur]

I've now tested against the 2873 terminfo files on my system, and I'm finding seven cause any problems (all of this sort):

(Base) julia> terms = let terms = String[]
                  for (root, dirs, files) in walkdir("/usr/share/terminfo")
                      append!(terms, files)
                  end
                  terms
              end;

(Base) julia> for term in terms
              try
                  Base.load_terminfo(term)
              catch err
                  @error "Failed with $term" err
              end
              end
┌ Error: Failed with screen-bce.xterm-new
│   err = DimensionMismatch: arrays could not be broadcast to a common size; got a dimension with lengths 75 and 76
└ @ Base REPL[3]:5
┌ Error: Failed with screen.putty-m1
│   err = DimensionMismatch: arrays could not be broadcast to a common size; got a dimension with lengths 7 and 8
└ @ Base REPL[3]:5
┌ Error: Failed with screen.putty-m1b
│   err = DimensionMismatch: arrays could not be broadcast to a common size; got a dimension with lengths 7 and 8
└ @ Base REPL[3]:5
┌ Error: Failed with screen.putty-m2
│   err = DimensionMismatch: arrays could not be broadcast to a common size; got a dimension with lengths 7 and 8
└ @ Base REPL[3]:5
┌ Error: Failed with screen.xterm-256color
│   err = DimensionMismatch: arrays could not be broadcast to a common size; got a dimension with lengths 75 and 76
└ @ Base REPL[3]:5
┌ Error: Failed with screen.xterm-new
│   err = DimensionMismatch: arrays could not be broadcast to a common size; got a dimension with lengths 75 and 76
└ @ Base REPL[3]:5
┌ Error: Failed with screen.xterm-xfree86
│   err = DimensionMismatch: arrays could not be broadcast to a common size; got a dimension with lengths 75 and 76
└ @ Base REPL[3]:5

[christiangnrd]

Can anyone else confirm that this was fixed by #51399? I tried to reproduce earlier today and found out that it was working.

[JeffBezanson]

This seems to work now; closing.