max_columns == 0 incorrectly wraps around for wide dfs

[bjonen]

According to the description of max_columns, should switch to "smaller" (at the beginning to info, nowadays to truncate) view when set to zero instead of wrapping around.

display.max_columns: [default: 20] [currently: 0]
: int
        max_rows and max_columns are used in __repr__() methods to decide if
        to_string() or info() is used to render an object to a string.  In case
        python/IPython is running in a terminal this can be set to 0 and pandas
        will correctly auto-detect the width the terminal and swap to a smaller
        format in case all columns would not fit vertically. The IPython notebook,
        IPython qtconsole, or IDLE do not run in a terminal and hence it is not
        possible to do correct auto-detection.
        'None' value means unlimited.

Originally introduced here #453 (comment)

Fixed after broken here: #2881

Behavior change (broken again) here #5550

[jreback]

so is this a bug wrt to the current doc string? or an API change?

[bjonen]

I personally find this behavior extremely useful when dealing with big dfs, so I would like to bring it back to life. Then the docs are fine again too. Perhaps it would be nice to make this functionality more visible.

The final point is an enhancement proposal: Instead of using the info view when the df exceeds the terminal width, use the truncate view but adjust the truncation points (max_cols and max_rows we use under the hood) depending on the terminal size.

I love the idea of setting max_rows=0 and max_columns=0 and just let the frame expand to the terminal size showing as many rows and columns as possible truncating centrally. Right now, one has to micromanage these options depending on the frame (and its column width) to view as much data as possible.

Apparently it worked in the notebook too, back when it was introduced... #453 (comment)

[jreback]

makes sense
marking for 0.14.1

ideally get in soon a after 0.14.0 release to get some battle testing in

[bjonen]

Sounds good

[jorisvandenbossche]

@bjonen I think whether the truncate or info view is shown should just depend on the pd.options.display.large_repr option.

@jreback Is it too late for 0.14 if @bjonen can look at this quickly. As it is actually a regression I think?

[jreback]

I think if we do the bug fix ok

maybe leave the enhancement for 0.14.1

[bjonen]

@jorisvandenbossche Determining truncate vs info from pd.options.display.large_repr sounds right. I think the docs should define "smaller format" to be the value of large_repr so that this becomes transparent.

Right now truncate will just show an empty df for max_columns==0: https://github.com/pydata/pandas/blob/master/pandas/core/format.py#L326 .

We could just default to info view for now (bug fix). I'm also happy to check out how much work it is to implement the desired behavior for truncate.

[jreback]

@bjonen fix for this?

[bjonen]

I can look into this on the weekend. This might not be an easy one.

[jreback]

@bjonen fix for this, or for 0.14.1?

[bjonen]

This will have to wait for 0.14.1

[jreback]

noted

[bjonen]

@takluyver @y-p Is this an intentional change?

From your commit message (takluyver@be80898)

Truncate wide HTML repr rather than changing format.

I would guess that for max_columns=0 we get a truncated df when the screen size is exceeded, but the df actually wraps around (with \) if pd.options.display.expand_frame_repr = True and just prints the whole df running over many lines otherwise.

Finally, is there any code that maps line width to number of columns that can be represented?

[bjonen]

To be precise, the change happens here: takluyver@be80898#diff-1e79abbbdd150d4771b91ea60a4e1cc7R450

I'll try to make max_columns adjust to the terminal width when max_columns == 0 and submit a PR.

[jreback]

@bjonen fix for this?

[bjonen]

I have not had time to look into this so far...

[bjonen]

I made some progress on this for the string representation. The get_terminal_size function does not work in the notebook/qtconsole however (https://github.com/pydata/pandas/blob/8ac0e11b59b65f0ac898dca2beebde5f87836649/pandas/util/terminal.py#L21).

Does anybody know another way to infer the width in the notebook? I think it would be nice to have this feature in the notebook as well. Scrolling is a good default, but a truncated view which fits the screen is usually all I actually need.

[jreback]

maybe ask on the IPython list

[bjonen]

Here are the responses from the mailing list: http://python.6.x6.nabble.com/Getting-the-size-of-ipython-notebook-cell-td5063264.html#a5063344

So we can only have this automatic behavior in the terminal.

[jreback]

closes by #7691