Enhancement request: read_html() parameter to control contents of DataFrame

[thedatadoc]

The current read_html() method is useful when parsing simple tables, however in practice, not every html table has been created with simple parsing in mind. It is often required that we identify the type or purpose of data in a table cell. HTML tables generally express information this via the assignment of a CSS class name to a th or td element.

I propose an optional read_html() string parameter called "get_attribute" that indicates if the parser should get the text of the cell or if it should get the value of an attribute. If get_attribute is None, behavior of read_html() is the same as its current functionality: th/td cell values will be read into the dataframe. If it is a string value, read_html() will retrieve attribute values rather than cell text. If a th or td element has no matching attribute, the value returned is None.

I would use this functionality as follows:

myTableData = pandas.read_html( table )[ 0 ]
myTableClasses = pandas.read_html( table, get_attribute = 'class' )[ 0 ]

The dataframe myTableClasses would contain information that I could use to map cell styling to the values in myTableData. I imagine it could be useful for other types of information as well, conveyed via other attributes.

[WillAyd]

Do you have examples of general sites where this would be useful? Off the top of my head seems like a very niche application that you would want to roll your own parser for, but if there are examples out there that illustrate the utility would be helpful for consideration

[thedatadoc]

I'll find a few examples. The specific data I am working with right now is non-public, but I've encountered this elsewhere on public sites.

I could roll my own parser, however, I really like the standard pandas functionality, and I want to minimize the risk of interpreting html differently while retrieving cells. E.g., colspan and rowspan, specifically. I'd like the same code to make those types of decisions and simply populate the table with the desired value.

[WillAyd]

OK thanks - if you can provide again would be helpful.

FWIW you might be able to override _text_getter in a subclass to get what you want without interfering with the rest of the logic:

pandas/pandas/io/html.py

Line 143 in 8d124ea

class _HtmlFrameParser:

[thedatadoc]

Agreed - already started down that path. I thought it might be a useful feature for others, so I thought I'd write an enhancement request. Having it in pandas is marginally better than maintaining that code myself, but I agree that it is not required for instances where tables are formatted as one would expect.
It is particularly useful when the table output is dynamic (so labels or row positions aren't always predictable) and rows can contain different types of information.

For example, an HTML table row may contain summary statistics that do not share the same meaning as the remainder of the table. They are generally styled differently to highlight the fact that they are different. In a perfect world, this data ends up in their own table, but that doesn't seem to be the case all of the time.

In any case, I'll find some examples and include them here and the issue can be prioritized accordingly.

[thedatadoc]

This article includes screenshots of tabular data represented like this: https://www.qresearchsoftware.com/market-research-guide-basic-data-analysis

(I'll still keep working to get live link examples.)

[WillAyd]

Cool thanks - looking forward to what you can find. Some other options include the match keyword and/or using something like the SoupStrainer with bs4 to remove what you don't want from intermixed tables

[thedatadoc]

When one needs an example of complexity in something that could otherwise be ordered, one need only look at a .gov website :) Here's a pretty good example, I think.

https://www.bls.gov/news.release/empsit.t09.htm

They aren't using the class attribute, but they do use a "headers" attribute on their td elements. If I were parsing this, I couldn't rely entirely on the text in the "Characteristic" column to identify the meaning of the row. I'd need to group by the first value in the headers attribute. In this case, if I had this feature, I think the easiest way generate a meaningful multilevel index would be to join the values dataframe with the headers dataframe on the row index so that I could access that value easily and re-index the table.

Clearly, a clever person can solve the problem ... but having the feature included in pandas helps minimize risk.

[topper-123]

do I understand you correctly, that you would like the option to retrieve a DataFrame of attribute values of the corresponding td element?

So, if you'd do df = pd.read_html('https://www.bls.gov/news.release/empsit.t09.htm', get_attribute = 'headers') you would get a data frame with frame cell values set to the "headers" attributes and if that attribute is not found, then set it to nan?

I can see the usefulness of this, but have the same concerns that @WillAyd has, i.e. is it not general enough to handle enough types of situations. E.g. (in your example table) what if they set the interesting attribute on the span or p inner elements, what if an desired attribute is set on the tr element, etc.

So, my concern is whether a possible solution could be made generalized enough, or conversely whether a sufficiently generalized solution would become so complex, that it would become a burden on Pandas to maintain it and it would be better to do it outside pandas.

[thedatadoc]

It is true that just as we don't control the page author's table structure when they choose to mix dissimilar data in a single table's rows or columns, we don't control their use of other HTML within table cells, either. In the case you describe where there is a span or p, it would probably be better if there was some table cell handler function that could override _text_getter altogether. To me, that's a different use case that seems more in line with the thinking behind the converters in read_csv, rather than simply specifying a single attribute. A converter-like parameter for read_html could cover both, however, if it could access the table cell element and all of its contents.

It seems the question here is whether or not the pandas maintainers intend for read_html to be as supremely configurable as read_csv via the use of method parameters. As @WillAyd mentions above, you can simply override via a subclass - but that is not as approachable for all users of pandas. I'm fine with it, but many of my colleagues whom I've introduced to pandas would struggle with that. They would likely choose to write hundreds of lines of difficult to maintain code.

[topper-123]

I do find this functionality useful , as I said, as I've dealt with tables, where understanding of data depends on html attributes (typically class or id). I only doubt if this isn't too complex a problem to push into pandas itself, and maybe it would be better to have in a separate package.

In the csv cases, there's never ambiguity what the input cell values are and converters is "just" operation on that value. In html, if you want to get out the attributes of a table, the interesting values may be in tr, th, td, or an attribue on the cell value itself (span etc.) or other locations. How would you make read_html navigate to the desired attribute in a generalized-but-not-too-complex way?

In the .gov table you showed, could you demonstrate various options? E.g. if the attribute is set on the tr , how do you point pandas to it, and in what data frame cell(s) does the attribute go into? I get a feeling such things get difficult to model.

[thedatadoc]

This hasn't dropped off my radar - just a bit swamped at work. I'll get back to this before Monday (have some work travel next week).

[thedatadoc]

@topper-123 I think you've convinced me that the attribute name alone is not sufficient for all instances. However, the idea of a converter-style handler would be much more appropriate. A method that overrides the default _text_getter and accepts the same obj that _text_getter accepts would be sufficient to access the element, its text, any attribute on the element, or any of its parent elements and their attributes.

In the case of the .gov table, I don't think I'd need to traverse the DOM to a parent to get what I need. However if I needed to, I could.

I can implement this if you're interested in seeing an example of how I'd solve it.

[topper-123]

_text_getter manipulates the DOM element, right? So are you thinking of a e.g. a dom_converters attribute, that would just take a callable, and/or perhaps a dict of callables, where the callable would take a DOM element for the relevant parser and call that inside _text_getter? So would work like this?

>>> url = 'https://www.bls.gov/news.release/empsit.t09.htm'
>>> df = pd.read_html(url, dom_converters=lambda obj:  obj.attrib.get('class'))  # get classs attribute for every element in frame
>>> df = pd.read_html(url, dom_converters={'td': lambda obj:  obj.attrib.get('class')})  # only work on td elements

I like that. Very general functionality and very simple for pandas to maintain.

@WillAyd ?

[thedatadoc]

Yep, exactly.