Read from multiple <tbody> within a <table> #20891
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WillAyd
requested changes
May 1, 2018
pandas/io/html.py
Outdated
| return self._parse_raw_data(res) | ||
| raw_data = [] | ||
|
|
||
| if len(tbodies) > 0: |
There was a problem hiding this comment.
For PEP8 compliance use the boolean-ness of the sequence, so just if tbodies:
pandas/tests/io/test_html.py
Outdated
| </tbody> | ||
| </table>''' | ||
| expected = DataFrame({'A': [1, 3], 'B': [2, 4]}) | ||
| result = self.read_html(StringIO(data)) |
There was a problem hiding this comment.
Move the indexer to this line and make the subsequent line just tm.assert_frame_equal(result, expected). Obviously doesn't change result just reads better with the rest of the tests
Codecov Report
@@ Coverage Diff @@
## master #20891 +/- ##
==========================================
+ Coverage 91.78% 91.78% +<.01%
==========================================
Files 153 153
Lines 49337 49338 +1
==========================================
+ Hits 45285 45286 +1
Misses 4052 4052
Continue to review full report at Codecov.
|
jreback
added
Enhancement
IO HTML
jreback
requested changes
May 1, 2018
pandas/tests/io/test_html.py
Outdated
| @@ -396,6 +396,34 @@ def test_empty_tables(self): | |||
| res2 = self.read_html(StringIO(data2)) | |||
| assert_framelist_equal(res1, res2) | |||
|
|
|||
| def test_multiple_tbody(self): | |||
| """ | |||
There was a problem hiding this comment.
add the issue number here, you can use use # and not a triple-quotes here for the description of the test
|
@adamhooper do you know if / when you'll be able to update? I'm cutting the RC soon (hopefully a couple hours from now). If you're busy now I can push the changes and then we'll get this merged before the RC. |
WillAyd
approved these changes
May 1, 2018
|
Test failures from travis were unrelated. Fixing in #20906 |
|
Failures should be fixed in masater. |
|
Thanks @adamhooper! |
|
Er ... finally I'm available to fix things. You beat me to it -- thanks! |
|
Thank you! I'm glad we could get this in before the release.
…On Tue, May 1, 2018 at 2:05 PM, Adam Hooper ***@***.***> wrote:
Er ... finally I'm available to fix things. You beat me to it -- thanks!
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#20891 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABQHIvxQ3bQ_7Hz1bb2hlEW2DG-4gkXgks5tuLIKgaJpZM4TtdTg>
.
|
adamhooper
added a commit
to adamhooper/pandas
that referenced
this pull request
Jun 14, 2018
This is essentially a rebased and squashed pandas-dev#17054 (mad props to @jowens for doing all the hard thinking). My tweaks: * test_computer_sales_page (see pandas-dev#17074) no longer tests for ParserError, because the ParserError was a bug caused by missing colspan support. Now, test that MultiIndex works as expected. * I respectfully removed the fill_rowspan argument from pandas-dev#17073. Instead, the virtual cells created by rowspan/colspan are always copies of the real cells' text. This prevents _infer_columns() from naming virtual cells as "Unnamed: ..." * I removed a small layer of abstraction to respect pandas-dev#20891 (multiple <tbody> support), which was implemented after @jowens' pull request. Now _HtmlFrameParser has _parse_thead_trs, _parse_tbody_trs and _parse_tfoot_trs, each returning a list of <tr>s. That let me remove _parse_tr, Making All The Tests Pass. * That caused a snowball effect. lxml does not fix malformed <thead>, as tested by spam.html. The previous hacky workaround was in _parse_raw_thead, but the new _parse_thead_trs signature returns nodes instead of text. The new hacky solution: return the <thead> itself, pretending it's a <tr>. This works in all the tests. A better solution is to use html5lib with lxml; but that might belong in a separate pull request.
5 tasks
adamhooper
added a commit
to adamhooper/pandas
that referenced
this pull request
Jun 26, 2018
This is essentially a rebased and squashed pandas-dev#17054 (mad props to @jowens for doing all the hard thinking). My tweaks: * test_computer_sales_page (see pandas-dev#17074) no longer tests for ParserError, because the ParserError was a bug caused by missing colspan support. Now, test that MultiIndex works as expected. * I respectfully removed the fill_rowspan argument from pandas-dev#17073. Instead, the virtual cells created by rowspan/colspan are always copies of the real cells' text. This prevents _infer_columns() from naming virtual cells as "Unnamed: ..." * I removed a small layer of abstraction to respect pandas-dev#20891 (multiple <tbody> support), which was implemented after @jowens' pull request. Now _HtmlFrameParser has _parse_thead_trs, _parse_tbody_trs and _parse_tfoot_trs, each returning a list of <tr>s. That let me remove _parse_tr, Making All The Tests Pass. * That caused a snowball effect. lxml does not fix malformed <thead>, as tested by spam.html. The previous hacky workaround was in _parse_raw_thead, but the new _parse_thead_trs signature returns nodes instead of text. The new hacky solution: return the <thead> itself, pretending it's a <tr>. This works in all the tests. A better solution is to use html5lib with lxml; but that might belong in a separate pull request.
adamhooper
added a commit
to adamhooper/pandas
that referenced
this pull request
Jun 27, 2018
This is essentially a rebased and squashed pandas-dev#17054 (mad props to @jowens for doing all the hard thinking). My tweaks: * test_computer_sales_page (see pandas-dev#17074) no longer tests for ParserError, because the ParserError was a bug caused by missing colspan support. Now, test that MultiIndex works as expected. * I respectfully removed the fill_rowspan argument from pandas-dev#17073. Instead, the virtual cells created by rowspan/colspan are always copies of the real cells' text. This prevents _infer_columns() from naming virtual cells as "Unnamed: ..." * I removed a small layer of abstraction to respect pandas-dev#20891 (multiple <tbody> support), which was implemented after @jowens' pull request. Now _HtmlFrameParser has _parse_thead_trs, _parse_tbody_trs and _parse_tfoot_trs, each returning a list of <tr>s. That let me remove _parse_tr, Making All The Tests Pass. * That caused a snowball effect. lxml does not fix malformed <thead>, as tested by spam.html. The previous hacky workaround was in _parse_raw_thead, but the new _parse_thead_trs signature returns nodes instead of text. The new hacky solution: return the <thead> itself, pretending it's a <tr>. This works in all the tests. A better solution is to use html5lib with lxml; but that might belong in a separate pull request.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
git diff upstream/master -u -- "*.py" | flake8 --diff