read_html hanging?

[jseabold]

Is this a local problem? I'm on g41d10b5. The following url appears never to return, or at least it's taking an inordinately long time.

http://www.nku.edu/~longa/geomed/ppa/doc/globals/Globals.htm

Same result with either the url or the raw html.

[jseabold]

Though it works if you just copy out the table code HTML, so it must be a parsing issue.

[jseabold]

Perhaps I just need to pass in some regex though. It seemed pretty simple, so I expected it to work.

[cpcloud]

I can take a look later today.

[cpcloud]

There's something strange going on with html5lib. lxml + bs4 "parses" it but who knows if it's correct.

[cpcloud]

HTML that lets you know you're in for some pain:

<meta content="Microsoft Word 97" name="Generator"/>

[jseabold]

Haha, indeed.

[cpcloud]

@jseabold I've sort of narrowed it down:

ipdb> list
    481                 return [element for element in generator
    482                         if isinstance(element, Tag)]
    483             # Optimization to find all tags with a given name.
    484             elif isinstance(name, basestring):
    485                 return [element for element in generator
--> 486                         if isinstance(element, Tag) and element.name == name]
    487             else:
    488                 strainer = SoupStrainer(name, attrs, text, **kwargs)
    489         else:
    490             # Build a SoupStrainer
    491             strainer = SoupStrainer(name, attrs, text, **kwargs)

ipdb> element
<ol><font face="Arial"><i><b>

</b></i><li>The input data file should contain the X,Y coordinates and the value at each point (x<sub>I</sub>).</li>
<li>Input whether you have a spatial weights matrix file.</li>
<li><font face="Arial">If you do not have a spatial weights matrix, you’ll be asked to enter the </font><i><font size="4">A </font></i><font face="Arial">and</font><i><font size="4"> m </font></i><font face="Arial">parameters (see below).</font></li></font><li><font face="Arial">If you do not have a spatial weights matrix, you’ll be asked to enter the </font><i><font size="4">A </font></i><font face="Arial">and</font><i><font size="4"> m </font></i><font face="Arial">parameters (see below).</font></li><font face="Arial">
<li>You will be asked to enter the maximum distance, the number of steps, and whether you want bands or increments.</li></font></ol>
ipdb> n
> /home/phillip/.virtualenvs/pandas/lib/python2.7/site-packages/bs4/element.py(485)_find_all()
    484             elif isinstance(name, basestring):
--> 485                 return [element for element in generator
    486                         if isinstance(element, Tag) and element.name == name]

ipdb> n
> /home/phillip/.virtualenvs/pandas/lib/python2.7/site-packages/bs4/element.py(486)_find_all()
    485                 return [element for element in generator
--> 486                         if isinstance(element, Tag) and element.name == name]
    487             else:

ipdb> element
<i><b>

</b></i>
ipdb> n
> /home/phillip/.virtualenvs/pandas/lib/python2.7/site-packages/bs4/element.py(485)_find_all()
    484             elif isinstance(name, basestring):
--> 485                 return [element for element in generator
    486                         if isinstance(element, Tag) and element.name == name]

ipdb> n
> /home/phillip/.virtualenvs/pandas/lib/python2.7/site-packages/bs4/element.py(486)_find_all()
    485                 return [element for element in generator
--> 486                         if isinstance(element, Tag) and element.name == name]
    487             else:

ipdb> element
<ol><font face="Arial"><i><b>

</b></i><li>The input data file should contain the X,Y coordinates and the value at each point (x<sub>I</sub>).</li>
<li>Input whether you have a spatial weights matrix file.</li>
<li><font face="Arial">If you do not have a spatial weights matrix, you’ll be asked to enter the </font><i><font size="4">A </font></i><font face="Arial">and</font><i><font size="4"> m </font></i><font face="Arial">parameters (see below).</font></li></font><li><font face="Arial">If you do not have a spatial weights matrix, you’ll be asked to enter the </font><i><font size="4">A </font></i><font face="Arial">and</font><i><font size="4"> m </font></i><font face="Arial">parameters (see below).</font></li><font face="Arial">
<li>You will be asked to enter the maximum distance, the number of steps, and whether you want bands or increments.</li></font></ol>

that <ol> element just keeps on parsing every time....

I would guess that the invalidity of the HTML is what's causing this, let me look a little deeper

[cpcloud]

For whatever reason, there's a cycle in the parse tree (which makes it not a tree anymore) so that's why this never finishes. One of the element's children is its parent, thus never finishing. This might take me a while to fix....as it looks like a bs4 problem....since i can parse successfully with html5lib alone

[jseabold]

Thanks for having a look. Not a huge priority for me or anything and obviously not a typical web site. Just wasn't sure what was going on.

[cpcloud]

[cpcloud]

@jseabold can we close this?

[jseabold]

Sure, if there isn't any way to fail gracefully on malformed HTML here. Not likely to come up often so probably not worth any effort.

[cpcloud]

eh i'll leave it... low prio tho