fix: Remove JavaScript from HTML reader output

[tomaarsen]

Closes #149

Hello!

Pull Request overview

• No longer turn JavaScript from script tags in an HTML document into an Element.

Bug

See #149 for details - it contains the following snippet:

import requests
from unstructured.partition.html import partition_html

url = "https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-december-13"
r = requests.get(url)
elements = partition_html(text=r.text)
print("\n\n".join([str(el) for el in elements[:5]]))

Which outputs:

Skip to main content

(function(d){
  var js, id = 'facebook-jssdk'; if (d.getElementById(id)) {return;}
  js = d.createElement('script'); js.id = id; js.async = true;
  js.src = "//connect.facebook.net/en_US/all.js#xfbml=1";
  d.getElementsByTagName('head')[0].appendChild(js);
}(document));

Search form

Home

Who We Are

Details

The bug was caused by the _construct_text function, which was called using a <td> tag. Within this tag, there is a <script> tag, containing the JS. The _construct_text function will happily loop through this <script> tag and consider all of its contents as "text".
This fix verifies whether the tag that we're looping over is not a <script> tag, but recognize that this is a bit of a bandaid solution on a bigger problem: there may be other tags for which we don't want the text to be included in the HTML reader output.

However, for the time being, this is a major improvement.

After this PR

Skip to main content

Search form

Home

Who We Are

Research

• Tom Aarsen

[MthwRobinson]

LGTM, pending one tweak to the CHANGELOG. Thanks!

[MthwRobinson]

Could you add this under 0.5.1-dev1 and add it as a new entry above 0.5.0?

[MthwRobinson]

Actually looks like there's a test failing. Please also add a new test that includes a script tags and tests that the script content isn't included in the output.

[tomaarsen]

Actually looks like there's a test failing. Please also add a new test that includes a script tags and tests that the script content isn't included in the output.

There is indeed! I converted it to draft until I resolve it.
I'll also tackle the CHANGELOG nitpick.

[MthwRobinson]

Could you also add a test to test_html.py to test that the script tag content gets removed?

[tomaarsen]

Will do

[tomaarsen]

Should be ready now. Apologies for forgetting to include the new test earlier. The test case is simple: it just checks whether the output elements contain function (. That should be sufficient to ensure that no regression is introduced in the future.
I want to point out that this test case fails when moved to main.

[MthwRobinson]

LGTM with the test added. Thanks!