fix(opinions): Add filter linebreaksbr to html

[flooie]

Fix #3377

Fix missing line breaks that sporadically affect our HTML with citations. Add django filter for line breaks where new lines exist and it cleans up our HTML beautifully.

Before

After

[mlissner]

Yeah, I don't know, Bill. You don't want to put zillions of <br> tags into the page. It'll make it slower, but even if that were negligible, it would also just make it uglier? Is there a better way to do this?

[flooie]

@mlissner

The problem we are seeing is that the HTML is not rendering correctly when new lines are inside <pre> tags or maybe other types. I'm sorry that I wasn't clear about what is happening here so let me back up and take a second stab at it.

The above example I used for illustration has a stretch of text that looks like this on our website.

liability. See Burlington Industries, Inc. v. Ellerth, 524 U.S. 742(1998); Faragher v. Boca Raton, 524 U.S. 775 (1998). Liability

All one line - no spacing. If you look at the PDF - it should look like this.

liability. See Burlington Industries, Inc. v. Ellerth, 524 U.S. 742
(1998); Faragher v. Boca Raton, 524 U.S. 775 (1998). Liability

with a break after the Citation and a new line starting at (1998).

I'm not an expert in HTML with citations being generated but it's clear what is happening here. Lets look at the html_with_citations content im going to add new lines for ease of reading

...\r\n\r\nliability. See Burlington Industries, Inc. v. Ellerth, </pre>
<span class="citation" data-id="118244">
<a href="/opinion/118244/burlington-industries-inc-v-ellerth/">524 U.S. 742</a></span>
<pre class="inline">\r\n(1998); 
Faragher v. Boca Raton, 
</pre><span class="citation" data-id="118245">
<a href="/opinion/118245/faragher-v-boca-raton/">524 U.S. 775</a>
</span><pre class="inline"> (1998). 
Liability\r\nunder Monell v. New York City Department of Social Services,\r\n</pre>

See the <pre class="inline">\r\n(1998); <- those returns are what are being dropped and causing our runoffs.

[mlissner]

OK, sorry, I definitely misunderstood how linebreaksbr works. Somehow I thought it was going to convert all whitespace to br's.

I'm a bit concerned though because usually HTML is supposed to ignore line returns and white space. Have you tried this on a bunch of different content types and sources to make sure it doesn't have any weird effects?

[flooie]

I've looked at maybe a dozen or so examples, but there are more than that. Though I suspect that was enough to get a sense. I saw nothing other than it nicely processing the HTM - Do you want me to do a much more extensive review?

My expectation is that this is only for scraped opinions.

[mlissner]

So the first change you made is focused on things that have source of C. That seems mostly OK, because those are mostly PDFs with bad plain text, but even some of those come in as HTML where we don't control the line returns.

The second and third changes below are not from the scrapers and worry me. My concern is that if we scrape some HTML and it has a bunch of line returns in it, we'll convert those line returns to <br> tags and make the content really long without meaning to. In general line returns in HTML should be ignored, but this does the opposite.

So I think:

• The first one might be fine. At least, for PDFs it should be.
• The second and third ones worry me and seem like they could cause trouble.

[flooie]

great. So lets drop the second and third one - leave the first - which was my primary focus at the start and just keep a watchful eye. its still a very easy and quick revert @mlissner yes?

[mlissner]

Sounds good.

[flooie]

@mlissner this I think is ready for your approval. It failed a web hook test - but I am rerunning and it and would suggest that its likely not related and deals with some other bug in web hooks?

[mlissner]

Cool. Let's see how this goes.