regression: IOException: Mark invalid after upgrade to 1.12.2

[surli]

Hi,

we are using jsoup inside XWiki to perform some page validation as part of our integrations tests and we noticed a regression after the upgrade from jsoup 1.12.1 to 1.12.2. When accessing a particular page, we obtained the following:

Exception in thread "main" java.io.IOException: Mark invalid
	at org.jsoup.parser.CharacterReader.rewindToMark(CharacterReader.java:148)
	at org.jsoup.parser.Tokeniser.consumeCharacterReference(Tokeniser.java:192)
	at org.jsoup.parser.TokeniserState$38.read(TokeniserState.java:759)
	at org.jsoup.parser.Tokeniser.read(Tokeniser.java:59)
	at org.jsoup.parser.TreeBuilder.runParser(TreeBuilder.java:55)
	at org.jsoup.parser.TreeBuilder.parse(TreeBuilder.java:47)
	at org.jsoup.parser.Parser.parseInput(Parser.java:35)
	at org.jsoup.helper.DataUtil.parseInputStream(DataUtil.java:170)
	at org.jsoup.helper.DataUtil.load(DataUtil.java:66)
	at org.jsoup.Jsoup.parse(Jsoup.java:118)
	at Main.main(Main.java:47)

I've been able to isolate and reproduce in a "small" setup, but it still needs an XWiki instance.
Here's the small Java code I'm using for reproduction:

GetMethod method = new GetMethod("http://127.0.0.1:8080/xwiki/bin/admin/XWiki/XWikiPreferences?editor=globaladmin&section=Groups");
method.setFollowRedirects(true);
method.getParams().setSoTimeout(30000);
method.setDoAuthentication(true);
method.addRequestHeader("Authorization","Basic " + new String(Base64.encodeBase64("Admin:admin".getBytes())));
new HttpClient().executeMethod(method);
byte[] response = method.getResponseBody();
method.releaseConnection();

Jsoup.parse(new ByteArrayInputStream(response), null, "");

Before running this, you need to download an instance of XWiki 12.0 (http://maven.xwiki.org/releases/org/xwiki/platform/xwiki-platform-distribution-flavor-jetty-hsqldb/12.0/xwiki-platform-distribution-flavor-jetty-hsqldb-12.0.zip), unzip it, start it using the bash script (./start_xwiki.sh) and open a browser with http://localhost:8080 to be sure everything's loaded.

I checked with 1.12.1 and I confirm I didn't reproduce with it.

[jhy]

Hi,

Thanks for the replication detail. Does this replicate if you save the response byte[] to a file and then open that? (Which I expect it should, as that's all jsoup is getting input from.) If so, can you attach that as a file. It will simplify the repro.

[surli]

Does this replicate if you save the response byte[] to a file and then open that?

Indeed, here's the test file which reproduces the failure: test.html.txt

I now reproduce with:

Jsoup.parse(new FileInputStream("/tmp/test.html.txt"), null, "");

[jhy]

Thanks for that file. I have replicated and fixed, and it will be in 1.13.1. The issue was that during bufferUp, if a mark existed it would get cleared, and then it couldn't be rewound. This instance happened to require a rewind, so it would blow up. The fix is just to keep the mark when shuffling in new data.

You can build from HEAD to verify.

Also, out of interest, is there a reason to not use Jsoup.connect(url).get() for this? The benefit of using that is that the whole HTTP response does not need to be buffered into memory, and parsing can occur while the network stream in coming in. Happy to take feedback on anything missing in Connect.

[surli]

You can build from HEAD to verify.

Thanks, I'll check that.

is there a reason to not use Jsoup.connect(url).get() for this?

Well I did not write our validation tests where we use jsoup so I'd have to check properly but I can already imagine 2 reasons:

1. we just don't only use jsoup for the validation, so we have a common method to return the input stream of a page that we use for other validations too
2. we use HttpClient at several places in our code so we just reused it there again

[Nikos410]

Hi @jhy,

can you give an ETA on when a new release containing this fix will be published? If finishing version 1.13.1 is still going to take a while, we would really appreciate a hotfix release ❤️