[BUG] Text (in contents stream) with parentheses decoded/encoded incorrectly

[mknox74]

Can I report a bug here? I'm not quite sure this is a bug of iText or just RUPS, and since there's no dedicated RUPS forum (or at least I did not find one), I decided to write here.

The other day I was hand-editing contents streams of a pdf (for whatever reason, it is not important). Everything was fine, but for one page - after editing one line of text was a total mess in a pdf viewer. With trial and error I isolated the bug (I mean I've found a way to reproduce this bug)

There's this simple pdf, displaying one line of text: "Hello, world! (from PDF :-))"
out.pdf

In the contents stream it is:
(Hello, world! \(from PDF :-\)\)) Tj

When edited in RUPS, it is displayed as:
(Hello, world! (from PDF :-))) Tj (notice the lack of backslashes)

In RUPS, I just updated the stream and saved the file
outr.pdf

Magically the pdf viewer now displays nothing. That's because the contents stream now contains:
(Hello, world! (from PDF :-))) Tj - as previously seen in RUPS stream editor
Yes, you see it right, no backslashes, and the parentheses do not match. It's even worse in TJ command, as kerning offsets may be mistakenly added to text to display (that's what happened to me in my original pdf, btw, hence "a total mess" I mentioned)

I think it's quite serious bug - be it in iText or RUPS.

I was using RUPS 7.2.3, but today I tested it on 7.2.4 - to the same result.

[MatthiasValvekens]

It's definitely a bug in RUPS, not iText Core.

This is actually an artefact of the interaction between iText's stream processing and RUPS's Swing UI. RUPS has to render the content stream in a Swing document to make it editable and highlightable, and then "recompile" it after editing to save the stream content in the PDF. Until recently this was done very naively (just dumping the pane content into the stream without caring too much about lossy conversions). This messed up all kinds of encoding-related things as well, since PDF content streams contain mixed binary/textual data.

I rewrote most of that logic about half a year ago (when I still worked for iText) to deal with the encoding issues, but it looks like I forgot to fix this particular problem.

The way to deal with this is actually not completely obvious. Some unstructured ramblings/observations:

• We need to ensure that this method properly deals with escaping parentheses in string literals that aren't hexadecimally encoded. The editor "knows" where existing string literals are, so that is actually relatively easy.
• ...but the internal representation is less obvious: we currently "magically" deal with strings that have an explicit encoding by decoding & displaying the string content while also keeping track of the original encoding in the UI metadata, so the editor only has to deal with text, not encoding minutiae. But maybe we should preserve literal \ escapes for parentheses in the editor pane as well, since it matters to the user's editing view.
• ... but also that has sharp edges: we could escape all parentheses on ingestion here. But this requires care as well: if RUPS later decides to encode that string hexadecimally, then the escapes need to be removed first! I don't think we currently do that anywhere, but still: there's a potential in-band encoding issue here. There needs to be a way to allow editing with escapes without conflating escapes with string data in the internal representation for the editor.

TL;DR: Getting this right will likely require some trial and error, but I think the areas of the code base I linked are where the changes would have to happen.