Directly adjacent instances of raw source cause parsing error

[ghost]

When inserting raw LaTeX using the new marker system for doing so, if two instances of insertion are done back to back then the parser seems to fail to recognise one of the spans:

 This will render `\bfseries{`{=latex}properly`}`{=latex}.

And for a failing case:

This will cause `\`{=latex}`bfseries{`{=latex}errors`}`{=latex}.

The second example will produce the following output:

Unknown token type: 121
* (121) 55:8	'{=latex}'
	* (204) 55:2	'{='
	* (216) 57:5	'latex'
	* (206) 62:1	'}'
This will cause \texttt{\textbackslash{}`{=latex\}}bfseries\{\texttt{{=latex\}errors}\}`.

[ghost]

I should also add that it would help to report where in the source file the problem was encountered. That might be happening, but if so it is using a system that isn't easy to follow up on (total bytes from start of file?). It might be better to report line numbers at least.

[fletcher]

It did report exactly where the problem was encountered (in byte offsets, which is how MMD "thinks"), and even showed you the exact text at the problem area.

But an error report means that I left a particular token type out of the output processing, since in theory there should never be any errors in a MMD document, just things that don't work the user thought they would due to a misunderstanding of the syntax rules.

In this case, there is a bug, but it's not what you think. The final {=latex} causes the error message because it is not attached to a code span, and I failed to include an option for outputting that token when it was used incorrectly. From your message, I think you miscounted the backticks (or forgot to account for the fact that you escaped the second backtick?).

But the error message is easily resolved and will be fixed shortly.

[ghost]

Ah, I think I see what you mean, I hadn't consciously factored in that \ would not be a naked backslash and would rather be an escaped backtick. This was a solution ported directly from the older HTML comment method, where the code would have been <!--\--><!--bfseries{-->. Doing a blanket replace of HTML to backtick nomenclature caused the imbalance to appear.

So the question then becomes, how can one insert a backslash all by itself like that? I should be able to work around having done things that way in the past with a few tweaks to my generator, but just out of curiosity:

`\\`{=latex}

Rightly inserts two backslashes into the output.

Or for that matter (though it was not what I was trying to do), how can one get a backtick into the LaTeX output, since:

`\`\`this''`{=latex}

Produces:

\`\`this''

[fletcher]

You use a backslash normally inside code spans. The only thing you can't do (I think) is a backslash immediately before the final backtick -- you can use a space to separate the backslash from the backtick, e.g.:

`foo\ `

[fletcher]

(By normally I mean as you want it to appear in the output.)

[ghost]

Yes, that's exactly what I mean though, if for some reason you need to terminate the block at a backslash it seems the only way around the problem is to avoid ever having to do so. Probably not too big of a constraint all things considered. It's a bit of an odd case as I had two different operations generating different pieces of code. I can combine their total output effect to avoid this happening.

[fletcher]

See #133 for further discussion on this