Support transpiling to Rhai with #line or equiv?

[barries]

Hi!

Forgive (& clue bat) me if I missed something in the docs about this, thanks!

One of the use cases for embeddable scripting languages like Rhai is to be the intermediate language between a DSL with a custom, possibly pretty alien-to-rhai, syntax. For example, the (a) end user writes a template file, markdown or YAML file (like an OpenAPI spec), or a field in a GUI (say, something like a form or spreadsheet), then (b) some Rust code transpiles that source syntax into Rhai as the intermediate form, and (c) executes it using the Rhai engine.

In this use case, since the user is manipulating something other than the generated Rhai code, when errors happen, it's nice to have them refer to the original position (typically file, line and column, but could be a spreadsheet cell or the path to a node in an OpenAPI spec, for example).

If feasible, could Rhai support something like the #line directive (though, perhaps styled more like a Rust macro....) to set the Rhai compiler's idea of source location (string, line and column, say), and then have the errors report that location?

C and Perl have #line and it makes them really nice targets for transpilers. Last I did this in Python or Lua, etc., they don't support anything like this, which makes it hard for users to understand error messages, hints, debug output, etc., when using them as transpiler targets.

One possible example, might be to take a template like this:

<ul>
%%for name in names %{
    <li>%<=name | html_escape %></li>
%}
</ul>

and transpile it to something like this using source_loc!() to tell the Rhai parser the name of the source file and loc!() to tell it where the enclosed tokens occur in that source. They are just one simple syntax (and more powerful than the hoary old #line!), there are probably better.

source_loc!("my_template.tpl");
emit(loc!(1, 1, "ul\n"));
loc!(2, 0);for loc!(2, 7);name in names {loc!(2, 21);{
    emit(loc!(3, 2, "<li>")); emit(loc!(3, 17, html_escape)(loc!((3, 10, name))); emit(loc!(3, 31, "</li>\n"));
}
emit(loc!(4, 1, "</ul>\n"));
};

In this case, loc!(line_num, col_num, <tokens>) is a bit like the quote!() macro we know and love from proc macros...

[schungx]

I see the use for this. Let me think a bit on how to provide this functionality without disrupting the existing syntax.

I would probably use a special comment syntax, like //# to mark source map location...

Or copy JavaScript's map files...

[barries]

Thanks! The map files are a bit problematic in terms of managing them if they're separate files, but maybe the //# comments would allow inlining them?

The //# comments are nice, one key would be to support (say) /# ... #/ for inline location specs, like the loc!()s in the straw man above, because it turns out that things change position on lines a lot in intermediate form (Rhai) relative to the input (DSL) form. DSLs are often a bit more succinct than the intermediate form so things "move" left on the line a lot. For things that move right, space padding works well.

[barries]

For your map solution, adding a set of //# lines at the end of the source string would nicely declutter the main Rhai source.

Maybe lines like this?

//# "foo.dsl", <dsl line>, <dsl col>, <rhai line>, <rhai col>

I'll be interested to see what you come up with.

[schungx]

I'm actually thinking just having the source file/line/col info before each transcribed Rhai block of code.

Then Rhai will parse the source with positions all mapped back to the original source.

If there is an error, for example, the position will be in the source instead of in the transcribed Rhai code.

//# file.dsl:10,25
let foo = 42;
//# file.dsl:11,16
let bar = "hello";

//# file.dsl:13,8
process(
    //# file.dsl:13,15
    foo,
    //# file.dsl:13,20
    bar + 1
);

[barries]

Nice!

Could there be a way of allowing DSL character ranges on a line to be compressed, stretched or reordered?

For example:

DSL:

<%// some comment to spend a line...
<%for name in values %{

Rhai:

for name in injected_struct.values {

where:

Rhai code	source: line, col
for	my_file.dsl: 2,1
name in	my_file.dsl: 2,6
injected_struct.	prelude.dsl:
values	my_file.dsl: 2,14
{	my_file.dsl: 2,21

Or, say the DSL is a prefixy syntax:

DSL:

values.for_each(|foo| ...)

Rhai:

for foo in injected_struct.values {
    ...
}

Rhai code	source: line, col
for	my_file.dsl:1,8...1,15
foo	my_file.dsl:2,18
injected_struct.	prelude.dsl
values	my_file.dsl:1,1
{	my_file.dsl:1,22
...	...
'}'	my_file.dsl:1,26

These pretend DSLs are based on things I've actually had to codegen from. Not being able to map DSL character ranges into different locations and widths in the intermediate code was a pain point... I ended up having to break lines of the intermediate code across multiple lines every time an input span moved left relative to the current "spoofed" location in the intermediate file so as to inject a #line directive to move the spoofed location leftwards. And sometimes ranges shrink or expand (usually the latter--DSLs are often more concise than the intermediate form.

Anyway, this is all blue sky, based on my past pain; not sure how useful to your design process it is. Hope it helps, is all.

[schungx]

It shouldn't be a problem. Rhai would simply take your position info (regardless of ordering and whether it goes backwards) as the current position for the next token.

That means the AST parsed will contain positions that map exactly to the DSL source without necessarily needing to be strictly increasing.

Except that the source file info cannot be captured into the AST because it does not have it. Therefore it can only be line/col.

I toy with the idea of also having /*# ... #*/ for embedded position info but it seems quite easy to embed a newline for a line-based comment instead. What is your view on this?

[barries]

The /*#...#*/ would make it easier to read the generated code, I think; it becomes challenging when "logical" lines are broken across many physical lines. But /*#...#*/ is for sure syntactic sugar, however sweet. If it's easy given logic needed to support //#, seems like it'd be useful. If it's a lot of extra work, it's probably not worth it.

[barries]

Cool, just tripped over this in the overview, nice to see this use case is a first-class use case :).

• Scripts are optimized – useful for template-based machine-generated scripts.