Initial source-map support #675

Diggsey · 2015-02-23T21:18:58Z

This is still quite rough around the edges and could do with a lot more testing, but it works.

Pass "-m/--map" to lsc when compiling to generate source-maps.

vendethiel · 2015-02-23T21:41:43Z

Wow, very impressive work, nicely done :-).

Diggsey · 2015-02-23T21:50:47Z

Currently there's no way to specify the output filename when compiling and running livescript on-the-fly, so the sourcemap you get back won't work properly. For now it will only work when compiling livescript to .js files ahead of time. However, the sourcemaps for those .js files should work just fine in the browser if they are being served as static assets.

Diggsey · 2015-02-23T22:41:39Z

Having looked into it, it's seems that a trival change to the "liveify" plugin for browserify would enable this to work with browserify (as per coffeeify):
https://github.com/jnordberg/coffeeify/blob/master/index.js#L64

For the browser, coffeescript seems to use data-uris to specify the sourcemap, but I haven't figured out how it specifies the original source yet (or if it even does).

blvz · 2015-02-23T23:49:18Z

I've also quickly tried to make it work with browserify, with no luck. Then, I've tried to see if browserify could also bundle all generated maps, again, with no luck. I'll have to try again with a little more time.

Diggsey · 2015-02-23T23:54:18Z

Source maps should work in the browser now with no extra work:

<html>
<head>
</head>
<body>
<script type="text/javascript" src="browser/livescript.js"></script>
<script type="text/ls">
    debugger
    a = [1,2,3]
    b = [x*2 for x in a]
    if b[1] > 2
        console.log "Hello, world!"
</script>
<script type="text/javascript">
    require("LiveScript").go();
</script>
</body>
</html>

Diggsey · 2015-02-24T01:33:13Z

@blvz I've improved the command line options, so you can now pass "--map=embedded" which will embed both the source map and the original livescript inside the generated javascript file. This is the only form which browserify will accept AFAIK.

If you use that option, and also pass "--debug" to browserify, it will attempt to generate a combined source-map for the entire set of inputs. However, in my tests it's been generating invalid source-maps as output, and I have yet to figure out why.

…andles newlines correctly - Add debug output mode for source-maps

blvz · 2015-02-25T13:02:19Z

Very nice. Thanks for your effort. I've test it against a front-end app that uses mithril with some views deeply nested. Indeed, sourcemaps generated with browserify are slightly incorrect.

On Chrome Dev Tools, setting breakpoints in some-file.ls doesn't work, but setting them in bundle.js do. Of course, the breakpoint now points to the slightly wrong place in some-file.ls.

Enabling "pause on exceptions" made Dev Tools crash, but disabling it made the exception get printed in the console with correct file name and line number. Clicking on those, I could navigate to it and set a breakpoint, pausing the exception accordingly.

In the sample below, the correct file name and line number are shown in console, but navigating to it and setting a breakpoint fails.

...
  onclick: ->
    return if console.log \something # setting a breakpoint on this line is useless
...

gkz · 2015-02-25T21:30:31Z

Great work!
As soon as the kinks are worked out I am ready to merge this in!

Diggsey · 2015-02-25T22:10:57Z

At least part of the problem is due to the very old version of jison being used - I've been trying to update to a newer version, but that's proving difficult, as the jison project doesn't appear to maintain any release notes or even decent documentation, and LiveScript does some unusual things such as replacing the lexer and applying some regexes to the generated code, so would appreciate some help with that side of things (even just an explanation of what it's doing!).

The old version of jison causes two problems:

Multi-token grammar rules will pick up the line/column information for the last token in the rule, rather than the first
Column information is not tracked at all, so I had to hack that in by abusing one of the other fields

Additionally, my changes to the custom lexer in LiveScript to get it to track column information are still imperfect, although even the original line tracking was not perfect either. The problem is that they're tracked independently from the input source, and so it's very easy for them to get out of sync. Ideally it would be rewritten so that location information and input source are kept in lock-step at all times.

Once the information produced by the lexer is accurate and jison is updated, then the only possible source of errors will be within the source-node tree constructed by the compiler, and that can be improved incrementally over time, on a case-by-case basis, as it will for the most part be subjective, and is easy to change.

vendethiel · 2015-02-25T22:48:10Z

I think coffee updated the json version. That might prove interesting to check out.

Diggsey · 2015-02-26T04:44:11Z

Turned out that half the problem was that I'd updated to a buggy version of jison. With that solved, the update was successful, and the source-maps are MUCH better.

I've been using http://sokra.github.io/source-map-visualization/#custom to visualise the source maps if anyone's interested.

Diggsey · 2015-02-27T02:44:21Z

Sourcemap generation is pretty much perfect now (within the limits of v3 sourcemaps) at least for the common AST nodes.

gkz · 2015-02-27T07:29:32Z

Awesome, great work Diggsey!

raine · 2015-02-27T07:36:41Z

Can this be used to get stack traces with .ls source directly instead of having to compile and then check
the line numbers?

vendethiel · 2015-02-27T09:38:45Z

src/lexer.ls

@@ -292,9 +304,10 @@ exports import
             then @doHeredoc code, index, q
             else @strnum q+q; 2
    if q is \"
+      debugger


stray debugger here :)

Diggsey · 2015-02-27T11:01:45Z

@vendethiel Fixed!
@raine Yes. If you are running in a browser this should work off the bat, assuming you have sourcemaps enabled in the browser config. For node, you'll have to install the source-map-support npm package. In both cases you must tell lsc to generate source-maps using "--map=linked/linked-src/embedded" (unless you're compiling code on-the-fly in the browser, in which case it will generate source-maps by default).

igl · 2015-02-27T14:42:43Z

Awesome! Always bet on LS. :)

I am going to try this out with node-source-map-support tonight.

raine · 2015-02-27T20:37:42Z

Thank you @Diggsey, got the sourcemaps working and they seem accurate! Great work.

Here's a simple project with sourcemaps enabled if someone's interested.

Diggsey · 2015-03-06T17:32:03Z

@gkz Anything else need doing?

gkz · 2015-03-06T19:27:42Z

I just need to check it over and merge it in

dead-claudia · 2015-03-16T02:06:14Z

I'm already excited.

gkz · 2015-03-16T05:03:24Z

Sorry guys, I've been busy with work and school. I'll take a look at this in the coming week.

gkz · 2015-03-21T02:57:33Z

Works great! Awesome work @Diggsey!

vendethiel · 2015-03-21T03:07:10Z

🍰

awkay · 2015-04-05T23:55:59Z

This officially improves my chances of getting it adopted where I work! Thanks!

blvz · 2015-04-08T22:06:58Z

I didn't care much about it at first, but this is opening lots of opportunities, like code coverage:
gotwarlost/istanbul#212

So thanks again, @Diggsey.

gkz · 2015-04-10T01:03:41Z

Hi @Diggsey, I was wondering if you could give some comment on how the new source map functionality works. Ie. if I want to create a new ast node, what do I need to do? I see sn(this, ....args) used, anything special about how the args are laid out?

Diggsey · 2015-04-10T02:11:05Z

"sn(this, ...args)" creates a new sourcemap node: it takes the line and column information from 'this' (an AST node), and the text content for the node is given by recursively concatenating the content of sub-nodes passed in as '...args' (with plain js strings being the leaf nodes)

The aim when generating sourcenodes is to associate the location information of an AST node with precisely the javascript which that AST node generates, and it's a little more tricky than it sounds:

When generating source-nodes for the expression '1 + 2' in some simple language, it might involve two types of AST nodes, Literal and Addition. Literal is easy, it just returns "sn(this, value)", whereas Addition is more complicated: "sn(null, left, sn(this, ' + '), right)". The "null" means the sourcenode is just used for grouping - it doesn't supply any location information.

You might be tempted to just do this: "sn(this, left, ' + ', right)", and that's what I started out doing, but the problem is that "left" or "right" might contain code which is not associated with any location in the input - that code would get erroniously associated with the location information from 'this'.

gkz · 2015-04-15T02:50:53Z

Thanks for the info!

Would you be able to give a brief description of what the different options do (linked, linked-src, embedded, debug) to add to the docs in for the upcoming release?

Diggsey · 2015-04-15T12:12:05Z

Sure, there are three files involved when generating sourcemaps:
a) The original LiveScript source
b) The sourcemap
c) The generated javascript source

'a' can optionally be embedded inside 'b', and 'b' can optionally be embedded inside 'c' via a comment.

linked: No embedding, 'c' links to 'b' via a relative path, and same for 'b' to 'a'
linked-src: 'b' is embedded inside 'c', but 'a' is linked to
embedded: Everything is embedded inside 'c'
debug: Same as linked, but will also output a human readable representation of the source-node tree (similar to the output from the 'ast' option) to a '.map.debug' file.

Use linked or linked-src if you are directly serving the output from lsc to the browser - (ie. not performing further processing). They keep the original source separate, so the javascript file is still small. linked-src just means you have one fewer file to carry around at the expense of increasing the size of the javascript file.

Use embedded for everything else - it's self-contained and it's the only form most other tools such as browserify will accept as input. The file will be significantly larger, but you can remedy this by running a separate tool at the end of your build pipeline to split the output back into the 'linked' form.

dead-claudia · 2015-04-20T14:57:40Z

You can open a PR in the gh-pages branch. That's where this information is most useful.

gkz · 2015-04-26T21:50:13Z

I'll add the information to gh-pages, but I would appreciate if people who are more familiar with source-maps than me could add some tests using the --map option to https://github.com/gkz/LiveScript/blob/master/test/cli.ls

Diggsey added 3 commits February 21, 2015 04:03

Added code to track column indices through lexer and parser

60198cf

Basic source-map support

31d866c

Add missing package dependency

cba37cf

Source-maps now work in the browser

5f57b96

Improved command-line options for source-map generation

4b116f1

- Replace source-node -> source-map converter with custom one which h…

12de169

…andles newlines correctly - Add debug output mode for source-maps

Updated jison version, resulting in hugely improved source-maps

6002e18

Diggsey added 2 commits February 26, 2015 19:34

Improvements to source-map precision

90ceb92

General improvements to location tracking and source-node generation

791bbbc

vendethiel reviewed Feb 27, 2015
View reviewed changes

Remove stray "debugger" statement

07a463b

Remove mystery code which breaks line numbers

b68af4f

Minor improvements to source-map precision

ffc6f74

gkz merged commit ffc6f74 into gkz:master Mar 21, 2015

hzamani mentioned this pull request May 13, 2015

Source Maps appedemic/livescript-loader#3

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial source-map support #675

Initial source-map support #675

Diggsey commented Feb 23, 2015

vendethiel commented Feb 23, 2015

Diggsey commented Feb 23, 2015

Diggsey commented Feb 23, 2015

blvz commented Feb 23, 2015

Diggsey commented Feb 23, 2015

Diggsey commented Feb 24, 2015

blvz commented Feb 25, 2015

gkz commented Feb 25, 2015

Diggsey commented Feb 25, 2015

vendethiel commented Feb 25, 2015

Diggsey commented Feb 26, 2015

Diggsey commented Feb 27, 2015

gkz commented Feb 27, 2015

raine commented Feb 27, 2015

vendethiel Feb 27, 2015

Diggsey commented Feb 27, 2015

igl commented Feb 27, 2015

raine commented Feb 27, 2015

Diggsey commented Mar 6, 2015

gkz commented Mar 6, 2015

dead-claudia commented Mar 16, 2015

gkz commented Mar 16, 2015

gkz commented Mar 21, 2015

vendethiel commented Mar 21, 2015

awkay commented Apr 5, 2015

blvz commented Apr 8, 2015

gkz commented Apr 10, 2015

Diggsey commented Apr 10, 2015

gkz commented Apr 15, 2015

Diggsey commented Apr 15, 2015

dead-claudia commented Apr 20, 2015

gkz commented Apr 26, 2015

Initial source-map support #675

Initial source-map support #675

Conversation

Diggsey commented Feb 23, 2015

vendethiel commented Feb 23, 2015

Diggsey commented Feb 23, 2015

Diggsey commented Feb 23, 2015

blvz commented Feb 23, 2015

Diggsey commented Feb 23, 2015

Diggsey commented Feb 24, 2015

blvz commented Feb 25, 2015

gkz commented Feb 25, 2015

Diggsey commented Feb 25, 2015

vendethiel commented Feb 25, 2015

Diggsey commented Feb 26, 2015

Diggsey commented Feb 27, 2015

gkz commented Feb 27, 2015

raine commented Feb 27, 2015

vendethiel Feb 27, 2015

Choose a reason for hiding this comment

Diggsey commented Feb 27, 2015

igl commented Feb 27, 2015

raine commented Feb 27, 2015

Diggsey commented Mar 6, 2015

gkz commented Mar 6, 2015

dead-claudia commented Mar 16, 2015

gkz commented Mar 16, 2015

gkz commented Mar 21, 2015

vendethiel commented Mar 21, 2015

awkay commented Apr 5, 2015

blvz commented Apr 8, 2015

gkz commented Apr 10, 2015

Diggsey commented Apr 10, 2015

gkz commented Apr 15, 2015

Diggsey commented Apr 15, 2015

dead-claudia commented Apr 20, 2015

gkz commented Apr 26, 2015