Add lexical information to scalar tokens in JSON documents

[antoniogarrote]

Right now, we are only emitting lexical information for 'object' nodes in the YAML and JSON ASTs.

The result of this limitation is that some nodes are not selectable in the web app if the AST element they come from is not a 'object' node in the ast.

For example:

schema: MyType

Will not generate lexical info. On the contrary:

schema:
  type: MyType

Will generate information.

We need to add this missing information into the AST we are generating. The information is generated by the parsers but since we are using Clojure meta-data to annotate the produced Clojure data structures with the location information, we cannot annotate scalar objects.

To overcome this limitation and fix the problem, we need to wrap scalar nodes into some data structure in the generated Clojure AST data structure.

Parsers will need then to unwrap these nodes and extract the scalar value and the lexical information for the token.

[postatum]

Antonio Garrote
You can test it in the interface when we try to select elements of the tree but we don't find lexical info and we cannot scroll
Don't worry about this issue, I'm planning to take care of it in the airplane
I will push changes in the stop-over in Toronto

[antoniogarrote]

Pushed my changes into the lexical info branch.
Unfortunately I couldn't finish it.

The JS parser now emits all tokens with lexical info, but some tests are still failing.
To give it a try, run the JS tests.

It required important changes in the parsers. Not a trivial modification.

[antoniogarrote]

Fixed for YAML/RAML. Only OpenAPI/JSON documents are missing lexical info.