Implement V8dotNet to make typescript-addin platform independent

[chrisber]

As a	Developer
I want to	use the typescript-addin on different platforms *nix/Osx/Windows
to	write Typescript code with Monodevelop

Done When

• Implement CsharpTypescriptLanguageService
  • Remove Typescript types from the project
  • Remove Json marshaling of return values
  • Update code base to Typescript 1.5
  • refs [Task] parser unit tests #7 Add unit tests

Using v8dotnet with the typescriptbinding
In this issue I will explain the assumptions, changes and thoughts that I made while investigating the support of v8dotnet for the typescript-binding. I will use FMC models to explain my thoughts. Feel free to correct me if I made a wrong assumption about something or share your opinion with me.

TypescriptBinding
I observed that the TypescriptBinding consists of the following building blocks: V8, Hosting, Provider and Project (MD) related belongings.

Providers

Provider seperate responisbilties of the Typescript language service (ts service api) The Typescript service API can be separated in different responsibilities like parsing, formatting, diagnostics, completion, info, compile. We implemented for each responsibilitiy one provider.

• Parsing
  • getNavigateToItems()
  • getNavigationBarItems()
  • cleanupSemanticCache()
  • dispose()
• Diagnostics
  • getSyntacticDiagnostics()
  • getSemanticDiagnostics()
  • getCompilerOptionsDiagnostics()
  • //returns comment, identifier, keyword,number,operator,string ,.. used for hilighting???
  • getSyntacticClassifications()
  • getSemanticClassifications()
• Completion TypeScriptCompletionItemProvider
  • Provides complition info when a user types a point after an object or uses strg+spacebar. The provider uses the Typescript context to call:
    • getCompletionsAtPosition()
    • getCompletionEntryDetails()
    • getQuickInfoAtPosition()
    • getNameOrDottedNameSpan()
    • getBreakpointStatementAtPosition() //Get the breakpoint span in given sourceFile
    • getSignatureHelpItems()
• Formatting
  • getFormattingEditsForRange()
  • getFormattingEditsForDocument()
  • getFormattingEditsAfterKeystroke()
• Info
  • getRenameInfo() //Information about the item to be renamed
  • findRenameLocations()
  • getDefinitionAtPosition()
  • getReferencesAtPosition()
  • getOccurrencesAtPosition()
  • getOutliningSpans()
  • getTodoComments()
  • getBraceMatchingAtPosition()
  • getIndentationAtPosition()
  • getSourceFile()
• Compile
  When it comes to compiling we have two possibilities. One solution is to use getEmitOutput() function which takes a filename returns the result, or we construct a compiler with getProgram() which takes a file and returns diagnostics. But then we have to implement an additional CompileHostEnvironment(). This is needed because the language service and the compiler using different host environments. The compiler uses:
  Compiler API , Host , EmitHost) , while the service uses TS API and Host.
  - getEmitOutput()
  - getProgram()
• Debugger
  • The debugger uses websockets to communicate with the chromium dev tools. We use it to obtain values, set and release breakpoints.
  • The debugger provider maybe is also responsible to start the KestrelHttpServer to serve web content.
  • The debugger should be able to use chromium dev tools to live reload on file save.

Note: On VS providers are executed on different threads threads

V8dotnet

V8dotnet is a wrapper around the chromium V8 project. Different then other similar projects like Javascript.Net it uses P/Invoke to focus on platform independence. V8dotnet is developed by James Wilkins and he is mostly the only maintainer of v8dotnet. V8dotnet has a similar syntax like the native V8 library. To start with v8donet I used the following resources.

• connecting-c-to-javascript-via-googles-v8
• https://v8dotnet.codeplex.com/discussions
• https://developers.google.com/v8/embed

An usage example of the of the v8dotnet library:

function foo() {
return 42;
}

Native v8:

HandleScope scope;
LocalContext context;

Script::Compile(String::New("function foo() { return 42; }"))->Run();

Local fun = Local::Cast(context->Global()->Get(String::New("foo")));

int argc = 0;
Handle argv[] = NULL;

Handle result = fun->Call(fun, argc, argv); // argc and argv are your standard arguments to a function

With V8dotnet:

Handle handle;
V8Engine v8Engine = new V8Engine();
int number = v8Engine.Execute(@"function foo() { return 42; }");

Or an native approach:

string number = v8Engine.Execute(@"function foo() { return 42; }");
InternalHandle handle = v8Engine.GlobalObject.GetProperty("foo");
var temp = handle.Call(null);

Calling an object

v8Engine.Execute(@"fooObj = { foo : function (bar) { return bar + 42; } }");
InternalHandle handle = v8Engine.GlobalObject.GetProperty("fooObj");

Handle varHandle = v8Engine.CreateValue("V8dotnet=");
var temp = handle.Call("foo",null,varHandle);

CsharpTypeScriptLanguageService
The CsharpTypeScriptLanguageService excites of 3 projects. TypeScriptLanguageServiceTranspiler,TypeScriptLanguageService and TypeScriptLanguageServiceTranspilerTest,
We will use the CsharpTypeScriptLanguageService to reflect the TypeScriptLanguageService API on the managed side.
While investigating V8dotnet and TypeScript and Monodevelop addin development. I recognized that the TypeScript API changes frequently. It is hard to observe which Types changed or what functionality was added. I decided to use TypeScript itself to generate classes/interfaces ( services.ts, types.ts ) that have c# syntax. This is done in the TypeScriptLanguageServiceTranspiler which is a rudimentary implementation to transpile typescript interfaces to c# classes. It also generates the TypeScriptLanguageService API from services.ts. The TypeScriptLanguageService uses an utility function to map the v8 context objects to managed objects. This is done with reflection in the TypeMapper() function.

Like Javascript.Net, v8dotnet is also able to use callbacks. We can just use an attribute on a function and v8donet is able to call the managed function. But this has the disadvantage that we need to annotate all classes which are part of the serialize and deserialize process (return and call parameter). One solution to avoid this problem is to use Jason.Stringify(), but serializing/deserializing is time consuming.

We have two possibilities to avoid serializing based on the same approach:

Static approach

public CompletionInfo GetCompletionsAtPosition( string filename, int offset){

//create call parameters
Handle filenameHandle = v8Engine.CreateValue(filename);
Handle positionHandle = v8Engine.CreateValue(offset);

//get language service object
var ls = v8Engine.GlobalObject.GetProperty("ls");
var resultHandle = ls.Call("getCompletionsAtPosition", null,filenameHandle,positionHandle);

bool isMemberCompletion = resultHandle.GetProperty("isMemberCompletion");
ObjectHandle entriesHandle = resultHandle.GetProperty("entries");
int arrayLength = entriesHandle.ArrayLength;
CompletionEntry[] entries = new CompletionEntry[arrayLength];
for (int i = 0; i < arrayLength; i++)
{
entries[i] = new CompletionEntry( 
entriesHandle.GetProperty(i).GetProperty("name"),
entriesHandle.GetProperty(i).GetProperty("kind"),
entriesHandle.GetProperty(i).GetProperty("kindModifiers")
);
}
return new CompletionInfo(isMemberCompletion, entries);
}         

This function constructs native call parameters and calls the getCompletionsAtPosition function of the Typescript language service. The return values are then marshaled into managed types. Here we are saving time because we don't need to serialize objects first.
To speed up development this can be implemented with an T4 template.

Second approach:

public CompletionInfo  GetCompletionsAtPosition (   string fileName,  int position) {

Handle fileNameHandle = v8Engine.CreateValue(fileName);

Handle positionHandle = v8Engine.CreateValue(position);
var resultHandle = languageService.Call("getCompletionsAtPosition", null ,fileNameHandle,positionHandle);
var result = utilities.TypeMapper<CompletionInfo>(resultHandle);
return result;

}

Since we already have classes we can use reflection to construct the object. The object obtains it's own properties and uses the handle to access the corresponding property in the v8 context. I read that reflection is also costly, when we need more performance we can abandon this approach and use the static generated approach, explained before.

There is also an issue with Uninon types, for instance var foo = string | string[ ]. This is easy we can just use string [ ] as property type and checking the length of the type to initializing it. But what can we do with this messageText: string | DiagnosticMessageChain; which comes from interface Diagnostics and DiagnosticMessageChain. Currently only the first type is selected.

Host environment

We have two host environments the managed host and the v8 related host.

• LangauageServiceHost.cs derived from ILanugageServiceHost
• V8LanguageServiceHost.cs

The managed host doesn't know anything about v8, while the V8LanguageServiceHost has an instance of the managed LangauageServiceHost
The TypescriptContext updates then the LanguageServiceHost with new files and file infos. Furthermore, the V8LanguageServiceHost calls the LanguageServiceHost to get this information.

IScriptSnapshotAdapter
This interface is used to exchange data between the managed LangauageServiceHost and the V8LanguageServiceHost to compute the get text change rate. Here we need an algorithm to calculate the exact line changes. Currently we search for the first line that changed.

CRLF vs LF
We need to make sure the all files have the same line endings before initializing the native typescript service. This option can not be changed afterwords.

Building the TypescriptBinding with CsharpTypeScriptLanguageService

• git clone

• git submodule update --init --recursive

  • if the remote contains changes
    • git submodule update --remote
  • if you want to revert submodule changes
  • git submodule init

  Project
  Monodevelop related stuff like addin-extensions for the project-options and monodevelop events.

  • which extension points do we use ?
  • We generated the CompilerOptions setter/getter for the project options but this needs more work.

  Naming
  There are a lot of service, typescript, shim host, parser names for objects and I introduced even more service, lanugeage ... I think it is good to have a naming convention.
  And Maybe a directory structure.

• Hosting

• Provider

  • Parser
  • Formating
  • Diagnostic
  • Complition
  • Debugger

• Binding // Extension points / gui related assets
  What about namespaces?

Unit testing
For easier unit testing we can use SimpleInjector for IoC which should be fast by design and work on different platforms, but is it usable within Monodevelop?

Switch bettween the config for windows ( x86) , linux and osx (x86)
CsharpTypeScriptLanguageService has currently no .nupkg. To use it, we have included the CsharpTypeScriptLanguageService to the TypescriptBinding project.

Change the current project config to windows:

• TypescriptBinding
  • use lib/V8dotnet/win.x86_x64/TypeScriptBinding.addin.xml
  • reference src/CSharpTypeScriptLanguageService/V8dotnet/win.x86_x64
    • V8.Net
    • V8.Net.Proxy.Interface
    • V8.Net.SharedTypes
• TypeScriptLanguageService
  • reference src/CSharpTypeScriptLanguageService/V8dotnet/win.x86_x64
    • V8.Net
    • V8.Net.Proxy.Interface
    • V8.Net.SharedTypes
  • add as content and mark copy to ouput
    • msvcp110.dll
    • msvcr110.dll
    • v8-ia32.dll
    • vccorlib110.dll
    • remove libV8_Net_Proxy.so

This libraries are costum builds of v8dotnet. V8dotnet uses an app start event to Load the dlls but this events get never called in mono.

contribution
This are some assumptions and ideas that I made while investigating v8dotnet for mono, if you like it would be nice if I can contribute something of this list to the project. We can start using gitter and hubord to keep track of things that needed to be done.

[mrward]

Being able to use the TypeScript addin in MonoDevelop cross platform would be fantastic. So thanks for looking into this.

I had a look at your fork of V8.NET and tried it on the Mac. It compiled but I could not get the console app to run. It was crashing with a EXC_BAD_ACCESS code=2 error when executing dump(this).

[chrisber]

You are right on this, saddly its a known issue, have a look at:

• v8dotnet osx
• v8dotnet linux
• gitter

Debugging the gives an exceptions at ObjectTemplateProxy::RegisterNamedPropertyHandlers

What I did so far is using the clang AddressSanitizer, it seams that there are no issues there, means no memory leaks are detected. What I will do next is printing the Handle addresses out on the managed site and then use lldb to observe if I can find something.

But in general I think v8dotnet will run in the future on Osx.

[chrisber]

Could you please have a look at the binaries that I have complied for OSX, seams to be working now. I also have created an addin for osx. I could not properly testing it from osx within vmware on an Ubuntu host because the GUI is not working fluently. Would be nice to have some feedback about the performance. I will also post some ideas tomorrow of the changes that I have made.

[mrward]

Are there any plans to support Linux x86? Would compiling the V8.NET libraries for x86 work for both x86 and x64?

[mrward]

I tried the V8.Net-Console.exe on the Mac and it looks good.

Not tried the TypeScript addin with these binaries. The latest release you have seems to be 12 days old and does not seem to work.

[chrisber]

I currently build v8dotnet on x86 and updated the releases. On windows I build the typescript addin with the type Any CPU and added the x86 libraries of v8dotnet. On Linux it doesn't allow me to choose which kind of library I want to use. It always complains about:

Mono: DllImport error loading library '/build/dump/v8dotnet_linux_x86/libV8_Net_Proxy.so': '/build/dump/v8dotnet_linux_x86/libV8_Net_Proxy.so: wrong ELF class: ELFCLASS32'.

Means I need to use the ia32.release build on x86 and x64.release on x64. I need to look further into this problem if there is a solution.

[mrward]

@chrisber - Just tried my port of the TypeScript addin over to V8.NET with your V8.NET binaries on the Mac and it works. Great work 😄

[chrisber]

This is nice hoping this will bring web development with Typescript+Angularjs and Asp vNext on Linux and Osx further.
I added my thoughts about the addin into the issue description. Would be nice if you could provide feedback about it. When you are allowing contributions to the addin I would be happy to implement some of these suggestions.

[chrisber]

For the problem to have a x86_x64 for Linux, this will not work see the answer here. I read something about dll maps in the mono docs. I will have a look at this.