Remove Lint.jl

[ZacLN]

I'm adding this as a medium term aim. The linter seems to cause more trouble than it's worth, I can't do any work without having it disabled and more often than not I'm uninterested in it's messages.

I've begun to strip/rewrite JuliaParser so that it's a bit less opaque and have started on implementing the lexer/parser that retains whitespace. It'll take a while but when it's done the parser in conjunction with the language server protocol 3.0 should be able to:

• provide a CST that can regenerate the source file.
• return specific (actionable) information on parse errors (i.e. error("expected 'end') -> insert end)
• be able to return some formatting hints (i.e. indent rules, spaces after commas, etc)
• keep track of the name space (a la namespace.jl) as it parses to warn of conflicting names, use of globals, etc

Eventually this should be able provide all the non-evaluated functionality that Lint.jl does.

[davidanthoff]

I completely agree, there seem to be a lot of false positives from Lint.jl...

I do think there is value in having a package that can lint a file outside of an IDE, eventually folks might want to integrate that e.g. in the build process. So if you end up writing a new linter that is based on a new JuliaParser, it might make sense to just keep that in mind and offer an option that this new linter can also be called and used independently from the LS?

I guess another option would be to improve Lint.jl and try to at least get rid of all these false positives, which I find the most annoying. Maybe we should just open issues over there and hope someone picks it up in the short run.

But completely on board with the medium term goal here.

[oxinabox]

@ZacLN For your interest I have a CST prototype based on Tokenizer.jl here: https://github.com/oxinabox/Expr2LaTeX.jl/blob/master/JuliaCST.ipynb

Reason I didn't start by stripping JuliaParser, is that it immediately discards all whitespace.

I'm not sure my prototype is directly useful to you, but poke me if you want to talk CST stuff.

[ZacLN]

Thanks @oxinabox, I'm pretty much done with the core parser now (just a matter of trawling through packages to pick up the odd parsing mismatch). I've ended up using pretty much the same patterns as JuliaParser (I think a recursive descent parser?).

Am I right in reading your approach as finding all boundaries of expressions before parsing them, i.e. top down in some sense? Does this make things easier in terms of incremental parsing and eliding syntax errors?

[oxinabox]

Yes, the approach was to get expression boundries, because what got me started was wanting to redisplay code as LaTeX, which can only be done on a per expression basis.
Expressions seems the logical building block, since you it is a "whole" thing.
Any expression can be evaled.
Any expression can be parsed to get an AST.

I am excited to hear that your CST Parser is basically done.
Where is the code for that? (I feel like is should be in its own package, no?)

[ZacLN]

https://github.com/ZacLN/Parser.jl is the repo.
Some basic functionality:

using Parser
str = readstring(joinpath(Pkg.dir("Tokenize"),"src","lexer.jl"))
x, ps = Parser.parse(ParseState(str), true) 

# convert to normal AST
x0 = Expr(x)

# Formatting hints
ps.hints
str2 = Parser.Hints.apply(ps.hints, str)

# formatting hints have no effect on AST
Expr(Parser.parse(str2, true)) == x0

# returns: the object, y, at byte 434
#              the stack of expressions that contain it, h
#              the sequence of nodes within these expressions, inds
y, h, inds = find(x, 434);

# Gets the scope at byte position 22514
S = Parser._get_full_scope(x, 22514);

# convert the names/signatures to standard AST, repetitions are where something is overwritten. 
(x->Expr(x.id)).(S)

EDIT: that example was broken, had to push a quick update

[ZacLN]

The internals are essentially the same as with normal AST except that we have IDENTIFIERs instead of symbols, LITERALS to capture all numbers/strings,etc so that all positional info can be stores. You can iterate across an EXPR in terms of the sub-expressions/lexemes in the order that they visually appear

[ZacLN]

#48