invalid byte sequence in US-ASCII #33

JeffLuckett · 2018-04-06T13:59:50Z

Environment Info:
MacOS High Sierra
VS Code: 1.22.1 (1.22.1)
Solargraph Gem: 0.18.1
Ruby Solargraph Plugin: 0.14.1
Ruby: 2.4.1p111 (Large-ish Rails app)

Let me know if you need more info, or if you have any suggestions.

castwide · 2018-04-06T17:46:36Z

Is there a particular file that gives you this error? The most likely cause is a literal string with escaped characters, e.g., "\xcf".

I'm not able to reproduce this error in Windows. I'll try on MacOS later.

JeffLuckett · 2018-04-07T13:19:18Z

@castwide - I'm guessing that we've got something like that somewhere in the code base, but unfortunately the error message just isn't specific about where the error occurred.

The project is ~250K lines of code in ~2K .rb files, not including gems, etc...

castwide · 2018-04-07T13:44:26Z

No problem. Even in the unlikely event that I can't reproduce the error, at least I can add the name of the offending file to the message.

TheTharin · 2018-04-10T13:23:50Z

But is there a way to ignore it in the future? I mean I can't just go and add #encoding to any file in the project solargraph finds offending.

castwide · 2018-04-10T13:32:33Z

Yes, I think I can make the parser ignore it. There are already similar character encoding issues that it's able to resolve. I'll still make it report the file name in the event of an edge case it can't handle.

castwide · 2018-04-10T18:39:28Z

Reproduced on MacOS. I'm looking into solutions.

castwide · 2018-04-10T19:44:19Z

A fix is pushed to the master branch. The parser should never fail. Even if the file isn't valid Ruby code, Solargraph should add it to the workspace and report the problems in diagnostics. Character encoding should be a non-issue. It's possible for Ruby code to contain string literals with invalid characters, but Solargraph shouldn't care as long as it doesn't try to write back to the file, which it never does.

I'll update the gem in the next day or so.

castwide · 2018-04-11T13:45:34Z

Version 0.18.3 is published.

JeffLuckett · 2018-04-12T14:28:46Z

Thanks @castwide, instead of the original error, I now receive [NoMethodError] undefined method 'node' for nil:NilClass

Would you prefer I open a new issue on this?

castwide · 2018-04-17T16:39:16Z

@JeffLuckett Are you still getting that error in the current version (0.19.1)?

JeffLuckett · 2018-04-18T17:16:06Z

@castwide - no, now I get:

However, I've only got about 2K files in the repo that are .rb files.

=> git ls-files | grep .rb | wc -l
    1981

castwide · 2018-04-19T02:00:18Z

Do you have a .solargraph.yml file? If so, does the include section select files that do not have the .rb extension?

JeffLuckett · 2018-04-19T12:46:05Z

There is no .solargraph.yml file in my project root, or in my home dir.

castwide · 2018-04-19T13:41:29Z

Is the project root your open folder in VSCode? Solargraph treats whatever folder you open as the workspace.

Does the project have cache or vendor/cache directories, or something similar that contains bundled gems? If so, you might need to create a .solargraph.yml and add them to the exclude section. You can generate a default config by running solargraph config from the command line or selecting Create a Solargraph config file from the VSCode command palette.

Also, this command should give you a more accurate idea of what gets added to the workspace:

find -name "*.rb" -not -path "./spec/*" -not -path "./test/*" | wc -l

The default is to include all ruby files except the spec and test folders.

JeffLuckett · 2018-04-19T14:28:08Z

I did solargraph config and then added - vendor/**/* to the excludes. That seems to have made it happy.

Thanks for sticking with me on this :D

castwide · 2018-04-19T14:45:38Z

Glad you got it working. This made me think of two more items for the to-do list.

Add vendor to the list of folders excluded by default.
Implement a solargraph subcommand called something like ls-workspace that lists all the files that would be mapped based on the current configuration.

JeffLuckett · 2018-04-26T16:45:33Z

This issue is resolved, thanks again ... closing.

AeroCross · 2018-05-14T05:15:08Z

Hi! I was wondering if I found an edge case.

I'm using Solargraph's 0.21 version and I am still getting this error. This is a huge codebase though (about 9k Ruby files). I've reduced the count to less than 5000 to make the server work, but I still struggle to get it working, since I'm finding a lot of non-ASCII characters in form of comments, tests, etc.

After looking for a way to check what non ASCII characters are present in my codebase, I got the following:

$ ag --ruby "[\x80-\xFF]" . --stats-only
3382 matches
149 files contained matches
9451 files searched
19659019 bytes searched
0.363241 seconds

The stuff I found are characters like ’, ™, ö, etc. In other codebase I work on (which doesn't have non-ASCII characters), this doesn't occur.

These are contained in Ruby files, so I can't just get rid of them (we're specifically testing for issues with those characters, so I can't change the code to remove that).

I also notice that autocompletion still works, but it works in a very barebones way — no inline documentation, no differentiation between methods, constants, variables, etc., no symbol information for Ruby files, etc.).

I am using VS Code, if this helps, on Mac OS X 10.12.6.

$ code --version
1.23.1
d0182c3417d225529c6d5ad24b7572815d0de9ac
x64

$ solargraph --version
0.21.0

747 · 2018-05-19T05:42:40Z

Similar error. I started to get "Invalid byte sequence in Windows-31J" after some recent update.

[Error - 14:29:36] Server initialization failed.
  Message: [ArgumentError] invalid byte sequence in Windows-31J
  Code: -32603

> solargraph -v
0.21.1
> code --version
1.23.1
d0182c3417d225529c6d5ad24b7572815d0de9ac
x64
> [System.Environment]::OSVersion

Platform ServicePack Version      VersionString
-------- ----------- -------      -------------
 Win32NT             10.0.16299.0 Microsoft Windows NT 10.0.16299.0

castwide · 2018-05-28T22:38:32Z

@747 I started #63 to track character encoding issues.

tekknovator · 2019-02-21T05:46:04Z

Hi!
#63 has been closed but the issues persist. I work with scripts generating some bash with osascript calls (among other stuff) wich requires "¬" characters at some line endings. Unfortunately there is no work around that.
So solargraph keeps failing with
"Failed to start Solargraph: Error: [ArgumentError] invalid byte sequence in UTF-8"
I might just add encode or switch back to rubyLocate. Solargraph is (feels) faster though, would be nice to have.
Any ideas anyone?

castwide · 2019-02-21T15:50:46Z

@tekknovator Can you provide a minimal example that triggers the error? I've found at least one bug in how certain UTF-8 characters get handled (including "¬"), but it doesn't make the server fail to start.

dwarfi09 · 2019-03-18T14:37:15Z

Problem seems to appear again with version 0.31.0 of solargraph gem.

I downgraded to version 0.30.2 and solargraph-server starts inside atom.

AlanWarren · 2019-03-29T19:05:13Z

Problem seems to appear again with version 0.31.0 of solargraph gem.

I downgraded to version 0.30.2 and solargraph-server starts inside atom.

Thanks, I experienced this with a rails project today using solargraph-0.31.3. Downgrading to 0.30.2 fixed the issue.

Is there anyway to view output of which files solargraph is parsing when it hits the invalid byte sequence and crashes?

castwide · 2019-03-30T20:00:54Z

@AlanWarren I'm working on an update that will report which file triggered the error. Hopefully it will also be able to skip offending files and parse the rest of the workspace.

Can anyone who's experiencing this error let me know if there's a backtrace or any additional information in the console log? (In VS Code, Help -> Toggle Developer Tools; in Atom, View -> Developer -> Toggle Developer Tools.)

dwarfi09 · 2019-03-31T14:58:54Z

@castwide I just sent the console-output via email to you.

castwide · 2019-04-02T19:55:58Z

Thanks, @kkneutgen. That was exactly what I needed. I was able to trace the problem to a regular expression match that choked on invalid character encoding.

To reproduce the error, I created a file that contained UTF-8 (e.g., '¬'), converted its contents to ANSI with an external tool, and ran the file through the Solargraph source mapper.

I think I'll be able to have a fix for this in the next minor release.

castwide · 2019-04-03T14:25:16Z

The master branch has an update that might resolve this issue. It fixes the problem in the variation of the error I reproduced yesterday.

castwide · 2019-04-03T14:52:24Z

One more change: exceptions during the mapping process will emit warnings that identify the file where the problem occurred.

castwide · 2019-04-04T11:00:20Z

Gem v0.32.0 includes the latest fix for invalid byte sequences and the new mapper exception handling.

dwarfi09 · 2019-04-04T15:30:01Z

Great, thank you. No more problems with strange encodings here.

tekknovator · 2019-05-16T17:37:32Z

Works here as well now. Sorry for not responding earlier. I used the characters inside of a heredoc.

sent-hil · 2019-11-08T05:27:43Z

I'm getting the same error in version 0.37.2. Only version I can get to work is 0.31.3.

~/w/f/bevel:develop$ code --version
1.40.0
86405ea23e3937316009fc27c9361deee66ffbf5
x64

ag --ruby "[\x80-\xFF]" . --stats-only
265 matches
11 files contained matches
1519 files searched
2217378 bytes searched
0.041453 seconds

ruby --version
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin18]

castwide · 2019-11-08T19:18:00Z

@sent-hil Can you give me a simple example of a file that causes the error, the character set it uses, and a sequence of steps to cause it if it doesn't happen on startup? I haven't been able to reproduce it.

apsoto · 2020-11-01T06:02:21Z

I see invalid byte sequence in UTF-8 (ArgumentError) with the following text

# DÌaz

Example:

$ solargraph --version
0.39.17
[/tmp/foo git:() ]
$ ls
fails.rb*
[/tmp/foo git:() ]
$ cat fails.rb
# D�az

[/tmp/foo git:() ]
$ solargraph scan
Traceback (most recent call last):
	15: from /Users/apsoto/.rvm/gems/ruby-2.7.0/bin/solargraph:5:in `<main>'
	14: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
	13: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
	12: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
	11: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
	10: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/shell.rb:172:in `scan'
	 9: from /Users/apsoto/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/benchmark.rb:293:in `measure'
	 8: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/shell.rb:173:in `block in scan'
	 7: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/api_map.rb:170:in `load'
	 6: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/api_map.rb:67:in `catalog'
	 5: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/api_map.rb:67:in `each'
	 4: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/api_map.rb:85:in `block in catalog'
	 3: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/source_map.rb:156:in `map'
	 2: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/source_map/mapper.rb:50:in `map'
	 1: from /Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/source_map/mapper.rb:27:in `map'
/Users/apsoto/.rvm/gems/ruby-2.7.0/gems/solargraph-0.39.17/lib/solargraph/source_map/mapper.rb:201:in `process_comment_directives': invalid byte sequence in UTF-8 (ArgumentError)

bubbavox · 2021-02-03T01:11:44Z

For what it's worth to other googlers, I was getting the following error in VS-Code, with Solargraph 0.40.2:
Failed to start Solargraph: Error: [ArgumentError] invalid byte sequence in UTF-8

And I traced the problem to a ruby file containing a non-ASCII character in a comment - �
This ruby file wasn't even open in VSCode -- it was just in a folder within the group of folders open in my VSCode workspace.

castwide mentioned this issue Apr 6, 2018

"invalid byte sequence in US-ASCII" on latest version castwide/vscode-solargraph#37

Closed

castwide mentioned this issue Apr 10, 2018

Solargraph seems to handle go-to-definition/peek-definition too slow castwide/vscode-solargraph#39

Closed

JeffLuckett closed this as completed Apr 26, 2018

castwide reopened this Feb 22, 2019

castwide added the help wanted label Feb 22, 2019

castwide mentioned this issue Feb 22, 2019

Invalid Byte Sequence #84

Closed

castwide added a commit that referenced this issue Apr 3, 2019

Source and Mapper handle invalid UTF-8 byte sequences (#33)

14ba14d

castwide closed this as completed May 9, 2019

henriquetatagiba mentioned this issue May 6, 2020

Formatter running infinetly castwide/vscode-solargraph#173

Closed

invalid byte sequence in US-ASCII #33

invalid byte sequence in US-ASCII #33

Comments

JeffLuckett commented Apr 6, 2018

castwide commented Apr 6, 2018

JeffLuckett commented Apr 7, 2018 • edited Loading

castwide commented Apr 7, 2018

TheTharin commented Apr 10, 2018 • edited Loading

castwide commented Apr 10, 2018

castwide commented Apr 10, 2018

castwide commented Apr 10, 2018

castwide commented Apr 11, 2018

JeffLuckett commented Apr 12, 2018 • edited Loading

castwide commented Apr 17, 2018

JeffLuckett commented Apr 18, 2018

castwide commented Apr 19, 2018

JeffLuckett commented Apr 19, 2018

castwide commented Apr 19, 2018

JeffLuckett commented Apr 19, 2018

castwide commented Apr 19, 2018

JeffLuckett commented Apr 26, 2018

AeroCross commented May 14, 2018 • edited Loading

747 commented May 19, 2018 • edited Loading

castwide commented May 28, 2018

tekknovator commented Feb 21, 2019

castwide commented Feb 21, 2019

dwarfi09 commented Mar 18, 2019

AlanWarren commented Mar 29, 2019

castwide commented Mar 30, 2019

dwarfi09 commented Mar 31, 2019

castwide commented Apr 2, 2019

castwide commented Apr 3, 2019

castwide commented Apr 3, 2019

castwide commented Apr 4, 2019

dwarfi09 commented Apr 4, 2019

tekknovator commented May 16, 2019

sent-hil commented Nov 8, 2019

castwide commented Nov 8, 2019

apsoto commented Nov 1, 2020

bubbavox commented Feb 3, 2021 • edited Loading

JeffLuckett commented Apr 7, 2018 •

edited

Loading

TheTharin commented Apr 10, 2018 •

edited

Loading

JeffLuckett commented Apr 12, 2018 •

edited

Loading

AeroCross commented May 14, 2018 •

edited

Loading

747 commented May 19, 2018 •

edited

Loading

bubbavox commented Feb 3, 2021 •

edited

Loading