Can not parse row data with a “SHIFT_JIS” SAS file.

[nixiejun]

Hi contributors:
I found my SAS file was not parsed correctly under "SHIFT_JIS" encoding.

This file seems has the compression method "SASYZCRL" however under this encoding the method name can not be recognized correctly. (The currengPageType is 0 so the subheader count is required not equal to 0.)

However , overriding "UTF-8" encoding to it I can import row data:
SasFileReaderImpl sasFileReader = new SasFileReaderImpl(fis, "UTF-8");
I found the compression method substring is correct.

[nixiejun]

@printsev @TimurMikhaliov
Could you please help to take a look? Thanks.

[printsev]

Hi @nixiejun -- would it be possible for you to share the source file? Then we would be able to take a look. Thanks.

[printsev]

we cannot do much here unfortunately without the data set so I'm putting the "nodataset" label to the issue

[FriedEgg]

Here are some datasets for this. I haven't got time to look into the issue more closely, but hopefully this is helpful to others.

charset-sjis.zip

[xantorohara]

This issue can be marked again as "nodataset".
Provided "charset-sjis.zip" contains a copy of existing file "https://github.com/epam/parso/blob/master/src/test/resources/sas7bdat/charset_sjis.sas7bdat", and Parso successfully reads it.
As I see, the problem reported by @nixiejun is in a combination of SASYZCRL compression and SHIFT_JIS encoding (looks like compression method name is broken in a first character after the parsing).

[xantorohara]

It is not correct to check compression method using String.contains() method over the header bytes converter to a multibyte string. Octets, related to different things, combined together and then converted to a multibyte string may form unpredictable characters.

See:

new String(new byte[]{-104, 'S', 'A', 'S', 'Y', 'Z', 'C', 'R', 'L'}, "Shift_JIS").contains("SASYZCRL"); //false for "牢ASYZCRL"
new String(new byte[]{-104, 'S', 'A', 'S', 'Y', 'Z', 'C', 'R', 'L'}, "UTF-8").contains("SASYZCRL"); //true for "�SASYZCRL"
new String(new byte[]{-104, 'S', 'A', 'S', 'Y', 'Z', 'C', 'R', 'L'}, "US-ASCII").contains("SASYZCRL"); //true for "�SASYZCRL"

So, the compression method like any other fields should be extracted from an explicit position.

[printsev]

closed thanks to xantorohara