BOM seems to confuse CS-Script

[UweKeim]

Having a "do-build-all.cs" file that starts like this:

//css_import ../_References/Submodules/zeta-shared-ci-assets/DevelopmentTools/CentralScripts/Afx.cs;

using ZetaSoftware.CentralScripts;

...

I've run it with version 4.8.18 of CS-Script but it always complained that "ZetaSoftware.CentralScripts" was unknown.

In fact I could even write gibberish in the very first line for the file path within the //css_import directive. No error that it cannot find the file, no nothing.

Out of ideas I converted the file from UTF-8 with BOM to UTF-8 without BOM. (BOM = Byte Order Mark).

Et voilà! It is working now.

Tried it multiple times by converting back and forth and it always behaves the same. As soon as there is a BOM, it does skip the //css_import directive.

Even adding a comment before as the very first line does not improve the behaviour.

My question

Could you please make CS-Script work with UTF-8 BOM files? 😊

[oleg-shilo]

Definitely.
Txs for reporting.
Will do.

[oleg-shilo]

Hey Uwe, I cannot reproduce it.
I have attached the working sample with the scripts in UTF-BOM:

Maybe you can share yours if somehow our editors do BOM inconsistently.

cs-script_#386.zip

[UweKeim]

Thanks, @oleg-shilo and sorry for not providing an initial example right from the start.

poc-389.zip

This example has two folders, one with UTF-8 with BOM, one with UTF-8 without BOM.

With BOM output:

c:\Ablage>c:\ablage\utf-8-with-bom\build-all.cmd

PUSHD

CD /d c:\Ablage\utf-8-with-bom\

dotnet tool update --global cs-script.cli
Das Tool "cs-script.cli" wurde mit der stabilen Version (Version "4.8.18" ) neu installiert.

css -config:set:LegacyNugetSupport=false
set: LegacyNugetSupport: False

css -nuget:restore "c:\Ablage\utf-8-with-bom\\do-build-all.cs"
Compile: 2 error(s)
c:\Ablage\utf-8-with-bom\do-build-all.cs(3,7): error CS0246:  Der Typ- oder Namespacename "ZetaSoftware" wurde nicht gefunden (m├Âglicherweise fehlt eine using-Direktive oder ein Assemblyverweis).
c:\Ablage\utf-8-with-bom\do-build-all.cs(3,7): error CS0246:  Der Typ- oder Namespacename "ZetaSoftware" wurde nicht gefunden (m├Âglicherweise fehlt eine using-Direktive oder ein Assemblyverweis).

css /dbg "c:\Ablage\utf-8-with-bom\\do-build-all.cs"
Error: Specified file could not be compiled.

c:\Ablage\utf-8-with-bom\do-build-all.cs(3,7): error CS0246:  Der Typ- oder Namespacename "ZetaSoftware" wurde nicht gefunden (m├Âglicherweise fehlt eine using-Direktive oder ein Assemblyverweis).
c:\Ablage\utf-8-with-bom\do-build-all.cs(3,7): error CS0246:  Der Typ- oder Namespacename "ZetaSoftware" wurde nicht gefunden (m├Âglicherweise fehlt eine using-Direktive oder ein Assemblyverweis).

Without BOM output:

c:\Ablage\utf-8-with-bom>c:\ablage\utf-8-without-bom\build-all.cmd

PUSHD

CD /d c:\Ablage\utf-8-without-bom\

dotnet tool update --global cs-script.cli
Das Tool "cs-script.cli" wurde mit der stabilen Version (Version "4.8.18" ) neu installiert.

css -config:set:LegacyNugetSupport=false
set: LegacyNugetSupport: False

css -nuget:restore "c:\Ablage\utf-8-without-bom\\do-build-all.cs"
Error: Specified file could not be compiled.

Cannot import "../_References/Submodules/zeta-shared-ci-assets/DevelopmentTools/CentralScripts/Afx.cs" from the "c:\Ablage\utf-8-without-bom\\do-build-all.cs" script.
Could not find file "../_References/Submodules/zeta-shared-ci-assets/DevelopmentTools/CentralScripts/Afx.cs".
Ensure it is in one of the CS-Script search/probing directories.

css /dbg "c:\Ablage\utf-8-without-bom\\do-build-all.cs"
Error: Specified file could not be compiled.

Cannot import "../_References/Submodules/zeta-shared-ci-assets/DevelopmentTools/CentralScripts/Afx.cs" from the "c:\Ablage\utf-8-without-bom\\do-build-all.cs" script.
Could not find file "../_References/Submodules/zeta-shared-ci-assets/DevelopmentTools/CentralScripts/Afx.cs".
Ensure it is in one of the CS-Script search/probing directories.

You can see that in the second call it correctly complains that it cannot find fhe //css_import-ed file whereas in the first example, it doesn't try at all to include.

[UweKeim]

Regarding your attached example, I can confirm that it works both with BOM and without BOM correctly.

So I'm really confused.

[oleg-shilo]

OK, I do have a clarity now.

Indeed there is a difference in the way editors implement Encoding.UTF8.GetPreamble() (encoding marker)
Your and my example are both recognized in Notepad++ as UTF8-BOM (see the prev post screenshot).

However, when I use Sublime it shows the files differently:

It's clear that notepad++ did not save the encoding marker correctly despite somehow indicating UTF8-BOM on the status bar.
Let's leave it for N++ team to solve.

Now, things are even trickier in the .NET land.
I could not believe my eyes when I saw this in the debugger:

But things become clearer when I found this section in the documentation
.NET starting from .NET4 stopped recognizing \uFEFF as a white character:

It explains why it upsets the CS-Script parser.

I will need to restore the original .NET interpretation of the white character. But in the meantime you can use this simple trick to full the system - insert a blank line at the top of the file.

Will do the patch release asap

[oleg-shilo]

Done. Release v4.8.19.0
NugetTool is ready but with Choco and Winget it will take a little time to index the release

[UweKeim]

Incredibly awesome! Thanks a lot, Oleg!

[oleg-shilo]

It was a nasty problem.
It just happens that you were the first one who discovered it since it was introduced in .NET 4.
It tells me that not that many people use BOM :)