Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent powershell from adding a BOM to test input #3595

Merged
merged 10 commits into from
Apr 7, 2022
Merged

Prevent powershell from adding a BOM to test input #3595

merged 10 commits into from
Apr 7, 2022

Conversation

martin-frbg
Copy link
Collaborator

@martin-frbg martin-frbg commented Apr 5, 2022

fixes #3594 as suggested by mmuetzel

@mmuetzel
Copy link
Contributor

mmuetzel commented Apr 6, 2022

I'm not sure I understand what the actual issue is. But the following PowerShell command should read a file (with or without BOM) and write it without BOM (as long as those files only contain ASCII characters):

Get-Content .\test_file.txt -Encoding UTF8 | Set-Content .\test_file_no_BOM.txt -Encoding ASCII

@martin-frbg
Copy link
Collaborator Author

The problem is that on "some" Windows hosts in Azure, powershell´s Get-Content prepends a BOM to anything it pipes into an external program, and none of the various internet recipes I tried got it to stop doing that. This behaviour breaks some of the BLAS tests where the name of the output file to create is written as the first parameter in the input file for the test. As the (Fortran) test code does not support UTF, it simply adds the character pattern of the BOM to the filename, the test succeeds but the subsequent check does not find the expected file and reports a failure.
What I tried here is also destined to fail, as the BOM is not there (yet) when piping to the built-in replace command, but gets added in the next stage. #3598 has a different approach where I just try to replace the non-portable file command with something equally external but always available on Windows.

@mmuetzel
Copy link
Contributor

mmuetzel commented Apr 7, 2022

By default on the system I'm testing, the input encoding is set to ibm850. That might be the reason I didn't see this locally.
I think I can reproduce this if I set the input encoding of the console to UTF-8 with BOM with this PowerShell command:

[Console]::InputEncoding = New-Object Text.UTF8Encoding $true

It might be a good idea to unconditionally set the console input encoding to UTF-8 without BOM in that .ps1 script. That way the pipes should behave pretty similarly to what they do on modern Linux:

[Console]::InputEncoding = New-Object Text.UTF8Encoding $false

@martin-frbg
Copy link
Collaborator Author

martin-frbg commented Apr 7, 2022

Thank you for the suggestion - it seems this was not among the many variations I tried in #3563.

@martin-frbg martin-frbg changed the title Strip any spurious BOM from test input Prevent powershell from adding a BOM to test input Apr 7, 2022
@martin-frbg martin-frbg merged commit 145ccf8 into OpenMathLib:develop Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CTest failure with windows build
2 participants