CRLF is still used as EOL in Windows

[muzimuzhi]

Currently when run in Windows, l3build save still uses CRLF as end-of-line in generated .tlg, despite the existing EOL normalizations in rewrite() (line 95 below)

l3build/l3build-check.lua

Lines 93 to 102 in 2c9e178

	local function rewrite(source,result,processor,...)
	local file = assert(open(source,"rb"))
	local content = gsub(file:read("*all") .. "\n","\r\n","\n")
	close(file)
	local new_content = processor(content,...)
	local newfile = open(result,"w")
	output(newfile)
	write(new_content)
	close(newfile)
	end

This brings inconvenience to the cooperation between Win and non-Win users, see for example lvjr/tabularray#385 (comment).

I did some tests in muzimuzhi/hello-github-actions#6 and found

• it's io.write() that converts \n to \r\n automatically in Windows and
• it seems opening the file in binary mode (open(source, "rb")) does the trick.

[josephwright]

I'm not following here. Saved .tlg files should have the right line endings for the platform being used, so CRLF for Windows. That's what happens as far as I know, and it works fine for us (@u-fischer always uses Windows, I have a mixed setup, others only using *nix). What behaviour are you expecting that's different from the present situation?

[davidcarlisle]

@josephwright I think the problem depends on how you configure your source control. if git checks out local files with windows line endings you want to save tlg with windows endings but if you configure git to not convert then you want to write the same (nix) format on all platforms otherwise files saved on windows have crnl in the repo and fail on the CI

[josephwright]

@davidcarlisle But isn't that the same as any other method where you've decided not to follow the system setting: managing line endings is down to you? I'm not sure how l3build is supposed to know to do anything non-standard - it's not dependent on the development set-up per se rather the choice of an individual when getting a set of files.

[davidcarlisle]

@josephwright not sure either is "non standard" git (like svn and cvs before it) have a boolean option to translate or not line ends. Certainly here we configure source control not to change line endings (which really messes things up if you have a shared file system) l3build would know if it had a boolean option settable in build.lua saveusinglocaleol=false or whatever.

[josephwright]

@josephwright not sure either is "non standard" git (like svn and cvs before it) have a boolean option to translate or not line ends. Certainly here we configure source control not to change line endings (which really messes things up if you have a shared file system) l3build would know if it had a boolean option settable in build.lua saveusinglocaleol=false or whatever.

I meant 'non-standard' as in 'not what the system plain text editor would do' (so Notepad on Windows, TextEdit on macOS, various possibles for Linux).

My worry with a control variable is it suggests the behaviour depends on the module development code, whereas it depends on the practice of the person working on the code. So it's really 'machine-dependent'.

[FrankMittelbach]

correct me if I'm wrong, but assuming there would be a control variable test-linefeed-behavior, say, and it would accept the values system, lf, crlf, cr then regardless of what I set them to, save and check on a single system should work just fine. What it does if a source control system comes into play then depends on how I set that up and I have to make sure that the setting works together. And it should also work fine if you have a shared file system to store the stuff and developers on different platforms would use save/check as long as all of them set the variable to the same setting (other than "system").

[davidcarlisle]

@josephwright well yes it's come up before that some settings might want to be personal and set from lua outside the repo (upload emails etc) But an option that is not always right seems better than a fixed setting that's always wrong (on windows with core.autocrlf off) a build.lua for a project with users with different settings could set the option after testing git config --get core.autocrlf .

Currently if the checked in files have \n line endings and you save on windows you get the "wrong" endings and rely on git to further normalise on commit.

[josephwright]

Well if the general feeling is it makes sense, I will of course sort it. I guess do a last-minute replacement of \n with <requested-ended> and use rb. So probably don't use a switch but rather a var which defaults to the system default?

[muzimuzhi]

Hmm I kinda changed my idea.

It seems by convention either inherited or system default EOLs are used by programs generating files that would be tracked by a version control software, and setting .gitattributes is a (if not "the") recommended solution, for example latex package author can add *.tlg text eol=lf to .gitattributes.

I consulted two front-end tools, `babel` and `npm`.

• From a babel issue opened in Oct 2018,

  Babel doesn't keep any whitespace information around after the file is parsed. In general we're reducing the amount of control over output formatting that we expose, because all we really want to guarantee is that the file we output will run properly, and CR/LF values don't affect that.
  CRLF line endings are converted to LF line endings. babel/babel#8921 (comment)

  We likely preserve CRLF in the output for literal tokens that contained newlines in the original source, like if you have CRLF in a comment or a template literal and don't transform them somehow, I think they would pass through with CRLF values still.
  CRLF line endings are converted to LF line endings. babel/babel#8921 (comment)

• From a npm PR opened and merged in Feb 2018,

  This also improves performance on Unix systems since it avoids replacing newlines when it's not necessary.

  Tests have been added to test that package[-lock].json files are written to disk with their original line endings.
  Consolidate code for stringifying package data and add tests npm/npm#19907

In the sense of inheritance, l3build may also be requested to respect existing EOLs in each test output when saving new outputs (so the first wins). Then adding a new variable (as suggested in current issue) to do project-wide control looks easier and more efficient.

[muzimuzhi]

Supporting info: Code formatter prettier has an option endOfLine: "<lf|crlf|cr|auto>" with default value lf. https://prettier.io/docs/en/options.html#end-of-line

[josephwright]

Looking back, I think handling this with some 'local config' makes sense, but I wonder how we make sure such a file can only be used for appropriate variables, e.g. yes for line ending but no for anything repo-dependent.