The Hard Tabs Issue

[basmith]

Hi,

(This report is based on the v0.1.1 Win64 binary artifact from the Zig website.)

I noticed that if I create a Zig source file in Windows with a native editor (eg Notepad), the compiler complains about line endings:

$ zig build-exe hello.zig
':\code\zig\first\hello.zig:1:30: error: invalid character: '
const io = @import("std").io;
                             ^

If I manually kill the newlines (resulting in the code being all on one line) it compiles.

I tried using Vim in a Cygwin shell and the file it wrote also compiled without complaint (presumably Unix-style newlines, as Notepad renders that file on one line while Vim looks correct).

[thejoshwolfe]

You need to configure your editor to use unix line endings to write Zig code. Additionally, you need to configure your editor to use spaces instead of hard tabs for indentation.

notepad.exe has neither of these features, and it can't even comprehend unix line endings. This has been a long standing bug/missing feature in the windows default plain text editor. Notepad is in fact so deficient as a text editor, that literally every single other text editor in popular use today can comprehend unix line endings. Notepad is the worst text editor in popular use, and has been for decades. Zig will not bend to accommodate Microsoft's gross incompetence or nefarious stunts in their inability or unwillingness to provide a decent default text editor to their paying consumer base. Notepad is the problem here, not Zig. (It's not just me. Here's other angry people complaining about Notepad.)

The rationale for only supporting unix line endings and no hard tabs is part of the "only one obvious way to do things" philosophy. From a practical perspective, never having windows line endings makes it easier to write tools that read zig source files. For example, a tool that searches for "\n\n// TODO" and replaces it with something else that includes newlines: it's much easier to do this without worrying about newline style. Furthermore, git and svn have strange features that convert newline styles at odd times, and now all that's irrelevant for Zig.

Variable newline style and variable indentation style are features that Zig does not support.

Is this documented anywhere, or are users just expected to run into cryptic errors like this? Like a CR character doesn't even print properly in a terminal.

[hasenj]

I'm not sure if Zig does this on purpose or it was overlooked, but as far as I'm concerned this is a good feature. Different styles of indentation and line endings cause endless headaches when working on a collaborative project, for example using source control software such as git, etc.

I know notepad is the default text editor in windows, but nearly all developers use something else to write code, such as notepad++ or visual studio.

VS Code is also a good option. It's developed by Microsoft, and it's free.

[PavelVozenilek]

Nim does it right:

Any of the standard platform line termination sequences can be used - the Unix form using ASCII LF (linefeed), the Windows form using the ASCII sequence CR LF (return followed by linefeed), or the old Macintosh form using the ASCII CR (return) character. All of these forms can be used equally, regardless of platform.

Multiline string should insert LF newlines, as in C. If someone wants CR he could add it via \r.

Correct newlines are is not just problem of stupid Notepad: if one copy pastes example from webpage he gets CR/LF too. Imagine someone failing with Hello World.

[thejoshwolfe]

if one copy pastes example from webpage he gets CR/LF too.

Pasting into what editor does that?

[PavelVozenilek]

@thejoshwolfe: Notepad, Sublime Text 2 and probably anything else. I do not know Windows editor which by default uses Unix line ending and converts to this style automatically.

[thejoshwolfe]

@PavelVozenilek Are you saying that when an editor is configured to use unix line endings or is editing a file that already has consistent unix line endings, then pasting from a webbrowser inserts the wrong kind of line ending? Or are you just saying that windows line endings are typically the default line ending style before you configure it in your editor?

[PavelVozenilek]

@thejoshwolfe: the editors I know (VC++, Sublime Text 2, Notepad ...) do not have configuration option to force Unix ending everywhere from now. At the best one switch it manually file by file.

Programable editors like vim, probably, but I tend to avoid tools smarter than me.

I do not understand why this is even a problem. Line ending chaos is real, it won't go away, pragmatic solution (accept all) is easy and then the mess disappears from the view of ordinary user.

[kyle-github]

@thejoshwolfe: I think it is the browser that does this, not the editor. HTML is defined as using \r\n not \n. Most browsers let you get away with it on input, but when you copy and paste I think it recreates "correct" HTML from the DOM. Not sure about this, but I have run into the problem consistently.

I think @PavelVozenilek has a point. Every useful editor can manage to translate the line endings just fine but few allow you to do it at a project level and change everything automatically. However, the two main platforms, Windows and Mac, do not use the line ending convention that Zig uses. I happen to use Linux, but that is a minority platform.

I also tend to like the use of tools like go-fmt simply because it completely eliminates an entire class of bike-shedding. I've wasted too much time fighting about formats over the years. It is not a winnable war unless you create something like go-fmt.

[thejoshwolfe]

I just did some experimentation on my Windows machine. Here's what I found:

Eclipse and Notepad++ normalize line endings when you paste text into a file. Each file is determined to be in a particular style, and anything you put into the file through typing or pasting gets normalized to that style.

In Visual Studio, when you press enter, it uses the newline style of the lines around your cursor. When you paste code with CRLF line endings into any file, you get CRLF line endings for the text that you're pasting without affecting the surrounding text. If you save the file, it saves with mixed line endings without warning you. (You can convert line endings while saving through the "Advanced Save Options".) Visual Studio has no option to automatically normalize line endings on paste or on save. If you want normalized line endings, you gotta do it every time you save.

When you open a file in Visual Studio that has mixed line endings, you get a dialog that prompts you to normalize all the line endings to one style or another.

This is not a bug thread for Visual Studio, but that is a Visual Studio bug. Why would they let you create mixed line endings without warning, but then warn you when you open a file that has mixed line endings? This leads to a "best practice" where you should close and reopen all your files before making a commit to make sure you're not committing files that will produce warnings, which is just silly. This is a bug/missing feature in Visual Studio.

I don't know about Sublime Text; it's not free.

Meanwhile in Linux, copying text out of a web browser always seems to result in unix line endings, not windows line endings. I don't know where you're getting the idea that HTML uses windows line endings; I don't see it in the spec. Maybe you mean HTTP headers? There are parts of the HTML spec that talk about normalizing to CRLF, but I can't figure out how to observe that as an end user. I tried copy-paste and drag-drop text from a Google search page and from the textarea editor I'm typing this in right now, but I always got unix line endings (tested in Chrome).

The so-called "Mac"-style line ending style actually refers to pre-2005 "classic" Mac OS-9 line endings. Modern Mac uses Unix line endings, just like Linux.

[thejoshwolfe]

I do not understand why this is even a problem. Line ending chaos is real, it won't go away, pragmatic solution (accept all) is easy and then the mess disappears from the view of ordinary user.

This kind of reasoning leads to JavaScript's automatic semicolon insertion. This kind of reasoning has proven to be very successful at getting widespread adoption. This kind of reasoning is also contrary to the Zen of Zig. In Zig, the code author is required to do more work so that code readers are required to do less work.

I also tend to like the use of tools like go-fmt simply because it completely eliminates an entire class of bike-shedding. I've wasted too much time fighting about formats over the years. It is not a winnable war unless you create something like go-fmt.

Some kind of zig-fmt tool is definitely within the scope of what we want to create. Some plans so far are to make the Zig compiler outright reject any source files that would be modified by zig-fmt. This not only establishes a clear precedent, but forces everyone to use it, or else your code won't compile, not even in debug mode. Whole classes of bikeshedding are gone with this strategy, and all working Zig code has a consistent style. This is already partly the case as we're discussing here, although there's no zig-fmt to fix these problems for you yet. But since the only formatting that's currently rejected is '\r' and '\t' characters, you can pretty trivially clean these up. (You could use this tool, for example.)

[kyle-github]

@thejoshwolfe, thanks for running the experiments on cut-and-paste from browsers. Interesting that you did not get the CRLN combos. It has been a while since I cared to check and I tend to set all my editor tools to use LN only on save. As you note, Visual Studio is perhaps not the example of what to do :-)

Not sure how I feel about the idea of having the compiler reject code that is not in the One True Format(tm). While I like the idea that all Zig code would be formatted the same, that might be a little too draconian. For Python this almost works because indentation matters and if someone enters code using both tabs and spaces the meaning is ambiguous.

I think your example of JavaScript's semicolon insertion is taking this a bit far. The semicolon insertion (IMO) is an abomination because it can be wrong and change the intended meaning of the code. I do not the see the same thing with handling CRLN, LN or CR as white space.

If format is so important that you would want to make it enforced by the compiler, then perhaps the syntax should be closer to Python? I mean this in all seriousness. I think Guido van Rossum did something really interesting when he decided to make the visual layout elements of Python have meaning at the language level. Python code is not formatted all the same, but even without a python-fmt tool, the code from different projects has more formatting similarities than code in most other languages. I think van Rossum made a mistake in allowing tabs.

[thejoshwolfe]

If format is so important that you would want to make it enforced by the compiler, then perhaps the syntax should be closer to Python? I mean this in all seriousness. I think Guido van Rossum did something really interesting when he decided to make the visual layout elements of Python have meaning at the language level.

Yes. My idea is to have both C-like curly braces and Pythonic indentation, and they must agree. Curly braces are arguably easier for tools to understand, and indentation is absolutely easier for humans to understand, so I want both. Curly braces enable things that you can't do with just indentation. And as for the compiler enforcing indentation rules, come on, you should always get indentation right; no excuse for wrong indentation; it's not that hard, and it makes a huge difference for readability.

A neat advantage of having strict indentation rules and curly brace block scopes is that you can have better compile errors for unbalanced curly braces, which is something that is especially chaotic in C and Java.

fn SomeClassThing(comptime T: type) -> type {
    struct {
        const Self = this;
        field: T,
        fn method(self: &const Self) {
            {var i = u32(0); while (i < self.field.len) {
                self.field.something(i);
            }
        } // ERROR: missing '}', or wrong indentation
        // At this point, the compiler can trust the indentation
        // rather than the curly braces for parsing the rest of the file.
    }
}

In practice, indentation tends to be more correct than curly brace balance. This is especially relevant for IDE's where the tooling is trying to follow along with you as you type. Unbalanced parentheses, quotes, curly braces, etc. are very common while you're in the midst of typing code. By contrast, wrong indentation is much less common. Usually the indentation is wrong if you past/move a bunch of code at once, and in that case, you can have an IDE hotkey to trust the curly braces and fix the indentation; then everything's back in agreement.

Generally there are two facets to code formatting: readable for tools and readable for humans. C leans toward readable for tools (curly braces, etc.); Python leans toward readable for humans (indentation, etc.); Zig wants to have it both ways, and so has two sets of formatting rules that must be in agreement for your code to compile. (As a reminder, this is an informal plan for a future version of Zig, not status quo.)

Related is #114.

I think van Rossum made a mistake in allowing tabs.

Absolutely agree. It's horrifying how ugly you can make "correct" indentation in Python by mixing spaces and tabs, even in the same line. What a mess.

Not sure how I feel about the idea of having the compiler reject code that is not in the One True Format(tm). While I like the idea that all Zig code would be formatted the same, that might be a little too draconian.

I have high hopes for this strategy. We've already seen some people scared away by Zig's decision to not support hard tabs, which is a shame. But on the plus side, all Zig code will be consistent with this kind of design philosophy.