[suggestion] revise unicode/binary mode decision

[avih]

As far as I can tell, POSIX awk is required to respect the current locale, but goawk doesn't do that. Instead, it behaves in byte mode by default, unless -c is specified, at which case it behaves in UTF-8 codepoints mode.

And while goawk probably can't do arbitrary locales, and ignoring bugs, it seems to have support for LC_CTYPE of either UTF-8 or plain bytes (ASCII/C ?).

So assuming it's desirable for goawk to try and respect the current locale where possible, I think it could look like this:

• Add argument support for -b for binary/bytes mode. gawk has the same-ish -b as alias for --characters-as-bytes.
• During arguments parsing, if -b or -c is specified (or replaced it with -u) then use the specified mode.
• Else try to deduce it from the environment, like so:
  • If any of LC_ALL, LC_CTYPE, LC_LANG, in this override order is defined - even if empty (in go: os.LookupEnv(name)), stop the search and use its value:
    • if its tolower includes .utf8 or .utf-8 - enable UTF-8 mode, else enable byte/binary mode.
• Else (no -b/-c and none of these vars is defined), pick some default, maybe depending on the platform (e.g. on Windows, and maybe also elsewhere, probably enable UTF-8 because that's what most text files are likely to be).

Does something like this make sense? I think it should be fairly trivial to implement, so the real question is whether such behavior is desirable, right?

Is there some assumption or empirical observation that awk scripts tend to behave better in goawk in one mode or the other?

Is there a meaningful performance impact depending on the unicode mode? I think in general bytes mode is typically faster, but considering that it might be hard for goawk to do regexp in bytes mode, does it still matter for goawk?

[benhoyt]

Hi @avih, thanks for the issue to track. Yes, goawk tries to behave in bytes mode by default, or chars mode (UTF-8 codepoints) if goawk -c is specified. That's the opposite of gawk, which is chars mode by default, and bytes mode if you specify gawk -b. I'd like GoAWK to be chars-by-default like Gawk, but there are performance issues with this (see also previous work in #252). I'm open to continued work on that (though I don't personally have time right now), but I don't want a huge slowdown for bytes mode.

However, even though GoAWK is a POSIX-compatible AWK in most ways (I followed the spec pretty closely, and it's tested against onetrueawk and gawk test cases), I'm not going to implement support for locales based on environment variables. I strongly dislike the fact that environment variables like LC_CTYPE means that tools behave differently in different environments, and I don't want that for GoAWK.

Again, I'm open to improving the current state of affairs, to at least document the inconsistencies noted in #273. Ideally we'd figure out a not-too-invasive way to make chars the default and change to a Gawk-like chars-by-default.

[avih]

That's the opposite of gawk, which is chars mode by default, and bytes mode if you specify gawk -b.

Is it? I think that it defaults to current locale/env, while -b can force bytes mode. But indeed apparently there's no option to force unicode mode. But yeah, it differs.

I'd like GoAWK to be chars-by-default like Gawk, but there are performance issues with this (see also previous work in #252)

Right, I know that generally bytes mode tends to be faster, but I thought that because regexp is kinda stuck on UTF-8, it might carry less weight, but I guess it still does have meaningful impact. I looked at these links but didn't try to read them thoroughly.

Agreed that's a non negligible factor.

I'm not going to implement support for locales based on environment variables. I strongly dislike the fact that environment variables like LC_CTYPE means that tools behave differently in different environments, and I don't want that for GoAWK.

Interesting. So ideally you would have preferred to default to unicode mode, regardless of env, and only controlled by user options?

To be honest, I'm not sure what I prefer. I think I tend towards the "respect env if possible, like posix specifies" (and obviously user options have final say), but ultimately that's your decision to make, and I mostly care about being able to control it - which I am.

Do you happen to have some data on which goawk mode random awk scripts like more? I think both your links only refer to performance issue, but if many awk scripts just break up in unicode mode, then this could be another meaningful factor while deciding the default behavior.

Ideally we'd figure out a not-too-invasive way to make chars the default and change to a Gawk-like chars-by-default.

TBH I don't think it matters a lot, as long as it's documented and can be controlled.

In fact, the reason I realized the -c issues in #273 was that I checked the default unicode behavior in several awk implementations, using multibyte UTF-8 with /^.$/, and it matched in goawk, so I assumed it either respect the env or does unicode by default, but then I got really confused when --help showed that apparently it defaults to bytes, while -c enables unicode, so I dug a bit more, and then realized this match is a bug...

Anyway, I didn't know what was or wasn't considered in the past, so I'd thought to suggest a concrete path to push it a bit more towards POSIX, pending relevant factors.

But if you don't intend to respect the env, then I don't think that it matters a lot whether it defaults to unicode or bytes mode. The main factors in such case should probably be performance and compatibility with existing scripts, and you know about those with goawk much better than me.