Issues with matching `\s`

[smac89]

It seems very buggy atm. Strangely enough, using [ ] does not show the same behavior.

For example, given the following text:

# Here we go
This is a line

# Let's try this again
Here is another line

# Third time's the charm
Am I a line?

I will now run the following commands using the above text as input and a mix of [ ] and \s:

In all of the following instances, grep is aliased to ugrep --sort --basic-regexp --binary --empty --hidden

➜ clippaste | grep '\s*#'        
# Here we go
This is a line

# Let's try this again
Here is another line

# Third time's the charm

Clearly wrong output

➜ clippaste | grep '[ ]*#'
# Here we go
# Let's try this again
# Third time's the charm

Much better!

➜ clippaste | grep -v '\s*#'
Am I a line?

Nope!

➜ clippaste | grep -v '[ ]*#'
This is a line

Here is another line

Am I a line?

Yes!

---

I hope that's enough to track down this bug.

ugrep version:

ugrep 3.1.10 x86_64-pc-linux-gnu +avx2 +pcre2_jit +zlib +bzip2 +lzma +lz4
License BSD-3-Clause: <https://opensource.org/licenses/BSD-3-Clause>
Written by Robert van Engelen and others: <https://github.com/Genivia/ugrep>

[genivia-inc]

I have to look at this, but \s also matches newlines and anything else that is white space, so using \s* can certainly lead to unexpected matches!

PS. a better alternative is \h that matches space or tab only.

[genivia-inc]

Ugrep performs multi-line matching, as you can see in the output. The \s pattern matches any white space, including newline and Unicode white space characters. This means that the line before the line with # will be matched too:

LINE1\n\n#LINE3
     ^^^^^
     pattern match: LINE1\n\n# matches

This is clearly visible in the output with ugrep option -n:

ugrep -n '\s*#' lines.txt
1:# Here we go
2:This is a line
3|
4|# Let's try this again
5:Here is another line
6|
7|# Third time's the charm

where the | indicates multi-line matches (a ugrep feature). Not surprising then that the same happens with \n*# instead of \s*#:

ugrep -n '\n*#' lines.txt
1:# Here we go
2:This is a line
3|
4|# Let's try this again
5:Here is another line
6|
7|# Third time's the charm

There are two options to change ugrep:

• When using ugrep as a grep command, internally disable multiline matching altogether.
• Change \s to exclude \n, which is not according to the standard definition of \s and [[:space:]].

Both have pro and cons. The second could be conditional upon using ugrep as grep, but that can lead to confusion for sure.

[smac89]

Aha that makes sense now. I guess I was just used to line matching by default rather rather than multiple lines by default.

When using ugrep as a grep command, internally disable multiline matching altogether.

How do I do this please? I mostly followed the aliases defined here. I don't see an option that disables multiline match.

[genivia-inc]

When using ugrep as a grep command, internally disable multiline matching altogether.

How do I do this please? I mostly followed the aliases defined here. I don't see an option that disables multiline match.

I was talking about a possible update to ugrep to address this automatically.

There is no command-line option in ugrep to disable multiline matching. In fact, it should never be necessary to do so, because no "normal" regex pattern can match a newline and therefore produce multi-line matches, except when \n is explicitly used or when \s ([[:space:]]) is used in the pattern. By default, \s matches spacing in regex libraries, which includes newlines. Because grep is line-based, \s does not match a newline in grep. That's the only difference.

For now, use \h that matches space and tab in (GNU/BSD grep and ugrep).

For the future update of ugrep, the question is what to do about \s? I have to think about this...

[genivia-inc]

I will update ugrep very soon. I decided to make an exception for \s to exclude newline \n in the update. The other related patterns that could be changed are:

[[:space:]] POSIX ASCII space characters: space, tab, CR, LF, FF;
[[:cntrl:]] POSIX ASCII control characters;
\p{Space} Unicode space characters;
any other \p{X} that matches newline.

However, it gets quite complicated to change these patterns, so I decided not to change these patterns, putting the responsibility on the user to exclude \n when one of these specialized patterns is used, e.g. with:

ugrep `[\p{Space}--[\n]]` ...

the pattern above will be the same as the updated \s:

ugrep `\s` ...

Therefore, In the upcoming update, the \s pattern is \p{Space} but without newline in Unicode mode (the default) and [[:space:]] without newline in non-Unicode mode with option -U (in which case \p{Space} is actually the same as [[:space:]]).

I hope this decision makes the most sense and will not be confusing to users. It also allows users to fall back on using \p{Space} to match spacing including newlines as expected.

Note that all negated bracket patterns [^...] already exclude newlines \n in ugrep for the same reason as the problem you describe, such as [^a-z] that matches anything except a to z and newline.

[genivia-inc]

Updated in v3.1.11.