Problems reported when using NGINX by Ondrej PPA

[franbuehler]

The CRS team believes that strange CRS problems are reported when users use NGINX Ondrej PPA.

Reports:

• SO link to follow FIXME

What we think the problem is: FIXME

We would like to solve this problem (if any) or have it documented so users understand how to follow.

Statements from chat so that we don't lose them:

• We believe that the Ondrej PPA is responsible for the very strange errors we've seen reported over the past 2 years. E.g. "200 is not a number" and very odd things like that. (bearbeitet) (LINK?)
  We've had, I think, 4 separate reports.
• Yes, I think one remembers issues with that PPA and some bad dependencies. (LINK)
• yeah, there were some PCRE issue (LINK)
• Ondrey likes to use his own PCRE package (because of the PHP) (clarified the why below)

[dune73]

We have a history of poor cooperation with the Ondrej PPA lead developer. I think the best thing we can do is a blog post where we explain the situation and that it's hard for us to support this.

[fzipi]

Do we want to transfer this one to the website repo?

[dune73]

Sounds like a plan

[oerdnj]

We have a history of poor cooperation with the Ondrej PPA lead developer. I think the best thing we can do is a blog post where we explain the situation and that it's hard for us to support this.

I’m all ears…

[fzipi]

Now that we have @oerdnj attention 😄 (I'm surprised and glad you are here, 👋 btw), we probably should add what we think the problems (FIXME) were in the original description to see how/if things can be fixed, or just need better documentation on our side (or just the next step).

[oerdnj]

Now that we have @oerdnj attention 😄 (I'm surprised and glad you are here, 👋 btw), we probably should add what we think the problems (FIXME) were in the original description to see how/if things can be fixed, or just need better documentation on our side (or just the next step).

You know you could have tagged me and asked me before you start saying things about bad cooperation…. But let’s start afresh…

The PCRE2 library isn’t actually part of the nginx repository. I know that the nginx packages were (maybe still are) a mess in the past few months - it comes from the fact that Debian upstream maintainer unbundled the plugins and made some decisions that were not compatible with the PPAs. Unfortunately, this was inevitable as Ubuntu 24.04 picked on that, so I had to follow the suit, and it took me a while to figure out all the nifty details. And there’s apparently still something elusive that’s causing people problems on upgrades. I’m not omnipotent and I make mistakes.

And I am usually uncooperative only in case when people come and demand I do things for them for free. But I do try to listen to a solid argument.

[fzipi]

Of course, I assumed exactly that. This is, IMHO, the opposite of being "unresponsive" 😄 😄 .

Let's figure it out what happens... but as you said clearly here, and I totally agree: let's get a solid argument. I do not like not having a way to reproduce the problem. 💪

[airween]

The PCRE2 library isn’t actually part of the nginx repository.

It isn't. But it's part of your PHP repository.

In most cases (perhaps in all cases) the users who reported issues used some PHP application, and we assume all the used components installed from your repository. But it is also possible that the two have nothing to do with each other.

But the fact that there is a situation: if users install Nginx from your repository, then libmodsecurity3 produces the same weird behavior:

• #2752
• #3117
• #3181

and some other similar reports on mailing list/StackOverflow forums.

If users switch to Debian's version everything is solved.

The weird behavior is that the @rx operator (which does a simple regular expression pattern match) does not work as expected. Therefore, we can only think that the PCRE library used has some error.

I risk in all cases the reporter built the library (libmodsecurity3) from source, therefore the build flow used the system's PCRE library - so this is why we think that the user installed a non-default PCRE library.

[oerdnj]

The weird behavior is that the @rx operator (which does a simple regular expression pattern match) does not work as expected. Therefore, we can only think that the PCRE library used has some error.

So, what I am actually hearing is that it has nothing to do with NGINX repository, and it affects mod_security only when PCRE2 (I assume this is PCRE2, right?) is updated to the latest version? The PCRE2 maintainer is usually very responsive if there’s a test case.

Anyway, this should be fairly easy to test - Debian stable has PCRE2 10.42-1, Debian testing and Ubuntu 24.04 has PCRE2 10.42-4. My repositories has 10.42-3 (and the only difference between -3 and -4 is riscv64 support)

Can you observe the problematic behavior on those systems? Perhaps this boils down to - do you have means to reproduce the weird behavior?

[airween]

So, what I am actually hearing is that it has nothing to do with NGINX repository, and it affects mod_security only when PCRE2 (I assume this is PCRE2, right?) is updated to the latest version? The PCRE2 maintainer is usually very responsive if there’s a test case.

Unfortunately we don't know any details about that.

Here is a link that points the reporter's howto, but that's unavailable now. In the next comment I tried the request with same config, but it worked for me - but I was using Ubuntu's Nginx.

Btw I'm not sure it was built with PCRE2, because in Ubuntu 22.04 - as I remember - the default PCRE engine version was the 3, not the 2.

Anyway, this should be fairly easy to test - Debian stable has PCRE2 10.42-1, Debian testing and Ubuntu 24.04 has PCRE2 10.42-4. My repositories has 10.42-3 (and the only difference between -3 and -4 is riscv64 support)

Can you observe the problematic behavior on those systems? Perhaps this boils down to - do you have means to reproduce the weird behavior?

Based on the most valuable issue and its comment, I would try to follow this one. Install Nginx from your repository (onto an Ubuntu 22.04 - just for sure), install libmodsecurity3 from source - perhaps you should try with PCRE3 first - then install the connector. After that install CRS, and try to send the mentioned request.

If you need any help to configure/build the WAF just let us know.

[oerdnj]

Btw I'm not sure it was built with PCRE2, because in Ubuntu 22.04 - as I remember - the default PCRE engine version was the 3, not the 2.

That might be the problem - PCRE (libpcre3) is obsolete for couple of years now and it doesn't receive any updates and fixes anymore.

I also see this:

We had the same problem. It turned out that we did not supply --with-pcre2 during configuration.

And I see issues like this in modsecurity:

• PCRE2 support still requires PCRE1 owasp-modsecurity/ModSecurity#2966
• PCRE2 support still requires PCRE1 owasp-modsecurity/ModSecurity#2750

So, instead of saying the repository is at fault, I would suggest that ModSecurity uses PCRE2 by default, and for all bugreports where people compile libmodsecurity3 by hand, first suggest to use PCRE2 (libpcre2) instead of PCRE1 (libpcre3).

I understand it's very confusing that libpcre3 is PCRE1 (unsupported) and libpcre2 is PCRE2, but that's related to the SONAMEs of the libraries and is a different number than PCRE vs PCRE2.

[oerdnj]

Hmm, I am looking at coreruleset/coreruleset#3181 (comment) and this says the user compiled nginx from the source.