Improved collation support for non-ascii case-insensitive text-match

[tobixen]

RFC4791 requires the server to support searching according to two collations - i;octet for binary match and i;ascii-casemap allowing case-insensitive search, with the latter being the default. In caldav v2.2.2 there is test code covering both "case sensitive" and "case insensitive" searches. The problem with i;ascii-casemap is that it only works for ascii characters - causing mismatches between naïve and NAÏVE, cliché and CLICHÉ, smörgåsbord and SMØRGÅSBORD, not to forget millions of words in non-English languages, complete non-latin scripts, etc.

RFC4790 specifies a i;unicode-casemap collation, which may or may not be supported by the server. RFC4791 section 7.5.1 says that it's possible to ask the server what collations it support.

To resolve this issue ...

• Case-insensitive searches should work for non-ascii characters on all servers supporting it.
• Library should detect non-ascii characters and do workarounds for servers not supporting case insensitivity on non-ascii characters.

Locale support would be nice (i.e. "istanbul" should match with İstanbul in Turkish locale), but not required (this may be a very deep rabbit hole - one would like istanbul to match both İstanbul and Istanbul, at the other hand there may be too many false negatives if the matching is too liberal).

A good test-case may include English loan-words like crème brûlée and naïve, typical Scandinavian words like Smörgåsbord, Blåbærsyltetøy, some French and Turkish words, as well as Ukrainian text.

The i;unicode-casemap may not be sufficient to handle all languages, ref the Istanbul example above.

There is an example file in the example directory that may need brush-up as well.

[tobixen]

Claude claims this task will take 16-24 hour of work, I'm as always an optimist and believe it will take less (too much of an optimist, maybe, v2.0.0 -> v2.2.2 took me more than 230 hours, around 5x my estimate). Lets try to break it down ...

• Check should be added in caldav-server-checker to check if the server will do unicode case mapping or not when presented with the i;ascii-casemap collation (this is a tricky one - as I read the RFC it should not, so it's technically a breach of the standard to do so - but still I would consider it a nice feature). There is already test-case for search.text.case-sensitive in checks.py, so it would fit to throw it in there. The calls to the server is done via the library, it's a bit dangerous as changes in the library may alter what we're actually testing. One hour+
• Checks should be made in the caldav-server-checker to check if the sever supports RFC4791 section 7.5.1.. One hour+
• Collation support should be marked up in the features, some thinking needs to be done. One hour+
• Checks should be made to verify that it supports the collations it claims to be supporing. One hour+
• All servers should be checked. Problems with the check script needs to be ironed out. Two hours.
• Functional tests for the case insensitive unicode matching. One hour.
• Hammer out on the search.py-file. Three hours
• Make sure the test suite passes on all servers available. Two hours.

Totally: 12-16 hours.