bug/feature_request(urls): false positives for URLs

[Kristinita]

Related issues — #676, #2473, #3211.

1. Summary

When I run Codespell for my project, I get 14 false positives for URLs. I’m sure that default Codespell behavior for URLs need corrections.

2. MCVE

2.1. Reproducibility

I reproduced the problem on Ubuntu and Windows.

1. Ubuntu build on Travis CI
2. Windows build on AppVeyor CI

[NOTE] I get different behavior for Ubuntu and Windows. By default, for Ubuntu Codespell prints to console a colored output, for Windows — non-colored.

2.2. Files

KiraURLs.txt — the file with false positives from my real project:

https://browsersl.ist/
https://archive.fo/JaBJt
https://magisteria.ru/autor/andrey_vinogradov
https://woerterbuchnetz.de/?sigle=DWB&lemid=U14452
https://www.pucp.edu.pe/profesor/patricia-gonzales-gil
https://zoon.ru/msk/p-doctor/marina_vladimirovna_kuznetsova/
https://persona.rin.ru/view/f/0/25814/kolesov-evgenij-nikolaevich
https://broker.ru/studing/seminar/303
https://www.wirtschaftsdienst.eu/autor/walter-a-s-koch.html
https://web.archive.org/web/20160306044838/https://xpomo.com/ruskolan/tolpa/demagog.htm
https://base.garant.ru/10180093/
https://web.archive.org/web/20210509023444/https://portal.slac.stanford.edu/sites/inc_public/Pages/folder-file-names.aspx
https://portal.slac.stanford.edu/sites/inc_public/Pages/folder-file-names.aspx
https://www.digitalo.cz/dokument/qBhi6rSjJ87tFJ6y/plny-text.pdf#page=102

codespell.cfg:

[codespell]

ignore-regex = https?://[^\s`"<>^\\{|}]+

[NOTE] The uri-regex key with the same value has no effect. It’s a possible bug.

2.3. Commands

codespell --config codespell.cfg KiraURLs.txt
codespell KiraURLs.txt

2.4. Behavior

2.4.1. Expected — my regex

No output. Codespell ignores all URLs.

2.4.2. Non-desired — default Codespell behavior

codespell KiraURLs.txt
KiraURLs.txt:1: ist ==> is, it, its, it's, sit, list
KiraURLs.txt:2: fo ==> of, for, to, do, go
KiraURLs.txt:3: autor ==> author
KiraURLs.txt:4: sigle ==> single, sigil
KiraURLs.txt:5: profesor ==> professor
KiraURLs.txt:6: zoon ==> zoom
KiraURLs.txt:7: rin ==> ring, rink, rind, rain, rein, ruin, grin
KiraURLs.txt:8: studing ==> studying
KiraURLs.txt:9: autor ==> author
KiraURLs.txt:10: demagog ==> demagogue
KiraURLs.txt:11: garant ==> guarantor, guardant
KiraURLs.txt:12: slac ==> slack
KiraURLs.txt:13: slac ==> slack
KiraURLs.txt:14: dokument ==> document
Command exited with code 65

False positives for each line.

3. Regular expression for URLs

For my project I use this regular expression for URLs:

https?://[^\s`"<>^\\{|}]+

Perhaps it will help another users.

3.1. Advantages of my regex

1. This expression doesn’t match correctly examples of Mathias Bynens, but is sufficient for practical Codespell usage in my case.
2. It’s simple and shouldn’t slow down Codespell too much.

3.2. Explanation of my regex

It allowed any Unicode symbols in URLs except:

1. \s — for the Python regular expression syntax \s metacharacter “includes [ \t\n\r\f\v] and also many other characters, for example the non-breaking spaces mandated by typography rules in many languages”.
2. Another forbidden ASCII characters in URLs.

4. Environment

1. Operating system:

   1. Local — Microsoft Windows [Version 10.0.22621.3085]
   2. Travis CI — Ubuntu 24.04.2 LTS Noble Numbat
   3. AppVeyor CI — Microsoft Windows [Version 10.0.17763.6189]

2. Python:

   1. Local — 3.13.2
   2. Travis CI — 3.13.1
   3. AppVeyor CI — 3.12.8

3. Codespell 2.4.1

Thanks.

[DimitriPapadopoulos]

When I run Codespell for my project, I get 14 false positives for URLs. I’m sure that default Codespell behavior for URLs need corrections.

There's no specific behaviour for URLs, in the sense that codespell recognises but does not skip URLs. There's even a specific --uri-ignore-words-list option to ignore words within URIs.

Just ignore the false positives:

$ codespell KiraURLs.txt --uri-ignore-words-list ist,fo,autor,sigle,profesor,zoon,rin,studing,autor,demagog,garant,slac,dokument
$ 

[Kristinita]

Type: Questions ❓

codespell recognises but does not skip URLs

This behavior seems strange to me.

1. Why Codespell has this behavior? In my opinion, in most cases a purpose of checkers/linters/validators is to check my own content. External URLs are third-party content, I had no control over their creation. I don’t need to check external URLs like I don’t check third-party directories like node_modules and .venv.
2. Is the absence even the option like --skip-external-urls-checking a principled position of Codespell developers?

Thanks.

[DimitriPapadopoulos]

I don't know why codespell has this behaviour. It's probably too late to change the default behaviour, but an option such as --uri-ignore might be OK. On the other hand, I must admit I don't like the inflation of options that address issues that can be handled by simply ignoring a few false positives here and there - but then I'm not the maintainer.

I have found typos in URIs in the past, btw. On the other hand, except for sample non-existent URIs, link checkers are more suited to finding typos in URIs, no need for a spell checker for that!

[peternewman]

1. Why Codespell has this behavior? In my opinion, in most cases a purpose of checkers/linters/validators is to check my own content. External URLs are third-party content, I had no control over their creation. I don’t need to check external URLs like I don’t check third-party directories like node_modules and .venv.

A lot of URLs in projects I've been involved with may be mine, linking to documentation or other sites I control, so I may wish to spell check them too, but they might be using other tooling I can't just run codespell against directly.

Just ignore the false positives:

There's a special case for this, which I suspect is what you want?:

codespell/codespell_lib/_codespell.py

Lines 964 to 969 in 47337d2

	# If all URI spelling errors will be ignored, erase any URI before
	# extracting words. Otherwise, apply ignores after extracting words.
	# This ensures that if a URI ignore word occurs both inside a URI and
	# outside, it will still be a spelling error.
	if "*" in uri_ignore_words:
	line = uri_regex.sub(" ", line)

It's even documented!:

codespell/codespell_lib/_codespell.py

Lines 469 to 478 in 47337d2

	parser.add_argument(
	"--uri-ignore-words-list",
	action="append",
	metavar="WORDS",
	help="comma-separated list of words to be ignored "
	"by codespell in URIs and emails only. Words are "
	"case sensitive based on how they are written in "
	'the dictionary file. If set to "*", all '
	"misspelling in URIs and emails will be ignored.",
	)

3. Regular expression for URLs

For my project I use this regular expression for URLs:

https?://[^\s`"<>^\\{|}]+
Perhaps it will help another users.

3.1. Advantages of my regex

1. This expression doesn’t match correctly examples of Mathias Bynens, but is sufficient for practical Codespell usage in my case.
2. It’s simple and shouldn’t slow down Codespell too much.

It seems to miss all the other protocols the built in one handles, as well as capturing e.g. the label for a URL in markdown:

codespell/codespell_lib/_codespell.py

Lines 57 to 62 in 47337d2

	# While we want to treat characters like ( or " as okay for a starting break,
	# these may occur unescaped in URIs, and so we are more restrictive on the
	# endpoint. Emails are more restrictive, so the endpoint remains flexible.
	uri_regex_def = (
	r"(\b(?:https?|[ts]?ftp|file|git|smb)://[^\s]+(?=$|\s)|\b[\w.%+-]+@[\w.-]+\b)"
	)

In markdown its beforehand, but in other standards I guess it may be afterwards and would then get swallowed?
[The Example Website](http://example.com)

[Kristinita]

Type: Replies 💬

1. Checking own-controlled external URLs

A lot of URLs in projects I've been involved with may be mine, linking to documentation or other sites I control

1. OK, thanks. But there are also users like me who think that checking third-party content isn’t the purpose of linters and/or have significantly more third-party external URLs than their own external URLs. Codespell haven’t even the option like --skip-external-urls-checking for these users.

2. Regular expressions’ syntax allows excluding specific domains from regex if user will use the option like --skip-external-urls-checking.

3. Codespell could have the additional option like own-domains which values are the list of own domains. Codespell will check URLs on these domains, if a user run Codespell with the command-line argument --skip-external-urls-checking. For example, if a user have this configuration:

   own-domains = myexampledomain.com, mysecondexampledomain.com

   the regular expression for external URLs will be like this:

   https?://(?!(?:myexampledomain\.com|mysecondexampledomain\.com))[^\s`"<>^\\{|}]*

4. Codespell already has the command-line argument --check-filenames (why not --check-folders too?). Users can run Codespell with this argument for all websites they controlled instead of checking external URLs.

5. For checking URLs include own-controlled external URLs there are such tools like Linkchecker.

2. uri-ignore-words-list

Just ignore the false positives (…) There's a special case for this, which I suspect is what you want?

No.

1. As I wrote in the item 1 of my previous comment, I don’t think that checking third-party content is the purpose for checkers/linters/validators.
2. I can paste links on Latin from non-English sites to my English texts, and I can paste links with misspelled English words. I don’t want to be reminded of the option uri-ignore-words-list every time I do this. I want “just” ignore checking external URLs.
3. I don’t check projects with thousands of URLs use Codespell. Addition of new values to the list uri-ignore-words-list for these projects may take a time.

3. Regular expression for external URLs

1. I created a simple regular expression that shouldn’t slow down Codespell too much. The real default regular expression possibly must comply with the balance between accuracy and performance. But even if I use my regular expression, I haven’t problems on practice. I don’t exclude that they can be, but I didn’t encounter them.

2. It seems to miss all the other protocols the built in one handles

   By what criteria did Codespell developers define protocols for the uri_regex_def? I don’t use in my code, comments and texts protocols other than HTTPS and HTTP, therefore, I didn’t include another protocols in my regular expression. If another users want to exclude URLs with another protocols, and they don’t want Codespell to check them, of course, they need to modify the regular expression.

3. as well as capturing e.g. the label for a URL in markdown

   Yes, I knew about this problem when I posted this issue, but I decided not to complicate the regular expression, because I didn’t figure out what problems can Codespell users have because of it. Can you provide practical examples of possible problems due to the use of my regular expression?

Thanks.