Feature request: Support a "warning" level for linting

[lengau]

It would be really nice if, like ignore, we could select specific issues to turn into warnings rather than errors. That is, it would still output a warning to stdout, (perhaps separating errors and warnings), but it would still return with exit status 0.

Use case:

A large, older codebase that contains a lot of linting issues regarding specific rules. (For example, a codebase that predates type annotations.) By allowing warnings, we can make these issues visible to developers for fixing as they lint their changes, but we don't have to have a separate configuration for our pre-commit hooks, CI tools, etc. to ensure they ignore these issues.

[charliermarsh]

Yeah, this would be nice to have. It would also be helpful for editor integrations, since the LSP can differentiate between warnings and errors.

[charliermarsh]

Any thoughts on what the API should look like?

ESLint lets you map from rule name to level, like:

...
"eqeqeq": "error",
"curly": "warning",
...

Which makes sense, though they also use that system to turn errors on and off -- and we instead have the Flake8-like ignore and select system.

The most basic thing would be a map from check code prefix to level, like:

[tool.ruff.severity]
"F401" = "warning"
"F481" = "error"
# Probably also want to support prefixes.
"E" = "warning"

[lengau]

Not really sure... Naively, it appears to me that the most consistent way to configure it would be to have warn and extend-warn (or warning & extend-warning).

This file contains what I'm doing currently for the project that inspired this request.

I enabled more linting than we previously had (everything I viewed as potentially appropriate for this app), but I put everything that raised an error into the ignore field. With this feature, lines 78-102 would've been in a warn section instead of inside ignore, with the goal of eventually emptying the warn section by consensus.

[edgarrmondragon]

fwiw bandit has four severity levels: undefined, low, medium and high.

https://github.com/PyCQA/bandit/blob/af9f8dc09f7239f3c3481177a24a72dcb41ed450/bandit/core/constants.py#L8

[jfmengels]

I would recommend against the implementation of warning (or severity levels). I wrote about why in this blog post (and also in a as-of-yet-unreleased talk Why you don't trust your linter).

In my opinion, it's one of those features (along with disable comments) that gets copied over and over mostly because other linters have them. The fact that no linter docs ever recommend when to use these features is a sign of that in my opinion.

I can potentially see a benefit for this while seeing the errors in an editor, but it shouldn't IMO impact the behavior of the linter.

In elm-review there are no severity levels (nor disable comments): rules are enforced or they are not. Though to be taken with a grain of salt: the linter also has a lot less false positives than is probably possible for Python because of the target language.

A large, older codebase that contains a lot of linting issues regarding specific rules

This is exactly the kind of problem that the proposal in #1149 aims to solve, and is a much better solution.

[jacobmischka]

I can potentially see a benefit for this while seeing the errors in an editor, but it shouldn't IMO impact the behavior of the linter.

How would the editor show the proper type of diagnostic to the user if the linter doesn't provide that information?