Add --soft-fail option to capture Mypy output without blocking CI

[balihb]

Feature

Add a --soft-fail option to Mypy that allows capturing its output without causing a hard failure in CI/CD pipelines.

Pitch

Currently, Mypy exits with a 1 status code when type errors are found, which is expected behavior. However, in many CI/CD pipelines (especially those integrating with SonarQube or similar tools), it's useful to capture all Mypy output while still allowing the pipeline to continue in a controlled way.

A --soft-fail option would:

• Allow Mypy to run normally and produce full output, including errors and warnings.
• Exit with status code 0 so that CI/CD jobs don’t automatically fail.
• Still allow external tools like SonarQube to analyze and report issues without interfering with the pipeline’s flow.

I currently have a one liner workaround in tox:

https://gist.github.com/balihb/0b64435c0e8f553791071cea4a897c91

[tox]
min_version = 4.20
envlist =
# renovate: datasource=docker depName=python
    3.13

[testenv]
skip_install = true
pass_env =
    FORCE_COLOR
    MYPY_*
    NO_COLOR
    PIP_*
    PYTHONUNBUFFERED
    PY_COLORS
    TERM
deps =
    -r requirements.txt
    -r requirements-mypy.txt
commands =
commands =
    python -c "import os, re, shlex, subprocess, sys; \
    from pathlib import Path; \
    result = subprocess.run(['mypy'] + shlex.split('{posargs:src}'), check=False, text=True, capture_output=True); \
    result.stdout.strip() and sys.stdout.write(result.stdout); \
    result.stderr.strip() and sys.stderr.write(result.stderr); \
    output = Path(os.getenv('MYPY_TOX_OUTPUT', default='.report/mypy/output.txt')); \
    output.parent.mkdir(parents=True, exist_ok=True); \
    output.write_text(re.compile(r'\x1b\\[[0-9;]*?m').sub('', result.stdout)); \
    sys.exit(1 if result.returncode == 2 else 0)"

but this is far from ideal.

[JelleZijlstra]

Why can't you just add || true in the pipeline? This seems like something that's more easily handled outside of mypy.

[balihb]

try it on windows

[balihb]

also I want it to fail if there is a config error or I scan a non existing path.

[balihb]

refer to:

https://www.checkov.io/2.Basics/Hard%20and%20soft%20fail.html

[A5rocks]

also I want it to fail if there is a config error or I scan a non existing path.

I feel like this is somewhat of an ill-defined boundary. Should it error on syntax errors? Should mypy exit with failure for any blocking issues (i.e. there's an issue that prevents full checking) and success otherwise? Probably it's better for users to make the thing that exits differently instead of providing a ton of knobs.

[balihb]

how is this ill-defined? mypy already uses the return code 2. --soft-fail would make 0, 1 translate to 0 and 2 to 1. my workaround already using that, but a parameter would be nicer.

(.venv) PS C:\workspace\my-code> tox -e mypy
mypy: commands[0]> python -c "import os, re, shlex, subprocess, sys; from pathlib import Path; result = subprocess.run(['mypy'] + shlex.split('src'), check=False, text=True, capture_output=True); result.stdout.strip() and sys.stdout.write(result.stdout); result.stderr.strip() and sys.stderr.write(result.stderr, file=sys.stderr); output = Path(os.getenv('MYPY_TOX_OUTPUT', default='.report/mypy/output.txt')); output.parent.mkdir(parents=True, exist_ok=True); output.write_text(re.compile(r'\x1b\[[0-9;]*?m').sub('', result.stdout)); sys.exit(1 if result.returncode == 2 else 0)"
Generated Cobertura report: C:\workspace\my-code\.report\mypy\cobertura\cobertura.xml
Generated HTML report (via XSLT): C:\workspace\my-code\.report\mypy\html\index.html
Generated TXT report (via XSLT): C:\workspace\my-code\.report\mypy\txt\index.txt
Generated XML report: C:\workspace\my-code\.report\mypy\xml\index.xml
Success: no issues found in 112 source files
  mypy: OK (24.81=setup[0.45]+cmd[24.36] seconds)
  congratulations :) (25.31 seconds)

(.venv) PS C:\workspace\my-code> tox -e mypy
mypy: commands[0]> python -c "import os, re, shlex, subprocess, sys; from pathlib import Path; result = subprocess.run(['mypy'] + shlex.split('src'), check=False, text=True, capture_output=True); result.stdout.strip() and sys.stdout.write(result.stdout); result.stderr.strip() and sys.stderr.write(result.stderr, file=sys.stderr); output = Path(os.getenv('MYPY_TOX_OUTPUT', default='.report/mypy/output.txt')); output.parent.mkdir(parents=True, exist_ok=True); output.write_text(re.compile(r'\x1b\[[0-9;]*?m').sub('', result.stdout)); sys.exit(1 if result.returncode == 2 else 0)"
src\some_python_code.py:1:14: error: Incompatible types in assignment (expression has type "dict[Never, Never]", variable has type "list[Any]")  [assignment]
    alma: list = {}
                 ^~
Generated Cobertura report: C:\workspace\my-code\.report\mypy\cobertura\cobertura.xml
Generated HTML report (via XSLT): C:\workspace\my-code\.report\mypy\html\index.html
Generated TXT report (via XSLT): C:\workspace\my-code\.report\mypy\txt\index.txt
Generated XML report: C:\workspace\my-code\.report\mypy\xml\index.xml
Found 1 error in 1 file (checked 113 source files)
  mypy: OK (20.91=setup[0.14]+cmd[20.77] seconds)
  congratulations :) (21.05 seconds)

(.venv) PS C:\workspace\my-code> tox -e mypy -- bad-src                      
mypy: commands[0]> python -c "import os, re, shlex, subprocess, sys; from pathlib import Path; result = subprocess.run(['mypy'] + shlex.split('bad-src'), check=False, text=True, capture_output=True); result.stdout.strip() and sys.stdout.write(result.stdout); result.stderr.strip() and sys.stderr.write(result.stderr, file=sys.stderr); output = Path(os.getenv('MYPY_TOX_OUTPUT', default='.report/mypy/output.txt')); output.parent.mkdir(parents=True, exist_ok=True); output.write_text(re.compile(r'\x1b\[[0-9;]*?m').sub('', result.stdout)); sys.exit(1 if result.returncode == 2 else 0)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: TextIOWrapper.write() takes no keyword arguments
Generated Cobertura report: C:\workspace\my-code\.report\mypy\cobertura\cobertura.xml
Generated HTML report (via XSLT): C:\workspace\my-code\.report\mypy\html\index.html
Generated TXT report (via XSLT): C:\workspace\my-code\.report\mypy\txt\index.txt
Generated XML report: C:\workspace\my-code\.report\mypy\xml\index.xml
mypy: exit 1 (0.44 seconds) C:\workspace\my-code> python -c "import os, re, shlex, subprocess, sys; from pathlib import Path; result = subprocess.run(['mypy'] + shlex.split('bad-src'), check=False, text=True, capture_output=True); result.stdout.strip() and sys.stdo
ut.write(result.stdout); result.stderr.strip() and sys.stderr.write(result.stderr, file=sys.stderr); output = Path(os.getenv('MYPY_TOX_OUTPUT', default='.report/mypy/output.txt')); output.parent.mkdir(parents=True, exist_ok=True); output.write_text(re.compile(r'\x1b\[[0-9;]*?m').sub('', result.stdout)); sys.exit(1 if result.returncode == 2 else 0)" pid=48012
  mypy: FAIL code 1 (3.41=setup[2.97]+cmd[0.44] seconds)
  evaluation failed :( (3.53 seconds)

the first is with good code, the 2nd is problem with the code, the 3rd is problem with the execution.

[A5rocks]

Alright, I suppose that's a good enough definition. Though your reference allows configuring exit code based on issue codes which I don't think is mypy's responsibility.

[balihb]

I'm only talking about --soft-fail, not --soft-fail-on. just to allow the tool to generate the report if there weren't a catastrophic failure during the run, just detected problems with the scanned code.

[sterliakov]

As a precedent in a tool with good CLI interface, ruff has --exit-zero to translate exit code 1 to 0 and keep other codes (e.g. 2 - malformed config file) intact.

I agree that --exit-zero is a reasonable feature request. There's a big difference between "the tool produced some diagnostics" (exit code 1) and "the tool failed to run altogether" (code 2), and I think it's helpful to express "report something or die if you can't" with a single CLI flag, but rather low-prio as that feature is as simple as mypy || [ "$?" -lt 2] || exit 2 and probably nobody uses windows for CI/CD pipelines (or they have much bigger problems than mypy exit code nuisance).

[balihb]

"probably nobody uses windows for CI/CD pipelines" I can't support this assumption.

also I'm using mypy locally on a windows machine to check my code in tox which is a tool that is designed to run both locally and in the CI.

[balihb]

I've checked the mypy docs and it explicitly mentions how to properly use it on windows multiple times, so I assume windows is a supported platform for using the tool.

[balihb]

also FYI this is the regex sonar uses to parse the mypy output:

https://github.com/SonarSource/sonar-python/blob/master/python-commons/src/main/java/org/sonar/plugins/python/mypy/MypySensor.java#L49

obviously if mypy fails the pipeline/workflow before it can be parsed it is no help, but if there is a fundamental issue with the mypy run || true would hide that.