Support for pyproject.toml

[pslacerda]

PEP518 says that pyproject.toml is to support the minimum build requirements. I want to consider mypy as part of the minimum build on my package.

For example Pip uses a [tool.towncrier] section on pyproject.toml to generate a correctly updated NEWS.rst as part of its build. One can considers typing correctness as important to the distributed source pakcage as an well documented NEWS.rst. Other example is Black that also has it section on pyproject.toml.

Maybe you think that this is a silly feature request and that setup.cfg must be used instead, Pip or Flit or whatever can setup mypy after all. But an TOML module will be added to the standard library sooner or later, than could mypy accept this feature and antecipate TOML supporting pyproject.toml today?

[ilevkivskyi]

PEP 518 is accepted and specifies that pyproject.toml is preferred over setup.cfg, so I think we should support whatever the PEP specifies.

[gvanrossum]

Does adding pyproject.toml mean that we can remove other config files? Until that point I'm not excited.

[pslacerda]

I think that removing other config files would simplify config parsing even more. I would be very glad to change mypy.main.parse_config_file() accordingly.

If you agree that mypy==0.620 it is still in its early ages, would be exciting to remove the other config files. Otherwise I'm already coding the unexciting changes to support both legacy and pyproject.toml.

There is already lots of deprecated options:

--disallow-any
--strict-boolean
--dirty-stub
--use-python-pat
--silent-import
--almost-silen
--fast-parser
--no-fast-parser

So legacy config files (mypy.ini, setup.cfg and ~/.mypy.ini) could also be one of them.

[gvanrossum]

Note that mypy.ini serves a completely different purpose from setup.py, setup.cfg, and pyproject.toml. The mypy.ini file is read by mypy when it is about to type-check another program. The other files (IIUC) are used to build mypy itself. I don't think we can change mypy.ini much, though we can start removing deprecated flags (after they have been printing a warning for 1-2 release cycles). Please do that in a separate PR.

[pslacerda]

Even if setup.cfg was created originally for distutils or setuptools, everybody else seems to use it as an generic configuration file. PyTest for example don't care if is running from setup.py or whatever, it simply reads setup.cfg if it is found; many tools seems to do it (eg flake8, isort and even mypy).

If you are talking about to replace setup.py by pyproject.toml, yes it is indeed an legibility win.

Now taking about mypy config files when type-checking other programs, they are all equivalents and fallbacks of the next one (--config-file -> mypy.ini -> setup.cfg -> ~/.mypy.ini; http://mypy.readthedocs.io/en/latest/config_file.html. I'm talking to deprecate these ini-like config files and use toml, it is even more typed!

I will submit a PR to remove that flags later on.

[pslacerda]

To put it clear: enable sections like [tool.mypy] or [tool.mypy-somepackage] in pyproject.toml.

[gvanrossum]

Switching mypy.ini to a toml thing sounds like scope creep and we won't have the time and motivation to review PRs related to this, so please don't start on that. The rest is fine of course.

[pslacerda]

Seems that I misunderstood that to remove others would be necessary to the overral comunity need to target the same. Sorry.

Does adding pyproject.toml mean that we can remove other config files? Until that point I'm not excited.

[JelleZijlstra]

So what exactly is being proposed here?

There's a few things I can think of:

• Add support for reading mypy's configuration out of pyproject.toml. In addition, we should still support reading configs out of setup.cfg and mypy.ini for backward compatibility. We could perhaps deprecate setup.cfg support though.
• Move configurations within mypy's own repo to pyproject.toml. This could replace the current pytest.ini and setup.cfg files. Not sure whether all current tools that read these config files support pyproject.toml yet though.

[ethanhs]

Pytest seems to be stalled until someone works on it pytest-dev/pytest#1556

An issue for flake8 was opened yesterday https://gitlab.com/pycqa/flake8/issues/428

And the same for coverage https://bitbucket.org/ned/coveragepy/issues/664/allow-configuration-via-pyprojecttoml

I think we should add support for reading mypy's configuration from pyproject.toml, but it looks like we will need to wait to use it for our own config.

[pslacerda]

@JelleZijlstra

The first option. IMHO were some convergence towards each_project.ini then setup.cfg and now towards pyproject.toml. The messy that was tooling configuration seems to be finally ending. Indeed I have some sort of working code, a TOML library was required but pretty much can be packed together instead.

master...pslacerda:feature/pep518

But also the second option.

@ethanhs, seems that as more and more projects adopt it, more sense has to put all that config files centralized. There is divergent opinions for projects with very large pytest.ini, flake8.ini and so on. So maybe individual config files should continue where they are.

[pslacerda]

Maybe that branch isn't so rudimentary and maybe you allow I submmit this PR.

EDIT: there is a bug that nested tables dont flatten correctly here. I need to work more on it.

[gvanrossum]

If you do want to continue this discussion please do submit the PR. Please understand that doesn't mean it will be accepted.

[gvanrossum]

Thanks -- I finally understand that TOML is meant as an alternative to mypy.ini. My apologies for not understanding this earlier. There is no particular urgency for this, is there? The PR is nice to have but we'll need to find time to review it carefully. Also, did you add tests?

[pslacerda]

Take your time to review, there is no urgency. I adapted some tests and also put my opinions on the PR.

[gvanrossum]

I've rejected the PR, and I'm going to close this issue. I don't think this solves a real problem for mypy. Sorry!

[pslacerda]

I'm saying that mypy is a build/test time requirement in my projects.

[gvanrossum]

PEP 518 doesn't say that all build/test requirements must use pyproject.toml for their own configuration. It only says that the list of packages to be installed must be specified there (as opposed to in a parameter to the setup() call in setup.py).

[and-semakin]

Hey, it would be nice to remove setup.cfg and completely switch to using fresh modern pyproject.toml but we can't do this because of mypy 😞

[JelleZijlstra]

There's actually some risk in doing that. http://github.com/python/black only supports pyproject.toml as its configuration file, but this has caused grief for various users because the mere presence of pyproject.toml causes some versions of pip to process packages differently, even if pyproject.toml contained only Black configuration files.

[chancecarey]

There's actually some risk in doing that. http://github.com/python/black only supports pyproject.toml as its configuration file, but this has caused grief for various users because the mere presence of pyproject.toml causes some versions of pip to process packages differently, even if pyproject.toml contained only Black configuration files.

I don't think that's a reasonable concern. PEP 518 indicates that pyproject.toml is "the way forward". That some tools fail to parse it correctly is on them, it shouldn't be used as evidence that implementing support for pyproject.toml is a bad thing.

[chancecarey]

@gvanrossum

I've rejected the PR, and I'm going to close this issue. I don't think this solves a real problem for mypy. Sorry!

Perhaps not for mypy, but I think having a good standard format and place to store config for tools that are used during development is a boon. PEP 518 does specify a [tool] section, and comments on it:

The [tool] table is where any tool related to your Python project, not just build tools, can have users specify configuration data as long as they use a sub-table within [tool], e.g. the flit tool would store its configuration in [tool.flit]

[prokher]

@gvanrossum

I've rejected the PR, and I'm going to close this issue. I don't think this solves a real problem for mypy. Sorry!

Indeed, MyPy's support of pyproject.toml does not solve a real problem now. But taking into account the inter-language competition we have (Python vs JS vs Go vs TypeScript vs ...) everything matters. Especially if we recall that the population of software engineers grow at a high rate and languages basically fight for newcomers. I think that today the beauty of the ecosystem is as important as the beauty of the language itself. Python is a beautiful and expressive language, but why should I have all this configuration-management-related files in the root of every Python project I work on (.pylintrc, .mypyrc, setup.py, setup.cfg, isort.cfg, ...)? It just looks ugly IMHO. I agree, it is not a serious problem to fight against, but pyproject.toml is already here and it is seems not to be a big deal to support it in MyPy. Moreover this could be a good example for other Python tools to follow. So one day we can end with a single configuration file for every Python project, it would be nice.

Just sharing.

[chancecarey]

@prokher

but pyproject.toml is already here and it is seems not to be a big deal to support it in MyPy

Indeed, and someone already has a working PR ...

[uranusjr]

Since PEP 631 is accepted (Dependency specification in pyproject.toml), it seems like pyproject.toml is cemented as the way to do project configuration and setup.cfg is going the way of the dodo.

PyPA’s position has always been, and likely will continue be that pyproject.toml is only cemented as the way to declare packaging metadata. We do not encourage projects to store non-packaging-related configuration in pyproject.toml (although we don’t stop projects if they insist). It would be perfectly appropriate if Mypy decides to not support configuring with pyproject.toml at all.

It is also not accurate to say setup.cfg is going to be obsolete. Packaging tools are still free to use their own configuration file for things specific to themselves (i.e. things not relevant to any other packaging tools), and I can see setuptools continuing to support setup.cfg for those usages even after they implement PEP 631. The thing that might loose popularity is using setup.cfg as the de-facto standard project-level configuration file. Mypy can choose to drop support to setup.cfg (because using setup.cfg in this way will no longer be popular), support pyproject.toml (because it’s okay to do it and people want it), or do neither or both.

[johnthagen]

We do not encourage projects to store non-packaging-related configuration in pyproject.toml

I'm not disagreeing with anything you said, but I do think it's worth pointing out that PEP 518 explicitly defines the [tool] table as an official mechanism for tools to define non-packaging-related configuration in pyproject.toml. To me that is at least an implicit nod towards projects doing this if they wish.

[uranusjr]

An earlier version of PEP 518 led to a lot of finger-pointing when Black and Towncrier (examples from the top of my head, there may be more) started to use the tool table. The language was soften to stop the argument, but PyPA’s stance does not change much.

[lig]

Let's face it, the "PyPA’s stance" is just an opinion of a small number of community members which clearly contradicts the process actually happening in the community and the request for that being highly spread.

[brettcannon]

I think a better way to phrase it is the PyPA is not mandating tools use pyproject.toml for tool configuration outside of what is already in PEPs (e.g. PEP 517 and 518), but the support for doing so is definitely there and available for tools that want to use it (hence the "do not encourage" from @uranusjr ; it's not the same as "discourage", just that the PyPA is not taking a stance since it's way beyond any mandate or purview of PyPA to do so).

Personally, I want tools to use it. 😉

[Jasha10]

Ok there has been much talk about this issue, and since it is re-opened, I'm guessing that the mypy maintainers would be willing to humor pull-requests from the community.

So, maybe it's time to shift the discussion from "gosh I wish this feature were implemented" to "how can this feature be implemented".

[JelleZijlstra]

Exactly. The next step here should be a pull request. I'll try to make time to review it, but first we need a pull request.

[Jasha10]

I am not familiar with mypy internals. Does mypy have some sort of config IR that is used as a destination for both mypy.ini files and command line arguments?
(I'm thinking that, if so, this config IR could be a destination for pyproject.toml too)

[hauntsaninja]

Refer to #5208 (review)

[patrickelectric]

Any plan to do a release with this feature ?

[JukkaL]

Hopefully there's a new release within the next month (as always, no promises).

[kthy]

I hope so, because the docs are already updated and as a new user I've just spent an hour wondering why I couldn't get it to work. Then I found this issue. 🤡

[jeffnyman]

I understand this is closed but I'm trying to understand why something I used before doesn't work. I was at least able to do this before:

poetry run mypy --config-file pyproject.toml tests

In my pyproject.toml I would have:

[mypy]

[mypy-expects]
ignore_missing_imports = true

This would work. The message about missing imports would be gone for the expects library. It no longer does when I'm using 0.910. In fact, it looks like it stopped working in 0.900. If I go back to 0.812, it works fine.

I was just calling this out to make sure if this was a deliberate change.

[Tinche]

I understand this is closed but I'm trying to understand why something I used before doesn't work. I was at least able to do this before:

poetry run mypy --config-file pyproject.toml tests

In my pyproject.toml I would have:

[mypy]

[mypy-expects]
ignore_missing_imports = true

This would work. The message about missing imports would be gone for the expects library. It no longer does when I'm using 0.910. In fact, it looks like it stopped working in 0.900. If I go back to 0.812, it works fine.

I was just calling this out to make sure if this was a deliberate change.

The syntax in pyproject.toml is a little different, you need to use [[tool.mypy.overrides]] sections. It's in the docs somewhere.

[jeffnyman]

The syntax in pyproject.toml is a little different, you need to use [[tool.mypy.overrides]] sections. It's in the docs somewhere.

Ah! Much thanks. Yes, I did find it here: https://mypy.readthedocs.io/en/stable/config_file.html#using-a-pyproject-toml-file