Generate recipe from project tree using pypa/build #541

dholth · 2024-07-12T15:52:36Z

Description

This is the start of something I've wanted: grayskull pypi . on a checkout to generate a recipe, suitable for committing to the project tree. Like https://github.com/conda/conda/blob/main/recipe/meta.yaml#L5-L7 where source is "../" or the parent directory of the meta.yaml.

pypa/build gets all project metadata in a standard way, including the build-system requires (setuptools, flit, poetry). If these are not already installed, we should add a "conda install " before project.metadata_path(...) or bail and let the user install those. We would also change the pip command used in the build: section so that we never accidentally pip install a build requirement.

Remaining to be done, process all possible metadata out of distribution and into recipe; possibly deal with late "we don't know the package name until after fetch_data" more elegantly since this breaks grayskull's current assumptions.

It's also a little awkward that the recipe output dir is always <given directory>/<project name>. These would be <checkout>/conda.recipe/meta.yaml when included as first-party recipes.

Fix #542

for more information, see https://pre-commit.ci

grayskull/strategy/py_build.py

for more information, see https://pre-commit.ci

dholth

Thanks for this useful project

dholth · 2024-07-12T17:48:54Z

grayskull/strategy/py_build.py

+                    f'#% set version = "{config.version}" %}}\n'
+                )
+
+                recipe["package"]["version"] = "{{ version }}"


This puts quotes around version: '{{ version }}'; how do we control the YAML more closely with conda-souschef's Recipe?

conda-souschef is just a wrapper around ruamel yaml, you can still manipulate the ruamel yaml object directly.
you can do something like:

recipe["package"]["version"].value = "{{ version }}"

dholth · 2024-07-12T17:49:46Z

grayskull/strategy/py_build.py

+                str(r).rsplit(";", 1)[0]
+                for r in requirements
+                if not r.marker or r.marker.evaluate()
+            ]


How do we include the marker as a YAML comment?

These are evaluated in the environment grayskull runs in, and might not be correct for the end user.

there is a property called .comment that you can use for that
a recipe/section object uses this mixin
https://github.com/marcelotrevisani/souschef/blob/92cee9159fb65fa035593f1a8b60a931413ab581/src/souschef/mixins.py#L14

dholth · 2024-07-12T17:50:39Z

grayskull/strategy/py_build.py

+            recipe["build"][
+                "script"
+            ] = "{{ PYTHON }} -m pip install . -vv --no-deps --no-build-isolation"
+            # XXX also --no-index?


It would be nice to be careful about not fetching pip requirements, even the build-system requirements (poetry, flit-core, hatchling, ...)

sorry, I think I didn't follow, what do you mean?

dholth · 2024-07-12T17:51:12Z

grayskull/strategy/py_build.py

+                for ep in distribution.entry_points
+                if ep.group == "console_scripts"
+            ]
+            recipe["test"] = compose_test_section(metadata_dict, [])


Maybe this sort of "copy all fields from one format to another" code is destined to be messy...

dholth · 2024-07-12T17:55:37Z

grayskull/utils.py

+def origin_is_tree(name: str) -> bool:
+    """Return True if it is a directory"""
+    path = Path(name)
+    return path.is_dir()


We could look for pyproject.toml or setup.py; it should work with either.

that can work, but you also need to look for setup.cfg

With the pypa/build method, pyproject.toml is the only necessary file to get the metadata.

I agree, but not all projects uses it :/

pypa/build works with setup.py or pyproject.toml. It asks the underlying build system to create a wheel with package.dist-info/METADATA. Then we read METADATA and not pyproject.toml. So with this technique we don't touch setup.py, setup.cfg or pyproject.toml; pypa/build writes METADATA and we read that.

for more information, see https://pre-commit.ci

grayskull/license/discovery.py

marcelotrevisani · 2024-07-28T19:57:54Z

is this still as a draft?

dholth · 2024-07-29T13:26:43Z

I would like feedback but it is still a draft, it's not clean yet.

beenje

Very happy to see this PR!

I currently use pyproject-build to create a sdist and then run grayskull to generate the recipe from that local sdist.
This will make the process even easier.

One issue I stumbled upon is to find the module name to import (in the test section). It can differ from the distribution name.
I have a workaround when setuptools is used: #551

Do you know if this information can be retrieved from build?

beenje · 2024-08-17T08:18:34Z

grayskull/strategy/py_build.py

+                recipe._yaml._yaml_get_pre_comment()[0].value = (
+                    f'#% set name = "{config.name}" %}}\n'
+                    f'#% set version = "{config.version}" %}}\n'
+                )


Instead of that, you can use

set_global_jinja_var(recipe, "name", config.name) set_global_jinja_var(recipe, "version", config.version)

beenje · 2024-08-17T08:25:41Z

grayskull/strategy/py_build.py

+
+import build
+from conda.exceptions import InvalidMatchSpec
+from conda.models.match_spec import MatchSpec


Having to import conda will make grayskull more difficult to install.
Is it worth adding conda as dependency just for that function?

I haven't checked how requirements are parsed in the current code.

beenje · 2024-08-17T08:28:35Z

grayskull/strategy/py_build.py

+
+            about = {
+                "summary": metadata["summary"],
+                "license": metadata["license"],


On an example I tested, license was set to the full license text content. The pyproject.toml was using license = {file = "LICENSE"}. In the recipe, we want the SPDX identifier.

dholth · 2024-08-17T17:01:18Z

@beenje I have been refining this technique here. It makes a wheel and immediately converts it to .conda without bothering with conda-build or a recipe. But it's focused on editable installs and there's no supporting code to manage build dependencies etc. In the future the metadata convert part might be shareable.

pyproject.toml doesn't give your imports in a generic way. You could inspect the contents of a wheel. You could have build-system-specific handling like how flit's name= must match the Python import.

beenje · 2024-08-19T07:06:09Z

@beenje I have been refining this technique here. It makes a wheel and immediately converts it to .conda without bothering with conda-build or a recipe. But it's focused on editable installs and there's no supporting code to manage build dependencies etc. In the future the metadata convert part might be shareable.

Nice! Another project I have to look into :-)

dholth and others added 2 commits July 12, 2024 11:41

generate recipe from project tree using pypa/build

bb44c0e

[pre-commit.ci] auto fixes from pre-commit.com hooks

09485b3

for more information, see https://pre-commit.ci

github-advanced-security bot found potential problems Jul 12, 2024

View reviewed changes

grayskull/strategy/py_build.py Fixed Show fixed Hide fixed

grayskull/strategy/py_build.py Fixed Show fixed Hide fixed

grayskull/strategy/py_build.py Fixed Show fixed Hide fixed

grayskull/strategy/py_build.py Fixed Show fixed Hide fixed

dholth and others added 6 commits July 12, 2024 13:32

approach grayskull (pypi)'s result

bbbf259

[pre-commit.ci] auto fixes from pre-commit.com hooks

c712fe9

for more information, see https://pre-commit.ci

allow missing conda dependency

e967d1c

[pre-commit.ci] auto fixes from pre-commit.com hooks

e262ec9

for more information, see https://pre-commit.ci

fallback if requires is None

ad98bb6

[pre-commit.ci] auto fixes from pre-commit.com hooks

8ef8e25

for more information, see https://pre-commit.ci

dholth commented Jul 12, 2024

View reviewed changes

dholth mentioned this pull request Jul 12, 2024

[FEATURE] Use pypa/build to generate project metadata locally from any build system #542

Open

dholth and others added 4 commits July 12, 2024 17:26

expect additional argument in spy

5f39732

find extras example

ec43149

[pre-commit.ci] auto fixes from pre-commit.com hooks

2734288

for more information, see https://pre-commit.ci

add simpler license matcher

9ba91d8

github-advanced-security bot found potential problems Jul 12, 2024

View reviewed changes

grayskull/license/discovery.py Fixed Show fixed Hide fixed

dholth added 3 commits July 12, 2024 18:02

remove newer typing

c6d2934

mention pkg_resources API

a6ded19

pyproject

523f893

dholth mentioned this pull request Jul 16, 2024

Convert to .conda as part of the pip install? conda-incubator/conda-pypi#28

Open

Merge branch 'main' into from-project

16a8933

beenje reviewed Aug 17, 2024

View reviewed changes

beenje mentioned this pull request Aug 17, 2024

Fix tests #552

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate recipe from project tree using pypa/build #541

Generate recipe from project tree using pypa/build #541

dholth commented Jul 12, 2024 •

edited

Loading

dholth left a comment

dholth Jul 12, 2024

marcelotrevisani Aug 4, 2024

dholth Jul 12, 2024

dholth Jul 12, 2024

marcelotrevisani Aug 4, 2024

dholth Jul 12, 2024

marcelotrevisani Aug 4, 2024

dholth Jul 12, 2024

dholth Jul 12, 2024

marcelotrevisani Aug 4, 2024

dholth Aug 5, 2024

marcelotrevisani Aug 18, 2024

dholth Aug 19, 2024

marcelotrevisani commented Jul 28, 2024

dholth commented Jul 29, 2024

beenje left a comment

beenje Aug 17, 2024

beenje Aug 17, 2024

beenje Aug 17, 2024

dholth commented Aug 17, 2024

beenje commented Aug 19, 2024

Generate recipe from project tree using pypa/build #541

Are you sure you want to change the base?

Generate recipe from project tree using pypa/build #541

Conversation

dholth commented Jul 12, 2024 • edited Loading

Description

dholth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcelotrevisani commented Jul 28, 2024

dholth commented Jul 29, 2024

beenje left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dholth commented Aug 17, 2024

beenje commented Aug 19, 2024

dholth commented Jul 12, 2024 •

edited

Loading