Using --package fails when the code is in a src directory

[mauritsvanrees]

For example checkout icalendar and try to generate its readme:

bin/python -mreadme_renderer -p icalendar
Traceback (most recent call last):
  File "/Users/maurits/.pyenv/versions/3.11.4/lib/python3.11/importlib/metadata/__init__.py", line 563, in from_name
    return next(cls.discover(name=name))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/maurits/community/zest.releaser/lib/python3.11/site-packages/readme_renderer/__main__.py", line 62, in <module>
    main()
  File "/Users/maurits/community/zest.releaser/lib/python3.11/site-packages/readme_renderer/__main__.py", line 26, in main
    message = metadata(args.input)
              ^^^^^^^^^^^^^^^^^^^^
  File "/Users/maurits/.pyenv/versions/3.11.4/lib/python3.11/importlib/metadata/__init__.py", line 998, in metadata
    return Distribution.from_name(distribution_name).metadata
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maurits/.pyenv/versions/3.11.4/lib/python3.11/importlib/metadata/__init__.py", line 565, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for icalendar

Then do cd src and the same command, and it works. There it finds (or creates) the egg-info directory.

See also this report in zest.releaser where I found this problem.

Maybe it makes sense to do os.chdir("src") if this directory exists? Or try that if metadata(args.input) fails at first?

[miketheman]

@mauritsvanrees Looks like you're trying to use -p to detect from an installed package.

You'd either need to install the package into the environment, or omit the flag and pass the README.rst file path.

[mauritsvanrees]

No, the package is not installed. This is a fresh checkout. For packages without a src-layout, it works. For example:

git clone 
git@github.com:plone/plone.memoize.git
cd plone.memoize
python -mreadme_renderer -p plone.memoize
...
importlib.metadata.PackageNotFoundError: No package metadata was found for plone.memoize

Huh? I am sure that worked earlier today. Now I am confused. Looks like I need to do more testing. Sorry.

Passing the README.rst path is not exactly enough for me. I want the full PyPI long description, which in my code is often README.rst plus CHANGES.rst. Or their Markdown variants. Which is why I was happy with both the new -p option and the feature to be able to read markdown.

[miketheman]

I want the full PyPI long description

In plone.memoize it looks like you're already doing that?
https://github.com/plone/plone.memoize/blob/b52b96111654644835852fa23fcda70a02a486cb/setup.py#L8-L12

It also looks like you're already performing a description validation by running twine check:
https://github.com/plone/plone.memoize/blob/b52b96111654644835852fa23fcda70a02a486cb/tox.ini#L140

I guess it comes down to "what kind of description are you trying to render, and why?"

[mauritsvanrees]

TLDR: I am working around it now, so if you don't see value in changing this (I could make a PR), then feel free to close this issue. Maybe I am using this tool in a way that is unintended.

I am trying to render the PyPI long description of a package I am currently developing, to see if it renders nicely on PyPI. twine check helps to check that the ReStructuredText or MarkDown is valid, but I still want to actually see it in a browser. In my usual development setup, readme_renderer would be globally installed (for example with pipx), and not in the same venv as the package under development.

This seems to be why PR #271 was made, adding a -p option to be able to check this long description in the browser. But this needs package metadata to be available already. Okay, so I call python setup.py egg_info or python -mbuild -s . and then the egg info directory is available.

So now I call python -mreadme_renderer -p icalendar. This fails:

importlib.metadata.PackageNotFoundError: No package metadata was found for icalendar

When I first do cd src, then readme_renderer works. That is what this issue is about: it would be useful to me if readme_renderer could do cd src first, in case it is needed. I would be happy to create a PR for that.

---

What I do now, is work around this small limitation in an own tool. I am using and developing zest.releaser, which includes a tool called longtest for testing the long description of a Python package. For years already this has been a wrapper around readme_renderer. So belatedly: thanks for this tool!

Until now, what longtest did, was this:

• Call python setup.py --long-description.
• Call readme_renderer.render over the result.
• Open the resulting html file in a browser.

Since the Python world is moving slowly away from setup.py, the first step will not work for all packages. It sounded like the new --package option would mean we could ask readme_renderer to render the readme of a package under development, but as we have seen, this only works when the egg info directory is in the current directory. Or if the package is in the same venv as readme_renderer.

We have fixed it for our use case in zestsoftware/zest.releaser#428 by doing the cd src there ourselves.

[miketheman]

@mauritsvanrees Thanks for your in-depth explanation - that's very helpful!

In regards to usage - the CLI is a little tricky, and was removed from documentation and more recently the --package feature was added. This is also apparent by the lack of any entry_points - it's not fully meant to operate as its own CLI tool.

The way --package works is by using the importlib.metadata.metadata() function to look up a distribution. The way it finds the correct package is by using the logic described here: https://docs.python.org/3/library/importlib.metadata.html#distribution-discovery

In your example, are you performing an editable install of the package under development?

$ pip install readme_renderer
...
$ pwd
/workspace/icalendar
$ pip install -e .
...
$ python -m readme_renderer --package icalendar | head -n 10
/workspace/icalendar/venc/lib/python3.11/site-packages/readme_renderer/markdown.py:44: UserWarning: Markdown renderers are not available. Install 'readme_renderer[md]' to enable Markdown rendering.
  warnings.warn(_EXTRA_WARNING)
<p>The <a href="https://pypi.org/project/icalendar/" rel="nofollow">icalendar</a> package is a <a href="https://www.ietf.org/rfc/rfc5545.txt" rel="nofollow">RFC 5545</a> compatible parser/generator for iCalendar
files.</p>
<hr class="docutils">
<dl class="field-list simple">
<dt>Homepage<span class="colon">:</span></dt>
<dd><p><a href="https://icalendar.readthedocs.io" rel="nofollow">https://icalendar.readthedocs.io</a></p>
</dd>
<dt>Code<span class="colon">:</span></dt>
<dd><p><a href="/collective/icalendar" rel="nofollow">https://github.com/collective/icalendar</a></p>
</dd>

When running the tool, it needs to be able to discover the package via name, so it if's outside the environment, it won't be able to discover it. If it's installed in editable mode, that means it would be added to the sys.path, and then be discoverable by --package parameter.