Skip to content

ripariancommit merge - Documentation#24

Merged
b3yc0d3 merged 45 commits intob3yc0d3:masterfrom
ripariancommit:dev/docs
May 31, 2025
Merged

ripariancommit merge - Documentation#24
b3yc0d3 merged 45 commits intob3yc0d3:masterfrom
ripariancommit:dev/docs

Conversation

@ripariancommit
Copy link
Contributor

This patchset is a merge of my documentation feature branch. It is based on the same refs as #23, and they represent the first 14 commits.

Changes

  • Adds sphinx-based project documentation under the :docs/ directory.
  • Adds complete module, class, and method documentation throughout the rule34Py package, in the Google python style.
  • Adds a top-level Makefile, to provide easier entrypoints for building the project, running the tests, and linting.
  • Moves developer-focused user content either into a new Developer Guide (in the docs) or a Contributing Guide (in the docs).
  • Deprecates the old docs README.
  • Converts the downloader example into a tutorial in the docs.
  • Adds a PR check github workflow, that should check that each repo PR builds the docs.
    • Naturally, I intend to extend this into running the unit tests and whatever other PR validation we want to do.
  • Adds a gh-pages workflow to build and publish the GH pages on each push (or merge) into the master ref.
  • Adds a linter - ruff - which seems to be the new hotness in python linting, and was helpful to lint my docstrings.
    • At the moment, I only have the documentation-related rules enabled. But I will enable the others in future patchsets.
  • Adds a user-guide for how to clear Cloudflare captchas using the featuers in Revitalize the client module (part 2 / 2) #23.

No changelog changes for this PR - because nothing here should impact users.

Testing

This patchset does not contain any intentional functional changes to the core module (apart from the contents of patchset #23).

  • With this change, all existing unit tests still pass.
(.venv) [0] $ make check
PYTHONPATH=rule34Py/.. python3 -m pytest tests/unit/
==================================================================== test session starts =====================================================================
platform linux -- Python 3.13.2, pytest-8.3.5, pluggy-1.5.0 -- <snip>/rule34Py/.venv/bin/python3
cachedir: .pytest_cache
rootdir: <snip>/rule34Py/tests
configfile: pytest.ini
collected 29 items                                                                                                                                           

tests/unit/test_html.py::test_ICamePage_init PASSED                                                                                                    [  3%]
tests/unit/test_html.py::test_ICamePage_top_chart_from_html PASSED                                                                                     [  6%]
tests/unit/test_html.py::test__PoolHistoryPage__events_from_html PASSED                                                                                [ 10%]
tests/unit/test_html.py::test__PoolPage__pool_from_html PASSED                                                                                         [ 13%]
tests/unit/test_html.py::test_TagMapPage_init PASSED                                                                                                   [ 17%]
tests/unit/test_html.py::test_TagMapPage_map_points_from_html PASSED                                                                                   [ 20%]
tests/unit/test_html.py::test_TopTagsPage PASSED                                                                                                       [ 24%]
tests/unit/test_html.py::test_TopTagsPage_top_tags_from_html PASSED                                                                                    [ 27%]
tests/unit/test_module.py::test_version PASSED                                                                                                         [ 31%]
tests/unit/test_readme.py::test_module_import PASSED                                                                                                   [ 34%]
tests/unit/test_readme.py::test_get_comments PASSED                                                                                                    [ 37%]
tests/unit/test_readme.py::test_get_post PASSED                                                                                                        [ 41%]
tests/unit/test_readme.py::test_icame PASSED                                                                                                           [ 44%]
tests/unit/test_readme.py::test_search PASSED                                                                                                          [ 48%]
tests/unit/test_readme.py::test_get_pool PASSED                                                                                                        [ 51%]
tests/unit/test_rule34Py.py::test_rule34Py_get_comments PASSED                                                                                         [ 55%]
tests/unit/test_rule34Py.py::test_rule34Py_get_pool PASSED                                                                                             [ 58%]
tests/unit/test_rule34Py.py::test_rule34Py_get_post PASSED                                                                                             [ 62%]
tests/unit/test_rule34Py.py::test_rule34Py_icame PASSED                                                                                                [ 65%]
tests/unit/test_rule34Py.py::test_rule34Py_iter_search PASSED                                                                                          [ 68%]
tests/unit/test_rule34Py.py::test__rule34Py__user_agent PASSED                                                                                         [ 72%]
tests/unit/test_rule34Py.py::test_rule34Py_random_post PASSED                                                                                          [ 75%]
tests/unit/test_rule34Py.py::test_rule34Py_random_post_id PASSED                                                                                       [ 79%]
tests/unit/test_rule34Py.py::test__rule34Py__request_limiter PASSED                                                                                    [ 82%]
tests/unit/test_rule34Py.py::test_rule34Py_search PASSED                                                                                               [ 86%]
tests/unit/test_rule34Py.py::test_rule34Py_tag_map PASSED                                                                                              [ 89%]
tests/unit/test_rule34Py.py::test_rule34Py_tagmap PASSED                                                                                               [ 93%]
tests/unit/test_rule34Py.py::test_rule34Py_top_tags PASSED                                                                                             [ 96%]
tests/unit/test_rule34Py.py::test_rule34Py_version PASSED                                                                                              [100%]

==================================================================== 29 passed in 26.05s =====================================================================

Process

  1. This patchset should be pulled after Revitalize the client module (part 2 / 2) #23 . If necessary I can rebase it after the former is merged, to make pulling easier.
  2. Before this patchset is merged, @b3yc0d3 should go into the mainline repo settings and enable the github pages feature. Then, on-merge I think the gh-pages workflow trigger in it should activate and automatically publish the gh-pages content to the gh-pages ref as on my repo. I didn't have to do anything else with tokens or otherwise. It just worked. :)

ripariancommit and others added 30 commits May 17, 2025 16:58
Clean up the get_comments() method. No major changes.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
The rule34Py.random_post() optionally accepts a list of tags. If tags
are provided, the method returns a random post from the first page of
tag search results. Otherwise, it functions like the website Posts >
Random function and returns a random post from the total population.

This alternating behavior is a little strange. The tagged algorithm
isn't really 'random', in the sense that it is limited to the latest
page of results. Someone who wants that behavior can simply do
random.choice(rule34Py.search([tags...])).

So remove the tags functionality from random_post() and make it always
behave like the website's function.

Further, make the rule34Py.random_post_id() method public and correct
its return type to 'int'. There is little reason that it should be
private, when other methods with similar levels of testing and support
are public.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Use the newer client _get() method when requesting server URLs.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
The get_post() method fetches information about a single post based on a
post_id.

Clean up the code a little.

Return None when a matching Post cannot be found, rather than an empty
List.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Rule34.xxx occassionally uses cloudflare's captcha defense system to
disuade bots from accessing the website (not the API server). Though we
should not 'defeat' this defense, we should add some capability for the
client to satisfy the captcha, once the user has proven their humanity.

Add a new 'captcha_clearance' attribute to the client. Its value, and
the 'user_agent' value will be taken from the R34_CAPTCHA_COMPLIANCE and
R34_USER_AGENT environment variables, if they are asserted. When
populated, the client will use these values in its requests.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
The rule34Py.get_pool() method returns a list of Post ids that are
assigned to a pool on R34. This method works, but fails to retrieve all
the useful information about the pool (it's name, description, etc.).

Implement a new Pool dataclass, that holds this information. Revitalize
the get_pool() method to parse this info from the pool page's HTML and
generate one of these new objects.

Also implement the backend for an additional dataclass (PoolHistoryEvent) where
we can eventually store events from the pool's history record. It is not
currently exposed.

Regenerate the responses cache.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Some portions of the client API use the base rule34.xxx site (ie. the
interactive PHP endpoint), rather than the api.rule34.xxx endpoint.
The base site isn't really designed to be accessed at full velocity by
programmic clients.

Be kind to the PHP endpoint by adding a requests adapter for it that
limits requests to about once per second.

The limiter can be disabled with the set_base_site_rate_limit() method.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Add the new requests-ratelimiter dependency to the project.

Add a section to track test dependencies.
Use tabs for indentation. Sort elements alphabetically.

This commit contains no functional changes.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
All of the client module methods have been reviewed at this point.
Hopefully most or all of the spelling mistakes have been corrected. If
not, they hopefully will be when we get to revitalizing the project
documentation.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Typo.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
The default user-agent string in the client contains a 'test' value and
is different from what it was previously.

Correct the user-agent string in the client to reflect what it was
previously. Add a test to confirm that the user-agent string is
well-formed and can be changed.

Deprecate the __vars__.__header__ and __useragent__ variables, as they
are no longer used.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Add an initial implementation for sphinx documentation. Auto-generate
module documentation for the rule34Py module.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Add the intersphinx extension to automatically link to external sphinx
docs. Also enhance the autodoc configuration.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
The current canonical project documentation is a README.rst file that is
rather out of date, and has been replaced by the sphinx-based
documentation.

Remove the old file.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Convert the previous downloader example into a piece of tutorial
documentation.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Add python dependencies necessary to build the project documentation.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Add a github workflow to build the documentation on Pull Requests, so
that we can confirm the quality of the changes.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Add a job to the PR Checks workflow which confirms that the project
builds on Windows runners. This workflow does not generate artifacts
because they should be the same as the linux run.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Add a simple Makefile that comprehends how to build the project wheel,
source distribution, and HTML docs; and how to run the unit tests and
clean the project.

Update the PR Checks to use the Makefile.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Add a workflow to build and publish the project documentation to Github
Pages, on each push to the master ref.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Revitalize the module and class documentation for the Post class.

This commit contains no intentional changes to functionality.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Sphinx already does a good-enough job autodocumenting class
initialization under the class's heading in the docs. There is no need
to override the default value and force explicit documentation.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Add a python linter to the Makefile and a new target - lint - that lints
the python files for quality. Currently, it only lints for
docstring-related issues.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
The ruff linter reports several issues with the project docstrings and
type hinting. Fix them.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
During the PR checks, run the project linter.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
* Use the README markdown as the basis for the main documentation index.
* Move most of the information that is of interest to developers, to an
  independent developer-guide document in the docs.
* Add section descriptions to some of the sections.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Create a new contributing guide within the documentation, and move the
contributing information from the README to it.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Since the autodocs config statement to document __init__() methods was removed,
parameter documentation for __init__ must now go under the class
docstring.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Users will probably be understandably confused as to how to use the
captcha-clearance feature of the client class. Add a user guide on how
to use it.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
The dev/docs trigger was added to the gh-pages workflow for testing.
Remove it, now that the patchset is ready. Pages should only be
published from the master ref.

Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
Signed-off-by: Riparian Commit <ripariancommit@protonmail.com>
@b3yc0d3 b3yc0d3 added documentation Improvements or additions to documentation enhancement New feature or request labels May 31, 2025
@b3yc0d3 b3yc0d3 merged commit b790977 into b3yc0d3:master May 31, 2025
b3yc0d3 added a commit that referenced this pull request Jun 9, 2025
### Added

- Added captcha-compliance capabilities. You can now pass the client your captcha clearance token by completing the captcha in your browser, opening the response header, and writing your `cf_clearance` token into the client's `rule34Py.captcha_clearance` attribute and setting the client's `user_agent` attribute to match your browser.
	- Alternatively, set your execution environment's `R34_CAPTCHA_CLEARANCE` and `R34_USER_AGENT` variables to the appropriate values. The client will read form them during initialization.
	- See [this user guide](https://b3yc0d3.github.io/rule34Py/guides/captcha-clearance.html) for more information. (#24)
- Added a rate limiter for requests to the rule34.xxx base site (the PHP endpoint). By default, the client will now limit API calls that use this endpoint to 1 per second.
	- This behavior can be disabled by setting `rule34Py.set_base_site_rate_limit(False)`.
	- There is no rate limit on the api.rule34.xxx endpoint, which is assumed to handle rate-limiting on the server-side.
	- This new feature requires the `requests-ratelimiter` module.
- Added project documentation via sphinx docs. See: https://b3yc0d3.github.io/rule34Py/. It can be built by calling `make html`. (#24)

### Changed

- Changed the behavior of the `rule34Py.random_post()` method to function more like the website. The method now accepts no `tag` parameters, and returns a random post ID from all posts on the site. Users who want to use the old behavior of returning a random post from the first 1000 tag-search results are directed to do something like `random.choice(rule34Py.search([tags...]))`.
- Changed the `rule34Py.get_pool()` method to return a `Pool` object containing more complete information about a Pool.
	- The Pool's post ids can be accessed via the `Pool.posts` attribute.
- Moved the `:examples/` python recipes into tutorial documentation in the new sphinx documentation (`:docs/tutorials`). (#24)

### Deprecated

- Deprecated support for python runtimes 3.5 to 3.8, as they are EOL. The minimum supported python version is now 3.9. (#27)

### Removed

- Removed the deprecated `legacy.setup.py` file. Users are instructed to upgrade their setuptools installation and use `pyproject.toml` to build the project.
- Removed the `debug.py` file. Users are instructed to use the `make check` Makefile target to run the project's test suite. (#27)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants