Setup Sphinx for developer-docs #427

dhruvkb · 2021-12-23T07:26:32Z

Description

This PR installs and configures Sphinx as a proof of concept to show how documentation for the API's Django codebase would work.

This PR

adds Sphinx, the Furo theme and an auto-reloading builder
configures Sphinx for Django support to autogenerate docs for the catalog module
lightly customises the Furo theme for branding with Openverse color palette
adds just recipes for building, hosting and live-reloading the docs site

As a proof-of-concept-ish stage right now, most of the documents in the repo right now, are auto-generated. We can add more long-form documentation .rst files later on.

Testing Instructions

Clone the repo and check out the sphinx branch.
*Rebuild the web image to install dependencies with Pipenv.
*Ensure that you have a populated .env file inside openverse-api/.
Run the live-reload server: just sphinx-live. Visit port 3000 on localhost.
Compile the docs: just sphinx-make.
Serve the compiled docs just sphinx-serve. Visit port 3000 on localhost.

Checklist

My pull request has a descriptive title (not a vague title like Update index.md).
My pull request targets the default branch of the repository (main) or a parent feature branch.
My commit messages follow best practices.
My code follows the established code style of the repository.
I added or updated tests for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

AetherUnbound

This is amazing!! 🥳 I love the work you've done tweaking the theme, it feels very "Openverse" 🙂

When trying to run this myself with the steps provided, I encountered the following errors:

Missing DJANGO_SECRET_KEY. I was able to resolve this by running DJANGO_SECRET_KEY=foobar just sphinx-make.

$ just sphinx-make
cd openverse_api && pipenv run sphinx-build -M html docs/ build/
Running Sphinx v4.3.2

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/sphinx/config.py", line 329, in eval_config_file
    exec(code, namespace)
  File "/home/aether/git-a8c/openverse-api/openverse_api/docs/conf.py", line 16, in <module>
    django.setup()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/settings.py", line 33, in <module>
    SECRET_KEY = config("DJANGO_SECRET_KEY")  # required
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/decouple.py", line 206, in __call__
    return self.config(*args, **kwargs)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/decouple.py", line 90, in __call__
    return self.get(*args, **kwargs)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/decouple.py", line 75, in get
    raise UndefinedValueError('{} not found. Declare it as envvar or define a default value.'.format(option))
decouple.UndefinedValueError: DJANGO_SECRET_KEY not found. Declare it as envvar or define a default value.

error: Recipe `sphinx-make` failed on line 179 with exit code 2

Connection to elasticsearch failed. I was able to fix this by just up - should we expect the stack to be up in order to compile the docs? If so, maybe we need to add it as a dependency for those just recipes?

Error 2

DJANGO_SECRET_KEY=foobar just sphinx-make
cd openverse_api && pipenv run sphinx-build -M html docs/ build/
Running Sphinx v4.3.2
/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/grequests.py:22: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['urllib3.util.ssl_ (/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/util/ssl_.py)', 'urllib3.util (/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/util/__init__.py)']. 
  curious_george.patch_all(thread=False, select=False)
GET http://localhost:9200/ [status:N/A request:0.004s]
Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/gevent/_socketcommon.py", line 607, in connect
    raise _SocketError(err, strerror(err))
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/http/client.py", line 1034, in _send_output
    self.send(msg)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/http/client.py", line 974, in send
    self.connect()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fb040bcc7f0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb040bcc7f0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/elasticsearch/connection/http_requests.py", line 166, in perform_request
    response = self.session.send(prepared_request, **send_kwargs)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb040bcc7f0>: Failed to establish a new connection: [Errno 111] Connection refused'))

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/gevent/_socketcommon.py", line 607, in connect
    raise _SocketError(err, strerror(err))
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/http/client.py", line 1034, in _send_output
    self.send(msg)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/http/client.py", line 974, in send
    self.connect()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fb040bcc7f0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb040bcc7f0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/elasticsearch/connection/http_requests.py", line 166, in perform_request
    response = self.session.send(prepared_request, **send_kwargs)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb040bcc7f0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/sphinx/config.py", line 329, in eval_config_file
    exec(code, namespace)
  File "/home/aether/git-a8c/openverse-api/openverse_api/docs/conf.py", line 16, in <module>
    django.setup()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/django/apps/config.py", line 301, in import_models
    self.models_module = import_module(models_module_name)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/models/__init__.py", line 2, in <module>
    from catalog.api.models.audio import (
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/models/audio.py", line 1, in <module>
    import catalog.api.controllers.search_controller as search_controller
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/controllers/search_controller.py", line 414, in <module>
    es = _elasticsearch_connect()
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/controllers/search_controller.py", line 410, in _elasticsearch_connect
    _es.info()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/elasticsearch/client/__init__.py", line 295, in info
    return self.transport.perform_request(
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/elasticsearch/transport.py", line 417, in perform_request
    self._do_verify_elasticsearch(headers=headers, timeout=timeout)
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
    raise error
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/elasticsearch/transport.py", line 569, in _do_verify_elasticsearch
    _, info_headers, info_response = conn.perform_request(
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/elasticsearch/connection/http_requests.py", line 194, in perform_request
    raise ConnectionError("N/A", str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb040bcc7f0>: Failed to establish a new connection: [Errno 111] Connection refused'))) caused by: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb040bcc7f0>: Failed to establish a new connection: [Errno 111] Connection refused')))

error: Recipe `sphinx-make` failed on line 179 with exit code 2

Warnings about missing Exempi. This didn't seem to affect the doc generation though.

Error 3

DJANGO_SECRET_KEY=foobar just sphinx-make
cd openverse_api && pipenv run sphinx-build -M html docs/ build/
Running Sphinx v4.3.2
/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/grequests.py:22: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['urllib3.util (/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/util/__init__.py)', 'urllib3.util.ssl_ (/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/urllib3/util/ssl_.py)']. 
  curious_george.patch_all(thread=False, select=False)
GET http://localhost:9200/ [status:200 request:0.006s]
GET http://localhost:9200/ [status:200 request:0.003s]
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 12 source files that are out of date
updating environment: [new config] 12 added, 0 changed, 0 removed
reading sources... [100%] index                                                                                                                                                                                                                                                            
/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/controllers/search_controller.py:docstring of catalog.api.controllers.search_controller.search:9: WARNING: Field list ends without a blank line; unexpected unindent.
/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/models/audio.py:docstring of catalog.api.models.audio.Audio:2: WARNING: Unexpected section title.

Inherited fields
================
/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/models/image.py:docstring of catalog.api.models.image.Image:2: WARNING: Unexpected section title.

Inherited fields
================
/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/models/models.py:docstring of catalog.api.models.models.ContentProvider:7: WARNING: Unexpected indentation.
/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/models/models.py:docstring of catalog.api.models.models.ContentProvider:9: WARNING: Block quote ends without a blank line; unexpected unindent.
/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/serializers/audio_serializers.py:docstring of catalog.api.serializers.audio_serializers.AudioReportSerializer.create:5: WARNING: Inline strong start-string without end-string.
/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/serializers/audio_serializers.py:docstring of catalog.api.serializers.audio_serializers.AudioReportSerializer.create:11: WARNING: Inline strong start-string without end-string.
/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/serializers/image_serializers.py:docstring of catalog.api.serializers.image_serializers.ImageReportSerializer.create:5: WARNING: Inline strong start-string without end-string.
/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/serializers/image_serializers.py:docstring of catalog.api.serializers.image_serializers.ImageReportSerializer.create:11: WARNING: Inline strong start-string without end-string.
WARNING: autodoc: failed to import module 'ccrel' from module 'catalog.api.utils'; the following exception was raised:
Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/sphinx/ext/autodoc/importer.py", line 70, in import_module
    return importlib.import_module(modname)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/utils/ccrel.py", line 5, in <module>
    import libxmp
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/__init__.py", line 50, in <module>
    from .core import XMPMeta, XMPIterator
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/core.py", line 50, in <module>
    from . import exempi as _cexempi
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/exempi.py", line 69, in <module>
    EXEMPI = _load_exempi()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/exempi.py", line 60, in _load_exempi
    raise ExempiLoadError('Exempi library not found.')
libxmp.ExempiLoadError: Exempi library not found.

/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/utils/oauth2_helper.py:docstring of catalog.api.utils.oauth2_helper.get_token_info:6: WARNING: Field list ends without a blank line; unexpected unindent.
/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/utils/watermark.py:docstring of catalog.api.utils.watermark.watermark:5: WARNING: Field list ends without a blank line; unexpected unindent.
/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/utils/exceptions.py:docstring of catalog.api.utils.exceptions.exception_handler:1: WARNING: Unknown target name: "drf docs".
WARNING: autodoc: failed to import module 'image_views' from module 'catalog.api.views'; the following exception was raised:
Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/sphinx/ext/autodoc/importer.py", line 70, in import_module
    return importlib.import_module(modname)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/views/image_views.py", line 4, in <module>
    import libxmp
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/__init__.py", line 50, in <module>
    from .core import XMPMeta, XMPIterator
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/core.py", line 50, in <module>
    from . import exempi as _cexempi
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/exempi.py", line 69, in <module>
    EXEMPI = _load_exempi()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/exempi.py", line 60, in _load_exempi
    raise ExempiLoadError('Exempi library not found.')
libxmp.ExempiLoadError: Exempi library not found.

WARNING: autodoc: failed to import module 'urls.auth_tokens' from module 'catalog'; the following exception was raised:
Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/sphinx/ext/autodoc/importer.py", line 70, in import_module
    return importlib.import_module(modname)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/urls/__init__.py", line 20, in <module>
    from catalog.api.views.image_views import ImageViewSet
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/views/image_views.py", line 4, in <module>
    import libxmp
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/__init__.py", line 50, in <module>
    from .core import XMPMeta, XMPIterator
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/core.py", line 50, in <module>
    from . import exempi as _cexempi
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/exempi.py", line 69, in <module>
    EXEMPI = _load_exempi()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/exempi.py", line 60, in _load_exempi
    raise ExempiLoadError('Exempi library not found.')
libxmp.ExempiLoadError: Exempi library not found.

WARNING: autodoc: failed to import module 'urls.swagger' from module 'catalog'; the following exception was raised:
Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/sphinx/ext/autodoc/importer.py", line 70, in import_module
    return importlib.import_module(modname)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/urls/__init__.py", line 20, in <module>
    from catalog.api.views.image_views import ImageViewSet
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/views/image_views.py", line 4, in <module>
    import libxmp
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/__init__.py", line 50, in <module>
    from .core import XMPMeta, XMPIterator
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/core.py", line 50, in <module>
    from . import exempi as _cexempi
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/exempi.py", line 69, in <module>
    EXEMPI = _load_exempi()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/exempi.py", line 60, in _load_exempi
    raise ExempiLoadError('Exempi library not found.')
libxmp.ExempiLoadError: Exempi library not found.

WARNING: autodoc: failed to import module 'urls' from module 'catalog'; the following exception was raised:
Traceback (most recent call last):
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/sphinx/ext/autodoc/importer.py", line 70, in import_module
    return importlib.import_module(modname)
  File "/home/aether/programs/anaconda3/envs/python3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/urls/__init__.py", line 20, in <module>
    from catalog.api.views.image_views import ImageViewSet
  File "/home/aether/git-a8c/openverse-api/openverse_api/catalog/api/views/image_views.py", line 4, in <module>
    import libxmp
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/__init__.py", line 50, in <module>
    from .core import XMPMeta, XMPIterator
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/core.py", line 50, in <module>
    from . import exempi as _cexempi
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/exempi.py", line 69, in <module>
    EXEMPI = _load_exempi()
  File "/home/aether/.local/share/virtualenvs/openverse_api-FJiT0ZIs/lib/python3.9/site-packages/libxmp/exempi.py", line 60, in _load_exempi
    raise ExempiLoadError('Exempi library not found.')
libxmp.ExempiLoadError: Exempi library not found.

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index                                                                                                                                                                                                                                                             
generating indices... genindex py-modindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 17 warnings.

The HTML pages are in build/html.

AetherUnbound · 2021-12-23T23:30:34Z

justfile

+sphinx-make out="html":
+    cd openverse_api && pipenv run sphinx-build -M {{ out }} docs/ build/
+
+# Serve the Sphinx documentation from the HTML output directory
+sphinx-serve port="3000":
+    cd openverse_api/build/html && pipenv run python -m http.server {{ port }}
+
+# Serve the Sphinx documentation via a live-reload server
+sphinx-live port="3000":
+    cd openverse_api && pipenv run sphinx-autobuild --port {{ port }} docs/ build/html/


I feel like we're likely to add documentation to the other projects as well, do you think the directory could be an argument here that gets defaulted to openverse_api?

openverse_api/catalog/api/serializers/audio_serializers.py

AetherUnbound · 2021-12-23T23:41:36Z

openverse_api/docs/catalog.urls.rst

@@ -0,0 +1,29 @@
+catalog.urls package


How much of this do we have to explicitly define? Is this something generated by sphinx that we just need to check in? Do you mind also adding some internal documentation that describes how to add a new module and what needs to be changed in the docs/ folder when changes are made (if anything)?

dhruvkb · 2021-12-25T10:48:37Z

The error for missing secret key occurs most likely because of a missing .env file in the Pipenv root. Adding it with the DJANGO_SECRET_KEY and DJANGO_SETTINGS_MODULE populated should fix it as Pipenv reads the file for all pipenv run ... calls. I added that to the "Testing Instructions" with an *.

The error for missing Exempi is to some extent warranted. It's a dependency of the API. My system had exempi installed (using Homebrew) but the right way to make it system-independent would be to spin up a Docker container with the API image to run Sphinx from.

The Django code tries to connect to ES on initialisation. This is not good and must be changed. But in the meantime, even importing from the Django codebase (which Sphinx does to generate the docs) leads to the ES connection which fails. To work around that, just up is recommended.

…o sphinx

obulat · 2021-12-26T04:29:57Z

I was trying to test this, but I don't know how:
I usually run the API in docker, by running just up in the api directory. How do I install the dependencies using pipenv in the Docker container after running just up?
I tried simply running the command from Madison's review, but it fails, as is probably expected since I don't have the dependencies installed:

DJANGO_SECRET_KEY=foobar just sphinx-make
docker-compose -f docker-compose.yml up -d
[+] Running 9/9
 ⠿ Container openverse-api_thumbs_1            Running                       0.0s
 ⠿ Container openverse-api_es_1                Running                       0.0s
 ⠿ Container openverse-api_upstream_db_1       Running                       0.0s
 ⠿ Container openverse-api_analytics_1         Running                       0.0s
 ⠿ Container openverse-api_cache_1             Running                       0.0s
 ⠿ Container openverse-api_db_1                Running                       0.0s
 ⠿ Container openverse-api_web_1               Running                       0.0s
 ⠿ Container openverse-api_indexer_worker_1    Running                       0.0s
 ⠿ Container openverse-api_ingestion_server_1  Running                       0.0s
docker-compose exec web sphinx-build -M html docs/ build/
OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "sphinx-build": executable file not found in $PATH: unknown

dhruvkb · 2021-12-26T07:01:51Z

@obulate you will need to rebuild the web image:

$ docker-compose build web

After this step, things will work normally. You won't need to pass the secret key in the command because Docker will set the environment vars.

obulat

Thank you for your suggestion, @dhruvkb , I managed to run the PR and see the docs! It looks 👌.
The sidebar on desktop could look better if it was full width, left-aligned or had a bigger right padding.
What is the reason for changing the API port?

dhruvkb · 2021-12-27T18:41:44Z

I had to add the 3000:3000 port to Docker Compose because the Sphinx server was running from inside the container.

AetherUnbound

This is seriously so great - I love it! Thanks for moving the other markdown files into these docs as well 🚀

I tested this locally with the .env files made and it all work 💯

openverse_api/catalog/api/docs/README.md

…o sphinx

Add the missing 'child' kwarg as it is not a regular arg

0e5271e

dhruvkb requested a review from a team as a code owner December 23, 2021 07:26

dhruvkb requested review from rbadillap and AetherUnbound December 23, 2021 07:26

dhruvkb added 🌟 goal: addition Addition of new feature 📄 aspect: text Concerns the textual material in the repository 🟩 priority: low Low priority and doesn't need to be rushed labels Dec 23, 2021

dhruvkb added 3 commits December 23, 2021 11:28

Install and configure Sphinx

d75952c

Customise template for beauty

715ef76

Add documentation pages

59e6ef7

dhruvkb force-pushed the sphinx branch from 519e3d7 to 59e6ef7 Compare December 23, 2021 07:28

AetherUnbound suggested changes Dec 23, 2021

View reviewed changes

Read DJANGO_SETTINGS_MODULE and DJANGO_SECRET_KEY from .env

065415e

dhruvkb added 4 commits December 25, 2021 23:39

Use the Docker Compose setup to compile and serve Sphinx

95d0876

Move Sphinx recipes into their own section

c9ab14b

Merge branch 'main' of https://github.com/WordPress/openverse-api int…

4f9fceb

…o sphinx

Organise docs sources into a hierarchy

3060a22

dhruvkb added 5 commits December 26, 2021 12:55

State the nature of consumer documentation

aa9c057

Update the title and author of the developer docs

451005d

Remove development instructions from the README.md file

fa95915

Enable support for Markdown with Sphinx

bdec0da

Add the development instructions

c34b2c9

dhruvkb mentioned this pull request Dec 27, 2021

Add kwarg 'child' required by seriaizers.ListField #434

Merged

7 tasks

dhruvkb added 3 commits December 27, 2021 15:07

Remove documentation leaving only links

8591d76

Rename 'Development' to 'Quickstart'

6f8f90f

Remove unused prerequisite

dcbd9cb

dhruvkb added 2 commits December 27, 2021 15:37

Add documentation for running and testing the API

53a5bbf

Create stencils for more documentation

66b8b08

obulat approved these changes Dec 27, 2021

View reviewed changes

AetherUnbound approved these changes Jan 3, 2022

View reviewed changes

openverse_api/catalog/api/docs/README.md Outdated Show resolved Hide resolved

dhruvkb mentioned this pull request Jan 4, 2022

Rename the API directory to api/ #438

Merged

7 tasks

Merge branch 'main' of https://github.com/WordPress/openverse-api int…

e411c62

…o sphinx

dhruvkb merged commit e3d9427 into main Jan 6, 2022

dhruvkb deleted the sphinx branch January 6, 2022 06:30

AetherUnbound mentioned this pull request Jan 27, 2022

Reference quickstart guide in top level README.md #488

Closed

1 task

AetherUnbound mentioned this pull request Feb 18, 2022

Let's talk documentation WordPress/openverse#162

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup Sphinx for developer-docs #427

Setup Sphinx for developer-docs #427

dhruvkb commented Dec 23, 2021 •

edited

Loading

AetherUnbound left a comment

AetherUnbound Dec 23, 2021

AetherUnbound Dec 23, 2021

dhruvkb commented Dec 25, 2021 •

edited

Loading

obulat commented Dec 26, 2021 •

edited

Loading

dhruvkb commented Dec 26, 2021 •

edited

Loading

obulat left a comment

dhruvkb commented Dec 27, 2021

AetherUnbound left a comment •

edited

Loading

Setup Sphinx for developer-docs #427

Setup Sphinx for developer-docs #427

Conversation

dhruvkb commented Dec 23, 2021 • edited Loading

Description

Testing Instructions

Checklist

Developer Certificate of Origin

AetherUnbound left a comment

Choose a reason for hiding this comment

AetherUnbound Dec 23, 2021

Choose a reason for hiding this comment

AetherUnbound Dec 23, 2021

Choose a reason for hiding this comment

dhruvkb commented Dec 25, 2021 • edited Loading

obulat commented Dec 26, 2021 • edited Loading

dhruvkb commented Dec 26, 2021 • edited Loading

obulat left a comment

Choose a reason for hiding this comment

dhruvkb commented Dec 27, 2021

AetherUnbound left a comment • edited Loading

Choose a reason for hiding this comment

dhruvkb commented Dec 23, 2021 •

edited

Loading

dhruvkb commented Dec 25, 2021 •

edited

Loading

obulat commented Dec 26, 2021 •

edited

Loading

dhruvkb commented Dec 26, 2021 •

edited

Loading

AetherUnbound left a comment •

edited

Loading