Documentation infrastructure

[amitabhverma]

On the fly generation of docs (User Guide, API References, Examples, etc.) using sphinx and readthedocs integration.

A preview can be viewed here: https://waveorder.readthedocs.io/en/latest/index.html

• The html pages are rendered from github markdown (.md) files and some are included in .rst files.
• The API References (Settings and Stokes) are rendered based on docstrings.
• The Examples are pulled directly from source code files.

Further customization is possible as required. Some restructuring of the docs files within the repo (guides, examples, etc.) would benefit users.

[talonchandler]

First look comments:

• Looking good! I think the major infrastructure pieces are present here which is great.
• I would suggest changing the base branch from main to gui_update (within github is fine...no need to work through a rebase imo). As is this PR includes changes from both branchs, e.g. jobs_mgmt.py belongs to the other PR. This will help focus @srivarra's attention.
• No need for a duplicate poincare_swing.svg, correct?

[talonchandler]

Just approved and merged gui_update, so this is ready to review now.

[amitabhverma]

First look comments:

• Looking good! I think the major infrastructure pieces are present here which is great.
• I would suggest changing the base branch from main to gui_update (within github is fine...no need to work through a rebase imo). As is this PR includes changes from both branchs, e.g. jobs_mgmt.py belongs to the other PR. This will help focus @srivarra's attention.
• No need for a duplicate poincare_swing.svg, correct?

Thanks for the comments

• Changed the base branch to /gui_update - I agree, it will make the reviewing easier
• I have not moved the existing .md files and docs/images but duplicated them under the sphinx_source dir. So yes, there is some duplication of files at the moment which needs to be addressed. I assumed there might be a need to keep a structure that works on github as well.

[talonchandler]

I have not moved the existing .md files and docs/images but duplicated them under the sphinx_source dir. So yes, there is some duplication of files at the moment which needs to be addressed. I assumed there might be a need to keep a structure that works on github as well.

If they're still .md files, they're still viewable in github so I'm fine if they live in the sphinx_source directory. My main concern is that duplicated files will be difficult to maintain.

[srivarra]

This is a super great start. Just a few corner cases, and tiny things here and there!

[amitabhverma]

This is a super great start. Just a few corner cases, and tiny things here and there!

Thanks @srivarra for the detailed feedback. This will be super useful for Friday's meeting. I'm not replying to each comment (for now) since we will be discussing this but from my understanding most are related to current limitations of the renderer/formatting, hotlinking video files, etc.

One roadblock I'm facing is the .md links within a .md files do not convert to .html when the html page is being generated causing linking between files an issue unless the .md links are changed to .html. However, in that case they wouldn't work on github. I have tried looking for a solution but could not find something clean.

[talonchandler]

One roadblock I'm facing is the .md links within a .md files do not convert to .html when the html page is being generated causing linking between files an issue unless the .md links are changed to .html. However, in that case they wouldn't work on github. I have tried looking for a solution but could not find something clean.

Thanks for flagging this @amitabhverma. I think this feature is important, because as is many of the links are broken in the guides. e.g. see the links on this page:
https://waveorder.readthedocs.io/en/latest/guide.html

I've implemented a potential fix in #469.

My requests after this round of review are:

• make sure the links work in the guides in both github and the docs website (I think my PR fixes this)
• remove duplicate dependencies, ideally with a single source of truth
• remove or fix the execution times script
• were you able to successfully render a jupyter notebook-style page similar to the iohub examples like we discussed on Friday? No need to make major modifications to the notebook itself...I'm mostly looking for proof of concept.

Thank you @amitabhverma.

[amitabhverma]

• were you able to successfully render a jupyter notebook-style page similar to the iohub examples like we discussed on Friday? No need to make major modifications to the notebook itself...I'm mostly looking for proof of concept.

@talonchandler
Sorry but can you please refresh me on this. Did you mean the iohub examples (which we did confirm previously rendered correctly as expected) or you had something else in mind ?

The #469 related conversation, let's follow up there since we agree that a landing page fix is essential.

@srivarra @talonchandler
I think all the other points have now been covered but in case I missed something please do make a note. Thanks !

Update: #464 needs attention before merge.
Do you feel any of the other [DOCS] related issues are also relevant before merge ?

[talonchandler]

@talonchandler Sorry but can you please refresh me on this. Did you mean the iohub examples (which we did confirm previously rendered correctly as expected) or you had something else in mind?

My fault I made another error...I didn't see that the headlines were being rendered correctly in the .html. Great! All good here.

I just followed up on #469 with a video showing a broken link.

I agree #464 could be fixed here.

All else looks good. Thanks @amitabhverma.

[amitabhverma]

@talonchandler @srivarra

• I have restructured the docs and removed the sphinx_source directory with the intention when on github repo and navigating to docs you have the landing page show up. The same consistent behavior is on RTD. Also, sphinx can only work with docs under its directories and parent references don't work for toctree.

• The .md links are now being fixed via a post build job. The regex needed some tweaking to work with in-page target links (.md#target)

• For the video host I looked for solutions that would work for both cases (github repo & RTD) but any solution for RTD would not work on github. In the end I'm using a post build job to copy the video from github to RTD and replace the link which are currently defined below. Let me know if this solutions works. This can be automated if there is a need.
  

  waveorder/docs/scripts/fix_md_links.py

  Lines 7 to 18 in 9bc3ceb

  	# Github video link, filename on RTD, download once boolean
  	[
  	"https://github.com/user-attachments/assets/a0a8bffb-bf81-4401-9ace-3b4955436b57",
  	"waveorder-overview.mp4",
  	False,
  	],
  	[
  	"https://github.com/user-attachments/assets/4f9969e5-94ce-4e08-9f30-68314a905db6",
  	"figure-slideshow.mp4",
  	False,
  	],
  	]

https://waveorder.readthedocs.io/en/latest/index.html has the latest version for review.

[talonchandler]

This is great @amitabhverma! Thanks for all of this.

Your solution for video files is working well in my tests thanks.

Minor request: if you can turn off the ads on the docs website, please do so. If you need payment details, please let me know and we will start the process.

I'll approve and you can feel free to merge.

[amitabhverma]

This is great @amitabhverma! Thanks for all of this.

Your solution for video files is working well in my tests thanks.

Minor request: if you can turn off the ads on the docs website, please do so. If you need payment details, please let me know and we will start the process.

I'll approve and you can feel free to merge.

Thanks @talonchandler I'll request @srivarra for a re-review just to make sure he is up to speed on the recent changes.

Regarding turning off the ads on RTD, it would be best if this is done at @mattersoflight level. I believe as Gold member it should then apply to all the repos/projects under /mehta-lab. Src: https://app.readthedocs.org/accounts/gold/

[srivarra]

I think this looks great!