Make fast html #2013
Make fast html #2013
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2013 +/- ##
===========================================
+ Coverage 36.79% 92.47% +55.67%
===========================================
Files 68 68
Lines 9373 9379 +6
===========================================
+ Hits 3449 8673 +5224
+ Misses 5924 706 -5218 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
|
||
| # Import normalize_axis_index - handle compatibility | ||
| try: | ||
| from pytensor.npy_2_compat import normalize_axis_index | ||
| except ImportError: | ||
| # PyTensor 2.35+ no longer exports this, use numpy directly | ||
| try: | ||
| from numpy._core.numeric import normalize_axis_index | ||
| except (ImportError, AttributeError): | ||
| # Older numpy versions | ||
| from numpy.core.numeric import normalize_axis_index |
There was a problem hiding this comment.
This was "fixed" in #2015 We want to keep up to datw with PyMC latest versions, where they dropped NumPy < 2 support, see #2010 (comment)
|
Thanks! I think you can rebase and we can merge once the CI checks are green! |
Implements a fast build mode for the documentation to speed up development. This mode skips notebook execution and heavy API generation, significantly reducing build time. Adds Makefile targets for fast builds with different configurations (skipping notebooks, API generation, or both). Also adds a .gitignore rule to exclude the jupyter cache directory.
Handles compatibility issues with PyTensor's `normalize_axis_index` function. PyTensor 2.35+ no longer exports `normalize_axis_index`, so the code now attempts to import it from `numpy._core.numeric` or `numpy.core.numeric` if the initial import fails.
lfiaschi
force-pushed
the
make-fasthtml
branch
from
bcace83 to
6806c4e
Compare
|
pre-commit.ci autofix |
|
Related to #650 |
|
Hey @williambdean what is the status on this PR? seems like some checks are not working but I am unsure what this pre-commit.ci does. |
|
The |
| nb_execution_mode = "auto" | ||
| nb_execution_excludepatterns = ["*.ipynb"] | ||
| # Use cache mode for faster subsequent builds (only re-executes modified notebooks) | ||
| # Use "off" in fast build mode to skip all notebook execution | ||
| if SKIP_NOTEBOOKS: | ||
| nb_execution_mode = "off" | ||
| else: | ||
| nb_execution_mode = "cache" # Changed from "auto" for better performance | ||
|
|
||
| nb_execution_cache_path = ".jupyter_cache" # Persistent cache directory | ||
| nb_execution_timeout = 600 # 10 minutes per notebook | ||
| nb_execution_allow_errors = False | ||
| nb_execution_raise_on_error = True | ||
| nb_execution_excludepatterns = [ | ||
| # Heavy notebooks that take too long - execute manually when needed | ||
| "notebooks/mmm/mmm_case_study.ipynb", | ||
| "notebooks/mmm/mmm_multidimensional_example.ipynb", | ||
| "notebooks/mmm/mmm_tvp_example.ipynb", | ||
| "notebooks/mmm/mmm_time_varying_media_example.ipynb", | ||
| "notebooks/clv/dev/*.ipynb", # Development notebooks | ||
| ] |
There was a problem hiding this comment.
TL; DR. Before this PR, ipynb notebooks were never executed because we had this combination of settings:
nb_execution_mode = "auto"
nb_execution_excludepatterns = ["*.ipynb"]The reason including the notebooks as part of the build slows it down isn't so much execution as it is processing and rendering them. There are a lot of notebooks which are usually long and have a lot of code and images, so processing them and converting to html is already somewhat expensive (but on a whole other scale than executing them).
Full context:
Looking at the blame of these two lines, I see I added that 3 years ago when setting up the docs and it hasn't been updated since. I probably copied it from ArviZ because it is the pattern we use there, but without a comment it isn't very clear what is happening. The sphinx extension we use to process and render notebooks supports multiple formats for notebooks, not only .ipynb. This pattern will ensure non ipynb notebooks are executed and therefore have outputs included in the rendered docs while at the same time skipping execution of ipynb notebooks because they already have the outputs stored in them.
We then use .ipynb format for the more expensive notebooks and .md format for lighter notebooks so we don't need to run them manually and they are always kept in sync. For example this page comes from one of these lighter notebooks in markdown format.
After the change, only a handful of the notebooks are excluded, with the rest being executed fully. This makes much more expensive the first time we build the docs. After that first time the notebook outputs are cached so it will be faster, but I don't think we want executing notebooks to be a part of doc building at all.
If we do want to have notebook execution be a step in building the html docs, we should make sure this doesn't happen on readthedocs. On readthedocs we build on their servers and unless we subscribed to one of their paid plans, builds have a time limit limited resources. MCMC sampling is beyond the compute power of our readthedocs plan.
Note relevant to the discussion. There is already a GitHub Action that runs all the notebooks with mock sampling to check for potential errors, so there is already infrastructure in place to ensure notebooks always run.
Available Commands Now
Fast build
make fasthtml # ~30-60 seconds
Alternative builds
make fasthtml-nb # Skip notebooks only (~5-10 min)
make fasthtml-api # Skip API only (~15-25 min)
Full build
make html # Complete build with cache
Cleanup
make cleandocs # Clean everything
make cleancache # Clean cache only