Empower Developers to Build Rich, Automated Tutorials

[jmchilton]

Empower Developers to Build Rich, Automated Tutorials

This PR introduces a "Stories" feature that automatically generates visual documentation from Selenium & Playwright tests and standalone scripts built on galaxy-selenium. Stories interleave screenshots with markdown narrative to create tutorial-quality documentation in markdown, HTML, PDF, and zip formats. The same automation code now serves dual purpose: validating Galaxy functionality through tests while generating tutorials. This approach maintains a single source of truth for both testing and documentation.

Key Features

Dual-Purpose Automation

• Test documentation: Generate visual test reports showing what tests verify
• User tutorials: Create user-facing guides from the same automation code
• Developer docs: Document workflows and features with actual screenshots

Story API

Three new methods added to browser context:

• screenshot(label, caption=None) - Take screenshot with optional caption for story
• document(markdown_content) - Add markdown narrative between actions
• document_file(file_path, caption=None) - Include file contents in story with syntax highlighting

Output Formats

Stories automatically generate four artifact types:

• story.md - Markdown source with embedded screenshots
• story.html - Self-contained HTML with styling
• story.pdf - Publication-ready PDF (requires weasyprint)
• {story_name}.zip - Complete archive of all artifacts

Clean Architecture

Split across two packages for maximum reusability:

• packages/selenium: General-purpose story infrastructure (no test dependencies)
  • Reusable in tests, Jupyter notebooks, standalone tutorial scripts
  • Story class, NoopStory pattern, StoryProtocol typing
• packages/test_selenium: Test framework integration
  • Wires stories into test lifecycle via selenium_test decorator
  • Enabled via GALAXY_TEST_STORIES_DIRECTORY environment variable

Examples Included

Test Suite: Workbook Import Stories

Added test_workbook_import.py with four test cases demonstrating the new workbook functionality from #20288:

1. test_dataset_import_from_workbook - Import datasets with automatic column mapping
2. test_simple_list_collection_import - Create list collections from workbooks
3. test_list_of_pairs_collection_import - Import paired data with automatic pairing
4. test_nested_list_of_pairs_import - Build complex nested collection structures

These tests serve dual purpose: validating the workbook import feature while generating tutorial documentation.

Run tests with stories:

export GALAXY_TEST_STORIES_DIRECTORY=/tmp/test_stories
pytest lib/galaxy_test/selenium/test_workbook_import.py -v
# Check /tmp/test_stories for generated markdown, HTML, PDF, zip

Tutorial Generator: Rule Builder Guide

Added generate_rule_builder_tutorial.py demonstrating standalone tutorial generation at scale. Generates a story with 10 different examples included and documented (the 4 new workbook examples + 6 older rules examples that power training material tutorials) by reusing test helper methods with added narrative context.

Generate tutorials:

cd packages/selenium
# Outline mode (fast, displays rule structure and builder but doesn't wait for uploads)
uv run python galaxy/selenium/scripts/generate_rule_builder_tutorial.py --galaxy_url http://localhost:8081/ --story-output ./rules_tutorial

# Full mode (complete with uploads and screenshots)
uv run python galaxy/selenium/scripts/generate_rule_builder_tutorial.py --galaxy_url http://localhost:8081/ --story-output ./rules_tutorial --perform-uploads  --timeout-multiplier 6

These vs. Training Material

Despite Claude's flowery language and excitement - I think the Training Material remains the repository for much richer, more usable, better supported tutorials and this work will never supplant that - but it does make it really easy to build tutorial outlines that can be readily translated into training materials (a task that can be easily delegated to an agent once the Markdown and screenshots are in place) and it makes it really easy to regenerate all the screenshots for older tutorials generated from these stubs (again a task that could be easily delegated to an agent).

I have a plan to create tutorials for these test cases - this was the compare and contrast of the differences between what I have done here and what needs to be done for the training materials:

Selenium-generated documentation:

• Purpose: Test validation + reference documentation
• Audience: Developers and power users
• Content: Technical explanations of what Galaxy detects and how
• Screenshots: Focus on rule builder interface states

Final GTN tutorial (to be written):

• Purpose: Teaching beginners how to use auto-detection
• Audience: Scientists and researchers new to Galaxy
• Content: Step-by-step instructions with biological context
• Additional elements needed:
  • Clearer learning objectives and prerequisites
  • More biological context for the examples
  • Troubleshooting tips
  • Export template workbook demonstration
  • Links to intermediate/advanced tutorials
  • Exercises or challenges for practice

The selenium docs provide the foundation and validation, but the GTN tutorial needs additional pedagogy and narrative.

Implementation Highlights

NoopStory Pattern

Uses null object pattern instead of None checks for cleaner code and type safety:

if GALAXY_TEST_STORIES_DIRECTORY:
    self.story = Story(title, description, output_dir)
else:
    self.story = NoopStory()  # All methods are no-ops

Test Retry Support

Stories reset on test retry to avoid accumulating content from failed attempts:

def reset_driver_and_session(self):
    self.story.reset()  # Clear elements and counters
    # ... reset driver

Dual-Save Screenshots

When both GALAXY_TEST_STORIES_DIRECTORY and GALAXY_TEST_SCREENSHOTS_DIRECTORY are set, screenshots save to both locations for backward compatibility.

Data Helpers

Centralized example data access via galaxy.selenium.stories.data:

from galaxy.selenium.stories.data import WORKBOOK_EXAMPLE_1
self.workbook_upload(WORKBOOK_EXAMPLE_1)  # No fragile relative paths

Dependencies

Required (already in Galaxy)

• markdown - Already used for markdown to HTML
• zipfile - Python stdlib

Optional (graceful degradation)

• weasyprint - PDF generation (warns if unavailable)

Refactored for Cleaner Dependencies

Extracted markdown/PDF utilities from galaxy.managers.markdown_util to galaxy.util.markdown so selenium package doesn't depend on managers layer.

Design Rationale

Why Not Existing Test Reporting Solutions?

We evaluated several existing Python testing libraries before implementing a custom solution:

pytest-html (already in Galaxy)

• Designed for test pass/fail reports, not narrative documentation
• No built-in story/step structure for sequential narratives
• Limited formatting options for tutorial-style content
• Cannot generate markdown or PDF outputs
• Inherently test-focused - cannot be reused outside test context for user documentation

Allure Framework

• Industry-standard with rich interactive UI and screenshot support
• Requires Java-based report generator (additional infrastructure)
• Focused on test analytics and reports rather than narrative documentation
• Less control over output format
• Not suitable for generating user manuals or tutorials

ReportPortal

• Enterprise test management platform with ML-powered analytics
• Requires separate server infrastructure
• Aimed at test analytics dashboards, not documentation generation
• Excessive complexity for generating standalone tutorial content

pytest-bdd / Robot Framework

• Behavior-driven development frameworks with built-in reporting
• Would require rewriting all existing tests in different format/syntax
• Framework migration too invasive for this feature
• Focused on specifications rather than visual documentation

Seleniumbase

• Complete Selenium framework with built-in reporting and dashboards
• Would conflict with Galaxy's existing test abstractions
• Requires significant refactoring of test infrastructure
• Too opinionated about test structure

Why Not Extend pytest-html?

Galaxy's test architecture uses unittest.TestCase subclasses with pytest as a test runner, not pure pytest tests. Test lifecycle is already managed by the selenium_test decorator. A pytest plugin approach would:

• Create redundant lifecycle management (pytest hooks + existing decorator)
• Add coupling to pytest internals when Galaxy uses unittest
• Be permanently coupled to test framework, preventing reuse for user documentation
• Introduce debugging complexity with two layers managing the same lifecycle
• Still require custom code for markdown/PDF generation and narrative structure

What pytest-html provides: Test pass/fail reports with screenshot attachments
What we need: Narrative documentation with sequential screenshots, markdown/HTML/PDF output, story structure, and reusability for user tutorials and manuals

Why Custom Implementation?

Reusability Beyond Testing
The Story class in packages/selenium has zero test framework dependencies, enabling:

• User manual generation from the same automation code
• Interactive tutorial creation in Jupyter notebooks
• Standalone documentation scripts
• Developer workflow documentation with actual screenshots

Example: Standalone Tutorial Generation

See generate_rule_builder_tutorial.py as an example. This is clearly cleaner without the Pytest pieces involved and/or extra test-focused infrastructure like Allure.

Clean Architecture

• packages/selenium: General-purpose story infrastructure (no test dependencies)
• packages/test_selenium: Test framework integration via selenium_test decorator
• Clear separation allows independent testing and evolution of Story class

Fits Galaxy's Existing Architecture

• Works seamlessly with Galaxy's unittest-based tests
• Simple lifecycle management in one place (decorator)
• No conflicts with existing test infrastructure

Leverages Existing Galaxy Utilities

• Markdown to HTML: galaxy.util.markdown.to_html()
• HTML to PDF: galaxy.util.markdown.to_pdf_raw()
• weasyprint: Already in Galaxy dependencies
• Screenshot infrastructure: Existing methods and paths

Full Control Over Output

• Generate exactly the formats needed (markdown, HTML, PDF, zip)
• Narrative focus: Optimized for sequential documentation, not pass/fail reports
• Interleaved documentation and screenshots for tutorial quality

Testing

All functionality manually verified:

• ✅ Story generation in test context
• ✅ Standalone tutorial script execution
• ✅ Markdown, HTML, PDF, zip artifact generation
• ✅ Screenshot embedding and sequential numbering
• ✅ Test retry handling (story.reset())
• ✅ Type safety with StoryProtocol
• ✅ NoopStory pattern (zero impact when disabled)

Future Enhancements

Potential additions (deferred for future PRs):

• Video recording of test execution
• Gallery view for browsing test stories
• CI integration to publish stories as artifacts
• Visual diff highlighting between test runs

Related Work

• Builds on workbook import feature from Empower Users to More Pragmatically Import Datasets & Collections From Tables #20288
• Four test cases validate and document that feature
• Tutorial generator shows same code serving tests and documentation

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.