feat: api plugin system & offline versioned dependency vendoring

[pujitm]

• New Features

  • Created a dynamic plugin system for the API to enable community augmentation of GraphQL, CLI, and Cron functionalities capabilities.
  • Included an example plugin under packages/unraid-api-plugin-health that adds a new graphql query for API health checks.
  • Added rc.unraid-api commands for backing up, restoring, and installing production dependencies, streamlining maintenance and deployment.
  • Improved dependency vendoring by bundling a versioned pnpm store (instead of node_modules). Versioning will allow users to add plugins to a specific api release without requiring an internet connection on subsequent reboots.

• Chores

  • Upgraded build workflows and versioning processes to ensure more reliable artifact handling and production packaging.

Summary by CodeRabbit

• New Features

  • Enhanced automated workflows streamline versioning, artifact management, and dependency packaging.
  • Dynamic CLI and plugin integration now enable seamless extension of functionality.
  • A new health check plugin provides a simple GraphQL endpoint to verify system responsiveness.
  • Improved asset management delivers a more reliable deployment experience.
  • New functions for managing the PNPM store, including backup and restore capabilities.
  • New entries for Node.js and PNPM in the plugin configuration improve installation processes.

• Chores

  • Refined installation, backup, and restore processes enhance overall dependency management.

[coderabbitai]

Walkthrough

This pull request implements enhancements across CI/CD, API, CLI, and plugin ecosystems. The GitHub Actions workflow now includes steps for managing the PNPM store and generating API versions. The API is augmented with new functions for reading package metadata and configuring GraphQL with plugins, while the CLI now supports dynamic module registration with plugin integrations. Additionally, the plugin builder, symlink management, and workspace configurations have been updated to streamline asset handling and dependency management.

Changes

File(s)	Change Summary
.github/workflows/main.yml	Added new workflow steps: "Upload PNPM Store to Github artifacts", "Get API Version", "Download PNPM Store", plus listing directory contents for enhanced debugging.
api/package.json	Added runtime dependency "change-case": "^5.4.4" and dev dependency "type-fest": "^4.37.0".
api/scripts/build.ts	Enhanced build process by updating log messages, adjusting dependency installation (omitting dev dependencies), adding store directory configuration, cleaning up node_modules, and bundling & removing the PNPM store.
api/src/cli.ts	Introduced dynamic CLI module registration via a new DynamicCliModule that extends CliModule after calling CliModule.registerWithPlugins(), altering the control flow before invoking CommandFactory.run().
api/src/environment.ts	Replaced the old getPackageJsonVersion with new functions: readPackageJson, getPackageJson, and getPackageJsonDependencies to improve error handling and modularity for package.json access; updated API_VERSION accordingly.
api/src/graphql/schema/loadTypesDefs.ts	Modified loadTypeDefs to accept an additional parameter (additionalTypeDefs) to merge user-provided GraphQL type definitions.
api/src/store/index.ts	Added new type alias: export type ApiStore = typeof store;.
api/src/unraid-api/app/app.module.ts	Integrated PluginModule by importing it and invoking PluginModule.registerPlugins() within the module imports.
api/src/unraid-api/cli/cli.module.ts	Updated the module by replacing the providers array with constants and adding static methods getCommands() and registerWithPlugins() to support plugin-based command registration.
api/src/unraid-api/graph/graph.module.ts	Enhanced GraphQL configuration by injecting PluginService to retrieve dynamic plugin configurations (type definitions and resolvers).
api/src/unraid-api/plugin/plugin.interface.ts	Introduced new plugin interfaces and types, including PluginMetadata, apiPluginSchema, ApiPluginDefinition, and ConstructablePlugin.
api/src/unraid-api/plugin/plugin.module.ts	Added a new global PluginModule that includes a static registerPlugins() method returning a dynamic module with plugin providers.
api/src/unraid-api/plugin/plugin.service.ts	Created new PluginService with methods to load plugins, retrieve GraphQL/REST configurations, services, cron jobs, and dynamically register plugins.
packages/unraid-api-plugin-health/**	Introduced a new health plugin package with an index file defining its metadata, GraphQL resolvers/type definitions, lifecycle hooks, and accompanied by a new package.json.
plugin/**	- .gitignore: Added ignore rules for packed-pnpm-store.txz and source/.../install/doinst.sh. - builder/build-plugin.ts: Enhanced asset management with getAssetUrl, conditionally skipped Git check, and integrated bundlePnpmStore() into the main flow. - builder/build-pnpm-store.ts: Introduced functions getVersion(), getPnpmBundleName(), and bundlePnpmStore() for PNPM store archiving. - builder/build-txz.ts: Updated to import apiDir from utils/paths. - builder/utils/bucket-urls.ts: Consolidated URL generation via getAssetUrl (with getPluginUrl retained) and refactored getMainTxzUrl. - builder/utils/changelog.ts: Modified error handling to log failures and return fallback values. - builder/utils/paths.ts: Added new constant apiDir. - docker-compose.yml: Added a new volume mapping for the PNPM store archive. - plugins/dynamix.unraid.net.plg: Enhanced PNPM binary and vendor store configuration. - source/dynamix.unraid.net/etc/rc.d/rc.unraid-api: Added new functions for backing up, restoring, and installing dependencies using pnpm. - source/dynamix.unraid.net/install/doinst.sh: Overhauled symlink creation for numerous binaries.
pnpm-workspace.yaml	Expanded workspace scope by adding the entry ./packages/* to include additional packages.

Sequence Diagram(s)

sequenceDiagram
    participant Repo as GitHub Repository
    participant Workflow as GitHub Actions
    participant BuildAPI as Build-API Job
    participant BuildPlugin as Build-Plugin Job

    Repo->>Workflow: Push/Commit triggers workflow
    Workflow->>BuildAPI: Execute Build-API job
    BuildAPI->>BuildAPI: Run "Upload PNPM Store" step and list directory
    Workflow->>BuildPlugin: Execute Build-Plugin job
    BuildPlugin->>BuildPlugin: Run "Get API Version" step
    BuildPlugin->>BuildPlugin: Run "Download PNPM Store" step

Loading

sequenceDiagram
    participant CLI as CLI Entry Point
    participant CM as CliModule
    participant CF as CommandFactory

    CLI->>CM: Call registerWithPlugins()
    CM-->>CLI: Return dynamic module configuration
    CLI->>CF: Invoke run(DynamicCliModule, config)
    CF-->>CLI: Execute command with plugin support

Loading

Possibly related PRs

• Feat/split plugin builds #1151: The changes in the main PR regarding the GitHub Actions workflow in .github/workflows/main.yml are related to the modifications in the retrieved PR, which also involves significant updates to the CI/CD workflow in the same file, including job additions and renaming. Both PRs focus on enhancing the build and deployment processes within the GitHub Actions framework.
• feat: update release-please config, allow releases from main.yml #1177: The changes in the main PR, which enhance the GitHub Actions workflow with new steps for managing the PNPM store and API versioning, are related to the modifications in the retrieved PR that also involve updates to the GitHub Actions workflow, specifically enhancing the release process. Both PRs focus on improving the functionality and control flow within the .github/workflows/main.yml file.
• feat: begin building plugin with node instead of bash #1120: The changes in the main PR, which enhance the GitHub Actions workflow with new steps for managing the PNPM store and API versioning, are related to the modifications in the retrieved PR that also involve updates to the GitHub Actions workflow, specifically in the build-plugin job where the build process has transitioned to using npm commands.

Poem

In the code, new steps align and blend,
PNPM stores and dynamic plugins on the mend.
Workflows whisper secrets in each console log,
Modules extend like branches through the fog.
A dance of logic, crisp and smart –
Change unfolds with a coder’s heart! 🚀

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro (Legacy)

📥 Commits

Reviewing files that changed from the base of the PR and between 2169aaf and d5d9d9c.

📒 Files selected for processing (1)

• plugin/plugins/dynamix.unraid.net.plg (6 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• plugin/plugins/dynamix.unraid.net.plg

⏰ Context from checks skipped due to timeout of 90000ms (4)

• GitHub Check: Test API
• GitHub Check: Build API
• GitHub Check: Build Web App
• GitHub Check: Cloudflare Pages

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai plan to trigger planning for file edits and PR creation.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[elibosley]

This looks really close, a couple more q's

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

plugin/builder/build-plugin.ts (1)

111-111: Added pnpm store bundling step

The addition of the bundlePnpmStore() call ensures that the versioned pnpm store is bundled as part of the plugin build process, which supports the offline dependency vendoring feature.

Consider adding specific error handling for this step, as failures here might require different recovery actions than other build failures.

- await bundlePnpmStore();
+ try {
+   await bundlePnpmStore();
+   console.log("Successfully bundled pnpm store");
+ } catch (error) {
+   console.error("Failed to bundle pnpm store:", error);
+   throw error;
+ }

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro (Legacy)

📥 Commits

Reviewing files that changed from the base of the PR and between 8a83082 and aec5ae1.

📒 Files selected for processing (2)

• plugin/builder/build-plugin.ts (4 hunks)
• plugin/plugins/dynamix.unraid.net.plg (6 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• plugin/plugins/dynamix.unraid.net.plg

🧰 Additional context used

🧬 Code Definitions (1)

plugin/builder/build-plugin.ts (3)

plugin/builder/utils/bucket-urls.ts (1)

• getAssetUrl (33-37)

plugin/builder/build-pnpm-store.ts (2)

• getPnpmBundleName (24-27)
• bundlePnpmStore (38-42)

plugin/builder/utils/cleanup.ts (1)

• cleanupPluginFiles (13-17)

⏰ Context from checks skipped due to timeout of 90000ms (4)

• GitHub Check: Build Web App
• GitHub Check: Build API
• GitHub Check: Test API
• GitHub Check: Cloudflare Pages

🔇 Additional comments (5)

plugin/builder/build-plugin.ts (5)

6-6: New import for asset URL handling

The addition of getAssetUrl import is appropriate for the new functionality that references external assets in the plugin.

---

15-15: New pnpm store bundling functions imported

The import of bundlePnpmStore and getPnpmBundleName correctly brings in the necessary functions for the pnpm store bundling feature.

---

64-65: New entity references for vendor store

The addition of VENDOR_STORE_URL and VENDOR_STORE_FILENAME entities enables the plugin to reference the pnpm store bundle. This is essential for the offline dependency vendoring feature mentioned in the PR objectives.

---

73-75: Improved error handling for missing entities

The enhanced error message now includes the full entities object, which will make debugging missing entity values much easier.

---

102-106: Skip git check for local plugin builds

The conditional logic to skip Git checks for local builds is a good enhancement that will improve the developer experience when working locally.

[github-actions]

This plugin has been deployed to Cloudflare R2 and is available for testing.
Download it at this URL:

https://preview.dl.unraid.net/unraid-api/tag/PR1252/dynamix.unraid.net.plg

resolved comments