feat: support recursive DAG discovery

[yohamta0]

Summary

• add opt-in recursive DAG discovery under the configured DAG directory
• keep nested DAG IDs stem-based while excluding file-stem and effective-name conflicts
• support nested CRUD, search, indexing, and scheduler updates as directories change
• skip workspace config directories, dot-directories, and symlink entries

Configuration

dag_discovery:
  recursive: true

The same setting is available through DAGU_DAG_DISCOVERY_RECURSIVE.

Testing

• Go unit tests for discovery, configuration, persistence, scheduler, command wiring, and API behavior
• race tests for discovery, persistence, and recursive scheduler refreshes
• golangci-lint on native and Windows targets

Closes #2400

---

Summary by cubic

Adds opt-in recursive DAG discovery under paths.dags_dir, with the scheduler watching nested directories and emitting add/update/delete events. Duplicate file stems or DAG names are excluded and surfaced in API/list/search results. Closes #2400.

• New Features

  • Opt-in via dag_discovery.recursive: true or DAGU_DAG_DISCOVERY_RECURSIVE=true.
  • Scans YAML in subfolders; skips workspaces/, dot-directories, and symlinked entries (supports a symlinked root).
  • Uses slash-normalized relative paths; stem-based DAG IDs; excludes duplicates by stem or effective name and reports issues in errors and scheduler logs.
  • Full support across CRUD, list, search, grep, and labels; explicit path lookups (e.g. team/service/dag) are supported; nested paths resolve transparently.
  • Scheduler watches all discovered dirs, debounces refreshes, and emits add/update/delete on changes.
  • NewEntryReader now takes a recursive flag; index entries and loading use normalized relative paths; paths.alt_dags_dir remains lookup-only.
  • Updated config schema to include dag_discovery.recursive.

• Migration

  • Default is off; enable via config or env var.
  • Resolve any duplicate stems or DAG names across the tree; conflicting files are ignored until fixed.
  • In recursive mode, creates must not conflict with any existing stem; renames keep the file in its current directory.

^{Written for commit c74544e. Summary will update on new commits.}

Summary by CodeRabbit

• New Features

  • Added optional recursive discovery of DAG files in nested directories.
  • Added configuration through dag_discovery.recursive or DAGU_DAG_DISCOVERY_RECURSIVE.
  • Extended DAG listing, lookup, execution, search, metadata, and renaming for nested files.
  • Added conflict detection for duplicate DAG names and file stems.
  • Scheduler monitoring now detects nested DAG additions, updates, and removals.

• Bug Fixes

  • Improved cross-platform path handling and safer relative-path renaming.

• Documentation

  • Documented discovery exclusions, conflict handling, and alternate DAG directory lookup behavior.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[coderabbitai]

📝 Walkthrough

Walkthrough

The change adds optional recursive DAG discovery under dagsDir. It introduces filesystem scanning and catalog-based storage, updates scheduler watching and event handling, wires configuration through runtime setup, and adds storage, scheduler, configuration, and API tests.

Changes

Recursive DAG discovery

Layer / File(s)	Summary
Discovery configuration and filesystem scanning internal/cmn/config/*, internal/cmn/schema/config.schema.json, internal/dagdiscovery/*, README.md	Adds dag_discovery.recursive and its environment binding. The scanner supports recursive traversal, exclusions, symlink handling, YAML filtering, metadata, errors, and deterministic results.
Catalog-backed recursive DAG storage internal/persis/file/dag/*, internal/persis/file/dag_store.go	Builds a normalized catalog from discovered files. DAG listing, lookup, search, labels, creation, rename, deletion, and updates support nested paths and duplicate detection.
Recursive scheduler watching and event reconciliation internal/service/scheduler/entryreader.go, internal/service/scheduler/entryreader_internal_test.go	Adds recursive directory watchers, debounced refreshes, registry reloads, watcher synchronization, and deterministic add, update, rename, and delete events.
Configuration wiring and end-to-end validation internal/cmd/process/scheduler.go, internal/intg/distr/fixtures_test.go, internal/test/*, internal/service/frontend/api/v1/dags_test.go, internal/persis/file/dag/store_test.go	Passes recursive discovery into stores and schedulers. Tests cover nested DAG operations, conflicts, recovery, API lookup, and successful execution by filename.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ConfigLoader
  participant DAGStore
  participant EntryReader
  participant FileWatcher
  participant API
  ConfigLoader->>DAGStore: configure recursive discovery
  ConfigLoader->>EntryReader: configure recursive scheduling
  DAGStore->>DAGStore: scan and catalog nested DAG files
  FileWatcher->>EntryReader: report file or directory changes
  EntryReader->>DAGStore: reload discovered DAG metadata
  EntryReader-->>EntryReader: emit deterministic schedule events
  API->>DAGStore: resolve DAG by filename
  DAGStore-->>API: return nested DAG details and path

Loading

Possibly related PRs

• dagucloud/dagu#2424: Related DAG lookup and path containment changes.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 30.95% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes satisfy issue #2400 by adding opt-in recursive discovery, conflict handling, and nested DAG support across listings and UI APIs.
Out of Scope Changes check	✅ Passed	The changes remain within the recursive DAG discovery objectives and include related configuration, persistence, scheduler, API, documentation, and tests.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding support for recursive DAG discovery.
Description check	✅ Passed	The description explains the changes, configuration, testing, and related issue; the missing checklist is non-critical.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Fix failing CI checks

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/recursive-dag-discovery

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (1)

internal/persis/file/dag/store_test.go (1)

153-161: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add a nested cursor-pagination case to the recursive search test.

This test calls SearchCursor once with Limit: 10, so it never exercises a returned cursor. The cursor skip logic in Storage.SearchCursor compares file stems, while recursive discovery orders entries by relative path. A test that sets Limit: 1 over nested DAGs whose stem order differs from their path order would cover that interaction.

Example: create a/zebra.yaml and b/apple.yaml, both matching the query, then page with Limit: 1 and assert that both DAGs are returned across the two pages.

This relates to the pagination concern raised in internal/persis/file/dag/store.go.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/persis/file/dag/store_test.go` around lines 153 - 161, Extend the
recursive search test around SearchCursor to create nested matching DAGs whose
filename stem order differs from relative-path order, such as a/zebra.yaml and
b/apple.yaml. Use Limit: 1, assert the first page returns one item and a cursor,
then request the next page with that cursor and assert the remaining DAG is
returned, covering Storage.SearchCursor pagination across nested paths.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/persis/file/dag/catalog.go`:
- Around line 23-31: The loadCatalog flow should cache the constructed catalog
keyed by the discovery inputs baseDir, searchPaths, and recursive, reusing it
across recursive lookup paths such as locateDAG, List, SearchCursor, LabelList,
and metadata/spec operations. Add cache reuse and ensure invalidation occurs
only after writes or base-config state changes, while preserving fresh scan
error handling when the cache is invalid or absent.

In `@internal/persis/file/dag/store.go`:
- Around line 752-762: Align SearchCursor pagination with the catalog’s
iteration order: update the filtering in the search loop around entryStem and
cursor.FileName to compare the same FilePath-based key used to sort
catalog.entries, or change catalog.newCatalog ordering to stem order
consistently. Ensure subsequent pages cannot skip entries when directory paths
and stems sort differently.

In `@internal/service/scheduler/entryreader_internal_test.go`:
- Around line 301-307: Update the os.WriteFile call creating the test DAG at
firstPath to use 0600 permissions instead of 0644, matching the repository’s
secure test-file permission convention.

In `@internal/service/scheduler/entryreader.go`:
- Around line 412-433: Update syncWatches to compute watcher additions and
removals without mutating watchedDirs, then acquire er.lock while applying all
watchedDirs changes; preserve existing watcher error handling and removal
behavior. Also guard the watchedDirs write in initRecursive with er.lock for
consistent synchronization with readers. Run the Go test suite with gotestsum
and race detection.

---

Nitpick comments:
In `@internal/persis/file/dag/store_test.go`:
- Around line 153-161: Extend the recursive search test around SearchCursor to
create nested matching DAGs whose filename stem order differs from relative-path
order, such as a/zebra.yaml and b/apple.yaml. Use Limit: 1, assert the first
page returns one item and a cursor, then request the next page with that cursor
and assert the remaining DAG is returned, covering Storage.SearchCursor
pagination across nested paths.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 45efacc9-c967-4039-9935-9f1d962c62da

📥 Commits

Reviewing files that changed from the base of the PR and between 01c2336 and dbc1b39.

📒 Files selected for processing (20)

• README.md
• internal/cmd/process/scheduler.go
• internal/cmn/config/config.go
• internal/cmn/config/definition.go
• internal/cmn/config/loader.go
• internal/cmn/config/loader_test.go
• internal/dagdiscovery/scan.go
• internal/dagdiscovery/scan_test.go
• internal/intg/distr/fixtures_test.go
• internal/persis/file/dag/catalog.go
• internal/persis/file/dag/dagindex/dagindex.go
• internal/persis/file/dag/store.go
• internal/persis/file/dag/store_test.go
• internal/persis/file/dag_store.go
• internal/service/frontend/api/v1/dags_test.go
• internal/service/scheduler/entryreader.go
• internal/service/scheduler/entryreader_internal_test.go
• internal/service/scheduler/manger_test.go
• internal/test/helper.go
• internal/test/scheduler.go

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[yohamta0]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[yohamta0]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.