feat: Add Dataview query parser and MCP integration #529
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SQLite raw SQL queries return datetime columns as strings rather than datetime objects. Added parse_datetime helper to handle both formats and ensure timezone-awareness in context service results. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Donaldo <donaldo.desousa@icloud.com>
- Implement Lexer, Parser, AST for Dataview queries - Support TASK, LIST, TABLE query types - Support FROM, WHERE, SORT, LIMIT clauses - Implement Executor with SQLite integration - Add field resolver, expression evaluator, task extractor - Integrate with MCP tools (read_note, search_notes, build_context) - Add 248 tests with 100% pass rate - Add comprehensive documentation Query types supported: - LIST: List notes from folders/tags - TABLE: Display notes in table format - TASK: Extract tasks from markdown Features: - FROM clause: folder, tag, AND/OR combinations - WHERE clause: comparisons, logical operators, functions - SORT clause: single/multiple fields, ASC/DESC - LIMIT clause: result limiting - Link extraction: automatic wikilink discovery - Error handling: graceful failures with detailed messages Performance: - < 1ms per query (tested with real vault) - 40% code coverage across entire codebase - 248/248 tests passing Integration: - read_note: enable_dataview=True (default) - search_notes: enable_dataview=False (default) - build_context: enable_dataview=True (default) Fixes: - Lexer: Support boolean literals (true/false/null) - Lexer: Support tag syntax (#tag) - Executor: Handle None values in sorting - Expression evaluator: Proper error messages for unknown functions - Tests: Correct line tracking expectations
Adds a new MCP tool that allows users to force a full reindex of the search index without modifying or deleting any notes. Useful when: - Search returns empty results for content that exists - Search index appears stale or out of sync - After recovering from database issues The tool calls the existing /search/reindex API endpoint and provides clear feedback about the operation status.
- Fix _filter_by_from() to support nested file structure - Fix file_content desync after frontmatter update - Add refresh_dataview_relations() for auto-update - Add dataview_link relation type - Add comprehensive E2E and integration tests Closes US-001, US-002
Skip datetime.date and other non-string objects returned by Dataview queries when resolving links to create dataview_link relations. Prevents TypeError when processing Milestone.due_date fields that return date objects instead of strings.
- Add DataviewRefreshManager for automatic refresh - Debounce 5s to batch rapid changes - Only refresh entities impacted by changes (based on FROM clauses) - Cache entities with Dataview queries for performance - Integrate into SyncService.sync_markdown_file() No more manual refresh needed - relations update automatically.
…ueries - Add GET /entities/dataview endpoint to knowledge_router - Add list_entities_for_dataview method to KnowledgeClient - Pass notes_provider to create_dataview_integration in build_context - Add TDD tests for build_context Dataview integration Fixes: Dataview queries in build_context were returning 0 results because no notes_provider was passed to the DataviewIntegration.
- Pass notes_provider to create_dataview_integration() in read_note - Include result_markdown in build_context output instead of summary - Add file.path to query results for proper link resolution - Support nested file structure in FieldResolver with fallback - Read entity content from disk in dataview_refresh_manager - Call force_refresh_all() after sync to update dataview relations Closes US-003
- Add test_read_note_dataview.py (4 tests) - Add test_build_context_dataview.py (8 tests) - Fix test_dataview_relations_persistence assertions
|
|
Member
|
Hi thanks @DonaldoDes for the PR. It looks pretty cool. I will check it out. Can you tell me more about how you've been using BM with the Dataview info from Obsidian? I haven't had a chance to explore that yet. You can also dm me on our discord or email me at paul@basicmemory.com. There are couple issues with the files in your PR that we' d need to address
We can clean this up as needed. Also, would you mind signing off on your commits? This helps us to ensure we have all the copyright stuff ironed out correctly. To sign off Your AI can also sign off existing commits and do a force push. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds comprehensive Dataview query support to Basic Memory, enabling Obsidian-style queries in markdown notes with automatic relation tracking and MCP tool integration.
Features
Core Dataview Engine
MCP Integration
Automatic Relation Tracking
Additional Tools
Testing
Commits
fix: handle datetime strings from SQLite raw SQL queriesfeat: Add Dataview query parser and executor with MCP integrationfeat(mcp): add force_reindex tool to rebuild search indexchore: add crush config and test filesfeat(dataview): persist discovered links as relationsfix(dataview): filter non-string targets in refresh_dataview_relationsfeat(dataview): automatic refresh of Dataview relations with debouncingfix(dataview): provide notes_provider to build_context for Dataview queriesfeat(dataview): provide notes to MCP tools for query executiontest(dataview): add tests for MCP tools dataview integrationDocumentation
DATAVIEW_QUICKSTART.md: Getting started guideDATAVIEW_IMPLEMENTATION_SUMMARY.md: Technical overviewDATAVIEW_TEST_REPORT.md: Test coverage reportsrc/basic_memory/dataview/README.md: API documentationsrc/basic_memory/dataview/INTEGRATION.md: Integration guideBreaking Changes
None. This is a purely additive feature.
Related Issues
Closes #[issue-number] (if applicable)