Skip to content

[WIP] Project Memory & Distributed indexing - #59

Open
RomanEmreis wants to merge 52 commits into
mainfrom
feature/project-memory
Open

[WIP] Project Memory & Distributed indexing#59
RomanEmreis wants to merge 52 commits into
mainfrom
feature/project-memory

Conversation

@RomanEmreis

@RomanEmreis RomanEmreis commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Completes the local repository intelligence work through RG4 and RG5.

  • Adds independently revisioned project memory built from approved specifications, outcomes, archive metadata, and bounded runtime provenance.
  • Adds deterministic repository-memory links plus bounded federated search and context through CLI and read-only MCP tools.
  • Introduces a vendor-neutral distributed indexing prototype with immutable filtered inputs, durable leased jobs, stateless workers, encrypted tenant-scoped storage, independent CAS publication, authenticated snapshot-pinned queries, retention, and deletion.
  • Keeps local SQLite behavior as the default. The distributed prototype does not start a network service or make cloud connectivity part of the Ferrus runtime.
  • Splits oversized runtime, graph, project, HQ, TUI, and test modules into smaller semantic submodules.
  • Reduces the README and moves detailed operational and architectural material into docs/.

Type

  • Bug fix
  • Feature
  • Enhancement
  • Performance
  • Documentation
  • Refactor
  • Security
  • Breaking change

Workflow impact (if applicable)

Does this change affect the Supervisor -> Executor -> Reviewer flow?

  • No impact
  • Minor change (does not alter lifecycle semantics)
  • Changes behavior of an existing step
  • Introduces new state/transition

If yes, briefly explain:

Spec archival now performs a best-effort project-memory refresh after the archive commit. New repository and project-context tools are read-only and do not change task, run, lease, review, or approval state.

Checklist

  • I added/updated tests where it makes sense
  • I updated docs/examples if needed
  • This change is backwards-compatible (or clearly marked as breaking)
  • I ran formatting/lints locally (if applicable)

Notes for reviewers

Suggested review order:

  1. RG4/RG5 specifications and architecture documents.
  2. src/project_memory/ and the federated retrieval surface.
  3. src/distributed/ contracts, security boundaries, workers, publication, and query adapters.
  4. Mechanical module and test extraction.

Please pay particular attention to tenant isolation, privacy filtering before storage, independent graph/memory publication, and the absence of any distributed dependency in the default local workflow.

@RomanEmreis RomanEmreis self-assigned this Aug 8, 2026
@RomanEmreis RomanEmreis added documentation Improvements or additions to documentation enhancement New feature or request refactor chore security platform idea labels Aug 8, 2026
@RomanEmreis RomanEmreis changed the title Project Memory & Distributed indexing [WIP] Project Memory & Distributed indexing Aug 8, 2026
@RomanEmreis

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a08c58a670

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/distributed/api_sqlite.rs
Comment thread src/distributed/maintenance_sqlite.rs Outdated
unrelated jobs during partial project deletion
@RomanEmreis

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 193d29a4ac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/project_memory/query_sqlite.rs Outdated
@RomanEmreis

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 39f01b159e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/project_memory/query_sqlite.rs Outdated
@RomanEmreis

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d4616b5b6c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/project_memory/query_sqlite.rs Outdated
Comment thread src/project_memory/federation_service/support.rs Outdated
@RomanEmreis

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7acb283793

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/distributed/source.rs
Comment thread src/distributed/worker.rs Outdated
bounded reads for every worker source object
@RomanEmreis

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 92c71b9b78

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/distributed/worker.rs
Comment thread src/hq/workspace.rs
untracked symlinks when seeding worktrees
@RomanEmreis

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 14eac18467

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/distributed/worker.rs
@RomanEmreis

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e763b8c51e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/distributed/api_sqlite/support.rs
@RomanEmreis

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cb105613aa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/distributed/maintenance_sqlite.rs Outdated
Comment thread src/distributed/api_sqlite.rs Outdated
mutable-view resolution by the query deadline
@RomanEmreis

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2c93f101a9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/project_memory/query_sqlite.rs Outdated
Comment thread src/distributed/api_sqlite.rs Outdated
@RomanEmreis

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 122ba0156f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +482 to +484
let byte_len = self
.database
.query_row(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clamp object metadata lookup to the worker deadline

When object-store.db is locked and the remaining job duration is shorter than the store's fixed five-second busy timeout, this metadata query can block past the deadline before the subsequent time check returns DeadlineExceeded. Fresh evidence after the earlier bounded-read reports is that the new read_verified_bounded boundary still performs this SQLite lookup without installing a deadline-aware busy timeout or progress handler, so every manifest and source-object worker read can exceed its duration cap under contention.

AGENTS.md reference: AGENTS.md:L142-L146

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore documentation Improvements or additions to documentation enhancement New feature or request idea platform refactor security

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant