Current-state reference for FormicOS institutional memory: entry lifecycle, Bayesian confidence, Thompson Sampling retrieval, composite scoring, decay, and maintenance. Code-anchored to Wave 59.
Every knowledge entry is a MemoryEntry (Pydantic frozen model, extra="forbid")
defined in core/types.py. Persisted as plain dicts on MemoryEntryCreated events
for replay safety.
| Field | Type | Default | Purpose |
|---|---|---|---|
id |
str |
— | mem-{colony_id}-{type[0]}-{index} |
entry_type |
MemoryEntryType |
— | skill or experience |
sub_type |
EntrySubType | None |
None |
Granular classification (see below) |
status |
MemoryEntryStatus |
candidate |
Trust lifecycle position |
polarity |
MemoryEntryPolarity |
positive |
positive, negative, neutral |
decay_class |
DecayClass |
ephemeral |
Confidence decay rate |
Skills: technique, pattern, anti_pattern, trajectory.
Experiences: decision, convention, learning, bug.
EntrySubType is a StrEnum in core/types.py. The trajectory sub-type
(Wave 58) stores compressed tool-call sequences in trajectory_data:
each dict has {tool: str, agent_id: str, round_number: int}.
| Field | Type | Default | Purpose |
|---|---|---|---|
title |
str |
— | Short descriptive title |
content |
str |
— | Full actionable knowledge |
summary |
str |
"" |
One-line for search display |
domains |
list[str] |
[] |
Domain tags (normalized: lowercase, underscores) |
tool_refs |
list[str] |
[] |
Tool names referenced |
primary_domain |
— | — | Stamped after model_dump() (not a model field) |
| Field | Type | Purpose |
|---|---|---|
source_colony_id |
str |
Colony that produced this entry |
source_artifact_ids |
list[str] |
Artifact IDs from source colony |
source_round |
int |
Round number of source material |
workspace_id |
str |
Workspace scope |
thread_id |
str |
Thread scope (empty = workspace-wide) |
created_at |
str |
ISO timestamp |
playbook_generation |
str |
Content-hash of playbooks at extraction time |
| Field | Type | Default | Constraint |
|---|---|---|---|
conf_alpha |
float |
5.0 |
gt=0 |
conf_beta |
float |
5.0 |
gt=0 |
confidence |
float |
0.5 |
Initial scalar (legacy display) |
scan_status |
ScanStatus |
pending |
Security scan result tier |
Default prior: Beta(5, 5) — prior strength 10, split evenly. Point estimate
confidence = alpha / (alpha + beta).
candidate → verified → stale
↓ ↓
rejected rejected
| Status | Meaning |
|---|---|
candidate |
Newly extracted, awaiting verification |
verified |
Validated by operator or system |
rejected |
Explicitly rejected |
stale |
Decayed below relevance threshold |
Transitions happen via events: MemoryEntryStatusChanged,
MemoryEntryInvalidated (operator), MemoryEntryReinstated (operator).
| Class | Gamma (γ) | Half-life | Use |
|---|---|---|---|
ephemeral |
0.98 | ~34 days | Task-specific observations, workarounds |
stable |
0.995 | ~139 days | Domain knowledge, architectural decisions |
permanent |
1.0 | ∞ | Verified definitions, immutable truths |
Gamma-decay is applied at query time, not stored in the CRDT. A 180-day gamma cap prevents entries from decaying beyond recovery.
Entries with decay_class stable or permanent get a 20% gap-recovery bonus
when re-observed after significant decay (current_alpha < peak_alpha * 0.5).
peak_alpha is tracked in projections via the MemoryConfidenceUpdated handler.
Six signals, all normalized to [0, 1]. Default weights defined in
surface/knowledge_constants.py:
COMPOSITE_WEIGHTS = {
"semantic": 0.38, # vector similarity
"thompson": 0.25, # Thompson Sampling exploration score
"freshness": 0.15, # exponential decay, 90-day half-life
"status": 0.10, # verified=1.0, active=0.8, candidate=0.5, stale=0.0
"thread": 0.07, # 1.0 bonus for same-thread entries
"cooccurrence": 0.05, # sigmoid: 1 - e^(-0.6w)
}Semantic: Raw vector similarity from Qdrant hybrid search (dense + BM25 + RRF).
Thompson Sampling: exploration_score(alpha, beta, total_observations, ucb_weight)
from engine/scoring_math.py. Draws from Beta(alpha, beta) posterior and adds
UCB exploration bonus.
Freshness: 2.0^(-age_days / 90.0) — exponential decay with 90-day half-life.
Computed in engine/context.py:_compute_freshness().
Status bonus: {verified: 1.0, active: 0.8, candidate: 0.5, stale: 0.0}.
Defined in surface/knowledge_catalog.py:_STATUS_BONUS.
Thread bonus: Same-thread entries receive _thread_bonus = 1.0, weighted at 0.07.
Applied in KnowledgeCatalog.search() for thread-scoped retrieval.
Co-occurrence: Sigmoid normalization 1 - e^(-0.6w) where w is the raw
co-occurrence weight from the co-occurrence graph. Defined in
surface/knowledge_catalog.py:_sigmoid_cooccurrence().
Pin boost: Pinned entries get retrieval preference via _pin_boost.
Federation penalty: federated_retrieval_penalty(item) from surface/trust.py
penalizes weak foreign entries. Uses Bayesian PeerTrust (10th percentile of
Beta posterior). Hop discount: 0.7^hop with 0.5 cap.
Workspace-scoped weight overrides via configure_scoring MCP tool, stored in
WorkspaceConfigChanged events. Falls back to global defaults. At standard/full
retrieval tier, results include score_breakdown and ranking_explanation.
Four tiers control how much content is returned per result:
| Tier | Tokens/result | Content |
|---|---|---|
summary |
~15 | Title + one-line summary |
standard |
~75 | Title + summary + preview + domains + decay |
full |
~200+ | Everything including content, Beta params, merge/cluster info |
auto |
varies | Starts at summary, escalates if coverage is thin |
Formatting functions in engine/context.py: _format_tiered_catalog_item().
Entries are classified into confidence tiers for display (computed in
engine/context.py:_confidence_tier()):
| Tier | Criteria |
|---|---|
HIGH |
mean ≥ 0.7 and CI width < 0.20 |
MODERATE |
mean ≥ 0.45 |
LOW |
mean < 0.45 |
EXPLORATORY |
fewer than 3 real observations |
STALE |
status == "stale" |
CI width: 1.96 * sqrt(mean * (1 - mean) / (alpha + beta + 1)).
surface/memory_extractor.py:build_extraction_prompt() builds the LLM prompt
for dual skill + experience extraction. Called by colony_manager.py after
ColonyCompleted/ColonyFailed. Fire-and-forget — does not block colony lifecycle.
Extraction modes:
- Legacy (no existing entries): Returns
{"skills": [...], "experiences": [...]}. - Curating (Wave 59, existing entries provided): Returns
{"actions": [{type: "CREATE"|"REFINE"|"MERGE"|"NOOP", ...}]}.
Parameters include task_class (Wave 58.5) for primary_domain tagging and
existing_entries (Wave 59) for curation context.
primary_domain is stamped onto entry dicts after model_dump() (not a
MemoryEntry field, since the model uses extra="forbid").
is_environment_noise_text() filters out entries about workspace configuration,
missing tools, import errors, etc. Uses phrase matching (16 phrases) combined
with context + error pattern matching.
Second extraction pass on full colony transcript (Wave 33). Classifies turns
as bug, decision, convention, or learning. Maps to EntrySubType values.
Called at hook position 4.5 in the post-colony pipeline.
5-axis scanning: prompt injection, data exfiltration, credential leakage,
code safety, credential detection (via detect-secrets). See
surface/credential_scan.py.
All registered in surface/app.py via service_router.register_handler().
surface/maintenance.py. Finds near-duplicate entries via vector similarity,
asks LLM to merge. Emits MemoryEntryMerged events with
merge_source ∈ {"dedup", "federation", "extraction"}.
Marks entries as stale when confidence drops below threshold.
Identifies contradicting entries. Includes suggested_colony for auto-dispatch.
Resets priors on entries with excessive prediction errors.
Selects entries with access ≥ 5 and confidence < 0.65 (popular but unexamined).
Uses resolve_model("archivist", workspace_id) for LLM calls. Emits
MemoryEntryRefined events with refinement_source="maintenance".
Quality gates: entry exists, new_content ≥ 20 chars, content actually changed.
15 deterministic rules in surface/proactive_intelligence.py (no LLM calls):
- 7 knowledge-health rules: confidence decline, contradiction, federation trust drop, coverage gap, stale cluster, merge opportunity, federation inbound
- 4 performance rules: strategy efficiency, diminishing rounds, cost outlier, knowledge ROI
- Adaptive evaporation signal
- Branching-factor stagnation signal
- Earned-autonomy recommendation
Three rules (contradiction, coverage gap, stale cluster) include
suggested_colony configurations for auto-dispatch.
Dense co-occurrence clusters (≥5 entries, avg weight >3.0) are flagged as
distillation candidates. Archivist colonies synthesize into higher-order entries
(KnowledgeDistilled event). Distilled entries get decay_class="stable" and
elevated alpha (capped at 30).
| File | Purpose |
|---|---|
core/types.py |
MemoryEntry model, all enums |
surface/knowledge_catalog.py |
Retrieval, composite scoring, thread boost |
surface/knowledge_constants.py |
COMPOSITE_WEIGHTS, workspace overrides |
engine/context.py |
Freshness, confidence tiers, tiered formatting |
engine/scoring_math.py |
Thompson Sampling exploration_score |
surface/memory_extractor.py |
Extraction prompt, parsing, noise filter |
surface/maintenance.py |
Dedup, stale sweep, contradiction, curation |
surface/proactive_intelligence.py |
15 deterministic briefing rules |
surface/colony_manager.py |
Post-colony extraction orchestration |
surface/credential_scan.py |
5-axis security scanning |
surface/trust.py |
Federation penalty, peer trust |