Skip to content

Commit 1a0aa8a

Browse files
iAmGiGiAmGiG
authored
Merge pull request #473 from iAmGiG/feature/469-471-data-caching
feat: Database-first caching architecture (#469, #471)
2 parents d2e781f + d42efb2 commit 1a0aa8a

File tree

6 files changed

+968
-7
lines changed

6 files changed

+968
-7
lines changed

‎docs/04_development/04_cache_developer_guide.md‎

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Cache System Developer Guide
22

33
**Audience**: Developers integrating with or extending the cache system
4-
**Last Updated**: November 2025 (Issue #336)
4+
**Last Updated**: December 2025 (Issue #469)
55

66
---
77

@@ -61,6 +61,44 @@ cache_adapter.set_market_data(
6161
)
6262
```
6363

64+
### Using UnifiedBrokerCache (Issue #469)
65+
66+
For broker state caching (account, positions, orders) with database-first architecture:
67+
68+
```python
69+
from src.data_sources.cache import unified_broker_cache
70+
71+
# Get cached account (fetches if stale, returns from DB)
72+
account = unified_broker_cache.get_account(
73+
account_id="paper_main",
74+
fetcher=lambda: alpaca_monitor.get_account_status(),
75+
max_age_seconds=60
76+
)
77+
78+
# Get cached positions
79+
positions = unified_broker_cache.get_positions(
80+
account_id="paper_main",
81+
fetcher=lambda: alpaca_monitor.get_positions()
82+
)
83+
84+
# Store position snapshots for historical tracking
85+
unified_broker_cache.store_position_snapshot("paper_main", positions)
86+
87+
# Audit display events
88+
unified_broker_cache.audit_display(
89+
display_type="portfolio",
90+
data=positions,
91+
cache_source="cached",
92+
cache_age_seconds=30.5
93+
)
94+
95+
# Get cache info
96+
info = unified_broker_cache.get_cache_info("paper_main")
97+
print(f"Account cache: {info['cache_entries'].get('account', {})}")
98+
```
99+
100+
See [Database-First Caching Design](../design/469-database-first-caching.md) for architecture details.
101+
64102
---
65103

66104
## API Reference

‎docs/design/469-database-first-caching.md‎

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
# Database-First Caching Architecture Design
2+
3+
**Issue:** #469
4+
**Status:** Design Phase
5+
**Author:** Claude Code
6+
**Date:** 2025-12-04
7+
8+
## Executive Summary
9+
10+
This document outlines the design for implementing a database-first caching architecture where all data flows through the database before being displayed to users, ensuring consistency between stored and displayed data.
11+
12+
## Current State Analysis
13+
14+
### Data Flow Patterns Identified
15+
16+
| Path | Data Type | Current Pattern | Cache Type | Issues |
17+
|------|-----------|-----------------|------------|--------|
18+
| Market Data | OHLCV bars | fetch -> cache -> display | SQLite | Cache key mismatch (fixed) |
19+
| Account Status | Equity, buying power | fetch -> memory cache -> display | In-memory 60s | No persistence |
20+
| Positions | Holdings, P&L | fetch -> memory cache -> display | In-memory 60s | No audit trail |
21+
| Orders | Open/filled orders | fetch -> display (merge local) | None + JSON | Dual source of truth |
22+
| Analysis | MACD/RSI signals | calculate -> database -> display | SQLite | **Only DB-first path** |
23+
24+
### Current vs Proposed Architecture
25+
26+
```mermaid
27+
flowchart TB
28+
subgraph Current[Current Architecture]
29+
A1[Alpaca API] --> B1[In-Memory Cache]
30+
A1 --> C1[Display to User]
31+
B1 -.-> D1[Maybe Store]
32+
end
33+
34+
subgraph Proposed[Proposed Architecture Database-First]
35+
A2[Alpaca API] --> B2[UnifiedBrokerCache]
36+
B2 --> C2[(SQLite Database)]
37+
C2 --> D2[Display to User]
38+
end
39+
```
40+
41+
### Detailed Data Flow Sequence
42+
43+
```mermaid
44+
sequenceDiagram
45+
participant User
46+
participant CLI
47+
participant Cache as UnifiedBrokerCache
48+
participant DB as SQLite
49+
participant API as Alpaca API
50+
51+
User->>CLI: Request portfolio
52+
CLI->>Cache: get_positions(account_id)
53+
Cache->>DB: Check cache freshness
54+
alt Cache Fresh
55+
DB-->>Cache: Return cached data
56+
Cache-->>CLI: Return positions
57+
else Cache Stale
58+
Cache->>API: Fetch fresh data
59+
API-->>Cache: Return positions
60+
Cache->>DB: Store to database
61+
DB-->>Cache: Confirm stored
62+
Cache-->>CLI: Return positions
63+
end
64+
CLI->>Cache: audit_display()
65+
Cache->>DB: Log audit entry
66+
CLI-->>User: Display portfolio
67+
```
68+
69+
### Cache State Machine
70+
71+
```mermaid
72+
stateDiagram-v2
73+
[*] --> Empty
74+
Empty --> Fresh: API Fetch + Store
75+
Fresh --> Stale: TTL Expired
76+
Stale --> Fresh: Refresh
77+
Fresh --> Fresh: Cache Hit
78+
Stale --> Stale: API Fail Serve Stale
79+
```
80+
81+
### Database Schema - Entity Relationship
82+
83+
```mermaid
84+
erDiagram
85+
broker_state_cache {
86+
int id PK
87+
string account_id
88+
string state_type
89+
json data_json
90+
}
91+
92+
position_snapshots {
93+
int id PK
94+
string account_id FK
95+
string symbol
96+
float qty
97+
float unrealized_pnl
98+
}
99+
100+
order_snapshots {
101+
int id PK
102+
string account_id FK
103+
string order_id
104+
string symbol
105+
string status
106+
}
107+
108+
display_audit_log {
109+
int id PK
110+
datetime display_time
111+
string display_type
112+
json data_json
113+
}
114+
115+
broker_state_cache ||--o{ position_snapshots : has
116+
broker_state_cache ||--o{ order_snapshots : has
117+
```
118+
119+
## SQL Schema Definitions
120+
121+
### broker_state_cache Table
122+
123+
```sql
124+
CREATE TABLE IF NOT EXISTS broker_state_cache (
125+
id INTEGER PRIMARY KEY AUTOINCREMENT,
126+
account_id TEXT NOT NULL,
127+
state_type TEXT NOT NULL,
128+
data_json TEXT NOT NULL,
129+
fetched_at TEXT NOT NULL,
130+
expires_at TEXT NOT NULL,
131+
UNIQUE(account_id, state_type)
132+
);
133+
```
134+
135+
## Implementation Plan
136+
137+
### Phase 1: Core Infrastructure (This PR)
138+
139+
1. Create UnifiedBrokerCache class - SQLite-backed broker state caching
140+
2. Fix cache_adapter.py - Fixed cache key mismatch between GET and SET
141+
3. Create BrokerSnapshotManager - Store position/order snapshots
142+
143+
### Phase 2: Integration
144+
145+
1. Replace BrokerStateCache usage with UnifiedBrokerCache
146+
2. Update portfolio_tools.py to query from cache
147+
3. Update order_tools.py to use unified orders table
148+
149+
### Migration Strategy
150+
151+
```mermaid
152+
flowchart TD
153+
A[Start Migration] --> B{Feature Flag Enabled?}
154+
B -->|No| C[Use Existing JSON]
155+
B -->|Yes| D[Write to Both DB + JSON]
156+
D --> E{Validation Pass?}
157+
E -->|No| F[Log Discrepancy]
158+
F --> C
159+
E -->|Yes| G[Read from DB Only]
160+
G --> H[Remove JSON Fallback]
161+
```
162+
163+
## Success Metrics
164+
165+
- Cache hit rate greater than 80 percent for broker state
166+
- Display latency less than 500ms
167+
- 100 percent audit coverage for displayed data
168+
- Zero discrepancy between displayed and stored data
169+
170+
## Timeline Estimate
171+
172+
- Phase 1 (Core): 2-3 hours
173+
- Phase 2 (Integration): 3-4 hours
174+
- Phase 3 (Display): 2-3 hours
175+
- Phase 4 (Validation): 1-2 hours
176+
177+
Total: approximately 10 hours of implementation work

‎src/data_sources/cache/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,13 @@
33
from .cache_adapter import CacheAdapter, cache_adapter
44
from .news_cache import NewsCache
55
from .sqlite_cache import TradingCacheManager
6+
from .unified_broker_cache import UnifiedBrokerCache, unified_broker_cache
67

78
__all__ = [
89
"NewsCache",
910
"TradingCacheManager", # SQLite-based cache (production)
1011
"CacheAdapter",
1112
"cache_adapter",
13+
"UnifiedBrokerCache", # Database-first broker state cache (#469)
14+
"unified_broker_cache",
1215
]

‎src/data_sources/cache/cache_adapter.py‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -55,10 +55,13 @@ def get_market_data(
5555
"""
5656
# Combine source with timeframe to create unique cache key
5757
# This ensures different timeframes are cached separately
58-
cache_source = f"{source}_{timeframe}" if source not in ("any", "auto") else None
59-
if cache_source is None and timeframe != "1Day":
60-
# For non-daily timeframes with 'any' source, include timeframe
61-
cache_source = f"any_{timeframe}"
58+
# Must match the logic in set_market_data() for consistency
59+
if source in ("any", "auto"):
60+
# No source filter for generic requests
61+
cache_source = None if timeframe == "1Day" else f"any_{timeframe}"
62+
else:
63+
# Specific source: use {source}_{timeframe} for non-daily, just source for daily
64+
cache_source = f"{source}_{timeframe}" if timeframe != "1Day" else source
6265

6366
# First try SQLite cache
6467
data = self.cache.get(symbol, start_date, end_date, source=cache_source)

0 commit comments

Comments
 (0)