darwinex prep, portfolioworth fix

Author: JustinGuese

.claude/plans/lets-implement-livetrading-for-dazzling-wigderson.md

@@ -0,0 +1,87 @@
+# Darwinex DXtrade Live Trading Integration
+
+## Context
+
+The repo already supports three live brokers (Collective2, Interactive Brokers, eToro) via a shared `LiveBroker` abstraction in [tradingbot/livetrade/broker.py](tradingbot/livetrade/broker.py). The user just provisioned a Darwinex DXtrade CFD demo account (`dxtrade.darwinex.com`, master password auth) and wants a fourth broker added that mirrors the eToro/C2 wiring so existing bot weights can be copied to Darwinex via the same `LiveTradeCopier`.
+
+User decisions (locked in via clarifying Qs):
+- **API flavor**: official `/dxsca-web` REST + WebSocket (session-token auth via username + password)
+- **Symbol mapping**: catalog search first, fall back to a manual `symbol_mappings.json` entry — Darwinex is CFD-only so equity tickers like `QQQ` may need `QQQ.US`-style codes
+- **Default `dry_run`**: `false` (live from first successful run, like Collective2)
+- **Credentials**: only DXtrade username + master password — no OAuth consumer key
+
+Out of scope: DARWIN Info API (separate product), FIX API, MT4/MT5 paths.
+
+## DXtrade `/dxsca-web` API surface (confirmed via Devexperts SDK + community refs)
+
+- **Base URL**: `https://dxtrade.darwinex.com/dxsca-web`
+- **Login**: `POST /login` with `{"username", "password", "domain": "default"}` → returns session token in JSON; subsequent requests pass `Authorization: DXAPI <token>` header. Token must be refreshed on 401.
+- **Account/portfolio**: `GET /accounts/{accountId}/metrics` (cash, equity, margin) and `GET /accounts/{accountId}/portfolio` (positions list with `instrumentCode`, `quantity`, `openPrice`).
+- **Orders**: `POST /accounts/{accountId}/orders` with `{instrument, quantity, side: "BUY"|"SELL", type: "MARKET", legs: [...]}`. Close-by-position handled by submitting an opposing market order against `instrument` for the position quantity (DXtrade nets automatically).
+- **Cancel**: `DELETE /accounts/{accountId}/orders/{orderId}` (only relevant for non-market orders; we submit MARKET so cancellation is rarely needed).
+- **Instruments**: `GET /instruments?symbol=<query>` returns matching instrument descriptors (`symbol`, `description`, `type`, `currency`, etc.). Used for catalog search in `map_symbol`.
+- **Live quotes**: native quotes are WebSocket-only (`wss://dxtrade.darwinex.com/dxsca-web/md?format=JSON`). For v1 we **do not** open the WebSocket — `_get_native_price()` returns `0.0` and the base class falls back to yfinance, identical to how Collective2 behaves today. Worth revisiting once equities are confirmed CFD-mapped.
+
+> All endpoint paths are best-effort from public docs / community SDKs and must be verified against the live `/dxsca-web/specs` Swagger on first run. The plan deliberately keeps each endpoint isolated in a `_get`/`_post` helper so any path correction is a one-line fix.
+
+## Files to create
+
+### 1. `tradingbot/livetrade/darwinex.py` — broker class
+Mirror [tradingbot/livetrade/etoro.py](tradingbot/livetrade/etoro.py) structure. Class `DarwinexBroker(LiveBroker)`:
+- `__init__(username, password, account_id=None, demo=True, symbol_mapper=None, data_service=None)` — store creds, build `httpx.Client(base_url=...)`, set `self.name = "darwinex"`.
+- `_login()` — POST `/login`, store `self._session_token` and `self._token_expires_at`.
+- `_ensure_session()` — called at the top of every `_get/_post`; re-login on 401 or near-expiry.
+- `_get(path, params)` / `_post(path, json_data)` / `_delete(path)` — auth header injection + `raise_for_status()`.
+- `_resolve_account_id()` — if `account_id` not supplied, GET `/users/{username}/accounts` and pick the first (mirrors how IB picks the configured account).
+- `get_cash()` → `/accounts/{id}/metrics` → `balance`.
+- `get_total_equity()` → `/accounts/{id}/metrics` → `equity`.
+- `get_positions()` → `/accounts/{id}/portfolio` → dict of `instrumentCode → signed quantity`. Cache `instrumentCode → positionId` if needed for closes.
+- `_get_native_price()` → return `0.0` (yfinance fallback). Leave a `# TODO: WebSocket md subscription for live quotes` line since this is a known v1 limitation, not dead code.
+- `place_order(broker_symbol, quantity, side, symbol_type)` → POST `/accounts/{id}/orders` with `{instrument: broker_symbol, quantity, side, type: "MARKET"}`. Respect `dry_run` upstream (copier handles it; broker doesn't need to know).
+- `map_symbol(yf_symbol)` — **two-step** per the user's choice:
+  1. Check `self.symbol_mapper.map_symbol(yf_symbol, "darwinex")` — if a manual mapping exists in `symbol_mappings.json`, return it.
+  2. Otherwise `GET /instruments?symbol={yf_symbol}` and `{yf_symbol}.US` (covers the common equity-CFD naming). Return the first exact-match `{symbol, description, type}` dict; `None` if no match.
+  - Cache results in `self._instrument_cache` to avoid hammering the search endpoint each sync.
+- `search_symbol(query)` → thin wrapper over `GET /instruments?symbol={query}`.
+- `cancel_open_orders()` → list `/accounts/{id}/orders?status=PENDING`, DELETE each, return count. Matches eToro behavior.
+- `print_account_summary()` → log cash/equity/positions (copy from C2 verbatim, swap field names).
+
+### 2. `tradingbot/livetrade_darwinex.py` — runner/CLI entry
+Copy [tradingbot/livetrade_etoro.py](tradingbot/livetrade_etoro.py) and swap:
+- Env vars: `DARWINEX_USERNAME`, `DARWINEX_PASSWORD`, `DARWINEX_ACCOUNT_ID` (optional), `DARWINEX_DEMO` (default `"true"`).
+- Default for `LIVETRADE_DRY_RUN`: `"false"` (per user decision — matches C2 runner).
+- Keep the existing DB validation block from the eToro runner unchanged (validates `LIVETRADE_BOT_WEIGHTS` against the Bot table).
+- Instantiate `DarwinexBroker(...)`, then `LiveTradeCopier(...).sync()`.
+
+### 3. `tests/test_livetrade_darwinex.py`
+Mirror [tests/test_livetrade_etoro.py](tests/test_livetrade_etoro.py). Mock `httpx.Client` at the broker; cover:
+- `_login` stores token from response
+- `get_cash` / `get_total_equity` parse the metrics response
+- `get_positions` aggregates portfolio response into `{symbol: qty}`
+- `map_symbol` (a) hits the manual map first, (b) falls back to instrument search, (c) returns None on no match
+- `place_order` posts the right JSON body for BUY and SELL
+
+### 4. `helm/tradingbots/templates/cronjob-livetrade-darwinex.yaml`
+Copy `cronjob-livetrade-etoro.yaml`, swap broker name, env-var names, and gate by `.Values.liveTrade.darwinex.enabled`.
+
+## Files to modify
+
+- **`helm/tradingbots/values.yaml`** — add a `liveTrade.darwinex` block under the existing `liveTrade.etoro` block (`enabled: false`, `schedule`, `demo: true`, optional `portfolioFraction`, `accountId` optional).
+- **`README.md`** — add Darwinex to the supported-brokers list, the env-var quick-start block, and the copier/Helm flag table (alongside the existing eToro additions).
+- **`docs/guides/live-trading.md`** — add a Darwinex section parallel to the eToro one; extend the configuration reference table to include `DARWINEX_*` rows; update the alpha-notice line to include Darwinex.
+- **`AGENTS.md`** — add a Darwinex API quirks subsection under the existing eToro #10 quirks block (session-token auth, CFD-only catalog, no native price endpoint, two-step symbol resolution).
+- **`symbol_mappings.json`** — add a small starter map for the most common bot-universe tickers (e.g. `QQQ → QQQ.US`, `SPY → SPY.US`) so the very first sync isn't entirely catalog-search dependent. Concrete codes to be confirmed against `/instruments` on first connect.
+
+## Verification (end-to-end)
+
+1. **Unit tests**: `pytest tests/test_livetrade_darwinex.py -v` — all green with mocked httpx.
+2. **Live demo dry-run sanity**: temporarily export `LIVETRADE_DRY_RUN=true` and run `python -m tradingbot.livetrade_darwinex` against the demo. Expect: login OK → metrics fetched → equity > 0 → positions empty → bot weights resolved → mapping log shows which yf tickers map to which DXtrade instruments → orders printed but not sent.
+3. **Live demo wet run**: with `LIVETRADE_DRY_RUN=false`, confirm a single small BUY (e.g. `EUR/USD` with min order size) lands in the DXtrade UI.
+4. **Mapping coverage spot-check**: run a one-off helper invoking `broker.map_symbol("QQQ")`, `("AAPL")`, `("BTC-USD")`, `("EURUSD=X")` and log results — surfaces which of your active bot tickers are catalog-resolvable vs need manual `symbol_mappings.json` entries.
+5. **Lint/type**: whatever the repo runs in CI for the eToro module (no special needs here).
+
+## Known v1 gaps (worth flagging in commit/PR)
+
+- No WebSocket live quotes — equity calculation uses yfinance for position notional, fine for equities/crypto but inaccurate for FX/CFD-only instruments off-hours.
+- DXtrade nets positions automatically; the copier's SELL-then-BUY ordering still works but `get_positions()` returns the *net* position, not per-trade lots. eToro had the inverse problem (per-position lots). Should be a non-issue but worth noting.
+- Endpoint paths are best-effort from community SDKs; first live run will likely surface a 404 on one or two of them, easily corrected via `/dxsca-web/specs`.

.env.example

@@ -41,6 +41,12 @@ ETORO_API_KEY=
 ETORO_USER_KEY=guestros
 ETORO_DEMO=true
 
+# Darwinex DXtrade (https://dxtrade.darwinex.com/)
+DARWINEX_USERNAME=
+DARWINEX_PASSWORD=
+DARWINEX_ACCOUNT_ID=
+DARWINEX_DEMO=true
+
 LIVETRADE_BOT_WEIGHTS={"AdaptiveMeanReversionBot": 1.0}
 LIVETRADE_COPY_OPEN_TRADES=true
 LIVETRADE_MIN_ORDER_USD=50

.vscode/launch.json

@@ -58,6 +58,33 @@
         "PYTHONPATH": "${workspaceFolder}/tradingbot"
       }
     },
+    {
+      "name": "LiveTrade: Darwinex (paper)",
+      "type": "debugpy",
+      "request": "launch",
+      "program": "${workspaceFolder}/tradingbot/livetrade_darwinex.py",
+      "console": "integratedTerminal",
+      "cwd": "${workspaceFolder}/tradingbot/",
+      "envFile": "${workspaceFolder}/.env",
+      "env": {
+        "LIVETRADE_DRY_RUN": "false",
+        "DARWINEX_DEMO": "true",
+        "PYTHONPATH": "${workspaceFolder}/tradingbot"
+      }
+    },
+    {
+      "name": "Print Portfolio: Darwinex",
+      "type": "debugpy",
+      "request": "launch",
+      "program": "${workspaceFolder}/tradingbot/livetrade/darwinex.py",
+      "console": "integratedTerminal",
+      "cwd": "${workspaceFolder}/tradingbot/",
+      "envFile": "${workspaceFolder}/.env",
+      "env": {
+        "DARWINEX_DEMO": "true",
+        "PYTHONPATH": "${workspaceFolder}/tradingbot"
+      }
+    },
     {
       "name": "Print Portfolio: eToro",
       "type": "debugpy",

AGENTS.md

@@ -863,7 +863,35 @@ The eToro Public API (`https://public-api.etoro.com`, `/api/v1` prefix) has a nu
 - Body must include `{"InstrumentId": <int>}` even though the position ID is in the path. Missing instrument ID → HTTP 400 "InstrumentId: The instrument id does not exist".
 - httpx with `json=None` omits the body **and** the `Content-Type` header → eToro returns 415 Unsupported Media Type. Always pass at least `{}` (or the InstrumentId payload above).
 
-### 11. `POSTGRES_URI` Required Even for Non-DB Tests
+### 11. Darwinex (DXtrade) Broker — API Quirks
+
+The Darwinex DXtrade API (`/dxsca-web`) has its own set of nuances.
+
+**Auth**: Session-token based.
+- `POST /login` with `{"username", "password", "domain": "default"}` returns `sessionToken`.
+- Subsequent requests require `Authorization: DXAPI <token>` header.
+- Tokens expire (typically 24h, but v1 implementation refreshes on 401 or after 2h).
+
+**Account Selection**: A single user may have multiple accounts.
+- Use `GET /users/{username}/accounts` to list them.
+- If `DARWINEX_ACCOUNT_ID` is not provided, the broker picks the first one from this list.
+
+**Portfolio & Metrics**:
+- Metrics (`GET /accounts/{id}/metrics`): Returns `balance` (cash) and `equity` (MTM value).
+- Portfolio (`GET /accounts/{id}/portfolio`): Returns a list of `positions`.
+- DXtrade nets positions automatically; `get_positions()` aggregates by `instrumentCode`.
+- Position quantity is signed based on `side` (BUY = positive, SELL = negative).
+
+**Symbol Mapping (CFD Catalog)**:
+- Darwinex is a CFD-only broker. Equities are often suffixed with `.US` (e.g., `AAPL.US`).
+- `map_symbol()` first checks the manual `symbol_map.json`, then performs a catalog search (`GET /instruments?symbol=...`) for both the bare ticker and the `.US` variant.
+- Catalog search results can be a list or wrapped in an `instruments` key; implementation handles both.
+
+**No Native Last Price**:
+- The REST API does not provide a simple snapshot "last price" endpoint. Quotes are WebSocket-only (`/md`).
+- `_get_native_price()` returns `0.0`, triggering the base class's **yfinance fallback** for all order sizing and equity calculations.
+
+### 12. `POSTGRES_URI` Required Even for Non-DB Tests
 
 **Problem**: `pytest tests/...` fails with `KeyError: 'Set POSTGRES_URI or (POSTGRES_HOST + POSTGRES_PASSWORD) for database connection'` even when running tests that don't touch the DB (e.g. pure-mock tests under `tests/test_livetrade.py`).
 

README.md

@@ -377,12 +377,12 @@ telegramMonitor:
 
 See [Telegram Monitor Guide](docs/guides/telegram-monitor.md) for full setup instructions.
 
-## 📈 Live Trading (Collective2, Interactive Brokers & eToro)
+## 📈 Live Trading (Collective2, Interactive Brokers, eToro & Darwinex)
 
 > [!WARNING]
 > **DISCLAIMER:** This software is for educational and research purposes only. Trading involves significant risk of loss and is not suitable for all investors. Use of "Live Trading" features is strictly at your own risk. The authors and contributors are not liable for any financial losses, damages, or unintended trades incurred. Always test strategies thoroughly in a paper-trading environment before deploying real capital.
 
-The framework can mirror your paper-bot portfolios to a live brokerage account. Supported brokers: **Collective2** (World API v4), **Interactive Brokers** (via IB Gateway / `ib_async`), and **eToro** (Public REST API).
+The framework can mirror your paper-bot portfolios to a live brokerage account. Supported brokers: **Collective2** (World API v4), **Interactive Brokers** (via IB Gateway / `ib_async`), **eToro** (Public REST API), and **Darwinex** (DXtrade API).
 
 ### 1. Configure Environment
 
@@ -404,6 +404,12 @@ ETORO_API_KEY=your_public_key
 ETORO_USER_KEY=your_user_key
 ETORO_DEMO=true             # true for demo/paper, false for live
 
+# Darwinex (DXtrade API)
+DARWINEX_USERNAME=your_username
+DARWINEX_PASSWORD=your_password
+DARWINEX_ACCOUNT_ID=12345   # optional
+DARWINEX_DEMO=true          # true for demo/paper, false for live
+
 # Shared
 LIVETRADE_BOT_WEIGHTS='{"adaptivemeanreversionbot": 1.0}'
 LIVETRADE_DRY_RUN=false
@@ -425,6 +431,9 @@ uv run python tradingbot/livetrade/interactive_brokers.py
 
 # eToro (reads ETORO_API_KEY, ETORO_USER_KEY, ETORO_DEMO from .env)
 uv run python tradingbot/livetrade/etoro.py
+
+# Darwinex (reads DARWINEX_USERNAME, DARWINEX_PASSWORD, DARWINEX_DEMO from .env)
+uv run python tradingbot/livetrade/darwinex.py
 ```
 
 ### 2. Map Your Tickers
@@ -455,9 +464,12 @@ uv run python tradingbot/livetrade_interactive_brokers.py
 
 # eToro
 uv run python tradingbot/livetrade_etoro.py
+
+# Darwinex
+uv run python tradingbot/livetrade_darwinex.py
 ```
 
-Each broker is its own Helm CronJob gated by an independent flag in `values.yaml` — enable them separately (`liveTrade.enabled` for Collective2, `liveTradeIB.enabled` for IBKR, `liveTradeEToro.enabled` for eToro), so you can run any combination or none. All default to `false`. You can also cap how much of the account each broker mirrors via `LIVETRADE_PORTFOLIO_FRACTION` (default `1.0` = full account; e.g. `0.5` = half).
+Each broker is its own Helm CronJob gated by an independent flag in `values.yaml` — enable them separately (`liveTrade.collective2.enabled`, `liveTrade.interactiveBrokers.enabled`, `liveTrade.etoro.enabled`, `liveTrade.darwinex.enabled`), so you can run any combination or none. All default to `false`. You can also cap how much of the account each broker mirrors via `LIVETRADE_PORTFOLIO_FRACTION` (default `1.0` = full account; e.g. `0.5` = half).
 
 See the [Live Trading Guide](docs/guides/live-trading.md) for advanced configuration and mapping rules.