|
| 1 | +# OpenShock API Backend |
| 2 | + |
| 3 | +## Project Overview |
| 4 | +OpenShock backend: REST API, real-time WebSocket gateway (LCG), and scheduled jobs (Cron). Controls shock devices via ESP32 hubs over FlatBuffers WebSocket protocol with Redis pub/sub message routing. |
| 5 | + |
| 6 | +## Tech Stack |
| 7 | +- .NET 10, ASP.NET Core (SlimBuilder), C# latest |
| 8 | +- PostgreSQL via Npgsql + EF Core (DbContext pooling + factory) |
| 9 | +- Redis Stack (Redis.OM for documents, RediSearch indexing, pub/sub, keyspace notifications) |
| 10 | +- SignalR (WebSockets only, custom Redis backplane with `local#` prefix optimization) |
| 11 | +- FlatBuffers (hub↔LCG binary WebSocket protocol) |
| 12 | +- MessagePack (Redis pub/sub serialization) |
| 13 | +- Serilog (console + Grafana Loki + OpenTelemetry) |
| 14 | +- OpenTelemetry (Prometheus exporter at `/metrics`) |
| 15 | +- Asp.Versioning — URL-based: `/{version:apiVersion}/...` |
| 16 | +- OneOf discriminated unions for service return types |
| 17 | +- BCrypt.Net for password hashing |
| 18 | +- Fluid (Liquid) for email templates |
| 19 | +- HybridCache (memory + distributed, 5-min expiry) |
| 20 | +- TUnit for tests, Testcontainers for integration tests |
| 21 | + |
| 22 | +## Solution Projects |
| 23 | + |
| 24 | +| Project | Purpose | |
| 25 | +|---|---| |
| 26 | +| `Common` | Shared: DB context, entities, auth handlers, hubs, Redis models, services, middleware | |
| 27 | +| `API` | Main REST API for user-facing operations | |
| 28 | +| `LiveControlGateway` | WebSocket gateway for real-time hub (ESP32) connections | |
| 29 | +| `Cron` | Hangfire scheduled jobs | |
| 30 | +| `MigrationHelper` | Standalone EF Core migration tooling | |
| 31 | +| `SeedE2E` | E2E test data seeder (Bogus fakers) | |
| 32 | +| `API.IntegrationTests` | TUnit integration tests | |
| 33 | +| `Common.Tests` | TUnit unit tests for Common | |
| 34 | + |
| 35 | +## Project Structure |
| 36 | + |
| 37 | +### Common (`Common/`) |
| 38 | +``` |
| 39 | +Authentication/ |
| 40 | + AuthenticationHandlers/ # UserSession, ApiToken, Hub auth handlers |
| 41 | + ControllerBase/ # AuthenticatedSessionControllerBase (IActionFilter) |
| 42 | + Services/ # IUserReferenceService |
| 43 | + Attributes/ # TokenPermissionAttribute |
| 44 | +Constants/ # AuthConstants, HardLimits |
| 45 | +DeviceControl/ # ControlSender, device command routing |
| 46 | +Errors/ # Static error factories |
| 47 | +ExceptionHandle/ # OpenShockExceptionHandler (IExceptionHandler) |
| 48 | +Extensions/ # ConfigurationExtensions, HttpContextExtensions |
| 49 | +Hubs/ # UserHub, PublicShareHub + interfaces |
| 50 | +JsonSerialization/ # JsonOptions, SemVersionJsonConverter |
| 51 | +Migrations/ # EF Core migrations |
| 52 | +Models/ # Domain enums (PermissionType, RoleType, ControlType, ShockerModelType) |
| 53 | +Models/WebSocket/ # WebSocket message DTOs |
| 54 | +OpenShockDb/ # OpenShockContext + all entity classes |
| 55 | +Options/ # DatabaseOptions, FrontendOptions, MetricsOptions, etc. |
| 56 | +Problems/ # OpenShockProblem (RFC 7807), ValidationProblem |
| 57 | +Redis/ # DeviceOnline, LoginSession, DevicePair, LcgNode (Redis.OM docs) |
| 58 | +Redis/PubSub/ # MessagePack models for device messages |
| 59 | +Services/ # SessionService, ControlSender, BatchUpdateService, etc. |
| 60 | +Utils/ # HashingUtils, CryptoUtils, GravatarUtils |
| 61 | +Validation/ # CharsetMatchers, UsernameValidator |
| 62 | +Websocket/ # WebsocketBaseController<T>, FlatbuffersWebsocketBaseController |
| 63 | +OpenShockServiceHelper.cs # Central service registration (DB, Redis, auth, rate limiting) |
| 64 | +OpenShockControllerBase.cs # Base controller with Problem(), LegacyDataOk() |
| 65 | +``` |
| 66 | + |
| 67 | +### API (`API/`) |
| 68 | +``` |
| 69 | +Controller/ |
| 70 | + Account/ # Login(V1/V2), Signup(V1/V2), Logout, Activate, PasswordReset |
| 71 | + Account/Authenticated/ # ChangeEmail, ChangePassword, ChangeUsername, Deactivate |
| 72 | + Admin/ # User management, config, webhooks, blacklists |
| 73 | + Device/ # Hub-authenticated: GetSelf, Pair, AssignLCG |
| 74 | + Devices/ # User-authenticated: CRUD, OTA, shockers, pair codes |
| 75 | + OAuth/ # Discord/Google/Twitter OAuth flow |
| 76 | + Public/ # Stats, public share links |
| 77 | + Sessions/ # List, delete, self |
| 78 | + Shares/ # Public links, user shares, share codes |
| 79 | + Shockers/ # CRUD, control, logs, pause, share management |
| 80 | + Tokens/ # API token CRUD, self, reporting |
| 81 | + Users/ # GetSelf, LookupByName |
| 82 | + Version/ # GET / — server info |
| 83 | +Errors/ # API-specific error statics |
| 84 | +Services/ # AccountService, TurnstileService, EmailService, etc. |
| 85 | +Realtime/RedisSubscriberService.cs # Hosted service: Redis keyspace + device-status listener |
| 86 | +``` |
| 87 | + |
| 88 | +### LiveControlGateway (`LiveControlGateway/`) |
| 89 | +``` |
| 90 | +Controllers/ |
| 91 | + HubV1Controller.cs # /{v}/ws/device — FlatBuffers v1 (HubToken auth) |
| 92 | + HubV2Controller.cs # /{v}/ws/device — FlatBuffers v2 (HubToken auth) |
| 93 | + LiveControlController.cs # /{v}/ws/live/{hubId} — JSON WebSocket (user auth) |
| 94 | +LifetimeManager/ # HubLifetimeManager — tracks connected hubs, routes commands |
| 95 | +``` |
| 96 | + |
| 97 | +## Key Architectural Patterns |
| 98 | + |
| 99 | +### Partial Controllers |
| 100 | +Each controller is a `sealed partial class`. `_ApiController.cs` declares the class with attributes + DI fields. Individual actions are separate `.cs` files as partial continuations. |
| 101 | + |
| 102 | +### OneOf Discriminated Unions |
| 103 | +Service methods return `OneOf<Success, Error1, Error2, ...>`. Callers use `.Match()` or `.TryPickT0()`. No exceptions for expected failure paths. |
| 104 | + |
| 105 | +### RFC 7807 Problem Details |
| 106 | +All errors use `OpenShockProblem` (extends `ProblemDetails`). Static factory classes create specific instances. `.ToObjectResult()` for controllers, `.WriteAsJsonAsync()` for auth handlers/WebSocket contexts. |
| 107 | + |
| 108 | +### WebSocket Base Controllers |
| 109 | +`WebsocketBaseController<T>` uses `Channel<T>` for thread-safe send queuing with background `MessageLoop`. `FlatbuffersWebsocketBaseController` extends it for binary FlatBuffers (LCG hub connections). |
| 110 | + |
| 111 | +### BatchUpdateService |
| 112 | +Singleton that batches async `last_used` timestamp updates for sessions and API tokens to prevent N+1 DB writes. |
| 113 | + |
| 114 | +## Authentication |
| 115 | + |
| 116 | +| Scheme | Header/Cookie | Handler | Notes | |
| 117 | +|---|---|---|---| |
| 118 | +| `UserSessionCookie` | Cookie `openShockSession` or header `OpenShockSession` | Looks up `LoginSession` in Redis → user from DB | Default scheme, expands TTL | |
| 119 | +| `ApiToken` | Header `OpenShockToken` | SHA-256 hash lookup in DB | Permission-checked via `TokenPermissionAttribute` | |
| 120 | +| `HubToken` | Header `DeviceToken` | Looks up `Device` in DB | For ESP32 hubs | |
| 121 | + |
| 122 | +Combined scheme `"UserSessionCookie,ApiToken"` for endpoints accepting either. |
| 123 | + |
| 124 | +All handlers write RFC 7807 JSON on 401 challenge (no redirect). |
| 125 | + |
| 126 | +## Database (PostgreSQL + EF Core) |
| 127 | + |
| 128 | +Key entities: `User`, `Device`, `Shocker`, `ApiToken`, `UserShare`, `PublicShare`, `ShockerControlLog`, `DeviceOtaUpdate`, `LoginSession` (Redis), `DeviceOnline` (Redis) |
| 129 | + |
| 130 | +PostgreSQL enums: `control_type`, `permission_type`, `role_type`, `shocker_model_type`, `ota_update_status` |
| 131 | + |
| 132 | +Collation: `ndcoll` (ICU case-insensitive) on `users.name` and username blacklist. |
| 133 | + |
| 134 | +Both `AddDbContextPool` and `AddPooledDbContextFactory` registered (factory used by LCG per-message scopes). |
| 135 | + |
| 136 | +## Redis |
| 137 | + |
| 138 | +Four Redis.OM indexed document types: |
| 139 | +- `LoginSession` — user sessions with TTL |
| 140 | +- `DeviceOnline` — connected device state (TTL-based offline detection via keyspace notifications) |
| 141 | +- `DevicePair` — pairing code ↔ device mapping |
| 142 | +- `LcgNode` — active LCG gateway nodes |
| 143 | + |
| 144 | +Pub/sub channels: |
| 145 | +- `device-msg:{deviceId}` — control commands → consumed by LCG |
| 146 | +- `device-status` — online/offline events → consumed by API's `RedisSubscriberService` |
| 147 | + |
| 148 | +## SignalR Hubs |
| 149 | + |
| 150 | +| Hub | Path | Auth | |
| 151 | +|---|---|---| |
| 152 | +| `UserHub` | `/1/hubs/user` | UserSessionCookie or ApiToken | |
| 153 | +| `PublicShareHub` | `/1/hubs/share/link/{id:guid}` | Optional session | |
| 154 | + |
| 155 | +WebSockets only. Custom `OpenShockRedisHubLifetimeManager` supports `local#` prefix for same-node optimization. |
| 156 | + |
| 157 | +## Configuration |
| 158 | + |
| 159 | +Environment variable prefix: `OPENSHOCK__` (double underscore for nested). Key sections: |
| 160 | +- `OpenShock:DB` → `DatabaseOptions` (Conn, SkipMigration) |
| 161 | +- `OpenShock:Redis` → connection (Conn string or Host/Port/User/Password) |
| 162 | +- `OpenShock:Frontend` → `FrontendOptions` (BaseUrl, ShortUrl, CookieDomain — comma-separated) |
| 163 | +- `OpenShock:Turnstile` → `TurnstileOptions` (Enabled, SecretKey, SiteKey) |
| 164 | +- `OpenShock:Mail` → `MailOptions` (type: MAILJET or SMTP) |
| 165 | + |
| 166 | +Under integration tests (`ASPNETCORE_UNDER_INTEGRATION_TEST=1`): only environment variables loaded. |
| 167 | + |
| 168 | +## Rate Limiting |
| 169 | + |
| 170 | +Global sliding window: 1000 req/min unauthenticated, 120 req/min per user, unlimited for Admin/System. |
| 171 | +Named policies: `"auth"` (10/min fixed window by IP), `"token-reporting"` (concurrency 5), `"shocker-logs"` (concurrency 10). |
| 172 | + |
| 173 | +## Commands |
| 174 | + |
| 175 | +```bash |
| 176 | +# Build |
| 177 | +dotnet build API/API.csproj |
| 178 | +dotnet build LiveControlGateway/LiveControlGateway.csproj |
| 179 | +dotnet build Cron/Cron.csproj |
| 180 | + |
| 181 | +# Test (requires Docker for Testcontainers) |
| 182 | +dotnet test Common.Tests/Common.Tests.csproj |
| 183 | +dotnet test API.IntegrationTests/API.IntegrationTests.csproj |
| 184 | + |
| 185 | +# Migrations |
| 186 | +dotnet ef migrations add <Name> --project Common --startup-project MigrationHelper |
| 187 | + |
| 188 | +# Docker |
| 189 | +docker build -f docker/API.Dockerfile . |
| 190 | +docker build -f docker/LiveControlGateway.Dockerfile . |
| 191 | +docker build -f docker/Cron.Dockerfile . |
| 192 | +``` |
| 193 | + |
| 194 | +## CI/CD |
| 195 | +- `ci-build.yml`: test → build Docker → promote → deploy (master→prod, develop→staging) |
| 196 | +- `ci-tag.yml`: semver tag releases, multi-arch (amd64+arm64) |
| 197 | +- Images: `ghcr.io/openshock/{api,live-control-gateway,cron}` |
| 198 | +- GitOps: dispatch to `openshock/kubernetes-cluster-gitops` |
| 199 | + |
| 200 | +## Integration Test Patterns |
| 201 | +- TUnit with `[ClassDataSource<WebApplicationFactory>(Shared = SharedType.PerTestSession)]` |
| 202 | +- Testcontainers: PostgreSQL + Redis Stack (shared per session) |
| 203 | +- `InterceptedHttpMessageHandler` mocks Cloudflare Turnstile (`"valid-token"` → success) and MailJet |
| 204 | +- `TestHelper` bypasses signup/login for test setup (direct DB + Redis session creation) |
| 205 | +- Rate limiting disabled in test host |
| 206 | +- Cookie domain includes `localhost` for test server |
| 207 | +- Auth helpers: `CreateAuthenticatedClient`, `CreateApiTokenClient`, `CreateHubTokenClient` |
0 commit comments