Create data for Kpitest

[MarkSpectarium]

Implementation Prompt: Create Kpitest Seed Data from LiveTest Data

Context

The Kpitest project (C:/Users/Aracos/Desktop/Kpitest) is a KPI Dashboard Analytics Platform that uses SQLite with seed data for testing. Currently it uses synthetic data generated in backend/src/db/seed.ts.

This project (codebase-knowledge-extractor) has real game data in data/LiveTest/:

• live.json (5MB) - 127 Player entities from Metaplay
• chars.json (57MB) - 210 PlayerCharacter entities

The goal is to create a script that transforms this real game data into Kpitest seed data, replacing the synthetic data with analytics derived from actual player/character state.

LiveTest Data Structure

Player entities (live.json):

{
  "entities": [{
    "entityId": "Player:05cNvDUpiq",
    "persistedAt": "2025-12-18T05:48:00Z",
    "payload": {
      "playerId": "...",
      "playerName": "...",
      "playerLevel": 1,
      "stats": {
        "createdAt": "2025-12-12T09:42:54Z",
        "lastLoginAt": "2025-12-18T05:47:22Z",
        "totalLogins": 12
      },
      "loginHistory": [{ "clientPlatform": "Android", "deviceModel": "..." }],
      "characterRoster": { "characterIds": [...] },
      "wallet": {...}
    }
  }]
}

Character entities (chars.json):

{
  "entities": [{
    "entityId": "PlayerCharacter:054rS3hU3W",
    "payload": {
      "playerId": "...",
      "character": {
        "characterClassId": "ChacChel_Class",
        "levelSystem": { "currentLevel": 5, "currentXp": 0 },
        "questSystem": { 
          "questsById": { "FTUE_Quest_1": {...}, "DungeonGrindQuest": {...} },
          "completedQuests": {...}
        },
        "portalTrackComponent": {...},
        "inventory": {...},
        "wallet": {...}
      }
    }
  }]
}

Note: Files have a UTF-8 BOM that must be stripped before parsing:

let content = fs.readFileSync(path, 'utf8');
if (content.charCodeAt(0) === 0xFEFF) content = content.slice(1);
const data = JSON.parse(content);

Kpitest Schema

The Kpitest database has these tables (see Kpitest/backend/src/db/schema.sql):

Table	Purpose
users	Player profiles (user_id, first_seen, last_seen, device_type, platform, player_level)
sessions	Session tracking (user_id, session_id, start_time, end_time, duration_seconds)
ftue_progress	FTUE step completions (user_id, step_name, step_order, completed_at, time_spent_seconds)
quest_completions	Quest completions (user_id, quest_id, quest_name, completed_at)
dungeon_runs	Dungeon activity (user_id, portal_track, boss_name, started_at, completed_at, completed, xp_earned)
economy_transactions	Currency flow (user_id, transaction_type, currency_type, amount, timestamp, source)
gear_drops	Loot drops (user_id, rarity, item_type, dropped_at, equipped)
crashes	Crash events (user_id, session_id, timestamp, crash_type)
events	Generic events (event_type, user_id, timestamp, properties)

Deliverables

Create a new file Kpitest/scripts/import-livetest.ts that:

1. Reads LiveTest data from ../../codebase-knowledge-extractor/data/LiveTest/

   • Handle BOM in JSON files
   • Parse both live.json and chars.json

2. Maps Player → users table:

   • entityId → user_id (extract ID after "Player:")
   • stats.createdAt → first_seen
   • stats.lastLoginAt → last_seen
   • loginHistory[0].clientPlatform → platform (lowercase: "android", "ios")
   • loginHistory[0].deviceModel → device_type ("phone" or "tablet" based on heuristics)
   • Max character level from linked characters → player_level

3. Maps Character data → character-level metrics:

   • Link characters to players via payload.playerId
   • Use levelSystem.currentLevel for level data
   • Extract quest progress from questSystem

4. Generates synthetic events from observed state:

   • Sessions: Generate stats.totalLogins sessions spread between createdAt and lastLoginAt
   • FTUE: Check quest status for FTUE_Quest_* quests and infer completed steps
   • Quests: Extract completed quests from questSystem.completedQuests
   • Dungeons: Infer dungeon runs from portal track progress and quest objectives like CompleteNDungeonsQuestObjective
   • Economy: Generate transactions based on wallet state and inventory

5. Outputs to SQLite using better-sqlite3 (same pattern as existing seed.ts)

Technical Constraints

• Use TypeScript (follow existing patterns in Kpitest/backend/src/)
• Add script to Kpitest/package.json: "import-livetest": "npx tsx scripts/import-livetest.ts"
• Clear existing data before import (TRUNCATE or DELETE FROM tables)
• Handle missing/null data gracefully
• Log progress: "Importing X players, Y characters..."

Data Generation Strategy

Since LiveTest data is point-in-time state (not event logs), synthesize events:

1. Sessions: Spread totalLogins sessions evenly between createdAt and lastLoginAt. Each session 5-60 minutes.

2. FTUE: Check if FTUE quests are completed. Map to these steps:

   • tutorial_start (order 1)
   • character_creation (order 2)
   • first_battle (order 3)
   • inventory_tutorial (order 4)
   • first_quest_accept (order 5)
   • first_quest_complete (order 6)
   • ability_unlock (order 7)
   • portal_introduction (order 8)
   • first_dungeon_enter (order 9)
   • ftue_complete (order 10)

3. Dungeons: Look at portalTrackComponent status and CompleteNDungeonsQuestObjective.numDungeonsCompleted for run counts.

4. Economy: Calculate reasonable XP/gold earned based on character level and dungeon completions.

Success Criteria

1. Script runs without errors: npm run import-livetest
2. All 127 players imported to users table
3. Character data linked correctly (player_level reflects max character level)
4. Sessions generated proportional to totalLogins
5. FTUE progress reflects actual quest completion state
6. Kpitest dashboard shows realistic data when viewing imported data
7. No TypeScript errors: npm run typecheck passes

Running the Script

cd Kpitest
npm run import-livetest
npm run dev  # Verify data in dashboard

[MarkSpectarium]

Original issue description:

Can you check out the Desktop/Kpitest project and see if you can generate data for that project based on the LiveTest data in this one?