InsightHub

Description

InsightHub is a full-stack web application that provides a streamlined workflow for data exploration and visualization. Users can upload CSV/JSON datasets, construct interactive dashboards with intelligent chart configurations, apply sophisticated aggregation logic with automatic data preprocessing, reorganize dashboard layouts with drag-and-drop functionality, and gain insights through real-time visualizations. Once complete, dashboards can be saved to the app's library or exported as professionally formatted PDF files for presentations.

Live Links

• Production: https://insighthub-frontend-nine.vercel.app/
• Code Base: https://github.com/agworkgit/insighthub

Research

The initial research phase focused on defining the core requirements for a lightweight but capable data-analysis web application. The goal was to identify what users actually need when exploring datasets and distil those needs into features that can be delivered efficiently in a solo-developer environment.

Key Research Areas

1. Data Upload & Parsing

• Compared client-side vs server-side parsing approaches.
• Chose server-side parsing with PapaParse to ensure consistent handling of large files and reduce frontend load.
• Evaluated how different libraries handled malformed CSV/JSON and edge-cases.
• Implemented intelligent type inference system that automatically detects quantitative, categorical, and temporal data types.

2. Charting Libraries

• Tested multiple charting solutions, including Chart.js, Nivo, Victory, and Recharts.

• Recharts was selected due to:

  • Strong TypeScript support
  • Responsive components
  • Minimal configuration for common chart types
  • Easy integration with dynamic data structures
  • Excellent support for series/grouped visualizations

3. Data Processing & Aggregation

• Developed a robust preprocessing pipeline that handles:
  • Multiple aggregation types (SUM, AVG, COUNT, MIN, MAX)
  • Series grouping for stacked and grouped charts
  • Automatic limiting of high-cardinality data to top 10 categories
  • Smart field validation to prevent performance issues
• Implemented Map-based aggregation for optimal performance with large datasets

4. Dashboard Layout & Interaction

• Reviewed drag-and-drop libraries: React Beautiful DnD, dnd-kit, and SortableJS.

• dnd-kit was chosen for its:

  • Modern architecture
  • Extensibility
  • Smooth interactions
  • Strong TypeScript support
  • Sortable grid layouts with customizable sizing

5. Authentication & UX Flow

• Studied typical auth flows for dashboard apps and compared libraries like Clerk, Auth0, and custom JWT solutions.
• Opted for custom JWT authentication to maintain full backend control and reduce cost/complexity.
• Implemented protected routes with automatic token validation.

6. Exporting Visualizations

• Investigated PDF export solutions.

• Selected html2canvas + jsPDF because:

  • It supports exporting dashboard components directly
  • It performs reliably across modern browsers
  • It avoids maintaining a separate server rendering layer

• Enhanced export functionality to:

  • Export only chart visuals (no controls or UI elements)
  • Maintain aspect ratios
  • Fit 2 charts per page with proper margins
  • Generate multi-page PDFs automatically

7. UI, Theming & Styling

• Compared CSS Modules, Tailwind, and SASS.
• Chose SASS with global variables to maintain a clean, structured, fully-custom design system.
• Designed a Theme Context to support persistent light/dark mode across the application.
• Implemented themed toast notifications that adapt to light/dark modes
• Used Lucide React for consistent, customizable icons

8. Data Quality & User Experience

• Implemented validation system to prevent common data visualization pitfalls:

  • Warns users when selecting quantitative fields for categorical axes
  • Limits high-cardinality fields (>50 unique values) to prevent UI overflow
  • Shows real-time feedback with warning banners and toast notifications

• Smart default configurations that adapt to chart type:

  • Pie charts automatically use categorical X-axis and hide series options
  • All charts default to AVG aggregation for meaningful insights
  • Charts default to full-width for better visibility

Outcome of the Research Phase

The research validated the project's direction: a responsive, intuitive dashboard app powered by a modular frontend and a stable backend. The chosen stack ensures a balance between flexibility, performance, and development speed — allowing future enhancements without architectural bottlenecks.

Development Timeline

Development started on 05.10.2025 and ended on 14.11.2025.

Features

Core Functionality

User Authentication

• Secure registration and login with JWT-based authentication
• Protected routes with automatic token validation
• Session persistence with localStorage

File Management

• Upload CSV and JSON files with automatic parsing
• Intelligent type inference (quantitative, categorical, temporal)
• Column metadata extraction with unique value counting
• File storage with user isolation

Dashboard Builder

Interactive chart creation with 5 visualization types:

• Bar charts (single and grouped)
• Line charts (single and multi-series)
• Area charts with stacking support
• Pie charts with automatic top-10 limiting
• Stacked bar charts for comparative analysis

Smart field selection with type-aware filtering:

• Only quantitative fields shown for Y-axis
• Categorical fields for X-axis and series grouping
• Real-time validation with visual warnings
• Advanced aggregation options (SUM, AVG, COUNT, MIN, MAX)
• Series plotting for multi-dimensional analysis
• Flexible chart sizing (full-width or half-width)
• Drag-and-drop chart reordering with @dnd-kit
• Inline dashboard naming with keyboard shortcuts
• Auto-save indicators and unsaved changes warnings

Data Processing

• Automatic data preprocessing with optimized aggregation
• Top-10 category limiting to prevent UI overflow
• Decimal precision control (2 decimal places max)
• High-cardinality field detection and warnings
• Series limiting (top 10) for stacked/grouped charts

Export & Sharing

Professional PDF export with:

• Chart-only rendering (no UI controls)
• Maintained aspect ratios
• 2 charts per page with proper margins
• Multi-page support for large dashboards
• Dashboard persistence with MongoDB

Library Management

• Saved dashboards with full configuration restoration
• Uploaded files with metadata (row/column counts)
• Sortable lists (name, date, size)
• Quick load and delete actions
• Data file association with dashboards

User Interface

• Collapsible sidebar with hover tooltips
• Navigation blocking for unsaved changes
• Light/Dark mode toggle with CSS variable theming
• Themed toast notifications (success, error, loading)
• Empty states with helpful onboarding messages
• Responsive grid layouts for all screen sizes

Styling & Design

SASS-based modular styling system with:

• Global variables for colors, spacing, typography
• Theme-aware components using CSS variables
• Responsive breakpoints (mobile, tablet, desktop)
• Smooth transitions and hover effects
• Accessibility-tested color palettes (WCAG AAA compliant)
• Custom scrollbars and focus states
• Animated canvas background on auth pages

Developer Experience

• Full TypeScript coverage for type safety
• Comprehensive Jest test suite with:
• Auth route testing (registration, login, validation)
• Dashboard CRUD operations
• Data file upload/management
• User isolation and authorization checks
• ESLint configuration for code quality
• Hot module replacement with Vite

Tech Stack

Frontend

Core

• React 19.2.0 (TypeScript)
• React Router DOM 6.30.1
• Vite 7.2.2 (build tool)

State Management & Context

• Custom React Context for Auth, Dashboard, DataFiles, Theme
• UUID 13.0.0 for unique identifiers

Data Visualization

• Recharts 3.4.1 (charts and graphs)
• Custom data preprocessing utilities
• Type inference system

UI & Interaction

• @dnd-kit (drag and drop) - core 6.3.1, sortable 10.0.0
• Lucide React 0.553.0 (icon library)
• React Hot Toast 2.6.0 (notifications)
• SASS 1.93.3 (styling)

Export & Utils

• html2canvas 1.4.1 (screenshot capture)
• jsPDF 3.0.3 (PDF generation)
• Axios 1.13.2 (HTTP client)

Backend

Core

• Express (TypeScript)
• Node.js with ts-node-dev
• MongoDB with Mongoose

Authentication & Security

• JWT (jsonwebtoken) for authentication
• Bcrypt for password hashing
• Custom auth middleware

File Processing

• Multer (file uploads with memory storage)
• PapaParse (CSV parsing)
• JSON parsing with validation

Validation & Error Handling

• Zod (schema validation)
• express-validator
• Custom error middleware

Utilities

• dotenv (environment variables)
• Morgan (HTTP logging)
• CORS configuration

Dev Tools & Testing

Frontend

• ESLint 9.39.1 with TypeScript parser
• Vite with React plugin
• TypeScript 5.9.3

Backend

• Jest 29.x with ts-jest
• Supertest for API testing
• MongoDB Memory Server for test isolation
• ts-node-dev for development

Installation

Clone the repository

git clone https://github.com/agworkgit/insighthub.git
cd insighthub

Install dependencies

Frontend

cd frontend
npm install
npm run dev

Backend

cd backend
npm install
npm run dev

Environment Variables

Create a .env file in /backend:

MONGO_URI=your_mongo_connection_string
JWT_SECRET=your_jwt_secret
PORT=5000

Usage

Getting Started

1. Register or log in to create your account
2. Upload datasets - Support for CSV and JSON files
3. Build dashboards by adding charts and selecting data fields
4. Configure visualizations:
   • Choose chart type (bar, line, pie, area, stacked)
   • Select X and Y axes from your data
   • Apply aggregations (AVG, SUM, COUNT, MIN, MAX)
   • Add series for multi-dimensional analysis
   • Adjust chart size (full or half width)
5. Organize layout - Drag and drop charts to reorder
6. Save your work - Name and save dashboards to your library
7. Export as PDF - Generate professional reports with your visualizations
8. Toggle theme - Switch between light and dark mode for visual comfort

Tips for Best Results

• Use categorical fields (text/categories) for X-axis
• Use quantitative fields (numbers) for Y-axis
• The app automatically limits high-cardinality data to top 10 items
• Charts default to AVG aggregation for meaningful insights
• Warnings appear if you select inappropriate field types
• All values are rounded to 2 decimal places for readability

Project Structure

insighthub
├──backend
│   ├──src
│   │   ├──config
│   │   │   └──db.ts
│   │   ├──middleware
│   │   │   └──auth.ts
│   │   ├──models
│   │   │   ├──Dashboard.ts
│   │   │   ├──DataFile.ts
│   │   │   └──User.ts
│   │   ├──routes
│   │   │   ├──auth.ts
│   │   │   ├──dashboards.ts
│   │   │   ├──data.ts
│   │   │   └──user.ts
│   │   ├──types
│   │   │   └──express.d.ts
│   │   ├──index.ts
│   │   └──server.ts
│   ├──tests
│   │   ├──auth.test.ts
│   │   ├──dashboard.test.ts
│   │   └──data.test.ts
│   ├──jest.config.cjs
│   ├──package.json
│   └──tsconfig.json
├──frontend
│   ├──src
│   │   ├──components
│   │   │   ├──auth
│   │   │   │   └──ProtectedRoute.tsx
│   │   │   ├──background
│   │   │   │   └──AnimatedGridLines.tsx
│   │   │   ├──dashboard
│   │   │   │   ├──ChartCard.tsx (smart validation, type-aware fields)
│   │   │   │   ├──ChartRenderer.tsx (5 chart types, series support)
│   │   │   │   └──DashboardGrid.tsx (drag-drop, 4-column responsive)
│   │   │   ├──layout
│   │   │   │   ├──Layout.tsx (themed toasts, sidebar state)
│   │   │   │   ├──Navbar.tsx (file upload, theme toggle)
│   │   │   │   └──Sidebar.tsx (collapsible, tooltips)
│   │   │   └──ui
│   │   │       └──Spinner.tsx
│   │   ├──context
│   │   │   ├──AuthContext.tsx (JWT management, protected routes)
│   │   │   ├──DashboardContext.tsx (chart CRUD, save/load)
│   │   │   ├──DataFilesContext.tsx (file processing, type inference)
│   │   │   └──ThemeContext.tsx (light/dark mode persistence)
│   │   ├──pages
│   │   │   ├──Builder.tsx (dashboard creation, PDF export)
│   │   │   ├──Library.tsx (saved dashboards, file management)
│   │   │   ├──Login.tsx
│   │   │   └──Register.tsx
│   │   ├──services
│   │   │   └──api.ts (axios configuration, API endpoints)
│   │   ├──styles
│   │   │   ├──base
│   │   │   │   ├──_globals.scss
│   │   │   │   ├──_themes.scss (CSS variables for theming)
│   │   │   │   └──_variables.scss
│   │   │   ├──components (modular SASS files)
│   │   │   └──index.scss
│   │   ├──types
│   │   │   └──data.ts (TypeScript interfaces for data structures)
│   │   ├──utils
│   │   │   ├──dataProcessor.ts (aggregation, top-10 limiting)
│   │   │   └──fileProcessor.ts (parsing, type inference)
│   │   ├──App.tsx
│   │   └──main.tsx
│   ├──package.json
│   └──vite.config.ts
└──README.md

Tests

Backend (JEST)

Comprehensive test suite covering:

• Authentication (registration, login, validation)
• Dashboard CRUD operations
• Data file upload and management
• User isolation and authorization
• Error handling and edge cases

cd backend
npm test                 # Run all tests
npm run test:watch       # Watch mode
npm run test:coverage    # With coverage report

Test Coverage:

• 5+ test cases for auth routes
• 10+ test cases for dashboard operations
• 6+ test cases for data file handling
• User isolation verification
• Error scenarios (401, 404, 400, 500)

Deployment

Frontend

Platform: Vercel

• Build Command: npm run build
• Output Directory: dist
• Environment Variables: VITE_API_URL

Backend

Platform: Render

• Build Command: npm run build
• Start Command: npm start
• Environment Variables: MONGO_URI, JWT_SECRET, PORT

Key Technical Achievements

Data Processing

• Intelligent Type Inference: Automatically detects quantitative, categorical, and temporal fields
• Smart Aggregation: Optimized Map-based grouping with 5 aggregation types
• Performance Optimization: Top-10 limiting for both categories and series to prevent UI overflow
• Precision Control: All values rounded to 2 decimal places for clean display

User Experience

• Real-time Validation: Warns users about inappropriate field selections
• Smart Defaults: Charts automatically configure based on data types
• Responsive Design: 4-column grid that adapts to mobile, tablet, and desktop
• Drag-and-Drop: Smooth chart reordering with visual feedback
• PDF Export: Professional multi-page exports with maintained aspect ratios

Architecture

• Type Safety: Full TypeScript coverage in both frontend and backend
• Modular Design: Separate utilities for data processing, file parsing, and chart rendering
• Context Pattern: Efficient state management without prop drilling
• Test Coverage: Comprehensive Jest tests for all API endpoints

Screenshots

Builder - Dark Mode

Library - Dark Mode

Builder - Light Mode - Closed Sidebar

Login - Light Mode

Register - Light Mode

Future Enhancements

Planned Features

• Additional chart types (scatter plots, box plots, heatmaps)
• Advanced filtering options for data preprocessing
• Collaborative dashboards with sharing capabilities
• Real-time data updates via WebSocket integration
• Custom color schemes for charts
• Dashboard templates for common use cases
• Data transformation utilities (pivot, merge, filter)
• Export to multiple formats (PNG, SVG, CSV)

Performance Improvements

• Virtual scrolling for large datasets
• Server-side aggregation for datasets >10,000 rows
• Chart caching and memoization
• Lazy loading for dashboard components

Contributing

• Contributions are welcome! Please feel free to submit a Pull Request.

Author

Alex Grigore

• GitHub: @agworkgit
• Email: agworksbox@gmail.com

Licence

MIT License

Copyright (c) 2025 Alex Grigore agworksbox@gmail.com

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.