Skip to content
BranchesTags

Repository files navigation

Test Data Factory

CI Pipeline TypeScript Node.js Docker PostgreSQL Redis Faker.js License: MIT

Schema-driven test data generation and environment seeding platform — realistic relationship-aware data, compliance masking, snapshot/reset lifecycle, bulk streaming, and REST API for on-demand provisioning.

Features

  • Schema-Driven Generation — Define data models in YAML with relationships auto-resolved (users → orders → payments)
  • Relationship Integrity — Foreign keys, referential integrity, cross-table consistency guaranteed
  • Compliance Masking — PII/PHI anonymization for HIPAA/GDPR (hash, mask, synthetic replace)
  • Snapshot & Reset — Save environment state, restore to known-good baseline between test runs
  • Bulk Streaming — Generate millions of records for performance testing with streaming output
  • Environment Seeding API — REST API to seed any environment on demand (dev, staging, CI)
  • Industry Templates — Pre-built schemas for FinTech, Healthcare, E-commerce, SaaS
  • Custom Generators — Extend with domain-specific data generators
  • CI/CD Integration — GitHub Action to seed before test runs, teardown after
  • Multi-Format Output — JSON, CSV, SQL INSERT, PostgreSQL COPY

Architecture

┌──────────────────────────────────────────────────────────────────┐
│                    Test Data Factory                              │
│                                                                  │
│  ┌─────────────────┐    ┌──────────────────┐                    │
│  │  YAML Schema    │───▶│  Schema Parser   │                    │
│  │  Definitions    │    │  & Validator     │                    │
│  └─────────────────┘    └────────┬─────────┘                    │
│                                  │                               │
│                       ┌──────────▼─────────┐                    │
│                       │  Dependency Graph  │                    │
│                       │  Resolver (DAG)    │                    │
│                       └──────────┬─────────┘                    │
│                                  │                               │
│           ┌──────────────────────┼──────────────────┐           │
│           ▼                      ▼                  ▼           │
│  ┌─────────────────┐  ┌─────────────────┐  ┌──────────────┐   │
│  │  Faker.js       │  │  Custom         │  │  Template    │   │
│  │  Generators     │  │  Generators     │  │  Generators  │   │
│  └────────┬────────┘  └────────┬────────┘  └──────┬───────┘   │
│           └─────────────────────┼──────────────────┘            │
│                                 ▼                               │
│                    ┌────────────────────────┐                    │
│                    │  Compliance Engine     │                    │
│                    │  (Mask/Hash/Anonymize) │                    │
│                    └────────────┬───────────┘                    │
│                                 │                               │
│              ┌──────────────────┼──────────────────┐            │
│              ▼                  ▼                   ▼           │
│  ┌────────────────┐  ┌─────────────────┐  ┌──────────────┐    │
│  │  JSON / CSV    │  │  SQL INSERT /   │  │  Direct DB   │    │
│  │  File Output   │  │  COPY Output    │  │  Seeding     │    │
│  └────────────────┘  └─────────────────┘  └──────────────┘    │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  REST API  │  Snapshot Manager  │  Redis Cache/Queue    │    │
│  └─────────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────────┘

Project Structure

test-data-factory/
├── .github/workflows/
│   ├── ci.yml                         # CI pipeline (lint, unit, integration)
│   └── example-seed.yml               # Example: seed before E2E tests
├── action.yml                         # Custom GitHub Action definition
├── src/
│   ├── schemas/
│   │   ├── parser.ts                  # YAML schema parser
│   │   ├── validator.ts               # Zod-based schema validation
│   │   ├── resolver.ts                # Dependency graph (DAG) resolver
│   │   └── types.ts                   # Schema type definitions
│   ├── generators/
│   │   ├── base.generator.ts          # Abstract generator interface
│   │   ├── faker.generator.ts         # Faker.js-based field generation
│   │   ├── custom.generator.ts        # User-defined generator support
│   │   ├── reference.generator.ts     # Foreign key reference resolver
│   │   └── index.ts                   # Generator registry
│   ├── compliance/
│   │   ├── masker.ts                  # PII/PHI field masking
│   │   ├── anonymizer.ts             # Data anonymization strategies
│   │   └── rules.ts                   # HIPAA/GDPR compliance rules
│   ├── output/
│   │   ├── json.writer.ts             # JSON file output
│   │   ├── csv.writer.ts              # CSV file output
│   │   ├── sql.writer.ts             # SQL INSERT/COPY output
│   │   └── stream.writer.ts          # Streaming bulk output
│   ├── engine/
│   │   ├── factory.ts                 # Core data factory orchestrator
│   │   ├── bulk.engine.ts             # Bulk generation with backpressure
│   │   └── snapshot.manager.ts        # Snapshot save/restore lifecycle
│   ├── api/
│   │   ├── server.ts                  # Express REST API
│   │   ├── routes/
│   │   │   ├── generate.routes.ts     # POST /generate
│   │   │   ├── seed.routes.ts         # POST /seed, DELETE /seed
│   │   │   ├── snapshot.routes.ts     # POST/GET/DELETE /snapshots
│   │   │   └── health.routes.ts       # GET /health
│   │   └── middleware/
│   │       ├── auth.middleware.ts      # API key auth
│   │       └── error.middleware.ts     # Error handler
│   ├── templates/
│   │   ├── ecommerce.yml              # E-commerce data model
│   │   ├── fintech.yml                # FinTech data model
│   │   ├── healthcare.yml             # Healthcare/HIPAA data model
│   │   └── saas.yml                   # SaaS multi-tenant data model
│   ├── storage/
│   │   ├── postgres.client.ts         # PostgreSQL connection
│   │   ├── redis.client.ts            # Redis caching & queue
│   │   └── migrations/
│   │       ├── 001_create_snapshots.sql
│   │       └── 002_create_jobs.sql
│   ├── config/
│   │   ├── loader.ts                  # Environment config
│   │   ├── logger.ts                  # Winston logger
│   │   └── defaults.ts               # Default settings
│   └── index.ts                       # CLI entry point
├── tests/
│   ├── unit/
│   │   ├── schemas/
│   │   │   ├── parser.test.ts
│   │   │   └── resolver.test.ts
│   │   ├── generators/
│   │   │   ├── faker.generator.test.ts
│   │   │   └── reference.generator.test.ts
│   │   ├── compliance/
│   │   │   └── masker.test.ts
│   │   └── engine/
│   │       └── factory.test.ts
│   ├── integration/
│   │   └── seed-flow.test.ts
│   └── fixtures/
│       ├── ecommerce-schema.yml
│       └── simple-schema.yml
├── docker-compose.yml
├── Dockerfile
├── package.json
├── tsconfig.json
├── tsconfig.eslint.json
├── jest.config.ts
├── .eslintrc.json
├── .prettierrc
├── .env.example
├── .gitignore
├── LICENSE
└── README.md

Getting Started

Prerequisites

  • Node.js >= 20.0.0
  • Docker & Docker Compose
  • PostgreSQL 15+ (or use Docker)
  • Redis 7+ (or use Docker)

Installation

# Clone the repository
git clone https://github.com/Djones-qa/test-data-factory.git
cd test-data-factory

# Install dependencies
npm install

# Copy environment template
cp .env.example .env

# Start infrastructure (PostgreSQL + Redis)
docker compose up -d postgres redis

# Run database migrations
npm run migrate

# Start the engine
npm run dev

Quick Start (Docker)

docker compose up -d
# API available at http://localhost:3001

Running Tests

# Run all tests
npm test

# Run unit tests only
npm run test:unit

# Run integration tests
npm run test:integration

# Run tests with coverage
npm run test:coverage

# Run linter
npm run lint

# Format code
npm run format

Schema Definition

Data models are defined in YAML with relationship references:

# schemas/ecommerce.yml
version: "1.0"
name: "E-commerce"

entities:
  users:
    count: 100
    fields:
      id:
        type: uuid
        primary: true
      email:
        type: faker
        method: internet.email
        unique: true
      name:
        type: faker
        method: person.fullName
      created_at:
        type: faker
        method: date.past

  products:
    count: 50
    fields:
      id:
        type: uuid
        primary: true
      name:
        type: faker
        method: commerce.productName
      price:
        type: faker
        method: commerce.price
        options: { min: 9.99, max: 999.99 }
      sku:
        type: faker
        method: string.alphanumeric
        options: { length: 10 }
        unique: true

  orders:
    count: 500
    fields:
      id:
        type: uuid
        primary: true
      user_id:
        type: reference
        entity: users
        field: id
      status:
        type: enum
        values: [pending, processing, shipped, delivered, cancelled]
      total:
        type: faker
        method: commerce.price
        options: { min: 19.99, max: 2999.99 }
      created_at:
        type: faker
        method: date.recent
        options: { days: 90 }

  order_items:
    count: 1500
    fields:
      id:
        type: uuid
        primary: true
      order_id:
        type: reference
        entity: orders
        field: id
      product_id:
        type: reference
        entity: products
        field: id
      quantity:
        type: faker
        method: number.int
        options: { min: 1, max: 5 }

compliance:
  mask:
    - entity: users
      field: email
      strategy: hash
    - entity: users
      field: name
      strategy: synthetic

Compliance Masking

Built-in strategies for PII/PHI protection:

Strategy Description Example
hash SHA-256 hash with salt john@email.com → a3f2...@masked.com
synthetic Replace with realistic fake John Smith → Maria Garcia
mask Partial character masking 555-1234 → 555-****
redact Full redaction SSN: 123-45-6789 → SSN: [REDACTED]
tokenize Reversible token replacement john@email.com → tok_8f3a2b

CLI Usage

# Generate data from a schema file
npx test-data-factory generate --schema schemas/ecommerce.yml --output ./data

# Generate with compliance masking
npx test-data-factory generate --schema schemas/ecommerce.yml --mask --output ./data

# Seed a database directly
npx test-data-factory seed --schema schemas/ecommerce.yml --db postgresql://localhost:5432/testdb

# Bulk generate (streaming, 1M records)
npx test-data-factory bulk --schema schemas/ecommerce.yml --entity orders --count 1000000 --output ./bulk

# Snapshot current database state
npx test-data-factory snapshot save --name "baseline" --db postgresql://localhost:5432/testdb

# Reset database to snapshot
npx test-data-factory snapshot restore --name "baseline" --db postgresql://localhost:5432/testdb

API Endpoints

Method Endpoint Description
POST /api/generate Generate data from schema
POST /api/seed Seed a target database
DELETE /api/seed Teardown seeded data
POST /api/snapshots Create a snapshot
GET /api/snapshots List snapshots
POST /api/snapshots/:name/restore Restore a snapshot
DELETE /api/snapshots/:name Delete a snapshot
GET /api/templates List industry templates
GET /api/health Health check

GitHub Action Usage

- name: Seed Test Data
  uses: Djones-qa/test-data-factory@v1
  with:
    schema: schemas/ecommerce.yml
    database-url: postgresql://postgres:postgres@localhost:5432/testdb
    count-multiplier: 2
    mask-pii: true

- name: Run E2E Tests
  run: npm run test:e2e

- name: Teardown Test Data
  uses: Djones-qa/test-data-factory@v1
  with:
    action: teardown
    database-url: postgresql://postgres:postgres@localhost:5432/testdb

Industry Templates

Template Entities Use Case
ecommerce.yml users, products, orders, payments, reviews Checkout flows, inventory
fintech.yml accounts, transactions, loans, payments, KYC Banking, payment processing
healthcare.yml patients, providers, appointments, records, prescriptions HIPAA-compliant testing
saas.yml tenants, users, subscriptions, invoices, features Multi-tenant apps

Configuration

Environment Variables

Variable Description Default
PORT API server port 3001
DATABASE_URL PostgreSQL connection string postgresql://localhost:5432/test_data
REDIS_URL Redis connection string redis://localhost:6379
API_KEY API authentication key —
DEFAULT_LOCALE Faker.js locale en
BATCH_SIZE Records per batch for bulk ops 1000
MASK_SALT Salt for hash masking —

CI/CD Pipeline

The GitHub Actions workflow runs on push to main/develop and pull requests:

  1. Lint & Type Check — ESLint + TypeScript compiler
  2. Unit Tests — Jest with coverage reporting
  3. Integration Tests — Schema parsing, generation, seeding flow
  4. Docker Build — Multi-stage production image verification

Author

Darrius Jones

License

MIT © 2026 Darrius Jones

See LICENSE for details.

About

Schema-driven test data generation and environment seeding platform — realistic relationship-aware data, compliance masking (HIPAA/GDPR), snapshot/reset lifecycle, bulk streaming, REST API. TypeScript/Node.js + PostgreSQL + Redis + Docker + GitHub Actions CI.

Topics

compliancedata-factorydockerfakergdprhipaamaskingnodejspostgresqlqa-engineeringredisseedingtest-automationtest-datatypescript

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages