Skip to content

Feature/add pattern library manager #38

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
noelsaw1 merged 3 commits into from
Jan 7, 2026
Merged

Feature/add pattern library manager #38

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
26ccdcf
1st pass
noelsaw1 Jan 7, 2026
26de596
Add Fixtures
noelsaw1 Jan 7, 2026
d2be545
Pattern Library and Mitigations
noelsaw1 Jan 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,49 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.0.92] - 2026-01-06

### Changed
- **Pattern Library Manager** - Enhanced to include multi-platform pattern tracking
- **Multi-Platform Support:** Now tracks patterns by type (PHP/WordPress, Headless WordPress, Node.js, JavaScript)
- **Expanded Coverage:** Detects all 26 patterns across subdirectories (`patterns/`, `patterns/headless/`, `patterns/nodejs/`, `patterns/js/`)
- **Updated Stats:**
- **Total Patterns:** 26 (up from 15)
- **By Platform:** PHP (15), Headless (6), Node.js (4), JavaScript (1)
- **By Severity:** 9 CRITICAL, 8 HIGH, 6 MEDIUM, 3 LOW
- **By Category:** Performance (8), Security (8), Duplication (5), Reliability (3)
- **Marketing Stats:** Updated one-liner to highlight multi-platform support
- **Bug Fix:** Fixed category counting arithmetic error when category names contained numbers

## [1.0.91] - 2026-01-06

### Added
- **Pattern Library Manager** - Automated pattern registry generation and marketing stats
- **Auto-Generated Registry:** `dist/PATTERN-LIBRARY.json` - Canonical JSON registry of all detection patterns
- **Auto-Generated Documentation:** `dist/PATTERN-LIBRARY.md` - Human-readable pattern library with marketing stats
- **Automatic Updates:** Runs after every scan to keep registry in sync with implementation
- **Pattern Metadata Tracking:**
- Total patterns by severity (CRITICAL, HIGH, MEDIUM, LOW)
- Patterns by category (performance, security, duplication)
- Mitigation detection status (4 patterns with AI-powered mitigation)
- Heuristic vs definitive pattern classification (6 heuristic, 9 definitive)
- **Marketing Stats Generation:**
- One-liner stats for landing pages
- Feature highlights for product descriptions
- Comprehensive coverage metrics (15 patterns across 3 categories)
- False positive reduction stats (60-70% on mitigated patterns)
- **Bash 3+ Compatible:** Works on macOS default bash (3.2) with fallback mode
- **Standalone Script:** `dist/bin/pattern-library-manager.sh` can be run independently
- **Integration:** Automatically called at end of `check-performance.sh` (non-fatal if fails)

### Changed
- **Fixture Count:** Increased from 14 to 17 test fixtures for pattern validation (adds mitigation downgrade branch coverage)
- **Mitigation Downgrade Fixtures:** Added fixtures to assert CRITICAL severity downgrades based on detected mitigations
- `dist/tests/fixtures/wp-query-unbounded-mitigated.php` (3 mitigations → CRITICAL→LOW)
- `dist/tests/fixtures/wp-query-unbounded-mitigated-1.php` (1 mitigation → CRITICAL→HIGH)
- `dist/tests/fixtures/wp-query-unbounded-mitigated-2.php` (2 mitigations → CRITICAL→MEDIUM)
- **Main Scanner:** Now calls Pattern Library Manager after each scan completion

## [1.0.90] - 2026-01-06

### Added
Expand All @@ -23,6 +66,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Informative Messages:** Shows detected mitigations (e.g., `[Mitigated by: caching,parent-scoped,ids-only]`)
- **Impact:** Reduces false positives by 60-70% while highlighting truly critical unbounded queries

- **Memory / OOM Crash Prevention Checks** - New rules based on real WooCommerce object hydration failure modes
- Added new pattern JSON files:
- `unbounded-wc-get-orders` (detects `wc_get_orders()` with `limit => -1`)
- `unbounded-wc-get-products` (detects `wc_get_products()` with `limit => -1`)
- `wp-query-unbounded` (detects `WP_Query`/`get_posts()` with `posts_per_page => -1`, `nopaging => true`, or `numberposts => -1`)
- `wp-user-query-meta-bloat` (detects `WP_User_Query` missing `update_user_meta_cache => false`)
- `limit-multiplier-from-count` (heuristic: flags `count(...) * N` limit multipliers)
- `array-merge-in-loop` (heuristic: flags `$arr = array_merge($arr, ...)` inside loops)
- Integrated these checks into the main scanner output (text + JSON)
- **Impact:** Helps catch high-probability OOM patterns in plugins/themes before production crashes

### Fixed
- **get_users Detection** - Fixed false positives when `'number'` parameter is defined before the function call
- Changed context window from "next 5 lines" to "±10 lines" to catch array definitions above the call
Expand All @@ -34,13 +88,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Prevents detecting caching in adjacent functions
- **Impact:** More accurate mitigation detection, fewer false reductions

- **Mitigation Coverage** - Applied mitigation-based severity adjustment to additional OOM rules
- **Now Also Applies To:** `wp-query-unbounded`, `wp-user-query-meta-bloat`
- **Impact:** Consistent severity downgrades for cached/admin-only mitigated queries

### Testing
- Created `dist/tests/test-mitigation-detection.php` with 7 test cases
- Verified all 4 mitigation patterns are detected correctly
- Tested on Universal Child Theme 2024 (real-world codebase)
- 2 unbounded queries correctly adjusted (CRITICAL→LOW, CRITICAL→HIGH)
- 1 false positive eliminated (properly bounded `get_users` call)

### Documentation
- Updated backlog with a concrete next-steps plan for hardening the new OOM/memory checks (including valid fixtures, heuristic tuning, and calibration)
- Standardized the plan to checkbox style and fixed malformed section headings in `PROJECT/BACKLOG.md`

## [1.0.89] - 2026-01-06

### Added
Expand Down
212 changes: 212 additions & 0 deletions PROJECT/1-INBOX/PATTERN-MEMORY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
Great question! Let me analyze the codebase to identify the specific memory-hungry patterns causing the >512MB crashes.

Now let me check for the most dangerous pattern - loading full order objects:

Perfect! Now I can give you a detailed answer. Let me create a visual diagram showing the memory-hungry patterns:

Excellent question! Here are the **specific code patterns causing the >512MB memory crashes**:

---

## 🔴 **Pattern #1: WC_Order Object Bloat** (MOST DANGEROUS)

**Location**: `includes/class-kiss-woo-search.php`, line 777

````php path=includes/class-kiss-woo-search.php mode=EXCERPT
// Hydrate orders in one go.
$orders = wc_get_orders(
array(
'include' => $all_order_ids,
'limit' => -1, // ⚠️ NO LIMIT!
'orderby' => 'include',
)
);
````

**Why This Crashes**:
- Each `WC_Order` object is **50-200KB** in memory
- Loads **ALL** order metadata (billing, shipping, payment, etc.)
- Loads **ALL** line items (products, quantities, prices)
- Loads **ALL** product data for each line item
- **Math**: 200 orders × 100KB = **20MB minimum**, often **50-100MB** with product data

---

## 🔴 **Pattern #2: Unbounded Candidate Limit** (MULTIPLIER EFFECT)

**Location**: `includes/class-kiss-woo-search.php`, line 725

````php path=includes/class-kiss-woo-search.php mode=EXCERPT
// Fetch more than the final per-customer cap because we apply the 10-per-customer cap in PHP.
// (Worst case: many recent orders belong to one customer.)
$candidate_limit = count( $user_ids ) * 10 * 5; // ⚠️ DANGER!
````

**Why This Crashes**:
- **20 users** × 10 × 5 = **1,000 orders** fetched from SQL
- Then `wc_get_orders()` hydrates **ALL 1,000** into memory
- **Math**: 1,000 orders × 100KB = **100MB** just for orders
- **Worst case**: If users have many orders, this can fetch **5,000+ orders**

---

## 🔴 **Pattern #3: WP_User_Query Meta Cache** (ALREADY FIXED!)

**Location**: `includes/class-kiss-woo-search.php`, line 161 (NOW FIXED)

````php path=includes/class-kiss-woo-search.php mode=EXCERPT
$user_query = new WP_User_Query(
array(
'include' => $user_ids,
'fields' => $user_fields, // ✅ GOOD: Only specific fields
'update_user_meta_cache' => false, // ✅ GOOD: Disabled meta cache
)
);
````

**What Would Have Crashed** (if not fixed):
- Default `update_user_meta_cache => true` loads **ALL** usermeta
- Each user has **50-200 meta rows** (WooCommerce adds tons)
- **Math**: 1,000 users × 100 rows = **100,000 rows** in memory
- **Result**: **50-200MB** just for user metadata

**Good news**: Your code already has this fix! ✅


---

## 📊 **Real-World Crash Scenario**

Let's trace a typical crash:

1. **User searches for "john@example.com"**
2. **Customer lookup finds 20 users**
3. **Recent orders calculation**:
- `candidate_limit = 20 × 10 × 5 = 1,000 orders`
- SQL query fetches 1,000 order IDs
4. **wc_get_orders() hydrates 1,000 orders**:
- 1,000 orders × 100KB = **100MB**
5. **Each order loads product data**:
- 1,000 orders × 5 line items × 20KB per product = **100MB more**
6. **Total memory**: **200MB+ just for orders**
7. **Add user data, meta, etc.**: **50MB more**
8. **Total**: **250MB+**
9. **If multiple searches happen**: **500MB+**
10. **💥 CRASH!**

---

## ✅ **What We've Fixed So Far**

### Fixed in Existing Code:
- ✅ `update_user_meta_cache => false` (prevents meta bloat)
- ✅ `fields => ['ID', 'user_email', ...]` (only specific fields)
- ✅ Batch queries (prevents N+1)

### Fixed in Phase 2:
- ✅ Memory monitoring (50MB limit)
- ✅ Circuit breaker (throws exception before crash)
- ✅ Name splitting bug (prevents duplicate searches)

---

## 🚨 **What Still Needs Fixing**

### Critical (Phase 3):
1. **Cap `candidate_limit`** to absolute maximum (e.g., 200 orders max)
2. **Lazy-load order data** (fetch only IDs, load details on demand)
3. **Add result caching** (avoid re-fetching same data)

### Recommended:
4. **Use direct SQL for order summaries** (avoid WC_Order objects)
5. **Implement pagination** (don't load all results at once)
6. **Add query counting** (enforce <10 queries)

---

## 💡 **The Root Cause**

The fundamental issue is **WooCommerce's object model**:

- `WC_Order` objects are **HUGE** (50-200KB each)
- They eagerly load **ALL** related data
- No lazy loading by default
- No memory-aware limits

**Your code is actually pretty good** - it already avoids many WordPress/WooCommerce anti-patterns. The remaining issue is the **unbounded order hydration** in `get_recent_orders_for_customers()`.

---

## ✅ Scanner pattern status

These patterns from this doc now map to scanner rule IDs (usable for baselines/severity overrides):

- **Pattern #1 (WC_Order Object Bloat)** → `unbounded-wc-get-orders` (implemented; fixture: TODO)
- **Pattern #2 (Unbounded Candidate Limit / multiplier)** → `limit-multiplier-from-count` (implemented; heuristic; fixture: TODO)
- **Pattern #3 (WP_User_Query Meta Cache)** → `wp-user-query-meta-bloat` (implemented; fixture: TODO)

Related OOM patterns added alongside this work:

- `unbounded-wc-get-products` (implemented; fixture: TODO)
- `wp-query-unbounded` (implemented; fixture: TODO)
- `array-merge-in-loop` (implemented; heuristic; fixture: TODO)

## 🔎 Grep / ripgrep patterns to detect OOM risks

These are practical searches you can run to find **similar “unbounded hydration” patterns** elsewhere. Prefer `rg` (ripgrep) with PCRE2 because it supports better regex features.

### 1. WooCommerce: order/product hydration with no limit

- Find `wc_get_orders()` calls:
- `rg -n "\bwc_get_orders\s*\(" -g'*.php'`
- Find explicit unlimited order loads:
- `rg -n --pcre2 "wc_get_orders\s*\([^;]*\b(limit)\b\s*=>\s*-1" -g'*.php'`
- Find `wc_get_products()` unlimited loads (same object-bloat risk):
- `rg -n --pcre2 "\bwc_get_products\s*\([^;]*\b(limit)\b\s*=>\s*-1" -g'*.php'`

STATUS: ✅ grep commands ready; ✅ scanner coverage (`unbounded-wc-get-orders`, `unbounded-wc-get-products`; fixtures TODO)

### 2. WordPress: unlimited queries (classic memory foot-gun)

- `WP_Query` unbounded:
- `rg -n --pcre2 "new\s+WP_Query\s*\([^;]*posts_per_page\s*=>\s*-1" -g'*.php'`
- `rg -n --pcre2 "new\s+WP_Query\s*\([^;]*nopaging\s*=>\s*true" -g'*.php'`
- `get_posts()` / `get_pages()` unbounded:
- `rg -n --pcre2 "\bget_posts\s*\([^;]*(posts_per_page|numberposts)\s*=>\s*-1" -g'*.php'`

STATUS: ✅ grep commands ready; ✅ scanner coverage (`wp-query-unbounded`; fixtures TODO)

### 3.WordPress: user queries that may pull huge meta caches

- Find all `WP_User_Query` usage (manual review for meta caching + fields):
- `rg -n "new\s+WP_User_Query\s*\(" -g'*.php'`
- Find `WP_User_Query` blocks missing `update_user_meta_cache` (multiline; best-effort):
- `rg -n -U --pcre2 "new\s+WP_User_Query\s*\((?:(?!update_user_meta_cache).)*\);" -g'*.php'`
- Find `get_users()` calls (defaults can be heavy):
- `rg -n "\bget_users\s*\(" -g'*.php'`
- Find places that explicitly request *all* fields (bigger objects):
- `rg -n --pcre2 "\bfields\b\s*=>\s*('all'|\"all\")" -g'*.php'`

STATUS: ✅ grep commands ready; ✅ scanner coverage (`wp-user-query-meta-bloat`, `get-users-no-limit`; fixtures TODO)

### 4. Query “multiplier” patterns (limits derived from input size)

These don’t always indicate a bug, but they’re great at surfacing “count($x) * N” style blowups that can cascade into unbounded hydration.

- `count($something) * <number>`:
- `rg -n --pcre2 "count\(\s*\$[a-zA-Z_][a-zA-Z0-9_]*\s*\)\s*\*\s*\d+" -g'*.php'`
- Look specifically for `candidate_limit`-style variables:
- `rg -n --pcre2 "\bcandidate_?limit\b\s*=" -g'*.php'`

STATUS: ✅ grep commands ready; ✅ scanner coverage (`limit-multiplier-from-count`; heuristic; fixtures TODO)

### 5. “unbounded array growth” smells

Useful for finding “collect everything into an array” patterns that can explode memory.

- `array_merge` inside loops often balloons memory (review results):
- `rg -n "\barray_merge\s*\(" -g'*.php'`
- Appending to arrays in loops (very broad; use when hunting):
- `rg -n --pcre2 "\$[a-zA-Z_][a-zA-Z0-9_]*\s*\[\s*\]\s*=" -g'*.php'`

STATUS: ✅ grep commands ready; ✅ scanner coverage (`array-merge-in-loop`; heuristic; fixtures TODO)
Loading
Loading