Unexpected Association Rule Mining (Prototype)

A portfolio-ready, small-scale reimplementation of the 2024 project "Unexpected Association Rule Mining using Spark-based model" (NEclatClosed + unexpected-rule detection). The original Spark codebase cannot be made public, so this repository is an independent prototype that reproduces the paper's core methodology on a runnable, easy-to-demo scale.

Original paper focus: mining frequent closed itemsets with an Eclat-style vertical algorithm (NEclatClosed) on Apache Spark, then deriving unexpected association rules — rules that contradict known/expected patterns. This prototype keeps that algorithmic flow in pure Python.

---

What This Demonstrates

Paper concept	Implementation here
Vertical data layout (tidsets)	build_tidsets maps each item to a set of transaction IDs
Eclat-style DFS frequent mining	eclat_mine recursive tidset intersection
Closed pattern mining (NEclatClosed)	extract_closed_itemsets keeps itemsets with no equal-support superset
Association rules (support/confidence/lift)	generate_rules
Unexpected rule detection	detect_unexpected_rules (antecedent cosine similarity + contradictory consequent)
NEclat vs PFP comparison	closed rules vs all-frequent (PFP-style) baseline, with rule-reduction %

Definition of an "unexpected" rule (from the paper)

A mined rule X -> Y is unexpected relative to a known rule A -> B when:

1. Antecedents X and A are statistically similar (cosine similarity above a threshold), and
2. The consequents Y and B are contradictory (e.g. outcome=safe vs outcome=risk).

---

Project Structure

data/
  sample_transactions.csv     # Traffic-accident-style dataset
  known_rules.json            # Expected rules for the traffic domain
  medical_transactions.csv    # Medical-diagnosis dataset (denser)
  medical_known_rules.json    # Expected rules for the medical domain
src/
  unexpected_mining_prototype.py   # Core pipeline + CLI
  compare_and_visualize.py         # Multi-dataset comparison + charts
charts/                            # Generated PNGs (portfolio screenshots)

---

Quick Start

Run a single dataset:

python3 src/unexpected_mining_prototype.py \
  --data data/sample_transactions.csv \
  --known-rules data/known_rules.json \
  --min-support 0.2 \
  --min-confidence 0.6 \
  --min-lift 1.0 \
  --similarity-threshold 0.6

Run the full comparison across both datasets and generate charts:

pip install -r requirements.txt
python3 src/compare_and_visualize.py

---

Example Results

Comparison table (NEclatClosed vs PFP-style baseline):

Dataset               Txns   Freq  Closed  Baseline  ClosedR  Reduce  Unexp  Time(ms)
Traffic Accidents       20     70      26       131       54    59%      1     ~1.0
Medical Diagnosis       24     61      12       105       28    73%      1     ~0.7

Key takeaways mirrored from the paper:

• Closed-pattern mining removes redundancy: NEclatClosed produces far fewer rules than the PFP-style baseline (about 60-73% fewer here) while retaining the meaningful associations.

• Unexpected rules surface hidden insights, e.g. in the medical dataset:

  high fever & cough & sneeze -> allergy
    conflicts with expected:  high fever & cough & fatigue -> flu
    (antecedent similarity 0.67, confidence 1.00)
  

  The expected pattern says high fever + cough implies flu, but the prototype flags a similar context that instead leads to allergy.

Generated Charts

• charts/rule_counts_comparison.png — rule count: NEclatClosed vs baseline
• charts/top_unexpected_rules.png — top unexpected rules by confidence
• charts/confidence_distribution.png — confidence distribution of closed rules

---

Input Format

Transactions CSV:

• tid: unique transaction id
• items: semicolon-separated items (excluding the outcome)
• outcome: target outcome label

The loader appends outcome=<value> as an item inside each transaction.

Known-rules JSON:

[
  {
    "antecedent": ["weather=sunny", "road=dry"],
    "consequent": "outcome=safe",
    "source": "domain_knowledge"
  }
]

---

Why This Is A Prototype

• The original project targeted Spark-based scalability on large datasets (chess and Belgian traffic-accident benchmarks).
• This portfolio version is intentionally small, dependency-light, and runs locally so it can be demoed in seconds.
• The algorithmic flow stays faithful to the paper's methodology.

Possible Extensions

• Port the miner to PySpark RDD/DataFrame for large-scale data.
• Use GraphX to distribute set-enumeration-tree traversal (paper's future work).
• Add the original chess/accidents benchmark datasets in simplified form.