LogPulse - Operating System Log Collector

Real-time, cross-platform log monitoring and analysis tool with comprehensive OS log collection

🚀 Features

• Cross-Platform OS Log Collection: Windows Event Logs, macOS Unified Logs, Linux journalctl & syslog
• Security Log Support: Dedicated security log collection and querying
• Multi-threaded Architecture: Optimized for i7-13400HX (12 cores, 16 threads) with 8 worker threads
• Intelligent Checkpointing: Incremental log tailing with position tracking
• SQLite Storage: Raw JDBC implementation (no ORM, no Spring, no GUI)
• Robust Parser: Normalizes logs from all platforms into unified schema
• Real-time CLI: Interactive command-line interface with 10 commands

📋 Quick Start

Prerequisites

• Java JDK 11+
• Administrator/root privileges (for full log access)

Build & Run

Windows:

cd Build
build.bat

cd ..\run
run.bat

macOS/Linux:

cd Build
./build.sh

cd ../run
./run.sh

🎯 CLI Commands

collect-system          # Collect all OS logs from detected sources
recent                  # Show last 20 log entries
filter <INFO|WARN|ERROR> # Filter logs by level
filter-security         # Show security logs only
stats                   # Display log statistics
sources                 # List detected log sources
search <keyword>        # Search logs by keyword
verbose                 # Toggle verbose mode
help                    # Show help
exit                    # Graceful shutdown

📊 Example Session

╔════════════════════════════════════════════════╗
║          LogPulse v1.0                         ║
║   Real-time Log Monitoring & Analysis          ║
╚════════════════════════════════════════════════╝

Running on: Windows 10 10.0

Database initialized successfully.
Initialized LogIngestionManager with 8 worker threads
Starting 8 log writer threads...
✓ Log ingestion system active

LogPulse> collect-system

╔════════════════════════════════════════════════╗
║       Starting System Log Collection           ║
╚════════════════════════════════════════════════╝
Detected OS: Windows

→ Collecting Windows Event Logs...
  • Collecting: System
    ✓ Collected 200 events from System
  • Collecting: Application
    ✓ Collected 200 events from Application
  • Collecting: Security
    ✓ Collected 200 events from Security

✓ Collection completed in 15.23 seconds

LogPulse> stats

╔════════════════════════════════════════════════╗
║           Log Statistics                       ║
╚════════════════════════════════════════════════╝

Total Logs:     1543

INFO:        1234  (80.0%)
WARN:         245  (15.9%)
ERROR:         64  (4.1%)

🏗️ Architecture

Component Overview

• OSLogCollector: Multi-platform log source detection and collection
• LogIngestionManager: Multi-threaded processing with BlockingQueue (10k capacity)
• DatabaseHandler: Raw JDBC SQLite operations
• LogParser: Multi-pattern parser supporting 5+ log formats
• ConsoleInterface: Interactive CLI with command routing

Threading Model

• Main Thread: CLI and user input
• 8 Worker Threads: Concurrent database writes
• Temporary Collection Threads: OS command execution

Database Schema

logs table:

CREATE TABLE logs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT NOT NULL,
    level TEXT,
    source TEXT,
    message TEXT NOT NULL
);

security_logs table:

CREATE TABLE security_logs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT NOT NULL,
    event_type TEXT,
    source TEXT,
    message TEXT NOT NULL
);

log_checkpoints table:

CREATE TABLE log_checkpoints (
    source TEXT PRIMARY KEY,
    last_position TEXT NOT NULL
);

🔍 Log Sources

Windows

• System, Application, Security Event Logs (wevtutil)
• PowerShell Operational logs
• C:\Windows\System32\LogFiles\
• C:\ProgramData\ directories
• Windows Error Reporting (WER)

macOS

• Unified logging (log show --style syslog)
• /var/log/*.log files
• Apple System Log (ASL) if available

Linux

• journalctl logs (systemd)
• /var/log/syslog, /var/log/auth.log, /var/log/kern.log
• Generic /var/log/*.log files

⚡ Performance

• Throughput: 1,000-5,000 events/second
• Collection Time: 30-120 seconds (full system scan)
• Memory Usage: 100-300 MB
• CPU Optimization: Scales with cores (8+ recommended)

📁 Project Structure

Logpulse/
├── bin/                    # Compiled classes
├── Build/                  # Build scripts
├── lib/                    # SQLite JDBC driver
├── run/                    # Run scripts
├── src/main/java/com/logpulse/
│   ├── Main.java
│   ├── collector/          # OS log collection
│   ├── console/            # CLI interface
│   ├── database/           # SQLite operations
│   ├── model/              # Data models
│   ├── parser/             # Log parsing
│   └── worker/             # Multi-threaded ingestion
└── LOGPULSE_GUIDE.md      # Comprehensive guide

📚 Documentation

See LOGPULSE_GUIDE.md for comprehensive documentation including:

• Detailed architecture
• Parser implementation
• Checkpoint system
• Security considerations
• Troubleshooting guide
• Development instructions

🔐 Security & Permissions

Windows: Run as Administrator for Security Event Log access
macOS: Use sudo for unified logging and /var/log access
Linux: Use sudo or root for journalctl and protected log files

All data stored locally in logpulse.db - no network transmission.

🛠️ Technology Stack

• Language: Java 11+
• Database: SQLite 3.51.1 (via JDBC)
• Architecture: Multi-threaded with ExecutorService
• Dependencies: SQLite JDBC driver only (no frameworks)

📝 License

See LICENSE file for details.

🤝 Contributing

This is a comprehensive system log collector designed for Windows, macOS, and Linux. Contributions welcome!

---

LogPulse - Comprehensive OS Log Collection & Analysis Tool