[Feature]  "Audit Log of Log Access" (Meta-logging)

## Feature Description
Track and log all user interactions with the logging system itself: who accessed which logs, when, what searches were performed, what data was downloaded, and what configuration changes were made. This "meta-logging" provides a complete audit trail for compliance, security incident response, and accountability.

## Problem/Use Case
**Current problem:**
- No visibility into who accessed sensitive logs and when
- Cannot prove compliance with data access policies
- Security incidents leave no trail of who investigated what
- No accountability for configuration changes (who modified retention policies?)
- Cannot detect unauthorized log access or insider threats
- Compliance frameworks (SOC 2, ISO 27001, HIPAA) require audit logs

**Real-world scenarios:**

**Scenario 1: Compliance audit**
```
Auditor asks: "Who accessed customer payment logs in Q4 2024?"

Without audit logs:
❌ "We don't track that"
→ Compliance violation, potential fine

With audit logs:
✓ Generate report showing:
  - User: john@company.com
  - Accessed: payment-logs source
  - Date: Dec 15, 2024, 14:32 UTC
  - Query: transaction_id:txn_12345
  - Action: Viewed 47 logs
  - IP: 192.168.1.100
```

**Scenario 2: Security incident investigation**
```
Breach detected: Customer data leaked

Questions:
- Who accessed customer data recently?
- What did they search for?
- Did they download anything?
- Were there unusual access patterns?

Without audit logs:
❌ No way to know
→ Cannot identify insider threat

With audit logs:
✓ Discover:
  - Employee X accessed 10,000 customer logs (unusual)
  - Downloaded export at 2am (red flag)
  - Used personal device (policy violation)
  - Searched for "email:*@company.com" (suspicious)
→ Incident contained, employee investigated
```

**Scenario 3: Accountability for changes**
```
Production alert stops firing, critical outage goes unnoticed

Question: "Who disabled the alert?"

Without audit logs:
❌ "Someone must have disabled it, but we don't know who"
→ No accountability, team tension

With audit logs:
✓ Alert was disabled by: sarah@company.com
  Date: Jan 10, 2025, 16:45 UTC
  Reason: "Testing, will re-enable"
  Never re-enabled
→ Accountability restored, process improved
```

## Proposed Solution

**Implement comprehensive audit logging for all Logtide interactions:**

**Events to track:**

**1. Log Access**
```
- User viewed logs from source X
- User searched for query Y
- User viewed specific log entry Z
- User downloaded log export
- User accessed logs via API
```

**2. Configuration Changes**
```
- User created/modified/deleted alert rule
- User changed retention policy
- User added/removed source
- User modified PII masking rules
- User changed organization settings
```

**3. User Management**
```
- User logged in/out
- User invited new member
- User changed permissions
- User deleted account
- API key created/revoked
```

**4. Data Modification**
```
- User manually deleted logs
- User triggered manual retention purge
- User exported data
```

**Audit log entry structure:**
```json
{
  "id": "audit_abc123",
  "timestamp": "2025-01-15T14:32:15.123Z",
  "event_type": "log_access",
  "event_action": "search",
  "user_id": "user_xyz789",
  "user_email": "john@company.com",
  "organization_id": "org_123",
  "ip_address": "192.168.1.100",
  "user_agent": "Mozilla/5.0...",
  "resource_type": "source",
  "resource_id": "source_production_api",
  "details": {
    "query": "level:error AND user_id:12345",
    "results_count": 47,
    "time_range": {
      "start": "2025-01-15T14:00:00Z",
      "end": "2025-01-15T14:30:00Z"
    }
  },
  "session_id": "sess_abc123"
}
```

**UI for audit logs:**
```
┌─────────────────────────────────────────────────────┐
│ Audit Log                                           │
├─────────────────────────────────────────────────────┤
│                                                     │
│ Filters:                                            │
│ User: [All Users ▼]  Event: [All Events ▼]        │
│ Date Range: [Last 7 days ▼]                        │
│                                                     │
│ ┌─────────────────────────────────────────────────┐ │
│ │ 2025-01-15 14:32:15                             │ │
│ │ john@company.com searched production-api logs   │ │
│ │ Query: level:error                              │ │
│ │ Results: 47 logs                                │ │
│ │ IP: 192.168.1.100                               │ │
│ │ [View Details]                                  │ │
│ └─────────────────────────────────────────────────┘ │
│                                                     │
│ ┌─────────────────────────────────────────────────┐ │
│ │ 2025-01-15 14:30:42                             │ │
│ │ sarah@company.com modified alert rule           │ │
│ │ Alert: "High Error Rate"                        │ │
│ │ Change: Threshold 100 → 200                     │ │
│ │ IP: 192.168.1.105                               │ │
│ │ [View Details] [Revert]                         │ │
│ └─────────────────────────────────────────────────┘ │
│                                                     │
│ ┌─────────────────────────────────────────────────┐ │
│ │ 2025-01-15 13:45:10                             │ │
│ │ admin@company.com downloaded log export         │ │
│ │ Source: payment-logs                            │ │
│ │ Records: 1,247                                  │ │
│ │ IP: 192.168.1.103                               │ │
│ │ ⚠️ Large export flagged for review              │ │
│ │ [View Details]                                  │ │
│ └─────────────────────────────────────────────────┘ │
│                                                     │
│ [Export Audit Log] [Generate Report]               │
└─────────────────────────────────────────────────────┘
```

## Alternatives Considered

1. **No audit logging**
   - ✗ Compliance violations
   - ✗ Security blind spots
   - ✗ No accountability

2. **External audit logging service**
   - ✗ Additional cost
   - ✗ Data leaves Logtide (privacy concern)
   - ✗ Integration complexity

3. **Application logs only (not dedicated audit)**
   - ✗ Mixed with regular logs (hard to filter)
   - ✗ No guaranteed retention (might be purged)
   - ✗ Not immutable (can be tampered with)

4. **Database triggers only**
   - ✗ Doesn't capture all events (API calls, views)
   - ✗ Hard to correlate with user sessions

**Chosen approach:** Dedicated audit log system with separate storage, guaranteed retention, and compliance-focused reporting

## Implementation Details (Optional)

**Technical implementation:**

**1. Database schema**
```sql
-- Dedicated audit log table (separate from regular logs)
CREATE TABLE audit_logs (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  event_type VARCHAR(50) NOT NULL,
  event_action VARCHAR(50) NOT NULL,
  user_id UUID REFERENCES users(id),
  user_email VARCHAR(255),
  organization_id UUID REFERENCES organizations(id),
  ip_address INET,
  user_agent TEXT,
  session_id VARCHAR(100),
  resource_type VARCHAR(50),
  resource_id VARCHAR(255),
  details JSONB,
  
  -- Compliance fields
  retention_guaranteed_until TIMESTAMPTZ, -- Never auto-purge before this
  exported BOOLEAN DEFAULT false,
  export_timestamp TIMESTAMPTZ
);

-- Indexes for common queries
CREATE INDEX idx_audit_logs_timestamp ON audit_logs (timestamp DESC);
CREATE INDEX idx_audit_logs_user ON audit_logs (user_id, timestamp DESC);
CREATE INDEX idx_audit_logs_organization ON audit_logs (organization_id, timestamp DESC);
CREATE INDEX idx_audit_logs_event_type ON audit_logs (event_type, timestamp DESC);
CREATE INDEX idx_audit_logs_resource ON audit_logs (resource_type, resource_id);

-- TimescaleDB for efficient time-series queries
SELECT create_hypertable('audit_logs', 'timestamp');

-- Make audit logs append-only (no updates/deletes)
CREATE OR REPLACE RULE audit_logs_no_update AS 
  ON UPDATE TO audit_logs DO INSTEAD NOTHING;

CREATE OR REPLACE RULE audit_logs_no_delete AS 
  ON DELETE TO audit_logs DO INSTEAD NOTHING;
```

**2. Audit logging middleware**
```typescript
class AuditLogger {
  async logEvent(event: AuditEvent): Promise<void> {
    // Never block the main request
    setImmediate(async () => {
      try {
        await db.query(`
          INSERT INTO audit_logs (
            event_type,
            event_action,
            user_id,
            user_email,
            organization_id,
            ip_address,
            user_agent,
            session_id,
            resource_type,
            resource_id,
            details,
            retention_guaranteed_until
          ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
        `, [
          event.type,
          event.action,
          event.user?.id,
          event.user?.email,
          event.organization?.id,
          event.ip,
          event.userAgent,
          event.sessionId,
          event.resource?.type,
          event.resource?.id,
          JSON.stringify(event.details),
          this.calculateRetentionDate(event),
        ]);
      } catch (error) {
        // Audit logging failure should not break the app
        console.error('Failed to write audit log:', error);
      }
    });
  }
  
  private calculateRetentionDate(event: AuditEvent): Date {
    // Compliance-driven retention
    const retentionDays = {
      'log_access': 365,        // 1 year
      'config_change': 2555,    // 7 years (SOX requirement)
      'user_management': 2555,  // 7 years
      'data_modification': 2555, // 7 years
      'login': 90,              // 90 days
    }[event.type] || 365;
    
    const date = new Date();
    date.setDate(date.getDate() + retentionDays);
    return date;
  }
}

// Global audit logger instance
const auditLogger = new AuditLogger();

// Middleware for HTTP requests
app.use(async (req, res, next) => {
  // Capture request details
  res.on('finish', () => {
    auditLogger.logEvent({
      type: 'api_call',
      action: req.method,
      user: req.user,
      organization: req.organization,
      ip: req.ip,
      userAgent: req.headers['user-agent'],
      sessionId: req.session?.id,
      resource: {
        type: 'api_endpoint',
        id: req.path,
      },
      details: {
        method: req.method,
        path: req.path,
        query: req.query,
        statusCode: res.statusCode,
      },
    });
  });
  
  next();
});
```

**3. Event tracking examples**

**Log search tracking:**
```typescript
async function searchLogs(query: SearchQuery, user: User): Promise<SearchResult> {
  const results = await executeSearch(query);
  
  // Log the search
  await auditLogger.logEvent({
    type: 'log_access',
    action: 'search',
    user: user,
    organization: user.organization,
    resource: {
      type: 'source',
      id: query.sourceId,
    },
    details: {
      query: query.query,
      timeRange: query.timeRange,
      resultsCount: results.total,
    },
  });
  
  return results;
}
```

**Configuration change tracking:**
```typescript
async function updateAlertRule(ruleId: string, updates: Partial<AlertRule>, user: User) {
  const oldRule = await getAlertRule(ruleId);
  const newRule = await db.updateAlertRule(ruleId, updates);
  
  // Log the change
  await auditLogger.logEvent({
    type: 'config_change',
    action: 'update_alert',
    user: user,
    organization: user.organization,
    resource: {
      type: 'alert_rule',
      id: ruleId,
    },
    details: {
      changes: diffObjects(oldRule, newRule),
      oldValues: oldRule,
      newValues: newRule,
    },
  });
  
  return newRule;
}
```

**Data export tracking:**
```typescript
async function exportLogs(exportRequest: ExportRequest, user: User): Promise<string> {
  const exportPath = await generateExport(exportRequest);
  
  // Log the export
  await auditLogger.logEvent({
    type: 'data_modification',
    action: 'export',
    user: user,
    organization: user.organization,
    resource: {
      type: 'source',
      id: exportRequest.sourceId,
    },
    details: {
      recordCount: exportRequest.recordCount,
      format: exportRequest.format,
      timeRange: exportRequest.timeRange,
      exportPath: exportPath,
    },
  });
  
  // Alert if large export
  if (exportRequest.recordCount > 10000) {
    await notifyAdmins({
      type: 'large_export',
      user: user.email,
      count: exportRequest.recordCount,
    });
  }
  
  return exportPath;
}
```

**4. Audit log API**
```typescript
// Read-only API for audit logs
app.get('/api/audit-logs', requireRole('admin'), async (req, res) => {
  const { userId, eventType, startDate, endDate, limit = 100 } = req.query;
  
  const auditLogs = await db.query(`
    SELECT *
    FROM audit_logs
    WHERE organization_id = $1
      AND ($2::uuid IS NULL OR user_id = $2)
      AND ($3::varchar IS NULL OR event_type = $3)
      AND ($4::timestamptz IS NULL OR timestamp >= $4)
      AND ($5::timestamptz IS NULL OR timestamp <= $5)
    ORDER BY timestamp DESC
    LIMIT $6
  `, [
    req.organization.id,
    userId,
    eventType,
    startDate,
    endDate,
    limit,
  ]);
  
  res.json(auditLogs);
});

// Export audit logs for compliance
app.post('/api/audit-logs/export', requireRole('admin'), async (req, res) => {
  const { format, startDate, endDate } = req.body;
  
  const exportPath = await exportAuditLogs({
    organizationId: req.organization.id,
    format,
    startDate,
    endDate,
  });
  
  // Log the audit log export (meta-meta logging!)
  await auditLogger.logEvent({
    type: 'data_modification',
    action: 'export_audit_logs',
    user: req.user,
    organization: req.organization,
    details: {
      format,
      dateRange: { start: startDate, end: endDate },
      exportPath,
    },
  });
  
  res.json({ exportPath });
});
```

**5. Anomaly detection on audit logs**
```typescript
// Detect suspicious patterns in audit logs
class AuditAnomalyDetector {
  async detectAnomalies(): Promise<void> {
    // Check for unusual access patterns
    await this.detectUnusualVolume();
    await this.detectUnusualTime();
    await this.detectSuspiciousQueries();
    await this.detectLargeExports();
  }
  
  async detectUnusualVolume(): Promise<void> {
    // Find users accessing far more logs than usual
    const result = await db.query(`
      WITH user_baselines AS (
        SELECT 
          user_id,
          AVG(daily_count) as avg_daily_access
        FROM (
          SELECT 
            user_id,
            DATE(timestamp) as date,
            COUNT(*) as daily_count
          FROM audit_logs
          WHERE event_type = 'log_access'
            AND timestamp > NOW() - INTERVAL '30 days'
          GROUP BY user_id, DATE(timestamp)
        ) daily
        GROUP BY user_id
      )
      SELECT 
        al.user_id,
        al.user_email,
        COUNT(*) as today_count,
        ub.avg_daily_access
      FROM audit_logs al
      JOIN user_baselines ub ON al.user_id = ub.user_id
      WHERE al.event_type = 'log_access'
        AND al.timestamp > CURRENT_DATE
      GROUP BY al.user_id, al.user_email, ub.avg_daily_access
      HAVING COUNT(*) > ub.avg_daily_access * 5
    `);
    
    for (const anomaly of result) {
      await notifySecurityTeam({
        type: 'unusual_access_volume',
        user: anomaly.user_email,
        todayCount: anomaly.today_count,
        avgCount: anomaly.avg_daily_access,
      });
    }
  }
  
  async detectUnusualTime(): Promise<void> {
    // Access outside business hours
    const result = await db.query(`
      SELECT 
        user_email,
        COUNT(*) as access_count,
        ARRAY_AGG(DISTINCT timestamp::time) as access_times
      FROM audit_logs
      WHERE event_type = 'log_access'
        AND timestamp > NOW() - INTERVAL '7 days'
        AND (EXTRACT(hour FROM timestamp) < 6 OR EXTRACT(hour FROM timestamp) > 22)
      GROUP BY user_email
      HAVING COUNT(*) > 10
    `);
    
    for (const anomaly of result) {
      await notifySecurityTeam({
        type: 'unusual_access_time',
        user: anomaly.user_email,
        count: anomaly.access_count,
        times: anomaly.access_times,
      });
    }
  }
}
```

## Priority
- [ ] Critical - Blocking my usage of LogTide
- [ ] High - Would significantly improve my workflow
- [ ] Medium - Nice to have
- [x] Low - Minor enhancement

**Rationale:** Important for **compliance and security**, but not blocking for most users. Higher priority for regulated industries (finance, healthcare) and enterprises with strict audit requirements.

## Target Users
- [x] DevOps Engineers (incident investigation)
- [ ] Developers
- [x] Security/SIEM Users (primary: security audits)
- [x] System Administrators (compliance)
- [ ] All Users

**Primary benefit:** Security teams, compliance officers, and organizations requiring SOC 2, ISO 27001, or HIPAA compliance.

## Additional Context

**Compliance frameworks requiring audit logs:**

```
SOC 2 (Trust Services Criteria):
→ CC6.3: "Logs system activities"
→ CC7.2: "Monitors system components"

ISO 27001:
→ A.12.4.1: "Event logging"
→ A.12.4.3: "Administrator and operator logs"

HIPAA:
→ §164.308(a)(1)(ii)(D): "Information system activity review"
→ §164.312(b): "Audit controls"

PCI-DSS:
→ Requirement 10: "Track and monitor all access to network resources and cardholder data"

GDPR:
→ Article 30: "Records of processing activities"
```

**Marketing angle:**

> "Complete audit trail included. Logtide tracks who accessed what, when, and why – meeting SOC 2 and ISO 27001 requirements out of the box."

**Real compliance scenario:**

**SOC 2 audit:**
```
Auditor: "Show me who accessed customer data in Q4 2024"

With audit logs:
→ Generate report in 2 minutes
→ Export to CSV
→ Pass audit requirement

Without audit logs:
→ "We don't have that information"
→ Fail audit requirement
→ No SOC 2 certification
→ Lost enterprise deals
```

**Future enhancements:**
- Real-time anomaly alerting (suspicious access patterns)
- Integration with SIEM tools (forward audit logs)
- Blockchain/tamper-proof audit logs (cryptographic signing)
- Automated compliance reports (SOC 2, ISO 27001 templates)
- Data lineage tracking (who created/modified/deleted what)

## Contribution
- [ ] I would like to work on implementing this feature



Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] "Audit Log of Log Access" (Meta-logging) #94

Feature Description

Problem/Use Case

Proposed Solution

Alternatives Considered

Implementation Details (Optional)

Priority

Target Users

Additional Context

Contribution

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature] "Audit Log of Log Access" (Meta-logging) #94

Description

Feature Description

Problem/Use Case

Proposed Solution

Alternatives Considered

Implementation Details (Optional)

Priority

Target Users

Additional Context

Contribution

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions