Harbinger - We are the Harbinger of your destiny

Harbinger is the core Go-written API for NetSocial’s backend services. It manages key functionalities related to user-API interactions, algorithms, image detection, and more.

Requirements

• Go version 1.18.4 or higher.
• Properly configured config.yaml for optimal performance.
• Ensure the uuid-ossp extension is available on your database by running:

  CREATE EXTENSION "uuid-ossp";

  This extension is required for generating UUIDs in PostgreSQL.

Architecture

Harbinger consists of one main process and three assistant processes:

• Harbinger – Manages API interactions, database logging, and environment-related logging. Handles configuration generation, OpenAPI documentation, and orchestrates all assistant processes. It is responsible for routing, middleware orchestration, and the main HTTP server lifecycle.
• Gracey – Handles graceful service shutdowns, OS signal handling, and sequence-related tasks to ensure safe termination and cleanup. It ensures all database and cache connections are closed properly and logs shutdown events.
• Seedey – Manages database seeding and migrations. Now also supports database backup and restore operations, making it easier to maintain and recover your data. Seedey can automatically detect and apply new .sql files, and provides warnings for complex SQL statements. It parses SQL files to identify tables, enums, and indexes, and only applies migrations if the object does not already exist.
• Algor – Drives feed recommendations and basic post moderation. Integrates with AI models (like Ollama) for content filtering, spam detection, and recommendation logic. Algor can fetch the currently running AI model and version, and supports toggling features like image filtering and mass mention detection. It can be enabled or disabled via configuration.

---

Internal Functions & Features

• Request Lifecycle:
  Each HTTP request passes through a series of middleware layers for logging, error reporting (including Discord webhook integration), CORS, and request throttling. The database and cache connections are injected into the request context for easy access in handlers.
  Modular route registration is handled via the Chi router, with each feature (admin, user, coterie, etc.) registering its own endpoints.

• Automatic Configuration & OpenAPI Generation:
  On first run, Harbinger generates a config.yaml and an openapi.json reflecting the current API structure. The configuration file is validated for required fields and environment-specific checks.

• Dynamic Routing:
  Uses the Chi router for modular route registration, including admin, user, notification, coterie, authentication, stats, blogs, and partner endpoints. Each route group is isolated in its own Go file for maintainability.

• Middleware Stack:
  Includes recovery, real IP, path cleaning, logging, request ID, CORS, rate limiting, and Discord error reporting. Database and Redis connections are attached to the request context for use in handlers.

• Error Reporting:
  Integrates with Discord webhooks for real-time error notifications, with sensitive data redaction. All unhandled errors are logged and reported, including stack traces and request metadata.

• YAML/JSON Parsing:
  Custom parsers for configuration and OpenAPI output, supporting nested structures and type reflection. Configuration is loaded and validated at startup, with fatal errors for missing or invalid fields.

• Environment Checks:
  In production mode, Harbinger checks server hardware and environment compatibility before starting. This includes port, worker count, AES key length, and required environment variables.

• Database Object Detection & Seeding:
  Seedey parses SQL files to identify tables, enums, and indexes, and only applies migrations if the object does not already exist. It logs warnings for complex or ambiguous SQL statements.
  Backup and restore commands are available for disaster recovery, and all seed operations are idempotent.

• AI Integration & Moderation:
  Algor can fetch and log the running AI model and version, and exposes toggles for enabling/disabling moderation features. It can be used for content filtering, spam detection, and feed recommendations.

• Graceful Shutdown:
  Gracey listens for OS signals and ensures all HTTP, database, and cache connections are closed cleanly, logging the shutdown sequence.

---

Request Lifecycle

1. Incoming Request:
   The request enters the Chi router and passes through middleware for logging, CORS, error reporting, and context injection.
2. Context Injection:
   Database and Redis connections are attached to the request context for use in all handlers.
3. Routing:
   The request is routed to the appropriate handler based on the URL and HTTP method.
4. Handler Execution:
   Handlers access the database, perform business logic, and return JSON responses. Errors are reported to Discord if enabled.
5. Response:
   The response is serialized as JSON and sent to the client. If an error occurs, a structured error message is returned.

---

Setup

1. Compile Harbinger for your server and run it.
2. On the first run, it will generate a config.yaml and an openapi.json. Configure the configuration file; otherwise, it won't work.
3. Ensure all requirements are met. If your environment is set to "production," it will check if your server meets the hardware requirements to host Harbinger.
4. Ensure the database is seeded correctly. Although Seedey is good at its job, it might doze off sometimes, so double-check!
5. For backup and restore, use Seedey's new commands to create and load database backups as needed.

Contribution Guidelines

For the current period, we are only allowing bug and vulnerability reports and no direct contributions.

Contributors

Star History