🛡 NetSentinel

Secure Network Intelligence Platform 🌐

🔍 Overview

NetSentinel is a modular platform and microservice that delivers IP geolocation, reputation scoring and lightweight threat intelligence. This repository is the central overview for the project: it explains how the microservice works, how infrastructure is modeled with modular Terraform, how monitoring is assembled, and how DevSecOps practices are enforced across build and deploy pipelines. Use this file as the single source of truth when understanding the project at a glance.

🕵️ What NetSentinel does

• Accepts HTTP requests to evaluate an IP address (geolocation + reputation).
• Returns structured JSON ready for logs and telemetry pipelines.
• Exposes Prometheus metrics for request counts and latency.
• Designed to run in Kubernetes with health checks, resource requests/limits, HPA and PDB for safe operations.
• Integrates with Loki/Grafana for log visibility and Prometheus for metrics/alerts.

🧠 Microservice: core behavior

• Built with FastAPI (async) to keep I/O non-blocking and scale efficiently.
• Key endpoints:
  • GET /check_ip?ip=<ip> — orchestration endpoint that:
    1. Increments Prometheus counter per endpoint.
    2. Calls internal client to fetch geolocation (IP-API) and reputation (AbuseIPDB when configured).
    3. Returns { ip, country, asn, reputation, blacklisted }.
    4. Always records latency in a histogram (even on error).
  • GET /health — readiness/liveness probe target ({"status":"ok"}).
  • GET /metrics — Prometheus exposition endpoint.
• Logging: structured JSON lines to stdout to allow Promtail/Fluentd ingestion and field extraction.
• Failure handling:
  • Geolocation failures degrade gracefully to Unknown values.
  • Reputation uses AbuseIPDB when ABUSEIPDB_KEY is provided; otherwise a safe deterministic/random fallback is used for dev testing.
  • External call timeouts and errors are logged and surfaced as 500 responses when necessary.

🛠️ Source layout

• src/ — microservice code:
  • app.py — FastAPI app, metrics, endpoints.
  • ip_client.py — async HTTP client to IP-API and AbuseIPDB, result normalization.
  • logger.py — JSON stdout logger.
• k8s/ — Kubernetes manifests for the app (Deployment, Service, HPA, PDB, Namespace).
• terraform/ — modular Terraform stacks (modules for Netsentinel and monitoring, envs for local/dev/prod).
• monitoring/ — Helm values and ServiceMonitor/RBAC snippets for Loki/Prometheus/Grafana.
• Dockerfile, requirements.txt, CI scripts and pipeline definitions.

🌱 Terraform modular infrastructure summary

• Modular approach: separate modules for application (modules/netsentinel) and observability (modules/monitoring).
• envs/<env> folders contain environment-specific variables and backend config (local backend for single-developer, remote backends for teams).
• Modules encapsulate Kubernetes resource creation:
  • deployment.tf controls container image, probes, env var injection from Secrets, resource requests/limits.
  • service.tf exposes ClusterIP (internal) or other service types depending on env.
  • hpa.tf uses CPU utilization targets (metrics-server required).
  • pdb.tf ensures minAvailable replicas for safe upgrades.
• Recommended workflow:
  1. Build & verify Docker image locally (or via CI).
  2. Run terraform init in the chosen env.
  3. terraform plan -var-file=terraform.tfvars then terraform apply.
  4. Validate in-cluster resources (kubectl get pods,svc,hpa).

📊 Monitoring & observability

• Prometheus: scrapes /metrics and any ServiceMonitor-managed targets. Use kube-prometheus-stack or operator to manage Prometheus instances.
• Loki + Promtail: collect application logs (structured JSON) from pods; Promtail processes pipeline stages to extract fields (level, request_id, client_ip, method, path, status_code).
• Grafana: dashboards for metrics and logs; data sources configured for Prometheus and Loki. Use Secrets for admin credentials and enable TLS/OAuth in production.
• ServiceMonitor and RBAC resources are provided to let Prometheus automatically discover and scrape targets in selected namespaces.

🔗 CI/CD and DevSecOps posture

NetSentinel enforces security and compliance gates early in the pipeline:

• Image security: Docker images built in CI are scanned with Trivy. The pipeline fails on HIGH/CRITICAL findings.
• IaC security: Terraform is validated and scanned with Checkov to detect misconfigurations and insecure defaults.
• Secrets management: CI uses GitHub Actions Secrets (or equivalent); runtime uses Kubernetes Secrets or an external secret manager (Vault) for production.
• Fail-fast policy: any failing security check prevents promotion and push-to-registry.
• Best-practice checks: ensure resource requests exist (HPA compatibility), enforce minimal privileges, and avoid embedding secrets in code or tfvars.

🏃 Local development & quick verification

1. Start local cluster (recommended: kind or minikube).
2. Build image locally and load into cluster (for kind: kind load docker-image netsentinel:local).
3. Apply Kubernetes manifests or use Terraform to provision development namespace.
4. Port-forward the service to test:
   • kubectl -n dev port-forward svc/netsentinel 8000:80
   • curl http://127.0.0.1:8000/health
   • curl "http://127.0.0.1:8000/check_ip?ip=1.1.1.1"
   • curl http://127.0.0.1:8000/metrics | head -n 40
5. Verify logs appear in Loki/Grafana if monitoring is deployed.

✅ Example response (IP = 1.1.1.1)

{
  "ip": "1.1.1.1",
  "country": "United States",
  "asn": "AS13335 Cloudflare, Inc.",
  "reputation": 0,
  "blacklisted": false
}

🔒 Security recommendations (must-follow)

• Do not commit secrets into the repository. Use kubectl create secret for local testing and an external secret manager for production.

• Rotate API keys and credentials regularly and revoke unused keys.

• Use TLS and authentication (OAuth/LDAP) for Grafana in production.

• Enable rate limiting and backoff for external API calls (IP-API/AbuseIPDB).

• Add caching for frequent IP lookups to reduce external API usage and lower latency.

❌ Troubleshooting quick hits

• If HPA is not scaling: verify metrics-server is running and kubectl top pods returns metrics.

• If Prometheus is not scraping: confirm ServiceMonitor exists and the Prometheus operator watches the monitoring namespace.

• If Grafana errors on datasource provisioning: ensure only one datasource is marked isDefault: true.

• If Loki/Promtail don't show logs: verify Promtail DaemonSet mounts /var/log/containers and application logs are valid JSON.

License

This project is licensed under CC BY-NC-ND (custom). See LICENSE.md for details.