ElastiFlow on Docker Swarm

This repository deploys ElastiFlow NetObserv Flow with a dedicated OpenSearch Dashboards service. It reuses the three-node OpenSearch 2.19.5 cluster from the separate Graylog deployment through the same encrypted external backend overlay. OpenSearch remains unpublished; only Dashboards is routed through the existing Traefik Swarm provider, protected by an external authentication middleware.

The collector is pinned to elastiflow/flow-collector:7.26.2. ElastiFlow's current Docker image and OpenSearch output reference should be reviewed before every upgrade. OpenSearch Dashboards stays pinned to the same 2.19.5 version as the shared OpenSearch nodes.

Architecture and prerequisites

• Linux Docker Engine in Swarm mode and access to a manager node.
• The sanitized Graylog OpenSearch stack deployed first, with services named graylog-search_opensearch-1 through -3 or .env adjusted to their actual fully-qualified Swarm service names.
• The encrypted OpenSearch backend overlay and Traefik overlay already created.
• Traefik v3 configured with the Swarm provider, HTTPS entrypoint, certificate resolver, access to the configured Traefik network, and a mandatory authentication middleware named by TRAEFIK_AUTH_MIDDLEWARE.
• DNS for ELASTIFLOW_DASHBOARDS_HOSTNAME pointing at Traefik.
• Bash and enough node capacity for the reservations in .env.
• UDP kernel tuning on the collector node. ElastiFlow publishes recommended light/moderate and heavy-ingest sysctls.

OpenSearch credentials are deliberately absent because the reused Graylog cluster has its security plugin disabled and is isolated on an encrypted overlay with no published REST or transport port. If that cluster is later secured, configure TLS, PKI/users/roles, the collector, Dashboards, and every Graylog client together; do not expose the current unauthenticated API during migration.

Communication flow and entrypoints

flowchart LR
  subgraph external["External clients and flow sources"]
    exporters["Flow exporters<br/>sFlow, NetFlow, and IPFIX"]
    users["Dashboard users"]
  end

  subgraph traefikStack["Existing Traefik stack"]
    https["HTTPS entrypoint<br/>TRAEFIK_HTTPS_ENTRYPOINT<br/>normally TCP 443"]
    router["HTTP router<br/>Host: ELASTIFLOW_DASHBOARDS_HOSTNAME"]
    auth["Authentication middleware<br/>TRAEFIK_AUTH_MIDDLEWARE"]
    certs["TLS certificate resolver<br/>TRAEFIK_CERTRESOLVER"]
  end

  subgraph elastiflowStack["ElastiFlow Swarm stack"]
    udp["Host-mode UDP entrypoint<br/>6343 default<br/>2055 and 4739 optional"]
    collector["ElastiFlow flow collector<br/>API and metrics: TCP 8080 internal"]
    dashboards["OpenSearch Dashboards<br/>stateless service in this stack<br/>TCP 5601 internal"]
    secretWrapper["Secret entrypoint wrapper"]
  end

  subgraph nodeStorage["Collector node storage"]
    maxmind["MaxMind databases<br/>GeoLite2-ASN.mmdb<br/>GeoLite2-City.mmdb optional"]
    state["Persistent collector state<br/>ELASTIFLOW_DATA_PATH"]
    license["Docker Swarm secret<br/>commercial license key optional"]
  end

  subgraph searchStack["Existing Graylog OpenSearch stack"]
    opensearch["Shared OpenSearch 2.19.5 cluster<br/>graylog-search_opensearch-1<br/>graylog-search_opensearch-2<br/>graylog-search_opensearch-3"]
    graylog["Graylog server<br/>separate stack and shared consumer"]
  end

  prometheus["Prometheus optional<br/>attached to backend overlay"]

  exporters -->|"UDP 6343 by default<br/>direct to the selected Swarm node"| udp
  udp -->|"Decoded flow packets"| collector

  users -->|"HTTPS"| https
  certs -.->|"TLS certificate"| https
  https --> router
  router --> auth
  auth -->|"Traefik overlay<br/>HTTP TCP 5601"| dashboards

  collector -->|"Encrypted backend overlay<br/>HTTP TCP 9200"| opensearch
  dashboards -->|"Encrypted backend overlay<br/>HTTP TCP 9200"| opensearch
  graylog -->|"Encrypted backend overlay<br/>HTTP TCP 9200"| opensearch
  prometheus -->|"Backend only<br/>HTTP TCP 8080 /metrics"| collector

  maxmind -->|"Read-only bind mount"| collector
  state <-->|"Read/write bind mount"| collector
  license -->|"/run/secrets/elastiflow_license_key"| secretWrapper
  secretWrapper -->|"EF_LICENSE_KEY when present"| collector

Loading

Communication path	Port or interface	Exposure
Flow exporters to collector	UDP 6343 by default; UDP 2055/4739 opt-in	Published in mode: host only on the selected collector node; does not pass through Traefik
Browser to Traefik	TRAEFIK_HTTPS_ENTRYPOINT, normally HTTPS/TCP 443	Public, with TLS and TRAEFIK_AUTH_MIDDLEWARE required
Traefik to OpenSearch Dashboards	HTTP/TCP 5601	Internal Traefik overlay; no host port is published
Collector to OpenSearch	HTTP/TCP 9200	Encrypted backend overlay only
OpenSearch Dashboards to OpenSearch	HTTP/TCP 9200	Encrypted backend overlay only
Prometheus to collector	HTTP/TCP 8080 at /metrics	Optional and backend-only; no host port is published
Collector to MaxMind	Read-only filesystem access	Bind mount from ELASTIFLOW_MAXMIND_PATH; no network connection
Collector to persistent state	Read/write filesystem access	Bind mount from ELASTIFLOW_DATA_PATH
Secret wrapper to collector	Process environment populated from /run/secrets	Optional commercial license key; never published or stored in .env

Configure nodes, storage, and networks

Copy the example and replace every environment-specific value:

cp .env.example .env
chmod 600 .env

Apply the configured labels. The collector label must select exactly the node whose UDP/6343 host port exporters target and where its persistent paths exist:

docker node update --label-add exampleorg.elastiflow.collector=true FLOW_NODE
docker node update --label-add exampleorg.elastiflow.dashboards=true DASHBOARDS_NODE
docker node update --label-add exampleorg.zone=zone-a FLOW_NODE
docker node update --label-add exampleorg.zone=zone-a DASHBOARDS_NODE

Create the paths on FLOW_NODE. The pinned image runs as UID/GID 1000:

sudo install -d -o 1000 -g 1000 -m 0750 /srv/exampleorg/elastiflow/flowcoll
sudo install -d -o 1000 -g 1000 -m 0750 /srv/exampleorg/elastiflow/maxmind

Download current GeoLite2-ASN.mmdb (and optionally GeoLite2-City.mmdb) from MaxMind under its current license into ELASTIFLOW_MAXMIND_PATH. Database files are runtime data and must never be committed. Set the matching enrichment flag to false when its file is absent.

Verify the existing networks rather than creating a second isolated backend:

docker network inspect exampleorg_graylog_backend >/dev/null
docker network inspect exampleorg_traefik_public >/dev/null

Validate and deploy

Validation never prints the rendered configuration:

./scripts/validate.sh
./scripts/deploy.sh config

Deploy the shared OpenSearch stack first and wait for all three nodes to be healthy. Swarm does not provide Compose-style depends_on; health checks and restart policies handle temporary startup failures.

cd ../graylog
./scripts/deploy.sh search
docker stack services graylog-search

cd ../elastiflow
./scripts/deploy.sh deploy
docker stack services elastiflow

The direct deployment equivalent is:

set -a; . ./.env; set +a
docker stack deploy --detach=true --with-registry-auth --compose-file stack.yml "$ELASTIFLOW_STACK_NAME"

If flow-inputs.yml or stack-license.yml is enabled, include each additional --compose-file in the same order as scripts/deploy.sh.

Dashboards and Traefik

The collector itself has no public web UI. Its API/Prometheus endpoint remains on the backend overlay and publishes no host port. The separate Dashboards service is the only HTTP service attached to Traefik; all labels are under deploy.labels, TLS and the configured auth middleware are mandatory, and port 5601 is not published directly.

Create the auth middleware in the Traefik project before deployment. Prefer an SSO/forward-auth middleware; secret-backed basic auth is an acceptable fallback. Keep identity-provider client secrets or password hashes in the Traefik stack's Docker secrets/dynamic configuration, never in this stack's labels. Because the shared OpenSearch security plugin is disabled, do not bypass this middleware or attach Dashboards to any other public network.

For OpenSearch 2.19, import ElastiFlow's 2.14.x and later ECS saved objects from the official OpenSearch dashboard download table. Export customized saved objects before overwriting them during an upgrade. The Dashboards payload limit is set to 8 MiB for these imports.

Prometheus can join the backend overlay and scrape elastiflow_flow-collector:8080/metrics; the endpoint must not be routed through the public Traefik router.

Flow input exposure

Only sFlow UDP/6343 is enabled and published by default, preserving the original project's intended input. Host publishing bypasses the Swarm routing mesh and requires exporters or an upstream load balancer to target the selected collector node directly. ElastiFlow documents source-address loss across a normal Docker bridge as a particular problem for NetFlow/IPFIX, so UDP flow traffic is not sent through Traefik. Traefik remains responsible for the HTTP Dashboards route.

To add NetFlow 2055 or IPFIX 4739, keep only the required entries:

cp flow-inputs.yml.example flow-inputs.yml
${EDITOR:-vi} flow-inputs.yml
${EDITOR:-vi} .env
./scripts/deploy.sh deploy

Add the same ports to ELASTIFLOW_FLOW_PORTS, for example 6343,2055,4739. Open only those UDP ports in host/network firewalls. Do not publish the collector API, OpenSearch 9200/9300, or Dashboards 5601.

Optional commercial license secret

Community deployments do not need a license-key secret. If a commercial key is required, create a versioned external Swarm secret interactively; input is hidden, the value is piped directly to Docker, and existing secrets are never overwritten:

./scripts/create-secrets.sh
cp stack-license.yml.example stack-license.yml
./scripts/deploy.sh deploy

The entrypoint reads /run/secrets/elastiflow_license_key and exports the value only inside the collector process. Rotate by creating a new versioned secret name, updating .env, redeploying, verifying, and only then removing the old secret. Never put a license key, OpenSearch password, API credential, SNMP community, or cloud token in .env or stack YAML.

Data lifecycle, backup, and restore

The output uses rollover, three primary shards (one per OpenSearch data node), one replica, best compression, and the elastiflow ISM policy. These template settings affect new indices only. Inspect the generated policy in OpenSearch, set retention from measured ingest rate and available storage, and test rollover before production. Do not assume a generated default retention period matches your recovery or compliance requirements.

A complete recovery set includes:

• OpenSearch snapshots containing ElastiFlow indices and cluster state;
• exported OpenSearch Dashboards saved objects;
• a backup of the collector's ELASTIFLOW_DATA_PATH while the task is stopped;
• .env settings and a secure inventory of external secret names outside Git;
• separately licensed MaxMind databases or a documented redownload procedure;
• Traefik ACME/certificates backed up and rotated from the Traefik project.

Test restore in an isolated Swarm. Restore OpenSearch before the collector, import Dashboards only after the indices/templates are available, and verify flow ingestion plus rollover before reopening exporters.

Upgrade and security notes

• Back up first and upgrade one component family at a time.
• Keep OpenSearch Dashboards on the exact OpenSearch version.
• Read ElastiFlow release notes for configuration changes, re-check the image entrypoint used by the secret wrapper, and import matching dashboard objects.
• Recalculate CPU from measured flow type/rate and memory from enrichment cache use; monitor queue saturation, UDP drops, output failures, and disk growth.
• Keep ELASTIFLOW_INDEX_TEMPLATE_OVERWRITE=false until a planned template migration has been reviewed. Changing it does not update existing indices.
• Review the redacted history and rotation requirements in SECURITY_AUDIT.md before publishing this repository.