NetFlow-Writer Error

[cherbold24]

Didn't know if I should comment on the last issue reported or start a new one.
I have everything installed and pretty much working but I am not getting NetFlow data into the Dashboard. When looking at the Docker logs for netflow-writer I get the following logs as well as Template Sequence numbering issues.

2021-02-13 16:30:14.792 | ERR | [EXPORTERIPHERE] Error decoding v9 flow. Contents: {'IN_BYTES': 7200, 'OUT_BYTES': 7200, 'IN_PKTS': 120, 'OUT_PKTS': 120, 'FIRST_SWITCHED': 3363530250, 'LAST_SWITCHED': 3363590270, 'INPUT_SNMP': 0, 'OUTPUT_SNMP': 7, 'ICMP_TYPE': 2048, 'PROTOCOL': 1, 'APPLICATION_TAG': 368934934542806941696, 'UNKNOWN_FIELD_TYPE': 2, 'FORWARDING_STATUS': 64, 'IPV4_SRC_ADDR': 'SOURCEIPHERE', 'IPV4_DST_ADDR': 'DESTINATIONIPHERE'}
Traceback (most recent call last):
File "./netflowwriter.py", line 217, in _get_data
KeyError: 'L4_DST_PORT'

I assume it is some odd field that Fortinet has on it's netflows that is unable to be parsed but I am not a programmer. More than happy to help get you the information to help with this error.

[grafolean]

Hi @cherbold24!

It looks like the device is not sending L4_DST_PORT and L4_SRC_PORT fields... I have made a release (v0.3.2) of a netflow bot which handles such omissions gracefully.

This sequence of commands should upgrade it:

$ docker-compose pull
$ docker-compose down
$ docker-compose up -d

Can you maybe give it a go and report back if it works?

[cherbold24]

@grafolean Thanks for the quick reply. I am no longer getting the errors above however I am still getting template issues logged and no data showing. Attached is the log output of "docker logs grafolean-netflow-writer".
Grafolean.txt

[grafolean]

Is this just at the start or after you leave the system running for some time?

There are two errors visible:

• the missing packets should only happen at the start (a single time) or when exporter is restarted
• (some of) the templates are missing (namely 1 and 256), leading to dropped packets because they can't be parsed correctly

Can you maybe check also the logs for grafolean-netflow-collector if there are any warnings there?

Alternatively, you can also set debugging output for netflow-writer by adding the DEBUG environment variable to docker-compose.yml:

    image: grafolean/grafolean-netflow-bot:v0.3.1-rc.3
    container_name: grafolean-netflow-collector
    depends_on:
      db:
        condition: service_healthy
    environment:
      - NAMED_PIPE_FILENAME=/shared-grafolean/netflow.pipe
      # listen on UDP port: (2055 by default)
      - NETFLOW_PORT=2055
      - DEBUG=true

Are you at liberty to share what kind of a device this is?

[cherbold24]

I am using a FortiGate 100D. These errors seem to be constant even if I leave it up. Nothing concerning was on the collector logs. With Debug the logs are the following - Collector at the top and Writer Bottom (more or less the same as before)
Grafolean.txt

[grafolean]

I'm sorry, I made a mistake - can you maybe enable debug or the netflow-writer?

  netflowwriter:
    # Reads netflow data from named pipe and writes it to DB.
    image: grafolean/grafolean-netflow-bot
    container_name: grafolean-netflow-writer
    depends_on:
      db:
        condition: service_healthy
    environment:
      - NAMED_PIPE_FILENAME=/shared-grafolean/netflow.pipe
      - DB_HOST=db
      - DB_DATABASE=${DB_NAME:-grafolean}
      - DB_USERNAME=${DB_USER:-admin}
      - DB_PASSWORD=${DB_PASS:-admin}
      - DEBUG=${DEBUG:-true}

(alternatively, you could also set DEBUG=true in .env)

Either the firewall is not sending all of the packets (or they are not reaching netflow-collector), or there is some error decoding them... If you are willing to help debug this issue further, I'd be happy to help.

[cherbold24]

I have no problem assisting debugging this. I can get some PCAPS if needed as well. Here are the logs for the writer. This project seems interesting. I have some other flow collectors running but they are very resource intensive. I need to learn to code a little better and I could throw all sorts of things into this. I like the API aspect and the ability to easily modularize things.
Grafolean.txt

[grafolean]

If you can get the PCAPs, that would be great! The logs don't reveal the cause of the problem, and by inspecting the code I can't see how that could happen (assuming the traffic is correct, of course). If you can provide PCAP, I can hopefully replay it and debug exactly what goes wrong.

Otherwise, thank you for the kind words! I tried to make everything as extensible as possible, and quite some time went into performance optimizations. If you see something missing (or find some other obstacle), I'd be happy to hear about it - this way i can help and at the same time make the project better. If you have trouble starting with customizations, ditto.

[grafolean]

Thank you for sending the PCAP files! This device seems to be sending "NetFlow Version 9 Options Template Fields" which are not supported (yet) by the library used to decode NetFlow. I will try to fix this (and upstream when I'm done).

[grafolean]

It turns out that support for Options Templates and Options Data Records is not mandatory, I just needed to provide better error handling:

Options data record - the options data record is a special type of data record (based on an options template) with a reserved template ID that provides information about the NetFlow process itself.

https://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9.html

It does look like an interesting option though.

I made a version now that properly handles this traffic and it seems to work - I get nice charts :-) (not posting here because of privacy and security concerns). It is still a prerelease though, because there are some performance improvements that I didn't upstream yet and need to include them now manually. Still, this should work easily, unless you have lots of traffic.

To use it:

• change the version of netflowwriter (or in all 3 places - netflowcollector, netflowwriter and netflowbot) to v0.3.3-rc.1, so that it looks like this:

    netflowwriter:
      # Reads netflow data from named pipe and writes it to DB.
      image: grafolean/grafolean-netflow-bot:v0.3.3-rc.1
  

Note that it might take a few minutes for charts to be visible.

[cherbold24]

Looks like it is starting to display data. I'll let it sit around and see how it goes. Thanks for looking into it and working on it.

[grafolean]

Glad to hear it works for you.

I will close this issue as is seems solved now, but feel free to open another one if there is anything else.

Enjoy!