Skip to content

Collgamer0008/NexusFlowMeter

BranchesTags

Repository files navigation

NexusFlowMeter

PyPI version Python Version License: MIT Downloads

NexusFlowMeter is a high-performance network flow analysis tool that converts packet capture (PCAP) files into comprehensive flow-based insights. Unlike traditional packet-by-packet analysis, NexusFlowMeter aggregates network traffic into flows, providing deeper understanding of network behavior, communication patterns, and traffic characteristics.

  Contributions are welcome! Please feel free to submit pull requests, report bugs, or suggest features.

Architecture

graph LR
    %% Input Layer
    subgraph "Input Layer"
        A[Single PCAP File] 
        B[Directory of PCAPs]
        C[Large PCAPs >1GB]
    end

    %% Input Processing
    subgraph "Input Processing"
        D[File Size Check]
        E[PCAP Discovery]
        F[File Splitting]
        G[Chunk Generation]
    end

    %% Core Engine
    subgraph "Core Engine"
        H[PCAPToFlowConverter]
        I[Streaming Reader: PcapReader]
        J[Non-Streaming Reader: rdpcap]
        K[Flow Key Extractor]
        L[Packet Parser]
    end

    %% Flow Management
    subgraph "Flow Management"
        M[FlowKey Generator]
        N[NetworkFlow Objects]
        O[Flow Statistics]
        P[TCP State Analysis]
        Q[Timing Analysis]
    end

    %% Parallel Processing
    subgraph "Parallel Processing"
        R[ProcessPoolExecutor]
        S[Worker 1]
        T[Worker 2]
        U[Worker N]
        V[Results Merger]
    end

    %% Protocol Analysis
    subgraph "Protocol Analysis"
        W[TCP Handler]
        X[UDP Handler]
        Y[ICMP Handler]
        Z[ARP Handler]
        AA[DNS Handler]
    end

    %% Data Aggregation
    subgraph "Data Aggregation"
        BB[Flow Merger]
        CC[Statistics Calculator]
        DD[DataFrame Converter]
        EE[Protocol Filter]
        FF[Flow Limiter]
    end

    %% Output Processing
    subgraph "Output Processing"
        GG[CSV Exporter]
        HH[JSON Exporter]
        II[Excel Exporter]
        JJ[Report Generator]
        KK[Summary Statistics]
    end

    %% Batch Modes
    subgraph "Batch Modes"
        LL[Separate Files Mode]
        MM[Merged Mode]
        NN[Source File Tracking]
    end

    %% Output Layer
    subgraph "Output Layer"
        OO[Individual CSV Files]
        PP[Merged CSV File]
        QQ[JSON Output]
        RR[Excel Files]
        SS[Analysis Reports]
        TT[Processing Logs]
    end

    %% Flow connections
    A --> D
    B --> E
    C --> F
    D --> H
    E --> H
    F --> G
    G --> R
    H --> I --> L
    H --> J --> L
    L --> K --> M --> N
    N --> O & P & Q
    G --> S & T & U
    S --> V
    T --> V
    U --> V
    L --> W & X & Y & Z & AA
    V --> BB
    O & P & Q --> CC
    CC --> DD --> EE --> FF
    FF --> GG & HH & II
    DD --> JJ
    CC --> KK
    B --> LL & MM
    MM --> NN
    LL --> OO
    MM --> PP
    GG --> OO & PP
    HH --> QQ
    II --> RR
    JJ --> SS
    KK --> TT

    %% Styling
    classDef input fill:#d9edf7,stroke:#31708f,stroke-width:1px;
    classDef process fill:#e6e6fa,stroke:#6a1b9a,stroke-width:1px;
    classDef core fill:#fff2cc,stroke:#b36b00,stroke-width:1px;
    classDef flow fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px;
    classDef parallel fill:#fde0dc,stroke:#b71c1c,stroke-width:1px;
    classDef proto fill:#ede7f6,stroke:#4527a0,stroke-width:1px;
    classDef data fill:#f1f8e9,stroke:#33691e,stroke-width:1px;
    classDef outputProc fill:#f0f4c3,stroke:#827717,stroke-width:1px;
    classDef batch fill:#ffe0b2,stroke:#e65100,stroke-width:1px;
    classDef finalOut fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px;

    class A,B,C input
    class D,E,F,G process
    class H,I,J,K,L core
    class M,N,O,P,Q flow
    class R,S,T,U,V parallel
    class W,X,Y,Z,AA proto
    class BB,CC,DD,EE,FF data
    class GG,HH,II,JJ,KK outputProc
    class LL,MM,NN batch
    class OO,PP,QQ,RR,SS,TT finalOut
Loading

Installation

From PyPI (Recommended)

pip install nexusflowmeter

From Source

git clone https://github.com/Collgamer0008/NexusFlowMeter.git
cd NexusFlowMeter
pip install -r requirements.txt

Dependencies

  • scapy - Packet manipulation library
  • pandas - Data analysis and manipulation
  • openpyxl - Excel file support (optional)

NexusFlowMeter CLI Usage

 nexusflowmeter [-h] [-d DIRECTORY] [-c OUTPUT_DIR] [--merge] [--protocols PROTOCOLS] [--max-flows MAX_FLOWS]
                      [--output-format {csv,json,xlsx}] [--quick-preview QUICK_PREVIEW] [--split-by-protocol]
                      [--stream] [--chunk-size CHUNK_SIZE] [--flow-timeout FLOW_TIMEOUT] [--max-workers MAX_WORKERS]
                      [--verbose]
                      [pcap_file] [output_file]

Convert PCAP files to flow-based analysis (CSV, JSON, Excel)

positional arguments:
  pcap_file                               Input PCAP file path (for single file processing)
  output_file                             Output file path (for single file processing)

options:
  -h, --help                              show this help message and exit
  -d DIRECTORY, --directory DIRECTORY     Input directory containing PCAP files (for batch processing)
                                      
  -c OUTPUT_DIR, --output-dir OUTPUT_DIR  Output directory (for batch processing)
                        
  --merge                                 Merge all PCAP files into single output (directory processing only)
  --protocols PROTOCOLS                   Comma-separated list of protocols to include (tcp,udp,icmp,arp,dns,all)
                        
  --max-flows MAX_FLOWS                   Maximum number of flows to analyze
                        
  --output-format {csv,json,xlsx}, -of {csv,json,xlsx}
                                          Output file format (default: csv)

  --quick-preview QUICK_PREVIEW           Show first N flows before conversion (single file only)
                        
  --split-by-protocol                     Create separate files for each protocol (single file only)
  --stream                                Use streaming mode (PcapReader) instead of loading the whole file into memory
  --chunk-size CHUNK_SIZE                 Chunk size in MB for large files (default: 1024MB = 1GB)
                        
  --flow-timeout FLOW_TIMEOUT             Flow timeout in seconds (default: 60)
                        
  --max-workers MAX_WORKERS               Maximum parallel workers for chunk processing (default: 4)
                        
  --report-dir DIRECTORY               Directory to save analysis reports (default: in the same directory as csv)

  --verbose                               Enable verbose logging

Positional Arguments

Argument Description
pcap_file Input PCAP file path (for single file processing)
output_file Output file path (for single file processing)
-d /folder_path/ Input folder path containing PCAPs (for multiple file processing)
-c /folder_path/ Output folder path for extracted CSVs (for multiple file processing)
--report-dir /folder_path/ Directory to save analysis reports (default: in the same directory as csv)

Examples

    Single file conversion:
      nexusflowmeter capture.pcap flows.csv

    Directory processing:
      nexusflowmeter -d ./pcaps/ -c ./out/                    # Convert each PCAP in the folder to separate CSV
      nexusflowmeter -d ./pcaps/ -c ./out/ --merge            # Merge all PCAPs into one CSV
      nexusflowmeter -d ./pcaps/ -c ./out/ --output-format json  # Convert to JSON format

    Save report to desired location:
      nexusflowmeter capture.pcap flows.csv --report-dir ./reports/

Options

Option Description
-h, --help Show help message and exit
-d DIRECTORY, --directory DIRECTORY Input directory containing PCAP files (for batch processing)
-c OUTPUT_DIR, --output-dir OUTPUT_DIR Output directory (for batch processing)
--merge Merge all PCAP files into single output (directory processing only)
--protocols PROTOCOLS Comma-separated list of protocols to include (tcp,udp,icmp,arp,dns,all)
--max-flows MAX_FLOWS Maximum number of flows to analyze
--output-format {csv,json,xlsx}, -of {csv,json,xlsx} Output file format (default: csv)
--quick-preview QUICK_PREVIEW Show first N flows before conversion (single file only)
--split-by-protocol Create separate files for each protocol (single file only)
--stream Use streaming mode (PcapReader) instead of loading the whole file into memory (recommended for memory constrained systems)
--chunk-size CHUNK_SIZE Chunk size in MB for large files (default: 1024MB = 1GB)
--flow-timeout FLOW_TIMEOUT Flow timeout in seconds (default: 60)
--max-workers MAX_WORKERS Maximum parallel workers for chunk processing (default: 4)
--report-dir DIRECTORY Directory to save analysis reports (default: in the same directory as csv)
--verbose Enable verbose logging

Examples

    nexusflowmeter capture.pcap flows.json --output-format json  OR
    nexusflowmeter capture.pcap flows.json -of json


    nexusflowmeter capture.pcap flows --split-by-protocol
    nexusflowmeter capture.pcap flows.csv --quick-preview 5
    nexusflowmeter capture.pcap tcp_flows.xlsx --output-format xlsx --protocols tcp

Flow Features Extracted

NexusFlowMeter generates 34 per-flow statistics useful for traffic analysis, anomaly detection, and ML feature extraction.

Feature Description
flow_id Unique identifier for the flow (src_ip:src_port <-> dst_ip:dst_port protocol)
src_ip, dst_ip Source and destination IP addresses
src_port, dst_port Source and destination ports
protocol Transport or network protocol (TCP, UDP, ICMP, etc.)
start_time, end_time Timestamps marking the beginning and end of the flow
duration_seconds Duration of the flow
total_packets Total number of packets in the flow
forward_packets, backward_packets Packets in forward (src→dst) and backward (dst→src) directions
is_bidirectional Indicates if packets exist in both directions
forward_backward_ratio Ratio of forward to backward packet counts
total_bytes Total bytes transferred in the flow
forward_bytes, backward_bytes Bytes in forward and backward directions
total_payload_bytes Total payload size excluding headers
forward_payload_bytes, backward_payload_bytes Payload bytes split by direction
packets_per_second Packet transmission rate
bytes_per_second Byte transfer rate
min_packet_size, max_packet_size Smallest and largest packet sizes observed
avg_packet_size Average packet size
avg_inter_arrival_ms Average packet inter-arrival time (milliseconds)
min_inter_arrival_ms, max_inter_arrival_ms Minimum and maximum inter-arrival times
tcp_flags Summary of TCP flags observed (e.g., SYN, FIN, ACK)
syn_count, fin_count, rst_count, ack_count Count of specific TCP flags
connection_state State of the TCP connection (e.g., Established, Closed)

What makes this tool unique?

NexusFlowMeter includes several productivity-focused options which makes using this tool very convinient and easy.

Directory & Batch Processing

  • Feature: Process an entire folder of PCAP files at once.
  • Example: 
    convert all pcaps in the folder to individual csvs
  nexusflowmeter -d ./pcaps/ -c ./out/

convert all pcaps in the folder into a single file
  nexusflowmeter -d ./pcaps/ -c ./out/ --merge

Protocol Filtering

  • Feature: Focus only on specific protocols (TCP, UDP, ICMP, ARP, DNS, or all) i.e, extracted file contains flow with mentioned protocol only.
  • Example: 
    nexusflowmeter capture.pcap tcp_flows.csv --protocols tcp

Quick Preview

  • Feature: View the first N flows before committing to full conversion.
  • Example: 
    nexusflowmeter capture.pcap flows.csv --quick-preview 5
    Prints the first 5 flows in the console.

Split by Protocol

  • Feature: Automatically create separate output files for each protocol.
  • Example: 
    nexusflowmeter capture.pcap flows --split-by-protocol
    Generates files like flows_tcp.csv, flows_udp.csv, etc.

Streaming Mode

  • Feature: Processes packets using a stream reader instead of loading the entire PCAP into memory (1000 pkts/sec). Useful for large captures and memory contrained systems.
  • Example: 
    nexusflowmeter capture.pcap flows.csv --stream

Chunked Processing

  • Feature: Split very large PCAPs into chunks (in MB) for parallel or memory-efficient processing (default : 1024 MB).

  • Example:

    nexusflowmeter capture.pcap flows.csv --chunk-size 512

    Processes the file in 512 MB chunks.

    nexusflowmeter capture.pcap flows.csv --chunk-size 512 --max-workers 5

    Processes the file in 512 MB chunks parallely , here 5 seperate processes.

    Output Formats

  • Feature: Export results in CSV, JSON, or Excel (XLSX).

  • Example:

     nexusflowmeter capture.pcap flows.json --output-format json
 nexusflowmeter capture.pcap flows.xlsx --output-format xlsx

About

A Network tool used for extracting flow data from PCAP files

Resources

Readme

License

MIT license
Activity

Stars

1 star

Watchers

1 watching

Forks

0 forks
Report repository

Releases

1 tags

Packages

No packages published

Languages