High memory usage and long startup time in ndpiReader with large category lists (-G) – proposal for binary .ndb backend

[fabiodepin]

While working with large custom category datasets, we observed significant scalability limitations in ndpiReader when using the current text-based category loading (-G).

Test setup
• Total domains: ~7,848,883
• Total categories: 96
• Source: /etc/categories/*
• Environment: Linux

Results

Baseline (no categories loaded)
Total memory allocated: 9.03 MB
Setup Time: 107 ms

Using -G (text-based categories)
Total memory allocated: 892.46 MB
Setup Time: 7834 ms

Observations:
• Extremely high memory consumption (~892 MB)
• Very long initialization time (~7.8 seconds)

Using binary .ndb backend (prototype)
Total memory allocated: 9.06 MB
Setup Time: 1128 ms

Observations:
• Memory usage reduced to near baseline levels
• Startup time significantly improved compared to -G

⸻

Problem summary

The current text-based category loading (-G):
• Does not scale well for large datasets (millions of domains)
• Consumes large amounts of RAM due to in-memory structures (AC, strings, etc.)
• Has high initialization overhead
• Negatively impacts runtime performance

⸻

Proposed solution

Introduce (or support upstream) a binary category database format (.ndb) with:
• Precompiled domain index
• mmap-based loading
• Suffix-based lookup (domain → parent domains)
• Fixed category IDs (aligned with ndpi_protocol_category_t)

Key advantages
• Drastically reduced memory usage (no large in-memory AC structures)
• Faster startup (no parsing or building of runtime structures)
• Improved lookup performance
• Better scalability for NGFW / large deployments

⸻

Additional notes
• The prototype maintains compatibility with existing behavior:
• Built-in categories unchanged
• IP-based classification unchanged (Patricia trees)
• Optional hybrid mode (ndb + legacy fallback)
• Normalization and suffix matching follow a consistent contract between generator and runtime

⸻

Use case

This is particularly relevant for:
• NGFW / DPI systems
• Large-scale URL/category filtering
• Deployments with millions of domains

⸻

Conclusion

The current -G approach works well for small datasets, but does not scale to millions of domains.

The .ndb approach demonstrates:
• ~100x reduction in memory usage
• ~7x faster startup vs -G
• Better runtime performance

We believe this could be a valuable addition to nDPI for high-scale environments.

[fabiodepin]

As an additional note, while investigating this further, I observed that in ndpiReader the legacy memory usage appears to grow significantly with multiple interfaces / reader threads, while .ndb remains much flatter in practice.

This is separate from the single-interface results above, but it may be relevant when evaluating larger or multi-interface deployments.

[IvanNardi]

As an additional note, while investigating this further, I observed that in ndpiReader the legacy memory usage appears to grow significantly with multiple interfaces / reader threads, while .ndb remains much flatter in practice.

That should be very unexpected. Pretty much all data structures (with default configuration, at least) are per-context, i.e. every module has its own copy of everything. So if 1 module = 100MB --> 5 modules = 500MB (roughly speaking).
I do expect the same behavior with the ndb patch

[fabiodepin]

Yes, that was my expectation as well based on the per-context design.

However, in practice I observed different behavior with .ndb in ndpiReader.

Even though each module loads its own .ndb (via mmap), the RSS growth is much smaller compared to the legacy path when adding interfaces/threads. It does not scale close to linearly like the legacy structures do.

My current assumption is that this comes from the fact that .ndb relies on mmap(MAP_SHARED), so even if each module creates its own mapping, the underlying file-backed pages are shared by the OS page cache, while the legacy path builds independent heap structures per module.

So from the library perspective it is still per-context, but from a physical memory standpoint the behavior is different.

I can try to provide a more detailed breakdown (e.g. RSS vs virtual memory, per-thread measurements) if that would be useful.

[fabiodepin]

I did a more detailed check with smaps, and the behavior now makes sense.

You are right that the design is still per-context: with .ndb, each module/thread creates its own mapping of the file.

However, what differs from the legacy path is the physical memory behavior.

Legacy

With -G, memory grows almost linearly with the number of interfaces / reader threads:

• 1 interface: ~1.0 GiB
• 2 interfaces: ~1.9 GiB
• 3 interfaces: ~3.0 GiB

In smaps_rollup, this is mostly Private_Dirty, so these are effectively independent heap allocations per context.

.ndb

With .ndb, I also see one mapping per interface/thread, but the mapped dataset pages are shared by the kernel:

• 1 interface: ~384 MiB
• 2 interfaces: ~397 MiB
• 3 interfaces: ~410 MiB

Looking at /proc//smaps for the .ndb mappings:

• with 2 interfaces, each .ndb mapping shows Rss=375800 kB, Pss=187900 kB, Shared_Clean=375800 kB
• with 3 interfaces, each .ndb mapping shows Rss=375800 kB, Pss=125266 kB, Shared_Clean=375800 kB

So the library-side ownership is still per-context, as expected, but the file-backed pages of the .ndb are shared across contexts, while the legacy path builds separate heap structures per context.

That explains why the .ndb memory stays almost flat in practice, whereas the legacy path grows roughly linearly.

[IvanNardi]

Thanks for your analysis

but the file-backed pages of the .ndb are shared across contexts

I need to think a little bit if that is a good thing or not... 🤔

[fabiodepin]

Yes — I see it as a trade-off as well.

The design is still fully per-context (each module has its own mapping and reload path), but since the dataset is read-only and file-backed, the kernel ends up sharing the physical pages across contexts.

So effectively:

• logically per-context
• physically shared for the read-only dataset pages

The main benefit is avoiding large duplicated heap structures per context (as happens with -G), especially with multiple reader threads.

The trade-offs are mainly around:

• temporary coexistence of old/new mappings during reload
• memory behavior partially depending on page cache
• less strict isolation at the physical memory level

From my side, this seemed acceptable for large-scale deployments, but I agree it’s worth validating whether this aligns with the intended design goals.

Happy to adjust if a different trade-off is preferred.