[Tracking] Opening multiple event sources in the same Falco instance

[jasondellaluce]

Motivation

The plugin system allows Falco to open new kinds of event sources that go beyond the historical syscall use case. Recently, this has been leveraged to port the k8s audit log event source to a plugin (see: https://github.com/falcosecurity/plugins/tree/master/plugins/k8saudit, and #1952). One of the core limitation that comes from the plugin system implementation of the libraries, is that a given Falco instance is capable of opening only one event source. In the example above, this implies that Falco instances are not able to ingest both syscalls and k8s audit logs together. This can instead be accomplished by deploying two distinct Falco instances, one for each event source.

Feature Requirements

• (R1) A single Falco instance should become able to open more than one event source at once and in parallel
• (R2) There should be feature parity and performance parity between having 2+ source active in parallel in a single Falco instances and having 2+ single-source Falco instances with the same event sources

Proposed Solution

Release Goals

To be defined. This is out of reach for Falco 0.32.1. Falco 0.33.0

Terminology

• Capture Mode: A configuration of sinsp inspectors that reads event from a trace file
• Live Mode: A configuration of sinsp inspectors that reads event from one of the supported modes (kmod, ebpf, gvisor, plugin)

Design

• (D1) The feature is implemented in Falco only, and mostly only affects the codebase of falcosecurity/falco. Both libsinsp and libscap will keep working in single-source mode
• (D2) Falco manages multiple sinsp instances, one in each thread
• (D3) Falco manages one or more instances of sinsp inspectors
  • If the # of inspectors is 1, everything runs in the main thread just like now
  • If the # of inspectors is 2+, each inspector runs in its own a separate thread (see (R1)). The whole event data path happens in parallel within each thread (event production, data enrichment, event-rule matching, and output formatting)
• (D4) If in capture mode, Falco runs a only 1 inspector configured to read events from a trace file
• (D5) If in live mode, Falco runs 1 inspector for each active event source
  • If an event source terminates due to EOF being reached, Falco waits for the other event sources to terminate too
  • If an event source terminates with an error, Falco forces the termination of all the other event sources
• (D6) There is 1 instance of the Falco Rule Engine (just like now), and we leverage/enforce thread-safety guarantees to make sure it is safe and non-blocking for different threads to perform event-rule matching
• (D7) There is 1 instance of the Falco Output Engine (just like now), and we leverage/enforce thread-safety guarantees to make sure it is safe for different threads to send alerts when an event-rule match is found
  • Non-blocking guarantees are less of a concern here, because the number of alerts is orders of magnitudes lower than the number of events

Technical Limitations of the Design

• (L1) There cannot be 2+ event sources with the same name active at the same time
  • This would defeat the thread-safety guarantees of the Rule Engine, which are based on the notion of event source partitioning
  • Potential Workarounds (for the future, just in case):
    • Have more than one instances of the Rule Engine to handle the increased event source cardinality. For example, the second Rule Engine instance would cover all the second event source replicas, the third Rule Engine instance will handle the third replicas, and so on
    • We make the Rule Engine thread safe without the event source <-> thread 1-1 mapping assumptions. This is hardly achievable, because this would imply making the whole filtercheck system of libsinsp thread-safe too. Another naive solution would be to create one mutex for each event source to protect the access to the Rule Engine. In both scenarios, this would be hard to manage and performance would be sub-optimal
    • We have one Rule Engine for each source, which could become harder to manager. For example, rule files would need to be loaded by all the rule engines, which makes the initialization phase and hot-reloading slower too. However, this is something we can consider for the future.
• (L2) Filterchecks cannot be shared across different event sources to guarantee thread-safety in the Rule Engine. The direct implication is that if an plugin with extractor capability is compatible with 2+ active event sources (e.g. json can extract from both aws_cloudtrail and k8s_audit), we need to create and initialize two different instances of the plugin (1 for each inspector)
  • Practically, this means that a given plugin instance will always extract fields coming from the same event source (a.k.a. subsequent calls to plugin_extract_fields will never receive events from two distinct event sources for the same initialized pluginstate)
  • This limitation can actually be turned as a by-design feature, because doing the contrary would violate (R2)
  • Potential Workarounds (for the future, just in case):
    • Make field extraction thread-safe (hardly doable, see points in (L1)

Technical Blockers

This is the list of things we mandatorily need to work on to see this initiative happen.

• (B1) The rule engine <-> inspector source index mapping needs to be handle in different ways for capture mode and live mode
  • In capture mode, the rule engine source index is the same as the source index in the plugin manager of the single inspector used in capture mode (with the exception of the syscall source, which is by convention the last source index after all the plugin ones)
  • In live mode, each rule engine source index should be uniquely assigned to each inspector in live mode that runs in its own thread
  • update: support running multiple event sources in parallel #2182
• (B2) Plugins can potentially be loaded multiple times in order to be registered to each live mode inspector
  • In live mode, a single plugin with field extraction capability can be registered to all the inspectors configured with an event source they are compatible with
    • Note: This also applies to plugins with both field extraction and event sourcing capabilities. In this case, the plugin is register and used both its capabilities only with the inspector in which its event source is active, whereas it is just registered for its field extraction capability in all other event-source-compatible inspectors.
  • update: support running multiple event sources in parallel #2182
• (B3) The plugin API and the Plugin SDK Go should be revised to support multi-thread and concurrency assumptions (will likely be the only change needed outside of falcosecurity/falco)
  • Plugin API:
    • Most API symbols will need to support being called concurrently, with every call having distinct ss_plugin_t*s
    • update(userspace/plugin): bump plugin API version to 2.0.0 libs#547
  • Plugin SDK Go:
    • Tracking the discussion in another issue: [tracking] supporting concurrent consumers plugin-sdk-go#62
      • refactor(pkg/sdk/symbols/extract): support concurrent consumers in async extraction optimization plugin-sdk-go#65
• (B4) Print-only Falco actions (e.g. list fields, list events, etc...) are dependent on the app state inspector. These needs to be stateless and can be implemented by allocating a sinsp inspector on-the-fly because they just access static information
  • refactor(userspace/falco): reduce app actions dependency on app state and inspector #2097
• (B5) The Falco StatsWriter (-s option) is not thread-safe
  • refactor(userspace/falco): re-design stats writer and make it thread-safe #2109
• (B6) The Falco Rule Engine is does not provide any thread-safety guarantee
  • refactor(userspace/engine): increase const coherence in falco engine #2081 (non-blocker)
  • refactor(userspace/engine): strengthen and document thread-safety guarantees of falco_engine::process_event #2082
• (B7) Signal-based actions (termination, restart, and output reopening) are not thread safe
  • refactor(userspace/falco): make signal handlers thread safe #2091
• (B8) The Falco Output framework is not entirely thread safe
  • refactor!(userspace/falco): remove output rate limiter and make output engine explicitly thread-safe #2080
  • refactor(userspace/falco)!: make output rate limiter optional and output engine explicitly thread-safe #2139
• (B9) Libsinsp and libscap have a bunch of global and static variables that limit our freedom of having multiple inspectors running in parallel (g_infotables, g_logger, g_initializer, g_filterlist, g_decoderlist, s_cri_xxx, libsinsp::grpc_channel_registry::s_channels, s_callback, g_event_info, g_ppm_events, g_chisel_dirs, g_chisel_initializer, g_syscall_code_routing_table, g_syscall_table, g_syscall_info_table, g_json_error_log)
  • This may seem like a lot, but we should good to go as-is because most of these are read-only tables or objects. The ones that actually bundle some logic are either thread-safe (g_logger), or used only by inspectors running the syscall event source. Since due to (L1) we don't allow two inspectors running the syscall source at the same time, it should be safe to assume that no concurrent access will happen to the syscall-related globals.
• (B10) The whole Falco application logic should be revised to support multiple inspectors, multiple filtercheck factories, and to distinguish the capture-mode and live-mode use cases as defined in the Design section. This will required all previous (BXX) points to be satisfied first
  • update: support running multiple event sources in parallel #2182

Nice to Have

• (N1) Add a new -–enable-source=xxx option as a dual to -–disable-source=xxx. This design implies that the active event sources are chosen in an opt-out fashion: every loaded source gets activated, with the exception of the disabled ones. The -–enable-source option will make the UX better to define the only source users want to activate
  • new: add option to enable event sources selectively #2085
• (N2) Improve the regression testing framework falco_test.py to support selecting the active event source. Without it, all non-syscall tests will hang or fail, because the syscall event source is implicitly activated along with the testing-subject one and will cause Falco to not terminate (example: k8s audit tests)
  • new: add option to enable event sources selectively #2085
• (N3) Reduce threadiness of /healtz webserver (based on cpp-httplib). The webserver library documents that the default threadiness is 8 or std::thread::hardware_concurrency(). This is OK, but since we go on a multi-threaded model we should consider limiting the number of threads spawned by Falco. In my 8-core setup, I Falco spawned 30 threads (only few were active, luckily) with a simple test with syscalls and 2 Go plugins loaded.
  • update(userspace/falco): make webserver threadiness configurable #2090

Linked Discussions

• https://kubernetes.slack.com/archives/CMWH3EH32/p1655762074442099
• https://kubernetes.slack.com/archives/CMWH3EH32/p1655209558876649
• https://kubernetes.slack.com/archives/CMWH3EH32/p1647969834426869
• https://kubernetes.slack.com/archives/CMWH3EH32/p1646148516502339?thread_ts=1646061690.259599&cid=CMWH3EH32
• https://kubernetes.slack.com/archives/CMWH3EH32/p1645610199932789?thread_ts=1645396079.322259&cid=CMWH3EH32
• https://kubernetes.slack.com/archives/CMWH3EH32/p1645038583559959?thread_ts=1645034669.067289&cid=CMWH3EH32
• k8s_audit_rules can't be loaded #2110

[jasondellaluce]

I think this is something we should aim for over the next few months. So, I propose to turn this issue into a tracking issue where I'll share the though process of my proposed solution and all the steps and PRs involved. I expect this to be the place in which we can have discussions about this solution, and perhaps converge to something we all agree on.

/milestone 0.33.0

[jasondellaluce]

I'm gonna share the thought process that led me to the proposed solution documented in the issue description. @leogr has been involved in the early discussion of this topic.

Since the end goal is to run multiple sources at the same time, many execution models have been hypothesized, but each of them has been discarded in favor of the proposed solution. I'm gonna proceed and document briefly their design and cost/benefits tradeoffs.

• (H1) Single-threaded model, with multiple event sources opened by libscap
  • This would be simple to implement, but would require lots of refactorings of the libraries
  • This would not work in the general case, because performance would be tremendously degraded depending on the event consumption policy of libscap (e.g. round-robin)
  • Each event source would be blocked by the other ones. Since there are no buffering or multi-threading, every event production request is sequential. This means that "slow" event sources would become the bottleneck for all the other ones. In the case of syscalls, this would be totally unacceptable in terms of event drop metrics
• (H2) Multi-threaded model, with parallelization at the libscap level: libscap would become able to open multiple event sources in parallel and consume all of them from a single thread
  • Not too hard to implement
  • "Slow" event sources would not impact the others anymore
  • Producing events in parallel would require a queue-like synchronization system, similar to our syscall ringbuffer, which would also require backpressure control support and memory copies. Both of these would degrade performance to a non-acceptable level
• (H3) Multi-threaded model, with parallelization at the libsinsp level: libsinsp manages multiple scap instances in parallel
  • Costs and benefits are totally analoguous to (H2). In this case, libsinsp would be come the sequential bottleneck

Reasonably, going up in the stack the next synchronization point would be Falco itself, which is exactly the solution we proposed. This is the only design we came up with that ensure total and non-blocking parallelization of each event source processing routine. The only sequential bottleneck becomes the Falco Output Engine, which is natively designed to be thread-safe(~ish), and has the benefits that the number of alerts is orders of magnitude lower than the number of events produced by each event source (syscalls in particular), which ensure acceptable performance and negligible synchronization overhead.

[araujof]

Nice feature and write-up, @jasondellaluce!

For D6, have you thought about issues of fairness, etc. when the rate of system calls far exceeds the rate of other log sources? I think that some queue management will likely be necessary to avoid contention/blocking. A thread pool in the rules engine could also help with throughput, at the expense of allowing events to be exported out of order.

[jasondellaluce]

@araujof thanks for reviewing, those are all good points!

For D6, we can achieve good thread safety guarantees from the Rule Engine with the assumption that every thread gets assigned different event source. This stands before the Rule Engine is entirely partitioned by event source. See this for more details: #2082. Assuming that the Rule Engine level can provide wait-free concurrent access, I'd say fairness should be guaranteed at that level.

Then, once alerts are triggered they go through the Output Engine, where fairness and ordering are a big topic instead. Luckily, thanks to @leogr we already have a concurrent ordered queue there. There may be a fairness point if the queue gets filled-up with alerts coming from one event source, but that's an unlikely scenario (If Falco fills up the queue, it means that either the outputs are all blocked, or that some rule is really noisy). A simple workaround would be to increase the queue size. We only have one worker on the other side of the queue though, and we might consider using a thread pool maybe in the future.

What do you think? Does this address your concerns?

[araujof]

Hi @jasondellaluce,

What do you think? Does this address your concerns?

Sounds good to me! Thanks for the pointer to #2082.