How to reduce file source ingestion latency for near real-time log collection?Q&A

[ArkIiii-9526]

Question

I am using Vector 0.56.0 file source to collect application logs and I am trying to reduce the latency between a log line being appended to a file and the event being available downstream.

Current environment:

• Vector version: 0.56.0
• Source: file
• Filesystem: ext4
• Logs are appended to existing files

The observed end-to-end latency is around 0.5~2 seconds.

I investigated the delay and found that the file source uses a polling model rather than filesystem notifications. In the source code, the FileServer applies an exponential backoff when no new bytes are read:

backoff_cap = if global_bytes_read == 0 {
    cmp::min(2_048, backoff_cap.saturating_mul(2))
} else {
    1
};

My understanding is that this can cause a newly appended log line to wait until the next polling cycle.

I have tried tuning:

• glob_minimum_cooldown_ms
• max_read_bytes
• sink batch timeout

but these do not appear to affect this specific delay.

My goal is to achieve lower latency for interactive log viewing use cases.

Are there recommended configurations or deployment patterns for reducing file source ingestion latency?

Would another Vector source type or architecture be more appropriate for this use case?

Vector Config

apiVersion: v1
kind: ConfigMap
metadata:
name: vector-config
namespace: monitoring

data:
vector.yaml: |
data_dir: /var/lib/vector

api:
  enabled: true
  address: 0.0.0.0:8687

sources:

  application_logs:
    type: file

    include:
      - /var/log/app/*.log

    read_from: beginning

    fingerprint:
      strategy: device_and_inode

    # Tried lowering this value, but it does not affect
    # latency for appending to existing files.
    glob_minimum_cooldown_ms: 10

    max_read_bytes: 1048576

    multiline:
      mode: halt_before
      start_pattern: '^\d{4}-\d{2}-\d{2}'
      condition_pattern: '^\d{4}-\d{2}-\d{2}'
      timeout_ms: 1000


transforms:

  parse_logs:
    type: remap
    inputs:
      - application_logs

    source: |
      .message = to_string!(.message)


sinks:

  debug_output:
    type: console

    inputs:
      - parse_logs

    encoding:
      codec: json

---

apiVersion: apps/v1
kind: DaemonSet

metadata:
name: vector-agent
namespace: monitoring

spec:

selector:
matchLabels:
app: vector-agent

template:

metadata:
  labels:
    app: vector-agent

spec:

  containers:

    - name: vector

      image: timberio/vector:0.56.0-debian

      args:
        - --config
        - /etc/vector/vector.yaml


      resources:

        requests:
          cpu: 200m
          memory: 256Mi

        limits:
          cpu: 1000m
          memory: 1Gi


      volumeMounts:

        - name: config
          mountPath: /etc/vector

        - name: logs
          mountPath: /var/log/app
          readOnly: true

        - name: vector-data
          mountPath: /var/lib/vector


  volumes:

    - name: config

      configMap:
        name: vector-config


    - name: logs

      hostPath:
        path: /var/log/app
        type: Directory


    - name: vector-data

      emptyDir: {}

The above configuration is a simplified version of my production setup.

The actual deployment sends events to an HTTP sink, but the observed delay remains the same when using console sink, which indicates that the delay happens before the sink stage.

Vector Logs

No response

[paulodearaujo]

The first thing I would A/B test is the multiline block. In this config, it can account for roughly 1 second of the delay by itself.

timeout_ms is the maximum time the multiline aggregator waits for another line before flushing the buffered event. With halt_before, the last event is released when the next timestamp arrives or when that timeout expires. Vector 0.56 describes that behavior directly in MultilineConfig.

Try this first:

multiline:
  mode: halt_before
  start_pattern: '^\d{4}-\d{2}-\d{2}'
  condition_pattern: '^\d{4}-\d{2}-\d{2}'
  timeout_ms: 100

Or temporarily remove multiline entirely and measure the console sink again. If the delay drops, raise the timeout only as far as needed to avoid splitting slow stack traces.

The file polling delay you found is real too. In 0.56.0, reads of existing files use an adaptive sleep that doubles while idle and is hard-capped at 2,048 ms (read loop, backoff). glob_minimum_cooldown_ms controls discovery/globbing, not how often an already-open file is read. max_read_bytes controls fairness and throughput once data is available, so neither setting removes this idle backoff.

That gives you two separate latency sources: up to 1 second from multiline buffering and up to about 2 seconds from an idle file watcher. Lowering the multiline timeout should improve the usual case, but there is no configuration knob for the file read backoff in 0.56.0.

If consistently sub-100 ms delivery is a hard requirement, I would avoid the file source for that path and have the application write directly to a local socket or another push-based Vector source. Keep file tailing for durability or fallback rather than as the interactive transport.

[ArkIiii-9526]

Thanks for the explanation. I understand that achieving sub-100ms latency is not the target use case for the file source, and that a push-based source would be more appropriate for that scenario.

My current requirement is less strict: I am trying to keep interactive log viewing latency consistently around or below 1 second, while still keeping the reliability benefits of file-based collection.

From my tests:


• multiline.timeout_ms: 1000 adds approximately 1 second of delay.
• Reducing it to 100ms removes that part of the latency.
• With multiline disabled, I still observe around 1.8s p50 latency for sparse writes.
  
  Before considering a different ingestion architecture, I would like to understand whether there are any recommended configurations or operational patterns for getting the file source closer to ~1s latency.
  
  For example:
  
• Are there any file source scheduling parameters that are relevant for this case?
• Does keeping the file continuously active significantly reduce latency compared with sparse writes?
• Is the remaining ~1-2s latency considered the expected behavior boundary of the file source?

[paulodearaujo]

Your measurements match the 0.56.0 read loop.

There is no file-source scheduling setting that changes this behavior. After a successful read, the backoff resets to 1 ms. Each empty poll then doubles it until it reaches 2,048 ms. glob_minimum_cooldown_ms only affects file discovery, and max_read_bytes only affects how much is read once data is available.

So yes, a naturally busy file stays on the fast path and will usually have much lower polling delay. A sparse file eventually reaches the 2,048 ms sleep, so a new line can wait close to two seconds before Vector sees it. I would not add artificial heartbeat lines just to keep the reader active, since that changes the log stream and only works around the polling behavior.

That makes roughly 1–2 seconds a realistic boundary for an idle file, plus any multiline or downstream buffering. It is not a documented latency guarantee. A periodic sparse-write benchmark can also line up with the polling cycle, so randomizing the interval between test writes is useful when comparing p50 and p95.

For the setup you described, I would keep multiline.timeout_ms: 100, verify the sink is not batching, and compare active-file versus idle-file measurements. But 0.56.0 cannot guarantee consistently <=1 second for sparse writes through configuration alone. Meeting that hard bound would require either lowering the polling cap in the file source or using a push-based source for the interactive path.

The same hard-coded polling limit is also the subject of issue #19585.

[ArkIiii-9526]

Thanks for the detailed explanation.

I reran the benchmark after finding a clock skew issue between the test host and the Kubernetes node. The corrected results now match the behavior you described:

• Active file with continuous writes:

  • p50: ~123ms
  • p95: ~487ms

• Sparse writes:

  • p50: ~980ms
  • p95: ~1015ms

• Idle file (15s idle, then append):

  • p50: ~3.3s

This confirms that active files stay on the fast path, while sparse/idle files are affected by the adaptive polling backoff.

I also verified the multiline impact:

• timeout_ms: 1000 adds approximately 1 second of latency.
• timeout_ms: 100 reduces that overhead to around 100ms.

Based on these results, I will keep the file source approach and tune multiline.timeout_ms only after checking real Java stack trace line intervals, since lowering it too much may split slow-emitting stack traces.

For our use case, the main goal is interactive log viewing rather than strict real-time delivery, so accepting the file source behavior for sparse/idle files is reasonable. If we need a hard sub-second guarantee in the future, we will consider a push-based ingestion path.

Thanks for helping clarify the design trade-offs of the file source.