As soon as I enable schema.log_namespace, no events/logs make it through a type: reduce transform step ... but they do make it through type: remap!?! Why!?!

[kquinsland]

Question

I am trying to use vector to parse the system.log from apache cassandra. The lines in system.log are very well formed with one annoying exception. To handle this exception, I have a transforms.cassandra_system_log_merge.type: reduce.

Then, I have a transforms.cassandra_parser.type: remap which uses parse_regex() to give me useful fields.

Finally, things are sent off to an elastic search cluster.

That all works. Right up until I turned on schema.log_namespace. Then everything stopped working. I was able to figure out why things stopped working ... at least inside the remap transform. I still don't know why my reduce transform is broken.

Here's the poc config that I've been using:

---
timezone: "UTC"

api:
  enabled: false

schema:
  log_namespace: true

sources:
  journald:
    type: journald
    since_now: true
    extra_args:
      - "--utc"

  cassandra_system_log:
    type: file
    ignore_older_secs: 600
    read_from: end
    include:
      - /var/log/cassandra/system.log

transforms:
  # merge multi-line
  cassandra_system_log_merge:
    type: reduce
    inputs:
      - cassandra_system_log
    starts_when:
      type: "vrl"
      # The particular multi-line issue that I am dealing with consists of 3 lines. The first ends with `... Normal Tokens:`
      #      then the next line is a list of integers like: [1, 2, -9, 8 ....]
      #      and the third line is just an empty line.
      # The goal here is to turn that into a single line before parsing:
      #    `... Normal Tokens: [1, 2, -9, 8 ....]` is what _should_ get passed along to the next step, ideally.
      source: |
        event = to_string(.) ?? "SomethingWentWrong"
        match(event, r'Normal Tokens:\s')
    max_events: 3
    merge_strategies:
      message: concat

  cassandra_parser:
    type: remap
    inputs:
      # Works, misses the multi-line log entries from time to time :(
      - cassandra_system_log
      # Unclear why, but when the _merge input is used ... we get nothing
      # - cassandra_system_log_merge
    # When namespace is enabled, we have a slightly different syntax to access the log line / event
    ##
    file: "/etc/vector/cassandra_parser.ns.vrl"
    reroute_dropped: true
    drop_on_error: true
    drop_on_abort: true

sinks:
  # Useful for debugging
  cassandra_system_log_json:
    type: file
    path: /var/log/cassandra/system.log.json
    encoding:
      codec: json
      pretty: true
    inputs:
      # The output of the parser is written to disk as expected
      - cassandra_parser
      # But when dumping the 'merged' data to disk, only empty events are written!?!
      #
      #     $ tail -f system.log.json
      #     {}
      #     {}
      #     {}
      ##
      # - cassandra_system_log_merge

  proto_es:
    type: elasticsearch
    inputs:
      - journald
      - cassandra_parser

    endpoints:
      # TLS termination forwards over 9200
      - https://elastic-hou-dev.k.flightaware.com

    auth:
      strategy: basic
      user: <...>
      password: <...>

    acknowledgements:
      enabled: true

    api_version: auto
    mode: data_stream
    data_stream:
      # When true, the name will be programmatic: <type>-<dataset>-<namespace>
      auto_routing: true
      # But if the type/dataset/namespace are not set for each source/transform then we have defaults:
      type: logs
      dataset: vector_poc
      namespace: test-namespace

    encoding:
      timestamp_format: rfc3339

And here is the content of the cassandra_parser.ns.vrl file:

# Make it super clear what we're working on...
event = .

# Regex to extract columns via capture groups
matches, err = parse_regex(event, r'^(?P<level>\w+)\s+\[(?P<thread>[^\]]+)\]\s+(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\s+(?P<file>\S+):(?P<line>\d+)\s+-\s+(?P<msg>.+)$')
if err != null {
  abort
}

# Extract named capture groups into fields, coerce as needed
##
.level = matches.level
.thread = matches.thread

.file = matches.file
.line = to_int(matches.line) ?? null

# The actual "meat" of each line
.message = matches.msg

# Clean up the timestamp we extracted
.timestamp = matches.ts
.ts_parsed = parse_timestamp!(.timestamp, format: "%Y-%m-%d %H:%M:%S,%3f")
.ts_iso8601 = format_timestamp!(.ts_parsed, format: "%+")
.timestamp = .ts_iso8601

# Delete intermediates
del(.ts_parsed)
del(.ts_iso8601)

# Then add a bit of provenance
# The % symbol represents the root of the metadata that accompanies the event.
.metadata = {
  "source": {
    "host": %file.host,
    "path": %file.path,
  }
}

The Above Configuration Works but the occasional multi-line log entry is not handled correctly.
Not the end of the world, but it is annoying.

The Problem

I have brief notes in the comments but when I add the the transform.cassandra_system_log_merge into the chain, the sinks.cassandra_system_log_json log file is filled with empty objects, one per line.

{}
{}
{}

As soon as I remove transform.cassandra_system_log_merge from the chain, the sinks.cassandra_system_log_json file is populated with the expected data:

{"file":"AuthCache.java","level":"INFO","line":177,"message":"(Re)initializing CredentialsCache (validity period/update interval/max entries) (2000/2000/1000)","metadata":{"source":{"host":"af3fa405ad60","path":"/var/log/cassandra/system.log"}},"thread":"main","timestamp":"2025-08-05T21:20:56.205+00:00"}
{"file":"NativeTransportService.java","level":"INFO","line":73,"message":"Netty using Java NIO event loop","metadata":{"source":{"host":"af3fa405ad60","path":"/var/log/cassandra/system.log"}},"thread":"main","timestamp":"2025-08-05T21:20:56.261+00:00"}

The Question

The docs for the reduce transform say that namespaces change things:

Warning
The fields shown below will be different if log namespacing is enabled. See Log Namespacing for more details

But they don't say how things change.
The link to the Log Namespacing docs is not helpful either; it's a very brief announcement of the feature and there's a FIXME at the bottom that says:

[//]: # ([global config]: /docs/reference/configuration/global-options/#schema.log_namespace) TODO FIX

So what do I need to do so that I can use namespacing and still have the reduce transform work?

Vector Config

No response

Vector Logs

Nothing significant shows up in the logs.

I am running vector this way:

root@af3fa405ad60:/etc/vector# vector --config=/etc/vector/poc.ns.yaml

[pront]

Hi @kquinsland, please take a look at this guide https://vector.dev/guides/level-up/log_namespace/ (if you haven't already). Enabling the log_namespace changes the shape of the events thus your reduce configuration will have to be adjusted.

[kquinsland]

Thanks for that link.

That doesn’t explain what new shape is needed for reduce and why that’s somehow different from remap, though.

Ignore - for a moment - the reduce transform. In my remap, I have this:

# Make it super clear what we're working on...
event = .

Matches = parse_regex(event, ….)

And when namespace is NOT turned on, I have to:

Matches = parse_regex(.message, ….)

That makes sense; when the new namespace feature is on, what used to be .message is now . and all of the other keys that were in the top of . have been moved to a new object %. So with namespace on, an “event” is now a tuple: (.,%)

——

Assuming that understanding is correct, here’s what happens when I have the reduce before the remap:

# <…>
      source: |
        event = to_string(.) ?? "SomethingWentWrong"
        match(event, r'Normal Tokens:\s')

that only ever emits empty events: {} but I am still piping the same . into the match() call.
At no point am I ever changing . so as long as the source: … evaluates to TRUE, the reduce logic runs for the next 3 lines and then emits them all as a single event, right?

So when the reduce has collected everything, it still emits a (., %) pair, right?