Q&A How to use memory table insides tests

[lchabert]

Question

Hello,

I'm using memory enrichment tables inside a pipeline to add metadata to my syslog input
The metadata are stored inside a s3 bucket, and loaded each hour to fill up the memory table.

How i can test the syslog pipeline without any errors. When i was using csv files, it was working great, but now with memory table, i got an issue and tests does not work anymore.

Vector Config

sources:
  enrich_loader:
    type: http_client
    endpoint: https://mys3.com/bucket_files/enrichment_file.json
    decoding:
      codec: json
    scrape_interval_secs: 3600
    scrape_timeout_secs: 60
  syslog_ampq:
    type: amqp
    connection_string: amqp://admin:admin@localhost/
    decoding:
      codec: json
    queue: log_queue

enrichments_tables:
  metadata_cache:
    type: memory
    inputs: ["enrich_generator"]
    ttl: 3660
    flush_interval: 5

transforms:
  enrich_generator:
    type: "remap"
    inputs: ["enrich_loader"]
    source: |
      data = .
      . = set!(value: {}, path: [upcase!(get!(data, path: ["mac_address"]))], data: data)
  log_parser:
    type: remap
    reroute_dropped: true
    drop_on_error: true
    drop_on_abort: true
    inputs:
      - syslog_ampq
    source: |
      row, err = get_enrichment_table_record("metadata_cache", {"mac_address" : mac_address}, case_sensitive: false)
      . = merge(., row.value) 

sinks:
  console:
    type: console
    inputs: ["log_parser"]

enrichment_file.json

[
    {
        "mac_address": "37:ac:16:7e:af:48",
        "entity": "HOMELAB"
    }
]

Test content

tests:
  - name: "Syslog merge out"
    inputs:
      - insert_at: log_parser
        type: log
        log_fields:
          "@timestamp": "2024-08-20T09:37:22.245597628Z"
          ."mac_address": "37:ac:16:7e:af:48"
          ."action: "accept"
          ."dst-zone": "lan"
    outputs:
      - extract_from: fbx_fw_logging_transform
        conditions:
          - type: vrl
            source: |-
              assert!(.mac_address == "37:AC:16:7E:AF:48")
              assert!(.entity == "HOMELAB")
              assert!(.action == "accept")

Vector Logs

Failed to build test 'Syslog merge out':
  Input "enrich_generator" for sink "metadata_cache" doesn't match any components..

[pront]

cc @esensar in case you tried something like this in the past

[esensar]

I haven't tried it out in tests. It seems vector test needs some changes to correctly work with these updated tables, because they are not considered as components with inputs/ouputs and the subgraph used for tests ends up being incomplete. I will check if I can add a patch for that, shouldn't be too hard I think.

[pront]

Thank you!

[lchabert]

Nice to ear this. Waiting for this patch to secure our pipelines with unittest

[esensar]

PR is ready for this: #24081