Ability to inject events into sources for unit testing

[s-at-ik]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

I would like to ensure a change in multiline patterns will not break known multiline logs.

Attempted Solutions

I've tried writing a test case with input of type raw and insert_at the source component. However that's not supported:

sources:
  test_source:
    type: file
    include:
      - /var/log/example.log
    multiline:
      start_pattern: '^#'
      mode: halt_before
      condition_pattern: '^#'
      timeout_ms: 100

sinks:
  console:
    type: console
    target: stdout
    inputs:
      - test_source
    encoding:
      codec: json

tests:
  - name: test 1
    inputs:
      - insert_at: test_source
        type: raw
        value: |-
          # line start 1
          foo
          bar
          # line start 2
          foo
          bar
    outputs:
      - extract_from: test_source
        conditions:
          - type: vrl
            source: |-
              assert_eq!(.message, "# line start 1\nfoo\nbar")

Running tests
2024-09-06T13:11:18.960534Z ERROR vector::unit_test: Failed to execute tests:
Failed to build test 'test 1':
  Invalid extract_from target in test 'test 1': 'test_source' does not exist.

(adding a dummy transform just yields the same error on the input)

Proposal

As seen in my attempt above, one way to test multilines would be to allow source components as input_at and extract_from targets. Not sure it makes sense in any other case beside multiline though.

References

No response

Version

vector 0.40.1 (x86_64-unknown-linux-gnu a9392b0 2024-08-26 14:35:19.223750502)

[jszwedko]

Ah, yes, it isn't possible to provide inputs to sources for unit tests. Let me reword this title to be a bit broader.

As a workaround, instead of using the multiline config you could use a reduce transform, which would be unit testable.