Provide a demo mode for all components that depend on an external dependency

[binarylogic]

The ability to demo any Vector configuration without external dependencies would greatly reduce the friction of demoing for the purposes of education, sales, and development of tools that depend on Vector (i.e., dashboards, a UI, etc.).

Currently, we offer a vector-demos repository but this spins up a full environment with mock dependencies. It is great for integration testing, or demonstrating the installation onto platforms like Kubernetes, but it is not great for demonstrating very specific use cases (i.e., reducing cost of AWS VPC logs coming from AWS S3 and going to Splunk). Instead, Vector should offer "demo modes" for all of its components that depend on external dependencies.

Example

Taking the example above (reducing cost of AWS VPC logs coming from AWS S3 and going to Splunk), we could provide the following configuration. Because the new demo mode more or less mocks I/O, it could be run locally, on any supported platform, without the need to provision any external services:

[sources.aws_s3]
type = "aws_s3"
region = "us-east-1"
strategy = "sqs"
demo.enabled = true
demo.formats = ["aws_vpc"] # defaults to all formats

[transforms.remap]
inputs = ["aws_s3"]
type = "remap"
sources = '''
...
'''

[sinks.splunk_hec]
inputs = ["remap"]
type = "splunk_hec"
demo.enabled = true

1. The aws_s3 source would emit fake logs that mimic reading AWS VPC logs from S3.
2. The remap transform could demonstrate dropping extraneous attributes for cost cutting.
3. The splunk_hec sink would simply drop the data it receives.

Use cases

1. Help sales to demonstrate use cases specific to the prospect.
2. Help with support in certain situations (i.e., downloading a users Vector config, enabling demo mode, and running it locally).
3. Enable follow-along education in tutorials and guides.
4. Help with development of tools that depend on Vector's internal telemetry.
5. Generating reproducible set of logs for testing.

Open questions

Is it best to embed a demo option into each component?

The thinking here was to require every source to provide some way of demoing, and make it easy to flip existing components into demo mode. You could also imagine a global demo option that does this across the board. The problem is that this gets into some strange issues:

1. Do we continue to require fields that are irrelevant when the demo mode is enabled? (i.e., the aws_s3.region option).
2. Do we have an obvious place in the code that we could short-circuit I/O without other unintended side-effects, like failing health checks, etc.

Should we include transforms that interface with an external service?

For example, the ec2_metadata transform could enrich events with fake data.