Logs to s3

[shamj]

I am using awss3 sink to send the logs using vector. However I am trying to figure how to read the logs from s3 for a specific time period, is there a way I can do it. I tried ClickHouse , but it says invalid format. Looks like it didn't like the format the way vector writes to s3.
Please help.

[pront]

how to read the logs from s3 for a specific time period, is there a way I can do it.

The aws_s3 source polls SQS for bucket notifications, which means new objects. For older data you would have to implement a custom solution.

Looks like it didn't like the format the way vector writes to s3.

This depends on your configuration. Especially the source decoder and sink encoder.

[shamj]

Thank you. And is there a way to read the data that is written to s3 by vector and send it to grafana. We tested some other tools to read the data, but they didn't like the format and the way vector writes the logs to s3.

[pront]

And is there a way to read the data that is written to s3 by vector and send it to grafana.

Yes, you can use an aws_s3 source and a Loki sink. Note the AWS S3 source requires a SQS queue configured to receive S3 bucket notifications and this applies to new events as they come in while your pipeline is ready to read.

but they didn't like the format and the way vector writes the logs to s3

In general, the remap transform is powerful enough to remediate the data as they go through a Vector pipeline to satisfy the expectations of downstream systems (assuming the out-of-the-box codecs do not work). You will need to understand the event format when it comes out of the source and the format the downstream expects.

[shamj]

I was checking aws s3 source, does it have the option to read the files based on the object and folder structure in s3?
Lets say we have a folder in s3 as vector-logs that has subfolders as date/ hour/min . Can I read the logs from the folder saying read the logs from vector-logs/date1/hr1/min1? there are other folders also under vector-logs. and the logs are already written in the past. Based on your previous comment , it looks like it's not possible. It needs custom solution. Is that right?

[harshithvd-devops]

@shamj hi, i am using vector to send logs to both s3 and loki but i am facing many issues can you help me out in this
logs per sec = 3-5mb
2026-04-29T04:56:06.632306Z WARN sink{component_kind="sink" component_id=s3_out component_type=aws_s3}:request{request_id=4622}: vector::sinks::util::retries: Retrying after error. error=dispatch failure
2026-04-29T05:34:07.502392Z WARN sink{component_kind="sink" component_id=loki_out component_type=loki}:request{request_id=29583}: vector::sinks::util::retries: Retrying after error. error=Server responded with an error: 500 Internal Server Error
2026-04-29T05:34:08.218702Z WARN sink{component_kind="sink" component_id=loki_out component_type=loki}:request{request_id=29583}: vector::sinks::util::retries: Internal log [Retrying after error.] is being suppressed to avoid flooding.
2026-04-29T06:00:28.308212Z WARN sink{component_kind="sink" component_id=s3_out component_type=aws_s3}:request{request_id=5121}: vector::sinks::util::retries: Retrying after error. error=dispatch failure
2026-04-29T06:10:15.926636Z WARN sink{component_kind="sink" component_id=s3_out component_type=aws_s3}:request{request_id=5199}: vector::sinks::util::retries: Retrying after error. error=dispatch failure
2026-04-29T07:27:55.280605Z WARN sink{component_kind="sink" component_id=loki_out component_type=loki}:request{request_id=35896}: vector::sinks::util::retries: Internal log [Retrying after error.] has been suppressed 1 times.
2026-04-29T07:27:55.280620Z WARN sink{component_kind="sink" component_id=loki_out component_type=loki}:request{request_id=35896}: vector::sinks::util::retries: Retrying after error. error=Server responded with an error: 500 Internal Server Error
2026-04-29T08:48:02.817956Z WARN sink{component_kind="sink" component_id=loki_out component_type=loki}:request{request_id=39957}: vector::sinks::util::retries: Retrying after error. error=Server responded with an error: 500 Internal Server Error
2026-04-29T10:14:38.403139Z WARN sink{component_kind="sink" component_id=s3_out component_type=aws_s3}:request{request_id=7036}: vector::sinks::util::retries: Retrying after error. error=dispatch failure

for some namespace logs are appearing wait in loki and s3 nearly 30 mins to 1 hr

[harshithvd-devops]

@pront plz help me out here

[pront]

For anyone interested in this thread:

• Vector’s aws_s3 source consumes new S3 object notifications from SQS. It cannot scan existing objects or select them by prefix or time range. Reading historical objects requires an external replay process.
• Vector does not use a proprietary file format—the format depends on the S3 sink’s configured encoding and compression. Any reader must use matching settings.
• The later S3/Loki retry errors are a separate issue; please open a new discussion with the Vector version and configuration.

Marking this answered.