Is there a problem with multi-level directory collection? #21319
Labels
meta: awaiting author
Pull requests that are awaiting their author.
source: file
Anything `file` source related
type: bug
A code related bug.
A note for the community
Problem
Collection link
file -vector -kafka
The directory hierarchy for data collection is 10
Problem phenomenon
There is a problem of data loss during data collection
A total of 3000 files, with one line written to each file, only 949 lines were collected
kafka logs:
{"@timestamp":"2024-09-19T05:49:44.874343145Z","file":"/opt/file1/20240919/1/2/tmp_20240919_1.log","host":"760b54e990e1","message":"20240919134945_1","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874380045Z","file":"/opt/file1/20240919/1/2/tmp_20240919_10.log","host":"760b54e990e1","message":"20240919134945_10","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874396715Z","file":"/opt/file1/20240919/1/2/tmp_20240919_2.log","host":"760b54e990e1","message":"20240919134945_2","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874406722Z","file":"/opt/file1/20240919/1/2/tmp_20240919_3.log","host":"760b54e990e1","message":"20240919134945_3","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874416272Z","file":"/opt/file1/20240919/1/2/tmp_20240919_4.log","host":"760b54e990e1","message":"20240919134945_4","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874425632Z","file":"/opt/file1/20240919/1/2/tmp_20240919_5.log","host":"760b54e990e1","message":"20240919134945_5","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874434169Z","file":"/opt/file1/20240919/1/2/tmp_20240919_6.log","host":"760b54e990e1","message":"20240919134945_6","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874444929Z","file":"/opt/file1/20240919/1/2/tmp_20240919_7.log","host":"760b54e990e1","message":"20240919134945_7","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874454575Z","file":"/opt/file1/20240919/1/2/tmp_20240919_8.log","host":"760b54e990e1","message":"20240919134945_8","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874466055Z","file":"/opt/file1/20240919/1/2/tmp_20240919_9.log","host":"760b54e990e1","message":"20240919134945_9","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874474832Z","file":"/opt/file1/20240919/1/tmp_20240919_100.log","host":"760b54e990e1","message":"20240919134945_100","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874483879Z","file":"/opt/file1/20240919/1/tmp_20240919_11.log","host":"760b54e990e1","message":"20240919134945_11","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874494239Z","file":"/opt/file1/20240919/1/tmp_20240919_12.log","host":"760b54e990e1","message":"20240919134945_12","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874502815Z","file":"/opt/file1/20240919/1/tmp_20240919_13.log","host":"760b54e990e1","message":"20240919134945_13","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874511982Z","file":"/opt/file1/20240919/1/tmp_20240919_14.log","host":"760b54e990e1","message":"20240919134945_14","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874520819Z","file":"/opt/file1/20240919/1/tmp_20240919_15.log","host":"760b54e990e1","message":"20240919134945_15","offset":0,"source_type":"file"}
{"@timestamp":"2024-09-19T05:49:44.874531702Z","file":"/opt/file1/20
加载更多
Configuration
Version
vector 0.40.1
Debug Output
No response
Example Data
20240919134945_1
Additional Context
is Vector running in docker
References
No response
The text was updated successfully, but these errors were encountered: