Cleanup checkpoints for file source

[anton-ryzhov]

As I understand checkpoint files are never removed from the data_dir. This could cause denial of service because all inodes will be in use.

This even couldn't be handled externally because the application always rewrites these files with newer timestamps — impossible to remove old ones.

Vector should remove checkpoints for nonexistent files after some time.

[binarylogic]

Thanks! I agree. We'll get this implemented.

[anton-ryzhov]

I also don't really get why do you store offsets this way.

According to strace you're doing lots of syscalls. mkdir then open+close for each file to store checkpoints. And then lstat, open and getdents64 the directory, unlink for every file to remove the folder before saving new checkpoints.

That creates lots of IO requests, inodes are used more and more.

Isn't it better to store checkpoints in one file (as json or whatever) and update its content. For that case garbage collection won't be so necessary, only one inode will be used.

[binarylogic]

@anton-ryzhov that's very possible. I agree, how we store offsets is not ideal and we absolutely will be improving this. We've also discussed using leveldb since we include that for disk buffers. I like your idea of a single JSON file as well.