An IO error occurs when doing checkpoint (db_paths, cf_paths not supported for checkpoint+backups)

[zaorangyang]

Expected behavior

Checkpoint should be executed correctly when db_paths specified

Actual behavior

It throws a IO error -- "IO error: No such file or directory: while link file to checkpont_dir.tmp/000009.sst: main/000009.sst: No such file or directory"

Steps to reproduce the behavior

Take a look at the following example：

#include "rocksdb/db.h"
#include "rocksdb/utilities/checkpoint.h"
#include <iostream>

int main() {
    rocksdb::DB* db;
    rocksdb::Options options;
    options.create_if_missing = true;
    options.enable_blob_files = true;
    options.min_blob_size = 0;
    std::string dbname = "main";
    options.wal_dir = dbname + "_wal";
    options.db_paths.emplace_back(dbname + "_db", std::numeric_limits<uint64_t>::max());
    rocksdb::Status status = rocksdb::DB::Open(options, dbname, &db);
    assert(status.ok());
    std::string key = "key1";
    std::string value = "value1";
    status = db->Put(rocksdb::WriteOptions(), key, value);
    assert(status.ok());
    rocksdb::FlushOptions flush_opts;
    status = db->Flush(flush_opts);
    assert(status.ok());
    status = db->Get(rocksdb::ReadOptions(), key, &value);
    assert(status.ok());
    
    std::string checkpoint_dir = "checkpont_dir";
    rocksdb::Checkpoint* checkpoint;
    std::cout << "Start Checkpoint" << std::endl;
    rocksdb::Checkpoint::Create(db, &checkpoint);
    status = checkpoint->CreateCheckpoint(checkpoint_dir);
    std::cout << status.ToString() << std::endl;
    delete db;
    delete checkpoint;
}

[zaorangyang]

The root cause is that the path of source SST file is just ‘dbname_/sstname’:

rocksdb/utilities/checkpoint/checkpoint_impl.cc

Line 341 in e3a96c4

s = link_file_cb(db_->GetName(), src_fname, type);

This way can not get the right path of SST/blob file when db_paths specified.

[zaorangyang]

Hi, @ajkr. I'm trying to fix it, please assign it to me. Thank you :)

[autopear]

The same issue may also apply to db_paths, cf_paths, wal_dir and db_log_dir, or any potential path options.

A few things to note in checkpoint_impl:

• How to correctly get the source file path to copy/link.
• Where (target directory) should the file be copied/linked.
• What to do if the source directory is the same as the target directory (copy/link a file to itself).
• What to do if the target directory is inside the checkpoint directory.
  • E.g, checkpoint_dir = "/tmp/checkpoint", wal_dir = "/tmp/checkpoint/wal"

[zaorangyang]

The same issue may also apply to db_paths, cf_paths, wal_dir and db_log_dir, or any potential path options.

A few things to note in checkpoint_impl:

• How to correctly get the source file path to copy/link.

• Where (target directory) should the file be copied/linked.

• What to do if the source directory is the same as the target directory (copy/link a file to itself).

• What to do if the target directory is inside the checkpoint directory.

  • E.g, checkpoint_dir = "/tmp/checkpoint", wal_dir = "/tmp/checkpoint/wal"

How to correctly get the source file path to copy/link.

I think that introducing a new API maybe a good choice. The new API retrieves the all files in the database, just like GetLiveFiles. Different from GetLiveFiles, the files returned by the new API are put into a map. The key is files path, the value is a list of all files in the path.

Where (target directory) should the file be copied/linked.

I think that all the files should be copied/linked the checkpoint dir directly, regardless of the checkpoint dir’s hierarchy. The first reason is simplicity. Besides, The option file generated by checkpoint can not be rewritten currently.

What to do if the source directory is the same as the target directory (copy/link a file to itself).
What to do if the target directory is inside the checkpoint directory.

In fact, the checkpoint dir must not exist in current implementation. So we don't need to think about these cases.

[autopear]

Where (target directory) should the file be copied/linked.

I think that all the files should be copied/linked the checkpoint dir directly, regardless of the checkpoint dir’s hierarchy. The first reason is simplicity. Besides, The option file generated by checkpoint can not be rewritten currently.

I feel it's better to have a CheckpointOptions to allow overriding certain DBOptions when you create a checkpoint. If wal_dir or db_paths or cf_paths are set to some absolute location, it's probably because of some requirements of file organization or performance concern. Then we should provide the options to put those files in a separate location. Of course, this requires rewritten the options file, and that's why we think we can have a lower level API to serialize/deserialize options file #8648 (comment).

What to do if the source directory is the same as the target directory (copy/link a file to itself).
What to do if the target directory is inside the checkpoint directory.

In fact, the checkpoint dir must not exist in current implementation. So we don't need to think about these cases.

My concern is, if we allow overriding those path options, it's possible a user sets them to the same directory. We may assume he/she knows the consequence of this behavior. But in checkpoint_impl, files are only copied/linked from a path specified in the original db's options, to the same options specified in the checkpoint's options. I believe the CopyFile and LinkFile in file_util.cc do not check if the source and destination path are the same. So some validation may need to be added.

Also a minor issue, checkpointing will write to the checkpoint_path + ".tmp". It does not check if there is a file/directory at checkpoint_path + ".tmp". We can change the suffix to a random non-exist location rather than a fixed one.

These are just my findings when I tried to fix this issue. Hope they can help you a little.

[zaorangyang]

Where (target directory) should the file be copied/linked.

I think that all the files should be copied/linked the checkpoint dir directly, regardless of the checkpoint dir’s hierarchy. The first reason is simplicity. Besides, The option file generated by checkpoint can not be rewritten currently.

I feel it's better to have a CheckpointOptions to allow overriding certain DBOptions when you create a checkpoint. If wal_dir or db_paths or cf_paths are set to some absolute location, it's probably because of some requirements of file organization or performance concern. Then we should provide the options to put those files in a separate location. Of course, this requires rewritten the options file, and that's why we think we can have a lower level API to serialize/deserialize options file #8648 (comment).

What to do if the source directory is the same as the target directory (copy/link a file to itself).
What to do if the target directory is inside the checkpoint directory.

In fact, the checkpoint dir must not exist in current implementation. So we don't need to think about these cases.

My concern is, if we allow overriding those path options, it's possible a user sets them to the same directory. We may assume he/she knows the consequence of this behavior. But in checkpoint_impl, files are only copied/linked from a path specified in the original db's options, to the same options specified in the checkpoint's options. I believe the CopyFile and LinkFile in file_util.cc do not check if the source and destination path are the same. So some validation may need to be added.

Also a minor issue, checkpointing will write to the checkpoint_path + ".tmp". It does not check if there is a file/directory at checkpoint_path + ".tmp". We can change the suffix to a random non-exist location rather than a fixed one.

These are just my findings when I tried to fix this issue. Hope they can help you a little.

Hi, autopear. Thanks for your opinion. I agree with you that it's better to have a CheckpointOptions to specify the hierarchy of checkpoint dir. I suggest creating a new issue which introduce the CheckpointOptions after #8648 is addressed.

[mrambacher]

Where (target directory) should the file be copied/linked.

I think that all the files should be copied/linked the checkpoint dir directly, regardless of the checkpoint dir’s hierarchy. The first reason is simplicity. Besides, The option file generated by checkpoint can not be rewritten currently.

I feel it's better to have a CheckpointOptions to allow overriding certain DBOptions when you create a checkpoint. If wal_dir or db_paths or cf_paths are set to some absolute location, it's probably because of some requirements of file organization or performance concern. Then we should provide the options to put those files in a separate location. Of course, this requires rewritten the options file, and that's why we think we can have a lower level API to serialize/deserialize options file #8648 (comment).

What to do if the source directory is the same as the target directory (copy/link a file to itself).
What to do if the target directory is inside the checkpoint directory.

In fact, the checkpoint dir must not exist in current implementation. So we don't need to think about these cases.

My concern is, if we allow overriding those path options, it's possible a user sets them to the same directory. We may assume he/she knows the consequence of this behavior. But in checkpoint_impl, files are only copied/linked from a path specified in the original db's options, to the same options specified in the checkpoint's options. I believe the CopyFile and LinkFile in file_util.cc do not check if the source and destination path are the same. So some validation may need to be added.
Also a minor issue, checkpointing will write to the checkpoint_path + ".tmp". It does not check if there is a file/directory at checkpoint_path + ".tmp". We can change the suffix to a random non-exist location rather than a fixed one.
These are just my findings when I tried to fix this issue. Hope they can help you a little.

Hi, autopear. Thanks for your opinion. I agree with you that it's better to have a CheckpointOptions to specify the hierarchy of checkpoint dir. I suggest creating a new issue which introduce the CheckpointOptions after #8648 is addressed.

@autopear @zaorangyang I do not believe there is a reason to try to specify the directory paths via CheckpointOptions as you suggest. First, I believe this can get outrageously complicated to support. Each ColumnFamily can say where its files are for each level, meaning you would have to have a map of Paths to attempt to cover all of the cases.

[pdillinger]

#8968 would fix this to fail at checkpoint (backup) creation time and declare this as unsupported in the API comments.

I think the best path to basic support for db_paths/cf_paths would be to allow files to be in any configured path, with search starting with the path id in the manifest. This way, you would at least have a functioning DB in the checkpoint directory, or if backup is restored to a single directory.

But there are many places where we need to inject this search of cf_paths, requiring more than a drop-in replacement for TableFileName. 😪