Skip to content

BR proceeds when there are running jobs #4975

Closed
@nebula-bots

Description

@nebula-bots

Please check the FAQ documentation before raising an issue

Describe the bug (required)

We use br to backup the cluster when there are running balance data jobs(we ensure that the job is started before br backup and still running after backup finished), we expecte br to report error and exit, but it just proceeds and continue backup, the backup data might be inconsistent in this situation:

"time":"2022-12-02T15:29:46.055Z","uri":"s3://test/graph1/BACKUP_2022_12_02_15_29_16/data/71/103"}
{"file":"github.com/vesoft-inc/nebula-agent@v0.1.2-0.20220615054614-af5197e7d961/pkg/client/client.go:49","func":"github.com/vesoft-inc/nebula-agent/pkg/client.New","level":"debug","msg":"Dialing to address store5:8888.","time":"2022-12-02T15:29:46.055Z"}
{"file":"github.com/vesoft-inc/nebula-agent@v0.1.2-0.20220615054614-af5197e7d961/pkg/storage/s3.go:312","func":"github.com/vesoft-inc/nebula-agent/pkg/storage.(*S3).GetDir","level":"debug","msg":"Get backend for s3://test/graph1/BACKUP_2022_12_02_15_29_16/data/71/133 successfully.","time":"2022-12-02T15:29:46.055Z","uri":"s3://test/graph1/BACKUP_2022_12_02_15_29_16/data/71/133"}
{"file":"github.com/vesoft-inc/nebula-agent@v0.1.2-0.20220615054614-af5197e7d961/pkg/client/client.go:49","func":"github.com/vesoft-inc/nebula-agent/pkg/client.New","level":"debug","msg":"Dialing to address store1:8888.","time":"2022-12-02T15:29:46.055Z"}
{"file":"github.com/vesoft-inc/nebula-agent@v0.1.2-0.20220615054614-af5197e7d961/pkg/storage/s3.go:312","func":"github.com/vesoft-inc/nebula-agent/pkg/storage.(*S3).GetDir","level":"debug","msg":"Get backend for s3://test/graph1/BACKUP_2022_12_02_15_29_16/data/71/186 successfully.","time":"2022-12-02T15:29:46.056Z","uri":"s3://test/graph1/BACKUP_2022_12_02_15_29_16/data/71/186"}
{"file":"github.com/vesoft-inc/nebula-agent@v0.1.2-0.20220615054614-af5197e7d961/pkg/client/client.go:49","func":"github.com/vesoft-inc/nebula-agent/pkg/client.New","level":"debug","msg":"Dialing to address store4:8888.","time":"2022-12-02T15:29:46.056Z"}
{"file":"github.com/vesoft-inc/nebula-agent@v0.1.2-0.20220615054614-af5197e7d961/pkg/storage/s3.go:312","func":"github.com/vesoft-inc/nebula-agent/pkg/storage.(*S3).GetDir","level":"debug","msg":"Get backend for s3://test/graph1/BACKUP_2022_12_02_15_29_16/data/71/74 successfully.","time":"2022-12-02T15:29:46.056Z","uri":"s3://test/graph1/BACKUP_2022_12_02_15_29_16/data/71/74"}
{"file":"github.com/vesoft-inc/nebula-agent@v0.1.2-0.20220615054614-af5197e7d961/pkg/client/client.go:49","func":"github.com/vesoft-inc/nebula-agent/pkg/client.New","level":"debug","msg":"Dialing to address store4:8888.","time":"2022-12-02T15:29:46.056Z"}
{"file":"github.com/vesoft-inc/nebula-agent@v0.1.2-0.20220615054614-af5197e7d961/pkg/storage/s3.go:312","func":"github.com/vesoft-inc/nebula-agent/pkg/storage.(*S3).GetDir","level":"debug","msg":"Get backend for s3://test/graph1/BACKUP_2022_12_02_15_29_16/data/71/158 successfully.","time":"2022-12-02T15:29:46.056Z","uri":"s3://test/graph1/BACKUP_2022_12_02_15_29_16/data/71/158"}
{"file":"github.com/vesoft-inc/nebula-agent@v0.1.2-0.20220615054614-af5197e7d961/pkg/client/client.go:49","func":"github.com/vesoft-inc/nebula-agent/pkg/client.New","level":"debug","msg":"Dialing to address store5:8888.","time":"2022-12-02T15:29:46.056Z"}
{"file":"github.com/vesoft-inc/nebula-agent@v0.1.2-0.20220615054614-af5197e7d961/pkg/storage/s3.go:312","func":"github.com/vesoft-inc/nebula-agent/pkg/storage.(*S3).GetDir","level":"debug","msg":"Get backend for s3://test/graph1/BACKUP_2022_12_02_15_29_16/data/71/203 successfully.","time":"2022-12-02T15:29:46.056Z","uri":"s3://test/graph1/BACKUP_2022_12_02_15_29_16/data/71/203"}
{"data":"s3://test/graph1/BACKUP_2022_12_02_15_29_16/data","file":"github.com/vesoft-inc/br-ent@/pkg/backup/full.go:118","func":"github.com/vesoft-inc/br-ent/pkg/backup.(*Backup).FullBackup","level":"info","msg":"Upload data backup successfully.","name":"BACKUP_2022_12_02_15_29_16","time":"2022-12-02T15:29:51.398Z"}
{"file":"github.com/vesoft-inc/br-ent@/pkg/backup/full.go:134","func":"github.com/vesoft-inc/br-ent/pkg/backup.(*Backup).FullBackup","level":"info","msg":"Write meta data to local tmp file successfully.","name":"BACKUP_2022_12_02_15_29_16","time":"2022-12-02T15:29:51.399Z","tmp path":"/tmp/nebula-br/BACKUP_2022_12_02_15_29_16.meta"}
{"file":"github.com/vesoft-inc/nebula-agent@v0.1.2-0.20220615054614-af5197e7d961/pkg/storage/s3.go:172","func":"github.com/vesoft-inc/nebula-agent/pkg/storage.(*S3).uploadToStorage","level":"debug","msg":"Upload from /tmp/nebula-br/BACKUP_2022_12_02_15_29_16.meta to graph1/BACKUP_2022_12_02_15_29_16/BACKUP_2022_12_02_15_29_16.meta successfully.","time":"2022-12-02T15:29:51.401Z"}
{"file":"github.com/vesoft-inc/br-ent@/pkg/backup/full.go:143","func":"github.com/vesoft-inc/br-ent/pkg/backup.(*Backup).FullBackup","level":"info","msg":"Upload tmp backup meta file to remote.","name":"BACKUP_2022_12_02_15_29_16","remote path":"s3://test/graph1/BACKUP_2022_12_02_15_29_16/BACKUP_2022_12_02_15_29_16.meta","time":"2022-12-02T15:29:51.401Z"}
{"file":"github.com/vesoft-inc/br-ent@/pkg/backup/full.go:151","func":"github.com/vesoft-inc/br-ent/pkg/backup.(*Backup).FullBackup","level":"info","msg":"Drop backup in cluster and local tmp folder successfully.","name":"BACKUP_2022_12_02_15_29_16","time":"2022-12-02T15:29:56.828Z"}
Full backup succeed.

Your Environments (required)

  • OS: uname -a
  • Compiler: g++ --version or clang++ --version
  • CPU: lscpu
  • Commit id (e.g. a3ffc7d8) af513f9621154870e02ee865e67bf5bed5326b77 release-3.3

How To Reproduce(required)

Steps to reproduce the behavior:

  1. Step 1
  2. Step 2
  3. Step 3

Expected behavior

Additional context

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions