[skipci]feat: [curve/toos-v2] add bs list dir #2082

Sindweller · 2022-11-17T17:01:02Z

What problem does this PR solve?

Issue Number: #2038

Problem Summary:

What is changed and how it works?

What's Changed:

support list dir command in curve tool

Usage:  curve bs list dir [flags]

list directory information in curvebs

Flags:
  -c, --conf string           config file (default is $HOME/.curve/curve.yaml or /etc/curve/curve.yaml)
      --dir string            directory path
  -f, --format string         output format (json|plain) (default "plain")
  -h, --help                  print help
      --mdsaddr string        mds address, should be like 127.0.0.1:6700,127.0.0.1:6701,127.0.0.1:6702
      --password string       user password
      --rpcretrytimes int32   rpc retry times (default 1)
      --rpctimeout duration   rpc timeout (default 10s)
      --showerror             display all errors in command
      --user string           user name
      --verbose               show some log

Examples:
$ curve bs list dir --dir /

How it Works:

curve bs list dir --dir /

root@de7603f17cf9:/# ./curve bs list dir --dir /
+------+-------------+----------+-----------------+------------+---------------------+---------------+-------------+
|  ID  |  FILENAME   | PARENTID |    FILETYPE     |   OWNER    |        CTIME        | ALLOCATEDSIZE |  FILESIZE   |
+------+-------------+----------+-----------------+------------+---------------------+---------------+-------------+
| 1    | /RecycleBin | 0        | INODE_DIRECTORY | root       | 2022-11-12 16:38:25 | size:0 B      | size:0 B    |
+------+-------------+          +-----------------+------------+---------------------+               +-------------+
| 1004 | /dir11      |          | INODE_PAGEFILE  | test       | 2022-11-18 14:39:30 |               | size:20 GiB |
+------+-------------+          +                 +            +---------------------+               +             +
| 1005 | /dir12      |          |                 |            | 2022-11-18 14:44:00 |               |             |
+------+-------------+          +                 +------------+---------------------+               +-------------+
| 3    | /playground |          |                 | playground | 2022-06-27 17:35:28 |               | size:10 GiB |
+------+-------------+          +                 +------------+---------------------+               +-------------+
| 1003 | /test       |          |                 | test       | 2022-11-18 14:36:17 |               | size:20 GiB |
+------+-------------+          +                 +            +---------------------+               +-------------+
| 1006 | /test22     |          |                 |            | 2022-11-18 14:44:12 |               | size:10 GiB |
+------+-------------+----------+-----------------+------------+---------------------+---------------+-------------+

Side effects(Breaking backward compatibility? Performance regression?):

Check List

Relevant documentation/comments is changed or added
I acknowledge that all my contributions will be made under the project's license

Sindweller · 2022-11-17T17:30:19Z

I'm trying to implement this cmd refer to

curve_ops_tool list -fileName=test
  output:
fileName: "RecycleBin"
parentId: 0
fileType: INODE_DIRECTORY
owner: "root"
ctime: *********
allocated size: 0GB
file size: 0GB

To get the dir info I use ListDir interface in proto. It returns with []*FileInfo

type FileInfo struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Id          *uint64     `protobuf:"varint,1,opt,name=id" json:"id,omitempty"`
	FileName    *string     `protobuf:"bytes,2,opt,name=fileName" json:"fileName,omitempty"`
	ParentId    *uint64     `protobuf:"varint,3,opt,name=parentId" json:"parentId,omitempty"`
	FileType    *FileType   `protobuf:"varint,4,opt,name=fileType,enum=curve.mds.FileType" json:"fileType,omitempty"`
	Owner       *string     `protobuf:"bytes,5,opt,name=owner" json:"owner,omitempty"`
	ChunkSize   *uint32     `protobuf:"varint,6,opt,name=chunkSize" json:"chunkSize,omitempty"`
	SegmentSize *uint32     `protobuf:"varint,7,opt,name=segmentSize" json:"segmentSize,omitempty"`
	Length      *uint64     `protobuf:"varint,8,opt,name=length" json:"length,omitempty"`
	Ctime       *uint64     `protobuf:"varint,9,opt,name=ctime" json:"ctime,omitempty"`
	SeqNum      *uint64     `protobuf:"varint,10,opt,name=seqNum" json:"seqNum,omitempty"`
	FileStatus  *FileStatus `protobuf:"varint,11,opt,name=fileStatus,enum=curve.mds.FileStatus" json:"fileStatus,omitempty"`
	//用于文件转移到回收站的情况下恢复场景下的使用,
	//RecycleBin（回收站）目录下使用/其他场景下不使用
	OriginalFullPathName *string `protobuf:"bytes,12,opt,name=originalFullPathName" json:"originalFullPathName,omitempty"`
	// cloneSource 当前用于存放克隆源(当前主要用于curvefs)
	// 后期可以考虑存放 s3相关信息
	CloneSource *string `protobuf:"bytes,13,opt,name=cloneSource" json:"cloneSource,omitempty"`
	// cloneLength 克隆源文件的长度，用于clone过程中进行extent
	CloneLength    *uint64             `protobuf:"varint,14,opt,name=cloneLength" json:"cloneLength,omitempty"`
	StripeUnit     *uint64             `protobuf:"varint,15,opt,name=stripeUnit" json:"stripeUnit,omitempty"`
	StripeCount    *uint64             `protobuf:"varint,16,opt,name=stripeCount" json:"stripeCount,omitempty"`
	ThrottleParams *FileThrottleParams `protobuf:"bytes,17,opt,name=throttleParams" json:"throttleParams,omitempty"`
	Epoch          *uint64             `protobuf:"varint,18,opt,name=epoch" json:"epoch,omitempty"`
}

I was confused in allocated size: 0GB and file size: 0GB, how should I get or compute these two field?

Maybe I should call GetAllocatedSize and GetFileSize for each fileinfo from tools-v2/pkg/cli/command/curvebs/query/file/?

Any other comments on my code are welcome, many thanks.

Cyber-SiKu · 2022-11-18T02:33:16Z

curve bs list dir --fileName /test --> curve bs list dir --dir=/ is better?

Cyber-SiKu · 2022-11-18T02:42:11Z

PLS add some output, let's see the result.

Sindweller · 2022-11-18T02:47:46Z

curve bs list dir --fileName /test --> curve bs list dir --dir=/ is better?

Agree. I'm working on it. When it works as expected, I'll complete the output.

thx.

Cyber-SiKu · 2022-11-18T02:50:35Z

I'm trying to implement this cmd refer to

curve_ops_tool list -fileName=test
  output:
fileName: "RecycleBin"
parentId: 0
fileType: INODE_DIRECTORY
owner: "root"
ctime: *********
allocated size: 0GB
file size: 0GB

To get the dir info I use ListDir interface in proto. It returns with []*FileInfo

type FileInfo struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Id          *uint64     `protobuf:"varint,1,opt,name=id" json:"id,omitempty"`
	FileName    *string     `protobuf:"bytes,2,opt,name=fileName" json:"fileName,omitempty"`
	ParentId    *uint64     `protobuf:"varint,3,opt,name=parentId" json:"parentId,omitempty"`
	FileType    *FileType   `protobuf:"varint,4,opt,name=fileType,enum=curve.mds.FileType" json:"fileType,omitempty"`
	Owner       *string     `protobuf:"bytes,5,opt,name=owner" json:"owner,omitempty"`
	ChunkSize   *uint32     `protobuf:"varint,6,opt,name=chunkSize" json:"chunkSize,omitempty"`
	SegmentSize *uint32     `protobuf:"varint,7,opt,name=segmentSize" json:"segmentSize,omitempty"`
	Length      *uint64     `protobuf:"varint,8,opt,name=length" json:"length,omitempty"`
	Ctime       *uint64     `protobuf:"varint,9,opt,name=ctime" json:"ctime,omitempty"`
	SeqNum      *uint64     `protobuf:"varint,10,opt,name=seqNum" json:"seqNum,omitempty"`
	FileStatus  *FileStatus `protobuf:"varint,11,opt,name=fileStatus,enum=curve.mds.FileStatus" json:"fileStatus,omitempty"`
	//用于文件转移到回收站的情况下恢复场景下的使用,
	//RecycleBin（回收站）目录下使用/其他场景下不使用
	OriginalFullPathName *string `protobuf:"bytes,12,opt,name=originalFullPathName" json:"originalFullPathName,omitempty"`
	// cloneSource 当前用于存放克隆源(当前主要用于curvefs)
	// 后期可以考虑存放 s3相关信息
	CloneSource *string `protobuf:"bytes,13,opt,name=cloneSource" json:"cloneSource,omitempty"`
	// cloneLength 克隆源文件的长度，用于clone过程中进行extent
	CloneLength    *uint64             `protobuf:"varint,14,opt,name=cloneLength" json:"cloneLength,omitempty"`
	StripeUnit     *uint64             `protobuf:"varint,15,opt,name=stripeUnit" json:"stripeUnit,omitempty"`
	StripeCount    *uint64             `protobuf:"varint,16,opt,name=stripeCount" json:"stripeCount,omitempty"`
	ThrottleParams *FileThrottleParams `protobuf:"bytes,17,opt,name=throttleParams" json:"throttleParams,omitempty"`
	Epoch          *uint64             `protobuf:"varint,18,opt,name=epoch" json:"epoch,omitempty"`
}

I was confused in allocated size: 0GB and file size: 0GB, how should I get or compute these two field?

Maybe I should call GetAllocatedSize and GetFileSize for each fileinfo from tools-v2/pkg/cli/command/curvebs/query/file/?

Any other comments on my code are welcome, many thanks.

I'm trying to implement this cmd refer to

curve_ops_tool list -fileName=test
  output:
fileName: "RecycleBin"
parentId: 0
fileType: INODE_DIRECTORY
owner: "root"
ctime: *********
allocated size: 0GB
file size: 0GB

To get the dir info I use ListDir interface in proto. It returns with []*FileInfo

type FileInfo struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Id          *uint64     `protobuf:"varint,1,opt,name=id" json:"id,omitempty"`
	FileName    *string     `protobuf:"bytes,2,opt,name=fileName" json:"fileName,omitempty"`
	ParentId    *uint64     `protobuf:"varint,3,opt,name=parentId" json:"parentId,omitempty"`
	FileType    *FileType   `protobuf:"varint,4,opt,name=fileType,enum=curve.mds.FileType" json:"fileType,omitempty"`
	Owner       *string     `protobuf:"bytes,5,opt,name=owner" json:"owner,omitempty"`
	ChunkSize   *uint32     `protobuf:"varint,6,opt,name=chunkSize" json:"chunkSize,omitempty"`
	SegmentSize *uint32     `protobuf:"varint,7,opt,name=segmentSize" json:"segmentSize,omitempty"`
	Length      *uint64     `protobuf:"varint,8,opt,name=length" json:"length,omitempty"`
	Ctime       *uint64     `protobuf:"varint,9,opt,name=ctime" json:"ctime,omitempty"`
	SeqNum      *uint64     `protobuf:"varint,10,opt,name=seqNum" json:"seqNum,omitempty"`
	FileStatus  *FileStatus `protobuf:"varint,11,opt,name=fileStatus,enum=curve.mds.FileStatus" json:"fileStatus,omitempty"`
	//用于文件转移到回收站的情况下恢复场景下的使用,
	//RecycleBin（回收站）目录下使用/其他场景下不使用
	OriginalFullPathName *string `protobuf:"bytes,12,opt,name=originalFullPathName" json:"originalFullPathName,omitempty"`
	// cloneSource 当前用于存放克隆源(当前主要用于curvefs)
	// 后期可以考虑存放 s3相关信息
	CloneSource *string `protobuf:"bytes,13,opt,name=cloneSource" json:"cloneSource,omitempty"`
	// cloneLength 克隆源文件的长度，用于clone过程中进行extent
	CloneLength    *uint64             `protobuf:"varint,14,opt,name=cloneLength" json:"cloneLength,omitempty"`
	StripeUnit     *uint64             `protobuf:"varint,15,opt,name=stripeUnit" json:"stripeUnit,omitempty"`
	StripeCount    *uint64             `protobuf:"varint,16,opt,name=stripeCount" json:"stripeCount,omitempty"`
	ThrottleParams *FileThrottleParams `protobuf:"bytes,17,opt,name=throttleParams" json:"throttleParams,omitempty"`
	Epoch          *uint64             `protobuf:"varint,18,opt,name=epoch" json:"epoch,omitempty"`
}

I was confused in allocated size: 0GB and file size: 0GB, how should I get or compute these two field?

Maybe I should call GetAllocatedSize and GetFileSize for each fileinfo from tools-v2/pkg/cli/command/curvebs/query/file/?

Any other comments on my code are welcome, many thanks.

Yep, you need to add the corresponding command

ilixiaocui · 2022-11-18T02:51:44Z

I was confused in allocated size: 0GB and file size: 0GB, how should I get or compute these two field?

Maybe I should call GetAllocatedSize and GetFileSize for each fileinfo from tools-v2/pkg/cli/command/curvebs/query/file/?

Any other comments on my code are welcome, many thanks.

Yes，call GetAllocatedSize like

curve/src/tools/namespace_tool.cpp

Line 213 in f856f13

int NameSpaceTool::GetAndPrintAllocSize(const std::string& fileName) {

and GetFileSize like

curve/src/tools/namespace_tool.cpp

Line 198 in f856f13

int NameSpaceTool::GetAndPrintFileSize(const std::string& fileName) {

.

Sindweller · 2022-11-18T10:50:15Z

PTAL @Cyber-SiKu @ilixiaocui 👀

tools-v2/pkg/cli/command/curvebs/list/dir/dir.go

tools-v2/pkg/config/bs.go

Sindweller · 2022-11-22T05:18:26Z

Done.

root@de7603f17cf9:/# ./curve bs list dir
+------+-------------+----------+-----------------+------------+---------------------+---------------+----------+
|  ID  |  FILENAME   | PARENTID |    FILETYPE     |   OWNER    |        CTIME        | ALLOCATEDSIZE | FILESIZE |
+------+-------------+----------+-----------------+------------+---------------------+---------------+----------+
| 3    | /playground | 0        | INODE_PAGEFILE  | playground | 2022-06-27 17:35:28 | 0 B           | 10 GiB   |
+------+-------------+          +-----------------+------------+---------------------+---------------+----------+
| 1    | /RecycleBin |          | INODE_DIRECTORY | root       | 2022-11-12 16:38:25 | 0 B           | 0 B      |
+------+-------------+          +-----------------+------------+---------------------+---------------+----------+
| 1004 | /dir11      |          | INODE_PAGEFILE  | test       | 2022-11-18 14:39:30 | 0 B           | 20 GiB   |
+------+-------------+          +                 +            +---------------------+---------------+----------+
| 1005 | /dir12      |          |                 |            | 2022-11-18 14:44:00 | 0 B           | 20 GiB   |
+------+-------------+          +                 +            +---------------------+---------------+----------+
| 1003 | /test       |          |                 |            | 2022-11-18 14:36:17 | 0 B           | 20 GiB   |
+------+-------------+          +                 +            +---------------------+---------------+----------+
| 1006 | /test22     |          |                 |            | 2022-11-18 14:44:12 | 0 B           | 10 GiB   |
+------+-------------+----------+-----------------+------------+---------------------+---------------+----------+

Signed-off-by: Sindweller <sindweller5530@gmail.com>

Sindweller force-pushed the master branch from 443ac8b to cde95ec Compare November 17, 2022 17:05

ilixiaocui self-requested a review November 18, 2022 01:43

Sindweller force-pushed the master branch from b0f5b73 to 9709b62 Compare November 18, 2022 10:43

Sindweller changed the title ~~feat: [curve/toos-v2] add bs list dir~~ [skipci]feat: [curve/toos-v2] add bs list dir Nov 18, 2022

Sindweller marked this pull request as ready for review November 18, 2022 10:46

Cyber-SiKu approved these changes Nov 22, 2022

View reviewed changes

Sindweller force-pushed the master branch 2 times, most recently from 7266dc4 to 00de601 Compare November 22, 2022 03:07

Cyber-SiKu reviewed Nov 22, 2022

View reviewed changes

tools-v2/pkg/cli/command/curvebs/list/dir/dir.go Outdated Show resolved Hide resolved

tools-v2/pkg/cli/command/curvebs/list/dir/dir.go Outdated Show resolved Hide resolved

tools-v2/pkg/config/bs.go Show resolved Hide resolved

feat: [tools-v2] bs list dir

372d8ce

Signed-off-by: Sindweller <sindweller5530@gmail.com>

Sindweller force-pushed the master branch from edc98bc to 372d8ce Compare November 22, 2022 05:19

ilixiaocui approved these changes Nov 22, 2022

View reviewed changes

ilixiaocui merged commit d92e770 into opencurve:master Nov 22, 2022

ilixiaocui mentioned this pull request Nov 22, 2022

[2022 Q4 Developer Activities]: Call For Participation！ #2017

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[skipci]feat: [curve/toos-v2] add bs list dir #2082

[skipci]feat: [curve/toos-v2] add bs list dir #2082

Sindweller commented Nov 17, 2022 •

edited

Loading

Sindweller commented Nov 17, 2022

Cyber-SiKu commented Nov 18, 2022

Cyber-SiKu commented Nov 18, 2022

Sindweller commented Nov 18, 2022

Cyber-SiKu commented Nov 18, 2022

ilixiaocui commented Nov 18, 2022 •

edited

Loading

Sindweller commented Nov 18, 2022

Sindweller commented Nov 22, 2022

[skipci]feat: [curve/toos-v2] add bs list dir #2082

[skipci]feat: [curve/toos-v2] add bs list dir #2082

Conversation

Sindweller commented Nov 17, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Sindweller commented Nov 17, 2022

Cyber-SiKu commented Nov 18, 2022

Cyber-SiKu commented Nov 18, 2022

Sindweller commented Nov 18, 2022

Cyber-SiKu commented Nov 18, 2022

ilixiaocui commented Nov 18, 2022 • edited Loading

Sindweller commented Nov 18, 2022

Sindweller commented Nov 22, 2022

Sindweller commented Nov 17, 2022 •

edited

Loading

ilixiaocui commented Nov 18, 2022 •

edited

Loading