Put Storagegroup failed #2488

vvarg229 · 2023-08-09T16:21:33Z

neofs-cli storagegroup put failed:

COMMAND: neofs-cli --config /home/runner/work/neofs-node/neofs-node/neofs-testcases/wallet_config.yml storagegroup put --rpc-endpoint 's01.neofs.devenv:8080' --wallet '/home/runner/work/neofs-node/neofs-node/neofs-testcases/TemporaryDir/7744cda6-e7da-4f7e-84e0-670fcc23cda7.json' --cid '7mpsMFCYq69qVyYsbvFFDGEbhqbf5axyV8p5BmAxqhcY' --members 'ALPamBLkRXvMV8M94TRxvADtJKceGKLZQ6wyJJz2UNbp' --lifetime 10
RETCODE: 1

STDOUT:
could not collect storage group members: data must be 64-bytes long

STDERR:

Start / End / Elapsed	 18:15:12.153743 / 18:15:12.575976 / 0:00:00.422233

See in the report:
https://http.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/424-1691517028/index.html#suites/69b447004f3c682f67955ec900a09cf0/e7df9c93693e0c5/

Expected Behavior

neofs-cli storagegroup put failed

Current Behavior

neofs-cli storagegroup put should not fall

Steps to Reproduce (for bugs)

Run any TestStorageGroup test

Regression

Yes
PR# that caused this regression:
#2470

Your Environment

neo-go 0.101.1
neofs-s3-authmate 0.27.1
neofs-cli d67d5d3
neofs-adm d67d5d3
AWS aws-cli/2.13.5 Python/3.11.4 Linux/5.15.0-1042-azure exe/x86_64.ubuntu.22 prompt/off

The text was updated successfully, but these errors were encountered:

Previously, `storagegroup.CollectMembers` function could return error with text `data must be 64-bytes long`. According to the message it is not clear which data is incorrect. Now context has been added to the error so that it is clear at what stage it occurs. Refs #2488. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>

Previously, `storagegroup.CollectMembers` function could return error with text `data must be 64-bytes long` if any of the split members were without homomorphic checksum or with the invalid one. Including the bypass of the splitting chain, and the error occurred already after at stage `tz.Concat`. The correct format of the homomorphic checksum is required to form the group, so it is required to do an in-place check. Check each element of `IterateSplitLeaves` iterator and break traversal on any checksum problem. Refs #2488. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>

Previously, `storagegroup.CollectMembers` function ignored absence of the object ID in the member's child header. This could lead to incorrect storage group formation, and also hid problem objects in the system (ID is a required field for any object). Immediately return error from `CollectMembers` if any child of any member is without ID. Refs #2488. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>

Previously, `storagegroup.CollectMembers` function could return error with text `data must be 64-bytes long`. According to the message it is not clear which data is incorrect. Now context has been added to the error so that it is clear at what stage it occurs. Refs #2488. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>

Previously, `storagegroup.CollectMembers` function could return error with text `data must be 64-bytes long` if any of the split members were without homomorphic checksum or with the invalid one. Including the bypass of the splitting chain, and the error occurred already after at stage `tz.Concat`. The correct format of the homomorphic checksum is required to form the group, so it is required to do an in-place check. Check each element of `IterateSplitLeaves` iterator and break traversal on any checksum problem. Refs #2488. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>

Previously, `storagegroup.CollectMembers` function ignored absence of the object ID in the member's child header. This could lead to incorrect storage group formation, and also hid problem objects in the system (ID is a required field for any object). Immediately return error from `CollectMembers` if any child of any member is without ID. Refs #2488. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>

cthulhu-rider · 2023-08-10T13:15:19Z

homomorphic checksums of group members are empty or inaccessible (rather the first)

$ neofs-cli -c bin/cli.yml storagegroup put --cid 9oVtcNVCHquor6Qz2xMQy2RFGmx4jC9n3jvv32yRFNLw --lifetime 100 --members 4Qu97DCjz1HnwLdzETA2VQoQQ9nwzSGrjqwq5v6nKnfk
could not collect storage group members: collect split-chain for member #0: missing homomorphic checksum in member's child header '4Qu97DCjz1HnwLdzETA2VQoQQ9nwzSGrjqwq5v6nKnfk'

$ neofs-cli -c bin/cli.yml container get --cid 9oVtcNVCHquor6Qz2xMQy2RFGmx4jC9n3jvv32yRFNLw
container ID: 9oVtcNVCHquor6Qz2xMQy2RFGmx4jC9n3jvv32yRFNLw
owner ID: Nhfg3TbpwogLvDGVvAvqyThbsHgoSUKwtn
basic ACL: 1c8c8ccc (private)
       RangeHASH    Range      Search     Delete     Put        Head       Get
0 1    1 1 0 0      1 0 0 0    1 1 0 0    1 0 0 0    1 1 0 0    1 1 0 0    1 1 0 0
X F    U S O B      U S O B    U S O B    U S O B    U S O B    U S O B    U S O B
  X-Sticky F-Final U-User S-System O-Others B-Bearer
created: 2023-08-10 16:19:11 +0400 +04
attributes:
	Timestamp=1691669951
placement policy:
REP 1

$ neofs-cli -c bin/cli.yml object head --cid 9oVtcNVCHquor6Qz2xMQy2RFGmx4jC9n3jvv32yRFNLw --oid 4Qu97DCjz1HnwLdzETA2VQoQQ9nwzSGrjqwq5v6nKnfk
ID: 4Qu97DCjz1HnwLdzETA2VQoQQ9nwzSGrjqwq5v6nKnfk
CID: 9oVtcNVCHquor6Qz2xMQy2RFGmx4jC9n3jvv32yRFNLw
Owner: Nhfg3TbpwogLvDGVvAvqyThbsHgoSUKwtn
CreatedAt: 10
Size: 8
HomoHash: <empty>
Checksum: 491ae7d9f42782128f8e635e42e54433b22694c8337cfd07638420a6d30affd9
Type: REGULAR
Attributes:
  FileName=VERSION
  Timestamp=1691669983 (2023-08-10 16:19:43 +0400 +04)
ID signature:
  public key: 021312fdacd6510603578cdc4e03753780de1c0e308e26077ab55bb1f29726c2d6
  signature: 04bcdab0b79b369d0973619428ba83c3a420a78722067c7007010420b0cdf0683d074e95e542ec59c5d5e47032b79c30dedf7c038ce1609b49a16bbac4a8df36e4

$ neofs-cli -c bin/cli.yml netmap netinfo
Epoch: 23
Network magic: [net 0x3c2d] 15405
Time per block: 1s
NeoFS network configuration (system)
  Audit fee: 10000
  Storage price: 100000000
  Container fee: 1000
  EigenTrust alpha: 0.1
  Number of EigenTrust iterations: 4
  Epoch duration: 240
  Inner Ring candidate fee: 10000000000
  Maximum object size: 67108864
  Withdrawal fee: 100000000
  Homomorphic hashing disabled: false
  Maintenance mode allowed: false
NeoFS network configuration (other)
  SystemDNS: 636f6e7461696e6572

seems like homomorphic checksums are not stamped in the split-chain elements. I'll investigate why

Previously, storage nodes didn't check homomorphic checksum correctness in incoming objects. When homomorphic hashing is required in the container, nodes accepted objects without homomorphic hash header. This could lead to subsequent problems with storage group creation because it expects checksums in the members' headers. Make storage nodes to check that homomorphic checksum of the new object payload is set and has correct format according to TillichZemor algo. If homomorphic hashing is disabled, the header is unchecked. Exact TZ hash value is unchecked for now due to performance issues. Refs #2488. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>

Previously, storage nodes didn't calculate and set homomorphic payload checksum for the sliced objects even when it was required. This could affect Data Audit subsystem working with homomorphic hashes. Bypass hashing option from `slicingTarget` to the underlying `Slicer`. Fixes #2488. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>

Previously, storage nodes didn't check homomorphic checksum correctness in incoming objects. When homomorphic hashing is required in the container, nodes accepted objects without homomorphic hash header. This could lead to subsequent problems with storage group creation because it expects checksums in the members' headers. Make storage nodes to check that homomorphic checksum of the new object payload is set and has correct format according to TillichZemor algo. If homomorphic hashing is disabled, the header is unchecked. Exact TZ hash value is unchecked for now due to performance issues. Refs #2488. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>

vvarg229 added bug Something isn't working triage labels Aug 9, 2023

cthulhu-rider self-assigned this Aug 9, 2023

roman-khimov added this to the v0.38.0 milestone Aug 9, 2023

cthulhu-rider mentioned this issue Aug 10, 2023

go.mod: Upgrade to latest github.com/nspcc-dev/neofs-sdk-go #2489

Merged

cthulhu-rider removed the triage label Aug 10, 2023

roman-khimov closed this as completed in 98a0a7e Aug 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Put Storagegroup failed #2488

Put Storagegroup failed #2488

vvarg229 commented Aug 9, 2023

cthulhu-rider commented Aug 10, 2023

Put Storagegroup failed #2488

Put Storagegroup failed #2488

Comments

vvarg229 commented Aug 9, 2023

Expected Behavior

Current Behavior

Steps to Reproduce (for bugs)

Regression

Your Environment

cthulhu-rider commented Aug 10, 2023