Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot delete big objects with dynamic session #1978

Closed
anikeev-yadro opened this issue Oct 26, 2022 · 8 comments · Fixed by #1989 or #2196
Closed

Cannot delete big objects with dynamic session #1978

anikeev-yadro opened this issue Oct 26, 2022 · 8 comments · Fixed by #1989 or #2196
Assignees
@carpawell
Labels
bug Something isn't working U3 Regular

Comments

@anikeev-yadro
Copy link

Expected Behavior

Big objects can be deleted if we use dynamic session

Current Behavior

Cannot delete the big object if we use dynamic session

Steps to Reproduce (for bugs)

  1. Create dynamic session
  2. Create container
COMMAND: neofs-cli --config /jenkins/workspace/sbercloud_test/tmp.togrBj08uO/neofs-testcases/wallet_config.yml container create --rpc-endpoint '172.26.160.33:8080' --wallet '/jenkins/workspace/sbercloud_test/tmp.togrBj08uO/neofs-testcases/TemporaryDir/3b64acdb-7573-484d-a0d5-d0d5b451def9.json' --await --policy 'REP 1 IN LOC_STO_PLACE CBF 1 SELECT 1 FROM LOC_STO AS LOC_STO_PLACE FILTER "UN-LOCODE" EQ "SE STO" AS LOC_STO'
RETCODE: 0

STDOUT:
container ID: 9t3D3hnc7SZ8RA35Y1Rw9hnq73VzXhsf37J72BCPSxEd
awaiting...
container has been persisted on sidechain

STDERR:

Start / End / Elapsed	 07:13:21.057703 / 07:13:30.470498 / 0:00:09.412795
  1. Put object
COMMAND: neofs-cli --config /jenkins/workspace/sbercloud_test/tmp.togrBj08uO/neofs-testcases/wallet_config.yml object put --rpc-endpoint '172.26.160.33:8080' --wallet '/jenkins/workspace/sbercloud_test/tmp.togrBj08uO/neofs-testcases/TemporaryDir/3b64acdb-7573-484d-a0d5-d0d5b451def9.json' --cid '9t3D3hnc7SZ8RA35Y1Rw9hnq73VzXhsf37J72BCPSxEd' --file '/jenkins/workspace/sbercloud_test/tmp.togrBj08uO/neofs-testcases/TemporaryDir/5b4d7ab7-f5a8-414d-8d40-a780570160c6' --no-progress
RETCODE: 0

STDOUT:
[/jenkins/workspace/sbercloud_test/tmp.togrBj08uO/neofs-testcases/TemporaryDir/5b4d7ab7-f5a8-414d-8d40-a780570160c6] Object successfully stored
  OID: dFezaajq1qwLbQTDQpNptx5o3JkpgD5Fb5jeFESGGtM
  CID: 9t3D3hnc7SZ8RA35Y1Rw9hnq73VzXhsf37J72BCPSxEd

STDERR:

Start / End / Elapsed	 07:13:46.596161 / 07:14:02.332016 / 0:00:15.735855
  1. Try to delete object
COMMAND: neofs-cli --config /jenkins/workspace/sbercloud_test/tmp.togrBj08uO/neofs-testcases/wallet_config.yml object delete --rpc-endpoint '172.26.163.34:8080' --wallet '/jenkins/workspace/sbercloud_test/tmp.togrBj08uO/neofs-testcases/TemporaryDir/3b64acdb-7573-484d-a0d5-d0d5b451def9.json' --cid '9t3D3hnc7SZ8RA35Y1Rw9hnq73VzXhsf37J72BCPSxEd' --oid 'dFezaajq1qwLbQTDQpNptx5o3JkpgD5Fb5jeFESGGtM' --session '/jenkins/workspace/sbercloud_test/tmp.togrBj08uO/neofs-testcases/TemporaryDir/c6a24be7-f9dc-43d8-8143-8a836e6e733a'
RETCODE: 1

STDOUT:
rpc error: remove object via client: status: code = 2049 message = object not found
STDERR:

Start / End / Elapsed	 07:14:18.312092 / 07:14:19.097065 / 0:00:00.784973

Logs: fdcca35015d969d6.zip
Versions:

NeoFS Storage node
Version: v0.33.0-27-g7d456cb4

Your Environment
Server setup and configuration:
cloud, 4 VMs, 4 SN, 4 http qw, 4 s3 gw

Operating System and version (uname -a):
linux vedi 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux
@anikeev-yadro
Copy link
Author

May be related to #1894

@cthulhu-rider cthulhu-rider self-assigned this Oct 26, 2022
@cthulhu-rider
Copy link
Contributor

cthulhu-rider commented Oct 26, 2022 •
edited
Loading

Request fails because currently dynamic session includes only root (virtual, "big") object. The remote server doesn't have rights to form the tombstone since container is private. So, in used enviroment, the container must be at least public-read.

Overall, there are several problems:

1.Dynamic session in CLI doesn't include all physical ("small") objects from the removing object's split-chain
2. Dynamic sessions are opened in background which is really unexpected. I would expect them only if some flag like --trust is provided.
3. Intermediate node (outside the container) tries to form the object instead of forwarding the request to the container nodes.

I'm going to start with (1) but other problems also must be resolved.

@anikeev-yadro
Copy link
Author

There is the same old issue #1697

cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Oct 27, 2022
@cthulhu-rider
[nspcc-dev#1978] cli/object: Gather all related object in delete session
cb86e08 
Object removal session should reflect all objects related to the
removing one.

Make `OpenSessionViaClient` to gather the split members of the original
object in order to spread the session to them.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Oct 27, 2022
@cthulhu-rider
[nspcc-dev#1978] cli/object: Gather all related object in delete session
988475a 
Object removal session should reflect all objects related to the
removing one.

Make `OpenSessionViaClient` to gather the split members of the original
object in order to spread the session to them.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
@cthulhu-rider cthulhu-rider mentioned this issue Oct 27, 2022
Merged
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Oct 27, 2022
@cthulhu-rider
[nspcc-dev#1978] cli/object: Gather all related object in delete session
24dd213 
Object removal session should reflect all objects related to the
removing one.

Make `OpenSessionViaClient` to gather the split members of the original
object in order to spread the session to them.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
cthulhu-rider added a commit to cthulhu-rider/neofs-node that referenced this issue Oct 28, 2022
@cthulhu-rider
[nspcc-dev#1978] cli/object: Gather all related object in delete session
224ef4c 
Object removal session should reflect all objects related to the
removing one.

Make `OpenSessionViaClient` to gather the split members of the original
object in order to spread the session to them.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
@cthulhu-rider cthulhu-rider mentioned this issue Oct 31, 2022
Open
@anikeev-yadro
Copy link
Author

Maybe related to #1970

@anikeev-yadro anikeev-yadro mentioned this issue Nov 3, 2022
Closed
@anikeev-yadro
Copy link
Author

Reproduced on version 0.34.0-6-gabf4a635 at the devenv and virtual environment.

Steps to Reproduce:
1.Create dynamic session
2.Create container

COMMAND: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.XA2o0JzjWd/neofs-testcases/wallet_config.yml container create --rpc-endpoint '172.26.160.238:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.XA2o0JzjWd/neofs-testcases/TemporaryDir/06af1c2d-703d-47b6-bd17-566106a1e1a3.json' --await --policy 'REP 1 IN LOC_SPB_PLACE CBF 1 SELECT 1 FROM LOC_SPB AS LOC_SPB_PLACE FILTER "UN-LOCODE" EQ "RU LED" AS LOC_SPB'
RETCODE: 0

STDOUT:
container ID: EuC3GAUXDiwV45qLr53mjQ4WfTiycVUhRiRYh9y6fkGD
awaiting...
container has been persisted on sidechain

STDERR:

Start / End / Elapsed	 18:27:22.237665 / 18:27:31.650867 / 0:00:09.413202

3.Put object

COMMAND: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.XA2o0JzjWd/neofs-testcases/wallet_config.yml object put --rpc-endpoint '172.26.163.228:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.XA2o0JzjWd/neofs-testcases/TemporaryDir/06af1c2d-703d-47b6-bd17-566106a1e1a3.json' --cid 'EuC3GAUXDiwV45qLr53mjQ4WfTiycVUhRiRYh9y6fkGD' --file '/jenkins/workspace/sbercloud_functional_tests/tmp.XA2o0JzjWd/neofs-testcases/TemporaryDir/7e4f71bb-d83d-4a5a-ad8f-89b7476e8107' --no-progress --session '/jenkins/workspace/sbercloud_functional_tests/tmp.XA2o0JzjWd/neofs-testcases/TemporaryDir/ccfb3347-d24f-40a2-a0d7-383af8f77527'
RETCODE: 0

STDOUT:
[/jenkins/workspace/sbercloud_functional_tests/tmp.XA2o0JzjWd/neofs-testcases/TemporaryDir/7e4f71bb-d83d-4a5a-ad8f-89b7476e8107] Object successfully stored
  OID: B7YJBM6tREFctRdEAJHyxJXeMWo4HxhaciSNF5iTwXn
  CID: EuC3GAUXDiwV45qLr53mjQ4WfTiycVUhRiRYh9y6fkGD

STDERR:

Start / End / Elapsed	 18:28:01.293864 / 18:28:15.935664 / 0:00:14.641800

4.Try to delete object

COMMAND: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.XA2o0JzjWd/neofs-testcases/wallet_config.yml object delete --rpc-endpoint '172.26.163.228:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.XA2o0JzjWd/neofs-testcases/TemporaryDir/06af1c2d-703d-47b6-bd17-566106a1e1a3.json' --cid 'EuC3GAUXDiwV45qLr53mjQ4WfTiycVUhRiRYh9y6fkGD' --oid 'jRJFCJHxeh8HbYYNVTJ7UZnEuAxvMtDiwGVcRFHgMHL' --session '/jenkins/workspace/sbercloud_functional_tests/tmp.XA2o0JzjWd/neofs-testcases/TemporaryDir/ccfb3347-d24f-40a2-a0d7-383af8f77527'
RETCODE: 1

STDOUT:
rpc error: remove object via client: status: code = 2049 message = object not found
STDERR:

Start / End / Elapsed	 18:28:15.937799 / 18:28:16.768377 / 0:00:00.830578

Logs: ecb16cff9939217d.zip

@carpawell carpawell assigned carpawell and unassigned cthulhu-rider Nov 9, 2022
@alexvanin alexvanin mentioned this issue Nov 9, 2022
Closed
@carpawell carpawell mentioned this issue Nov 9, 2022
Merged
@alexchetaev alexchetaev added U3 Regular and removed U0 Needs to be resolved immediately 2022Q4 labels Nov 11, 2022
@fyrchik
Copy link
Contributor

fyrchik commented Nov 12, 2022

Closed via #2036

@anikeev-yadro
Copy link
Author

Reproduced today on version v0.34.0-142-g8744d6ab-dirty on devenv an virtual environments.
Test log: b7f5d073bbf670ae.txt
Nodes log: 723a3185ab902f73.zip

@carpawell carpawell mentioned this issue Jan 9, 2023
Merged
@carpawell
Copy link
Member

Should be closed via #2196.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@carpawell carpawell

Labels
bug Something isn't working U3 Regular
Projects
None yet
Milestone
No milestone
5 participants
@fyrchik @carpawell @alexchetaev @cthulhu-rider @anikeev-yadro