Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PUT input-output errors don't affect shard error counter #1819

Closed
anikeev-yadro opened this issue Sep 28, 2022 · 0 comments · Fixed by #1824
Closed

PUT input-output errors don't affect shard error counter #1819

anikeev-yadro opened this issue Sep 28, 2022 · 0 comments · Fixed by #1824
Assignees
Labels
bug Something isn't working U2 Seriously planned

Comments

@anikeev-yadro
Copy link

I see errors in log:

сен 28 07:12:01 vedi neofs-node[1348]: 2022-09-28T07:12:01.632Z        warn        engine/put.go:129        could not put object in shard        {"shard": "XbWbxKBWqJkioqN4CCwzcP", "error": "could not put object to BLOB storage: write /data/neofs/data2/7/o/o/N/zFqsXZfxBMaoRBhtF46QaaMTtGXkxyAcw8FCiJNu.G6W2M5L6gUJLz2U46Hra4XuLf2FSKMwwoERQHBZKbrc5: input/output error"}
сен 28 07:12:02 vedi neofs-node[1348]: 2022-09-28T07:12:02.881Z        warn        engine/put.go:129        could not put object in shard        {"shard": "XbWbxKBWqJkioqN4CCwzcP", "error": "could not put object to BLOB storage: write /data/neofs/data2/D/N/q/V/vDP7R1t5iAqRaXuWwb2LgfKtXShv6bhDmpBqudnG.DVqGsHRw5YxMQBdz3d7ouoNN9sVUAJV9p3vkFnzG6x8F: input/output error"}
сен 28 07:12:11 vedi neofs-node[1348]: 2022-09-28T07:12:11.397Z        warn        engine/put.go:129        could not put object in shard        {"shard": "XbWbxKBWqJkioqN4CCwzcP", "error": "could not put object to BLOB storage: write /data/neofs/data2/7/k/i/P/5RigUi6rCGvQwrwvqi1eMqwXcLwVCthsKkGNmGeR.DVqGsHRw5YxMQBdz3d7ouoNN9sVUAJV9p3vkFnzG6x8F: input/output error"}
сен 28 07:12:21 vedi neofs-node[1348]: 2022-09-28T07:12:21.252Z        warn        engine/put.go:129        could not put object in shard        {"shard": "XbWbxKBWqJkioqN4CCwzcP", "error": "could not put object to BLOB storage: write /data/neofs/data2/A/Z/H/N/EYo6tzicFFipWE3zwJwJ7nBHNpZxuUSTtUPsRKpT.AZHjbXXssZperzZbRjVXjgNamnuMeFfZCuUYM9bwBzg7: input/output error"}
сен 28 07:13:40 vedi neofs-node[1348]: 2022-09-28T07:13:40.011Z        warn        engine/put.go:129        could not put object in shard        {"shard": "XbWbxKBWqJkioqN4CCwzcP", "error": "could not put object to BLOB storage: write /data/neofs/data2/K/w/h/v/Nw1GZzuSTjDt4ZSyH48cV8WiakuCNJhv77nDk1k.GGBYm5zFoGXf874N6LV4yiYhEkT7kBUzFtNo5sd2pmaT: input/output error"}

but it doesn't lead to grow errors count on shard:

admin@vedi:~$ sudo neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json control shards list
Enter password >
Shard XbWbxKBWqJkioqN4CCwzcP:
Mode: read-write
Metabase: /srv/neofs/meta0/metabase2.db
Blobstor: /data/neofs/data2
Write-cache: /srv/neofs/meta0/write_cache2
Pilorama: /srv/neofs/meta0/pilorama2.db
Error count: 0

Expected Behavior

Any input-output errors on shard should be counted for put shard in degradated mode when count will exceed threshold.

Current Behavior

Shard input-output errors doesn't counted.

Log: bug_shard_errors.zip

Versions:

NeoFS Storage node
Version: v0.32.0-12-gc7f85994
GoVersion: go1.18.4

Your Environment

Server setup and configuration:
cloud, 4 VMs, 4 SN, 4 http qw, 4 s3 gw

Operating System and version (uname -a):
linux vedi 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

aprasolova pushed a commit to aprasolova/neofs-node that referenced this issue Oct 19, 2022
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this issue Oct 19, 2022
Add a common error for this case because it is not an error
which should increase error counter. Single error simplifies checks on
the call-site.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this issue Oct 19, 2022
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working U2 Seriously planned
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants