Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bbolt's read-only status is interpreted as a disk error #2057

Closed
carpawell opened this issue Nov 14, 2022 · 3 comments
Closed

Bbolt's read-only status is interpreted as a disk error #2057

carpawell opened this issue Nov 14, 2022 · 3 comments
Assignees
@carpawell
Labels
bug Something isn't working neofs-storage Storage node application issues U3 Regular

Comments

@carpawell
Copy link
Member

Some race is here. Also, I doubt moving DB to read-only mode by a node is a disk error.

Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.149Z        debug        blobovniczatree/control.go:24        read-only mode, skip blobovniczas initialization...
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.149Z        info        engine/engine.go:105        shard is moved in degraded mode due to error threshold        {"shard_id": "UdNgukPGsHGaTvDp7tGtz3", "error count": 30}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.158Z        warn        engine/engine.go:153        could not put object to active blobovnicza        {"shard_id": "UdNgukPGsHGaTvDp7tGtz3", "error count": 31, "error": "database not open"}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.158Z        debug        blobovnicza/control.go:17        creating directory for BoltDB        {"component": "Blobovnicza", "path": "/srv/neofs/data0/blobovnicza/3/0", "ro": true}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.158Z        debug        blobovnicza/control.go:31        opening BoltDB        {"component": "Blobovnicza", "path": "/srv/neofs/data0/blobovnicza/3/0", "permissions": "-rwxrwxrwx"}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.158Z        debug        blobovniczatree/blobovnicza.go:201        blobovnicza successfully activated        {"path": "3/0"}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.158Z        info        engine/engine.go:105        shard is moved in degraded mode due to error threshold        {"shard_id": "UdNgukPGsHGaTvDp7tGtz3", "error count": 31}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.169Z        warn        engine/engine.go:153        could not put object to active blobovnicza        {"shard_id": "UdNgukPGsHGaTvDp7tGtz3", "error count": 32, "error": "database is in read-only mode"}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.169Z        debug        blobovnicza/control.go:17        creating directory for BoltDB        {"component": "Blobovnicza", "path": "/srv/neofs/data0/blobovnicza/1/0", "ro": true}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.169Z        debug        blobovnicza/control.go:31        opening BoltDB        {"component": "Blobovnicza", "path": "/srv/neofs/data0/blobovnicza/1/0", "permissions": "-rwxrwxrwx"}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.169Z        debug        blobovniczatree/blobovnicza.go:201        blobovnicza successfully activated        {"path": "1/0"}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.169Z        info        engine/engine.go:105        shard is moved in degraded mode due to error threshold        {"shard_id": "UdNgukPGsHGaTvDp7tGtz3", "error count": 32}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.179Z        warn        engine/engine.go:153        could not put object to active blobovnicza        {"shard_id": "UdNgukPGsHGaTvDp7tGtz3", "error count": 33, "error": "database is in read-only mode"}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.179Z        debug        blobovnicza/control.go:17        creating directory for BoltDB        {"component": "Blobovnicza", "path": "/srv/neofs/data0/blobovnicza/2/0", "ro": true}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.179Z        debug        blobovnicza/control.go:31        opening BoltDB        {"component": "Blobovnicza", "path": "/srv/neofs/data0/blobovnicza/2/0", "permissions": "-rwxrwxrwx"}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.179Z        debug        blobovniczatree/blobovnicza.go:201        blobovnicza successfully activated        {"path": "2/0"}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.179Z        info        engine/engine.go:105        shard is moved in degraded mode due to error threshold        {"shard_id": "UdNgukPGsHGaTvDp7tGtz3", "error count": 33}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.190Z        warn        engine/engine.go:153        could not put object to active blobovnicza        {"shard_id": "UdNgukPGsHGaTvDp7tGtz3", "error count": 34, "error": "database is in read-only mode"}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.190Z        warn        engine/engine.go:153        can't flush an object to blobstor        {"shard_id": "UdNgukPGsHGaTvDp7tGtz3", "error count": 35, "error": "could not save the object in any blobovnicza"}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.190Z        debug        engine/engine.go:176        mode change is in progress, ignoring set-mode request        {"shard_id": "UdNgukPGsHGaTvDp7tGtz3", "error_count": 35}
Nov 14 02:42:42 az neofs-node[14468]: 2022-11-14T02:42:42.190Z        info        engine/engine.go:105        shard is moved in degraded mode due to error threshold        {"shard_id": "UdNgukPGsHGaTvDp7tGtz3", "error count": 34}
@carpawell carpawell added bug Something isn't working neofs-storage Storage node application issues 2022Q4 labels Nov 14, 2022
@alexchetaev alexchetaev added U3 Regular and removed 2022Q4 labels Nov 14, 2022
@fyrchik
Copy link
Contributor

fyrchik commented Nov 15, 2022

We could take mode mutex on read for all operations and return pre-defined ErrReadOnly error.

@carpawell carpawell added U2 Seriously planned and removed U3 Regular labels Nov 15, 2022
@carpawell carpawell self-assigned this Nov 15, 2022
@carpawell carpawell mentioned this issue Nov 15, 2022
Merged
carpawell added a commit that referenced this issue Nov 15, 2022
@carpawell
[#2057] blobstor: Block operations on a mode change
479601c 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
carpawell added a commit that referenced this issue Nov 15, 2022
@carpawell
[#2057] meta: Do not lock the whole meta on GET
e8d401e 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
carpawell added a commit that referenced this issue Nov 15, 2022
@carpawell
[#2057] meta: Fix concurrent mode changes
02676f0 
Includes:
1. mode change read lock operation in every exported method that r/w the
underlying database;
2. returning `ErrDegradedMode` logical error if any exported method is
called in degraded (without a metabase) mode.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
carpawell added a commit that referenced this issue Nov 15, 2022
@carpawell
[#2057] meta: Fail write operations in R/O mode
7cc0986 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
@fyrchik
Copy link
Contributor

fyrchik commented Nov 17, 2022

I coudn't come up with an example of then this can happen (after the fix from my previous comment).

@alexchetaev alexchetaev added U3 Regular and removed U2 Seriously planned labels Nov 18, 2022
@carpawell
Copy link
Member Author

Closed via #2059.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@carpawell carpawell

Labels
bug Something isn't working neofs-storage Storage node application issues U3 Regular
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@fyrchik @carpawell @alexchetaev