Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow to flush data from writecache #1806

Merged
merged 8 commits into from
Sep 28, 2022
Merged

Allow to flush data from writecache #1806

merged 8 commits into from
Sep 28, 2022

Conversation

fyrchik
Copy link
Contributor

@fyrchik fyrchik commented Sep 21, 2022

No description provided.

@codecov
Copy link

codecov bot commented Sep 21, 2022 •
edited
Loading

Codecov Report

Merging #1806 (6e7c1b4) into master (93742d3) will increase coverage by 0.04%.
The diff coverage is 15.66%.

❗ Current head 6e7c1b4 differs from pull request most recent head e5e08ef. Consider uploading reports for the commit e5e08ef to get more accurate results

@@            Coverage Diff             @@
##           master    #1806      +/-   ##
==========================================
+ Coverage   30.95%   31.00%   +0.04%     
==========================================
  Files         372      374       +2     
  Lines       25880    26175     +295     
==========================================
+ Hits         8012     8116     +104     
- Misses      17193    17377     +184     
- Partials      675      682       +7
Impacted Files Coverage Δ
pkg/local_object_storage/engine/writecache.go 0.00% <0.00%> (ø)
pkg/local_object_storage/shard/mode.go 100.00% <ø> (+8.00%) ⬆️
pkg/local_object_storage/shard/writecache.go 0.00% <0.00%> (ø)
pkg/local_object_storage/writecache/options.go 57.14% <ø> (+28.57%) ⬆️
pkg/local_object_storage/writecache/writecache.go 77.08% <ø> (ø)
pkg/services/control/convert.go 0.00% <0.00%> (ø)
pkg/services/control/ir/service.pb.go 22.42% <ø> (ø)
pkg/services/control/ir/service_grpc.pb.go 0.00% <ø> (ø)
pkg/services/control/ir/types.pb.go 35.00% <ø> (ø)
pkg/services/control/rpc.go 0.00% <0.00%> (ø)
... and 9 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

carpawell
carpawell reviewed Sep 22, 2022
View reviewed changes
Copy link
Member

@carpawell carpawell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last commit msg is not capitalized.

pkg/local_object_storage/writecache/flush.go Outdated
// Flush flushes all objects from the write-cache to the main storage.
// Write-cache must be in readonly mode to ensure correctness of an operation and
// to prevent interference with background flush workers.
func (c *cache) Flush() error {
Copy link
Member

@carpawell carpawell Sep 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need any IgnoreErrors like we have in the similar commands?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

pkg/local_object_storage/shard/writecache.go Outdated
return ErrWriteCacheDisabled
}

s.m.Lock()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is RLock enough here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, fixed

cthulhu-rider
cthulhu-rider previously approved these changes Sep 23, 2022
View reviewed changes
pkg/local_object_storage/shard/writecache.go Outdated

// ErrWriteCacheDisabled is returned when an operation on write-cache is performed,
// but write-cache is disabled.
var ErrWriteCacheDisabled = errors.New("write-cache is disabled")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we leave it unexported? I don't see where else we use it. Same for ErrMustBeReadOnly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

carpawell
carpawell previously approved these changes Sep 26, 2022
View reviewed changes
@fyrchik
Copy link
Contributor Author

fyrchik commented Sep 26, 2022

@carpawell Only updated changelog

carpawell
carpawell previously approved these changes Sep 26, 2022
View reviewed changes
@fyrchik
[nspcc-dev#1731] neofs-cli: Make --endpoint flag optional
a429d20 
Allow to provide it using config file.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
@fyrchik
[nspcc-dev#1806] writecache: Use interfaces for blobstor and metabase
2af5c80 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
@fyrchik
[nspcc-dev#1806] writecache: Allow to start flush manually
fd4834f 
Allow user to initiate flushing objects from a writecache.
We need this in 2 cases:
1. During writecache storage schema update, it should be flushed with
   the old version of node and started clean with a new one.
2. During SSD replacement, to avoid data loss.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
@fyrchik
[nspcc-dev#1806] engine: Allow to flush write-cache
c53cc60 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
@fyrchik
[nspcc-dev#1806] services/control: Allow to flush write-cache
b5b6312 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
@fyrchik
[nspcc-dev#1806] neofs-cli: Add control flush-cache command
2fa7345 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
@fyrchik
[nspcc-dev#1806] shard: Check each component mode when setting mode
e25c1ad 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
@fyrchik
[nspcc-dev#1806] writecache: Allow to ignore read errors during flush
e5e08ef 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
@fyrchik
Copy link
Contributor Author

fyrchik commented Sep 27, 2022

Rebased on master

@fyrchik fyrchik merged commit 0a41190 into nspcc-dev:master Sep 28, 2022
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
@fyrchik @aprasolova
[nspcc-dev#1806] writecache: Use interfaces for blobstor and metabase
c1d396b 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
@fyrchik @aprasolova
[nspcc-dev#1806] writecache: Allow to start flush manually
9f4bf6a 
Allow user to initiate flushing objects from a writecache.
We need this in 2 cases:
1. During writecache storage schema update, it should be flushed with
   the old version of node and started clean with a new one.
2. During SSD replacement, to avoid data loss.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
@fyrchik @aprasolova
[nspcc-dev#1806] engine: Allow to flush write-cache
d8ffba6 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
@fyrchik @aprasolova
[nspcc-dev#1806] services/control: Allow to flush write-cache
2f4935b 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
@fyrchik @aprasolova
[nspcc-dev#1806] neofs-cli: Add control flush-cache command
d56c278 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
@fyrchik @aprasolova
[nspcc-dev#1806] shard: Check each component mode when setting mode
e548d2d 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
@fyrchik @aprasolova
[nspcc-dev#1806] writecache: Allow to ignore read errors during flush
1c381a2 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@carpawell carpawell carpawell approved these changes

@realloc realloc Awaiting requested review from realloc

@cthulhu-rider cthulhu-rider Awaiting requested review from cthulhu-rider cthulhu-rider is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.33.0
Development

Successfully merging this pull request may close these issues.
3 participants
@fyrchik @carpawell @cthulhu-rider