Continuous replication #965
Conversation
Codecov Report
@@ Coverage Diff @@
## master #965 +/- ##
=======================================
Coverage 38.32% 38.32%
=======================================
Files 258 258
Lines 13473 13473
=======================================
Hits 5163 5163
Misses 7878 7878
Partials 432 432 Continue to review full report at Codecov.
|
cmd/neofs-node/config.go
Outdated
|
|
||
| pool.putRemote, err = ants.NewPool(pool.putRemoteCapacity, optNonBlocking) | ||
| if err != nil { | ||
| fatalOnErr(err) |
There was a problem hiding this comment.
if statement can be omitted.
| "go.uber.org/zap" | ||
| ) | ||
|
|
||
| // NodeLoader provides functions to fetch application load statistic. | ||
| type nodeLoader interface { |
There was a problem hiding this comment.
| type nodeLoader interface { | |
| type NodeLoader interface { |
There was a problem hiding this comment.
Should it be public? I am not sure it needs to be exported somewhere out of policer package.
pkg/services/policer/policer.go
Outdated
| "go.uber.org/zap" | ||
| ) | ||
|
|
||
| // NodeLoader provides functions to fetch application load statistic. |
There was a problem hiding this comment.
| // NodeLoader provides functions to fetch application load statistic. | |
| // NodeLoader provides application load statistics. |
| func (p *Policer) Run(ctx context.Context) { | ||
| defer func() { |
There was a problem hiding this comment.
Were these logs so useless?
There was a problem hiding this comment.
Nope, accidentally removed it. Put it back.
pkg/services/policer/process.go
Outdated
| addrs, cursor, err = p.jobQueue.Select(cursor, p.batchSize) | ||
| if err != nil { | ||
| if errors.Is(err, engine.ErrEndOfListing) { | ||
| time.Sleep(1 * time.Second) // finished whole cycle, sleep a bit |
There was a problem hiding this comment.
| time.Sleep(1 * time.Second) // finished whole cycle, sleep a bit | |
| time.Sleep(time.Second) // finished whole cycle, sleep a bit |
| time.Sleep(1 * time.Second) // finished whole cycle, sleep a bit | ||
| continue | ||
| } | ||
| p.log.Warn("failure at object select for replication", zap.Error(err)) |
There was a problem hiding this comment.
| p.log.Warn("failure at object select for replication", zap.Error(err)) | |
| p.log.Warn("failure at object select for replication", zap.String("error", err.Error())) |
and \n matters.
There was a problem hiding this comment.
Actually, \n are not produced anymore, even in wrapped errors! I think we simplified logger structure a while ago and it is not a case to worry now.
a := errors.New("deep error")
b := fmt.Errorf("included error: %w", a)
d := fmt.Errorf("included error two: %w", b)
c.log.Info("error", zap.Error(d)) // logger with config from neofs-dev-env
// 2021-11-24T09:50:09.841Z info neofs-node/main.go:59 error {"error": "included error two: included error: deep error"} There was a problem hiding this comment.
Sorry, I meant \n is source code.
There was a problem hiding this comment.
Buy anyway, with current logger configuration, zap.Error is totally legit, though it produced some ugly output before and we stopped using it.
| p.cache.Add(addrStr, time.Now()) | ||
| }) | ||
| if err != nil { | ||
| p.log.Warn("pool submission", zap.Error(err)) |
pkg/services/policer/policer.go
Outdated
| "go.uber.org/zap" | ||
| ) | ||
|
|
||
| // NodeLoader provides functions to fetch application load statistic. | ||
| type nodeLoader interface { | ||
| // ObjectServiceLoad returns object service load value. |
There was a problem hiding this comment.
| // ObjectServiceLoad returns object service load value. | |
| // ObjectServiceLoad returns object service load value in [0:1] range. |
|
|
||
| p.prevTask.undone-- | ||
| if p.taskPool.Cap() != newCapacity { |
There was a problem hiding this comment.
Required check? Thought maybe pool.Tune does nothing if same capacity is passed.
There was a problem hiding this comment.
This check is mostly for logger, to avoid log messages every second when capacity didn't change.
|
We can keep control of pool capacity value on the app-side, and on replicator's side only control the frequency of updates. In theory, you can even leave the periodicity on the side of the application. |
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Continues replication executed in separate pool of goroutines, so there is no need in worker to handle replication tasks asynchronously. Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Yes, both are viable options. I can't say what are pros to stick with one or another option, but they will definitely work fine. |
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Continues replication executed in separate pool of goroutines, so there is no need in worker to handle replication tasks asynchronously. Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Blocked by #948