Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix/put on busy shards #2965

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Fix/put on busy shards #2965

wants to merge 2 commits into from

Conversation

carpawell
Copy link
Member

No description provided.

@carpawell carpawell self-assigned this Oct 7, 2024
@codecov Codecov
Copy link

codecov bot commented Oct 7, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 38.46154% with 32 lines in your changes missing coverage. Please review.

Project coverage is 23.47%. Comparing base (c4bdae1) to head (12246c2).

Files with missing lines Patch % Lines
pkg/local_object_storage/engine/put.go 34.69% 31 Missing and 1 partial ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #2965      +/-   ##
==========================================
- Coverage   23.47%   23.47%   -0.01%     
==========================================
  Files         780      780              
  Lines       46655    46688      +33     
==========================================
+ Hits        10953    10958       +5     
- Misses      34833    34859      +26     
- Partials      869      871       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@carpawell
node/engine/put: if a shard cannot handle PUT, add its ID to logs
34d187a 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
@carpawell
node/engine: do not skip objects if shards are busy
12246c2 
If every shard's pool is overloaded with routines, choose the best one and try
to PUT an object to it 30 seconds. Closes #2871.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
@carpawell carpawell marked this pull request as ready for review October 16, 2024 15:40
@carpawell
Copy link
Member Author

Well, it works. There is not much we can do about testing currently. It is hard to test in general because it is time-dependent. But it is quite straightforward code, and the tests are okay.

@carpawell
Copy link
Member Author

Full run: https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/1997-1729097220/index.html

roman-khimov
roman-khimov reviewed Oct 22, 2024
View reviewed changes
Copy link
Member

@roman-khimov roman-khimov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's an improvement, but channels/routines would be even better.

pkg/local_object_storage/engine/put.go
}

func (e *StorageEngine) putToShardWithDeadLine(sh hashedShard, ind int, pool util.WorkerPool, addr oid.Address, prm PutPrm) error {
const deadline = 30 * time.Second
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make it configurable? Treat zero as old behavior?

pkg/local_object_storage/engine/put.go
err = errPutShard
err = e.putToShardWithDeadLine(bestShard, 0, bestPool, addr, prm)
if err != nil {
e.log.Warn("last stand to put object to the best shard",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't deserve a warn to me.

pkg/local_object_storage/engine/put.go
@@ -167,19 +214,20 @@ func (e *StorageEngine) putToShard(sh hashedShard, ind int, pool util.WorkerPool
return
}

e.reportShardError(sh, "could not put object to shard", err)
e.reportShardError(sh, "could not put object to shard", errGlobal)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be left as is?

pkg/local_object_storage/engine/put.go
defer timer.Stop()

const putCooldown = 100 * time.Millisecond
ticker := time.NewTicker(putCooldown)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the reasons I don't like pools we have now. Channels provide natural queueing and blocking/unblocking.

@carpawell
Copy link
Member Author

but channels/routines would be even better

@roman-khimov, what do you mean by that? Sending tasks via channels to every shard? What is the difference?

@roman-khimov
Copy link
Member

How can I get a "block for 30s" behavior with a pool? How can I get a queue with a pool? That's the difference.

@roman-khimov
Copy link
Member

MaxBlockingTasks allows to have some queue, maybe we can expose it through the config and test the behavior this way? I think we can have active threads with bigger queues at shard level (although it depends SSD/HDD) instead of current huge pools and no queue. Time to poll-block can be tuned too, we can make additional tests to see how it affects operation.

@carpawell carpawell mentioned this pull request Nov 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@roman-khimov roman-khimov roman-khimov left review comments

@cthulhu-rider cthulhu-rider Awaiting requested review from cthulhu-rider cthulhu-rider is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@carpawell carpawell

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@carpawell @roman-khimov