Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Share functionality of placement building between components #1803

Open
cthulhu-rider opened this issue Sep 20, 2022 · 4 comments
Open

Share functionality of placement building between components #1803

cthulhu-rider opened this issue Sep 20, 2022 · 4 comments
Labels
enhancement Improving existing functionality I4 No visible changes neofs-cli NeoFS CLI application issues neofs-ir Inner Ring node application issues neofs-storage Storage node application issues S2 Regular significance U3 Regular
Milestone

Comments

@cthulhu-rider
Copy link
Contributor

Object placement is build using netmap.Source and container.Source in most scenarios. Also there are some similar cases when we use same placement approach, e.g. shard evacuation introduced here and remote placement builder.

I propose to concentrate all placement functionality in single shared code space in order to easily reuse it in new tasks.

@fyrchik
Copy link
Contributor

fyrchik commented Sep 20, 2022

With a proper interface, tree server can use this component too.

@roman-khimov roman-khimov modified the milestones: v0.35.0, v0.36.0 Jan 26, 2023
@roman-khimov roman-khimov modified the milestones: v0.36.0, v0.38.0 Apr 7, 2023
@roman-khimov roman-khimov modified the milestones: v0.38.0, v0.39.0 Aug 15, 2023
@roman-khimov roman-khimov modified the milestones: v0.39.0, v0.41.0 Nov 28, 2023
@roman-khimov roman-khimov added enhancement Improving existing functionality U4 Nothing urgent S2 Regular significance I4 No visible changes and removed refactor U4 Nothing urgent labels Dec 21, 2023
@cthulhu-rider
Copy link
Contributor Author

caching is very meaningful for app performance and efficiency. Some attempts were made, but they are buggy, so lets also fix them

suggest to w8 4 prelim work finish

@roman-khimov roman-khimov added the blocked Can't be done because of something label Mar 5, 2024
@roman-khimov roman-khimov removed the blocked Can't be done because of something label Apr 1, 2024
@roman-khimov roman-khimov modified the milestones: v0.41.0, v0.42.0 Apr 1, 2024
@roman-khimov roman-khimov modified the milestones: v0.42.0, v0.43.0 May 22, 2024
@roman-khimov roman-khimov modified the milestones: v0.43.0, v0.44.0 Aug 5, 2024
cthulhu-rider added a commit that referenced this issue Aug 9, 2024
Continues 2f29338 for `ObjectService`'s
`Put` server handler. It shares the cache with other object processors.

Refs #1803.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Aug 9, 2024
Continues 5389a1e for `ObjectService`'s
`Put` server handler. It shares the cache with other object processors.

Refs #1803.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Aug 9, 2024
Continues 5389a1e for `ObjectService`'s
`Put` server handler. It shares the cache with other object processors.

Refs #1803.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Aug 19, 2024
Continues 5389a1e for `ObjectService`'s
`Put` server handler. It shares the cache with other object processors.

Refs #1803.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Aug 19, 2024
Continues 5389a1e for `ObjectService`'s
`Put` server handler. It shares the cache with other object processors.

Refs #1803.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Aug 21, 2024
Continues 5389a1e for `ObjectService`'s
`Put` server handler. It shares the cache with other object processors.

Refs #1803.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Aug 23, 2024
Continues 5389a1e for `ObjectService`'s
`Put` server handler. It shares the cache with other object processors.

Refs #1803.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Aug 30, 2024
Continues 5389a1e for `ObjectService`'s
`Put` server handler. It shares the cache with other object processors.

Refs #1803.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
@roman-khimov
Copy link
Member

Do we have anything left here?

@cthulhu-rider
Copy link
Contributor Author

for SN:

  1. ControlService.Evacuate
    nm, err := s.netMapSrc.GetNetMap(0)
    if err != nil {
    return err
    }
    c, err := s.cnrSrc.Get(cid)
    if err != nil {
    return err
    }
    ns, err := nm.ContainerNodes(c.Value.PlacementPolicy(), cid)
    if err != nil {
    return fmt.Errorf("can't build a list of container nodes")
    }
  2. object ACL
    cnrVectors, err := nm.ContainerNodes(cnr.PlacementPolicy(), idCnr)
  3. routing for container volume estimations
    func (l *loadPlacementBuilder) buildPlacement(epoch uint64, idCnr cid.ID) ([][]netmap.NodeInfo, *netmap.NetMap, error) {
    cnr, err := l.cnrSrc.Get(idCnr)
    if err != nil {
    return nil, nil, err
    }
    nm, err := l.nmSrc.GetNetMapByEpoch(epoch)
    if err != nil {
    return nil, nil, fmt.Errorf("could not get network map: %w", err)
    }
    cnrNodes, err := nm.ContainerNodes(cnr.Value.PlacementPolicy(), idCnr)
    if err != nil {
    return nil, nil, fmt.Errorf("could not build container nodes: %w", err)
    }
    return cnrNodes, nm, nil
    }
  4. Tree service (multi-op)
    nm, err := s.nmSource.GetNetMap(0)
    if err != nil {
    return nil, -1, fmt.Errorf("can't get netmap: %w", err)
    }
    cnr, err := s.cnrSource.Get(cid)
    if err != nil {
    return nil, -1, fmt.Errorf("can't get container: %w", err)
    }
    cidStr := cid.String()
    s.containerCache.Lock()
    if s.containerCache.nm != nm {
    s.containerCache.lru.Purge()
    } else if item, ok := s.containerCache.lru.Get(cidStr); ok {
    if item.cnr == cnr {
    s.containerCache.Unlock()
    return item.nodes, item.local, nil
    }
    }
    s.containerCache.Unlock()
    policy := cnr.Value.PlacementPolicy()
    cntNodes, err := nm.ContainerNodes(policy, cid)

object ACL should definitely use shared policy component. Evacuator is rarely used, but better to use it too. Tree and estimations... i wouldnt bother them tbh

finally, whole package should be purged

IR and CLI also apply storage policies to network maps for various needs. Caching doesn't help them, so I would leave everything as is

P.S. refs can be found by .ContainerNodes(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improving existing functionality I4 No visible changes neofs-cli NeoFS CLI application issues neofs-ir Inner Ring node application issues neofs-storage Storage node application issues S2 Regular significance U3 Regular
Projects
None yet
Development

No branches or pull requests

3 participants