New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge tree service in master #1607

Merged

fyrchik merged 46 commits into master from tree-service

Jul 21, 2022

Contributor

fyrchik commented Jul 18, 2022

Only the last 2 commits are fresh.
Tree service must now be explicitly enabled via tree.enabled config field.

codecov bot commented Jul 18, 2022 •

edited

Loading

Codecov Report

Merging #1607 (9f1c597) into master (198beae) will decrease coverage by 2.79%.
The diff coverage is 20.28%.

❗ Current head 9f1c597 differs from pull request most recent head d537e9e. Consider uploading reports for the commit d537e9e to get more accurate results

@@            Coverage Diff             @@
##           master    #1607      +/-   ##
==========================================
- Coverage   35.92%   33.13%   -2.80%     
==========================================
  Files         305      329      +24     
  Lines       18464    22468    +4004     
==========================================
+ Hits         6633     7444     +811     
- Misses      11273    14421    +3148     
- Partials      558      603      +45

Impacted Files	Coverage Δ
pkg/local_object_storage/engine/tree.go	`0.00% <0.00%> (ø)`
pkg/local_object_storage/pilorama/info.go	`0.00% <0.00%> (ø)`
pkg/local_object_storage/shard/info.go	`0.00% <ø> (ø)`
pkg/local_object_storage/shard/tree.go	`0.00% <0.00%> (ø)`
pkg/services/control/convert.go	`0.00% <0.00%> (ø)`
pkg/services/control/rpc.go	`0.00% <0.00%> (ø)`
pkg/services/control/service.go	`21.97% <0.00%> (-1.56%)`	⬇️
pkg/services/control/service_grpc.pb.go	`0.00% <0.00%> (ø)`
pkg/services/tree/cache.go	`0.00% <0.00%> (ø)`
pkg/services/tree/container.go	`0.00% <0.00%> (ø)`
... and 26 more

carpawell reviewed

View reviewed changes

cmd/neofs-node/tree.go Outdated Show resolved Hide resolved

carpawell previously approved these changes

View reviewed changes

Contributor Author

fyrchik commented Jul 18, 2022

Updated only 2 fixes for the linter.

carpawell previously approved these changes

View reviewed changes

Contributor Author

fyrchik commented Jul 19, 2022

Fixed conflicts with master.

carpawell previously approved these changes

View reviewed changes

fyrchik and others added 23 commits

July 21, 2022 13:43


          [#1324] local_object_storage: Implement tree service backend

0174d24

In this commit we implement algorithm for CRDT trees from
https://martin.klepmann.com/papers/move-op.pdf

Each tree is identified by the ID of a container it belongs to
and the tree name itself. Essentially, it is a sequence of operations
which should be applied in chronological order to get a usual tree
representation.

There are 2 backends for now: bbolt database and in-memory.
In-memory backend is here for debugging and will eventually act
as a memory-cache for the on-disk database.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1324] engine: Implement Forest interface for storage engine

41037dd

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1324] services/tree: Implement Object Tree Service

a4d933b

Object Tree Service allows changing trees assotiated with
the container in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1326] services/tree: Implement GetSubTree RPC

08a258f

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1344] pilorama: Use require.ErrorIs in tests

058586a

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1344] engine: Log errors in Tree* operations

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1344] pilorama: Document errors for Get* methods

2626bb8

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1342] neofs-node: Use the default endpoint for tree service

3e3dfc8

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1328] services/tree: Fix proto field numbers

f7e1435

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1328] services/tree: Implement access control

1dd7f49

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1328] services/tree: Fix grpc import path

036318d

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1404] services/tree: Fix nodeId in GetSubTree

7870adc

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1406] pilorama: Return parent from TreeGetMeta

4dae177

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1419] pilorama: Create new nodes in path if needed

6a3e0d1

Consider a node `{FileName: "dir", Attribute: "xxx"}`. In case we add
a new node by path `["dir", "file.txt"]`, create a new intermediate node
with a single attribute.

`GetByPath` now also considers only nodes with a single attribute while building a path.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1426] services/tree: Do not replicate to a local node

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1329] services/tree: Implement GetOpLog RPC

e043358

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1329] pilorama: Allow to benchmark all tree backends

f4011f8

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1329] services/tree: Synchronize from the last stored height

73f6c4b

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1431] engine: Add benchmark for Select vs TreeGetByPath

33bfcc8

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1431] pilorama: Cache attributes in the index

f65a15d

Currently to find a node by path we iterate over all the children on
each level. This is far from optimal and scales badly with the number of
nodes on a single level. Thus we introduce "indexed attributes" for
which an additional information is stored and which can be use in
`*ByPath` operations. Currently this set only includes `FileName`
attribute but this may change in future.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1431] pilorama: Use Batch for write transactions

3a79e5c

Helps a lot in case of concurrent request flow.

```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     78.0µs ± 9%    59.8µs ± 4%  -23.39%  (p=0.000 n=10+9)
ApplyReorderLast/bbolt-8     143µs ± 5%     113µs ±15%  -21.06%  (p=0.000 n=10+10)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     56.9kB ± 8%    28.9kB ± 3%  -49.22%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8    87.3kB ± 3%    40.9kB ±10%  -53.16%  (p=0.000 n=10+10)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        224 ±11%       262 ± 5%  +16.93%  (p=0.000 n=9+10)
ApplyReorderLast/bbolt-8       518 ± 4%       674 ±11%  +30.09%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1401] services/tree: Retransmit queries to container nodes

d602f21

Also fix a bug with replicator using the multiaddress instead of
<host>:<port> format expected by gRPC library.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1401] services/tree: Marshal public key once

b435502

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

fyrchik added 23 commits

July 21, 2022 13:43


          [#1442] services/tree: Fix log message for failed Apply

4fcabe4

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1442] pilorama: Generate timestamp based on node position in the co…

85a7aff

…ntainer

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1446] services/tree: Cache connections to the container nodes

f6b3704

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1444] pilorama: Optimize internal encoding/decoding

670d7ad

```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     55.5µs ± 4%    55.5µs ± 3%     ~     (p=1.000 n=10+7)
ApplyReorderLast/bbolt-8     108µs ± 6%     112µs ± 8%     ~     (p=0.077 n=9+9)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     28.8kB ± 3%    27.7kB ± 6%   -3.79%  (p=0.005 n=10+10)
ApplyReorderLast/bbolt-8    41.4kB ± 5%    38.9kB ± 5%   -6.19%  (p=0.001 n=10+9)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        262 ± 2%       235 ±10%  -10.41%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8       684 ± 6%       616 ± 7%  -10.04%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1427] services/tree: Parallelize replicator

ae49bbd

Before this commit the replication channel was quickly filled under
heavy load. This lead to the continuously increasing latency for all
write operations. Now it looks better.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1444] pilorama: Fix TreeMove in bbolt backend

ed82486

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1446] services/tree: Use grpc.WithInsecure only for nodes without…

e6708ce

… TLS

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1445] services/tree: Cache the list of container nodes

8a7f55b

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1456] services/tree: Wait some time before reconnecting after failure

cdbd8dd

In case node is down or failing for some reason, we can expect `Dial` to
fail. In case we actively try to replicate and `Dial` always takes 2
seconds, replication-related channels quickly become full. That affects
latency of all other write operations.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1481] pilorama: Fix TreeApply

53b9ac1

Current implementation prevents invalid operations to become valid at
some later point (consider adding a child to the non-existent parent and
then adding the parent). This seems to diverge from the paper algorithm
and complicates implementation. Make it simpler.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1333] local_object_storage: Support ReadOnly mode in pilorama

15ae261

The tricky part here is the engine itself: we stop iteration on
`ErrReadOnly` because it is better to synchronize the shard later than
to have partial trees stored in 2 shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1333] services/control: Return pilorama info in ListShards RPC

534b0dd

Do not return backend type from the service for now, because memory
backend is expected to vanish.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1333] engine: Increase error counter for pilorama errors

a661a5a

1. Modifying operations are not expected to fail, unless the shard is
   read-only.
2. `Get*` operations should increase error counter too, unless the
   error is `ErrTreeNotFound`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1333] neofs-node: Initialize tree service before the control one

cef2255

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1333] services/control: Allow to synchronize local trees

8c4cfe8

Do not check that a node indeed belongs to the container, because the
synchronization will fail in this case anyway.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1333] neofs-cli: Add control synchronize-tree command

64108aa

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1334] services/tree: Document *.proto files

c9ba81e

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1505] pilorama: Allow to customize database parameters

4b67164

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1505] pilorama: Provide timeout to bbolt.Open

ece78aa

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1607] pilorama: Enable tree service explicitly

1e72280

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1607] neofs-node: Initialize storage before other services

0a990a6

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1607] services/tree: allow to customize some parameters

688b188

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>


          [#1607] CHANGELOG.md: add tree service info

d537e9e

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

carpawell approved these changes

View reviewed changes

fyrchik merged commit 2455b72 into master

aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request


          [nspcc-dev#1607] pilorama: Enable tree service explicitly

949f7c3

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request


          [nspcc-dev#1607] neofs-node: Initialize storage before other services

e4e8eb8

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request


          [nspcc-dev#1607] services/tree: allow to customize some parameters

e1213c9

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request


          [nspcc-dev#1607] CHANGELOG.md: add tree service info

3836c41

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet