Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add snapshot processing for all astrid modes #12339

Merged
merged 124 commits into from
Oct 28, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
124 commits
Select commit Hold shift + click to select a range
b8812c9
initial refactor for spans and events
mh0lt Aug 30, 2024
b0da397
fixed compilation errors
mh0lt Sep 1, 2024
56a7436
removed block reader bor snapshot processing
mh0lt Sep 1, 2024
605a015
removed stage binary formating dependencies
mh0lt Sep 5, 2024
628dac0
removed stage binary formating dependencies
mh0lt Sep 5, 2024
f90382a
typo
mh0lt Sep 5, 2024
fddae8c
remove temp event code
mh0lt Sep 5, 2024
d424f28
Merge branch 'main' into astrid_snapshots
mh0lt Sep 5, 2024
468d26e
fixed compile errors
mh0lt Sep 5, 2024
5f11254
regen mock
mh0lt Sep 5, 2024
07e91bb
add eventfetcher to bridge
mh0lt Sep 6, 2024
0bfd9e6
Merge branch 'main' into astrid_snapshots
mh0lt Sep 6, 2024
77be1f8
Add tx adapter
mh0lt Sep 6, 2024
bfaf380
standalone running with snapshots
mh0lt Sep 6, 2024
24f2e6f
add age to head update log
mh0lt Sep 6, 2024
39a1be5
added adaptors for index building and range adaption
mh0lt Sep 7, 2024
5ff9fba
get checkpoint id from range index
mh0lt Sep 7, 2024
c381bfa
use bridge/heimdall for stages
mh0lt Sep 9, 2024
b198ccb
add tx to range queries
mh0lt Sep 9, 2024
648b154
add printer
mh0lt Sep 9, 2024
78bc39d
add error soure detail
mh0lt Sep 9, 2024
7231688
add error soure detail
mh0lt Sep 9, 2024
1b3a619
add error soure detail
mh0lt Sep 9, 2024
17daeb3
add error soure detail
mh0lt Sep 9, 2024
3f270bd
add printer
mh0lt Sep 9, 2024
96b28f6
add printer
mh0lt Sep 9, 2024
0e90d22
add printer
mh0lt Sep 9, 2024
c76d130
pass in range table name
mh0lt Sep 9, 2024
bb2a42e
remove printer
mh0lt Sep 9, 2024
0d54442
error formatting
mh0lt Sep 9, 2024
dcb6785
add printer
mh0lt Sep 9, 2024
e9c2ce9
add printer
mh0lt Sep 10, 2024
c572038
fix entity id from store
mh0lt Sep 10, 2024
534a62d
fix withtx return
mh0lt Sep 10, 2024
e5bb649
fix store type
mh0lt Sep 10, 2024
35c1e8f
add printer
mh0lt Sep 10, 2024
78b0fe9
Merge branch 'main' into astrid_snapshots
mh0lt Sep 10, 2024
f262c9d
post merge fixes
mh0lt Sep 10, 2024
929b58e
promote fetch log
mh0lt Sep 12, 2024
97f3d7e
promote put log
mh0lt Sep 12, 2024
2672d05
add printer
mh0lt Sep 12, 2024
a150c6b
add printer
mh0lt Sep 12, 2024
161d768
add printer
mh0lt Sep 12, 2024
74171e6
add printer
mh0lt Sep 12, 2024
a296a85
add printer
mh0lt Sep 12, 2024
5d96a38
add printer
mh0lt Sep 13, 2024
70e29ea
add printer
mh0lt Sep 13, 2024
fd475f6
add printer
mh0lt Sep 13, 2024
d8a2b28
split prune and unwid functionality
mh0lt Sep 13, 2024
08a5c9b
remove printers
mh0lt Sep 13, 2024
759d391
move log
mh0lt Sep 13, 2024
e1c03d1
move log
mh0lt Sep 13, 2024
31ccf9f
Fix blocks availibility for partial blocks
mh0lt Sep 16, 2024
9b438b8
remove printers
mh0lt Sep 16, 2024
f2e6a36
check file blocks available by type
mh0lt Sep 16, 2024
484ebfa
download not complete until db processing finished
mh0lt Sep 16, 2024
8885b95
get events from snapshots
mh0lt Sep 16, 2024
4748dd8
add lastprocessed event with frozen
mh0lt Sep 17, 2024
d782683
sync start up & event snap end fixes
mh0lt Sep 18, 2024
1e8b4af
adjust logging
mh0lt Sep 18, 2024
40a83c8
rationalize stage & astrid event access code
mh0lt Sep 18, 2024
8ba307c
Merge branch 'main' into astrid_snapshots
mh0lt Sep 19, 2024
d2c6882
post merge updates
mh0lt Sep 19, 2024
587b20c
base snapshots compiling
mh0lt Sep 19, 2024
21ed721
base merge compiling
mh0lt Sep 19, 2024
70346ee
sysc.stage at tip
mh0lt Sep 20, 2024
4b9611a
fix failt on reopen of active files
mh0lt Oct 4, 2024
579740b
Merge branch 'main' into astrid_snapshots
mh0lt Oct 4, 2024
a42371c
post merge updates
mh0lt Oct 5, 2024
d1d59f5
Merge branch 'main' into astrid_snapshots
mh0lt Oct 8, 2024
cba2228
remove printer
mh0lt Oct 8, 2024
4a4c109
fix panic due to nill receipt
mh0lt Oct 8, 2024
2cd71b7
fix close nil channel panic
mh0lt Oct 8, 2024
9bb1fcf
don't wait on zero timeout
mh0lt Oct 8, 2024
c15731a
remove races
mh0lt Oct 8, 2024
59229cb
remove races
mh0lt Oct 9, 2024
4faa819
ref count frozen
mh0lt Oct 9, 2024
3068555
fix race
mh0lt Oct 11, 2024
4c58e16
tidy lock locations
mh0lt Oct 11, 2024
325e08f
Merge branch 'main' into astrid_snapshots
mh0lt Oct 11, 2024
7ebfb56
post merge changes
mh0lt Oct 11, 2024
089f605
Merge branch 'main' into astrid_snapshots
mh0lt Oct 11, 2024
2856dbc
internalize option mechanics
mh0lt Oct 12, 2024
6b1d5c1
start from snapshots (db's removed)
mh0lt Oct 12, 2024
5d97017
make file recalc atomic
mh0lt Oct 16, 2024
a247986
Merge branch 'main' into astrid_snapshots
mh0lt Oct 16, 2024
7241e80
don't reopen index if refcount>0
mh0lt Oct 16, 2024
f188e53
add bridge store to sim test
mh0lt Oct 16, 2024
7db113a
Merge branch 'main' into astrid_snapshots
mh0lt Oct 17, 2024
9410aa7
post merge fixes
mh0lt Oct 17, 2024
f712241
add stores to tests
mh0lt Oct 17, 2024
3da6cc4
Merge branch 'main' into astrid_snapshots
mh0lt Oct 17, 2024
7ba2f3d
fix test validator signature
mh0lt Oct 17, 2024
6735807
fix prune db passing
mh0lt Oct 19, 2024
dc3cca1
move snapshot ready to engine ready
mh0lt Oct 19, 2024
e2b1497
Merge branch 'main' into astrid_snapshots
mh0lt Oct 19, 2024
4767200
updated including #12362
mh0lt Oct 19, 2024
7b2d520
fix startup from 0 with no-downloader
mh0lt Oct 19, 2024
faa6429
fix heimdall service_test regression
mh0lt Oct 21, 2024
5ee2684
review fixes
mh0lt Oct 21, 2024
c2f4ea2
review fixes
mh0lt Oct 21, 2024
41c43e1
review fixes
mh0lt Oct 21, 2024
d526ff2
review fixes
mh0lt Oct 21, 2024
1a7938c
Merge branch 'main' into astrid_snapshots
mh0lt Oct 21, 2024
86f22c9
added block changes
mh0lt Oct 21, 2024
870e988
adding bissing dbs
mh0lt Oct 21, 2024
d4083dc
consolidate bridge db code
mh0lt Oct 22, 2024
d20b40e
fexed start timing and remove block reader dependency
mh0lt Oct 22, 2024
ff2b0b7
remove unecessary polygon specific code
mh0lt Oct 22, 2024
c1575cf
Merge branch 'main' into astrid_snapshots
mh0lt Oct 22, 2024
a20e6d7
post merge fixes
mh0lt Oct 22, 2024
b5f6909
fix snapshot production after db changes
mh0lt Oct 23, 2024
3ceeccd
add missing channels
mh0lt Oct 23, 2024
80abb44
Merge branch 'main' into astrid_snapshots
mh0lt Oct 23, 2024
c4ee8ba
added syncronous error check and block insert logging
mh0lt Oct 24, 2024
8bb6c19
Merge branch 'main' into astrid_snapshots
mh0lt Oct 24, 2024
dc042f3
port merge changes
mh0lt Oct 24, 2024
17228c9
review fixes
mh0lt Oct 25, 2024
e9fc731
Merge branch 'main' into astrid_snapshots
mh0lt Oct 25, 2024
9d6a6de
post merge updates
mh0lt Oct 25, 2024
dd95339
remove redundant method
mh0lt Oct 25, 2024
893d988
set correct base seg
mh0lt Oct 25, 2024
6f9e254
lint fixes
mh0lt Oct 26, 2024
e6bf473
Merge branch 'main' into astrid_snapshots
mh0lt Oct 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added block changes
  • Loading branch information
mh0lt committed Oct 21, 2024
commit 86f22c93cb44806a7e58e6355cdb32d38a299c6b
2 changes: 1 addition & 1 deletion erigon-lib/downloader/downloadercfg/downloadercfg.go
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@ func loadSnapshotsEitherFromDiskIfNeeded(dirs datadir.Dirs, chainName string) (*
if err := dir.WriteFileWithFsync(preverifiedToml, snapcfg.GetToml(chainName), 0644); err != nil {
return nil, err
}
return snapcfg.KnownCfg(preverifiedToml), nil
return snapcfg.KnownCfg(chainName), nil
}

func getIpv6Enabled() bool {
Expand Down
10 changes: 4 additions & 6 deletions turbo/snapshotsync/snapshots.go
Original file line number Diff line number Diff line change
Expand Up @@ -453,7 +453,9 @@ func (s *Segments) SetMaxVisibleBlock(max uint64) {

func (s *Segments) BeginRo() *RoTx {
for _, seg := range s.VisibleSegments {
seg.src.refcount.Add(1)
if !seg.src.frozen {
seg.src.refcount.Add(1)
}
}
return &RoTx{segments: s, VisibleSegments: s.VisibleSegments}
}
Expand Down Expand Up @@ -482,10 +484,6 @@ func (s *RoTx) Close() {

refCnt := src.refcount.Add(-1)

if src.frozen {
continue
}

if refCnt == 0 && src.canDelete.Load() {
src.closeAndRemoveFiles()
}
Expand Down Expand Up @@ -1072,7 +1070,7 @@ func (s *RoSnapshots) rebuildSegments(fileNames []string, open bool, optimistic
})

if !exists {
sn = &DirtySegment{segType: f.Type, version: f.Version, Range: Range{f.From, f.To}, frozen: snapcfg.Seedable(s.cfg.ChainName, f)}
sn = &DirtySegment{segType: f.Type, version: f.Version, Range: Range{f.From, f.To}, frozen: snapcfg.IsFrozen(s.cfg.ChainName, f)}
}

if open {
Expand Down
Loading