Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Turbopack] improve aggregation implementation #69651

Closed
Closed
Changes from 1 commit
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
a47eabe
add lmdb persisting and restoring
sokra Aug 9, 2024
10b2688
pass test name to test_config to construct db name
sokra Aug 9, 2024
3320b98
continue uncompleted operations
sokra Aug 9, 2024
e389ce3
create dir and logging
sokra Aug 12, 2024
fba0487
improve error messages
sokra Aug 13, 2024
b6f946d
handle keys larger than 511 bytes
sokra Aug 13, 2024
d952e25
avoid storing transient tasks
sokra Aug 14, 2024
fdfe7e9
show lookup error
sokra Aug 14, 2024
3c3d2d2
handle state serialization
sokra Aug 14, 2024
4f6dffc
validate serialization and improve errors
sokra Aug 14, 2024
b1cff41
add turbo_tasks_backend to tracing, add tracing for restore
sokra Aug 16, 2024
d6a3594
disable TLS
sokra Aug 16, 2024
a16b7cb
print lookup error
sokra Aug 16, 2024
9338f2a
verify serialization
sokra Aug 16, 2024
f4aecd9
fix lookup deserialization
sokra Aug 16, 2024
de09ab8
replace bincode with pot
sokra Aug 16, 2024
7220910
fix restore data trace
sokra Aug 17, 2024
796656f
more tracing in db
sokra Aug 17, 2024
517f568
remove verify_serialization
sokra Aug 19, 2024
716fd63
fix race condition
sokra Aug 17, 2024
898a031
do not interrupt persisting while there is data
sokra Aug 19, 2024
def2e20
add persist trace
sokra Aug 19, 2024
82abf4f
improve task aggregation
sokra Sep 3, 2024
c58050e
restore task_pair
sokra Sep 4, 2024
b1093f5
verify persistent function only calls persistent functions
sokra Aug 19, 2024
40dde1d
clippy
sokra Sep 6, 2024
e6451d0
add more details to save_snapshot tracing
sokra Sep 20, 2024
1d8d5f8
store task data and aggregation separately
sokra Sep 25, 2024
21bcf0e
compare with old value before storing to db
sokra Sep 24, 2024
7f3b34f
Early stop snapshot interval when stopping
sokra Sep 24, 2024
3440d1b
parallel save_snapshot
sokra Sep 24, 2024
7bc39db
run snapshot in spawn_blocking
sokra Sep 25, 2024
957708d
save snapshot when starting idle
sokra Sep 25, 2024
9d0a7ed
wait until idle for a second before persisting
sokra Sep 25, 2024
909a230
increase max DB size
sokra Sep 26, 2024
cf9c9b0
remove db lookup tracing
sokra Sep 26, 2024
da79e6e
remove database println
sokra Sep 26, 2024
464adcc
sharded transaction log
sokra Sep 26, 2024
38f8d57
share read transition
sokra Oct 4, 2024
cfd75ba
remove aggregation update queue tracing
sokra Sep 27, 2024
2c2938e
improve messaging
sokra Sep 27, 2024
8ba023f
handle idle end event for idle detection in persisting
sokra Sep 27, 2024
291c838
improve imports
sokra Oct 4, 2024
d4d6ee6
clippy
sokra Oct 4, 2024
ff90deb
randomize port
sokra Oct 4, 2024
5aa1f81
add or_insert to AutoMap Entry
sokra Oct 4, 2024
d863284
add support for collectibles to new backend
sokra Oct 4, 2024
551df44
flag task as dirty when they get output set
sokra Oct 4, 2024
225a352
force root task when reading collectibles
sokra Oct 4, 2024
767cbc5
add collectibles test
sokra Oct 4, 2024
f111b51
make in progress tasks as stale
sokra Oct 4, 2024
df83503
skip updating output for stale tasks
sokra Oct 4, 2024
3e8b6f2
fix bug
sokra Oct 4, 2024
9a8148b
add remaining test cases
sokra Oct 4, 2024
0c58049
enable new backend
sokra Aug 13, 2024
5eb5fea
WIP: remove todos
sokra Aug 13, 2024
e11a7b0
print turbopack time
sokra Sep 5, 2024
053c178
add restore task tracing
sokra Sep 26, 2024
6a9f5c0
bolder time reporting
sokra Sep 27, 2024
adbd31b
avoid some serde skip and untagged
sokra Aug 14, 2024
12e6f34
log restored db entries
sokra Aug 14, 2024
7f777ff
WIP: print new tasks
sokra Aug 14, 2024
f6b2761
WIP: logging
sokra Sep 6, 2024
eedbc2f
clippy
sokra Sep 6, 2024
f2adfcb
invalidation tracing
sokra Aug 16, 2024
724a4f5
WIP: recompute from reading output tracing
sokra Sep 5, 2024
07a8e55
remove verbose tracing
sokra Aug 19, 2024
d37ca8b
WIP: logging
sokra Sep 5, 2024
26a9ca8
WIP
sokra Sep 6, 2024
565c16c
Revert "WIP"
sokra Sep 20, 2024
e13d82f
Revert "WIP: logging"
sokra Sep 20, 2024
3b08bce
Revert "WIP: logging"
sokra Sep 20, 2024
f802682
trace aggregation update
sokra Sep 23, 2024
26bea01
trace suspended operations
sokra Sep 23, 2024
eb54393
WIP: logging
sokra Sep 23, 2024
6378b96
Revert "WIP: logging"
sokra Sep 23, 2024
0212002
aggregation stats
sokra Sep 23, 2024
c2a8f1b
log tasks with many items
sokra Sep 23, 2024
9be0b98
WIP: logging
sokra Sep 24, 2024
4135634
reuse outdated children
sokra Sep 24, 2024
1264a17
Revert "WIP: logging"
sokra Sep 24, 2024
02c2e7c
Revert "reuse outdated children"
sokra Sep 24, 2024
ac28765
log tasks with many items
sokra Sep 24, 2024
9a6534d
improve once task handling
sokra Aug 16, 2024
b57490c
WIP: avoid overriden package.json
sokra Aug 17, 2024
ee27bc1
workaround for missing active tracking
sokra Aug 19, 2024
5be2ef5
Revert "workaround for missing active tracking"
sokra Sep 19, 2024
463aea0
strong reads
sokra Sep 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
log tasks with many items
  • Loading branch information
sokra committed Oct 7, 2024
commit ac28765b39ad55ec1683ebe2995fdd11c4d00032
Original file line number Diff line number Diff line change
Expand Up @@ -113,9 +113,10 @@ impl<'a> ExecuteContext<'a> {
if !task.persistance_state().is_restored(category) {
if items.len() > 10000 {
println!(
"{} restored {} items",
"{} restored {} items ({:?})",
self.backend.get_task_desc_fn(task_id)(),
items.len()
items.len(),
category
);
}
for item in items {
Expand Down