Clean up and streamline snapshot data structures #55906

nnethercote · 2018-11-12T22:55:42Z

These commits clean up the snapshot structures a bit, so they are more consistent with each other and with the ena crate.

They also remove the OpenSnapshot and CommittedSnapshot entries in the undo log, just like I did for the ena crate in rust-lang/ena#14. This PR in combination with that ena PR reduces instruction counts by up to 6% on benchmarks.

r? @nikomatsakis. Note that this isn't quite ready for landing, because the ena dependency in the first commit needs to be updated once rust-lang/ena#14 lands. But otherwise it should be good.

rust-highfive · 2018-11-12T23:01:34Z

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:0095aeb8:start=1542063484397050670,finish=1542063545869501563,duration=61472450893
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
[00:02:04] 
################################                                          44.9%
######################################################################## 100.0%
[00:02:04] extracting /checkout/obj/build/cache/2018-10-30/cargo-beta-x86_64-unknown-linux-gnu.tar.gz
[00:02:04] error: failed to read `/ena2/Cargo.toml`
[00:02:04] Caused by:
[00:02:04]   No such file or directory (os error 2)
[00:02:04] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:04] Build completed unsuccessfully in 0:00:14
[00:02:04] Build completed unsuccessfully in 0:00:14
[00:02:04] Makefile:81: recipe for target 'prepare' failed
[00:02:04] make: *** [prepare] Error 1
[00:02:05] Command failed. Attempt 2/5:
[00:02:05] error: failed to read `/ena2/Cargo.toml`
[00:02:05] Caused by:
[00:02:05]   No such file or directory (os error 2)
[00:02:05] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:05] Build completed unsuccessfully in 0:00:00
[00:02:05] Build completed unsuccessfully in 0:00:00
[00:02:05] make: *** [prepare] Error 1
[00:02:05] Makefile:81: recipe for target 'prepare' failed
[00:02:07] Command failed. Attempt 3/5:
[00:02:07] error: failed to read `/ena2/Cargo.toml`
[00:02:07] Caused by:
[00:02:07]   No such file or directory (os error 2)
[00:02:07] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:07] Build completed unsuccessfully in 0:00:00
[00:02:07] Build completed unsuccessfully in 0:00:00
[00:02:07] Makefile:81: recipe for target 'prepare' failed
[00:02:07] make: *** [prepare] Error 1
[00:02:10] Command failed. Attempt 4/5:
[00:02:10] error: failed to read `/ena2/Cargo.toml`
[00:02:10] Caused by:
[00:02:10]   No such file or directory (os error 2)
[00:02:10] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:10] Build completed unsuccessfully in 0:00:00

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

nikomatsakis · 2018-11-14T20:29:52Z

See also rust-lang/ena#15

nnethercote · 2018-11-15T01:34:15Z

I've updated the first commit so that it uses ena 0.11.0. I confirmed that I'm still getting the speed-ups of up to 6%.

nikomatsakis · 2018-11-16T20:02:58Z

@bors r+

bors · 2018-11-16T20:02:59Z

📌 Commit 350a15ab4ea8ee78336c2289f56a3f67c787aa69 has been approved by nikomatsakis

nnethercote · 2018-11-20T04:39:14Z

@bors try

bors · 2018-11-20T04:39:14Z

🙅 Please do not try after a pull request has been r+ed. If you need to try, unapprove (r-) it first.

bors · 2018-11-21T11:01:00Z

☔ The latest upstream changes (presumably #52591) made this pull request unmergeable. Please resolve the merge conflicts.

nnethercote · 2018-11-21T22:55:35Z

@bors try

bors · 2018-11-21T22:55:46Z

⌛ Trying commit bf614a8bb532c52c5a721f82b8e65dc8adb1e280 with merge 04eac86e3858e974b9e3bf99acf31e76ad4ac8af...

rust-highfive · 2018-11-21T22:59:08Z

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:01af613e:start=1542840983454530064,finish=1542840986009267361,duration=2554737297
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
######################################################################    97.8%
######################################################################## 100.0%
[00:02:11] extracting /checkout/obj/build/cache/2018-10-30/cargo-beta-x86_64-unknown-linux-gnu.tar.gz
[00:02:11]     Updating crates.io index
[00:02:18] error: the lock file /checkout/src/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:02:18] Build completed unsuccessfully in 0:00:23
[00:02:18] Makefile:81: recipe for target 'prepare' failed
[00:02:18] make: *** [prepare] Error 1
[00:02:19] Command failed. Attempt 2/5:
[00:02:19] Command failed. Attempt 2/5:
[00:02:19]     Updating crates.io index
[00:02:19] error: the lock file /checkout/src/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:02:19] Build completed unsuccessfully in 0:00:00
[00:02:19] make: *** [prepare] Error 1
[00:02:19] Makefile:81: recipe for target 'prepare' failed
[00:02:21] Command failed. Attempt 3/5:
[00:02:21] Command failed. Attempt 3/5:
[00:02:21]     Updating crates.io index
[00:02:22] error: the lock file /checkout/src/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:02:22] Build completed unsuccessfully in 0:00:00
[00:02:22] make: *** [prepare] Error 1
[00:02:22] Makefile:81: recipe for target 'prepare' failed
[00:02:25] Command failed. Attempt 4/5:
[00:02:25] Command failed. Attempt 4/5:
[00:02:25]     Updating crates.io index
[00:02:25] error: the lock file /checkout/src/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:02:25] Build completed unsuccessfully in 0:00:00
[00:02:25] make: *** [prepare] Error 1
[00:02:25] Makefile:81: recipe for target 'prepare' failed
[00:02:25] Makefile:81: recipe for target 'prepare' failed
[00:02:30] error: the lock file /checkout/src/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:02:30] Build completed unsuccessfully in 0:00:00
[00:02:30] Makefile:81: recipe for target 'prepare' failed
[00:02:30] make: *** [prepare] Error 1
[00:02:30] The command has failed after 5 attempts.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors · 2018-11-21T23:01:56Z

💔 Test failed - status-travis

rust-highfive · 2018-11-21T23:01:57Z

The job dist-x86_64-linux of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:2c771ba4
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
###                                                                        5.3%
######################################################################## 100.0%
[00:04:16] extracting /checkout/obj/build/cache/2018-10-30/cargo-beta-x86_64-unknown-linux-gnu.tar.gz
[00:04:16]     Updating crates.io index
[00:04:22] error: the lock file /checkout/src/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:04:22] Build completed unsuccessfully in 0:00:23
[00:04:22] make: *** [prepare] Error 1
[00:04:23] Command failed. Attempt 2/5:
[00:04:23]     Updating crates.io index
[00:04:23]     Updating crates.io index
[00:04:24] error: the lock file /checkout/src/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:04:24] Build completed unsuccessfully in 0:00:00
[00:04:24] make: *** [prepare] Error 1
[00:04:26] Command failed. Attempt 3/5:
[00:04:26]     Updating crates.io index
[00:04:26]     Updating crates.io index
[00:04:26] error: the lock file /checkout/src/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:04:26] Build completed unsuccessfully in 0:00:00
[00:04:26] make: *** [prepare] Error 1
[00:04:29] Command failed. Attempt 4/5:
[00:04:30]     Updating crates.io index
[00:04:30]     Updating crates.io index
[00:04:30] error: the lock file /checkout/src/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:04:30] Build completed unsuccessfully in 0:00:00
[00:04:30] make: *** [prepare] Error 1
          768M     0  768M   0% /var/ramfs
2616972 .

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

nnethercote · 2018-11-21T23:45:50Z

@bors try

bors · 2018-11-21T23:46:01Z

⌛ Trying commit f61f9ed138a073229e19fbbff1a38d2ddbe53a05 with merge 2f28ba5f132d52fb788e30c86134b325180d16c5...

bors · 2018-11-22T02:45:35Z

☀️ Test successful - status-travis
State: approved= try=True

nnethercote · 2018-11-22T02:59:00Z

@rust-timer build 2f28ba5f132d52fb788e30c86134b325180d16c5

bors · 2018-11-25T06:04:18Z

💡 This pull request was already approved, no need to approve it again.

This pull request previously failed. You should add more commits to fix the bug, or use retry to trigger a build again.
There's another pull request that is currently being tested, blocking this pull request: Decouple proc_macro from the rest of the compiler. #49219

bors · 2018-11-25T06:04:18Z

📌 Commit bfc1902 has been approved by nikomatsakis

bors · 2018-11-25T06:48:16Z

🔒 Merge conflict

This pull request and the master branch diverged in a way that cannot be automatically merged. Please rebase on top of the latest master branch, and let the reviewer approve again.

How do I rebase?

Assuming self is your fork and upstream is this repository, you can resolve the conflict following these steps:

git checkout rm-OpenSnapshot-CommittedSnapshot (switch to your branch)
git fetch upstream master (retrieve the latest master)
git rebase upstream/master -p (rebase on top of it)
Follow the on-screen instruction to resolve conflicts (check git status if you got lost).
git push self rm-OpenSnapshot-CommittedSnapshot --force-with-lease (update this PR)

You may also read Git Rebasing to Resolve Conflicts by Drew Blessing for a short tutorial.

Please avoid the "Resolve conflicts" button on GitHub. It uses git merge instead of git rebase which makes the PR commit history more difficult to read.

Sometimes step 4 will complete without asking for resolution. This is usually due to difference between how Cargo.lock conflict is handled during merge and rebase. This is normal, and you should still perform step 5 to update this PR.

Error message

warning: Cannot merge binary files: Cargo.lock (HEAD vs. heads/homu-tmp)
Auto-merging Cargo.lock
CONFLICT (content): Merge conflict in Cargo.lock
Automatic merge failed; fix conflicts and then commit the result.

This version has some significant speed-ups relating to snapshotting.

Because it's as useless as its name suggests. This commit also renames `UndoLog::Noop` as `UndoLog::Purged`, because (a) that's a more descriptive name and (b) it matches the name used in similar code in `librustc/infer/region_constraints/mod.rs`.

So that it matches `librustc_data_structures/snapshot_map/mod.rs` and the `ena` crate.

Because they shouldn't be reused. This provides consistency with the `ena` crate.

This makes the two snapshot implementations more consistent with each other and with crate `ena`.

…ap`. They're not strictly necessary, and they result in the `Vec` being allocated even for the trivial (and common) case where a `start_snapshot` is immediately followed by a `commit` or `rollback_to`.

…straintCollector`. They're not strictly necessary, and they result in the `Vec` being allocated even for the trivial (and common) case where a `start_snapshot` is immediately followed by a `commit` or `rollback_to`. The commit also removes a now-unnecessary argument of `pop_placeholders()`.

nnethercote · 2018-11-25T07:01:35Z

@bors r=nikomatsakis

bors · 2018-11-25T07:01:36Z

📌 Commit 94967ae has been approved by nikomatsakis

bors · 2018-11-25T12:20:25Z

⌛ Testing commit 94967ae with merge e9bca7a...

@nikomatsakis

… r=nikomatsakis Clean up and streamline snapshot data structures These commits clean up the snapshot structures a bit, so they are more consistent with each other and with the `ena` crate. They also remove the `OpenSnapshot` and `CommittedSnapshot` entries in the undo log, just like I did for the `ena` crate in rust-lang/ena#14. This PR in combination with that `ena` PR reduces instruction counts by up to 6% on benchmarks. r? @nikomatsakis. Note that this isn't quite ready for landing, because the `ena` dependency in the first commit needs to be updated once rust-lang/ena#14 lands. But otherwise it should be good.

bors · 2018-11-25T15:09:55Z

☀️ Test successful - status-appveyor, status-travis
Approved by: nikomatsakis
Pushing e9bca7a to master...

nnethercote · 2018-11-25T21:28:13Z

Just for posterity: this PR finally landed 8 days and 19 hours after the initial r+. I'm sure that's the slowest landing I've ever had. It was in the queue for almost all that time, mostly hovering around positions 6, 7 and 8, continually being bumped. I had to rebase it four or five times, and for one of those rebases I wasn't quick enough and so I missed an opening for landing, which delayed things by few hours more.

Centril · 2018-11-28T21:08:15Z

cc @kennytm re. @nnethercote's previous comment... what's up with that?

petrochenkov · 2018-11-28T21:22:52Z

First, prioritized edition-critical PRs are flying around between master and beta right now.
Second, rollups of small PRs are prioritized.
This PR is neither critical nor small enough to rollup - the saddest kind of PR to land.

nnethercote · 2018-11-28T21:37:09Z

Landing cycle time now exceeds 3 hours, which means there are about 7.5 slots per 24 hours. Combine that with the fact that it feels like intermittent failures are more frequent lately, and the number of successful landings per 24 hours is probably something like 5.

nnethercote · 2018-11-29T21:28:37Z

Final perf results -- wins of up to 5%, with almost every benchmark seeing at least some improvement.

rust-highfive assigned nikomatsakis Nov 12, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 12, 2018

nnethercote mentioned this pull request Nov 12, 2018

Remove OpenSnapshot and CommittedSnapshot markers. rust-lang/ena#14

Merged

nikomatsakis approved these changes Nov 14, 2018

View reviewed changes

nnethercote force-pushed the rm-OpenSnapshot-CommittedSnapshot branch from c189efa to 350a15a Compare November 15, 2018 01:31

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 16, 2018

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 21, 2018

nnethercote force-pushed the rm-OpenSnapshot-CommittedSnapshot branch from 350a15a to bf614a8 Compare November 21, 2018 22:54

nnethercote force-pushed the rm-OpenSnapshot-CommittedSnapshot branch from bf614a8 to f61f9ed Compare November 21, 2018 23:22

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 25, 2018

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 25, 2018

nnethercote added 8 commits November 25, 2018 17:54

Update to ena 0.11.0.

1e34dfc

This version has some significant speed-ups relating to snapshotting.

Remove insert_noop.

9847b5c

Because it's as useless as its name suggests. This commit also renames `UndoLog::Noop` as `UndoLog::Purged`, because (a) that's a more descriptive name and (b) it matches the name used in similar code in `librustc/infer/region_constraints/mod.rs`.

Rename UndoLogEntry as UndoLog.

c86bbd4

So that it matches `librustc_data_structures/snapshot_map/mod.rs` and the `ena` crate.

Replace a .truncate(0) call with .clear().

7fe09a6

Make commit and rollback_to methods take ownership of the snapshots.

f5624e4

Because they shouldn't be reused. This provides consistency with the `ena` crate.

Introduce in_snapshot and assert_open_snapshot methods.

f23c969

This makes the two snapshot implementations more consistent with each other and with crate `ena`.

Remove OpenSnapshot and CommittedSnapshot markers from `SnapshotM…

2d68fa0

…ap`. They're not strictly necessary, and they result in the `Vec` being allocated even for the trivial (and common) case where a `start_snapshot` is immediately followed by a `commit` or `rollback_to`.

nnethercote force-pushed the rm-OpenSnapshot-CommittedSnapshot branch from bfc1902 to 94967ae Compare November 25, 2018 07:01

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 25, 2018

bors merged commit 94967ae into rust-lang:master Nov 25, 2018

nnethercote deleted the rm-OpenSnapshot-CommittedSnapshot branch November 25, 2018 21:23

Clean up and streamline snapshot data structures #55906

Clean up and streamline snapshot data structures #55906

Uh oh!

Conversation

nnethercote commented Nov 12, 2018

Uh oh!

rust-highfive commented Nov 12, 2018

Uh oh!

nikomatsakis commented Nov 14, 2018

Uh oh!

nnethercote commented Nov 15, 2018

Uh oh!

nikomatsakis commented Nov 16, 2018

Uh oh!

bors commented Nov 16, 2018

Uh oh!

nnethercote commented Nov 20, 2018

Uh oh!

bors commented Nov 20, 2018

Uh oh!

bors commented Nov 21, 2018

Uh oh!

nnethercote commented Nov 21, 2018

Uh oh!

bors commented Nov 21, 2018

Uh oh!

rust-highfive commented Nov 21, 2018

Uh oh!

bors commented Nov 21, 2018

Uh oh!

rust-highfive commented Nov 21, 2018

Uh oh!

nnethercote commented Nov 21, 2018

Uh oh!

bors commented Nov 21, 2018

Uh oh!

bors commented Nov 22, 2018

Uh oh!

nnethercote commented Nov 22, 2018

Uh oh!

bors commented Nov 25, 2018

Uh oh!

bors commented Nov 25, 2018

Uh oh!

bors commented Nov 25, 2018

Uh oh!

nnethercote commented Nov 25, 2018

Uh oh!

bors commented Nov 25, 2018

Uh oh!

bors commented Nov 25, 2018

Uh oh!

bors commented Nov 25, 2018

Uh oh!

nnethercote commented Nov 25, 2018

Uh oh!

Centril commented Nov 28, 2018

Uh oh!

petrochenkov commented Nov 28, 2018

Uh oh!

nnethercote commented Nov 28, 2018

Uh oh!

nnethercote commented Nov 29, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants