VS deadlocked during up-to-date check

[davkean]

@jcouv attempted to build and VS appears to have deadlocked during up-to-date check.

Threads overview:

Dump is on [jcouv's dev box]\shared-dumps\devenv-thread-starvation.dmp

[davkean]

@srivatsn Put this in 15.3 as this a new path just introduced and probably related to the update-to-date.

[davkean]

I had a quick look at the code (not the dump - as I can't donwload it) nothing obvious - except maybe the options check, does that call back into the UI thread? Maybe that's not proceeding?

[panopticoncentral]

Wait... where is the dump file?

[panopticoncentral]

But, yes, the options check does switch to the main thread to call the settings manager. But I don't think the main thread blocks on up to date checks, so I'm not sure where the block might be.

[jcouv]

Here is the dump file (when VS was hung, I attached another VS and saved the dump): https://www.dropbox.com/s/gq42udarqzcenij/devenv-thread-starvation.dmp?dl=0
I also have it on my dev box.

[davkean]

It's wrapped in an ExecuteSynchronously, so it's definitely blocking, it's only going to let transitions to the UI thread that it thinks are part of the same chain.

[panopticoncentral]

This was on a local built project system instance, is it possible to get the symbols for it so we can see where we were in the up to date check when we hung? Or is there some trick to figure that out?

[jcouv]

I'm not sure. Assuming that project system is built with determinism, you should be able to get to the same state by building from your PR on FUTD check for ref assemblies (#2414).
Since there were multiple iterations, I'm not sure which iteration it was. Based on my recollection and the timestamps, I think it was 32b9b66 or the one before that.

[jcouv]

I have a new dump capturing the same issue (I think) and not involving any private binaries.
//jcouvdev/shared-dumps/devenv-hung2.dmp

Below is the capture of Parallel stacks for confirmation:

[panopticoncentral]

This is https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems?id=455018. The problem is that we used to have CPS wait for critical tasks to finish before calling the up to date check, and this would block the UI thread if a critical task didn't finish or took a long time. This was fixed in PR #2453.