Updating progress pane using timer causes race condition that would randomly crash powershell

[daxian-dbw]

With PR #2640, a timer is used to trigger updating progress pane every 100 ms to improve progress record rendering performance (commit: 760a9c2). Before the change, _progPane.Show and _pendingProgress.Update used to be called on the same thread, and now they will be called on different threads. This causes race conditions -- for all type members that are used by both methods, they might be changed at the same time and the states may be corrupted.

The screenshot below is from a recent CI build on master branch. The exception was raised from RenderHelper, and that was because it was actually enumerating over PendingProgress._topLevelNodes and at the same time new node was being added to/removed from _topLevelNodes by Update.

/cc @iSazonov @lzybkr

[iSazonov]

@daxian-dbw Is the #2771 addressed this?

[iSazonov]

@daxian-dbw I specially made restart the timer after completing _progPane.Show to exclude collisions but missed the need to control in HandleIncomingProgressRecord.
Perhaps it would be easier to add a lock control in the HandleIncomingProgressRecord and ProgressPaneUpdateTimerElapsed.
/cc @SteveL-MSFT

[lzybkr]

Does #2771 fix this?

[daxian-dbw]

Ah, I didn't know we already had an issue for this: #2756.
However, _topLevelNodes is not the only type member that is shared between the main pipeline thread and _progPane.Show running on the timer thread, _rawui and _progressRegion are another 2 examples. I think we need to analyze all those members in case any of them have similar race conditions.