High number of revisions causes system overload.

[bldcm]

I'm migrating from TFS to ADO using a VM with 64GB of memory and 6 processors. For most of our projects this has been plenty, until this one. Migration tool 16.0.4

Ran a migration which broke down to 503 work items and it ran for 22+ hours. One thing that appears to be different about this project is that the number of revisions per work item is in the 100s and not the 10s. I think this is because the do a lot of reordering on the backlog and every shuffle generates a revision.

I am looking for ways to optimize. I have read about a MaximumNumberOfRevisions (sp?) parameter, that takes the first revision and then the 'n-1' latest revisions. The problem with that is high probability that some user comments will be lost. I am ok with losing all state change type history, but we need the user comments.

I have seen reference to something called a "batch size", but even if I could find it, I am not sure if I could do batches smaller than 503 though when the projects contain 10s of thousands and sometimes 100s of thousands of work item.

Is there a way to keep the tool from loading all the work items into memory before starting the migration? At least this is what it looks like it is doing.

I have also heard of a "Rate Limiter" for ADO, but I am not sure where that can be found.

Your help is appreciated, and I have included my configuration and the logs for the slow migration.

configuration-TFS2ADOPreProd-eFlow-NOTO-upgraded-diag.json
20241009045127.zip

[MrHinsh]

Is there a way to keep the tool from loading all the work items into memory before starting the migration? At least this is what it looks like it is doing.

No. And Btahc Size is for the Rest API and not the Tfs Object Model.

I have never seen this type of issue before, and I have migrated 16k work items at once with 200-300 revisions per item. It sounds like something else is going on.

---

Funding for us doing work on this tool is generally through 1) customer funding, or 2) donated free time.

Sponcer a Feature | Hire us directly

[bldcm]

Any suggestions on how to debug it? This has currently only happen in one of are projects and like you, this is not even the biggest project we have migrated.

We have tried multiple VMs and PCs in the same network segment as the TFS and ADO server. All these server are used in all our other migrations. We have increased the memory to 128 GB with no improvements. We have migrated everything from 2000, to 75,000 work items previously with no issues. This occurs in both version 14 and 16.0.4.

[MrHinsh]

Id run it in Visual Studio and see where it's glitching...

Where does it get to in the log?

[connor-dawson]

@MrHinsh I did some debugging with Visual Studio. The tool slows down when loading the workitems as workitem data. This process begins in the ToWorkItemDataList method in TfsExtensions.cs, which calls the WorkItem.AsWorkItemData() method. This is where the work item fields, links, and revisions are read into memory.

During this process my computer became unresponsive. My system has 32GB of memory and the migration process was using about 22GB of it. As you can see from the log only ~2600 work items had been loaded so far, where with other migrations I've migrated 60K work items. Also the rate at which work items are loaded is significantly slower than other migrations I've ran, where we typically see about 500 work items loaded every 30 seconds.

I think this is due to the high number of revisions the work items have, unless there is another part of this process that is using memory but I didn't see any. It looks like the program loads all work item fields data for each revision into memory. Are there any improvements in this area that could be made to avoid loading so much data into memory? Perhaps comparing the current revision to the previous revision and only loading fields that have changed into memory?

Thanks
migration2024102311.log

[MrHinsh]

What's the revisions per work item? It would be good to know where the tipping point is.

This would be a lot of work to change. I can't think of an easy way to do this other than turning off revisions.

You could limit the revisions that are loaded, but I don't remember where in the cycle we dump the ones we don't want, I think it's after we load each item...

That might be somewhere we can optimise.