Backoffice slows down getting content with a lot of variants

[dawoe]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

10.6.1 to 12.x

Bug summary

We have a site with 6 languages. We also have uMarketing suite installed. For some pages multiple segment variants are created with uMarketing suite.

We noticed while adding more languages, and the customer creating personalized content with uMarketingSuite this slows down loading of content in the backoffice. The frontend runs just fine.

We noticed that getting a response from this API method is slow :

Umbraco-CMS/src/Umbraco.Web.BackOffice/Controllers/ContentController.cs

Line 460 in cd28556

public ActionResult<ContentItemDisplayWithSchedule?> GetById(int id)

On our live environment this takes up between 8 to 10 seconds to return a result. With the copy of the database of the live environment on our local machine this takes between 5 and 7 seconds.

Adding more variants (culture or segment) increases the response time. Removing variants decreases the response time.

We see the same behavior when use this database on a clean Umbraco install. So we can rule out uMarketing suite.

Specifics

I did a debugging session against the Umbraco source code and found where the majority of time is spent.

It happens in this method :

Umbraco-CMS/src/Umbraco.Core/Models/Mapping/ContentPropertyDisplayMapper.cs

Line 35 in cd28556

public override void Map(IProperty originalProp, ContentPropertyDisplay dest, MapperContext context)

This mapping function is called for each property in each variant. Because this happens in a loop we are executing code that is the same for all properties in all variants.
For example getting the datatype configuration. But this is the same for all variants, so there is no need to get it over and over again.

This can be probably be improved by getting all this information earlier in the mapping process and pass it on using the MappingContext.

Steps to reproduce

Create a site with a lot of culture variants. If needed install uMarketingSuite and some personalized content as well.

See the performance of the GetById degrading with each variant added.

Expected result / actual result

Adding more variants should not slow down the retrieval of content significantly.

[github-actions]

Hi there @dawoe!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[Migaroez]

Hey @dawoe, Thx for the clear and complete bug description and investigating the possible culprit, going to have a look at this and will get back to you asap.

[ronaldbarendse]

We've implemented performance improvements in mapping code for a similar issue in PR #10521 by storing/caching data in the MapperContext, exactly like you already mentioned. Calling DataTypeService.GetDataType() for every property (even if some use the same data type), multiplied by every culture and again multiplied by every segment does indeed cause a lot of unnecessary overhead...

[Migaroez]

Intermediate findings:
Tested on v10/dev(132935c)
Used the following code https://gist.githubusercontent.com/Migaroez/162faab59d4d04515d665e27f50e2735/raw/e25436167c3882118699760c0760912bf244ff78/ContentGenerateController
And payload

{
    "amount":3,
    "doctypeAliases":["fifteenCultureDocType"],
    "containingDoctypeAlias":"container",
    "containtingDoctypeUsesListview" : "false",
    "newDocTypePropertyMinAmount":30,
    "newDocTypePropertyMaxAmount":30,
    "languageIsoCodes":["en-us","nl-nl","nl-be","fr-fr","dk-dk","fr-be","fr-en","aa","aa-DJ","aa-ER","aa-ET","af","af-NA","af-ZA","agq"],
    "segments" :["segmentOne","segmentTwo","segmentThree","segmentFour","segmentFive"]
}

Takes around 426ms to get one of these nodes trough the Backoffice on SQL Server Dev 16
and 289ms on sqllite
Without cultures, it takes around 57ms on sqllite.

So there definitely is an impact related to enabling and using segments but not anything dramatic as Dave is seeing. Granted, this is on a limited data set with only textbox properties.

Will continue the investigation

[Migaroez]

Increasing the nodes to 1000+ does not seem to mess with the median response time, but there are some random spikes, this might indicate some caching issues.

Since reproducing this on a blank installation seems to be not straight forward, I am going to request some dedicated time to investigate this in a future sprint.

[dawoe]

Hi @Migaroez and @ronaldbarendse

Did some further digging myself. It's the mapping of BlockList properties that is taking up all the time on my end. We have several of them. And some of the blocks in them even have blocklist properties.

If I prevent the following lines being executed for the block list editor my page loads in about 500ms.

https://github.com/umbraco/Umbraco-CMS/blob/contrib/src/Umbraco.Core/Models/Mapping/ContentPropertyBasicMapper.cs#L94

https://github.com/umbraco/Umbraco-CMS/blob/contrib/src/Umbraco.Core/Models/Mapping/ContentPropertyDisplayMapper.cs#L50

https://github.com/umbraco/Umbraco-CMS/blob/contrib/src/Umbraco.Core/Models/Mapping/ContentPropertyDisplayMapper.cs#L78

So there is something fishy going on when getting the data for the block list editor.

[cornehoskam]

After trying out the possible pullrequest by @dawoe, the performance seems to be improved for all intents and purposes that use the ContentService to fetch content from a uMarketingSuite perspective, especially when fetching content items in bulk.

[dawoe]

I just did some digging. Our customer started reporting a slow backoffice after we upgraded from 10.2.x to 10.4.x

[jmcaveney]

I've looked a bit at this, since we're having similar issues. we have 6 languages and nodes with three different block editors on them, and the load times for GetById are sometimes 14+ seconds on Umbraco Cloud for those nodes.

As @ronaldbarendse mentioned, the loading of the datatypes, specifically this call on line 213, seem to be the cause:
https://github.com/umbraco/Umbraco-CMS/blob/release/10.8/src/Umbraco.Infrastructure/PropertyEditors/BlockEditorPropertyValueEditor.cs#L213

This is done multiple times for every block property, even if the data type configurations are shared across blocks. Checking the dictionary of IDataValueEditors before calling to the database to get the data types seems to improve the load times significantly.

        int dataTypeId = prop.Value.PropertyType.DataTypeId;
        if (!valEditors.TryGetValue(dataTypeId, out IDataValueEditor? valEditor) || valEditor == null)
        {
            IDataType? dataType = _dataTypeService.GetDataType(dataTypeId);
            if (dataType == null)
            {
                // deal with weird situations by ignoring them (no comment)
                row.PropertyValues.Remove(prop.Key);
                _logger.LogWarning(
                    "ToEditor removed property value {PropertyKey} in row {RowId} for property type {PropertyTypeAlias}",
                    prop.Key,
                    row.Key,
                    property.PropertyType.Alias);
                continue;
            }

            var tempConfig = dataType.Configuration;
            valEditor = propEditor.GetValueEditor(tempConfig);

            valEditors.Add(dataTypeId, valEditor);
        }
        // update the raw value since this is what will get serialized out
        row.RawPropertyValues[prop.Key] = valEditor.ToEditor(tempProp);

It's not, however, the entirety of the problem. It seems to knock the load times by about 30% but they're still pretty slow. I plan to look at it further next week when I'm actually in the office and on the same network as the SQL server I'm referencing.

I would assume we could cache those data type calls across all block editors per content request. Otherwise we're loading data types for every language and every block editor and then for both settings and content sections of each block editor.