Performance Degradation in Umbraco Backoffice After Upgrade to 15.3.0

[martinthogersen]

Which Umbraco version are you using?

15.3.0

Bug summary

We recently upgraded a website from Umbraco version 15.2.2 to 15.3.0. After the upgrade, the performance of opening pages in the Umbraco backoffice has significantly deteriorated. The image below illustrates the difference: on the left side, you can see the load times for Umbraco 15.3.0, while the right side shows the performance in version 15.2.2. Both versions are running in the same environment.

Here are the full paths to the resources that experience slow loading times:

• /umbraco/management/api/v1/document/f3d39119-032f-4496-8279-973a5798fd48
• /umbraco/management/api/v1/document/f3d39119-032f-4496-8279-973a5798fd48/published
• /umbraco/management/api/v1/document/urls?id=f3d39119-032f-4496-8279-973a5798fd48

This issue is not limited to a single page; it affects the entire backoffice. Additionally, the associated document type has no properties.

Specifics

No response

Steps to reproduce

See above.

Expected result / actual result

No response

---

This item has been added to our backlog AB#51636

[github-actions]

Hi there @martinthogersen!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[martinthogersen]

The database schema remains unchanged, except for the addition of two properties to the umbracoWebhook table:

[AndyButland]

Is there a possibility of privately sharing a copy of the database with us here @martinthogersen please, so we can more easily debug what's going on here?

[martinthogersen]

@AndyButland The database contains sensitive information that requires an NDA.
Would it be possible to debug the issue using screen sharing?

[AndyButland]

Understood, just thought I would ask as sometimes that is feasible and is a useful way to debug. I don't think screen sharing would really help, as we'd only see what you've already shared. My idea was to try to hook it up to the source code and debug to see what might be taking so long.

[nielslyngsoe]

Hi @martinthogersen

First, thanks for reporting this issue, and as well very nice that you have been able to pin-point the situation to concern 15.3.0.

I would like to ask a for a few more details in regards to your project, as that would help us figure out what could be the trigger in this project. But I can only be guessing, so ideally, you can append the details you see relevant.

What I'm mainly interested in knowing is which extensions you have in this project. And if you have any customizations, I'm specifically interested in C#/Backend customizations/additions.

Thanks in advance

[martinthogersen]

@nielslyngsoe We do not have any custom extensions that would impact performance.

Below are the references for the project:

• Total members: 298,672
• Documents in the internal index: 72,527
• Supported cultures: 16

Package References:

<PackageReference Include="Umbraco.Cms" Version="15.3.0" />
<PackageReference Include="CsvHelper" Version="33.0.1" />
<PackageReference Include="Stripe.net" Version="47.4.0" />
<PackageReference Include="Microsoft.Office.Interop.PowerPoint" Version="15.0.4420.1018" />
<PackageReference Include="MaxMind.GeoIP2" Version="5.2.0" />
<PackageReference Include="uSync" Version="15.1.5" />
<PackageReference Include="IronPdf" Version="2023.10.3" />
<PackageReference Include="SkiaSharp" Version="2.88.6" />
<PackageReference Include="SkiaSharp.Svg" Version="1.60.0" />
<PackageReference Include="IronPdf.Native.Chrome.MacOS.ARM" Version="2023.9.19098" />
<PackageReference Include="Microsoft.IdentityModel.Protocols.OpenIdConnect" Version="8.7.0" />
<PackageReference Include="SharpZipLib" Version="1.4.2" />

[AndyButland]

I've been investigating this @martinthogersen - mainly trying to pinpoint what may have changed between 15.2 and 15.3 that may have triggered this regression you see. But nothing has jumped out at me yet. I can see you have a lot of members, but the requests you are indicating are all for single documents, so it's not obvious that this should be having an effect.

It might be interesting to have you try a couple of things please, just to see if we can pinpoint what might be going on a bit more closely.

Firstly the database. The following is a SQL request for a single document when requested via the content service. Could you try it and see if the response comes back in a reasonable time on your 15.3 database, and, if possible for completeness, also your 15.2 (though I can't see any reason why they should be different given there's no related schema updates between versions).

DECLARE @id uniqueidentifier
SET @id = 'ca4249ed-2b23-4337-b522-63cabe5587d1'

SELECT [umbracoDocument].[nodeId] AS [NodeId], [umbracoDocument].[published] AS [Published], [umbracoDocument].[edited] AS [Edited]
, [umbracoContent].[nodeId] AS [ContentDto__NodeId], [umbracoContent].[contentTypeId] AS [ContentDto__ContentTypeId]
, [umbracoNode].[id] AS [ContentDto__NodeDto__NodeId], [umbracoNode].[uniqueId] AS [ContentDto__NodeDto__UniqueId], [umbracoNode].[parentId] AS [ContentDto__NodeDto__ParentId], [umbracoNode].[level] AS [ContentDto__NodeDto__Level], [umbracoNode].[path] AS [ContentDto__NodeDto__Path], [umbracoNode].[sortOrder] AS [ContentDto__NodeDto__SortOrder], [umbracoNode].[trashed] AS [ContentDto__NodeDto__Trashed], [umbracoNode].[nodeUser] AS [ContentDto__NodeDto__UserId], [umbracoNode].[text] AS [ContentDto__NodeDto__Text], [umbracoNode].[nodeObjectType] AS [ContentDto__NodeDto__NodeObjectType], [umbracoNode].[createDate] AS [ContentDto__NodeDto__CreateDate]
, [umbracoDocumentVersion].[id] AS [DocumentVersionDto__Id], [umbracoDocumentVersion].[templateId] AS [DocumentVersionDto__TemplateId], [umbracoDocumentVersion].[published] AS [DocumentVersionDto__Published]
, [umbracoContentVersion].[id] AS [DocumentVersionDto__ContentVersionDto__Id], [umbracoContentVersion].[nodeId] AS [DocumentVersionDto__ContentVersionDto__NodeId], [umbracoContentVersion].[versionDate] AS [DocumentVersionDto__ContentVersionDto__VersionDate], [umbracoContentVersion].[userId] AS [DocumentVersionDto__ContentVersionDto__UserId], [umbracoContentVersion].[current] AS [DocumentVersionDto__ContentVersionDto__Current], [umbracoContentVersion].[text] AS [DocumentVersionDto__ContentVersionDto__Text], [umbracoContentVersion].[preventCleanup] AS [DocumentVersionDto__ContentVersionDto__PreventCleanup]
, [pdv].[id] AS [PublishedVersionDto__Id], [pdv].[templateId] AS [PublishedVersionDto__TemplateId], [pdv].[published] AS [PublishedVersionDto__Published]
, [pcv].[id] AS [PublishedVersionDto__ContentVersionDto__Id], [pcv].[nodeId] AS [PublishedVersionDto__ContentVersionDto__NodeId], [pcv].[versionDate] AS [PublishedVersionDto__ContentVersionDto__VersionDate], [pcv].[userId] AS [PublishedVersionDto__ContentVersionDto__UserId], [pcv].[current] AS [PublishedVersionDto__ContentVersionDto__Current], [pcv].[text] AS [PublishedVersionDto__ContentVersionDto__Text], [pcv].[preventCleanup] AS [PublishedVersionDto__ContentVersionDto__PreventCleanup]
, COALESCE([ccv].[name],[umbracoNode].[text]) AS variantName
FROM [umbracoDocument]
INNER JOIN [umbracoContent]
ON [umbracoDocument].[nodeId] = [umbracoContent].[nodeId]
INNER JOIN [umbracoNode]
ON [umbracoContent].[nodeId] = [umbracoNode].[id]
INNER JOIN [umbracoContentVersion]
ON ([umbracoDocument].[nodeId] = [umbracoContentVersion].[nodeId])
INNER JOIN [umbracoDocumentVersion]
ON ([umbracoContentVersion].[id] = [umbracoDocumentVersion].[id])
LEFT JOIN [umbracoContentVersion] [pcv]
INNER JOIN [umbracoDocumentVersion] [pdv]
ON (([pcv].[id] = [pdv].[id]) AND [pdv].[published] = 1)
ON ([umbracoDocument].[nodeId] = [pcv].[nodeId])
LEFT JOIN [umbracoContentVersionCultureVariation] [ccv]
INNER JOIN [umbracoLanguage] [lang]
ON (([ccv].[languageId] = [lang].[id]) AND ([lang].[languageISOCode] = '[[[ISOCODE]]]'))
ON ([umbracoContentVersion].[id] = [ccv].[versionId])
WHERE (([umbracoNode].[nodeObjectType] = 'C66BA18E-EAF3-4CFF-8A22-41B16D66A972'))
AND ([umbracoContentVersion].[current] = 1)
AND (([umbracoNode].[uniqueId] = @id))

Then we might look at the service calls. This code that you can just place in a view will split the two calls made for the second endpoint you've highlighted of concern. You should restart to see the result, just to ensure any memory caches are cleared. And you'll need to adjust to use a GUID for one of your published documents.

@inject Umbraco.Cms.Core.Services.IContentService ContentService
@inject Umbraco.Cms.Api.Management.Factories.IDocumentPresentationFactory DocumentPresentationFactory;
@{
  var id = Guid.Parse("202CDC2F-1A45-40F4-A653-A7B321E1C54C");
  var stopWatch = new System.Diagnostics.Stopwatch();
  stopWatch.Start();
  
  var beforeMs = stopWatch.ElapsedMilliseconds;
  var content = ContentService.GetById(id);
  var afterMs = stopWatch.ElapsedMilliseconds;
  @if (content is null || content.Published is false)
  {
  	throw new InvalidOperationException("Could not get expected published content.");
  }
  
  <div>Got content in: @(afterMs - beforeMs)ms</div>
  
  beforeMs = stopWatch.ElapsedMilliseconds;
  var model = await DocumentPresentationFactory.CreatePublishedResponseModelAsync(content);
  afterMs = stopWatch.ElapsedMilliseconds;
  <div>Prepared model in: @(afterMs - beforeMs)ms</div>
}

Of course this isn't going to solve anything immediately, but it should be useful to help identify the area of further investigation. Thanks for your help with this.

[martinthogersen]

@AndyButland Thank you for the help.

The SQL query executed very quickly, so that doesn't appear to be the issue - both Umbraco 15.2 and 15.3 returned results promptly.

Here's the output from the code snippet you shared:

Got content in: 5ms
Prepared model in: 3872ms

[AndyButland]

Thanks - fairly clear that the issue isn't in the content retrieval, rather in something in the model preparation. Please could you try the same with the following (am just looking to break it down a little more).

    @inject Umbraco.Cms.Core.Services.IContentService ContentService
    @inject Umbraco.Cms.Core.Mapping.IUmbracoMapper UmbracoMappper;
    @inject Umbraco.Cms.Api.Management.Factories.IDocumentUrlFactory DocumentUrlFactory;
    @inject Umbraco.Cms.Core.Services.ITemplateService TemplateService;
    @{
        var id = Guid.Parse("202CDC2F-1A45-40F4-A653-A7B321E1C54C");
        var stopWatch = new System.Diagnostics.Stopwatch();
        stopWatch.Start();

        var beforeMs = stopWatch.ElapsedMilliseconds;
        var content = ContentService.GetById(id);
        var afterMs = stopWatch.ElapsedMilliseconds;
        @if (content is null || content.Published is false)
        {
            throw new InvalidOperationException("Could not get expected published content.");
        }

        <div>Got content in: @(afterMs - beforeMs)ms</div>

        beforeMs = stopWatch.ElapsedMilliseconds;
        var responseModel = UmbracoMappper.Map<Umbraco.Cms.Api.Management.ViewModels.Document.PublishedDocumentResponseModel>(content)!;
        afterMs = stopWatch.ElapsedMilliseconds;
        <div>Mapped model in: @(afterMs - beforeMs)ms</div>

        beforeMs = stopWatch.ElapsedMilliseconds;
        var urls = DocumentUrlFactory.CreateUrlsAsync(content);
        afterMs = stopWatch.ElapsedMilliseconds;
        <div>Mapped URLs in: @(afterMs - beforeMs)ms</div>

        @if (content.PublishTemplateId.HasValue)
        {
            beforeMs = stopWatch.ElapsedMilliseconds;
            var template = await TemplateService.GetAsync(content.PublishTemplateId.Value);
            afterMs = stopWatch.ElapsedMilliseconds;
            <div>Got template in: @(afterMs - beforeMs)ms</div>
        }
    }