Status: 413 (Request Entity Too Large)

[Shazwazza]

Describe the bug
When performing large indexing operations, this may result in Status: 413 (Request Entity Too Large)

License ID
N/A

To Reproduce
The batch size is set to 1000 which is the max batch size for Azure Search, however there may be cases where the actual byte size of the payload is too big.

Expected behavior
A configurable batch size would be nice to have. ExamineX can also try to split the problematic batch in half and recurse.

Screenshots
N/A

Versions

• 6.0.7

[Matthew-Wise]

Hi @Shazwazza this is the logs we see when this occurs. It does duplicate the message so I assume that's one retry

---- alot more of ensuring fields and missing x ---
[17:38:40 DBG] (ProcessIndexBatch) Processing index batch size 1000
[17:39:04 DBG] Ensuring fields
[17:39:04 DBG] Ensuring fields, missing 1
[17:39:04 DBG] (ProcessIndexBatch) Process index batch
[17:39:04 DBG] (ProcessIndexBatch) Processing index batch size 1000
[17:39:29 ERR] (ProcessIndexBatch) Unhandled exception occurred during indexing.
Azure.RequestFailedException: Service request failed.
Status: 413 (Request Entity Too Large)

Content:
The page was not displayed because the request entity is too large.

Headers:
Server: Microsoft-IIS/10.0
Strict-Transport-Security: REDACTED
Date: Mon, 22 Jul 2024 16:39:04 GMT
Connection: close
Content-Type: text/html
Content-Length: 67

   at Azure.Search.Documents.SearchClient.IndexDocumentsInternal[T](IndexDocumentsBatch`1 batch, IndexDocumentsOptions options, Boolean async, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](Task`1 task)
   at Azure.Search.Documents.SearchClient.IndexDocuments[T](IndexDocumentsBatch`1 batch, IndexDocumentsOptions options, CancellationToken cancellationToken)
   at ExamineX.AzureSearch.AzureSearchIndex.A[A,a](IEnumerable`1, Func`2, Func`3, CancellationToken, String callerMember)

[Shazwazza]

Thanks @Matthew-Wise. Version 6.0.8 has been published to resolve this. It will try to automatically reduce the batch size if it encounters this issue. There's also a new configuration option: AzureSearchOptions.IndexBatchSize which are configured via IOptions. This defaults to 1000 but can be adjusted either by code or by:

"ExamineX": {
  "AzureSearch": {
    "IndexBatchSize ": 999
  }
}

[Matthew-Wise]

Thanks Shannon for the fast fix!

[Shazwazza]

Thanks for the report and help :)

I've updated the docs in a couple places:

• https://examinex.online/media
• https://examinex.online/azure-search-limits

Also just FYI, there will be another version published hopefully later this week which will contain a new Umbraco Health Check to ensure that your blob storage media files are all tagged correctly. Without this health check, it can be a bit annoying to get any previously existing media in blob storage to have the correct metadata and if any of the files are missing the metadata, the Azure Search indexer will not run correctly.

And, in the near future we will be publishing support for all of the Image analysis features for Blob media so that you can have all of your images in Umbraco indexed with AI generated descriptions, tags, etc... 🚀