How to re-ingest years of data into FinOps-Hub after upgrade

[francesco1119]

PROBLEM: last week I had a problem in the FinOps-Hub and the solution was to upgrade from v12 to v13.

The problem was that after upgrade only the last 13 months were showing up while in reality I had 6 years of data.

The solution would have been re-export 6 years of data but life is too short to iron your underpants so...

SOLUTION: The AI built for me an Azure Automation Account which runbook

1. Iterate into the ingestion folder
2. Search for all the manifest.json file
3. Adds a blank space at the end of each file and saves

This will trig the ADF to re-ingest all data into ADX

Here the code:

<#
  .SYNOPSIS
      Re-triggers FinOps Hub ingestion by appending a trailing space to all
      manifest.json files in the ingestion container.

  .DESCRIPTION
      Azure Automation Runbook. Requires a System-assigned Managed Identity with
      "Storage Blob Data Contributor" role on the storage account.
      Run once on-demand whenever a full re-ingestion is needed.
  #>

  param (
      [string]$StorageAccountName = "Put-your-Storage-Account-name-here",
      [string]$ContainerName      = "ingestion",
      [string]$SubscriptionId     = "Put-your-Subscription-ID-here"
  )

  # --- Auth via Managed Identity ---
  Write-Output "Authenticating with Managed Identity..."
  Connect-AzAccount -Identity | Out-Null
  Set-AzContext -SubscriptionId $SubscriptionId | Out-Null

  # --- Storage context (Entra ID token, no keys) ---
  $ctx = New-AzStorageContext -StorageAccountName $StorageAccountName -UseConnectedAccount

  # --- List all manifest.json blobs ---
  Write-Output "Listing manifest.json files in container '$ContainerName'..."
  $blobs = Get-AzStorageBlob -Container $ContainerName -Context $ctx |
           Where-Object { $_.Name -like "*/manifest.json" }

  Write-Output "Found $($blobs.Count) manifest.json files. Starting update..."

  $success = 0
  $failed  = 0

  foreach ($blob in $blobs) {
      try {
          $content    = $blob.ICloudBlob.DownloadText([System.Text.Encoding]::UTF8)
          $newContent = $content.TrimEnd() + " "
          $bytes      = [System.Text.Encoding]::UTF8.GetBytes($newContent)
          $memStream  = [System.IO.MemoryStream]::new($bytes)
          $blob.ICloudBlob.UploadFromStream($memStream)
          $memStream.Dispose()

          Write-Output "  [OK] $($blob.Name)"
          $success++
      }
      catch {
          Write-Warning "  [FAIL] $($blob.Name) — $_"
          $failed++
      }
  }

  Write-Output ""
  Write-Output "Done. Success: $success | Failed: $failed"

Edit this parameters:

  param (
      [string]$StorageAccountName = "Put-your-Storage-Account-name-here",
      [string]$ContainerName      = "ingestion",
      [string]$SubscriptionId     = "Put-your-Subscription-ID-here"
  )

SUGGESTION: I'm not an ADF lover but I think we should put in place an ADF pipeline that is trigged every time the FinOps-Hub changes version.

Or ideally we can use this Azure Automation Account.
But for consistency I would prefer to put everything in ADF, right?