[ADF] config_InitializeHub: "HTTP redirects are not allowed" in KQL "externaldata" command – evaluate cleaner solution

[RolandKrummenacher]

🐛 Problem

When initializing the FinOps Hub the step that populates the PricingUnits table fails.
The KQL command uses externaldata to read PricingUnits.csv from a GitHub releases URL and returns

HTTP client error: redirects are not allowed

so the PricingUnits table is not created or updated.

The failing command:

.set-or-replace PricingUnits <|
    externaldata(x_PricingUnitDescription: string, AccountTypes: string, x_PricingBlockSize: real, PricingUnit: string)
    [@"https://github.com/microsoft/finops-toolkit/releases/download/v12/PricingUnits.csv"]
    with (format = "csv", ignoreFirstRecord = true)
    | project-away AccountTypes

This looks like the GitHub releases download URL now returns a redirect and Azure Data Explorer does not follow redirects for externaldata().

👣 Repro steps

1. Deploy the FinOps Hub (v12) with Azure Data Explorer as the hub database.
2. In Data Factory run pipeline config_InitializeHub.
3. Inside the pipeline the Azure Data Explorer command activity Upserts PricingUnits runs the command above.
4. The activity fails with a partial query failure and the error message HTTP client error: redirects are not allowed.

Optional confirmation:
5. Run the same .set-or-replace PricingUnits command directly in the Kusto Web UI against the hub database.
6. Observe the same error.

🤔 Expected

The pipeline config_InitializeHub should complete successfully and the PricingUnits table should be created or updated from PricingUnits.csv without errors.

📷 Screenshots

1. Kusto Web UI showing the failing query and error message.

2. Data Factory pipeline config_InitializeHub with the Azure Data Explorer command activity that runs the same statement.

ℹ️ Additional context

• Environment: FinOps Hub v12, Azure Data Explorer as hub, Data Factory based deployment.

• The error text in Kusto is:

  Partial query failure: Unspecified error (message: HTTP client error: redirects are not allowed, details: StgError { kind: Generic("HTTP client error: redirects are not allowed"), source: None })

• This started to happen when using the GitHub releases URL
  https://github.com/microsoft/finops-toolkit/releases/download/v12/PricingUnits.csv
  which appears to return a redirect to another host.

• As a workaround the CSV can be copied to a storage account and referenced from there with a direct URL that does not redirect.

• Same issue with fabric based deployments

• Same issue with Region.csv, ResourceTypes.csv, Services.csv

Suggested fix ideas:

• Host PricingUnits.csv at a non redirecting URL (for example under raw.githubusercontent.com or in the repo itself) and update the KQL script.
• Or include the pricing units as an inline datatable during initialization.

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) if you are also seeing the same failure in your FinOps Hub deployment.
2. Add comments if you have confirmation on which URLs work correctly with externaldata or if you have a better suggestion for hosting PricingUnits.csv.

[RolandKrummenacher]

Related to #1885

[RolandKrummenacher]

We could use this base url for PriceingUnits, Regions, ResourceTypes and Services Open Data: https://raw.githubusercontent.com/microsoft/finops-toolkit/refs/tags/v12/src/open-data/PricingUnits.csv