Fix Costs_v1_2 duplicate records from Services table join

[Copilot]

🛠️ Description

The Costs_v1_2() function produced duplicate cost records when the Services table contained multiple entries for the same x_ResourceType, causing inflated cost values in v12 dashboards compared to v11.

Root cause: join kind=leftouter without deduplication created N output rows for each input row when Services had N matching entries.

Fix: Changed to lookup kind=leftouter with summarize take_any() to ensure 1:1 enrichment:

// Before: Creates duplicates when Services has multiple rows per x_ResourceType
| join kind=leftouter (Services | where isnotempty(x_ResourceType) | project x_ResourceType, ServiceSubcategory, x_ServiceModel) on x_ResourceType

// After: Ensures single match per x_ResourceType
| lookup kind=leftouter (Services | where isnotempty(x_ResourceType) | summarize take_any(ServiceSubcategory), take_any(x_ServiceModel) by x_ResourceType) on x_ResourceType

This pattern already exists in CommitmentDiscountUsage_v1_2() line 27.

📷 Screenshots

The issue manifested when resource types like microsoft.security/pricings appeared multiple times in Services with different subcategories (Application Platforms, Containers, Security Posture Management, Virtual Machines), multiplying cost records 5x.

📋 Checklist

🔬 How did you test this change?

• 🤏 Lint tests
• 🤞 PS -WhatIf / az validate
• 👍 Manually deployed + verified
• 💪 Unit tests
• 🙌 Integration tests

🙋‍♀️ Do any of the following that apply?

• 🚨 This is a breaking change.
• 🤏 The change is less than 20 lines of code.

📑 Did you update docs/changelog.md?

• ✅ Updated changelog (required for dev PRs)
• ➡️ Will add log in a future PR (feature branch PRs only)
• ❎ Log not needed (small/internal change)

📖 Did you update documentation?

• ✅ Public docs in docs (required for dev)
• ✅ Internal dev docs in src (required for dev)
• ➡️ Will add docs in a future PR (feature branch PRs only)
• ❎ Docs not needed (small/internal change)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Cost Price differs in Finops 11 vs 12 Dashboard</issue_title>
<issue_description>

🐛 Problem

I have upgraded the Finops 11 to 12. After that I have deployee the FinOps 12 Dashboard.
Cost Price differs in Finops 11 vs 12 Dashboard

👣 Repro steps

Open Finops Dashboard 11 and 12
Compare the budgeting Under Quantify
Sumary for Billed Cost month wise.

🤔 Expected

Both Dashboard should have same datta.

📷 Screenshots

TODO: If applicable, add screenshots to help explain your problem. Remove if not applicable.

ℹ️ Additional context

I run the export again for historical data and its is getting corrected.

🙋‍♀️ Ask for the community

Should have same data in both the Dashboard
</issue_description>

Comments on the Issue (you are @copilot in this section)

@RolandKrummenacher Hi @gajadv . Do you only see this difference in this dashboard or in the data in general? For example by comparing the results of the following queries:

Costs_v1_0
| summarize
BilledCost = todouble(round(sum(BilledCost), 2)),
EffectiveCost = todouble(round(sum(EffectiveCost), 2)),
ContractedCost = todouble(round(sum(ContractedCost), 2)),
ListCost = todouble(round(sum(ListCost), 2))
by
ChargePeriodStart = startofmonth(ChargePeriodStart)

Costs_v1_2
| summarize
BilledCost = todouble(round(sum(BilledCost), 2)),
EffectiveCost = todouble(round(sum(EffectiveCost), 2)),
ContractedCost = todouble(round(sum(ContractedCost), 2)),
ListCost = todouble(round(sum(ListCost), 2))
by
ChargePeriodStart = startofmonth(ChargePeriodStart)</comment_new>
<comment_new>@RolandKrummenacher
@gajadv There is no good reason why querying Costs_v1_0 in your Hub database would return different values than Costs_v1_2. Both functions project the data from the exact same underlying tables, therefore the results should be identical under normal conditions.

If backfilling your data resolves the discrepancy that is of course fine, but it should not be necessary.

@RolandKrummenacher @flanakin I've found an issue in Costs_v1_2() function in the hub database. The following line is leading to duplicated records:

because:

A quick fix would be to join with this subquery, but I don't fully understand the intention of this join at all:

| lookup kind=leftouter(Services | where isnotempty(x_ResourceType) | summarize take_any(ServiceSubcategory), take_any(x_ServiceModel) by x_ResourceType | project x_ResourceType, ServiceSubcategory, x_ServiceModel) on x_ResourceType</comment_new>
<comment_new>@RolandKrummenacher
Also a kql-lookup would be much better suited than a kql-join.</comment_new>
<comment_new>@RolandKrummenacher
This also explains why backfilling resolved the issue for @gajadv . When the data was backfilled it landed in Costs_final_v1_2 (instead of Costs_final_1_0), where the problem does not exists since the Costs_v1_2() function does not have to make a transformation/projection on the fly.</comment_new>
<comment_new>@RolandKrummenacher

what should I use.

If you have done the backfill already, then you should be good. I recommend to use the Costs_v1_2 function.</comment_new>

• Fixes Cost Price differs in Finops 11 vs 12 Dashboard #1888

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[RolandKrummenacher]

@flanakin Is this a proper fix?