feat(hubs): Draft FOCUS 1.5 SkuPrices preview

docs-mslearn/toolkit/changelog.md

@@ -3,7 +3,7 @@ title: FinOps toolkit changelog
 description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more.
 author: MSBrett
 ms.author: brettwil
-ms.date: 09/11/2026
+ms.date: 10/26/2026
 ms.topic: reference
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -29,6 +29,7 @@ The following section lists features and enhancements that are currently in deve
 
 - **Added**
   - Added VNet and private network modes, including opt-in NAT Gateway support for private mode; NAT Gateway incurs additional cost when enabled ([#2163](https://github.com/microsoft/finops-toolkit/pull/2163)).
+  - Added a draft FOCUS 1.5 SKU Price preview (`SkuPrices_v1_5` and `SkuPricesWide_v1_5`) with validation and scenario scripts under `Analytics/preview/SkuPrices_v1_5`. The preview is not deployed with FinOps hubs; run it manually against a hub ([#2339](https://github.com/microsoft/finops-toolkit/pull/2339)).
 - **Changed**
   - Clarified that the FinOps toolkit exclusively manages the FinOps hub virtual network and documented customer-managed private endpoints as the preferred private-access topology, with virtual network peering as a secondary option ([#2156](https://github.com/microsoft/finops-toolkit/issues/2156)).
   - Replaced redundant `tolower()` comparisons in hub KQL with case-insensitive operators (`has`, `=~`, `!~`) so the engine can use the term index instead of scanning every row ([#2213](https://github.com/microsoft/finops-toolkit/issues/2213)).

docs-mslearn/toolkit/hubs/data-model.md

@@ -3,7 +3,7 @@ title: FinOps hubs data model
 description: Learn about the tables and functions available in FinOps hubs to build your own queries, reports, and dashboards.
 author: flanakin
 ms.author: micflan
-ms.date: 04/01/2026
+ms.date: 10/26/2026
 ms.topic: reference
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -642,6 +642,15 @@ Columns in the **Prices** managed dataset include:
 | SkuMeter                             | String    | New in FOCUS 1.2. Renamed from x_SkuMeterName.    |
 | SkuPriceId                           | String    |                                                   |
 | SkuPriceIdv2                         | String    | Source: Hubs add-on.                              |
+
+### FOCUS 1.5 SKU prices preview
+
+A draft of the FOCUS 1.5 SKU Price dataset is available in the `src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/preview/SkuPrices_v1_5` folder of the [FinOps toolkit repository](https://github.com/microsoft/finops-toolkit). It isn't deployed with FinOps hubs and may change before release. The preview builds 2 tables from **Prices_final_v1_2** in the **Ingestion** database:
+
+- **SkuPrices_v1_5** has one row per SKU price and price type (`x_UnitPriceType` of List, Base, Contracted, or Effective) with effective start and end dates.
+- **SkuPricesWide_v1_5** has the same prices with each price type in its own column.
+
+To try it, run `Build-SkuPrices.ps1` against your hub, then `Test-SkuPrices.ps1` to validate the results. See the README in that folder for mapping decisions and known gaps.
 | x_BaseUnitPrice                      | Real      | Source: Microsoft.                                |
 | x_BillingAccountAgreement            | String    | Source: Hubs add-on.                              |
 | x_BillingAccountId                   | String    | Source: Microsoft, MCA only.                      |

src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/preview/SkuPrices_v1_5/Build-SkuPrices.ps1

@@ -0,0 +1,92 @@
+<#
+.SYNOPSIS
+Rebuilds the FOCUS 1.5 SkuPrices preview tables on a hub from Prices_final_v1_2 and Costs_final_v1_2 using SkuPrices_v1_5.kql.
+
+.DESCRIPTION
+1. Creates or updates every function in the KQL file.
+2. Builds SkuPrices_v1_5_runs_table (runs of unchanged prices) for each price type and the wide layout.
+3. Builds SkuPrices_v1_5 (rows layout) one month and price type at a time, and SkuPricesWide_v1_5 (columns layout) one month at a time.
+4. Builds SkuPriceIdv2_map and CostsWithSkuPriceIdv2 (Costs copy with SkuPriceIdv2) one month at a time.
+Small steps keep each query within memory on small clusters.
+#>
+[CmdletBinding(SupportsShouldProcess)]
+param(
+  [string] $Cluster = 'https://ftk-dev.westus.kusto.windows.net',
+  [string] $Database = 'Ingestion',
+  [string] $Context = 'fh-dev',
+  [string] $KqlPath = "$PSScriptRoot/SkuPrices_v1_5.kql",
+  [switch] $SkipCosts
+)
+
+$ErrorActionPreference = 'Stop'
+
+$ctx = Get-AzContext -Name $Context
+if (-not $ctx) { throw "Az context '$Context' not found" }
+Set-AzContext -Context $ctx -Scope Process -WhatIf:$false | Out-Null  # This process only; leaves the saved default alone
+$secure = (Get-AzAccessToken -ResourceUrl $Cluster -AsSecureString).Token
+$headers = @{ Authorization = "Bearer $([System.Net.NetworkCredential]::new('', $secure).Password)"; 'Content-Type' = 'application/json' }
+
+function Invoke-Kusto {
+  param([string] $Csl, [string] $Endpoint = 'mgmt')
+  $body = @{ db = $Database; csl = $Csl; properties = @{ Options = @{ servertimeout = '01:00:00' } } } | ConvertTo-Json -Depth 5
+  (Invoke-RestMethod -Uri "$Cluster/v1/rest/$Endpoint" -Method Post -Headers $headers -Body $body -TimeoutSec 3600).Tables[0]
+}
+
+# Run one command, timing it; the first command for a table replaces it, the rest append
+function Invoke-Step([string] $Table, [string] $Query, [ref] $First, [string] $Label) {
+  $csl = if ($First.Value) { ".set-or-replace $Table with (folder = 'FOCUS 1.5 preview', recreate_schema = true) <| $Query" } else { ".append $Table <| $Query" }
+  if ($PSCmdlet.ShouldProcess($Table, $csl)) {
+    $sw = [Diagnostics.Stopwatch]::StartNew()
+    Invoke-Kusto $csl | Out-Null
+    Write-Host "$Label done in $([int]$sw.Elapsed.TotalSeconds)s"
+  }
+  $First.Value = $false
+}
+
+# 1. Functions from the KQL file (every .create-or-alter function block, in order)
+$raw = Get-Content -Path $KqlPath -Raw
+$pos = 0
+while (($start = $raw.IndexOf('.create-or-alter function', $pos)) -ge 0) {
+  $depth = 0; $end = -1
+  for ($i = $raw.IndexOf('{', $start); $i -lt $raw.Length; $i++) {
+    if ($raw[$i] -eq '{') { $depth++ }
+    elseif ($raw[$i] -eq '}') { $depth--; if ($depth -eq 0) { $end = $i + 1; break } }
+  }
+  if ($end -lt 0) { throw "Unbalanced function body in $KqlPath" }
+  $fn = $raw.Substring($start, $end - $start)
+  $name = [regex]::Match($fn, '\)\s*(\w+)\s*\(').Groups[1].Value
+  if ($PSCmdlet.ShouldProcess($name, '.create-or-alter function')) { Invoke-Kusto $fn | Out-Null; Write-Host "Updated $name" }
+  $pos = $end
+}
+
+$types = 'List', 'Base', 'Contracted', 'Effective'
+
+# 2. Runs of unchanged prices, per price type and for the wide layout
+$first = $true
+$parts = 8  # Hash partitions of keys, to stay within memory
+foreach ($type in $types + 'Wide') { for ($p = 0; $p -lt $parts; $p++) { Invoke-Step 'SkuPrices_v1_5_runs_table' "SkuPrices_v1_5_runs_build('$type', $p, $parts)" ([ref]$first) "Runs $type $p" } }
+
+# 3. Rows and columns layouts, one month (and price type) at a time
+$months = (Invoke-Kusto 'Prices_final_v1_2 | summarize by M = startofmonth(x_IngestionTime) | order by M asc' 'query').Rows | ForEach-Object { ([datetime]$_[0]).ToString('yyyy-MM-dd') }
+$firstRows = $true; $firstWide = $true
+foreach ($m in $months) {
+  foreach ($type in $types) { Invoke-Step 'SkuPrices_v1_5' "SkuPrices_v1_5_build('$type', datetime($m))" ([ref]$firstRows) "Rows $m $type" }
+  Invoke-Step 'SkuPricesWide_v1_5' "SkuPricesWide_v1_5_build(datetime($m))" ([ref]$firstWide) "Wide $m"
+}
+
+# 4. SkuPriceIdv2 map, then a Costs copy with SkuPriceIdv2, one month at a time
+if (-not $SkipCosts) {
+  $first = $true
+  Invoke-Step 'SkuPriceIdv2_map' 'SkuPriceIdv2_map_build()' ([ref]$first) 'Map'
+  $costMonths = (Invoke-Kusto 'Costs_final_v1_2 | summarize by M = startofmonth(ChargePeriodStart) | order by M asc' 'query').Rows | ForEach-Object { ([datetime]$_[0]).ToString('yyyy-MM-dd') }
+  $first = $true
+  foreach ($m in $costMonths) { Invoke-Step 'CostsWithSkuPriceIdv2' "CostsWithSkuPriceIdv2_build(datetime($m))" ([ref]$first) "Costs $m" }
+}
+
+# 5. Verify
+foreach ($t in 'SkuPrices_v1_5', 'SkuPricesWide_v1_5') {
+  $c = (Invoke-Kusto "$t | summarize Rows = count(), Current = countif(isnull(SkuPriceEffectiveEnd))" 'query').Rows[0]
+  $columns = (Invoke-Kusto "$t | getschema | project ColumnName" 'query').Rows | ForEach-Object { $_[0] }
+  $sorted = [string[]] $columns.Clone(); [Array]::Sort($sorted, [StringComparer]::Ordinal)
+  Write-Host "${t}: $($c[0]) rows ($($c[1]) current); $($columns.Count) columns; alphabetical: $(-not (Compare-Object $columns $sorted -SyncWindow 0))"
+}

src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/preview/SkuPrices_v1_5/README.md

@@ -0,0 +1,96 @@
+# FOCUS 1.5 SkuPrices (preview, draft)
+
+Draft FinOps hubs implementation of the FOCUS 1.5 **SKU Price** dataset ([FOCUS PR 2424](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec/pull/2424)), built from `Prices_final_v1_2`, with a comparison layout and scenarios to choose between them. Not wired into `app.bicep` or `.build.config` yet. Run manually against a hub.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `SkuPrices_v1_5.kql` | Functions + build commands for `SkuPrices_v1_5`, `SkuPricesWide_v1_5`, `CostsWithSkuPriceIdv2`, and helper tables (`SkuPrices_v1_5_runs_table`, `SkuPriceIdv2_map`) |
+| `Build-SkuPrices.ps1` | Creates the functions and builds all tables in small steps (key partitions, months, price types) |
+| `Test-SkuPrices.ps1` | Validation: duplicates, overlaps, current rows vs. latest sheet, rows vs. runs, price match, required columns, ranges |
+| `SkuPrices_v1_5_scenarios.kql` | 14 scenarios, each with the most efficient query found for each layout (plus today's monthly table for cost joins) |
+| `Test-SkuPriceScenarios.ps1` | Runs each query 3x, compares results, scores simplicity, time, CPU, memory, rows scanned, x_ usage, correctness |
+| `Render-SkuPriceReport.ps1` + `SkuPrices_v1_5_report.template.html` | Renders the side-by-side HTML report from scenario results and live table sizes |
+
+Scripts default to the `ftk-dev.westus` cluster, `Ingestion` database, and `fh-dev` Az context. Override with `-Cluster`, `-Database`, `-Context`.
+
+```powershell
+./Build-SkuPrices.ps1          # ~45 min on ftk-dev
+./Test-SkuPrices.ps1
+./Test-SkuPriceScenarios.ps1   # writes results JSON to the temp folder
+./Render-SkuPriceReport.ps1
+```
+
+## Layouts
+
+All three hold the same 14 months of prices.
+
+| | Today `Prices_final_v1_2` | Rows `SkuPrices_v1_5` | Columns `SkuPricesWide_v1_5` |
+|---|---|---|---|
+| Grain | Price per month | Price type per run of unchanged months | Price per contract per run of unchanged months |
+| Prices | Columns | `UnitPrice` + `x_UnitPriceType` (List, Base, Contracted, Effective) | `ListUnitPrice`, `ContractedUnitPrice`, `x_BaseUnitPrice`, `x_EffectiveUnitPrice` |
+| Dates | Month | Effective start/end per price type | One start/end per row (new row when any price changes) |
+| List price scope | n/a | Public: no contract, global eligibility | Scoped to the contract with the rest of the row |
+| Rows | 18.78M | 4.17M (3.89M current) | 1.79M (1.63M current) |
+| Extent | 2.34 GB | 0.64 GB | 0.33 GB |
+
+Rows by type (all / current): List 1,348,812 / 1,236,379; Contracted 1,326,353 / 1,236,379 (98% = List); Base 1,056,500 / 1,023,491 (94% = List); Effective 438,723 / 388,900.
+
+## Mapping decisions
+
+* `SkuPriceId` = `SkuPriceIdv2` (unique per price); source `SkuPriceId` kept as `x_SkuPriceIdv1`
+* `ContractId` = `/providers/microsoft.billing/billingaccounts/{x_BillingAccountId}` (EA enrollment / MCA billing account, not the profile)
+* Savings plan rows carry only the Effective price (list/base/contracted on those rows are copies of the on-demand meter's prices)
+* Effective dates are month-level runs of unchanged prices across monthly loads; current prices have no end
+* `PricingRegionId` = region name (`x_SkuRegion`); `PricingServiceName` = meter category; `SkuPriceCreated`/`LastUpdated` = first/last ingestion
+* `PurchaseDurationType` from `x_SkuTerm`; `PurchasePaymentModel` empty (not in price sheet)
+* Non-SKU Price source columns get an `x_` prefix; `CommitmentDiscountCategory` kept as a filter label; columns sorted alphabetically
+
+## Cost and Usage join
+
+Costs `SkuPriceId` doesn't match the price sheet's, and Costs lacks `x_SkuProductId`/`x_SkuMeterType`, so `SkuPriceIdv2` can't be rebuilt from Costs columns. `CostsWithSkuPriceIdv2` is a copy of `Costs_final_v1_2` with `SkuPriceIdv2` looked up from prices by price type + meter + offer + term:
+
+| Cost type | Price used | Rows matched | Cost matched |
+|---|---|---|---|
+| On-demand usage | On-demand | 100% | 100% |
+| Reservation-covered usage | On-demand (no reservation usage price) | 90.2% | 74.7% |
+| Savings-plan-covered usage | Savings plan for the term | 83.1% (68 rows have no meter) | 100% |
+| Reservation purchase | Reservation for the term (only when unique) | 16/16 | n/a |
+| Adjustment | None | 0% | 0% |
+
+Effective-dated prices have no range lookup in KQL. Scenarios spread each price over the months it was in effect and look up on `SkuPriceId` + `ContractId` + `PricingCurrency` + month: no range join and no duplicated cost rows. Today's monthly table needs no spreading.
+
+## Scenarios
+
+| Id | Scenario | Simpler | Faster | CPU | Memory | Rows scanned | FOCUS-only | Correct |
+|---|---|---|---|---|---|---|---|---|
+| S01 | List price on a date | Tie | Tie | Tie | Tie | Columns | Columns | Tie |
+| S02 | Negotiated discount by service | Columns | Columns | Columns | Columns | Columns | Columns | Tie |
+| S03 | Base price drift from list | Columns | Columns | Columns | Columns | Columns | Tie | Tie |
+| S04 | Reservation price vs. on-demand | Tie | Columns | Columns | Tie | Columns | Columns | Tie |
+| S05 | Savings plan rate vs. on-demand | Tie | Tie | Columns | Tie | Columns | Tie | Tie |
+| S06 | Price changes since July | Rows | Tie | Tie | Rows | Columns | Rows | Tie |
+| S07 | Cheapest region for a VM size | Tie | Tie | Tie | Tie | Columns | Columns | Tie |
+| S08 | Verify billed unit prices (+Monthly) | Tie | Tie | Tie | Tie | Columns | Columns | Tie |
+| S09 | Savings plan what-if for on-demand usage (+Monthly) | Tie | Monthly | Monthly | Tie | Columns | Tie | Tie |
+| S10 | All prices for one SKU | Tie | Tie | Tie | Tie | Columns | Rows | Tie |
+| S11 | Estimate a planned workload | Tie | Tie | Tie | Tie | Columns | Columns | Tie |
+| S12 | Public prices visible to each account | Tie | Rows | Rows | Tie | Columns | Columns | Rows |
+| S13 | Usage rates vs. purchase fees | Rows | Tie | Tie | Tie | Columns | Rows | Tie |
+| S14 | Backfill missing prices on cost data (+Monthly) | Monthly | Tie | Tie | Tie | Columns | Columns | Tie |
+| | **Wins (Rows / Columns / Monthly)** | 2 / 2 / 1 | 1 / 3 / 1 | 1 / 4 / 1 | 1 / 2 / 0 | 0 / 14 / 0 | 3 / 8 / 0 | 1 / 0 / 0 |
+
+Scores: winner per measure, or Tie within 15% of the runner-up. Simpler = operators + 2x joins + lets; Faster/CPU/Memory/Rows scanned = median of 3 server-measured runs; FOCUS-only = fewer distinct x_ columns referenced; Correct = which layout returns the right answer when they differ (S12: columns hide the public list price from accounts outside the agreement).
+
+Covers 9 of 13 FOCUS PR 2595 queries plus 5 more (S03, S05, S06, S10, S14). Not testable with this data: announced price changes, public vs. negotiated at a quantity, tier resolution, next tier (no future prices or tiers in EA price sheets), negotiated savings plan discount, multiple currencies, consumption currency. Cost scenarios use a price only when its block unit matches the charge's `PricingUnit`.
+
+## Gaps / TODO
+
+* [ ] `PricingUnit` and `x_PricingBlockSize` empty: `PricingUnits` open data table has 0 rows on ftk-dev, so `UnitPrice` is per block (e.g., per 10 hours); scenarios parse the block size from `x_PricingUnitDescription`
+* [ ] Savings plan list price ("list effective") is lost on ingestion: `Prices_transform_v1_2` nulls the SP `MarketPrice` and backfills on-demand prices; needed to measure negotiated SP discounts
+* [ ] Decide whether to skip Base/Contracted rows equal to List (lookups fall back to List)
+* [ ] MCA untested (collapses billing profiles into the billing account; check for profile-specific prices); latest month is global, not per account
+* [ ] Decide final layout once FOCUS settles the price type column (planned for 1.6)
+* [ ] Add `SkuPriceIdv2` to `Costs_transform_v1_2` (replaces `CostsWithSkuPriceIdv2`); find a reservation-covered usage price
+* [ ] Wire into `IngestionSetup_v1_5.kql` / `HubSetup_v1_5.kql`, `app.bicep`, `.build.config`; add `SkuPrices()` hub function, docs, changelog