Phase 1: portal roles, clean hosts before reuse, working scaling

[Paul Lizer (paullizer)]

A review of the admin side found several places where the solution did not do what it promised. It also found that the portal's access model made every signed-in tenant user an administrator:

• Scaling never started or stopped a VM. TriggerScalingLogic returned PoweredOn/PoweredOff, but the API matched PowerOn/PowerOff, so decisions were only recorded in SQL.
• Reconnect and cleanup timing was wrong.
  • The released-VM sweep used a fixed 30 minutes instead of the configured grace period.
  • A returned host could go to the next user before the previous user's account was removed, because userdel failures were masked.
  • The host agent killed the desktop as soon as a user disconnected, so "resume within the grace period" only meant getting the same VM and profile back.
• Anyone who could sign in to the portal had full control. Holding the delegated access_as_user scope was enough to delete VMs or change settings, and the portal's checkout sent the Linux password to the browser.
• Readiness and throughput problems. Started hosts could stay "unreachable" in the broker for up to an hour. The API ran one sync gunicorn worker and fetched the signing keys and the SSH key from Key Vault on every call.

This is Phase 1: correctness, access control and performance. Phases 2-4 (admin console, OS and desktop support, strategic scale) and a security-hardening backlog are designed in docs/ROADMAP.md.

What changed

Access control

• Every endpoint enforces the Reader, Operator or FullAccess app role, or its service role. The delegated scope grants nothing on its own.
• ALLOW_LEGACY_SCOPE_ACCESS (Bicep allowLegacyScopeAccess, default false) bridges an upgrade while roles are assigned.
• GET /api/me drives a role-aware portal: a No access page, a read-only banner and gated actions. A unit test pins every route to its roles.
• The portal no longer receives the Linux password or lease from checkout.

Release lifecycle

• New CleanupPending state. The sweep, a manual return and a failed checkout all claim cleanup, and the VM cannot be checked out until the user is actually gone from the host.
• Cleanup is retried automatically about every 2 minutes, or on demand with "Retry cleanup".
• manage-lease.sh keeps the lease while the user is signed in. Otherwise it ends leftover processes such as tmux or nohup jobs.
• The API only treats userdel exit codes 0, 6 and 12 as success.
• The sweep threshold is grace + reconcile interval + 60 s, measured from a new ReleasedDate.
• New opt-in host setting: "Keep sessions alive during the grace period".
  • It is off by default and cannot be combined with the screen lock.
  • It is left out of documents sent to hosts while off, so hosts that have not been migrated keep converging.
• Checkout and scaling now require an unassigned host to have Username IS NULL. Add VM now sends blank fields as null, and migration 040 repairs existing rows that have blank or leftover usernames.

Scaling

• TriggerScalingLogic is rewritten:
  • A single active rule (lowest RuleID); creating a second rule returns 409.
  • MinVMs >= 1, including recovery from zero running hosts.
  • Released and cleanup-pending hosts count as in use.
  • Booting hosts count as capacity and are never stopped. Maintenance, assigned and pending hosts are left alone.
  • Runs are serialized with sp_getapplock, and the activity log records the actual counts and the reason for each decision.
• The API reads power state from Azure before deciding, using instance_view per VM.
• It then acts by the rule's stop mode: Power off (default) or Deallocate. The portal warns about deallocation's cost, start-latency and capacity risks and asks for confirmation.
• If Azure refuses an operation, the API restores the recorded state.

Performance and readiness

• gunicorn gthread workers (API 2x8, portal 2x4).
• A per-process SQL connection cap (DB_MAX_CONCURRENCY, default 6, answers 503 when saturated). New sqlDatabaseSkuName parameter, default Basic.
• The signing keys, Graph token and SSH key are cached. Every outbound call has a timeout, and the Graph group check only runs when no role already authorizes the call.
• Checkout provisions the user in one SSH session (create-user.sh --password-stdin) instead of 6-8. Hosts running the older script fall back to the old sequence.
• Apply Now and sweep cleanups run in parallel with deadlines.
• The task function probes every 2 minutes, in parallel, and only writes changes, through the new /network-status endpoint.
• UIDs come from a SQL sequence instead of MAX(uid)+1.

Rollout

• Post-provision applies the SQL scripts before building images, and restarts the apps in the order API, task, front end.
• SQL changes are additive (040-066) and stay compatible with the previous API build.

Found along the way

• When no host was free, CheckoutVm rolled back pymssql's own transaction. SQL Server raised error 266, so a full pool returned 500 instead of 409. This bug predates this PR and is fixed in 042.

Upgrade notes

• Scaling will start and stop VMs for the first time. Review the scaling rule first; idle hosts above the minimum will be powered off.
• Assign portal roles to administrators before upgrading, or deploy once with allowLegacyScopeAccess=true and turn it off afterwards.
• Run deploy/Migrate-ExistingEnvironment.ps1 so the Linux hosts get the new scripts. Hosts that are not migrated keep their previous behavior.
• The full procedure is in deploy/DEPLOYMENT.md, under "Upgrading To Role-Based Access And Working Scaling".

Testing

• Run locally, all passing:
  • API: 155 unit tests, plus 21 integration tests against SQL Server 2022 through the real pymssql driver.
  • SQL contract: 6 tests; every script is applied twice with READ_COMMITTED_SNAPSHOT on.
  • Host scripts: 4 of 4 test files, in an ubuntu:24.04 container.
  • Task function: 8. Portal BFF: 153. Vitest: 110. Typecheck and production build pass.
  • main.json was regenerated with Bicep 0.44.1.
• New CI jobs: SQL (with a SQL Server service container), host scripts, and the task function.
• Not yet validated:
  • a real Azure deployment;
  • real Entra tokens carrying the new roles;
  • running the PowerShell deploy scripts;
  • keep-sessions-alive and create-user.sh on a real RHEL 9 host with GNOME and xrdp. docs/ROADMAP.md item 3.4 has a checklist.

Review focus

• sql_queries/059 (scaling), 043/044/045 (claiming and completing cleanup), and transaction handling under pymssql.
• api/app.py: token_required and the role mapping against the real callers, cleanup_remote_user, and trigger_scaling_logic.
• linux_host/create-user.sh and manage-lease.sh, which run as root through sudo.