Skip to content

Add configured and live audit instance counts to the usage report - #5938

Merged
johnsimons merged 2 commits into
masterfrom
john/audit_live_instances
Sep 29, 2026
Merged

johnsimons merged 2 commits into
masterfrom
john/audit_live_instances

Conversation

@johnsimons

Copy link
Copy Markdown
Member

This issue was discovered as part of https://github.com/Particular/CloudPlatformExperience/pull/962#issuecomment-5826623019

Adds two counts to the environment data in the usage report. Audit.ConfiguredInstances is the number of audit instances the primary instance is configured with. Audit.LiveInstances is how many of those responded as a ServiceControl audit instance at the last collection.

The version and transport summaries in AuditServicesData count every entry in the primary's ServiceControl/RemoteInstances setting, whether or not the remote responds. Unreachable remotes are counted under the version Unknown and an empty transport name. Deleting an audit instance does not remove it from the primary's list, so stale entries build up over time. Some reports show more configured audit instances than seem plausible. Nothing in the report separates stale entries from live ones.

A remote counts as live when the configuration it returns contains transport.audit_queue and its X-Particular-Version header parses as a version. Only audit instances return an audit queue, so a primary instance listed as a remote does not count.

The daily audit collection stores both counts in AuditServiceMetadata. The report writes them to EnvironmentData so that the AuditServicesData type from the Particular.LicensingComponent.Report package does not change.

Both counts are nullable. Metadata stored by an earlier version has neither count, so the report leaves both keys out until the first collection run, about 40 seconds after startup. Reports from earlier versions lack the keys too, so a missing key means the count was not reported. An earlier version reading the new metadata ignores the extra properties. Tests cover reading metadata in the old shape from RavenDB, SQL Server and PostgreSQL.

This PR also fixes a bug in AuditQuery.GetAuditRemotes. The queue name list was created once, before the loop, so every remote got the same list holding the queues of all remotes. The only code that read the list checks whether a queue name is in it, so the bug had no visible effect. Each remote now gets its own list.

Adds ConfiguredInstances and LiveInstances to AuditServiceMetadata so that throughput reports can include how many audit instances are configured versus how many are actively responding. A remote is considered a live audit instance only if its configuration includes an audit queue and it has a parseable semantic version. Compatibility tests confirm that existing persisted metadata without these fields reads back with null counts.
…ed list

Moving the queue list initialization inside the loop ensures each remote tracks only its own audit and audit log queues, rather than inheriting queues discovered from previous remotes. A new test confirms that two remotes each report only their own queues.

The fake configuration API is also generalized to accept arbitrary remote configurations via a params constructor, replacing the single-purpose named class.
@johnsimons johnsimons self-assigned this Sep 29, 2026
{
List<string> queues = [];

foreach (var remote in remotes)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parse each remote in its own try catch, so a single bad remote doesnt discard the entire list.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at this @warwickschroeder, and as far as I can tell, the code is already quite defensive, everywhere we are using try..., so as far as I can see, there is no reason for a try-catch here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern is that remote.Configuration.AsObject() would throw if any remote JSON cant be parsed, and the catch is outside of the foreach.

Comment on lines +5 to +6
public int? ConfiguredInstances { get; init; }
public int? LiveInstances { get; init; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some xml comments here would be handy explaining what these capture, and what null means.

@johnsimons
johnsimons merged commit 35edf27 into master Sep 29, 2026
36 checks passed
@johnsimons
johnsimons deleted the john/audit_live_instances branch September 29, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants