Skip to content

fix(metrics): use monotonic System.nanoTime and ignore non-positive durations (fixes #3480) - #4100

Open
D-H-O-R-A wants to merge 1 commit into
wavesplatform:version-1.6.xfrom
D-H-O-R-A:fix/kamon-timer-monotonic-ntp-skew
Open

D-H-O-R-A wants to merge 1 commit into
wavesplatform:version-1.6.xfrom
D-H-O-R-A:fix/kamon-timer-monotonic-ntp-skew

Conversation

@D-H-O-R-A

Copy link
Copy Markdown

Summary of Changes

Fixes #3480

Problem Description

As reported in #3480, nodes frequently log warnings from kamon.metric.Timer during block application or mining:

WARN  [ecution-context-global-XX] kamon.metric.Timer - Failed to record value [-70000000] on [span.processing-time,{operation=block-appender,error=false}] because the value is outside of the configured range. The recorded value was adjusted to the highest trackable value [3600000000000].

Root Cause

  1. In com.wavesplatform.metrics.TimerExt, startedTimer.stop() uses Kamon's clock, which relies on the system wall-clock time. When NTP synchronizes or shifts the system clock backwards during block execution, the calculated duration is negative.
  2. Kamon's histogram rejects negative values with an out-of-range warning and falls back to recording 3,600,000,000,000ns.
  3. Furthermore, timer allocations and measurements executed unconditionally even when metrics were disabled in user configuration.

Solution

  1. Monotonic Clock: Use System.nanoTime() (which uses CLOCK_MONOTONIC on Linux), guaranteeing that time never moves backwards even during NTP time synchronization.
  2. Non-positive duration guard: Ensure that only strictly positive durations (elapsed > 0) are recorded into Kamon metrics.
  3. Bypass on disabled metrics: Check Kamon.enabled() before measuring to execute f directly with zero allocations and zero CPU overhead when metrics are disabled.
  4. Unit Tests: Added TimerExtSpec verifying measure, measureSuccessful, and measureOptional behavior.

…urations (fixes wavesplatform#3480)

- Use monotonic System.nanoTime() to avoid clock skew and backwards time adjustments during NTP synchronization
- Skip metric recording when elapsed time is non-positive, preventing Kamon out-of-range warnings
- Bypass measurement when Kamon is disabled (Kamon.enabled() == false) to eliminate unnecessary CPU and object allocation overhead
- Add comprehensive unit tests in TimerExtSpec
@D-H-O-R-A
D-H-O-R-A requested a review from phearnot as a code owner October 23, 2026 02:25

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kamon.metric.Timer warns about negative values due to NTP time sync

1 participant