Skip to content

Default report frequencies and reject values below 100ms - #466

Merged
driv3r merged 5 commits into
mainfrom
fix-report-frequency-default
Nov 4, 2026
Merged

driv3r merged 5 commits into
mainfrom
fix-report-frequency-default

Conversation

@driv3r

@driv3r driv3r commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #214. Supersedes #215.

Problem

ProgressReportFrequency and StateReportFrequency previously had no defaults or validation. With a callback URI configured, zero or negative intervals made time.After fire immediately, repeatedly serializing and posting progress/state. The original copydb smoke observed 1,053 progress POSTs in 1.5 seconds when the frequency was omitted.

Policy

For each reporting frequency:

Configuration Result
Omitted or 0 Use the 60000 ms default
Nonzero value below 100 ms, callback URI set Return a validation error
100 ms or greater Preserve the configured value
Callback URI empty Reporting disabled; the minimum is not enforced

Zero is an unset/default sentinel, not a way to disable reporting. This follows the convention used by other optional numeric configuration fields. Explicit invalid nonzero values are rejected rather than silently clamped.

Library-path protection

The policy lives in one private Config.validateReportFrequencies helper, called by both Config.ValidateConfig and Ferry.Initialize.

Initialize invokes it after establishing the default error handler but before opening database connections. This protects the normal library Initialize → Start → Run lifecycle even when callers do not invoke Config.ValidateConfig, without duplicating policy in the reporting loops or revalidating unrelated settings inside Run.

Review updates

  • Share the reporting checks with library initialization.
  • Explicitly document zero/default semantics and URI-based disabling.
  • Move the changelog entry to Changed because rejecting previously accepted nonzero values is a compatibility change.
  • Replace the earlier frequency tests with a table-driven matrix for both fields: -1, 0, 99, 100, 500, and an unset callback URI. Tests assert acceptance/rejection and normalized or preserved values, not exact error wording.
  • Add initialization regressions proving zero intervals are defaulted and invalid intervals fail before source/target connections are opened while leaving the error handler available.

The library-path regressions failed before this correction: initialization retained zero frequencies and accepted 99 ms. They pass with the shared helper.

Verification

  • go test -count=1 -p=1 . ./test/go ./copydb/test ./sharding/test: all four suites passed.
  • Ruby test/integration/callbacks_test.rb: four tests, zero failures.
  • A standalone live library migration deliberately omitted Config.ValidateConfig. Initialize set both intervals to 60000 ms; the run copied three rows and emitted two progress POSTs and no state POSTs instead of flooding either callback.
  • The same smoke program confirmed that 99 ms is rejected during initialization, with neither database connection opened and the error handler initialized.

Updated the branch with current main and retained all independent changelog entries when resolving the merge conflict.

CI follow-up: interrupt/resume shutdown

The failed Go 8.0/GTID job stopped before testing because pulling the MySQL image returned HTTP 502. The two Ruby 8.4 jobs instead timed out in the interrupt/resume suite.

Two test callbacks paused the data iterator with an unbounded sleep. After the Go subprocess exited, WEBrick waited forever for those Ruby request handlers, leaving the test harness stuck in kill while joining its server thread. A focused real-process/HTTP probe reproduced that exact shutdown state.

The callbacks now wait for the subprocess thread to finish, so they release when the child exits. A queue barrier ensures both affected regressions actually enter the paused-copy path before sending TERM; no data/verification assertions were removed, and no retries or longer job timeouts were added.

Post-fix verification used isolated Percona 8.4.4 servers with the CI server flags:

Configuration CI seed Full Ruby suite
zerolog / file_position 1695 76 tests, 0 failures
logrus / gtid 1881 76 tests, 0 failures

Both were single-process bundle exec rake test runs. The two forced-pause regressions also passed individually in both modes. The standalone shutdown probe changed from child_alive=false, shutdown_finished=false to child_alive=false, shutdown_finished=true.

Temporary database-port overrides used for isolated verification were restored before committing.

With a progress or state callback configured and the frequency left at
0 (or set negative), the reporting loop waits on time.After(0) and posts
to the callback as fast as it can.
ProgressReportFrequency and StateReportFrequency now default to 60000ms
like other optional settings, and ValidateConfig rejects values below
100ms when the matching callback is configured.

Fixes #214.
@driv3r
driv3r requested review from grodowski and mtaner October 19, 2026 16:44

@shivnagarajan shivnagarajan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few comments. Nothing super critical, but just some minor adjustments suggested.

Comment thread config.go Resolved
Comment thread CHANGELOG.md Resolved
Comment thread config.go
if c.ProgressReportFrequency == 0 {
c.ProgressReportFrequency = defaultReportFrequencyMs
}
if c.ProgressCallback.URI != "" && c.ProgressReportFrequency < minReportFrequencyMs {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is a bit strange that 0 is handled differently from values < minReportFrequencyMs (100)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept 0 as “unset/use the 60-second default,” consistent with optional Go integer fields so it wouldn't break existing setups, but apply new default

Comment thread test/go/config_test.go Resolved
@driv3r
driv3r merged commit 933a266 into main Nov 4, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Setting Config.ProgressReportFrequency to a small or negative number would be bad

2 participants