Repository navigation
Mirror Linux mdsd errors and warnings to container stdout - #1830
Bilen Ahmet (PelagicGames) wants to merge 9 commits into
Conversation
Preserve diagnostic files for onboarding checks and follow log rotation in the main container and active Prometheus sidecar. Retain direct mdsd stderr and cover forwarding with shell regression tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
Default diagnostic file forwarding to disabled and parse a Linux-only boolean setting from the existing agent ConfigMap. Preserve original stderr behavior, document enablement, and cover configuration defaults and parser-to-shell behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
Add the default-off amalogs.enableMdsdLogForwarding Helm value to both chart sources and pass it to every Linux Arc agent role. Preserve the extension setting unless an explicit ConfigMap boolean overrides it, document publication requirements, and cover chart rendering and configuration precedence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
Use a single enablement variable and export only when enabled, matching custom metrics. Normalize ConfigMap values like process metrics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Chris Wedgwood (cwedgwood)
left a comment
There was a problem hiding this comment.
Requesting changes: with forwarding enabled the liveness probe can no longer detect a dead mdsd (inline).
| return 1 | ||
| fi | ||
| # Read from the beginning to include startup failures, and follow names across log rotation. | ||
| tail -q -n +1 -F "${MDSD_LOG}/mdsd.err" "${MDSD_LOG}/mdsd.warn" & |
There was a problem hiding this comment.
This tail defeats the liveness probe. build/linux/installer/scripts/livenessprobe.sh:58 treats any process matching grep "mdsd" as mdsd, and this command line contains .../log/mdsd.err. With forwarding enabled, a dead mdsd no longer fails the probe in ama-logs or ama-logs-prometheus. Match the process itself (e.g. pgrep -x mdsd) or keep mdsd out of the follower's command line, and add a probe test with forwarding enabled and mdsd stopped.
Also, nothing restarts the follower if it exits; forwarding then stops silently.
There was a problem hiding this comment.
Addressed by replacing the standalone tail with the agent's existing Fluent Bit machinery (f381c5d; scope follow-up 357e539). An opt-in diagnostics include tails only mdsd.err/mdsd.warn, records offsets in a dedicated DB, and emits JSON stdout under a separate tag. Both the extension and ConfigMap settings remain available and disabled by default.
The original liveness check is now unchanged. With no separate tail ...mdsd.err ...mdsd.warn process, forwarding no longer introduces the misleading command line that caused the false positive. Fluent Bit still runs with its ordinary config-file argument, not the diagnostic filenames. We therefore do not need a probe change to address the regression introduced by the earlier implementation.
There is also no independent follower process to die silently: file following and offset recovery belong to the already-running Fluent Bit process and its existing lifecycle. No custom supervisor or new follower-specific liveness condition is introduced.
Validation was rerun against the unchanged production probe in an isolated agent container: main, Prometheus-sidecar and ReplicaSet probes fail after mdsd stops while Fluent Bit continues forwarding diagnostics; the muted-sidecar exemption remains unchanged. Full repository validation passed (28 tests / 611 assertions), including both configuration paths, full role-config dry runs, error/warning-only output with info/QoS exclusion, routing isolation, rotation/truncation, and position-DB recovery after killing/restarting Fluent Bit without full-file replay. Recovery retains Fluent Bit's existing best-effort delivery semantics, not an exactly-once guarantee. Live Kubernetes/end-to-end ingestion validation remains outstanding.
Replace the standalone tail follower with an opt-in diagnostics input, dedicated offset database and stdout route in the existing Fluent Bit process. Match the mdsd process exactly in liveness checks. Cover settings, role configs, rotation, crash recovery, output isolation and actual probe behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
The standalone tail process is gone, so forwarding no longer adds mdsd filenames to a separate process command line. Keep the original probe and validate that it still detects daemon failure while Fluent Bit forwards diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
Summary
Supersedes #1803, which was closed for inactivity. Rebased onto the latest
ci_prod(the repository's default branch), resolving the config-parser conflict without undoing upstream shell escaping.amalogs.enableMdsdLogForwarding=trueon the Arc extension, or[agent_settings.mdsd_log_forwarding] enabled = truein the agent ConfigMap. Disabled by default.mdsd.errandmdsd.warn, records positions in a separate database, and routes a dedicated tag to JSON stdout. No standalone follower or custom restart/checkpoint implementation. Existing telemetry filters and routes are left unchanged.Validation
git diff --checkpassed.Release considerations
Both the updated chart and agent image must ship in an extension release. Maintainers should confirm whether publisher-side configuration-key validation outside this repository needs updating. Real onboarding-failure and downstream log-ingestion validation remain outstanding.
Forwarding starts with the existing Fluent Bit process and emits JSON records. Recovery uses its existing file-position/buffering semantics, not an exactly-once stdout-delivery guarantee.