Repository navigation
Mirror Linux mdsd errors and warnings to container stdout - #1803
Bilen Ahmet (PelagicGames) wants to merge 6 commits into
Conversation
Preserve diagnostic files for onboarding checks and follow log rotation in the main container and active Prometheus sidecar. Retain direct mdsd stderr and cover forwarding with shell regression tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Confirmed the log forwarding works with some manually generated test warning and error logs, for both ama-logs and ama-log-prometheus, by loading a locally-built image. |
Default diagnostic file forwarding to disabled and parse a Linux-only boolean setting from the existing agent ConfigMap. Preserve original stderr behavior, document enablement, and cover configuration defaults and parser-to-shell behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
Add the default-off amalogs.enableMdsdLogForwarding Helm value to both chart sources and pass it to every Linux Arc agent role. Preserve the extension setting unless an explicit ConfigMap boolean overrides it, document publication requirements, and cover chart rendering and configuration precedence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
Use a single enablement variable and export only when enabled, matching custom metrics. Normalize ConfigMap values like process metrics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
|
Updated in 5f5bfcd to match the existing config model: removed |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75cea555-9326-4853-af30-503e94c302b0
|
This PR is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
|
I can't remove the label manually, so commenting. This PR is still desired, but I am awaiting another round of review. This review is on hold while critical security issues are being addressed |
|
This PR is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
|
This PR was closed because it has been stalled for 12 days with no activity. |
|
Superseded by #1830, rebased onto the latest |
Motivation
When agent onboarding or configuration retrieval fails, recurring mdsd diagnostics are written to files inside the container rather than its stdout/stderr stream. A pod can remain running while
kubectl logsand independent node-level container log collectors have no visibility into those failures.Changes
amalogs.enableMdsdLogForwarding=true(defaultfalse), wired through both the legacy and unified Arc chart sources to the Linux DaemonSet main container, Prometheus sidecar, and ReplicaSet main container.[agent_settings.mdsd_log_forwarding] enabled = truein the existingcontainer-azm-ms-agentconfigConfigMap (agent-settingsdata). Like custom metrics, use one enablement variable and export only when enabled: ConfigMapfalseor an absent setting leaves the extension value unchanged. Values are normalized with.to_s.strip.downcase == "true", matching process metrics. To disable forwarding, disable both the extension and ConfigMap settings. Windows and the unified chart's AKS mode are unchanged.mdsd.errandmdsd.warnto stdout using the BusyBox-compatibletail -q -n +1 -Falready available in the Linux image.Informational and QoS log files remain file-only. Readiness, liveness, authentication, and configuration retrieval behavior are unchanged.
Validation
Configuration consistency update: parser-to-shell tests passed (10 tests, 319 assertions), covering single-variable enable-only behavior, boolean and case/whitespace-normalized string values, extension interaction, defaults, invalid config, unsupported schema, and Windows behavior. Helm tests passed (5 tests, 155 assertions), covering both charts and safe YAML deserialization. Bash and ciprod BusyBox forwarding tests also passed.
Opt-in update: Bash unit tests passed; parser-to-production-shell tests passed (5 tests, 120 assertions), covering missing/false/invalid values, malformed TOML, unsupported schema, Windows behavior, and explicit enablement. The 26-assertion forwarding suite also passed with the actual ciprod BusyBox.
Bash syntax checks and
git diff --checkpassed.Existing Bash unit-test runner passed, including the new forwarding suite.
Forwarding tests passed with BusyBox and GNU tail, covering startup and ongoing diagnostics, retained files, missing warning-file creation, copytruncate, file replacement, duplicate/header prevention, excluded informational logs, and surfaced preparation failures.
Built a local linux/amd64 derivative of published
ciprod:3.6.0, applying only this patch to the released startup script. The forwarding tests passed inside that image using its bundled BusyBox and patched entrypoint.Remaining validation / limitations
Extension configuration support requires publishing both the updated chart and agent image. Any configuration-key validation or publisher metadata maintained outside this repository must allow
amalogs.enableMdsdLogForwarding; maintainer confirmation is requested.Synthetic warning and error forwarding was verified in both agent containers on a simulator before the opt-in setting was added. End-to-end validation of a real onboarding failure and independent container-log ingestion remains outstanding; the simulator is no longer available.
This surfaces diagnostics but does not resolve onboarding failures or change the healthy/ready status of an agent that has never obtained configuration.
File forwarding follows the existing polling/copytruncate behavior; it does not introduce a lossless delivery guarantee around concurrent truncation.