Skip to content

feat/labels: Apply sourcegraph.labels to all objects in all four charts - #1023

Draft
marcleblanc2 wants to merge 18 commits into
marc/frontend-internal-service-labelsfrom
marc/labels-every-object
Draft

marcleblanc2 wants to merge 18 commits into
marc/frontend-internal-service-labelsfrom
marc/labels-every-object

Conversation

@marcleblanc2

@marcleblanc2 marcleblanc2 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Draft until

  1. It has been on a cluster: nothing here has since the split. Older heads of feat/labels: Add labels to PVCs, ServiceAccounts, and RBAC objects #1008 were upgraded on EKS with 0 objects recreated
  2. Decided: the collector's new --config argument replaces service.telemetry.metrics.readers in a custom command's config file. Should the readers be a Helm value instead?
  3. Decided: a label key set in both sourcegraph.labels and a per-component value renders twice, as it already does on main for workloads. Helm accepts it, kubeconform -strict does not

Problem

Solution

  • All four charts render sourcegraph.labels, the standard Helm labels, deploy, and app.kubernetes.io/component on every object
  • <serviceName>.labels and executor.labels cover more objects. New: storageClass.labels, priorityClasses[].labels, privateDockerRegistry.serviceLabels
  • Not labelled, on purpose: PVCs that Kubernetes creates from a StatefulSet's volume claim template still get only <serviceName>.storageLabels. Kubernetes rejects changes to those templates, so adding labels would fail the upgrade of existing releases
  • The per-pod scratch PVCs of ephemeral volumes (preciseCodeIntel and syntacticCodeIntel with storageType: pvc, dind executor.storage.type: ephemeral) get the same labels as the standalone PVCs, including helm.sh/chart and app.kubernetes.io/version
  • The dind executor chart's two PVCs take their own labels: new executor.storage.labels for the scratch PVC, and new privateDockerRegistry.storageLabels for the registry's volume claim template. Both default to empty. Its README has a new "PersistentVolumeClaim labels" section
  • The README's PersistentVolumeClaim labels table, added in feat/labels: Add labels to PVCs, ServiceAccounts, and RBAC objects #1008, lists which values label each kind of PVC
  • No _helpers.tpl gains a line: every helper this PR adds or changes is in a one-topic file under _helpers/. The dind chart's executor.deployment moved whole to _helpers/_executorDeployment.tpl, so its diff shows as a new file; its real changes are the pod labels and the scratch PVC's labels
  • A Prometheus labeldrop rule keeps the new labels off metrics
  • prometheus restarts when its config changes, and otel-collector does when openTelemetry.gateway.config.traces.exporters is set
  • Fixed along the way:
    • redisStore.serviceAccount.annotations was dropped
    • The OpenTelemetry Collector "pods available" panel and alert had no data
    • A user's own app.kubernetes.io/instance pod label left the Jaeger Services with no endpoints

Upgrade impact

  • Every pod restarts once. Nothing is recreated, and no selector changes
  • Later chart-only upgrades restart no pods: pods don't get helm.sh/chart or app.kubernetes.io/version. The exception is a pod with a scratch PVC (the settings above): its PVC labels are in the pod template and include the chart version, so it restarts on every chart version change
  • With prometheus.existingConfig, add the labeldrop rule yourself
  • A custom openTelemetry.gateway.command must pass the chart's last argument on; values.yaml says how

Tested

Amp threads

Changelog

  • sourcegraph.labels, the standard Helm labels, deploy, and app.kubernetes.io/component are now on every object in all four charts
  • New: storageClass.labels, priorityClasses[].labels, privateDockerRegistry.serviceLabels, and in the dind chart executor.storage.labels and privateDockerRegistry.storageLabels
  • With prometheus.existingConfig, add the new labeldrop rule to keep the labels off metrics
  • Fixed redisStore.serviceAccount.annotations, and the OpenTelemetry Collector "pods available" alert

@github-actions

Copy link
Copy Markdown

marcleblanc2 and others added 17 commits November 18, 2026 14:31
Services, ConfigMaps, Secrets, the Ingress, PodDisruptionBudgets,
PriorityClasses, and extraResources of the sourcegraph chart did not render
sourcegraph.labels. Neither did any object of the sourcegraph-executor-k8s
chart, nor the docker-config ConfigMap and private-docker-registry Service of
the sourcegraph-executor-dind chart.

Prometheus copies Service labels to the metrics it scrapes, so its bundled
configuration now drops the chart's standard labels: helm.sh/chart and
app.kubernetes.io/version change on every upgrade.

Each chart gets a test that renders every template and checks the labels, so
a template added later is covered too.

Amp-Thread-ID: https://ampcode.com/threads/T-c7c725c9-ed68-48e2-9f2d-0b8c31e76050
Co-authored-by: Amp <amp@ampcode.com>
The redis-store ServiceAccount was the only one that did not include the
annotations helper, so redisStore.serviceAccount.annotations was silently
dropped.

Amp-Thread-ID: https://ampcode.com/threads/T-c7c725c9-ed68-48e2-9f2d-0b8c31e76050
Co-authored-by: Amp <amp@ampcode.com>
PodDisruptionBudgets, PriorityClasses, the StorageClass, the test Pod, and
the k8s executor Secret now have deploy and app.kubernetes.io/component
labels, and the jaeger objects have app.kubernetes.io/instance.
<serviceName>.labels and executor.labels are also rendered on a component's
ConfigMaps, Secrets, and PodDisruptionBudget. Adds storageClass.labels,
priorityClasses[].labels, and privateDockerRegistry.serviceLabels (dind).

Amp-Thread-ID: https://ampcode.com/threads/T-c7c725c9-ed68-48e2-9f2d-0b8c31e76050
Co-authored-by: Amp <amp@ampcode.com>
…ice with its app

The bundled Prometheus copies Service labels to the metrics it scrapes. Now
that every Service has sourcegraph.labels, the labeldrop rule also drops
those keys, so the labels of Service-scraped metrics stay as they are on main.

The rule no longer drops app.kubernetes.io/managed-by: Helm adds that label
to every object it installs, so the metrics already had it.

The otel-collector Service had no app label, unlike every other scraped
Service, so its metrics had no app and job labels and the dashboard's
up{app=~".*otel-collector"} panel and alert had no data.

Amp-Thread-ID: https://ampcode.com/threads/T-c7c725c9-ed68-48e2-9f2d-0b8c31e76050
Co-authored-by: Amp <amp@ampcode.com>
The image's logging.yaml, and the ConfigMap the chart renders for custom
exporters, left the collector's metrics on 127.0.0.1:8888, where Prometheus
cannot scrape them. With the app label the Service gained, up was 0 and the
pods available alert fired instead of showing no data.

Pass the telemetry readers setting as a second --config to the default
command, and add it to the rendered ConfigMap. jaeger.yaml already has it.

Amp-Thread-ID: https://ampcode.com/threads/T-c7c725c9-ed68-48e2-9f2d-0b8c31e76050
Co-authored-by: Amp <amp@ampcode.com>
…rics to be scraped

openTelemetry.gateway.command replaces the default command, so it does not get
the argument that serves the collector's metrics on all interfaces.

Amp-Thread-ID: https://ampcode.com/threads/T-c7c725c9-ed68-48e2-9f2d-0b8c31e76050
Co-authored-by: Amp <amp@ampcode.com>
Pod templates now get `sourcegraph.labels`, the chart's standard labels,
and `app.kubernetes.io/component`, as every other object does. One helper,
`sourcegraph.podLabels`, merges them with `sourcegraph.podLabels`, the
component's `podLabels`, and the selector labels, so each key is rendered
once and a selector label always wins.

The Prometheus `labeldrop` rule now also drops
`app.kubernetes.io/managed-by` and also applies to the `kubernetes-pods`
job, so Helm and `helm template | kubectl apply` installs get the same
metric labels. No bundled dashboard or alert queries the dropped labels.

The changelog no longer asks for a manual Prometheus restart: the
`checksum/prometheus-config` annotation restarts it.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1cc03-93a7-7445-8f1f-3dd7123272bf
Co-authored-by: Amp <amp@ampcode.com>
The setting that serves the collector's own metrics on all interfaces was
part of the default command, so a command set in
`openTelemetry.gateway.command` did not get it, and Prometheus could not
scrape the collector. It is now always the last container argument, after
`openTelemetry.gateway.args`, so a replacement command gets it too. Last,
so that it is `$0` of a `sh -c` command whose script is the first argument.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1cc03-93a7-7445-8f1f-3dd7123272bf
Co-authored-by: Amp <amp@ampcode.com>
The collector does not reload a changed ConfigMap by itself, so a change to openTelemetry.gateway.config.traces took effect only at the pod's next restart. The pod template now has a checksum/otel-collector-config annotation over those values, which are all the ConfigMap's data is rendered from. It is set only when the chart renders the ConfigMap, so the default install does not change.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1cc03-93a7-7445-8f1f-3dd7123272bf
Co-authored-by: Amp <amp@ampcode.com>
helm.sh/chart and app.kubernetes.io/version change with every chart version. As pod labels they changed every pod template, so every pod restarted whenever the chart version changed, even when nothing else did. On main no pod template depends on the chart version.

Pod templates in all four charts now get the common labels without these two. The other objects keep them.

This also updates the pod label snapshots of the selector labels golden test: every pod gains app.kubernetes.io/managed-by and app.kubernetes.io/component, and no selector, volume claim template, or existing pod label changes.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1cc03-93a7-7445-8f1f-3dd7123272bf
Co-authored-by: Amp <amp@ampcode.com>
Each chart's label defines, with their comments, move unchanged out of
_helpers.tpl, next to the other one-topic helper files. Renders are
byte-identical.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1d360-e0b4-71d1-8ded-2e4851b93a8d
Co-authored-by: Amp <amp@ampcode.com>
The jaeger-collector and jaeger-query Services select on
app.kubernetes.io/instance: <release name>. main pins that label on the
jaeger pod. Once the pod also renders sourcegraph.podLabels and
jaeger.podLabels, a value for that label in either replaced the release
name, and both Services lost their endpoints.

Merge the release name last, and test that the pod keeps every label the
Services select on.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1902b-ece9-725c-863d-3e2bba35ee1d
…gument on

With the script as the first of openTelemetry.gateway.args after
command: ["/bin/sh", "-c"], the chart's argument is $0 and "$@" is empty,
so a wrapper that passes only "$@" drops it and the collector keeps serving
its metrics on 127.0.0.1. Document "$0" "$@" for that form and "$@" for a
script in command followed by a placeholder, and use the first in the test.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1902b-ece9-725c-863d-3e2bba35ee1d
executor.storage.labels labels the scratch PersistentVolumeClaim that is
created when executor.storage.type is ephemeral, as storageLabels does for
the PersistentVolumeClaims of the sourcegraph chart. It defaults to empty,
so existing releases render the same pods.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1902b-ece9-725c-863d-3e2bba35ee1d
…VC labels

privateDockerRegistry.storageLabels adds labels to the cache volume claim
template of the private-docker-registry StatefulSet, the last
PersistentVolumeClaim in the four charts that had no label value. It
defaults to empty, so existing releases render the same template.

The README now says which values label each PersistentVolumeClaim of the
chart, and that the registry's cannot change on an installed release.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1902b-ece9-725c-863d-3e2bba35ee1d
The scratch PersistentVolumeClaim of preciseCodeIntel and syntacticCodeIntel
(storageType: pvc) and of the dind executor (executor.storage.type:
ephemeral) now gets sourcegraph.labels, the standard Helm labels, deploy, and
app.kubernetes.io/component, as the standalone PersistentVolumeClaims do, in
addition to its own storageLabels or executor.storage.labels.

These labels are part of the pod template and include the chart version, so
with those settings the pods restart whenever the chart version changes.
Default renders are unchanged: the default volume is an emptyDir.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1902b-ece9-725c-863d-3e2bba35ee1d
@marcleblanc2
marcleblanc2 force-pushed the marc/labels-every-object branch from 6cda7a2 to 591d864 Compare November 18, 2026 20:32
sourcegraph.otelCollector.metricsReaders moves to
_helpers/_otelCollector.tpl, and the dind chart's executor.deployment moves
to _helpers/_executorDeployment.tpl, each with its comment and otherwise
unchanged. No _helpers.tpl gains a line on this branch any more: every
helper it adds or changes is in a one-topic file under _helpers/. Renders
are byte-identical.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1902b-ece9-725c-863d-3e2bba35ee1d

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant