diff --git a/charts/sourcegraph-executor/CHANGELOG.md b/charts/sourcegraph-executor/CHANGELOG.md index 5e2f4fb18..18d6f5e5a 100644 --- a/charts/sourcegraph-executor/CHANGELOG.md +++ b/charts/sourcegraph-executor/CHANGELOG.md @@ -3,6 +3,11 @@ ## Unreleased +- Added `executor.storage.labels` to the `sourcegraph-executor-dind` chart to label the per-pod PersistentVolumeClaim that is created when `executor.storage.type` is `ephemeral`. That claim now also gets `sourcegraph.labels`, the chart's standard labels, `deploy`, and `app.kubernetes.io/component`, so with `executor.storage.type: ephemeral` the executor pods restart whenever the chart version changes. Changing `executor.storage.labels` restarts them too. +- Added `privateDockerRegistry.storageLabels` to the `sourcegraph-executor-dind` chart to label the `cache` volume claim template of the `private-docker-registry` StatefulSet. It defaults to empty, so existing releases render the same template. Set it on the first install and never change it on an installed release: Kubernetes does not allow changing a StatefulSet's volume claim templates. See [PersistentVolumeClaim labels](dind/README.md#persistentvolumeclaim-labels). +- Every object the charts create now has the `deploy` and `app.kubernetes.io/component` labels: the `sourcegraph-executor-k8s` Secret did not have them. `executor.labels` is now also rendered on the `sourcegraph-executor-k8s` ConfigMap, Secret, ServiceAccount, Role, and RoleBinding, and on the `sourcegraph-executor-dind` `docker-config` ConfigMap, not only on the Deployment. Added `privateDockerRegistry.serviceLabels` to the `sourcegraph-executor-dind` chart to label the `private-docker-registry` Service. +- `sourcegraph.labels` and the charts' standard labels (`helm.sh/chart`, `app.kubernetes.io/name`, `app.kubernetes.io/instance`, `app.kubernetes.io/version`, `app.kubernetes.io/managed-by`) are now rendered on every object the charts create. The `sourcegraph-executor-k8s` chart rendered them on none of its objects (ConfigMap, Deployment, Role, RoleBinding, Secret, Service, ServiceAccount), and the `sourcegraph-executor-dind` chart did not render them on the `docker-config` ConfigMap and the `private-docker-registry` Service. Pods get them too, except `helm.sh/chart` and `app.kubernetes.io/version`, which change with every chart version and would restart every pod on every upgrade. The `private-docker-registry` pod gains `app.kubernetes.io/component`. Existing releases gain these labels on upgrade. No object is recreated. Every pod restarts once, on this upgrade. + ## 8.0.0 * Added a `network-policy` example to the `sourcegraph-executor-k8s` and `sourcegraph-executor-dind` charts, with `CiliumNetworkPolicy` resources which limit Executor and job pods to the frontend API. diff --git a/charts/sourcegraph-executor/dind/README.md b/charts/sourcegraph-executor/dind/README.md index 14bff60a9..e1868867a 100644 --- a/charts/sourcegraph-executor/dind/README.md +++ b/charts/sourcegraph-executor/dind/README.md @@ -38,6 +38,21 @@ helm repo add sourcegraph https://helm.sourcegraph.com/release Manifests rendered using the `helm template` command can be used for direct deployment using `kubectl`. +## PersistentVolumeClaim labels + +Kubernetes creates both of this chart's PersistentVolumeClaims from a template, so Helm does not manage them: + +| PersistentVolumeClaim | Created from | Labels | +| --- | --- | --- | +| The registry's `cache` | The `private-docker-registry` StatefulSet's volume claim template | `deploy`, `app.kubernetes.io/component`, and `privateDockerRegistry.storageLabels` | +| The executor's scratch volume, when `executor.storage.type` is `ephemeral` | The executor pod's ephemeral volume | `sourcegraph.labels`, the chart's standard labels, `deploy`, `app.kubernetes.io/component`, and `executor.storage.labels` | + +The registry's claim gets neither `sourcegraph.labels` nor the chart's standard labels: a chart version that added them would fail the upgrade of existing releases. + +Set `privateDockerRegistry.storageLabels` on the first install and never change it on an installed release. Kubernetes rejects any change to a StatefulSet's volume claim templates, so the upgrade fails. Kubernetes also never updates the labels of a claim that exists: label an existing one with `kubectl label pvc`. + +The scratch volume's claim is deleted and created again with its pod, so it always has the labels the chart renders. Its labels are part of the pod template: changing one restarts the executor pods, and so does every change of the chart version, which is in `helm.sh/chart`. + ## Configuration Options Reference the table below for available configuration parameters and consult [executor] documentation. @@ -83,6 +98,7 @@ In addition to the documented values, the `executor` and `private-docker-registr | executor.replicaCount | int | `1` | | | executor.resources | object | `{}` | Resource requests and limits for the executor container. Each queue can override this with its own resources field. | | executor.storage.class | string | `""` | StorageClass for the ephemeral volume. Only used when type is ephemeral. Defaults to the cluster default StorageClass when empty. | +| executor.storage.labels | object | `{}` | Optional labels to add to the ephemeral volume's PVC, in addition to `sourcegraph.labels`. Only used when type is ephemeral. Changing them restarts the executor pods. See [PersistentVolumeClaim labels](#persistentvolumeclaim-labels) | | executor.storage.size | string | `""` | Size of the scratch volume. emptyDir: sets sizeLimit (optional, leave empty for unlimited). ephemeral: sets the PVC storage request (required). | | executor.storage.type | string | `"emptyDir"` | Type of scratch volume for job workspaces. One of: emptyDir, ephemeral. emptyDir: plain emptyDir, no storage class required. ephemeral: per-pod PVC via the cluster default storage class; size is required. | | privateDockerRegistry.enabled | bool | `true` | Whether to deploy the private registry. Only one registry is needed when deploying multiple executors. More information: https://sourcegraph.com/docs/self-hosted/executors/deploy-executors#using-private-registries | @@ -90,6 +106,8 @@ In addition to the documented values, the `executor` and `private-docker-registr | privateDockerRegistry.image.registry | string | `"index.docker.io"` | | | privateDockerRegistry.image.repository | string | `"registry"` | | | privateDockerRegistry.image.tag | int | `3` | | +| privateDockerRegistry.serviceLabels | object | `{}` | Add extra labels to the `private-docker-registry` Service | +| privateDockerRegistry.storageLabels | object | `{}` | Labels to add to the `cache` volume claim template of the `private-docker-registry` StatefulSet. Kubernetes does not allow changing them on an installed StatefulSet. See [PersistentVolumeClaim labels](#persistentvolumeclaim-labels) | | privateDockerRegistry.storageSize | string | `"10Gi"` | | | queues | list | `[]` | Optional list of queues to deploy as standalone Deployments. When set, the single executor Deployment is not rendered. Each entry supports: name (required) — used as the deployment name suffix (executor-) queueName — sets EXECUTOR_QUEUE_NAME; defaults to name if omitted queueNames — sets EXECUTOR_QUEUE_NAMES (comma-joined); takes precedence over queueName when set replicaCount, resources, env (merged with executor.env, queue overrides) | | sourcegraph.affinity | object | `{}` | Affinity, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) | diff --git a/charts/sourcegraph-executor/dind/README.md.gotmpl b/charts/sourcegraph-executor/dind/README.md.gotmpl index fbc7da966..bad9036a2 100644 --- a/charts/sourcegraph-executor/dind/README.md.gotmpl +++ b/charts/sourcegraph-executor/dind/README.md.gotmpl @@ -38,6 +38,21 @@ helm repo add sourcegraph https://helm.sourcegraph.com/release Manifests rendered using the `helm template` command can be used for direct deployment using `kubectl`. +## PersistentVolumeClaim labels + +Kubernetes creates both of this chart's PersistentVolumeClaims from a template, so Helm does not manage them: + +| PersistentVolumeClaim | Created from | Labels | +| --- | --- | --- | +| The registry's `cache` | The `private-docker-registry` StatefulSet's volume claim template | `deploy`, `app.kubernetes.io/component`, and `privateDockerRegistry.storageLabels` | +| The executor's scratch volume, when `executor.storage.type` is `ephemeral` | The executor pod's ephemeral volume | `sourcegraph.labels`, the chart's standard labels, `deploy`, `app.kubernetes.io/component`, and `executor.storage.labels` | + +The registry's claim gets neither `sourcegraph.labels` nor the chart's standard labels: a chart version that added them would fail the upgrade of existing releases. + +Set `privateDockerRegistry.storageLabels` on the first install and never change it on an installed release. Kubernetes rejects any change to a StatefulSet's volume claim templates, so the upgrade fails. Kubernetes also never updates the labels of a claim that exists: label an existing one with `kubectl label pvc`. + +The scratch volume's claim is deleted and created again with its pod, so it always has the labels the chart renders. Its labels are part of the pod template: changing one restarts the executor pods, and so does every change of the chart version, which is in `helm.sh/chart`. + ## Configuration Options Reference the table below for available configuration parameters and consult [executor] documentation. diff --git a/charts/sourcegraph-executor/dind/templates/_helpers.tpl b/charts/sourcegraph-executor/dind/templates/_helpers.tpl index 8b87b5d24..9bfb63fa4 100644 --- a/charts/sourcegraph-executor/dind/templates/_helpers.tpl +++ b/charts/sourcegraph-executor/dind/templates/_helpers.tpl @@ -12,29 +12,6 @@ Create chart name and version as used by the chart label. {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} {{- end }} -{{/* -Common labels -*/}} -{{- define "sourcegraph.labels" -}} -helm.sh/chart: {{ include "sourcegraph.chart" . }} -{{ include "sourcegraph.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- if .Values.sourcegraph.labels }} -{{ toYaml .Values.sourcegraph.labels }} -{{- end }} -{{- end }} - -{{/* -Selector labels -*/}} -{{- define "sourcegraph.selectorLabels" -}} -app.kubernetes.io/name: {{ include "sourcegraph.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end }} - {{/* Create the name of the service account to use */}} @@ -107,13 +84,6 @@ executor-{{join "-" .Values.executor.queueNames }} {{- end }} {{- end }} -{{- define "executor.labels" -}} -app: {{ include "executor.name" . }} -deploy: sourcegraph -sourcegraph-resource-requires: no-cluster-admin -app.kubernetes.io/component: executor -{{- end}} - {{- define "dind.daemonConfig" -}} {{- $config := .Values.dind.daemonConfig | deepCopy }} {{- if .Values.dind.gVisor.enabled }} @@ -122,243 +92,6 @@ app.kubernetes.io/component: executor {{- $config | toPrettyJson }} {{- end }} -{{/* -Render a single executor Deployment. -Usage: include "executor.deployment" (dict "root" $ "name" "executor-foo" "queueName" "foo" "queueNames" (list) "replicaCount" 1 "resources" $res "env" $env) -*/}} -{{- define "executor.deployment" -}} -{{- $r := .root -}} -apiVersion: apps/v1 -kind: Deployment -metadata: - name: {{ .name }} - annotations: - description: Runs sourcegraph executors - kubectl.kubernetes.io/default-container: executor - labels: - {{- include "sourcegraph.labels" $r | nindent 4 }} - {{- if $r.Values.executor.labels }} - {{- toYaml $r.Values.executor.labels | nindent 4 }} - {{- end }} - app: {{ .name }} - deploy: sourcegraph - sourcegraph-resource-requires: no-cluster-admin - app.kubernetes.io/component: executor -spec: - selector: - matchLabels: - {{- include "sourcegraph.selectorLabels" $r | nindent 6 }} - app: {{ .name }} - minReadySeconds: 10 - replicas: {{ .replicaCount }} - revisionHistoryLimit: 10 - strategy: - rollingUpdate: - maxSurge: 1 - maxUnavailable: 1 - type: RollingUpdate - template: - metadata: - annotations: - kubectl.kubernetes.io/default-container: executor - checksum/docker-config: {{ include "dind.daemonConfig" $r | sha256sum }} - {{- if $r.Values.sourcegraph.podAnnotations }} - {{- toYaml $r.Values.sourcegraph.podAnnotations | nindent 8 }} - {{- end }} - {{- if $r.Values.executor.podAnnotations }} - {{- toYaml $r.Values.executor.podAnnotations | nindent 8 }} - {{- end }} - labels: - {{- include "sourcegraph.selectorLabels" $r | nindent 8 }} - {{- if $r.Values.sourcegraph.podLabels }} - {{- toYaml $r.Values.sourcegraph.podLabels | nindent 8 }} - {{- end }} - {{- if $r.Values.executor.podLabels }} - {{- toYaml $r.Values.executor.podLabels | nindent 8 }} - {{- end }} - app: {{ .name }} - deploy: sourcegraph - sourcegraph-resource-requires: no-cluster-admin - app.kubernetes.io/component: executor - spec: - {{- with $r.Values.executor.podSecurityContext }} - securityContext: - {{- toYaml . | nindent 8 }} - {{- end }} - containers: - - name: executor - image: {{ include "sourcegraph.image" (list $r "executor") }} - imagePullPolicy: {{ $r.Values.sourcegraph.image.pullPolicy }} - livenessProbe: - httpGet: - path: /healthz - port: http-debug - scheme: HTTP - initialDelaySeconds: 60 - timeoutSeconds: 5 - readinessProbe: - httpGet: - path: /healthz - port: http-debug - scheme: HTTP - periodSeconds: 5 - timeoutSeconds: 5 - ports: - - name: http-debug - containerPort: 8080 - terminationMessagePolicy: FallbackToLogsOnError - env: - - name: EXECUTOR_FRONTEND_URL - value: {{ $r.Values.executor.frontendUrl | quote }} - - name: EXECUTOR_FRONTEND_PASSWORD - {{- if $r.Values.executor.frontendExistingSecret }} - valueFrom: - secretKeyRef: - name: {{ $r.Values.executor.frontendExistingSecret }} - key: EXECUTOR_FRONTEND_PASSWORD - {{- else }} - value: {{ $r.Values.executor.frontendPassword | quote }} - {{- end }} - {{- if .queueNames }} - - name: EXECUTOR_QUEUE_NAMES - value: {{ join "," .queueNames | quote }} - {{- else }} - - name: EXECUTOR_QUEUE_NAME - value: {{ .queueName | quote }} - {{- end }} - - name: SRC_LOG_LEVEL - value: {{ $r.Values.executor.log.level | quote }} - - name: SRC_LOG_FORMAT - value: {{ $r.Values.executor.log.format | quote }} - - name: EXECUTOR_MAXIMUM_RUNTIME_PER_JOB - value: {{ $r.Values.executor.maximumRuntimePerJob | quote }} - - name: EXECUTOR_USE_FIRECRACKER - value: "false" - - name: EXECUTOR_USE_KUBERNETES - value: "false" - - name: EXECUTOR_HEALTH_SERVER_ADDR - value: ":8080" - - name: EXECUTOR_JOB_NUM_CPUS - value: "0" - - name: EXECUTOR_JOB_MEMORY - value: "0" - - name: DOCKER_HOST - value: tcp://localhost:2375 - - name: TMPDIR - value: /scratch - {{- range $name, $item := .env }} - - name: {{ $name }} - {{- $item | toYaml | nindent 14 }} - {{- end }} - volumeMounts: - - mountPath: /scratch - name: executor-scratch - {{- with .resources }} - resources: - {{- toYaml . | nindent 12 }} - {{- end }} - - name: dind - image: "{{ $r.Values.dind.image.registry }}/{{ $r.Values.dind.image.repository }}:{{ $r.Values.dind.image.tag }}" - imagePullPolicy: {{ $r.Values.sourcegraph.image.pullPolicy }} - securityContext: - {{- if $r.Values.dind.gVisor.enabled }} - {{- toYaml $r.Values.dind.gVisor.securityContext | nindent 12 }} - {{- else }} - privileged: true - {{- end }} - command: - {{- if $r.Values.dind.gVisor.enabled }} - {{- toYaml $r.Values.dind.gVisor.command | nindent 12 }} - {{- else }} - {{- toYaml $r.Values.dind.command | nindent 12 }} - {{- end }} - livenessProbe: - exec: - command: - - wget - - -qO- - - http://127.0.0.1:2375/_ping - initialDelaySeconds: 15 - periodSeconds: 5 - failureThreshold: 5 - readinessProbe: - exec: - command: - - wget - - -qO- - - http://127.0.0.1:2375/_ping - initialDelaySeconds: 20 - periodSeconds: 5 - failureThreshold: 5 - env: - - name: POD_NAME - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.name - ports: - - containerPort: 2375 - protocol: TCP - volumeMounts: - - mountPath: /scratch - name: executor-scratch - - mountPath: /etc/docker/daemon.json - subPath: daemon.json - name: docker-config - {{- if $r.Values.dind.gVisor.enabled }} - - mountPath: /var/lib/docker - name: docker - {{- end }} - enableServiceLinks: false - {{- if $r.Values.dind.gVisor.enabled }} - runtimeClassName: gvisor - {{- end }} - {{- with $r.Values.sourcegraph.nodeSelector }} - nodeSelector: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with $r.Values.sourcegraph.affinity }} - affinity: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with include "sourcegraph.priorityClassName" (list $r "executor") | trim }}{{ . | nindent 6 }}{{- end }} - {{- with $r.Values.sourcegraph.tolerations }} - tolerations: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with $r.Values.sourcegraph.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - volumes: - - name: executor-scratch - {{- if eq $r.Values.executor.storage.type "ephemeral" }} - ephemeral: - volumeClaimTemplate: - spec: - accessModes: ["ReadWriteOnce"] - {{- if $r.Values.executor.storage.class }} - storageClassName: {{ $r.Values.executor.storage.class }} - {{- end }} - resources: - requests: - storage: {{ required "executor.storage.size is required when storage.type is ephemeral" $r.Values.executor.storage.size }} - {{- else }} - emptyDir: - {{- if $r.Values.executor.storage.size }} - sizeLimit: {{ $r.Values.executor.storage.size }} - {{- end }} - {{- end }} - - name: docker-config - configMap: - defaultMode: 420 - name: docker-config - {{- if $r.Values.dind.gVisor.enabled }} - - name: docker - emptyDir: {} - {{- end }} -{{- end }} - {{/* Validate that an env dict does not contain managed environment variable names. Usage: include "executor.validateEnv" (list $envDict "label") diff --git a/charts/sourcegraph-executor/dind/templates/_helpers/_executorDeployment.tpl b/charts/sourcegraph-executor/dind/templates/_helpers/_executorDeployment.tpl new file mode 100644 index 000000000..bfa5e358b --- /dev/null +++ b/charts/sourcegraph-executor/dind/templates/_helpers/_executorDeployment.tpl @@ -0,0 +1,238 @@ +{{/* +Render a single executor Deployment. +Usage: include "executor.deployment" (dict "root" $ "name" "executor-foo" "queueName" "foo" "queueNames" (list) "replicaCount" 1 "resources" $res "env" $env) +*/}} +{{- define "executor.deployment" -}} +{{- $r := .root -}} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ .name }} + annotations: + description: Runs sourcegraph executors + kubectl.kubernetes.io/default-container: executor + labels: + {{- include "sourcegraph.labels" $r | nindent 4 }} + {{- if $r.Values.executor.labels }} + {{- toYaml $r.Values.executor.labels | nindent 4 }} + {{- end }} + app: {{ .name }} + deploy: sourcegraph + sourcegraph-resource-requires: no-cluster-admin + app.kubernetes.io/component: executor +spec: + selector: + matchLabels: + {{- include "sourcegraph.selectorLabels" $r | nindent 6 }} + app: {{ .name }} + minReadySeconds: 10 + replicas: {{ .replicaCount }} + revisionHistoryLimit: 10 + strategy: + rollingUpdate: + maxSurge: 1 + maxUnavailable: 1 + type: RollingUpdate + template: + metadata: + annotations: + kubectl.kubernetes.io/default-container: executor + checksum/docker-config: {{ include "dind.daemonConfig" $r | sha256sum }} + {{- if $r.Values.sourcegraph.podAnnotations }} + {{- toYaml $r.Values.sourcegraph.podAnnotations | nindent 8 }} + {{- end }} + {{- if $r.Values.executor.podAnnotations }} + {{- toYaml $r.Values.executor.podAnnotations | nindent 8 }} + {{- end }} + labels: + {{- include "sourcegraph.podLabels" (list $r $r.Values.executor) | nindent 8 }} + app: {{ .name }} + deploy: sourcegraph + sourcegraph-resource-requires: no-cluster-admin + app.kubernetes.io/component: executor + spec: + {{- with $r.Values.executor.podSecurityContext }} + securityContext: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: executor + image: {{ include "sourcegraph.image" (list $r "executor") }} + imagePullPolicy: {{ $r.Values.sourcegraph.image.pullPolicy }} + livenessProbe: + httpGet: + path: /healthz + port: http-debug + scheme: HTTP + initialDelaySeconds: 60 + timeoutSeconds: 5 + readinessProbe: + httpGet: + path: /healthz + port: http-debug + scheme: HTTP + periodSeconds: 5 + timeoutSeconds: 5 + ports: + - name: http-debug + containerPort: 8080 + terminationMessagePolicy: FallbackToLogsOnError + env: + - name: EXECUTOR_FRONTEND_URL + value: {{ $r.Values.executor.frontendUrl | quote }} + - name: EXECUTOR_FRONTEND_PASSWORD + {{- if $r.Values.executor.frontendExistingSecret }} + valueFrom: + secretKeyRef: + name: {{ $r.Values.executor.frontendExistingSecret }} + key: EXECUTOR_FRONTEND_PASSWORD + {{- else }} + value: {{ $r.Values.executor.frontendPassword | quote }} + {{- end }} + {{- if .queueNames }} + - name: EXECUTOR_QUEUE_NAMES + value: {{ join "," .queueNames | quote }} + {{- else }} + - name: EXECUTOR_QUEUE_NAME + value: {{ .queueName | quote }} + {{- end }} + - name: SRC_LOG_LEVEL + value: {{ $r.Values.executor.log.level | quote }} + - name: SRC_LOG_FORMAT + value: {{ $r.Values.executor.log.format | quote }} + - name: EXECUTOR_MAXIMUM_RUNTIME_PER_JOB + value: {{ $r.Values.executor.maximumRuntimePerJob | quote }} + - name: EXECUTOR_USE_FIRECRACKER + value: "false" + - name: EXECUTOR_USE_KUBERNETES + value: "false" + - name: EXECUTOR_HEALTH_SERVER_ADDR + value: ":8080" + - name: EXECUTOR_JOB_NUM_CPUS + value: "0" + - name: EXECUTOR_JOB_MEMORY + value: "0" + - name: DOCKER_HOST + value: tcp://localhost:2375 + - name: TMPDIR + value: /scratch + {{- range $name, $item := .env }} + - name: {{ $name }} + {{- $item | toYaml | nindent 14 }} + {{- end }} + volumeMounts: + - mountPath: /scratch + name: executor-scratch + {{- with .resources }} + resources: + {{- toYaml . | nindent 12 }} + {{- end }} + - name: dind + image: "{{ $r.Values.dind.image.registry }}/{{ $r.Values.dind.image.repository }}:{{ $r.Values.dind.image.tag }}" + imagePullPolicy: {{ $r.Values.sourcegraph.image.pullPolicy }} + securityContext: + {{- if $r.Values.dind.gVisor.enabled }} + {{- toYaml $r.Values.dind.gVisor.securityContext | nindent 12 }} + {{- else }} + privileged: true + {{- end }} + command: + {{- if $r.Values.dind.gVisor.enabled }} + {{- toYaml $r.Values.dind.gVisor.command | nindent 12 }} + {{- else }} + {{- toYaml $r.Values.dind.command | nindent 12 }} + {{- end }} + livenessProbe: + exec: + command: + - wget + - -qO- + - http://127.0.0.1:2375/_ping + initialDelaySeconds: 15 + periodSeconds: 5 + failureThreshold: 5 + readinessProbe: + exec: + command: + - wget + - -qO- + - http://127.0.0.1:2375/_ping + initialDelaySeconds: 20 + periodSeconds: 5 + failureThreshold: 5 + env: + - name: POD_NAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: metadata.name + ports: + - containerPort: 2375 + protocol: TCP + volumeMounts: + - mountPath: /scratch + name: executor-scratch + - mountPath: /etc/docker/daemon.json + subPath: daemon.json + name: docker-config + {{- if $r.Values.dind.gVisor.enabled }} + - mountPath: /var/lib/docker + name: docker + {{- end }} + enableServiceLinks: false + {{- if $r.Values.dind.gVisor.enabled }} + runtimeClassName: gvisor + {{- end }} + {{- with $r.Values.sourcegraph.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with $r.Values.sourcegraph.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with include "sourcegraph.priorityClassName" (list $r "executor") | trim }}{{ . | nindent 6 }}{{- end }} + {{- with $r.Values.sourcegraph.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with $r.Values.sourcegraph.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + volumes: + - name: executor-scratch + {{- if eq $r.Values.executor.storage.type "ephemeral" }} + ephemeral: + volumeClaimTemplate: + metadata: + labels: + {{- include "sourcegraph.labels" $r | nindent 18 }} + {{- with $r.Values.executor.storage.labels }} + {{- toYaml . | nindent 18 }} + {{- end }} + deploy: sourcegraph + app.kubernetes.io/component: executor + spec: + accessModes: ["ReadWriteOnce"] + {{- if $r.Values.executor.storage.class }} + storageClassName: {{ $r.Values.executor.storage.class }} + {{- end }} + resources: + requests: + storage: {{ required "executor.storage.size is required when storage.type is ephemeral" $r.Values.executor.storage.size }} + {{- else }} + emptyDir: + {{- if $r.Values.executor.storage.size }} + sizeLimit: {{ $r.Values.executor.storage.size }} + {{- end }} + {{- end }} + - name: docker-config + configMap: + defaultMode: 420 + name: docker-config + {{- if $r.Values.dind.gVisor.enabled }} + - name: docker + emptyDir: {} + {{- end }} +{{- end }} diff --git a/charts/sourcegraph-executor/dind/templates/_helpers/_labels.tpl b/charts/sourcegraph-executor/dind/templates/_helpers/_labels.tpl new file mode 100644 index 000000000..f4c0197ef --- /dev/null +++ b/charts/sourcegraph-executor/dind/templates/_helpers/_labels.tpl @@ -0,0 +1,47 @@ +{{/* +Common labels +*/}} +{{- define "sourcegraph.labels" -}} +helm.sh/chart: {{ include "sourcegraph.chart" . }} +{{ include "sourcegraph.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- if .Values.sourcegraph.labels }} +{{ toYaml .Values.sourcegraph.labels }} +{{- end }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "sourcegraph.selectorLabels" -}} +app.kubernetes.io/name: {{ include "sourcegraph.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Labels of a pod template: the common labels, then sourcegraph.podLabels, then +the component's podLabels, then the selector labels. The later one wins when a +key is set more than once, so each key is rendered once and the selector always +matches the pods. + +`helm.sh/chart` and `app.kubernetes.io/version` are left out: they change with +every chart version, and a changed pod template restarts the pods. + +{{ include "sourcegraph.podLabels" (list . .Values.executor) }} +*/}} +{{- define "sourcegraph.podLabels" -}} +{{- $top := index . 0 -}} +{{- $component := index . 1 -}} +{{- $labels := omit (include "sourcegraph.labels" $top | fromYaml) "helm.sh/chart" "app.kubernetes.io/version" -}} +{{- mergeOverwrite $labels ($top.Values.sourcegraph.podLabels | default dict) ($component.podLabels | default dict) (include "sourcegraph.selectorLabels" $top | fromYaml) | toYaml -}} +{{- end }} + +{{- define "executor.labels" -}} +app: {{ include "executor.name" . }} +deploy: sourcegraph +sourcegraph-resource-requires: no-cluster-admin +app.kubernetes.io/component: executor +{{- end}} diff --git a/charts/sourcegraph-executor/dind/templates/executor/docker-daemon.ConfigMap.yaml b/charts/sourcegraph-executor/dind/templates/executor/docker-daemon.ConfigMap.yaml index ecd96f99b..6ffb86250 100644 --- a/charts/sourcegraph-executor/dind/templates/executor/docker-daemon.ConfigMap.yaml +++ b/charts/sourcegraph-executor/dind/templates/executor/docker-daemon.ConfigMap.yaml @@ -2,6 +2,10 @@ apiVersion: v1 kind: ConfigMap metadata: labels: + {{- include "sourcegraph.labels" . | nindent 4 }} + {{- if .Values.executor.labels }} + {{- toYaml .Values.executor.labels | nindent 4 }} + {{- end }} app: executor deploy: sourcegraph app.kubernetes.io/component: executor diff --git a/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Service.yaml b/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Service.yaml index 356293ca8..cf472ab58 100644 --- a/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Service.yaml +++ b/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Service.yaml @@ -7,9 +7,13 @@ metadata: {{- toYaml .Values.privateDockerRegistry.serviceAnnotations | nindent 4 }} {{- end }} labels: + {{- include "sourcegraph.labels" . | nindent 4 }} app: private-docker-registry deploy: sourcegraph app.kubernetes.io/component: private-docker-registry + {{- if .Values.privateDockerRegistry.serviceLabels }} + {{- toYaml .Values.privateDockerRegistry.serviceLabels | nindent 4 }} + {{- end }} name: private-docker-registry spec: ports: diff --git a/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Statefulset.yaml b/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Statefulset.yaml index 71f3194e8..b1cda47a0 100644 --- a/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Statefulset.yaml +++ b/charts/sourcegraph-executor/dind/templates/private-docker-registry/private-docker-registry.Statefulset.yaml @@ -30,15 +30,10 @@ spec: {{- toYaml .Values.privateDockerRegistry.podAnnotations | nindent 8 }} {{- end }} labels: - {{- include "sourcegraph.selectorLabels" . | nindent 8 }} - {{- if .Values.sourcegraph.podLabels }} - {{- toYaml .Values.sourcegraph.podLabels | nindent 8 }} - {{- end }} - {{- if .Values.privateDockerRegistry.podLabels }} - {{- toYaml .Values.privateDockerRegistry.podLabels | nindent 8 }} - {{- end }} + {{- include "sourcegraph.podLabels" (list . .Values.privateDockerRegistry) | nindent 8 }} deploy: sourcegraph app: private-docker-registry + app.kubernetes.io/component: private-docker-registry spec: containers: - name: private-docker-registry @@ -92,6 +87,9 @@ spec: labels: deploy: sourcegraph app.kubernetes.io/component: private-docker-registry + {{- with .Values.privateDockerRegistry.storageLabels }} + {{- toYaml . | nindent 10 }} + {{- end }} spec: accessModes: - ReadWriteOnce diff --git a/charts/sourcegraph-executor/dind/tests/executor_test.yaml b/charts/sourcegraph-executor/dind/tests/executor_test.yaml index ebc890027..b550e6803 100644 --- a/charts/sourcegraph-executor/dind/tests/executor_test.yaml +++ b/charts/sourcegraph-executor/dind/tests/executor_test.yaml @@ -45,3 +45,81 @@ tests: apiVersion: apps/v1 name: executor-batches documentIndex: 1 + + # The chart version is part of the helm.sh/chart label, so pin it to compare whole label sets. + - it: should add sourcegraph.labels, the chart's labels, and executor.storage.labels to the per-pod PVC + template: executor/executor.Deployment.yaml + release: + name: executor + chart: + version: 1.2.3 + appVersion: 1.2.3 + set: + sourcegraph: + labels: + example.com/project: sourcegraph + executor: + queueName: batches + storage: + type: ephemeral + size: 50Gi + labels: + example.com/service: executor + asserts: + - equal: + path: spec.template.spec.volumes[0].ephemeral.volumeClaimTemplate.metadata + value: + labels: + helm.sh/chart: sourcegraph-executor-dind-1.2.3 + app.kubernetes.io/name: sourcegraph-executor-dind + app.kubernetes.io/instance: executor + app.kubernetes.io/version: 1.2.3 + app.kubernetes.io/managed-by: Helm + example.com/project: sourcegraph + example.com/service: executor + deploy: sourcegraph + app.kubernetes.io/component: executor + - equal: + path: spec.template.spec.volumes[0].ephemeral.volumeClaimTemplate.spec.resources.requests.storage + value: 50Gi + + - it: should add only the chart's own labels to the per-pod PVC when no label value is set + template: executor/executor.Deployment.yaml + release: + name: executor + chart: + version: 1.2.3 + appVersion: 1.2.3 + set: + executor: + queueName: batches + storage: + type: ephemeral + size: 50Gi + asserts: + - equal: + path: spec.template.spec.volumes[0].ephemeral.volumeClaimTemplate.metadata + value: + labels: + helm.sh/chart: sourcegraph-executor-dind-1.2.3 + app.kubernetes.io/name: sourcegraph-executor-dind + app.kubernetes.io/instance: executor + app.kubernetes.io/version: 1.2.3 + app.kubernetes.io/managed-by: Helm + deploy: sourcegraph + app.kubernetes.io/component: executor + + - it: should ignore executor.storage.labels on an emptyDir volume + template: executor/executor.Deployment.yaml + set: + executor: + queueName: batches + storage: + labels: + example.com/project: sourcegraph + asserts: + - equal: + path: spec.template.spec.volumes[0].name + value: executor-scratch + - notExists: + path: spec.template.spec.volumes[0].ephemeral diff --git a/charts/sourcegraph-executor/dind/tests/labels_test.yaml b/charts/sourcegraph-executor/dind/tests/labels_test.yaml new file mode 100644 index 000000000..9a9dfc7ed --- /dev/null +++ b/charts/sourcegraph-executor/dind/tests/labels_test.yaml @@ -0,0 +1,107 @@ +suite: sourcegraph.labels on every object +# No `templates` list, so every template in the chart is rendered, including +# ones added later. The test fails with "no manifest found" for a template that +# renders nothing, so enable a new optional template here. +tests: + - it: should add sourcegraph.labels to every object the chart renders + set: + sourcegraph: + labels: + example.com/installation: production + example.com/project: sourcegraph + executor: + queueName: batches + privateDockerRegistry: + enabled: true + asserts: + - isSubset: + path: metadata.labels + content: + app.kubernetes.io/managed-by: Helm + deploy: sourcegraph + example.com/installation: production + example.com/project: sourcegraph + - exists: + path: metadata.labels["helm.sh/chart"] + - exists: + path: metadata.labels["app.kubernetes.io/name"] + - exists: + path: metadata.labels["app.kubernetes.io/instance"] + - exists: + path: metadata.labels["app.kubernetes.io/version"] + - exists: + path: metadata.labels["app.kubernetes.io/component"] + - it: should add the labels of a component to every object of it + set: + executor: + queueName: batches + labels: + example.com/service: component + privateDockerRegistry: + enabled: true + labels: + example.com/service: component + serviceLabels: + example.com/service: component + asserts: + - equal: + path: metadata.labels["example.com/service"] + value: component + - it: should add sourcegraph.labels and the chart's labels to the pod template, and let the selector labels win + template: executor/executor.Deployment.yaml + set: + sourcegraph: + labels: + example.com/project: sourcegraph + podLabels: + example.com/project: pods + # A selector label: a different value would stop the selector from matching the pods. + app.kubernetes.io/instance: other + executor: + queueName: batches + asserts: + - isSubset: + path: spec.template.metadata.labels + content: + app.kubernetes.io/instance: RELEASE-NAME + app.kubernetes.io/managed-by: Helm + deploy: sourcegraph + example.com/project: pods + - notExists: + path: spec.template.metadata.labels["helm.sh/chart"] + - exists: + path: spec.template.metadata.labels["app.kubernetes.io/name"] + - notExists: + path: spec.template.metadata.labels["app.kubernetes.io/version"] + - exists: + path: spec.template.metadata.labels["app.kubernetes.io/component"] + - it: should add sourcegraph.labels and the chart's labels to the pod template, and let the selector labels win + template: private-docker-registry/private-docker-registry.Statefulset.yaml + set: + sourcegraph: + labels: + example.com/project: sourcegraph + podLabels: + example.com/project: pods + # A selector label: a different value would stop the selector from matching the pods. + app.kubernetes.io/instance: other + executor: + queueName: batches + privateDockerRegistry: + enabled: true + asserts: + - isSubset: + path: spec.template.metadata.labels + content: + app.kubernetes.io/instance: RELEASE-NAME + app.kubernetes.io/managed-by: Helm + deploy: sourcegraph + example.com/project: pods + - notExists: + path: spec.template.metadata.labels["helm.sh/chart"] + - exists: + path: spec.template.metadata.labels["app.kubernetes.io/name"] + - notExists: + path: spec.template.metadata.labels["app.kubernetes.io/version"] + - exists: + path: spec.template.metadata.labels["app.kubernetes.io/component"] diff --git a/charts/sourcegraph-executor/dind/tests/privateDockerRegistry_test.yaml b/charts/sourcegraph-executor/dind/tests/privateDockerRegistry_test.yaml index cbff736c7..bcaaa6ddd 100644 --- a/charts/sourcegraph-executor/dind/tests/privateDockerRegistry_test.yaml +++ b/charts/sourcegraph-executor/dind/tests/privateDockerRegistry_test.yaml @@ -26,3 +26,32 @@ tests: asserts: - hasDocuments: count: 0 + + - it: should label the cache volume claim template with only the chart's own labels by default + template: private-docker-registry/private-docker-registry.Statefulset.yaml + set: + sourcegraph: + labels: + example.com/project: sourcegraph + asserts: + - equal: + path: spec.volumeClaimTemplates[0].metadata.labels + value: + deploy: sourcegraph + app.kubernetes.io/component: private-docker-registry + + - it: should add privateDockerRegistry.storageLabels to the cache volume claim template + template: private-docker-registry/private-docker-registry.Statefulset.yaml + set: + privateDockerRegistry: + storageLabels: + example.com/project: sourcegraph + example.com/service: registry + asserts: + - equal: + path: spec.volumeClaimTemplates[0].metadata.labels + value: + deploy: sourcegraph + app.kubernetes.io/component: private-docker-registry + example.com/project: sourcegraph + example.com/service: registry diff --git a/charts/sourcegraph-executor/dind/values.yaml b/charts/sourcegraph-executor/dind/values.yaml index 571ebe624..43e98d936 100644 --- a/charts/sourcegraph-executor/dind/values.yaml +++ b/charts/sourcegraph-executor/dind/values.yaml @@ -131,6 +131,9 @@ executor: # -- StorageClass for the ephemeral volume. Only used when type is ephemeral. # Defaults to the cluster default StorageClass when empty. class: "" + # -- Optional labels to add to the ephemeral volume's PVC, in addition to `sourcegraph.labels`. Only used when type + # is ephemeral. Changing them restarts the executor pods. See [PersistentVolumeClaim labels](#persistentvolumeclaim-labels) + labels: {} # -- Pod-level security context applied to executor pods. # fsGroup: 101 ensures the scratch volume is writable by the sourcegraph user (uid=100, gid=101) # when running in Docker mode (dind). Set to {} to disable. @@ -217,6 +220,11 @@ privateDockerRegistry: repository: registry tag: 3 storageSize: 10Gi + # -- Labels to add to the `cache` volume claim template of the `private-docker-registry` StatefulSet. Kubernetes does + # not allow changing them on an installed StatefulSet. See [PersistentVolumeClaim labels](#persistentvolumeclaim-labels) + storageLabels: {} + # -- Add extra labels to the `private-docker-registry` Service + serviceLabels: {} # -- Extra environment variables to set on the registry container. # Registry can be configured with environment variables using the pattern # REGISTRY_
_ (uppercase). Full reference: diff --git a/charts/sourcegraph-executor/k8s/templates/_helpers.tpl b/charts/sourcegraph-executor/k8s/templates/_helpers.tpl index 70f75b28c..cb8bb442b 100644 --- a/charts/sourcegraph-executor/k8s/templates/_helpers.tpl +++ b/charts/sourcegraph-executor/k8s/templates/_helpers.tpl @@ -12,29 +12,6 @@ Create chart name and version as used by the chart label. {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} {{- end }} -{{/* -Common labels -*/}} -{{- define "sourcegraph.labels" -}} -helm.sh/chart: {{ include "sourcegraph.chart" . }} -{{ include "sourcegraph.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- if .Values.sourcegraph.labels }} -{{ toYaml .Values.sourcegraph.labels }} -{{- end }} -{{- end }} - -{{/* -Selector labels -*/}} -{{- define "sourcegraph.selectorLabels" -}} -app.kubernetes.io/name: {{ include "sourcegraph.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end }} - {{/* Create the name of the service account to use */}} @@ -106,9 +83,3 @@ executor-{{join "-" .Values.executor.queueNames }} {{- end }} {{- end }} -{{- define "executor.labels" -}} -app: {{ include "executor.name" . }} -deploy: sourcegraph -sourcegraph-resource-requires: no-cluster-admin -app.kubernetes.io/component: executor -{{- end}} diff --git a/charts/sourcegraph-executor/k8s/templates/_helpers/_labels.tpl b/charts/sourcegraph-executor/k8s/templates/_helpers/_labels.tpl new file mode 100644 index 000000000..f4c0197ef --- /dev/null +++ b/charts/sourcegraph-executor/k8s/templates/_helpers/_labels.tpl @@ -0,0 +1,47 @@ +{{/* +Common labels +*/}} +{{- define "sourcegraph.labels" -}} +helm.sh/chart: {{ include "sourcegraph.chart" . }} +{{ include "sourcegraph.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- if .Values.sourcegraph.labels }} +{{ toYaml .Values.sourcegraph.labels }} +{{- end }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "sourcegraph.selectorLabels" -}} +app.kubernetes.io/name: {{ include "sourcegraph.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Labels of a pod template: the common labels, then sourcegraph.podLabels, then +the component's podLabels, then the selector labels. The later one wins when a +key is set more than once, so each key is rendered once and the selector always +matches the pods. + +`helm.sh/chart` and `app.kubernetes.io/version` are left out: they change with +every chart version, and a changed pod template restarts the pods. + +{{ include "sourcegraph.podLabels" (list . .Values.executor) }} +*/}} +{{- define "sourcegraph.podLabels" -}} +{{- $top := index . 0 -}} +{{- $component := index . 1 -}} +{{- $labels := omit (include "sourcegraph.labels" $top | fromYaml) "helm.sh/chart" "app.kubernetes.io/version" -}} +{{- mergeOverwrite $labels ($top.Values.sourcegraph.podLabels | default dict) ($component.podLabels | default dict) (include "sourcegraph.selectorLabels" $top | fromYaml) | toYaml -}} +{{- end }} + +{{- define "executor.labels" -}} +app: {{ include "executor.name" . }} +deploy: sourcegraph +sourcegraph-resource-requires: no-cluster-admin +app.kubernetes.io/component: executor +{{- end}} diff --git a/charts/sourcegraph-executor/k8s/templates/executor.ConfigMap.yaml b/charts/sourcegraph-executor/k8s/templates/executor.ConfigMap.yaml index 6e773607b..9963cfea1 100644 --- a/charts/sourcegraph-executor/k8s/templates/executor.ConfigMap.yaml +++ b/charts/sourcegraph-executor/k8s/templates/executor.ConfigMap.yaml @@ -3,7 +3,11 @@ kind: ConfigMap metadata: name: {{ include "executor.name" . }} labels: -{{- include "executor.labels" . | nindent 4 }} + {{- include "sourcegraph.labels" . | nindent 4 }} + {{- include "executor.labels" . | nindent 4 }} + {{- if .Values.executor.labels }} + {{- toYaml .Values.executor.labels | nindent 4 }} + {{- end }} data: EXECUTOR_USE_FIRECRACKER: "false" EXECUTOR_FRONTEND_URL: "{{ .Values.executor.frontendUrl }}" diff --git a/charts/sourcegraph-executor/k8s/templates/executor.Deployment.yaml b/charts/sourcegraph-executor/k8s/templates/executor.Deployment.yaml index 1cb35d2c4..a001db219 100644 --- a/charts/sourcegraph-executor/k8s/templates/executor.Deployment.yaml +++ b/charts/sourcegraph-executor/k8s/templates/executor.Deployment.yaml @@ -6,6 +6,7 @@ metadata: description: Runs a sourcegraph executor kubectl.kubernetes.io/default-container: executor labels: + {{- include "sourcegraph.labels" . | nindent 4 }} {{- include "executor.labels" . | nindent 4 }} {{- if .Values.executor.labels }} {{- toYaml .Values.executor.labels | nindent 4 }} @@ -37,13 +38,7 @@ spec: {{- toYaml .Values.executor.podAnnotations | nindent 8 }} {{- end }} labels: - {{- include "sourcegraph.selectorLabels" . | nindent 8 }} - {{- if .Values.sourcegraph.podLabels }} - {{- toYaml .Values.sourcegraph.podLabels | nindent 8 }} - {{- end }} - {{- if .Values.executor.podLabels }} - {{- toYaml .Values.executor.podLabels | nindent 8 }} - {{- end }} + {{- include "sourcegraph.podLabels" (list . .Values.executor) | nindent 8 }} {{- include "executor.labels" . | nindent 8 }} spec: securityContext: diff --git a/charts/sourcegraph-executor/k8s/templates/executor.Role.yaml b/charts/sourcegraph-executor/k8s/templates/executor.Role.yaml index 7a3d1ef4d..68caac8a5 100644 --- a/charts/sourcegraph-executor/k8s/templates/executor.Role.yaml +++ b/charts/sourcegraph-executor/k8s/templates/executor.Role.yaml @@ -4,6 +4,10 @@ kind: Role metadata: name: sg-executor-role labels: + {{- include "sourcegraph.labels" . | nindent 4 }} + {{- if .Values.executor.labels }} + {{- toYaml .Values.executor.labels | nindent 4 }} + {{- end }} category: rbac deploy: sourcegraph sourcegraph-resource-requires: cluster-admin diff --git a/charts/sourcegraph-executor/k8s/templates/executor.RoleBinding.yaml b/charts/sourcegraph-executor/k8s/templates/executor.RoleBinding.yaml index f13311c5f..e7e6f2df7 100644 --- a/charts/sourcegraph-executor/k8s/templates/executor.RoleBinding.yaml +++ b/charts/sourcegraph-executor/k8s/templates/executor.RoleBinding.yaml @@ -4,6 +4,10 @@ kind: RoleBinding metadata: name: sg-executor-role-binding labels: + {{- include "sourcegraph.labels" . | nindent 4 }} + {{- if .Values.executor.labels }} + {{- toYaml .Values.executor.labels | nindent 4 }} + {{- end }} category: rbac deploy: sourcegraph sourcegraph-resource-requires: cluster-admin diff --git a/charts/sourcegraph-executor/k8s/templates/executor.Secret.yaml b/charts/sourcegraph-executor/k8s/templates/executor.Secret.yaml index 669afae55..936b4ea1a 100644 --- a/charts/sourcegraph-executor/k8s/templates/executor.Secret.yaml +++ b/charts/sourcegraph-executor/k8s/templates/executor.Secret.yaml @@ -2,6 +2,12 @@ apiVersion: v1 kind: Secret metadata: + labels: + {{- include "sourcegraph.labels" . | nindent 4 }} + {{- include "executor.labels" . | nindent 4 }} + {{- if .Values.executor.labels }} + {{- toYaml .Values.executor.labels | nindent 4 }} + {{- end }} name: {{ include "executor.name" . }} type: Opaque data: diff --git a/charts/sourcegraph-executor/k8s/templates/executor.Service.yaml b/charts/sourcegraph-executor/k8s/templates/executor.Service.yaml index 33d841b3b..ed865c0cb 100644 --- a/charts/sourcegraph-executor/k8s/templates/executor.Service.yaml +++ b/charts/sourcegraph-executor/k8s/templates/executor.Service.yaml @@ -6,6 +6,7 @@ metadata: {{- toYaml .Values.executor.serviceAnnotations | nindent 4 }} {{- end }} labels: + {{- include "sourcegraph.labels" . | nindent 4 }} {{ include "executor.labels" . | indent 4 }} {{- if .Values.executor.serviceLabels }} {{- toYaml .Values.executor.serviceLabels | nindent 4 }} diff --git a/charts/sourcegraph-executor/k8s/templates/executor.ServiceAccount.yaml b/charts/sourcegraph-executor/k8s/templates/executor.ServiceAccount.yaml index 6a0f44627..39f2a0fb7 100644 --- a/charts/sourcegraph-executor/k8s/templates/executor.ServiceAccount.yaml +++ b/charts/sourcegraph-executor/k8s/templates/executor.ServiceAccount.yaml @@ -4,6 +4,10 @@ kind: ServiceAccount metadata: name: sg-executor labels: + {{- include "sourcegraph.labels" . | nindent 4 }} + {{- if .Values.executor.labels }} + {{- toYaml .Values.executor.labels | nindent 4 }} + {{- end }} category: rbac deploy: sourcegraph sourcegraph-resource-requires: cluster-admin diff --git a/charts/sourcegraph-executor/k8s/tests/labels_test.yaml b/charts/sourcegraph-executor/k8s/tests/labels_test.yaml new file mode 100644 index 000000000..349649e74 --- /dev/null +++ b/charts/sourcegraph-executor/k8s/tests/labels_test.yaml @@ -0,0 +1,73 @@ +suite: sourcegraph.labels on every object +# No `templates` list, so every template in the chart is rendered, including +# ones added later. The test fails with "no manifest found" for a template that +# renders nothing, so enable a new optional template here. +tests: + - it: should add sourcegraph.labels to every object the chart renders + set: + sourcegraph: + labels: + example.com/installation: production + example.com/project: sourcegraph + executor: + queueName: batches + configureRbac: true + asserts: + - isSubset: + path: metadata.labels + content: + app.kubernetes.io/managed-by: Helm + deploy: sourcegraph + example.com/installation: production + example.com/project: sourcegraph + - exists: + path: metadata.labels["helm.sh/chart"] + - exists: + path: metadata.labels["app.kubernetes.io/name"] + - exists: + path: metadata.labels["app.kubernetes.io/instance"] + - exists: + path: metadata.labels["app.kubernetes.io/version"] + - exists: + path: metadata.labels["app.kubernetes.io/component"] + - it: should add executor.labels to every object, and serviceLabels to the Service + set: + executor: + queueName: batches + configureRbac: true + labels: + example.com/service: executor + serviceLabels: + example.com/service: executor + asserts: + - equal: + path: metadata.labels["example.com/service"] + value: executor + - it: should add sourcegraph.labels and the chart's labels to the pod template, and let the selector labels win + template: executor.Deployment.yaml + set: + sourcegraph: + labels: + example.com/project: sourcegraph + podLabels: + example.com/project: pods + # A selector label: a different value would stop the selector from matching the pods. + app.kubernetes.io/instance: other + executor: + queueName: batches + asserts: + - isSubset: + path: spec.template.metadata.labels + content: + app.kubernetes.io/instance: RELEASE-NAME + app.kubernetes.io/managed-by: Helm + deploy: sourcegraph + example.com/project: pods + - notExists: + path: spec.template.metadata.labels["helm.sh/chart"] + - exists: + path: spec.template.metadata.labels["app.kubernetes.io/name"] + - notExists: + path: spec.template.metadata.labels["app.kubernetes.io/version"] + - exists: + path: spec.template.metadata.labels["app.kubernetes.io/component"] diff --git a/charts/sourcegraph-migrator/templates/_helpers.tpl b/charts/sourcegraph-migrator/templates/_helpers.tpl index f6ce9b36d..2de962a1e 100644 --- a/charts/sourcegraph-migrator/templates/_helpers.tpl +++ b/charts/sourcegraph-migrator/templates/_helpers.tpl @@ -12,29 +12,6 @@ Create chart name and version as used by the chart label. {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} {{- end }} -{{/* -Common labels -*/}} -{{- define "sourcegraph.labels" -}} -helm.sh/chart: {{ include "sourcegraph.chart" . }} -{{ include "sourcegraph.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- if .Values.sourcegraph.labels }} -{{ toYaml .Values.sourcegraph.labels }} -{{- end }} -{{- end }} - -{{/* -Selector labels -*/}} -{{- define "sourcegraph.selectorLabels" -}} -app.kubernetes.io/name: {{ include "sourcegraph.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end }} - {{/* Create the name of the service account to use */}} diff --git a/charts/sourcegraph-migrator/templates/_helpers/_labels.tpl b/charts/sourcegraph-migrator/templates/_helpers/_labels.tpl new file mode 100644 index 000000000..a73d9a7bb --- /dev/null +++ b/charts/sourcegraph-migrator/templates/_helpers/_labels.tpl @@ -0,0 +1,40 @@ +{{/* +Common labels +*/}} +{{- define "sourcegraph.labels" -}} +helm.sh/chart: {{ include "sourcegraph.chart" . }} +{{ include "sourcegraph.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- if .Values.sourcegraph.labels }} +{{ toYaml .Values.sourcegraph.labels }} +{{- end }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "sourcegraph.selectorLabels" -}} +app.kubernetes.io/name: {{ include "sourcegraph.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Labels of a pod template: the common labels, then sourcegraph.podLabels, then +the component's podLabels, then the selector labels. The later one wins when a +key is set more than once, so each key is rendered once and the selector always +matches the pods. + +`helm.sh/chart` and `app.kubernetes.io/version` are left out: they change with +every chart version, and a changed pod template restarts the pods. + +{{ include "sourcegraph.podLabels" (list . .Values.migrator) }} +*/}} +{{- define "sourcegraph.podLabels" -}} +{{- $top := index . 0 -}} +{{- $component := index . 1 -}} +{{- $labels := omit (include "sourcegraph.labels" $top | fromYaml) "helm.sh/chart" "app.kubernetes.io/version" -}} +{{- mergeOverwrite $labels ($top.Values.sourcegraph.podLabels | default dict) ($component.podLabels | default dict) (include "sourcegraph.selectorLabels" $top | fromYaml) | toYaml -}} +{{- end }} diff --git a/charts/sourcegraph-migrator/templates/migrator/sourcegraph-migrator.Job.yaml b/charts/sourcegraph-migrator/templates/migrator/sourcegraph-migrator.Job.yaml index c77ec3b33..4e41b3a28 100644 --- a/charts/sourcegraph-migrator/templates/migrator/sourcegraph-migrator.Job.yaml +++ b/charts/sourcegraph-migrator/templates/migrator/sourcegraph-migrator.Job.yaml @@ -24,15 +24,10 @@ spec: {{- toYaml .Values.migrator.podAnnotations | nindent 8 }} {{- end }} labels: - {{- include "sourcegraph.selectorLabels" . | nindent 8 }} - {{- if .Values.sourcegraph.podLabels }} - {{- toYaml .Values.sourcegraph.podLabels | nindent 8 }} - {{- end }} - {{- if .Values.migrator.podLabels }} - {{- toYaml .Values.migrator.podLabels | nindent 8 }} - {{- end }} + {{- include "sourcegraph.podLabels" (list . .Values.migrator) | nindent 8 }} job: migrator deploy: sourcegraph + app.kubernetes.io/component: migrator spec: containers: - name: migrator diff --git a/charts/sourcegraph-migrator/tests/labels_test.yaml b/charts/sourcegraph-migrator/tests/labels_test.yaml new file mode 100644 index 000000000..47c6571a0 --- /dev/null +++ b/charts/sourcegraph-migrator/tests/labels_test.yaml @@ -0,0 +1,55 @@ +suite: sourcegraph.labels on every object +# No `templates` list, so every template in the chart is rendered, including +# ones added later. The test fails with "no manifest found" for a template that +# renders nothing, so enable a new optional template here. +tests: + - it: should add sourcegraph.labels to every object the chart renders + set: + sourcegraph: + labels: + example.com/installation: production + example.com/project: sourcegraph + asserts: + - isSubset: + path: metadata.labels + content: + app.kubernetes.io/managed-by: Helm + deploy: sourcegraph + example.com/installation: production + example.com/project: sourcegraph + - exists: + path: metadata.labels["helm.sh/chart"] + - exists: + path: metadata.labels["app.kubernetes.io/name"] + - exists: + path: metadata.labels["app.kubernetes.io/instance"] + - exists: + path: metadata.labels["app.kubernetes.io/version"] + - exists: + path: metadata.labels["app.kubernetes.io/component"] + - it: should add sourcegraph.labels and the chart's labels to the pod template, and let the selector labels win + template: migrator/sourcegraph-migrator.Job.yaml + set: + sourcegraph: + labels: + example.com/project: sourcegraph + podLabels: + example.com/project: pods + # A selector label: a different value would stop the selector from matching the pods. + app.kubernetes.io/instance: other + asserts: + - isSubset: + path: spec.template.metadata.labels + content: + app.kubernetes.io/instance: RELEASE-NAME + app.kubernetes.io/managed-by: Helm + deploy: sourcegraph + example.com/project: pods + - notExists: + path: spec.template.metadata.labels["helm.sh/chart"] + - exists: + path: spec.template.metadata.labels["app.kubernetes.io/name"] + - notExists: + path: spec.template.metadata.labels["app.kubernetes.io/version"] + - exists: + path: spec.template.metadata.labels["app.kubernetes.io/component"] diff --git a/charts/sourcegraph/CHANGELOG.md b/charts/sourcegraph/CHANGELOG.md index cb6d00a3f..1fa7a6b3f 100644 --- a/charts/sourcegraph/CHANGELOG.md +++ b/charts/sourcegraph/CHANGELOG.md @@ -10,7 +10,14 @@ Use `**BREAKING**:` to denote a breaking change - Added `frontend.internalServiceAnnotations` and `frontend.internalServiceLabels` for the `sourcegraph-frontend-internal` Service. That Service no longer renders `frontend.serviceAnnotations` and `frontend.serviceLabels`, which now apply to the `sourcegraph-frontend` Service only, so load balancer annotations or a service mesh label set for the external Service no longer land on the internal one. If you relied on them on the internal Service, set the new values. - The `correct-data-dir-permissions` init containers of `pgsql`, `codeintel-db`, and `codeinsights-db` now run the database's own image instead of `sourcegraph/alpine-3.14`, which was based on an end-of-life Alpine release. The chart no longer pulls `alpine-3.14`, so private registries no longer need to mirror it. `alpine.image` has been removed and is ignored if set; `alpine.containerSecurityContext`, `alpine.resources`, and `codeInsightsDB.init.containerSecurityContext` still apply. -- Added `storageLabels` on `blobstore`, `codeInsightsDB`, `codeIntelDB`, `pgsql`, `prometheus`, `redisCache`, and `redisStore` to label each component's PersistentVolumeClaim, and `.serviceAccount.labels` to label a component's ServiceAccount and, for `frontend` and `prometheus`, its Role, RoleBinding, ClusterRole, and ClusterRoleBinding. Also added `storageLabels` on `preciseCodeIntel` and `syntacticCodeIntel`, to label the per-pod `/tmp` scratch PersistentVolumeClaim when `storageType` is `pvc`; they default to empty, so no pod restarts. `sourcegraph.labels` and the chart's standard labels (`helm.sh/chart`, `app.kubernetes.io/name`, `app.kubernetes.io/instance`, `app.kubernetes.io/version`, `app.kubernetes.io/managed-by`) are now also rendered on these PersistentVolumeClaims, on every ServiceAccount, and on the `prometheus` RBAC objects, as they already are on every workload and on the `frontend` Role and RoleBinding, so existing releases gain these labels on upgrade. Labels on these objects can be changed in place, so nothing is recreated or restarted. See [Adopting existing workloads](README.md#adopting-existing-workloads). +- Fixed the "pods available" panel and alert of the OpenTelemetry Collector dashboard, which query `up{app=~".*otel-collector"}`, having no data. The `otel-collector` Service was missing the `app` label that every other scraped Service has, so its metrics had no `app` and `job` labels. They now have `app="otel-collector"` and `job="otel-collector"`. The collector also served its own metrics on 127.0.0.1 only, unless `jaeger.enabled` was set, so Prometheus could not scrape them. The chart now always passes the container one argument, after `openTelemetry.gateway.args`, that serves them on all interfaces, port 8888. A command set in `openTelemetry.gateway.command` gets the argument too, so a shell wrapper must pass it on to the collector: as `"$0" "$@"` when the script is the first of `openTelemetry.gateway.args` after `command: ["/bin/sh", "-c"]`, or as `"$@"` when the script is in `command`, followed by a placeholder for `$0`. The argument replaces `service.telemetry.metrics.readers` of the configuration files the command loads. +- Every object the chart creates now has the `deploy` and `app.kubernetes.io/component` labels. PodDisruptionBudgets get the component of their workload, PriorityClasses get `priority-class`, the StorageClass gets `storage` and keeps `deploy: sourcegraph-storage`, and the `sg-test-connection` Pod gets `test-connection`. `extraResources` get `deploy: sourcegraph` unless they set their own; set `app.kubernetes.io/component` on them yourself. The `jaeger` ConfigMap, Deployment, and Services gain `app.kubernetes.io/instance` and keep `app.kubernetes.io/name: jaeger`, which their selectors use. Labels on these objects can be changed in place; the `jaeger` pod restarts once, because its `checksum/jaeger-config` annotation covers the ConfigMap's labels. +- `.labels` is now also rendered on the component's ConfigMaps, Secrets, and PodDisruptionBudget, not only on its workload, so releases that set it gain these labels on upgrade. Added `storageClass.labels` and a `labels` field on each `priorityClasses` entry. +- Fixed `redisStore.serviceAccount.annotations` being ignored, so the `redis-store` ServiceAccount was created without them, e.g. without a workload identity annotation. They are now rendered, as on every other ServiceAccount. +- The `prometheus` pod now restarts when an upgrade changes the chart's `prometheus.yml`, through a `checksum/prometheus-config` pod annotation. Prometheus does not reload a changed ConfigMap by itself, so such a change took effect only at the pod's next restart. The annotation is not set with `prometheus.existingConfig`. +- The `otel-collector` pod now restarts when `openTelemetry.gateway.config.traces` changes, through a `checksum/otel-collector-config` pod annotation. The collector does not reload a changed ConfigMap by itself, so a changed exporter or processor took effect only at the pod's next restart. The annotation is set only with `openTelemetry.gateway.config.traces.exporters`, which is when the chart renders that ConfigMap, so the default install is not affected. +- `sourcegraph.labels` and the chart's standard labels (`helm.sh/chart`, `app.kubernetes.io/name`, `app.kubernetes.io/instance`, `app.kubernetes.io/version`, `app.kubernetes.io/managed-by`) are now rendered on every object the chart creates: also on pods, Services, ConfigMaps, Secrets, the Ingress, PodDisruptionBudgets, PriorityClasses, and `extraResources`, so existing releases gain these labels on upgrade. Pods do not get `helm.sh/chart` and `app.kubernetes.io/version`, which change with every chart version and would restart every pod on every upgrade, and they also gain `app.kubernetes.io/component`. The per-pod scratch PersistentVolumeClaim of `preciseCodeIntel` and `syntacticCodeIntel`, created when `storageType` is `pvc`, gets all of these labels, so with that setting those pods restart whenever the chart version changes. Labels that an `extraResources` entry, `serviceLabels`, or `podLabels` set take precedence, except that `podLabels` cannot change a selector label. No object is recreated. Every pod restarts once, on this upgrade. Prometheus copies the labels of a Service or pod to the metrics it scrapes from it, so the bundled Prometheus configuration now drops the five standard labels and the keys of `sourcegraph.labels`. No bundled dashboard or alert queries them. Metrics lose `app_kubernetes_io_managed_by`, which Helm added to every Service, `otel-collector` metrics lose the other standard labels, which its Service already had, and `cadvisor` and `jaeger` metrics, scraped from pods, lose `app_kubernetes_io_name` and `app_kubernetes_io_instance` and gain `app_kubernetes_io_component`. To add a label to the metrics of a Service, set it in `.serviceLabels`, with a key that is not in `sourcegraph.labels`. If you set `prometheus.existingConfig`, add the same `labeldrop` rule after each `__meta_kubernetes_service_label_(.+)` and `__meta_kubernetes_pod_label_(.+)` `labelmap` rule in your `prometheus.yml`, as in the [k3s example](examples/prometheus/prometheus-override-k3s.ConfigMap.yaml), and append the keys of `sourcegraph.labels` to its regex. +- Added `storageLabels` on `blobstore`, `codeInsightsDB`, `codeIntelDB`, `pgsql`, `prometheus`, `redisCache`, and `redisStore` to label each component's PersistentVolumeClaim, and `.serviceAccount.labels` to label a component's ServiceAccount and, for `frontend` and `prometheus`, its Role, RoleBinding, ClusterRole, and ClusterRoleBinding. Also added `storageLabels` on `preciseCodeIntel` and `syntacticCodeIntel`, to label the per-pod `/tmp` scratch PersistentVolumeClaim when `storageType` is `pvc`; they default to empty, so no pod restarts. `sourcegraph.labels` and the chart's standard labels (`helm.sh/chart`, `app.kubernetes.io/name`, `app.kubernetes.io/instance`, `app.kubernetes.io/version`, `app.kubernetes.io/managed-by`) are now also rendered on these PersistentVolumeClaims, on every ServiceAccount, and on the `prometheus` RBAC objects, as they already are on every workload and on the `frontend` Role and RoleBinding, so existing releases gain these labels on upgrade. Labels on these objects can be changed in place, so none is recreated. See [Adopting existing workloads](README.md#adopting-existing-workloads). - Fixed the `jaeger` ServiceAccount failing to render with `jaeger.serviceAccount.create: true`. Its name and annotations were read from a `tracing` value that does not exist, and are now read from `jaeger.serviceAccount`, as the `jaeger` Deployment's `serviceAccountName` already was. - Added `sourcegraph.selectorLabelsOverride`, `.extraSelectorLabels`, and `storageLabels` on `gitserver`, `grafana`, `indexedSearch`, and `searcher`, so the chart can render the selectors and volume claim templates of workloads created outside it, such as with Kustomize, and a Helm release can take them over without deleting them. All default to empty, so existing releases render the same selectors. Never change them on an installed release: Kubernetes does not allow changing these fields. See [Adopting existing workloads](README.md#adopting-existing-workloads). - Fixed `frontend.command` and `migrator.command` rendering a list as a single element, e.g. `["sh", "-c", "