From 1e46feebedebaf71fd314c243bb1cb023d9de44e Mon Sep 17 00:00:00 2001 From: fullsend-code <278716306+fullsend-ai-coder[bot]@users.noreply.github.com> Date: Mon, 26 Oct 2026 08:16:14 +0000 Subject: [PATCH 1/6] docs(#1094): unify SLO documentation Merge the content of the error-budgets research (SLOs-SLIs.md and users-expectations.md in packit/research) into the official Service Level Objectives page so there is a single authoritative document. The page now contains: - job-outcome definitions (success / failure / error) - the three SLOs with the existing official thresholds (SLO2 keeps "5 minutes after the Copr build has finished", not the research's 12 hours) - the SLIs per objective and how they are measured (Prometheus Pushgateway, histograms with buckets aligned to SLO thresholds, Grafana), checked against packit-service monitoring code - a rationale section explaining why each objective and indicator was chosen, based on the collected user expectations - guidance for other teams defining their own SLOs - a note that the page supersedes the research, which stays as historical background SLO headings keep their text, so existing anchors do not change. Note: the full site build (make import && yarn build) could not run in the sandbox because git clones fail on TLS verification. The page was compiled with @mdx-js/mdx 1.6.22 (the MDX version used by Docusaurus 2.4.1) instead. pre-commit could not fetch hook repos; check-merge-conflict and detect-private-key were run directly and passed; prettier excludes docs/. Closes #1094 Co-Authored-By: Claude Opus 5.5 --- docs/service-level-objectives.md | 130 +++++++++++++++++++++++++++++-- 1 file changed, 124 insertions(+), 6 deletions(-) diff --git a/docs/service-level-objectives.md b/docs/service-level-objectives.md index f22fdfb3bd..5172c9c6ef 100644 --- a/docs/service-level-objectives.md +++ b/docs/service-level-objectives.md @@ -11,6 +11,26 @@ objectives](https://sre.google/sre-book/service-level-objectives/) defined in Google's SRE book. It means that when we run out of error budget, we stop all the work on new features and start stabilizing the service to get our budget back. +This page is the official definition of Packit's Service Level Objectives +(SLOs) and the Service Level Indicators (SLIs) we use to track them. It also +explains why we picked them, so that other teams can use it as a guide when +defining their own SLOs. + +## Job outcomes + +The objectives below talk about jobs (builds and tests) ending with a status. +To make them unambiguous, we distinguish these outcomes: + +- **success**: the job finished and everything is awesome +- **failure**: the job finished and wasn't successful: the PR needs to be fixed +- **error**: infrastructure problems prevented the job from completing + +Only _success_ and _failure_ count as good outcomes for our objectives. Both +give you a real answer about your change. An _error_ means Packit (or a service +it depends on) let you down. + +## Objectives + We have 3 objectives defined for Packit right now. :::caution @@ -21,29 +41,127 @@ actual numbers. ::: -## SLO1: Changes to GitHub PRs receive the first status update within 15 seconds in 99% of cases +### SLO1: Changes to GitHub PRs receive the first status update within 15 seconds in 99% of cases We want you to know as soon as possible that Packit is working on your request. ![SLO 1](img/slo/slo1.png) -## SLO2: 98% of builds have status set to success or failure within 5 minutes after the Copr build has finished +### SLO2: 98% of builds have status set to success or failure within 5 minutes after the Copr build has finished Once Packit works on a build, you should see that the work is done within a reasonable time. ![SLO 2](img/slo/slo2.png) - -## SLO3: 95% of test runs have status set to success or failure within 12 hours +### SLO3: 95% of test runs have status set to success or failure within 12 hours The same as SLO2 except this is for tests. Testing Farm is a more complex system so we reduced the number here. ![SLO 3](img/slo/slo3.png) -If you are interested, here's the accompanying research: -[github.com/packit/research/../error-budgets](https://github.com/packit/research/tree/main/research/monitoring/error-budgets) +## Indicators + +To know how we are doing against the objectives, we measure these indicators: + +| Objective | Indicators | +| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| SLO1 | Time it takes Packit to set the first status on a PR or commit after receiving the event. Number of PRs or commits with no status from Packit after the time limit. | +| SLO2 | Number of Copr builds queued, started and finished. Time from a Copr build finishing to Packit reporting the final status. Time from accepting a Copr build to it finishing. | +| SLO3 | Number of test runs queued, started and finished. Time from submitting a test run to Packit setting its final status. | + +The counters tell us how many jobs entered and left the pipeline. If jobs are +queued or started but never finish, it shows up as a gap between the numbers. +The durations tell us how long our users wait. + +### How we measure + +All the other teams around us use the Prometheus + Grafana combo, so this is +our choice as well. Packit pushes the metrics to a Prometheus Pushgateway, and +the objectives are visualised in Grafana. + +Durations are recorded as Prometheus +[histograms](https://prometheus.io/docs/concepts/metric_types/#histogram). The +histogram buckets are aligned with the SLO thresholds (for example 15 seconds +for SLO1 and 12 hours for SLO3), so we can directly compute +[what percentage of requests](https://prometheus.io/docs/practices/histograms/) +fell under the threshold, instead of relying on averages. Unfortunately, we cannot share our Grafana instance with you since it's hidden behind a firewall. + +## Why these objectives and indicators + +Before defining the objectives, we asked the most active users of Packit +(projects such as anaconda, cockpit, systemd and others) what they expect from +the service in terms of latency, reliability and availability. The objectives +are derived from their answers. + +### SLO1: a quick first status + +The most consistent request was a quick "accepted" status: an indication that +Packit has seen the change and will deal with it. Users were fine with waiting +minutes or even hours for the results, as long as they knew the request had been +picked up. Their expectations for this status ranged from 5 seconds to a minute. +No status at all is frustrating, both for our users and for us, because it's +impossible to tell whether Packit is working or has dropped the event. Hence the +first objective is about the first status only, with a strict time limit and a +high percentage. + +### SLO2: builds end with a real answer + +Copr builds for pull requests are Packit's core functionality. What matters to +users is that a build ends with a real answer (success or failure) and is not +interrupted by an infrastructure error. Users told us an error rate of about 5 +to 10% is something they would tolerate at the beginning. + +Build duration is not something we control: some builds take minutes, others +take hours. An objective defined from the moment we accept the build would +therefore depend mostly on the project being built. That's why SLO2 measures +the part Packit is responsible for: reporting the final result once Copr has +finished the build. + +### SLO3: tests with a lower target + +Tests are similar to builds, but they run in Testing Farm, which is outside of +our control and is a more complex system (provisioning machines, running tests, +collecting logs). That's why the percentage is lower and the time limit covers +the whole test run. + +### Indicators follow the objectives + +We did not start by collecting all the metrics we could get. We picked +indicators that answer the question "are we meeting the objective?" directly: +how long it takes to set a status, and how many jobs entered and left each stage. +This keeps the dashboards small and the alerts meaningful. + +## Guidance for other teams + +If you are defining SLOs for your own service, this is what worked for us: + +- **Ask your users first.** Talk to the most frequent users of your service + and ask about latency, error rate, throughput and availability. Their answers + are more valuable than the current numbers in your dashboards. +- **Start small.** Some teams (for example + [CKI](https://cki-project.org/docs/hacking/rfcs/cki-004-slo/#created-slos-based-on-the-feedback)) + have a comprehensive set of SLOs. Instead of taking the same route, we started + with three objectives and add more objectives and indicators over time. +- **Measure what you control.** If a part of the workflow is owned by + another service (Copr builds, Testing Farm), either measure only your part or + set a lower target. +- **Use distributions, not averages.** "99% of requests within 15 seconds" says + much more than "15 seconds on average", which can hide problems in the tail. +- **Avoid 100%.** Insisting on perfection slows down development. The error + budget gives you room to ship new features while keeping the service + reliable. +- **Revisit the targets.** Loose targets are fine at the beginning. Tighten + them, or drop objectives nobody talks about, as you learn more. + +## Background + +This page supersedes the original research on error budgets, which is kept +only for historical reasons: +[github.com/packit/research/../error-budgets](https://github.com/packit/research/tree/main/research/monitoring/error-budgets). +It contains the full notes from Google's SRE book and the collected +[expectations of our users](https://github.com/packit/research/blob/main/research/monitoring/error-budgets/users-expectations.md). From 17df112036d3d79c7d2fac7da41026e74ff94075 Mon Sep 17 00:00:00 2001 From: fullsend-fix <278716306+fullsend-ai-coder[bot]@users.noreply.github.com> Date: Mon, 26 Oct 2026 08:33:20 +0000 Subject: [PATCH 2/6] docs(#1094): address review feedback on PR #1184 - State the real 25-second limit of the SLO1 "no status" counter in the indicators table and note it is looser than the 15-second SLO1 limit. - Capitalize the first word after the colon in the rationale subheadings to match the Objectives headings. Note: pre-commit could not fetch hook repositories (TLS verification failure in the sandbox). check-merge-conflict and detect-private-key (pre-commit-hooks v5.0.0) were run directly and passed; the prettier hook excludes docs/. Addresses #1184 Co-Authored-By: Claude Opus 5.5 --- docs/service-level-objectives.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/service-level-objectives.md b/docs/service-level-objectives.md index 5172c9c6ef..68746fa219 100644 --- a/docs/service-level-objectives.md +++ b/docs/service-level-objectives.md @@ -65,11 +65,11 @@ system so we reduced the number here. To know how we are doing against the objectives, we measure these indicators: -| Objective | Indicators | -| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| SLO1 | Time it takes Packit to set the first status on a PR or commit after receiving the event. Number of PRs or commits with no status from Packit after the time limit. | -| SLO2 | Number of Copr builds queued, started and finished. Time from a Copr build finishing to Packit reporting the final status. Time from accepting a Copr build to it finishing. | -| SLO3 | Number of test runs queued, started and finished. Time from submitting a test run to Packit setting its final status. | +| Objective | Indicators | +| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| SLO1 | Time it takes Packit to set the first status on a PR or commit after receiving the event. Number of PRs or commits with no status from Packit after 25 seconds (a looser limit than the 15 seconds of SLO1). | +| SLO2 | Number of Copr builds queued, started and finished. Time from a Copr build finishing to Packit reporting the final status. Time from accepting a Copr build to it finishing. | +| SLO3 | Number of test runs queued, started and finished. Time from submitting a test run to Packit setting its final status. | The counters tell us how many jobs entered and left the pipeline. If jobs are queued or started but never finish, it shows up as a gap between the numbers. @@ -98,7 +98,7 @@ Before defining the objectives, we asked the most active users of Packit the service in terms of latency, reliability and availability. The objectives are derived from their answers. -### SLO1: a quick first status +### SLO1: A quick first status The most consistent request was a quick "accepted" status: an indication that Packit has seen the change and will deal with it. Users were fine with waiting @@ -109,7 +109,7 @@ impossible to tell whether Packit is working or has dropped the event. Hence the first objective is about the first status only, with a strict time limit and a high percentage. -### SLO2: builds end with a real answer +### SLO2: Builds end with a real answer Copr builds for pull requests are Packit's core functionality. What matters to users is that a build ends with a real answer (success or failure) and is not @@ -122,7 +122,7 @@ therefore depend mostly on the project being built. That's why SLO2 measures the part Packit is responsible for: reporting the final result once Copr has finished the build. -### SLO3: tests with a lower target +### SLO3: Tests with a lower target Tests are similar to builds, but they run in Testing Farm, which is outside of our control and is a more complex system (provisioning machines, running tests, From 2bbfccb8f84878d5a505a5aa7e8a8d2188de878c Mon Sep 17 00:00:00 2001 From: fullsend-fix <278716306+fullsend-ai-coder[bot]@users.noreply.github.com> Date: Mon, 26 Oct 2026 08:49:38 +0000 Subject: [PATCH 3/6] docs(#1094): clarify what the SLO indicators measure Address the second round of review feedback on PR #1184: - Describe the SLO1 counter accurately: it counts PRs or commits whose first status took longer than 25 seconds, and neither SLO1 indicator sees events that never get a status. - Say that the histograms give the share of jobs that finished within the time threshold, whatever the outcome, so they approximate SLO2 and SLO3. Explain how error outcomes are handled for test runs (counted) and for builds (missing from the histogram). Verification: pre-commit could not fetch its hook repositories (TLS verification failure in the sandbox). The prettier hook excludes docs/. check-merge-conflict and detect-private-key were checked with equivalent pattern searches and passed. The page compiles with @mdx-js/mdx 1.6.22. Addresses #1184 Co-Authored-By: Claude Opus 5.5 --- docs/service-level-objectives.md | 33 +++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/docs/service-level-objectives.md b/docs/service-level-objectives.md index 68746fa219..4daf4247b6 100644 --- a/docs/service-level-objectives.md +++ b/docs/service-level-objectives.md @@ -65,16 +65,20 @@ system so we reduced the number here. To know how we are doing against the objectives, we measure these indicators: -| Objective | Indicators | -| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| SLO1 | Time it takes Packit to set the first status on a PR or commit after receiving the event. Number of PRs or commits with no status from Packit after 25 seconds (a looser limit than the 15 seconds of SLO1). | -| SLO2 | Number of Copr builds queued, started and finished. Time from a Copr build finishing to Packit reporting the final status. Time from accepting a Copr build to it finishing. | -| SLO3 | Number of test runs queued, started and finished. Time from submitting a test run to Packit setting its final status. | +| Objective | Indicators | +| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| SLO1 | Time it takes Packit to set the first status on a PR or commit after receiving the event. Number of PRs or commits whose first status took longer than 25 seconds (a looser limit than the 15 seconds of SLO1). | +| SLO2 | Number of Copr builds queued, started and finished. Time from a Copr build finishing to Packit reporting the final status. Time from accepting a Copr build to it finishing. | +| SLO3 | Number of test runs queued, started and finished. Time from submitting a test run to Packit setting its final status. | The counters tell us how many jobs entered and left the pipeline. If jobs are queued or started but never finish, it shows up as a gap between the numbers. The durations tell us how long our users wait. +Both SLO1 indicators are recorded only when Packit sets a first status. An event +that Packit drops completely, without setting any status, does not show up in +either of them. + ### How we measure All the other teams around us use the Prometheus + Grafana combo, so this is @@ -83,10 +87,21 @@ the objectives are visualised in Grafana. Durations are recorded as Prometheus [histograms](https://prometheus.io/docs/concepts/metric_types/#histogram). The -histogram buckets are aligned with the SLO thresholds (for example 15 seconds -for SLO1 and 12 hours for SLO3), so we can directly compute -[what percentage of requests](https://prometheus.io/docs/practices/histograms/) -fell under the threshold, instead of relying on averages. +histogram buckets are aligned with the SLO time thresholds (for example 15 +seconds for SLO1 and 12 hours for SLO3), so we can directly compute +[what percentage of the measured jobs](https://prometheus.io/docs/practices/histograms/) +finished within the threshold, instead of relying on averages. + +The histograms record only the time, whatever the outcome of the job. This means +the current indicators approximate SLO2 and SLO3 rather than compute them +exactly: + +- The time to the final test status is recorded for every finished test run, + including runs that ended with an error or were canceled. A Testing Farm + error reported within 12 hours therefore counts as within the threshold. +- The time to the final build status is recorded only when Packit reports + success or failure. A build that ends with an error on Packit's side is + missing from the histogram completely. Unfortunately, we cannot share our Grafana instance with you since it's hidden behind a firewall. From f9993fdf563a5b8fe932e2dfd81bc71896897667 Mon Sep 17 00:00:00 2001 From: fullsend-fix <278716306+fullsend-ai-coder[bot]@users.noreply.github.com> Date: Mon, 26 Oct 2026 09:05:50 +0000 Subject: [PATCH 4/6] docs(#1094): document SLO2 measurement gaps and SLO1 history Address the third round of review feedback on PR #1184: - List the remaining ways the SLO2 histogram differs from the objective: Packit reports every unsuccessful Copr build as a failure, including Copr infrastructure failures, so those errors are measured as failures. Failed builds are not reported or measured when the SRPM build already failed or a newer build for the same target exists. - Add a note on SLO1 history: the limit was raised to 30 seconds for all statuses in late 2022 (packit-service#1776), and in March 2023 (commit 76caf5b) SLO1 was changed to the first status within 15 seconds. The published threshold is unchanged; maintainers should still confirm it matches current monitoring. Verification: pre-commit could not fetch its hook repositories (TLS verification failure in the sandbox), and the hook executables are not installed. The prettier hook excludes docs/. check-merge-conflict and detect-private-key were checked with equivalent pattern searches and passed. The page compiles with @mdx-js/mdx 1.6.22. Addresses #1184 Co-Authored-By: Claude Opus 5.5 --- docs/service-level-objectives.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/service-level-objectives.md b/docs/service-level-objectives.md index 4daf4247b6..4f3a195f67 100644 --- a/docs/service-level-objectives.md +++ b/docs/service-level-objectives.md @@ -102,6 +102,12 @@ exactly: - The time to the final build status is recorded only when Packit reports success or failure. A build that ends with an error on Packit's side is missing from the histogram completely. +- Packit reports every unsuccessful Copr build as a failure, including builds + that failed because of a Copr infrastructure problem. Such builds are + measured as failures, even though they are errors by the definitions above. +- A failed build is not reported, and so not measured, when its SRPM build has + already failed (Packit has reported that failure already) or when a newer + build for the same target exists. Unfortunately, we cannot share our Grafana instance with you since it's hidden behind a firewall. @@ -124,6 +130,11 @@ impossible to tell whether Packit is working or has dropped the event. Hence the first objective is about the first status only, with a strict time limit and a high percentage. +In late 2022, the SLO1 limit was raised from 15 to 30 seconds to cover setting +all statuses ([packit-service#1776](https://github.com/packit/packit-service/pull/1776)). +In March 2023, SLO1 was changed to the current definition: the first status +within 15 seconds. + ### SLO2: Builds end with a real answer Copr builds for pull requests are Packit's core functionality. What matters to From ed21b1a0f0a5b6b4becb34ee2287b5be1944fd92 Mon Sep 17 00:00:00 2001 From: fullsend-fix <278716306+fullsend-ai-coder[bot]@users.noreply.github.com> Date: Mon, 26 Oct 2026 12:54:48 +0000 Subject: [PATCH 5/6] docs(#1094): split detailed SLO content into a separate page Follow the /fs-fix instruction on PR #1184: - Restore docs/service-level-objectives.md to the simple page for non-developer users. It is now the same as on main, except the link to the error-budgets research is replaced by a link to the new details page. - Add docs/service-level-objectives-details.md ("How our SLOs are built") for readers who want to know more. It covers job outcomes, the SLIs, how we measure them, why we picked them, guidance for other teams, and background. It names the Grafana dashboards that show the SLIs: Packit and Fedora CI. SLI screenshots were not added. The sandbox cannot download the example image from the PR comment, and Grafana sits behind a firewall. Screenshots from the Packit and Fedora CI dashboards need to be added by someone with access. Verification: pre-commit could not fetch its hook repositories (TLS verification failure). check-merge-conflict and detect-private-key from pre-commit-hooks v5.0.0 were run directly and passed. The prettier hook excludes docs/. Both pages compile with @mdx-js/mdx 1.6.22. Addresses #1184 Co-Authored-By: Claude Opus 5.5 --- docs/service-level-objectives-details.md | 156 ++++++++++++++++++++++ docs/service-level-objectives.md | 157 +---------------------- 2 files changed, 163 insertions(+), 150 deletions(-) create mode 100644 docs/service-level-objectives-details.md diff --git a/docs/service-level-objectives-details.md b/docs/service-level-objectives-details.md new file mode 100644 index 0000000000..d5491ea8a5 --- /dev/null +++ b/docs/service-level-objectives-details.md @@ -0,0 +1,156 @@ +--- +title: How our SLOs are built +sidebar_position: 12 +--- + +# How our SLOs are built + +This page is for those who want to know more about Packit's +[Service Level Objectives](./service-level-objectives.md) (SLOs): the Service +Level Indicators (SLIs) we use to track them, how we measure them and why we +picked them. Other teams can use it as a guide when defining their own SLOs. + +## Job outcomes + +The objectives talk about jobs (builds and tests) ending with a status. To make +them unambiguous, we distinguish these outcomes: + +- **success**: the job finished and everything is awesome +- **failure**: the job finished and wasn't successful: the PR needs to be fixed +- **error**: infrastructure problems prevented the job from completing + +Only _success_ and _failure_ count as good outcomes for our objectives. Both +give you a real answer about your change. An _error_ means Packit (or a service +it depends on) let you down. + +## Indicators + +To know how we are doing against the objectives, we measure these indicators: + +| Objective | Indicators | +| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| SLO1 | Time it takes Packit to set the first status on a PR or commit after receiving the event. Number of PRs or commits whose first status took longer than 25 seconds (a looser limit than the 15 seconds of SLO1). | +| SLO2 | Number of Copr builds queued, started and finished. Time from a Copr build finishing to Packit reporting the final status. Time from accepting a Copr build to it finishing. | +| SLO3 | Number of test runs queued, started and finished. Time from submitting a test run to Packit setting its final status. | + +The counters tell us how many jobs entered and left the pipeline. If jobs are +queued or started but never finish, it shows up as a gap between the numbers. +The durations tell us how long our users wait. + +Both SLO1 indicators are recorded only when Packit sets a first status. An event +that Packit drops completely, without setting any status, does not show up in +either of them. + +## How we measure + +All the other teams around us use the Prometheus + Grafana combo, so this is +our choice as well. Packit pushes the metrics to a Prometheus Pushgateway, and +the indicators are visualised in two Grafana dashboards: **Packit** and +**Fedora CI**. + +Durations are recorded as Prometheus +[histograms](https://prometheus.io/docs/concepts/metric_types/#histogram). The +histogram buckets are aligned with the SLO time thresholds (for example 15 +seconds for SLO1 and 12 hours for SLO3), so we can directly compute +[what percentage of the measured jobs](https://prometheus.io/docs/practices/histograms/) +finished within the threshold, instead of relying on averages. + +The histograms record only the time, whatever the outcome of the job. This means +the current indicators approximate SLO2 and SLO3 rather than compute them +exactly: + +- The time to the final test status is recorded for every finished test run, + including runs that ended with an error or were canceled. A Testing Farm + error reported within 12 hours therefore counts as within the threshold. +- The time to the final build status is recorded only when Packit reports + success or failure. A build that ends with an error on Packit's side is + missing from the histogram completely. +- Packit reports every unsuccessful Copr build as a failure, including builds + that failed because of a Copr infrastructure problem. Such builds are + measured as failures, even though they are errors by the definitions above. +- A failed build is not reported, and so not measured, when its SRPM build has + already failed (Packit has reported that failure already) or when a newer + build for the same target exists. + +Unfortunately, we cannot share our Grafana instance with you since it's hidden +behind a firewall. + +## Why these objectives and indicators + +Before defining the objectives, we asked the most active users of Packit +(projects such as anaconda, cockpit, systemd and others) what they expect from +the service in terms of latency, reliability and availability. The objectives +are derived from their answers. + +### SLO1: A quick first status + +The most consistent request was a quick "accepted" status: an indication that +Packit has seen the change and will deal with it. Users were fine with waiting +minutes or even hours for the results, as long as they knew the request had been +picked up. Their expectations for this status ranged from 5 seconds to a minute. +No status at all is frustrating, both for our users and for us, because it's +impossible to tell whether Packit is working or has dropped the event. Hence the +first objective is about the first status only, with a strict time limit and a +high percentage. + +In late 2022, the SLO1 limit was raised from 15 to 30 seconds to cover setting +all statuses ([packit-service#1776](https://github.com/packit/packit-service/pull/1776)). +In March 2023, SLO1 was changed to the current definition: the first status +within 15 seconds. + +### SLO2: Builds end with a real answer + +Copr builds for pull requests are Packit's core functionality. What matters to +users is that a build ends with a real answer (success or failure) and is not +interrupted by an infrastructure error. Users told us an error rate of about 5 +to 10% is something they would tolerate at the beginning. + +Build duration is not something we control: some builds take minutes, others +take hours. An objective defined from the moment we accept the build would +therefore depend mostly on the project being built. That's why SLO2 measures +the part Packit is responsible for: reporting the final result once Copr has +finished the build. + +### SLO3: Tests with a lower target + +Tests are similar to builds, but they run in Testing Farm, which is outside of +our control and is a more complex system (provisioning machines, running tests, +collecting logs). That's why the percentage is lower and the time limit covers +the whole test run. + +### Indicators follow the objectives + +We did not start by collecting all the metrics we could get. We picked +indicators that answer the question "are we meeting the objective?" directly: +how long it takes to set a status, and how many jobs entered and left each stage. +This keeps the dashboards small and the alerts meaningful. + +## Guidance for other teams + +If you are defining SLOs for your own service, this is what worked for us: + +- **Ask your users first.** Talk to the most frequent users of your service + and ask about latency, error rate, throughput and availability. Their answers + are more valuable than the current numbers in your dashboards. +- **Start small.** Some teams (for example + [CKI](https://cki-project.org/docs/hacking/rfcs/cki-004-slo/#created-slos-based-on-the-feedback)) + have a comprehensive set of SLOs. Instead of taking the same route, we started + with three objectives and add more objectives and indicators over time. +- **Measure what you control.** If a part of the workflow is owned by + another service (Copr builds, Testing Farm), either measure only your part or + set a lower target. +- **Use distributions, not averages.** "99% of requests within 15 seconds" says + much more than "15 seconds on average", which can hide problems in the tail. +- **Avoid 100%.** Insisting on perfection slows down development. The error + budget gives you room to ship new features while keeping the service + reliable. +- **Revisit the targets.** Loose targets are fine at the beginning. Tighten + them, or drop objectives nobody talks about, as you learn more. + +## Background + +This page supersedes the original research on error budgets, which is kept +only for historical reasons: +[github.com/packit/research/../error-budgets](https://github.com/packit/research/tree/main/research/monitoring/error-budgets). +It contains the full notes from Google's SRE book and the collected +[expectations of our users](https://github.com/packit/research/blob/main/research/monitoring/error-budgets/users-expectations.md). diff --git a/docs/service-level-objectives.md b/docs/service-level-objectives.md index 4f3a195f67..3b94d195ae 100644 --- a/docs/service-level-objectives.md +++ b/docs/service-level-objectives.md @@ -11,26 +11,6 @@ objectives](https://sre.google/sre-book/service-level-objectives/) defined in Google's SRE book. It means that when we run out of error budget, we stop all the work on new features and start stabilizing the service to get our budget back. -This page is the official definition of Packit's Service Level Objectives -(SLOs) and the Service Level Indicators (SLIs) we use to track them. It also -explains why we picked them, so that other teams can use it as a guide when -defining their own SLOs. - -## Job outcomes - -The objectives below talk about jobs (builds and tests) ending with a status. -To make them unambiguous, we distinguish these outcomes: - -- **success**: the job finished and everything is awesome -- **failure**: the job finished and wasn't successful: the PR needs to be fixed -- **error**: infrastructure problems prevented the job from completing - -Only _success_ and _failure_ count as good outcomes for our objectives. Both -give you a real answer about your change. An _error_ means Packit (or a service -it depends on) let you down. - -## Objectives - We have 3 objectives defined for Packit right now. :::caution @@ -41,153 +21,30 @@ actual numbers. ::: -### SLO1: Changes to GitHub PRs receive the first status update within 15 seconds in 99% of cases +## SLO1: Changes to GitHub PRs receive the first status update within 15 seconds in 99% of cases We want you to know as soon as possible that Packit is working on your request. ![SLO 1](img/slo/slo1.png) -### SLO2: 98% of builds have status set to success or failure within 5 minutes after the Copr build has finished +## SLO2: 98% of builds have status set to success or failure within 5 minutes after the Copr build has finished Once Packit works on a build, you should see that the work is done within a reasonable time. ![SLO 2](img/slo/slo2.png) -### SLO3: 95% of test runs have status set to success or failure within 12 hours + +## SLO3: 95% of test runs have status set to success or failure within 12 hours The same as SLO2 except this is for tests. Testing Farm is a more complex system so we reduced the number here. ![SLO 3](img/slo/slo3.png) -## Indicators - -To know how we are doing against the objectives, we measure these indicators: - -| Objective | Indicators | -| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| SLO1 | Time it takes Packit to set the first status on a PR or commit after receiving the event. Number of PRs or commits whose first status took longer than 25 seconds (a looser limit than the 15 seconds of SLO1). | -| SLO2 | Number of Copr builds queued, started and finished. Time from a Copr build finishing to Packit reporting the final status. Time from accepting a Copr build to it finishing. | -| SLO3 | Number of test runs queued, started and finished. Time from submitting a test run to Packit setting its final status. | - -The counters tell us how many jobs entered and left the pipeline. If jobs are -queued or started but never finish, it shows up as a gap between the numbers. -The durations tell us how long our users wait. - -Both SLO1 indicators are recorded only when Packit sets a first status. An event -that Packit drops completely, without setting any status, does not show up in -either of them. - -### How we measure - -All the other teams around us use the Prometheus + Grafana combo, so this is -our choice as well. Packit pushes the metrics to a Prometheus Pushgateway, and -the objectives are visualised in Grafana. - -Durations are recorded as Prometheus -[histograms](https://prometheus.io/docs/concepts/metric_types/#histogram). The -histogram buckets are aligned with the SLO time thresholds (for example 15 -seconds for SLO1 and 12 hours for SLO3), so we can directly compute -[what percentage of the measured jobs](https://prometheus.io/docs/practices/histograms/) -finished within the threshold, instead of relying on averages. - -The histograms record only the time, whatever the outcome of the job. This means -the current indicators approximate SLO2 and SLO3 rather than compute them -exactly: - -- The time to the final test status is recorded for every finished test run, - including runs that ended with an error or were canceled. A Testing Farm - error reported within 12 hours therefore counts as within the threshold. -- The time to the final build status is recorded only when Packit reports - success or failure. A build that ends with an error on Packit's side is - missing from the histogram completely. -- Packit reports every unsuccessful Copr build as a failure, including builds - that failed because of a Copr infrastructure problem. Such builds are - measured as failures, even though they are errors by the definitions above. -- A failed build is not reported, and so not measured, when its SRPM build has - already failed (Packit has reported that failure already) or when a newer - build for the same target exists. +If you want to know more about how these objectives are built, how we measure +them and why we picked them, see +[How our SLOs are built](./service-level-objectives-details.md). Unfortunately, we cannot share our Grafana instance with you since it's hidden behind a firewall. - -## Why these objectives and indicators - -Before defining the objectives, we asked the most active users of Packit -(projects such as anaconda, cockpit, systemd and others) what they expect from -the service in terms of latency, reliability and availability. The objectives -are derived from their answers. - -### SLO1: A quick first status - -The most consistent request was a quick "accepted" status: an indication that -Packit has seen the change and will deal with it. Users were fine with waiting -minutes or even hours for the results, as long as they knew the request had been -picked up. Their expectations for this status ranged from 5 seconds to a minute. -No status at all is frustrating, both for our users and for us, because it's -impossible to tell whether Packit is working or has dropped the event. Hence the -first objective is about the first status only, with a strict time limit and a -high percentage. - -In late 2022, the SLO1 limit was raised from 15 to 30 seconds to cover setting -all statuses ([packit-service#1776](https://github.com/packit/packit-service/pull/1776)). -In March 2023, SLO1 was changed to the current definition: the first status -within 15 seconds. - -### SLO2: Builds end with a real answer - -Copr builds for pull requests are Packit's core functionality. What matters to -users is that a build ends with a real answer (success or failure) and is not -interrupted by an infrastructure error. Users told us an error rate of about 5 -to 10% is something they would tolerate at the beginning. - -Build duration is not something we control: some builds take minutes, others -take hours. An objective defined from the moment we accept the build would -therefore depend mostly on the project being built. That's why SLO2 measures -the part Packit is responsible for: reporting the final result once Copr has -finished the build. - -### SLO3: Tests with a lower target - -Tests are similar to builds, but they run in Testing Farm, which is outside of -our control and is a more complex system (provisioning machines, running tests, -collecting logs). That's why the percentage is lower and the time limit covers -the whole test run. - -### Indicators follow the objectives - -We did not start by collecting all the metrics we could get. We picked -indicators that answer the question "are we meeting the objective?" directly: -how long it takes to set a status, and how many jobs entered and left each stage. -This keeps the dashboards small and the alerts meaningful. - -## Guidance for other teams - -If you are defining SLOs for your own service, this is what worked for us: - -- **Ask your users first.** Talk to the most frequent users of your service - and ask about latency, error rate, throughput and availability. Their answers - are more valuable than the current numbers in your dashboards. -- **Start small.** Some teams (for example - [CKI](https://cki-project.org/docs/hacking/rfcs/cki-004-slo/#created-slos-based-on-the-feedback)) - have a comprehensive set of SLOs. Instead of taking the same route, we started - with three objectives and add more objectives and indicators over time. -- **Measure what you control.** If a part of the workflow is owned by - another service (Copr builds, Testing Farm), either measure only your part or - set a lower target. -- **Use distributions, not averages.** "99% of requests within 15 seconds" says - much more than "15 seconds on average", which can hide problems in the tail. -- **Avoid 100%.** Insisting on perfection slows down development. The error - budget gives you room to ship new features while keeping the service - reliable. -- **Revisit the targets.** Loose targets are fine at the beginning. Tighten - them, or drop objectives nobody talks about, as you learn more. - -## Background - -This page supersedes the original research on error budgets, which is kept -only for historical reasons: -[github.com/packit/research/../error-budgets](https://github.com/packit/research/tree/main/research/monitoring/error-budgets). -It contains the full notes from Google's SRE book and the collected -[expectations of our users](https://github.com/packit/research/blob/main/research/monitoring/error-budgets/users-expectations.md). From dcdc396fd646b82f0299a2404215722b18d71622 Mon Sep 17 00:00:00 2001 From: fullsend-fix <278716306+fullsend-ai-coder[bot]@users.noreply.github.com> Date: Mon, 26 Oct 2026 13:10:31 +0000 Subject: [PATCH 6/6] docs(#1094): order the SLO details page after the overview Address the fourth round of review feedback on PR #1184: - Set sidebar_position to 12.5 on the details page. Before, it shared position 12 with service-level-objectives.md (and reproduce-locally.md), so the autogenerated sidebar could put the details page above the overview. Docusaurus 2.4.1 validates sidebar_position as a plain number and sorts by it, so fractional values work. Not changed in this commit (they need a human or another repository): the PR description still describes the first single-page version, the packit/research follow-up has no tracking issue, and the Grafana screenshots are still missing. Verification: pre-commit could not fetch its hook repositories (TLS verification failure). check-merge-conflict and detect-private-key from pre-commit-hooks v5.0.0 were run directly and passed. The prettier hook excludes docs/. The repository has no test suite, and the full site build needs network access for `make import`. Addresses #1184 Co-Authored-By: Claude Opus 5.5 --- docs/service-level-objectives-details.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/service-level-objectives-details.md b/docs/service-level-objectives-details.md index d5491ea8a5..eb6db7ca4a 100644 --- a/docs/service-level-objectives-details.md +++ b/docs/service-level-objectives-details.md @@ -1,6 +1,6 @@ --- title: How our SLOs are built -sidebar_position: 12 +sidebar_position: 12.5 --- # How our SLOs are built