diff --git a/.cspell.yml b/.cspell.yml index 9da23446..997f8c89 100644 --- a/.cspell.yml +++ b/.cspell.yml @@ -46,6 +46,7 @@ words: - Uchechukwu - webdev - Welsch + - llms - KubeVirt - kubevirt - Quickstart diff --git a/docs/analysis/criteria.md b/docs/analysis/criteria.md index 9045bb01..56e9871b 100644 --- a/docs/analysis/criteria.md +++ b/docs/analysis/criteria.md @@ -2,8 +2,6 @@ title: Assessment criteria and examples --- -# Assessment criteria and examples - We recommend reading the [CNCF sandbox project documentation primer](../sandbox-doc-primer.md) before reviewing these criteria. The primer contains key definitions concepts that are @@ -15,7 +13,7 @@ referred to in the criteria. How your technical information is organized, laid out, and presented. This includes structure (pages/subpages/sections/subsections), information types -(conceptual/instructional/reference), and matching documentation content to user +(conceptual/instructional/reference), and content that matches user expectations. We evaluate on the following: @@ -23,7 +21,8 @@ We evaluate on the following: - Is there high level conceptual content? - Is every product feature documented? - Does the documentation define all user roles (personas) for the product? -- Are there instructions (tasks, tutorials) documented for features? +- Do instructions (tasks, tutorials) cover the intended purposes of all + user-facing features? - Are there instructions for all major use cases for each user role? - Are tasks organized by user role and use case? - Does instructional content demonstrate atomicity — are individual tasks @@ -31,8 +30,8 @@ We evaluate on the following: - Are tasks written as numbered step-by-step instructions? - Do task descriptions in headings and the TOC describe the task with a verb phrase? -- Is the documentation free of any key features which are documented but missing - task documentation? +- Are there any features that are documented (for example, described + conceptually) but are missing task documentation? - Is the “happy path” — the most common use case — documented? - If the documentation does not suffice, is there a clear escalation path for users needing more help? @@ -41,6 +40,11 @@ We evaluate on the following: - Is content up to date and accurate? - Does the documentation separate conceptual, instructional, and reference information? +- If the product builds on or incorporates another product (such as Kubernetes), + are there external references where needed to install and configure those + products? +- Does the documentation organization and navigation need a high-level + restructuring? Example: @@ -60,7 +64,8 @@ We evaluate on the following: - Are different types of installation documented (development, test, production) if necessary? - If needed, are multiple OSes documented? -- Do users know where to go after reading the getting started guide? +- Is there guidance for what to read next after reading the getting started + guide? - Is your new user content clearly signposted on your site’s homepage or at the top of your information architecture? - Is there easily copy-pastable sample code or other example content? @@ -150,10 +155,14 @@ Example: We evaluate on the following: -- Are docs issues well-triaged? +- Are docs issues triaged with respect to effort level and required expertise? - Is there a clearly marked way for new contributors to make code or documentation contributions (i.e. a “good first issue” label)? -- Are issues well-documented (i.e., more than just a title)? +- Are issues documented with: + - What is missing, incorrect, or improperly presented? + - What impact the issue has on documentation users? + - A suggested remedy? + - An estimate of the effort required to address the issue? - Are issues maintained for staleness? Example: @@ -172,7 +181,7 @@ We evaluate on the following: - Do you have a community repository or section on your website? - Is there a document specifically welcoming new contributors and documenting a first contribution process? -- Can new users find where to get help? +- Are there prominently displayed resources for new users to find help? Example: @@ -184,7 +193,7 @@ One of the CNCF’s core project values is open governance. We evaluate on the following: -- Is project governance clearly documented? +- Does the project governance align with CNCF guidelines and values? Example: @@ -239,8 +248,8 @@ requirements for sandbox projects._ Incubating status. - Is there rudimentary **project documentation**, or a placeholder or substitute? - - It is acceptable at this maturity level to link out to documentation that - hasn't yet been integrated into the website. + - It is acceptable at this maturity level to link to documentation that hasn't + yet been integrated into the website. - _Example_: website with a single homepage, without any documentation or, as was mentioned above, linking out to an external (preexisting) source for docs. @@ -259,14 +268,14 @@ requirements for sandbox projects._ - Does **project documentation** meet these standards?: - Stakeholders / personas are identified and their core needs (in terms of docs) documented. - - Hosted directly on the website. + - Hosted, internally or externally, on the website. - The [single-source requirement][] met. - Doc is comprehensive, addressing most stakeholder needs. #### Graduated - Are follow-through actions from the [Docs assessment][] complete? -- Does **project documentation** fully addresses the needs of key stakeholders? +- Does **project documentation** fully addresses the needs of stakeholders? #### Archived @@ -274,7 +283,7 @@ requirements for sandbox projects._ - Is the archived status of the project obvious to those visiting the website, such as through the use of a prominent banner? - If a successor project exists, are there links to its website and/or migration - documentation. + documentation? [archived state]: https://docs.github.com/en/repositories/archiving-a-github-repository/archiving-repositories @@ -295,12 +304,8 @@ We evaluate on the following: - Is the website usable from mobile? - Are doc pages readable? -- Are all or most website features accessible from mobile -- such as the - top-nav, site search, and in-page table of contents? - Might a [mobile-first][] design make sense for your project? -- Are color contrasts significant enough for color-impaired readers? - Are most website features usable using a keyboard only? -- Does text-to-speech offer listeners a good experience? [mobile-first]: https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Responsive/Mobile_first @@ -310,8 +315,8 @@ Plan for suitable [accessibility][] measures for your website. We evaluate on the following: - Are color contrasts significant enough for color-impaired readers? -- Are most website features usable using a keyboard only? -- Does text-to-speech offer listeners a good experience? +- Are all vital website features usable using a keyboard only? +- Does text-to-speech offer listeners a usable experience? It is up to each project to set their own guidelines. @@ -333,7 +338,7 @@ Example: - https://helm.sh/ -### Case studies/social proof +### Case studies and social proof One of the best ways to advertise an open source project is to show other organizations using it. @@ -367,7 +372,7 @@ We evaluate on the following: - Is analytics disabled for all other deploys? - If your project used Google Analytics, have you migrated to GA4? - Can Page-not-found (404) reports easily be generated from you site - analytics? Provide a sample of the site's current top-10 404s. + analytics? - Is site indexing supported for the production server, while disabled for website previews and builds for non-default branches? - Is local intra-site search available from the website? @@ -391,6 +396,23 @@ Example: - https://kubernetes.io +### AI Optimization and Discoverability + +Tools and techniques are emerging that enable AI agents to use documentation as +input. For example, many repositories now include "AI-friendly" Markdown files +that enable LLMs to ingest content. We encourage projects to optimize for +AI-based documentation use. + +We evaluate on the following: + +- Is there an llms.txt file in the root of the website or documentation + repository? + + The leading proposal for an AI-enabling file for websites is at + [llms-txt](https://llmstxt.org/). + + Task can be accomplished manually or by tools. + ### Other - Is your website accessible via HTTPS? diff --git a/docs/analysis/howto.md b/docs/analysis/howto.md index 19a71825..1a00c5ec 100644 --- a/docs/analysis/howto.md +++ b/docs/analysis/howto.md @@ -218,7 +218,7 @@ Create issues in the project documentation GitHub repository for: - An umbrella issue that provides a context for the previously created individual issues. -[analyses]: ../../analyses/ +[analyses]: ../../analyses [criteria]: ./criteria.md [project maturity level]: https://www.cncf.io/project-metrics [templates]: ./templates/index.md diff --git a/docs/analysis/templates/analysis.md b/docs/analysis/templates/analysis.md index 2551255f..2847144a 100644 --- a/docs/analysis/templates/analysis.md +++ b/docs/analysis/templates/analysis.md @@ -3,6 +3,7 @@ title: _PROJECT_ Documentation Analysis tags: [_PROJECT_] created: YYYY-MM-DD modified: YYYY-MM-DD +toc_max_heading_level: 4 author: _NAME_ (@_HANDLE_) --- @@ -10,27 +11,27 @@ author: _NAME_ (@_HANDLE_) ## About this template -TO USE THIS TEMPLATE, search and replace the named IDs: +To use this template, use a copy of this file with the same `analysis.md` +filename and search and replace the named placeholder IDs: -- `_PROJECT_`: project name -- `YYYY-MM-DD`: creation and modification dates of the analysis document -- `_NAME_`: name of the analysis author -- `@_HANDLE_`: GitHub handle of the analysis author -- `_PROJECT-WEBSITE_`: landing page of the project's information website -- `_PROJECT-DOC-URL_`: main page of the technical documentation for the current - project revision; this might be on the main website server, for example as - _PROJECT-WEBSITE_/doc -- `_PROJECT-DOC-REPO_`: repository where the project technical documentation is - stored; this might be its own repo or a directory in the project main repo +| ID | Description | +| -------------------- | --------------------------------------------------- | +| `_PROJECT_` | Project name | +| `YYYY-MM-DD` | Creation & modification dates of this document | +| `_NAME_` | Author's, technical writer, name | +| `@_HANDLE_` | Author's GitHub handle | +| `_PROJECT-WEBSITE_` | Main Landing page of the project's website | +| `_PROJECT-DOC-URL_` | Main page of the Project's documentation | +| `_PROJECT-DOC-REPO_` | GitHub repository where the project's documentation | For the analysis procedure, see [Analysis how-to](../howto.md). -> Note: delete this "About this template" section after you have customized this -> template for a specific project. +> Author's Note: Delete this "About this template" section after you have +> customized this template for a specific project. ## Introduction -This document is an analyzes the effectiveness and completeness of the +This document is an analysis of the effectiveness and completeness of the [_PROJECT_][project-website] open source software (OSS) project's documentation and website. It is funded by the CNCF Foundation as part of its overall effort to incubate, grow, and graduate open source cloud native software projects. @@ -40,21 +41,34 @@ prerequisite for program graduation. The documentation analysis is the first step of a CNCF process aimed at assisting projects with their documentation efforts. +### AI assisted analysis + +> Author's note: Remove this section if not applicable, otherwise edit as +> needed. + +The following parts of this analysis were generated by AI: + +- Overall comments, following the section heading, on the areas in that section. +- Answers to the criteria questions for an area. +- Comments, including strengths and weaknesses, for an area. +- Recommendations for improvements in each area. + ### Purpose This document was written to analyze the current state of _PROJECT_ documentation. It aims to provide project leaders with an informed understanding -of potential problems in current project documentation. A second -[implementation][] document, , outlines an actionable plan for improvement. A +of potential problems in current project documentation. Optionally, a second +[implementation][] document, outlines an actionable plan for improvement and a third document is an [issues list][] of issues to be added to the project documentation repository. These issues can be taken up by contributors to improve the documentation. This document: -- Analyzes the current _PROJECT_ technical documentation and website -- Compares existing documentation against the CNCF’s standards -- Recommends a program of key improvements with the largest return on investment +- Analyzes the current _PROJECT_ technical documentation and website. +- Compares existing documentation against the CNCF’s standards. +- Recommends a program of key improvements with the largest return on + investment. ### Scope of analysis @@ -86,30 +100,40 @@ This document is divided into three sections that represent three major areas of concern: - **Project documentation:** concerns documentation for users of the _PROJECT_ - software, aimed at people who intend to use the project software + software, aimed at people who intend to use the project software. - **Contributor documentation:** concerns documentation for new and existing - contributors to the _PROJECT_ OSS project + contributors to the _PROJECT_ OSS project. - **Website:** concerns the mechanics of publishing the documentation, and - includes branding, website structure, and maintainability + includes branding, website structure, and maintainability. + +A fourth area of concern, AI-assisted analysis, might be included in this +analysis. Each section begins with summary ratings based on a rubric with appropriate [criteria] for the section, then proceeds to: -- **Comments**: observations about the existing documentation, with a focus on - how it does or does not help _PROJECT_ users achieve their goals. +- **Comments**: observations about the existing documentation, including answers + to questions from the criteria specification. Comments focus on how it does or + does not help _PROJECT_ users achieve their goals. - **Recommendations**: suggested changes that would improve the effectiveness of the documentation. -The accompanying [implementation] document breaks the recommendations down into -concrete actions that can be implemented by project contributors. Its focus is -on drilling down to specific, achievable work that can be completed in +> Author's note: We recommend writing implementation notes only for large +> projects. An implementation plan can bridge the gap between the high-level +> recommendations in this document and individual issues entered in the doc +> repo. Remove the sentences about the implementation doc if you didn't write +> one. + +An optional accompanying [implementation] document breaks the recommendations +down into concrete actions that can be implemented by project contributors. Its +focus is on drilling down to specific, achievable work that can be completed in constrained blocks of time. Ultimately, the implementation items are decomposed into a series of issues and entered on GitHub `PROJECT-DOC-WEBSITE/issues`. ### How to use this document Readers interested only in actionable improvements should skip this document and -read the [implementation][] plan and [issues list][]. +read the [implementation][] plan (if provided) and [issues list][]. Readers interested in the current state of the documentation and the reasoning behind the recommendations should read the section of this document pertaining @@ -136,12 +160,12 @@ to legal requirements such as copyright and licensing issues. ## Project documentation -> AUTHOR NOTE: Pick the CNCF maturity level of the project: +> Author's Note: Pick the CNCF maturity level of the project: _PROJECT_ is a **graduated** project of CNCF. This means that the project should have [_very high_][criteria] standards for documentation. -> AUTHOR NOTE: or +> Author's Note: or _PROJECT_ is an **incubating** project of CNCF. This means that the project should be [_developing_][criteria] professional-quality documentation alongside @@ -157,26 +181,26 @@ the project code. ### Comments -> AUTHOR NOTE: make any overall comments about the Project Documentation here. +> Author's Note: make any overall comments about the Project Documentation here. The following sections contain brief assessments of each element of the Project Documentation rubric. -> AUTHOR NOTE: For each heading below, discuss how well the in-scope items, and -> especially the technical documentation, meet these criteria. (Criteria are +> Author's Note: For each heading below, discuss how well the in-scope items, +> and especially the technical documentation, meet these criteria. (Criteria are > copied from criteria.md) #### Information architecture -The overall structure (pages/subpages/sections/subsections) of your project +The overall structure (pages/subpages/sections/subsections) of the project documentation. We evaluate on the following: - Is there high level conceptual/“About” content? Is the documentation feature complete? (i.e., each product feature is documented) - Are there step-by-step instructions (tasks, tutorials) documented for features? -- Are there any key features which are documented but missing task - documentation? +- Are there any features that are documented (for example, described + conceptually) but are missing task documentation? - Is the “happy path”/most common use case documented? Does task and tutorial content demonstrate atomicity and isolation of concerns? (Are tasks clearly named according to user goals?) @@ -184,6 +208,11 @@ documentation. We evaluate on the following: users needing more help? (FAQ, Troubleshooting) - If the product exposes an API, is there a complete reference? - Is content up to date and accurate? +- If the product builds on or incorporates another product (such as Kubernetes), + are there external references where needed to install and configure those + products? +- Does the documentation organization and navigation need a high-level + restructuring? #### New user content @@ -195,8 +224,8 @@ specifically for them. We evaluate on the following: - Is installation documented step-by-step? - If needed, are multiple OSes documented? - Do users know where to go after reading the getting started guide? -- Is your new user content clearly signposted on your site’s homepage or at the - top of your information architecture? +- Is the new user content clearly signposted on the site’s homepage or at the + top of the information architecture? - Is there sample code or other example content that can easily be copy-pasted? #### Content maintainability & site mechanics @@ -206,10 +235,10 @@ become large maintenance burdens, particularly if you don’t plan for them. We evaluate on the following: -- Is your documentation searchable? +- Is the documentation searchable? - Are you planning for localization/internationalization with regards to site directory structure? Is a localization framework present? -- Do you have a clearly documented method for versioning your content? +- Do you have a clearly documented method for versioning the content? #### Content creation processes @@ -218,9 +247,9 @@ requires the same kind of review and approval processes as code. We evaluate on the following: -- Is there a clearly documented (ongoing) contribution process for - documentation? -- Does your code release process account for documentation creation & updates? +- Is there a clear path to the contributor documentation for new users who want + to contribute to the project? +- Does the code release process account for documentation creation & updates? - Who reviews and approves documentation pull requests? - Does the website have a clear owner/maintainer? @@ -237,7 +266,7 @@ We evaluate on the following: ### Recommendations -> AUTHOR NOTE: Write general recommendations based on the comments from the +> Author's Note: Write general recommendations based on the comments from the > previous section. #### Information architecture @@ -252,12 +281,12 @@ We evaluate on the following: ## Contributor documentation -> AUTHOR NOTE: Pick the CNCF maturity level of the project: +> Author's Note: Pick the CNCF maturity level of the project: _PROJECT_ is a **graduated** project of CNCF. This means that the project should have [_very high_][criteria] standards for documentation. -> AUTHOR NOTE: or +> Author's Note: or _PROJECT_ is an **incubating** project of CNCF. This means that the project should be [_developing_][criteria] professional-quality documentation alongside @@ -272,15 +301,15 @@ the project code. ### Comments -> AUTHOR NOTE: make any overall comments about the Contributor Documentation +> Author's Note: make any overall comments about the Contributor Documentation > here. The following sections contain brief assessments of each element of the Contributor Documentation rubric. -> AUTHOR NOTE: For each heading below, discuss how well the in-scope items meet -> these criteria. Keep in mind that much of the contributor documentation might -> be contained in the documentation repository. (Criteria are copied from +> Author's Note: For each heading below, discuss how well the in-scope items +> meet these criteria. Keep in mind that much of the contributor documentation +> might be contained in the documentation repository. (Criteria are copied from > criteria.md) #### Communication methods documented @@ -291,8 +320,8 @@ to reach you. We evaluate on the following: - Is there a Slack/Discord/Discourse/etc. community and is it prominently linked - from your website? -- Is there a direct link to your GitHub organization/repository? + from the website? +- Is there a direct link to the GitHub organization/repository? - Are weekly/monthly project meetings documented? Is it clear how someone can join those meetings? - Are mailing lists documented? @@ -315,8 +344,8 @@ in easily? We evaluate on the following: -- Do you have a community repository or section on your website? -- Is there a document specifically for new contributors/your first contribution? +- Do you have a community repository or section on the website? +- Is there a document specifically for new contributors/the first contribution? - Do new users know where to get help? #### Project governance documentation @@ -329,7 +358,7 @@ We evaluate on the following: ### Recommendations -> AUTHOR NOTE: Write general recommendations based on the comments from the +> Author's Note: Write general recommendations based on the comments from the > previous section. #### Communication methods documented @@ -342,12 +371,12 @@ We evaluate on the following: ## Website and infrastructure -> AUTHOR NOTE: Pick the CNCF maturity level of the project: +> Author's Note: Pick the CNCF maturity level of the project: _PROJECT_ is a **graduated** project of CNCF. This means that the project should have [_very high_][criteria] standards for documentation. -> AUTHOR NOTE: or +> Author's Note: or _PROJECT_ is an **incubating** project of CNCF. This means that the project should be [_developing_][criteria] professional-quality documentation alongside @@ -359,7 +388,7 @@ the project code. | Meets min website req. (for maturity level) | [rating (1-5)] | | Usability, accessibility, and design | [rating (1-5)] | | Branding and design | [rating (1-5)] | -| Case studies/social proof | [rating (1-5)] | +| Case studies & social proof | [rating (1-5)] | | SEO, Analytics, and site-local search | [rating (1-5)] | | Maintenance planning | [rating (1-5)] | | A11y plan & implementation | [rating (1-5)] | @@ -372,16 +401,16 @@ the project code. ### Comments -> AUTHOR NOTE: make any overall comments about the Website and documentation +> Author's Note: make any overall comments about the Website and documentation > infrastructure here. The following sections contain brief assessments of each element of the Website and documentation infrastructure rubric. -> AUTHOR NOTE: for each heading below, discuss how well the in-scope items meet -> these criteria. Keep in mind that much of the website infrastructure criteria -> depend on the tools (static site generator, website framework and hosting, -> analytics tools, etc.) and processes (project CI, release procedures, +> Author's Note: for each heading below, discuss how well the in-scope items +> meet these criteria. Keep in mind that much of the website infrastructure +> criteria depend on the tools (static site generator, website framework and +> hosting, analytics tools, etc.) and processes (project CI, release procedures, > governance, etc.) used to produce the documentation. (Criteria are copied from > criteria.md) @@ -402,6 +431,11 @@ submodules][git-submodules]. If a project chooses to keep source files in multiple repos, they need a clearly documented strategy for managing mirrored files and new contributions. +We evaluate on the following: + +- Does the project have a single source for its documentation? If not, is there + a reason? + #### Minimal website requirements Listed here are the minimal website requirements for projects based on their @@ -425,22 +459,27 @@ only two levels for which a tech docs analysis can be requested.) https://github.com/cncf/toc/tree/main/process#ii-stages---definitions--expectations [cncf-servicedesk]: https://servicedesk.cncf.io +We evaluate on the following: + +- Are all of the applicable guidelines satisfied as described in the Website + guidelines and checklist? + #### Usability, accessibility and devices Most CNCF websites are accessed from mobile and other non-desktop devices at -least 10-20% of the time. Planning for this early in your website's design will +least 10-20% of the time. Planning for this early in the website's design will be much less effort than retrofitting a desktop-first design. - Is the website usable from mobile? - Are doc pages readable? - Are all / most website features accessible from mobile -- such as the top-nav, site search and in-page table of contents? -- Might a [mobile-first] design make sense for your project? +- Might a [mobile-first] design make sense for the project? [mobile-first]: https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Responsive/Mobile_first -Plan for suitable [accessibility][] measures for your website. For example: +Plan for suitable [accessibility][] measures for the website. For example: - Are color contrasts significant enough for color-impaired readers? - Are most website features usable using a keyboard only? @@ -462,7 +501,7 @@ We evaluate on the following: - Is the brand used across the website consistently? - Is the website’s typography clean and well-suited for reading? -#### Case studies/social proof +#### Case studies and social proof One of the best ways to advertise an open source project is to show other organizations using it. @@ -478,18 +517,18 @@ We evaluate on the following: #### SEO, Analytics and site-local search -SEO helps users find your project and it's documentation, and analytics helps -you monitor site traffic and diagnose issues like page 404s. Intra-site search, -while optional, can offer your readers a site-focused search results. +SEO helps users find the project and it's documentation, and analytics helps you +monitor site traffic and diagnose issues like page 404s. Intra-site search, +while optional, can offer the readers a site-focused search results. We evaluate on the following: - Analytics: - Is analytics enabled for the production server? - Is analytics disabled for all other deploys? - - If your project used Google Analytics, have you migrated to GA4? + - If the project used Google Analytics, have you migrated to GA4? - Can Page-not-found (404) reports easily be generated from you site - analytics? Provide a sample of the site's current top-10 404s. + analytics? - Is site indexing supported for the production server, while disabled for website previews and builds for non-default branches? - Is local intra-site search available from the website? @@ -503,20 +542,37 @@ project maintainers aren’t web developers. We evaluate on the following: -- Is your website tooling well supported by the community (i.e., Hugo with the +- Is the website tooling well supported by the community (i.e., Hugo with the Docsy theme) or commonly used by CNCF projects (our recommended tech stack?) - Are you actively cultivating website maintainers from within the community? - Are site build times reasonable? - Do site maintainers have adequate permissions? +#### AI Optimization and Discoverability + +Tools and techniques are emerging that enable AI agents to use documentation as +input. For example, many repositories now include "AI-friendly" Markdown files +that enable LLMs to ingest content. We encourage projects to optimize for +AI-based documentation use. + +We evaluate on the following: + +- Is there an llms.txt file in the root of the website or documentation + repository? + + The leading proposal for an AI-enabling file for websites is at + [llms-txt](https://llmstxt.org). + + Task can be accomplished manually or by tools. + #### Other -- Is your website accessible via HTTPS? +- Is the website accessible via HTTPS? - Does HTTP access, if any, redirect to HTTPS? ### Recommendations -> AUTHOR NOTE: Write general recommendations based on the comments from the +> Author's Note: Write general recommendations based on the comments from the > previous section. #### Single-source requirement @@ -527,7 +583,7 @@ We evaluate on the following: #### Branding and design -#### Case studies/social proof +#### Case studies and social proof #### SEO, Analytics and site-local search