[ENG-12064] Phase 2: SSR performance & reliability metrics

README.md

@@ -23,10 +23,14 @@ take up to 60 seconds once the docker build finishes.
 ### Recommended
 
 - [Docker Commands](docs/docker.md).
+- [Architecture](docs/arch.md).
 - [ESLint Strategy](docs/eslint.md).
 - [Git Conventions](docs/git-convention.md).
 - [i18n](docs/i18n.md).
+- [Models conventions](docs/models.md).
 - [NGXS Conventions](docs/ngxs.md).
+- [SSR](docs/ssr.md).
+- [SSR metrics](docs/ssr-metrics.md).
 - [Testing Strategy](docs/testing.md).
 - [Sentry error filtering](docs/sentry.md).
 
@@ -60,6 +64,6 @@ Install Volta from [volta](https://volta.sh/) and it will automatically pin Node
 
 ## Configuration
 
-OSF uses an `assets/config/config.json` file for any 3rd-party tokens. This file is not committed to the repo.
+OSF uses `src/assets/config/config.json` for third-party tokens and environment URLs. This file is not committed to the repo.
 
-There is a `assets/config/template.json` file that can be copied to `assets/config/config.json` to store any 3rd-party tokens locally.
+Copy `src/assets/config/template.json` to `src/assets/config/config.json` for local development. At runtime the app loads it from `/assets/config/config.json`.

docs/arch.md

@@ -79,6 +79,12 @@ See [NGXS State Management](./ngxs.md).
 
 ---
 
+## SSR
+
+Server-side rendering, route render modes, config, and bot traffic are documented in [SSR](./ssr.md). Render metrics are in [SSR metrics](./ssr-metrics.md).
+
+---
+
 ## 🚀 Dynamic File Generation (Schematics)
 
 Use Angular CLI for scaffolding:

docs/ssr-metrics.md

@@ -0,0 +1,146 @@
+# SSR metrics
+
+## Index
+
+- [Overview](#overview)
+- [What we track](#what-we-track)
+- [When a metric is sent](#when-a-metric-is-sent)
+- [When HTML is inspected](#when-html-is-inspected)
+- [is_complete rules](#is_complete-rules)
+- [content_type](#content_type)
+- [Payload and API](#payload-and-api)
+- [Related docs](#related-docs)
+
+---
+
+## Overview
+
+After the SSR server sends HTML to a crawler, it POSTs a JSON:API metric to the OSF API. Collection runs in the background and does not delay the response.
+
+Implementation: `src/server/ssr-metrics.middleware.ts`, `src/server/ssr-metrics.ts`, `src/server/ssr-html-metrics.ts`.
+
+In production, the SSR Node process receives **bot traffic only**. Metrics are emitted for HTML navigations on that server.
+
+---
+
+## What we track
+
+| Goal                | Field          | Notes                                                                                                                                 |
+| ------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
+| Render success rate | `status`       | Derive success as HTTP 2xx. Failures use `0` (no Angular response) or `500` (render exception).                                       |
+| Render speed        | `ttfb`         | Milliseconds from request start until Angular returns a response. Server-side render time, not browser network TTFB.                  |
+| Page completeness   | `is_complete`  | Checked from rendered HTML. See [is_complete rules](#is_complete-rules).                                                              |
+| Content type        | `content_type` | API resource type from `osf:type` meta (`nodes`, `registrations`, `preprints`, `files`, `users`). Null when not set or not inspected. |
+| Bot vs other        | `is_bot`       | From User-Agent regex in middleware.                                                                                                  |
+| Crawler identity    | `user_agent`   | Truncated to 512 characters.                                                                                                          |
+| Page                | `url`          | Full public URL (`webUrl` + path). **Query string is stripped** (no `view_only` or other params).                                     |
+
+We do **not** track Search Console index status. “Pages by content type” in SSR means **successful bot responses grouped by `content_type`**, not confirmed Google indexing.
+
+---
+
+## When a metric is sent
+
+Sent for **HTML navigations** handled by the metrics middleware.
+
+**Skipped** (no metric):
+
+- `/assets/*`, `/static/*`
+- `/.well-known/*`
+- Static file extensions (`.js`, `.css`, `.ico`, `.json`, images, fonts, …)
+
+**Included**:
+
+- Successful renders (any HTTP status Angular returns)
+- Missing Angular response → `status: 0`
+- Render exception → `status: 500`
+
+---
+
+## When HTML is inspected
+
+HTML is read only when:
+
+- User-Agent matches the bot regex (`SEARCH_BOT` in middleware)
+- HTTP status is **200**
+
+If not inspected, `is_complete` stays `false` and `content_type` stays `null` even when the page rendered successfully.
+
+On a bot-only SSR host, every 200 could be inspected; today the code also requires a bot regex match.
+
+Inspection runs **after** the response is sent. It clones the response body only when inspection will run.
+
+---
+
+## is_complete rules
+
+Implemented in `inspectSsrHtml` (`src/server/ssr-html-metrics.ts`).
+
+All must pass:
+
+1. **SSR marker** — `<osf-root>` has `ng-server-context` (page was server-rendered, not a bare CSR shell).
+2. **Non-empty root** — inner HTML of `<osf-root>` is not empty after whitespace is removed.
+3. **Meta tags or allowlisted path** — either:
+   - HTML contains `osf-dynamic-meta`, or
+   - path is on the meta-optional allowlist (search, discover, terms, user, institutions, meetings, collections, provider landing pages, etc.)
+
+---
+
+## content_type
+
+Read from rendered HTML:
+
+```html
+<meta name="osf:type" content="nodes" />
+```
+
+Set by `MetaTagsService` (`osfType`). Project, registration, preprint, and file pages go through `MetaTagsBuilderService`. Profile pages set `osfType: users` only.
+
+**Populated for:** projects, registrations, preprints, files, users (`/user/:id` and `/profile`).
+
+**Often null for:** institutions, meetings, collections, search, discover, and any page without `osf:type`. That is expected if those types are out of scope for the metric.
+
+---
+
+## Payload and API
+
+**Endpoint**
+
+```
+POST {apiDomainUrl}/_/metrics/events/ssr_metrics/
+```
+
+**Headers**
+
+- `Accept: application/vnd.api+json;version=2.20`
+- `Content-Type: application/vnd.api+json`
+- `X-Throttle-Token: {THROTTLE_TOKEN}` when env is set
+
+**Example body**
+
+```json
+{
+  "data": {
+    "attributes": {
+      "url": "https://osf.io/abc12/overview",
+      "ttfb": 842,
+      "is_bot": true,
+      "is_complete": true,
+      "content_type": "nodes",
+      "status": 200,
+      "user_agent": "Mozilla/5.0 (compatible; Googlebot/2.1; ...)"
+    }
+  }
+}
+```
+
+If `apiDomainUrl` is missing, the POST is skipped silently.
+
+Failed POSTs are logged with `console.error` in the middleware catch block.
+
+---
+
+## Related docs
+
+- [SSR overview](./ssr.md) — architecture, routes, config
+- [Architecture](./arch.md) — file layout

docs/ssr.md

@@ -0,0 +1,169 @@
+# SSR (Server-Side Rendering)
+
+## Index
+
+- [Overview](#overview)
+- [Production traffic](#production-traffic)
+- [Request flow](#request-flow)
+- [Key files](#key-files)
+- [Render modes](#render-modes)
+- [Configuration](#configuration)
+- [SEO-related app behavior](#seo-related-app-behavior)
+- [Local development](#local-development)
+- [Adding or changing SSR routes](#adding-or-changing-ssr-routes)
+- [Related docs](#related-docs)
+
+---
+
+## Overview
+
+OSF uses Angular SSR so search-engine crawlers receive fully rendered HTML for public pages (projects, registrations, preprints, discover pages, and similar).
+
+The SSR stack has two layers:
+
+1. **Angular SSR** — renders the app on the server (`app.config.server.ts`, `app.routes.server.ts`).
+2. **Express server** — serves static assets, delegates HTML to Angular, and sends render metrics after the response (`src/server.ts`, `src/server/*`).
+
+In production, **only bot traffic** is routed to the SSR build. Human traffic uses the client-only build. See [Production traffic](#production-traffic).
+
+---
+
+## Production traffic
+
+```
+Crawler  → cloud (bot detection) → SSR Node server → HTML + metrics
+Browser  → cloud                 → static/CSR build → SPA shell + client render
+```
+
+---
+
+## Request flow
+
+1. Request hits Express (`src/server.ts`).
+2. Static files (`/assets`, hashed JS/CSS, etc.) are served from `browser/` when possible.
+3. HTML navigations go through `createSsrMetricsMiddleware` (`src/server/ssr-metrics.middleware.ts`).
+4. Angular SSR renders the page (`AngularNodeAppEngine.handle`).
+5. HTML is sent to the client immediately.
+6. In the background (`setImmediate`):
+   - for bot + HTTP 200: HTML is inspected for completeness and `content_type`
+   - a metric payload is POSTed to the OSF API
+
+Bot timing is not blocked by steps 6. See [SSR metrics](./ssr-metrics.md).
+
+---
+
+## Key files
+
+| File                                           | Role                                                   |
+| ---------------------------------------------- | ------------------------------------------------------ |
+| `src/server.ts`                                | Express entry, static files, wires metrics middleware  |
+| `src/main.server.ts`                           | Angular server bootstrap                               |
+| `src/app/app.config.server.ts`                 | Server providers: routes, config, i18n loader          |
+| `src/app/app.routes.server.ts`                 | Per-route render mode (Server / Client / Prerender)    |
+| `src/server/ssr-metrics.middleware.ts`         | Render timing, HTML inspection trigger, metric queue   |
+| `src/server/ssr-html-metrics.ts`               | `is_complete` and `content_type` checks                |
+| `src/server/ssr-metrics.ts`                    | POST metric payload to API                             |
+| `src/server/ssr-server-config.ts`              | Load `config.json` + env for metrics                   |
+| `src/server/static-cache-headers.ts`           | Cache headers for static assets                        |
+| `src/app/shared/services/meta-tags.service.ts` | Dynamic SEO meta tags (`osf:type`, `osf-dynamic-meta`) |
+
+---
+
+## Render modes
+
+Defined in `src/app/app.routes.server.ts`:
+
+| Mode          | When to use                                                                          | Bot receives                                       |
+| ------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------- |
+| **Server**    | Public pages that should be indexed (project overview, preprint detail, discover, …) | Full SSR HTML                                      |
+| **Client**    | Auth, forms, dashboards, moderation, editors                                         | CSR shell; metrics may report `is_complete: false` |
+| **Prerender** | Static legal/help pages (`terms-of-use`, `privacy-policy`, …)                        | Pre-built HTML at build time                       |
+
+**Rules of thumb**
+
+- Public read-only detail or listing page → **Server**
+- Login, submit, edit, settings, “my …” pages → **Client**
+- Fixed copy that never changes at runtime → **Prerender**
+
+More specific routes must appear **before** broader patterns (e.g. `preprints/:providerId/:id` before `preprints/:providerId`). Unmatched paths fall through to `**` → **Client**.
+
+---
+
+## Configuration
+
+### `assets/config/config.json`
+
+Copied from `assets/config/template.json` for local dev. Deployed with the build. Used for URLs and third-party keys.
+
+Relevant to SSR / metrics:
+
+| Field          | Used by                                                             |
+| -------------- | ------------------------------------------------------------------- |
+| `apiDomainUrl` | SSR API calls (via `OSFConfigService`) and metrics POST target host |
+| `webUrl`       | Canonical URLs in meta tags; full URL in metric payloads            |
+
+### Environment variables (SSR Node process)
+
+| Variable         | Used by                                                                 |
+| ---------------- | ----------------------------------------------------------------------- |
+| `THROTTLE_TOKEN` | `X-Throttle-Token` on SSR API calls (auth interceptor) and metrics POST |
+| `API_DOMAIN_URL` | Fallback if `apiDomainUrl` missing from `config.json`                   |
+| `WEB_URL`        | Fallback if `webUrl` missing from `config.json`                         |
+| `PORT`           | Express listen port (default `4000`)                                    |
+
+**Throttle token:** SSR page-render API calls read `THROTTLE_TOKEN` from the process environment only (`app.config.server.ts`). It is not taken from `config.json` for auth. Metrics use the same env var via `loadSsrServerConfig`.
+
+### Angular SSR config loading
+
+On the server, `OSFConfigService` does not HTTP-fetch `config.json`. It uses `SSR_CONFIG`, populated at startup from disk in `app.config.server.ts`.
+
+Translations on SSR are loaded from `browser/assets/i18n/en.json` (cached at module load). Browser builds use the HTTP loader as usual.
+
+---
+
+## SEO-related app behavior
+
+### Meta tags
+
+`MetaTagsService` writes dynamic tags with class `osf-dynamic-meta`. Pages also emit `osf:type` (API type: `nodes`, `registrations`, `preprints`, `files`, `users`). Metrics read `osf:type` from the rendered HTML for `content_type`.
+
+Built in `MetaTagsBuilderService` for project, registration, preprint, and file pages. Profile pages call `updateMetaTags` with only `osfType: users` and `mergeDefaults: false`, so they emit a single `osf:type` tag.
+
+---
+
+## Local development
+
+| Command                 | What it does                                  |
+| ----------------------- | --------------------------------------------- |
+| `npm start`             | CSR dev server (port 4200)                    |
+| `npm run start:ssr`     | Dev server with SSR (`dev-ssr` configuration) |
+| `npm run build:ssr`     | Production SSR build → `dist/osf/`            |
+| `npm run serve:ssr:osf` | Run built Express server (port 4000)          |
+
+Typical local SSR test:
+
+```bash
+npm run build:ssr
+npm run serve:ssr:osf
+```
+
+Set `THROTTLE_TOKEN` in the shell if SSR API calls should bypass throttling locally.
+
+Docker `start:docker` runs the **development** configuration (CSR), not the SSR server. Use `build:ssr` + `serve:ssr:osf` or the Dockerfile `ssr` stage for SSR.
+
+---
+
+## Adding or changing SSR routes
+
+1. Add the route in `app.routes.ts` (browser routes).
+2. Add a matching entry in `app.routes.server.ts` with the correct `RenderMode`.
+3. If the page should contribute SEO meta, ensure the feature calls `MetaTagsService.updateMetaTags` and sets `osfType` where applicable.
+4. For public indexable pages, prefer **Server**. Do not SSR authenticated workflows unless there is a specific SEO need.
+
+---
+
+## Related docs
+
+- [SSR metrics](./ssr-metrics.md) — payload, `is_complete`, dashboards
+- [Architecture](./arch.md) — folder layout
+- [Docker](./docker.md) — container workflows