Adds the api-service stage job behind Varnish

operations/README.md

@@ -6,14 +6,14 @@ Nomad job definitions for running the services on the Anyone cluster. Deployment
 | ---- | ---- |
 | [geo-ip-service-stage.hcl](geo-ip-service-stage.hcl) | geo-ip-service in `stage-services`, on the `stage` pool |
 | [geo-ip-service-live.hcl](geo-ip-service-live.hcl) | geo-ip-service in `live-services`, on the `live-services` pool |
-| [api-service-parity-stage.hcl](api-service-parity-stage.hcl) | the ported api-service next to the running one in `stage-services`, for the [parity run](#api-service-parity-run-on-stage) |
+| [api-service-stage.hcl](api-service-stage.hcl) | api-service behind Varnish in `stage-services`, on the `stage` pool |
 
 ## Choosing a commit
 
-Every job takes `commit_sha`, the commit whose image it runs, and pins a default. Continuous integration publishes a service's image only for commits that changed the service or a package it uses, so pick a commit that did. For geo-ip-service, the latest one on the default branch is:
+Every job takes `commit_sha`, the commit whose image it runs, and pins a default. Continuous integration publishes a service's image only for commits that changed the service or a package it uses, so pick a commit that did. For a service under `apps/<service>`, the latest one on the default branch is:
 
 ```sh
-git log -1 --format=%H origin/master -- apps/geo-ip-service packages package.json bun.lock tsconfig.json .dockerignore
+git log -1 --format=%H origin/master -- apps/<service> packages package.json bun.lock tsconfig.json .dockerignore
 ```
 
 ```sh
@@ -120,38 +120,22 @@ nomad alloc exec -namespace <namespace> -task <task> <allocation> sh -c 'wget -q
 
 From outside the WireGuard network, the same address and port do not answer.
 
-## api-service parity run on stage
+## api-service
 
-The parity job runs the ported api-service next to `api-service-stage`, with the same upstreams, so the two can be compared before the ported one replaces it. It is reachable only on WireGuard: it has no Traefik tags and no Varnish, and no other job reads its Consul service `api-service-parity-stage`. `GEO_IP_CACHE_TTL` is `2m` so that cached locations go stale within the checks below.
+What the stage job sets up:
 
-A job reads only the Vault path named after it, so before the first run the database credentials go to `kv/stage-services/api-service-parity-stage`, under `DB_USER` and `DB_PASS`, the same values as `kv/stage-services/api-service-stage`. The copy lives only as long as the parity job; at the cutover this job definition becomes `api-service-stage`, which reads the original path.
+- One allocation on the `stage` pool with two tasks on one network: api-service on port 3000 and Varnish in front of it on port 8080. A new version starts as a canary next to the running one and is promoted once healthy, or reverted; it has ten minutes to become healthy.
+- The Consul service `api-service-stage` on a dynamic port on the `wireguard` host network, mapped to Varnish, with the Traefik tags that publish it as `api-stage.ec.anyone.tech`.
+- An HTTP check on `/healthz` sent to the allocation's own address through Varnish, so it covers both tasks. The service reports `degraded` rather than `down` while geo-ip-service is unavailable, so the check keeps passing then.
+- The geo-ip-service address in a file, as [Reaching it from another job](#reaching-it-from-another-job) describes.
+- The UNS indexer database credentials from Vault at `kv/stage-services/api-service-stage`, under `DB_USER` and `DB_PASS`.
 
-```sh
-nomad job run operations/api-service-parity-stage.hcl
-```
-
-### Parity
+Varnish:
 
-From inside the allocation, against the running service through its public hostname:
+- Caches successful and `4xx` responses for five minutes. A `5xx` is never cached, so the next request goes to the service again.
+- Never caches `/healthz`.
+- Accepts requests of up to 128 KB, above the service's own limit of 80 KB, so an oversized request gets `431` from the service instead of a dropped connection.
 
 ```sh
-nomad alloc exec -namespace stage-services -task api-service-parity-stage-task <allocation> \
-  bun apps/api-service/parity/parity.ts --old https://api-stage.ec.anyone.tech --new http://127.0.0.1:3000
+nomad job run -var commit_sha=<sha> operations/api-service-stage.hcl
 ```
-
-The [parity check](../apps/api-service/#parity-check) describes what it compares. Map differences for single relays usually mean the two services read different GeoLite2 releases; the running service loads its databases only when it starts.
-
-### geo-ip-service outages
-
-1. Request `/relay-map/` and `/fingerprint-map/` once, then stop geo-ip-service with `nomad job stop -namespace stage-services geo-ip-service-stage`. After more than two minutes both maps still answer `200` with the same relays, and `/healthz` reports `geoIp` as `degraded`.
-2. With geo-ip-service still stopped, restart the parity allocation. Both maps answer `503` with `Relay geolocation is unavailable`, and every other route answers as before.
-3. Run geo-ip-service again. It loads its databases from the volume and comes up on a new port. The parity job picks up the new address without restarting, the maps answer `200` again, and `/healthz` reports `geoIp` as `ok`.
-
-Afterwards, stop the parity job and delete every version of its copy of the credentials:
-
-```sh
-nomad job stop -purge -namespace stage-services api-service-parity-stage
-vault kv metadata delete kv/stage-services/api-service-parity-stage
-```
-
-`vault kv delete` is not enough: it marks only the latest version deleted, and the values stay recoverable.

operations/api-service-parity-stage.hcl → operations/api-service-stage.hcl

@@ -4,7 +4,7 @@ variable "commit_sha" {
   default     = "be5303e389fba3bf4cb96921de311be17304c1ff"
 }
 
-job "api-service-parity-stage" {
+job "api-service-stage" {
   datacenters = ["ator-fin"]
   type        = "service"
   namespace   = "stage-services"
@@ -14,32 +14,58 @@ job "api-service-parity-stage" {
     value     = "stage"
   }
 
-  group "api-service-parity-stage-group" {
+  group "api-service-stage-group" {
     count = 1
 
+    update {
+      max_parallel      = 1
+      canary            = 1
+      min_healthy_time  = "30s"
+      healthy_deadline  = "10m"
+      progress_deadline = "15m"
+      auto_revert       = true
+      auto_promote      = true
+    }
+
     network {
       mode = "bridge"
       port "http" {
-        to           = 3000
+        to           = 8080
         host_network = "wireguard"
       }
     }
 
     service {
-      name = "api-service-parity-stage"
+      name = "api-service-stage"
       port = "http"
-      tags = ["logging"]
+      tags = [
+        "logging",
+        "traefik-ec.enable=true",
+        "traefik-ec.http.routers.api-stage.rule=Host(`api-stage.ec.anyone.tech`)",
+        "traefik-ec.http.routers.api-stage.entrypoints=https",
+        "traefik-ec.http.routers.api-stage.tls=true",
+        "traefik-ec.http.routers.api-stage.tls.certresolver=anyoneresolver",
+        "traefik-ec.http.routers.api-stage.middlewares=api-stage-ratelimit",
+        "traefik-ec.http.middlewares.api-stage-ratelimit.ratelimit.average=1000",
+        "traefik-ec.http.middlewares.api-stage-ratelimit.ratelimit.burst=10"
+      ]
 
       check {
-        name     = "api-service health"
-        type     = "http"
-        path     = "/healthz"
-        interval = "10s"
-        timeout  = "5s"
+        name         = "api-service health"
+        type         = "http"
+        path         = "/healthz"
+        interval     = "10s"
+        timeout      = "5s"
+        address_mode = "alloc"
+
+        check_restart {
+          limit = 10
+          grace = "30s"
+        }
       }
     }
 
-    task "api-service-parity-stage-task" {
+    task "api-service-stage-task" {
       driver = "docker"
 
       config {
@@ -56,7 +82,6 @@ job "api-service-parity-stage" {
         HB_URL                  = "https://hb-stage.anyone.tech"
         DB_NAME                 = "uns_indexer"
         GEO_IP_SERVICE_URL_FILE = "/local/geo-ip-service-url"
-        GEO_IP_CACHE_TTL        = "2m"
         LOG_LEVEL               = "info"
       }
 
@@ -100,7 +125,7 @@ job "api-service-parity-stage" {
 
       template {
         data        = <<-EOH
-        {{ with secret "kv/stage-services/api-service-parity-stage" }}
+        {{ with secret "kv/stage-services/api-service-stage" }}
         DB_USER="{{ .Data.data.DB_USER }}"
         DB_PASS="{{ .Data.data.DB_PASS }}"
         {{ end }}
@@ -114,5 +139,52 @@ job "api-service-parity-stage" {
         memory = 512
       }
     }
+
+    task "varnish-cache-stage-task" {
+      driver = "docker"
+
+      config {
+        image   = "varnish:9.1.0"
+        args    = ["-p", "http_req_size=128k", "-p", "workspace_client=256k"]
+        volumes = ["local/default.vcl:/etc/varnish/default.vcl:ro"]
+      }
+
+      env {
+        VARNISH_HTTP_PORT = "8080"
+      }
+
+      template {
+        data        = <<-EOH
+        vcl 4.1;
+
+        backend default {
+            .host = "127.0.0.1";
+            .port = "3000";
+        }
+
+        sub vcl_recv {
+            if (req.url ~ "^/healthz/?(\?|$)") {
+                return (pass);
+            }
+        }
+
+        sub vcl_backend_response {
+            if (beresp.status >= 500) {
+                set beresp.uncacheable = true;
+                set beresp.ttl = 10s;
+                return (deliver);
+            }
+            set beresp.ttl = 5m;
+        }
+        EOH
+        destination = "local/default.vcl"
+        change_mode = "noop"
+      }
+
+      resources {
+        cpu    = 256
+        memory = 1024
+      }
+    }
   }
 }