diff --git a/operations/README.md b/operations/README.md index fcc3356..ec4d73a 100644 --- a/operations/README.md +++ b/operations/README.md @@ -6,14 +6,14 @@ Nomad job definitions for running the services on the Anyone cluster. Deployment | ---- | ---- | | [geo-ip-service-stage.hcl](geo-ip-service-stage.hcl) | geo-ip-service in `stage-services`, on the `stage` pool | | [geo-ip-service-live.hcl](geo-ip-service-live.hcl) | geo-ip-service in `live-services`, on the `live-services` pool | -| [api-service-parity-stage.hcl](api-service-parity-stage.hcl) | the ported api-service next to the running one in `stage-services`, for the [parity run](#api-service-parity-run-on-stage) | +| [api-service-stage.hcl](api-service-stage.hcl) | api-service behind Varnish in `stage-services`, on the `stage` pool | ## Choosing a commit -Every job takes `commit_sha`, the commit whose image it runs, and pins a default. Continuous integration publishes a service's image only for commits that changed the service or a package it uses, so pick a commit that did. For geo-ip-service, the latest one on the default branch is: +Every job takes `commit_sha`, the commit whose image it runs, and pins a default. Continuous integration publishes a service's image only for commits that changed the service or a package it uses, so pick a commit that did. For a service under `apps/`, the latest one on the default branch is: ```sh -git log -1 --format=%H origin/master -- apps/geo-ip-service packages package.json bun.lock tsconfig.json .dockerignore +git log -1 --format=%H origin/master -- apps/ packages package.json bun.lock tsconfig.json .dockerignore ``` ```sh @@ -120,38 +120,22 @@ nomad alloc exec -namespace -task sh -c 'wget -q From outside the WireGuard network, the same address and port do not answer. -## api-service parity run on stage +## api-service -The parity job runs the ported api-service next to `api-service-stage`, with the same upstreams, so the two can be compared before the ported one replaces it. It is reachable only on WireGuard: it has no Traefik tags and no Varnish, and no other job reads its Consul service `api-service-parity-stage`. `GEO_IP_CACHE_TTL` is `2m` so that cached locations go stale within the checks below. +What the stage job sets up: -A job reads only the Vault path named after it, so before the first run the database credentials go to `kv/stage-services/api-service-parity-stage`, under `DB_USER` and `DB_PASS`, the same values as `kv/stage-services/api-service-stage`. The copy lives only as long as the parity job; at the cutover this job definition becomes `api-service-stage`, which reads the original path. +- One allocation on the `stage` pool with two tasks on one network: api-service on port 3000 and Varnish in front of it on port 8080. A new version starts as a canary next to the running one and is promoted once healthy, or reverted; it has ten minutes to become healthy. +- The Consul service `api-service-stage` on a dynamic port on the `wireguard` host network, mapped to Varnish, with the Traefik tags that publish it as `api-stage.ec.anyone.tech`. +- An HTTP check on `/healthz` sent to the allocation's own address through Varnish, so it covers both tasks. The service reports `degraded` rather than `down` while geo-ip-service is unavailable, so the check keeps passing then. +- The geo-ip-service address in a file, as [Reaching it from another job](#reaching-it-from-another-job) describes. +- The UNS indexer database credentials from Vault at `kv/stage-services/api-service-stage`, under `DB_USER` and `DB_PASS`. -```sh -nomad job run operations/api-service-parity-stage.hcl -``` - -### Parity +Varnish: -From inside the allocation, against the running service through its public hostname: +- Caches successful and `4xx` responses for five minutes. A `5xx` is never cached, so the next request goes to the service again. +- Never caches `/healthz`. +- Accepts requests of up to 128 KB, above the service's own limit of 80 KB, so an oversized request gets `431` from the service instead of a dropped connection. ```sh -nomad alloc exec -namespace stage-services -task api-service-parity-stage-task \ - bun apps/api-service/parity/parity.ts --old https://api-stage.ec.anyone.tech --new http://127.0.0.1:3000 +nomad job run -var commit_sha= operations/api-service-stage.hcl ``` - -The [parity check](../apps/api-service/#parity-check) describes what it compares. Map differences for single relays usually mean the two services read different GeoLite2 releases; the running service loads its databases only when it starts. - -### geo-ip-service outages - -1. Request `/relay-map/` and `/fingerprint-map/` once, then stop geo-ip-service with `nomad job stop -namespace stage-services geo-ip-service-stage`. After more than two minutes both maps still answer `200` with the same relays, and `/healthz` reports `geoIp` as `degraded`. -2. With geo-ip-service still stopped, restart the parity allocation. Both maps answer `503` with `Relay geolocation is unavailable`, and every other route answers as before. -3. Run geo-ip-service again. It loads its databases from the volume and comes up on a new port. The parity job picks up the new address without restarting, the maps answer `200` again, and `/healthz` reports `geoIp` as `ok`. - -Afterwards, stop the parity job and delete every version of its copy of the credentials: - -```sh -nomad job stop -purge -namespace stage-services api-service-parity-stage -vault kv metadata delete kv/stage-services/api-service-parity-stage -``` - -`vault kv delete` is not enough: it marks only the latest version deleted, and the values stay recoverable. diff --git a/operations/api-service-parity-stage.hcl b/operations/api-service-stage.hcl similarity index 52% rename from operations/api-service-parity-stage.hcl rename to operations/api-service-stage.hcl index 2bfb538..f0e5daf 100644 --- a/operations/api-service-parity-stage.hcl +++ b/operations/api-service-stage.hcl @@ -4,7 +4,7 @@ variable "commit_sha" { default = "be5303e389fba3bf4cb96921de311be17304c1ff" } -job "api-service-parity-stage" { +job "api-service-stage" { datacenters = ["ator-fin"] type = "service" namespace = "stage-services" @@ -14,32 +14,58 @@ job "api-service-parity-stage" { value = "stage" } - group "api-service-parity-stage-group" { + group "api-service-stage-group" { count = 1 + update { + max_parallel = 1 + canary = 1 + min_healthy_time = "30s" + healthy_deadline = "10m" + progress_deadline = "15m" + auto_revert = true + auto_promote = true + } + network { mode = "bridge" port "http" { - to = 3000 + to = 8080 host_network = "wireguard" } } service { - name = "api-service-parity-stage" + name = "api-service-stage" port = "http" - tags = ["logging"] + tags = [ + "logging", + "traefik-ec.enable=true", + "traefik-ec.http.routers.api-stage.rule=Host(`api-stage.ec.anyone.tech`)", + "traefik-ec.http.routers.api-stage.entrypoints=https", + "traefik-ec.http.routers.api-stage.tls=true", + "traefik-ec.http.routers.api-stage.tls.certresolver=anyoneresolver", + "traefik-ec.http.routers.api-stage.middlewares=api-stage-ratelimit", + "traefik-ec.http.middlewares.api-stage-ratelimit.ratelimit.average=1000", + "traefik-ec.http.middlewares.api-stage-ratelimit.ratelimit.burst=10" + ] check { - name = "api-service health" - type = "http" - path = "/healthz" - interval = "10s" - timeout = "5s" + name = "api-service health" + type = "http" + path = "/healthz" + interval = "10s" + timeout = "5s" + address_mode = "alloc" + + check_restart { + limit = 10 + grace = "30s" + } } } - task "api-service-parity-stage-task" { + task "api-service-stage-task" { driver = "docker" config { @@ -56,7 +82,6 @@ job "api-service-parity-stage" { HB_URL = "https://hb-stage.anyone.tech" DB_NAME = "uns_indexer" GEO_IP_SERVICE_URL_FILE = "/local/geo-ip-service-url" - GEO_IP_CACHE_TTL = "2m" LOG_LEVEL = "info" } @@ -100,7 +125,7 @@ job "api-service-parity-stage" { template { data = <<-EOH - {{ with secret "kv/stage-services/api-service-parity-stage" }} + {{ with secret "kv/stage-services/api-service-stage" }} DB_USER="{{ .Data.data.DB_USER }}" DB_PASS="{{ .Data.data.DB_PASS }}" {{ end }} @@ -114,5 +139,52 @@ job "api-service-parity-stage" { memory = 512 } } + + task "varnish-cache-stage-task" { + driver = "docker" + + config { + image = "varnish:9.1.0" + args = ["-p", "http_req_size=128k", "-p", "workspace_client=256k"] + volumes = ["local/default.vcl:/etc/varnish/default.vcl:ro"] + } + + env { + VARNISH_HTTP_PORT = "8080" + } + + template { + data = <<-EOH + vcl 4.1; + + backend default { + .host = "127.0.0.1"; + .port = "3000"; + } + + sub vcl_recv { + if (req.url ~ "^/healthz/?(\?|$)") { + return (pass); + } + } + + sub vcl_backend_response { + if (beresp.status >= 500) { + set beresp.uncacheable = true; + set beresp.ttl = 10s; + return (deliver); + } + set beresp.ttl = 5m; + } + EOH + destination = "local/default.vcl" + change_mode = "noop" + } + + resources { + cpu = 256 + memory = 1024 + } + } } }