Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions DEV.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,13 @@ Flash uses YAML config files. Ibex OAuth2 credentials go in local config overrid

The base config is at `dev/config/base-config.yaml`. Secrets and local overrides go in `$CONFIG_PATH/dev-overrides.yaml` (default: `~/.config/flash/dev-overrides.yaml`).

**Option A — Run the interactive script:**
**Option A. Run the interactive script:**

```bash
./dev/config/set-overrides.sh
```

**Option B — Create manually:**
**Option B. Create manually:**

```yaml
# ~/.config/flash/dev-overrides.yaml
Expand Down Expand Up @@ -151,7 +151,7 @@ This runs four processes in parallel:
| Process | Port | Description |
|---------|------|-------------|
| `start-main` | 4012 (direct), 4002 (via oathkeeper) | Main GraphQL API |
| `start-trigger` | — | Event trigger processor |
| `start-trigger` | none | Event trigger processor |
| `start-ws` | 4000 | WebSocket server for subscriptions |
| `start-ibex-wh` | 4008 | Ibex webhook receiver |

Expand Down Expand Up @@ -234,7 +234,7 @@ TEST=01-connection make integration # runs 01-connection.spec.ts
```

**Known issues:**
- Integration tests are not fully idempotent — use `make reset-integration` between runs
- Integration tests are not fully idempotent; use `make reset-integration` between runs
- If tests timeout, increase: `JEST_TIMEOUT=120000 yarn test:integration`
- Use an SSD for Docker volumes (tests are disk-intensive)

Expand Down Expand Up @@ -287,19 +287,19 @@ Flash requires Node 24.x. Switch with `nvm use 24` or run `yarn install --ignore

### Docker warnings about unset variables

`IBEX_URL`, `HONEYCOMB_DATASET`, `HONEYCOMB_API_KEY` warnings in docker compose output are harmless in dev — these are only needed for the Docker-based server (not used by `make start`).
`IBEX_URL`, `HONEYCOMB_DATASET`, `HONEYCOMB_API_KEY` warnings in docker compose output are harmless in dev; these are only needed for the Docker-based server (not used by `make start`).

### `API key does not start with "SG."`

SendGrid isn't configured — email notifications won't work. Safe to ignore in dev unless you're testing email features. Add a real key to your `dev-overrides.yaml` if needed.
SendGrid isn't configured; email notifications won't work. Safe to ignore in dev unless you're testing email features. Add a real key to your `dev-overrides.yaml` if needed.

### Price service platform warning (Apple Silicon)

```
The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8)
```

The `lnflash/price:edge` image is amd64-only. On Apple Silicon Macs it runs under Rosetta emulation — this is harmless but slow. The warning is safe to ignore.
The `lnflash/price:edge` image is amd64-only. On Apple Silicon Macs it runs under Rosetta emulation; this is harmless but slow. The warning is safe to ignore.

### Server starts but crashes immediately

Expand Down
10 changes: 5 additions & 5 deletions docs/account-capabilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,14 @@ builds on the state machine defined here.
| L1 | Personal | **Verified** |
| L2 | Pro | **Verified** + *Bank payout* badge |
| L3 | Merchant | **Business** |
| — | International | *USD account* badge (capability, not a tier) |
| none | International | *USD account* badge (capability, not a tier) |

- "Pro" is retired (vague). "International" is retired (it's just a USD
account). "Merchant" is renamed **Business**.
- Levels (L1/L2/L3) are internal-only — derived from capabilities, never
- Levels (L1/L2/L3) are internal-only; derived from capabilities, never
shown as a tier the user picks.
- Product decision (2026-07-14): **light headline status**. The account leads
with one word — Trial → Verified → Business — with capability badges as
with one word, Trial → Verified → Business, with capability badges as
supporting detail (not the pure capability-badge approach).

## Capability model
Expand Down Expand Up @@ -50,7 +50,7 @@ Implementation: `src/domain/accounts/capabilities.ts`
operational value used by limits and permissions; transitions recompute it
through the state machine.

> **Note — derived level vs stored `level`.** The read model can imply a
> **Note:** derived level vs stored `level`. The read model can imply a
> level higher than the stored one: an L1 account with an approved bank
> account on file resolves `bankPayout: true`, which derives to L2. Both the
> stored `level` and the derived `capabilities`/`statusHeadline` are exposed
Expand Down Expand Up @@ -120,7 +120,7 @@ mutation {

The server derives the target level from current capabilities plus the
requested one and posts the existing ERPNext **Account Upgrade Request**
doctype — the human review flow in frappe-flash-admin is unchanged.
doctype; the human review flow in frappe-flash-admin is unchanged.
`businessAccountUpgradeRequest` (whole-tier, takes a `level`) remains for
existing clients but is superseded by this mutation.

Expand Down
34 changes: 17 additions & 17 deletions docs/api-keys/ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# API Keys — Architecture
# API Keys: Architecture

Operator and maintainer view of the FIP-07 API-key authentication system. For the developer-facing surface see [USAGE.md](./USAGE.md).

## Design principle: verify at the gateway, not in the app
## Design principle: verify at the gateway

Flash's public GraphQL API sits behind Ory Oathkeeper in **decision mode**: the nginx ingress issues an `auth_request` subrequest to oathkeeper's `/decisions` endpoint, oathkeeper authenticates the request and mints an `id_token` JWT, and nginx **replaces** the client's `Authorization` header with that JWT before proxying to the backend (`auth-response-headers: Authorization`). The backend's `expressjwt` layer then only ever trusts an oathkeeper-signed JWT.

Because of this, a raw `fk_…` key placed in `Authorization` could never reach the backend — it would be rejected as a non-JWT bearer before any application code ran. So API-key verification is done **at the gateway**, mirroring Blink's production pattern, and the key travels in a **separate `X-API-KEY` header** so it never collides with the Kratos session path. (This is the ADR recorded under ENG-107.)
Because of this, a raw `fk_…` key placed in `Authorization` could never reach the backend; it would be rejected as a non-JWT bearer before any application code ran. So API-key verification is done **at the gateway**, mirroring Blink's production pattern, and the key travels in a **separate `X-API-KEY` header** so it never collides with the Kratos session path. (This is the ADR recorded under ENG-107.)

## Request flow

Expand Down Expand Up @@ -49,7 +49,7 @@ It reads `X-API-KEY`, then `verifyApiKey` (`src/app/api-keys/verify-api-key.ts`)
4. Compare `sha256(secret)` against the stored hash in **constant time** (`timingSafeEqual`).
5. Enforce IP constraints (see below), fail-closed.
6. Resolve the owning account's Kratos user id.
7. Throttled `lastUsedAt` update (≤ once per key per minute — keeps verification read-only on hot keys).
7. Throttled `lastUsedAt` update (≤ once per key per minute; keeps verification read-only on hot keys).

On success it returns a **kratos-whoami-shaped** body so oathkeeper's existing `subject_from`/`extra_from` config and `id_token` mutator apply unchanged:

Expand All @@ -68,14 +68,14 @@ On success it returns a **kratos-whoami-shaped** body so oathkeeper's existing `
- `expires_at` is always empty: key expiry is enforced here on every verification, and a value would make session handling try to Kratos-extend the synthetic session id.
- `scope` (space-delimited) and `rate_limit` (numeric) become JWT claims via conditional segments in the mutator template.

Any failure returns `401 {"error":"invalid_api_key"}`, which oathkeeper surfaces as an explicit gateway `401` — the same result as an invalid Kratos token.
Any failure returns `401 {"error":"invalid_api_key"}`, which oathkeeper surfaces as an explicit gateway `401`; the same result as an invalid Kratos token.

## Scope enforcement (graphql-shield)

Scopes are enforced in-process with **graphql-shield**, deny-by-default, from `src/domain/api-keys/scope-map.ts`:

- **`apiKeyScopeForField`** — every authed root query/mutation mapped to the scope it requires, or `BLOCKED`. Non-API-key sessions pass through untouched; for an API-key session the shield rule requires the mapped scope, and any field **not** in the map (or mapped `BLOCKED`) is rejected. A completeness unit test fails CI if a new authed field is added without a mapping, so the deny-by-default guarantee can't silently regress.
- **`apiKeyNestedFieldScopes`** — type-level guards (`ConsumerAccount`, `UserContact`, `BTCWallet`, `UsdWallet`, `UsdtWallet`) so wallet/transaction data reachable *through* a low-scope root field (e.g. `me`, which only needs `read:user`) still requires `read:wallet` / `read:transactions`. This closes the nested-escalation path.
- **`apiKeyScopeForField`**: every authed root query/mutation mapped to the scope it requires, or `BLOCKED`. Non-API-key sessions pass through untouched; for an API-key session the shield rule requires the mapped scope, and any field **not** in the map (or mapped `BLOCKED`) is rejected. A completeness unit test fails CI if a new authed field is added without a mapping, so the deny-by-default guarantee can't silently regress.
- **`apiKeyNestedFieldScopes`**: type-level guards (`ConsumerAccount`, `UserContact`, `BTCWallet`, `UsdWallet`, `UsdtWallet`) so wallet/transaction data reachable *through* a low-scope root field (e.g. `me`, which only needs `read:user`) still requires `read:wallet` / `read:transactions`. This closes the nested-escalation path.

`hasApiKeyScope` implements the implication rules: `admin` grants everything, `write:X` implies `read:X`. Scope claims arrive on the GraphQL context via `session.ts`, which splits the space-delimited `scope` JWT claim into `ctx.scopes`.

Expand All @@ -87,7 +87,7 @@ The client IP the check endpoint enforces against comes from `X-Real-Ip`. On the
proxy_set_header X-Real-IP $remote_addr;
```

This overwrites any client-supplied value with the ingress-observed peer, making the header trustworthy on this path (the ENG-105 fix). The IP-constraint control **depends on this snippet** — see the warning comment in `api-key-check.ts`. In the local dev proxy topology (no ingress) `X-Real-Ip` is client-controlled, which is acceptable for dev only.
This overwrites any client-supplied value with the ingress-observed peer, making the header trustworthy on this path (the ENG-105 fix). The IP-constraint control **depends on this snippet**; see the warning comment in `api-key-check.ts`. In the local dev proxy topology (no ingress) `X-Real-Ip` is client-controlled, which is acceptable for dev only.

Matching (`src/domain/api-keys/ip-constraints.ts`, via `ipaddr.js`) handles bare IPs and CIDR ranges, normalizes IPv4-mapped IPv6, treats family mismatches as non-matches, and never throws.

Expand All @@ -99,23 +99,23 @@ Per-key request limiting lives in an express middleware (`src/servers/middleware
- Reads the key's limit from the `rate_limit` JWT claim (falling back to the config default).
- Consumes from a `rate-limiter-flexible` Redis limiter (`src/services/rate-limit/api-keys.ts`), keyed on `keyId`, one limiter instance per distinct limit value.
- On rejection, short-circuits with an HTTP `200` GraphQL error carrying `extensions.code = TOO_MANY_REQUESTS` plus `retryAfterSeconds` and `rateLimit { limit, remaining }`. It also sets `X-RateLimit-*` / `Retry-After` headers best-effort.
- **Fails open** on any internal/Redis error — rate limiting must never take the API down.
- **Fails open** on any internal/Redis error; rate limiting must never take the API down.

**Why a GraphQL error, not an HTTP `429`.** This `/graphql` server is a federation subgraph behind the Apollo router. The router forwards a subgraph response to the client only when it is HTTP `2xx` with a GraphQL body; a bare `429` from the subgraph is swallowed and re-emitted as an opaque `SUBREQUEST_HTTP_ERROR`, and subgraph response headers are dropped. Returning `200` + a GraphQL error with the retry metadata in `extensions` is the only shape that reaches the client intact through the router, and it matches how every other galoy rate limit surfaces (`TOO_MANY_REQUEST`). The `X-RateLimit-*` / `Retry-After` headers are still set for callers that reach the service directly (no router in between). This was confirmed end-to-end against the router: a bare `429` reached the client as `SUBREQUEST_HTTP_ERROR`, while the GraphQL-error form arrives as a clean `TOO_MANY_REQUESTS`.

Config: `apiKeys.defaultRequestsPerMinute` (default 120) and `apiKeys.maxKeysPerAccount` (default 10), read via `src/config/api-keys.ts`.

## Observability

**Metrics (ENG-103)** — prom-client counters on the default registry (`src/services/api-keys-metrics.ts`):
**Metrics (ENG-103)**: prom-client counters on the default registry (`src/services/api-keys-metrics.ts`):

- `galoy_api_key_verification_total{result,reason}` — `result` = `success`|`denied`, `reason` = the denial's error class (or `ok`).
- `galoy_api_key_verification_total{result,reason}`: `result` = `success`|`denied`, `reason` = the denial's error class (or `ok`).
- `galoy_api_key_rate_limited_total`
- `galoy_api_key_management_total{operation,result}` — `operation` = `create`|`revoke`|`rotate`|`list`.
- `galoy_api_key_management_total{operation,result}`: `operation` = `create`|`revoke`|`rotate`|`list`.

Verification, rate limiting, and management all run inside the **API pods** (not the standalone exporter), so a per-pod listener (`src/servers/api-key-metrics.ts`) exposes `/metrics` on `API_METRICS_PORT` (default **3001**), started from the main API entrypoint only — never in the admin/ws/trigger/exporter processes. The API deployment carries `prometheus.io/{scrape,path,port}` annotations for this port.
Verification, rate limiting, and management all run inside the **API pods** (not the standalone exporter), so a per-pod listener (`src/servers/api-key-metrics.ts`) exposes `/metrics` on `API_METRICS_PORT` (default **3001**), started from the main API entrypoint only; never in the admin/ws/trigger/exporter processes. The API deployment carries `prometheus.io/{scrape,path,port}` annotations for this port.

**Audit (ENG-104)** — a pino child logger (`module: api-keys-audit`, `src/services/api-keys-audit.ts`) emits fields-object-first events with a stable `event` discriminator: `api_key.created`, `api_key.revoked`, `api_key.rotated`, `api_key.denied`, `api_key.rate_limited`. **Only the public `keyId` is ever logged** — never the raw key, secret, or hash. These are structured logs (no new DB write path); usage history is intended to ride metrics + logs rather than a per-request DB write.
**Audit (ENG-104)**: a pino child logger (`module: api-keys-audit`, `src/services/api-keys-audit.ts`) emits fields-object-first events with a stable `event` discriminator: `api_key.created`, `api_key.revoked`, `api_key.rotated`, `api_key.denied`, `api_key.rate_limited`. **Only the public `keyId` is ever logged**; never the raw key, secret, or hash. These are structured logs (no new DB write path); usage history is intended to ride metrics + logs rather than a per-request DB write.

## Subscriptions / WebSocket

Expand All @@ -125,8 +125,8 @@ API keys **cannot** reach the GraphQL subscriptions (WS) server: the WS auth pat

The gateway wiring lives in the charts repo (`charts/flash`):

- **`values.yaml`** — the `flash-router` oathkeeper `accessRules`: the added `bearer_token` authenticator (`token_from: X-API-KEY`, `forward_http_headers: [X-API-KEY, X-Real-Ip]`, `force_method: GET`) pointing at `/auth/api-key/check`, and the `id_token` claims template extended with conditional `scope` and `rate_limit` claims.
- **`templates/api-ingress.yaml`** — the `auth-snippet` line forcing `X-Real-IP` from `$remote_addr` on the auth subrequest (ENG-105).
- **`templates/api-deployment.yaml`** — prometheus scrape annotations and the metrics container port for the per-pod API-key metrics listener.
- **`values.yaml`**: the `flash-router` oathkeeper `accessRules`: the added `bearer_token` authenticator (`token_from: X-API-KEY`, `forward_http_headers: [X-API-KEY, X-Real-Ip]`, `force_method: GET`) pointing at `/auth/api-key/check`, and the `id_token` claims template extended with conditional `scope` and `rate_limit` claims.
- **`templates/api-ingress.yaml`**: the `auth-snippet` line forcing `X-Real-IP` from `$remote_addr` on the auth subrequest (ENG-105).
- **`templates/api-deployment.yaml`**: prometheus scrape annotations and the metrics container port for the per-pod API-key metrics listener.

The dev equivalents live in `dev/ory/oathkeeper_rules.yaml` (the `galoy-backend` rule).
Loading
Loading