diff --git a/DEV.md b/DEV.md index 23bc0e021..b5cf89740 100644 --- a/DEV.md +++ b/DEV.md @@ -46,13 +46,13 @@ Flash uses YAML config files. Ibex OAuth2 credentials go in local config overrid The base config is at `dev/config/base-config.yaml`. Secrets and local overrides go in `$CONFIG_PATH/dev-overrides.yaml` (default: `~/.config/flash/dev-overrides.yaml`). -**Option A — Run the interactive script:** +**Option A. Run the interactive script:** ```bash ./dev/config/set-overrides.sh ``` -**Option B — Create manually:** +**Option B. Create manually:** ```yaml # ~/.config/flash/dev-overrides.yaml @@ -151,7 +151,7 @@ This runs four processes in parallel: | Process | Port | Description | |---------|------|-------------| | `start-main` | 4012 (direct), 4002 (via oathkeeper) | Main GraphQL API | -| `start-trigger` | — | Event trigger processor | +| `start-trigger` | none | Event trigger processor | | `start-ws` | 4000 | WebSocket server for subscriptions | | `start-ibex-wh` | 4008 | Ibex webhook receiver | @@ -234,7 +234,7 @@ TEST=01-connection make integration # runs 01-connection.spec.ts ``` **Known issues:** -- Integration tests are not fully idempotent — use `make reset-integration` between runs +- Integration tests are not fully idempotent; use `make reset-integration` between runs - If tests timeout, increase: `JEST_TIMEOUT=120000 yarn test:integration` - Use an SSD for Docker volumes (tests are disk-intensive) @@ -287,11 +287,11 @@ Flash requires Node 24.x. Switch with `nvm use 24` or run `yarn install --ignore ### Docker warnings about unset variables -`IBEX_URL`, `HONEYCOMB_DATASET`, `HONEYCOMB_API_KEY` warnings in docker compose output are harmless in dev — these are only needed for the Docker-based server (not used by `make start`). +`IBEX_URL`, `HONEYCOMB_DATASET`, `HONEYCOMB_API_KEY` warnings in docker compose output are harmless in dev; these are only needed for the Docker-based server (not used by `make start`). ### `API key does not start with "SG."` -SendGrid isn't configured — email notifications won't work. Safe to ignore in dev unless you're testing email features. Add a real key to your `dev-overrides.yaml` if needed. +SendGrid isn't configured; email notifications won't work. Safe to ignore in dev unless you're testing email features. Add a real key to your `dev-overrides.yaml` if needed. ### Price service platform warning (Apple Silicon) @@ -299,7 +299,7 @@ SendGrid isn't configured — email notifications won't work. Safe to ignore in The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) ``` -The `lnflash/price:edge` image is amd64-only. On Apple Silicon Macs it runs under Rosetta emulation — this is harmless but slow. The warning is safe to ignore. +The `lnflash/price:edge` image is amd64-only. On Apple Silicon Macs it runs under Rosetta emulation; this is harmless but slow. The warning is safe to ignore. ### Server starts but crashes immediately diff --git a/docs/account-capabilities.md b/docs/account-capabilities.md index a78a720d7..9a11a680b 100644 --- a/docs/account-capabilities.md +++ b/docs/account-capabilities.md @@ -12,14 +12,14 @@ builds on the state machine defined here. | L1 | Personal | **Verified** | | L2 | Pro | **Verified** + *Bank payout* badge | | L3 | Merchant | **Business** | -| — | International | *USD account* badge (capability, not a tier) | +| none | International | *USD account* badge (capability, not a tier) | - "Pro" is retired (vague). "International" is retired (it's just a USD account). "Merchant" is renamed **Business**. -- Levels (L1/L2/L3) are internal-only — derived from capabilities, never +- Levels (L1/L2/L3) are internal-only; derived from capabilities, never shown as a tier the user picks. - Product decision (2026-07-14): **light headline status**. The account leads - with one word — Trial → Verified → Business — with capability badges as + with one word, Trial → Verified → Business, with capability badges as supporting detail (not the pure capability-badge approach). ## Capability model @@ -50,7 +50,7 @@ Implementation: `src/domain/accounts/capabilities.ts` operational value used by limits and permissions; transitions recompute it through the state machine. -> **Note — derived level vs stored `level`.** The read model can imply a +> **Note:** derived level vs stored `level`. The read model can imply a > level higher than the stored one: an L1 account with an approved bank > account on file resolves `bankPayout: true`, which derives to L2. Both the > stored `level` and the derived `capabilities`/`statusHeadline` are exposed @@ -120,7 +120,7 @@ mutation { The server derives the target level from current capabilities plus the requested one and posts the existing ERPNext **Account Upgrade Request** -doctype — the human review flow in frappe-flash-admin is unchanged. +doctype; the human review flow in frappe-flash-admin is unchanged. `businessAccountUpgradeRequest` (whole-tier, takes a `level`) remains for existing clients but is superseded by this mutation. diff --git a/docs/api-keys/ARCHITECTURE.md b/docs/api-keys/ARCHITECTURE.md index 8a3667eee..699f8b33e 100644 --- a/docs/api-keys/ARCHITECTURE.md +++ b/docs/api-keys/ARCHITECTURE.md @@ -1,12 +1,12 @@ -# API Keys — Architecture +# API Keys: Architecture Operator and maintainer view of the FIP-07 API-key authentication system. For the developer-facing surface see [USAGE.md](./USAGE.md). -## Design principle: verify at the gateway, not in the app +## Design principle: verify at the gateway Flash's public GraphQL API sits behind Ory Oathkeeper in **decision mode**: the nginx ingress issues an `auth_request` subrequest to oathkeeper's `/decisions` endpoint, oathkeeper authenticates the request and mints an `id_token` JWT, and nginx **replaces** the client's `Authorization` header with that JWT before proxying to the backend (`auth-response-headers: Authorization`). The backend's `expressjwt` layer then only ever trusts an oathkeeper-signed JWT. -Because of this, a raw `fk_…` key placed in `Authorization` could never reach the backend — it would be rejected as a non-JWT bearer before any application code ran. So API-key verification is done **at the gateway**, mirroring Blink's production pattern, and the key travels in a **separate `X-API-KEY` header** so it never collides with the Kratos session path. (This is the ADR recorded under ENG-107.) +Because of this, a raw `fk_…` key placed in `Authorization` could never reach the backend; it would be rejected as a non-JWT bearer before any application code ran. So API-key verification is done **at the gateway**, mirroring Blink's production pattern, and the key travels in a **separate `X-API-KEY` header** so it never collides with the Kratos session path. (This is the ADR recorded under ENG-107.) ## Request flow @@ -49,7 +49,7 @@ It reads `X-API-KEY`, then `verifyApiKey` (`src/app/api-keys/verify-api-key.ts`) 4. Compare `sha256(secret)` against the stored hash in **constant time** (`timingSafeEqual`). 5. Enforce IP constraints (see below), fail-closed. 6. Resolve the owning account's Kratos user id. -7. Throttled `lastUsedAt` update (≤ once per key per minute — keeps verification read-only on hot keys). +7. Throttled `lastUsedAt` update (≤ once per key per minute; keeps verification read-only on hot keys). On success it returns a **kratos-whoami-shaped** body so oathkeeper's existing `subject_from`/`extra_from` config and `id_token` mutator apply unchanged: @@ -68,14 +68,14 @@ On success it returns a **kratos-whoami-shaped** body so oathkeeper's existing ` - `expires_at` is always empty: key expiry is enforced here on every verification, and a value would make session handling try to Kratos-extend the synthetic session id. - `scope` (space-delimited) and `rate_limit` (numeric) become JWT claims via conditional segments in the mutator template. -Any failure returns `401 {"error":"invalid_api_key"}`, which oathkeeper surfaces as an explicit gateway `401` — the same result as an invalid Kratos token. +Any failure returns `401 {"error":"invalid_api_key"}`, which oathkeeper surfaces as an explicit gateway `401`; the same result as an invalid Kratos token. ## Scope enforcement (graphql-shield) Scopes are enforced in-process with **graphql-shield**, deny-by-default, from `src/domain/api-keys/scope-map.ts`: -- **`apiKeyScopeForField`** — every authed root query/mutation mapped to the scope it requires, or `BLOCKED`. Non-API-key sessions pass through untouched; for an API-key session the shield rule requires the mapped scope, and any field **not** in the map (or mapped `BLOCKED`) is rejected. A completeness unit test fails CI if a new authed field is added without a mapping, so the deny-by-default guarantee can't silently regress. -- **`apiKeyNestedFieldScopes`** — type-level guards (`ConsumerAccount`, `UserContact`, `BTCWallet`, `UsdWallet`, `UsdtWallet`) so wallet/transaction data reachable *through* a low-scope root field (e.g. `me`, which only needs `read:user`) still requires `read:wallet` / `read:transactions`. This closes the nested-escalation path. +- **`apiKeyScopeForField`**: every authed root query/mutation mapped to the scope it requires, or `BLOCKED`. Non-API-key sessions pass through untouched; for an API-key session the shield rule requires the mapped scope, and any field **not** in the map (or mapped `BLOCKED`) is rejected. A completeness unit test fails CI if a new authed field is added without a mapping, so the deny-by-default guarantee can't silently regress. +- **`apiKeyNestedFieldScopes`**: type-level guards (`ConsumerAccount`, `UserContact`, `BTCWallet`, `UsdWallet`, `UsdtWallet`) so wallet/transaction data reachable *through* a low-scope root field (e.g. `me`, which only needs `read:user`) still requires `read:wallet` / `read:transactions`. This closes the nested-escalation path. `hasApiKeyScope` implements the implication rules: `admin` grants everything, `write:X` implies `read:X`. Scope claims arrive on the GraphQL context via `session.ts`, which splits the space-delimited `scope` JWT claim into `ctx.scopes`. @@ -87,7 +87,7 @@ The client IP the check endpoint enforces against comes from `X-Real-Ip`. On the proxy_set_header X-Real-IP $remote_addr; ``` -This overwrites any client-supplied value with the ingress-observed peer, making the header trustworthy on this path (the ENG-105 fix). The IP-constraint control **depends on this snippet** — see the warning comment in `api-key-check.ts`. In the local dev proxy topology (no ingress) `X-Real-Ip` is client-controlled, which is acceptable for dev only. +This overwrites any client-supplied value with the ingress-observed peer, making the header trustworthy on this path (the ENG-105 fix). The IP-constraint control **depends on this snippet**; see the warning comment in `api-key-check.ts`. In the local dev proxy topology (no ingress) `X-Real-Ip` is client-controlled, which is acceptable for dev only. Matching (`src/domain/api-keys/ip-constraints.ts`, via `ipaddr.js`) handles bare IPs and CIDR ranges, normalizes IPv4-mapped IPv6, treats family mismatches as non-matches, and never throws. @@ -99,7 +99,7 @@ Per-key request limiting lives in an express middleware (`src/servers/middleware - Reads the key's limit from the `rate_limit` JWT claim (falling back to the config default). - Consumes from a `rate-limiter-flexible` Redis limiter (`src/services/rate-limit/api-keys.ts`), keyed on `keyId`, one limiter instance per distinct limit value. - On rejection, short-circuits with an HTTP `200` GraphQL error carrying `extensions.code = TOO_MANY_REQUESTS` plus `retryAfterSeconds` and `rateLimit { limit, remaining }`. It also sets `X-RateLimit-*` / `Retry-After` headers best-effort. -- **Fails open** on any internal/Redis error — rate limiting must never take the API down. +- **Fails open** on any internal/Redis error; rate limiting must never take the API down. **Why a GraphQL error, not an HTTP `429`.** This `/graphql` server is a federation subgraph behind the Apollo router. The router forwards a subgraph response to the client only when it is HTTP `2xx` with a GraphQL body; a bare `429` from the subgraph is swallowed and re-emitted as an opaque `SUBREQUEST_HTTP_ERROR`, and subgraph response headers are dropped. Returning `200` + a GraphQL error with the retry metadata in `extensions` is the only shape that reaches the client intact through the router, and it matches how every other galoy rate limit surfaces (`TOO_MANY_REQUEST`). The `X-RateLimit-*` / `Retry-After` headers are still set for callers that reach the service directly (no router in between). This was confirmed end-to-end against the router: a bare `429` reached the client as `SUBREQUEST_HTTP_ERROR`, while the GraphQL-error form arrives as a clean `TOO_MANY_REQUESTS`. @@ -107,15 +107,15 @@ Config: `apiKeys.defaultRequestsPerMinute` (default 120) and `apiKeys.maxKeysPer ## Observability -**Metrics (ENG-103)** — prom-client counters on the default registry (`src/services/api-keys-metrics.ts`): +**Metrics (ENG-103)**: prom-client counters on the default registry (`src/services/api-keys-metrics.ts`): -- `galoy_api_key_verification_total{result,reason}` — `result` = `success`|`denied`, `reason` = the denial's error class (or `ok`). +- `galoy_api_key_verification_total{result,reason}`: `result` = `success`|`denied`, `reason` = the denial's error class (or `ok`). - `galoy_api_key_rate_limited_total` -- `galoy_api_key_management_total{operation,result}` — `operation` = `create`|`revoke`|`rotate`|`list`. +- `galoy_api_key_management_total{operation,result}`: `operation` = `create`|`revoke`|`rotate`|`list`. -Verification, rate limiting, and management all run inside the **API pods** (not the standalone exporter), so a per-pod listener (`src/servers/api-key-metrics.ts`) exposes `/metrics` on `API_METRICS_PORT` (default **3001**), started from the main API entrypoint only — never in the admin/ws/trigger/exporter processes. The API deployment carries `prometheus.io/{scrape,path,port}` annotations for this port. +Verification, rate limiting, and management all run inside the **API pods** (not the standalone exporter), so a per-pod listener (`src/servers/api-key-metrics.ts`) exposes `/metrics` on `API_METRICS_PORT` (default **3001**), started from the main API entrypoint only; never in the admin/ws/trigger/exporter processes. The API deployment carries `prometheus.io/{scrape,path,port}` annotations for this port. -**Audit (ENG-104)** — a pino child logger (`module: api-keys-audit`, `src/services/api-keys-audit.ts`) emits fields-object-first events with a stable `event` discriminator: `api_key.created`, `api_key.revoked`, `api_key.rotated`, `api_key.denied`, `api_key.rate_limited`. **Only the public `keyId` is ever logged** — never the raw key, secret, or hash. These are structured logs (no new DB write path); usage history is intended to ride metrics + logs rather than a per-request DB write. +**Audit (ENG-104)**: a pino child logger (`module: api-keys-audit`, `src/services/api-keys-audit.ts`) emits fields-object-first events with a stable `event` discriminator: `api_key.created`, `api_key.revoked`, `api_key.rotated`, `api_key.denied`, `api_key.rate_limited`. **Only the public `keyId` is ever logged**; never the raw key, secret, or hash. These are structured logs (no new DB write path); usage history is intended to ride metrics + logs rather than a per-request DB write. ## Subscriptions / WebSocket @@ -125,8 +125,8 @@ API keys **cannot** reach the GraphQL subscriptions (WS) server: the WS auth pat The gateway wiring lives in the charts repo (`charts/flash`): -- **`values.yaml`** — the `flash-router` oathkeeper `accessRules`: the added `bearer_token` authenticator (`token_from: X-API-KEY`, `forward_http_headers: [X-API-KEY, X-Real-Ip]`, `force_method: GET`) pointing at `/auth/api-key/check`, and the `id_token` claims template extended with conditional `scope` and `rate_limit` claims. -- **`templates/api-ingress.yaml`** — the `auth-snippet` line forcing `X-Real-IP` from `$remote_addr` on the auth subrequest (ENG-105). -- **`templates/api-deployment.yaml`** — prometheus scrape annotations and the metrics container port for the per-pod API-key metrics listener. +- **`values.yaml`**: the `flash-router` oathkeeper `accessRules`: the added `bearer_token` authenticator (`token_from: X-API-KEY`, `forward_http_headers: [X-API-KEY, X-Real-Ip]`, `force_method: GET`) pointing at `/auth/api-key/check`, and the `id_token` claims template extended with conditional `scope` and `rate_limit` claims. +- **`templates/api-ingress.yaml`**: the `auth-snippet` line forcing `X-Real-IP` from `$remote_addr` on the auth subrequest (ENG-105). +- **`templates/api-deployment.yaml`**: prometheus scrape annotations and the metrics container port for the per-pod API-key metrics listener. The dev equivalents live in `dev/ory/oathkeeper_rules.yaml` (the `galoy-backend` rule). diff --git a/docs/api-keys/README.md b/docs/api-keys/README.md index 3b26dd503..8d9795d11 100644 --- a/docs/api-keys/README.md +++ b/docs/api-keys/README.md @@ -2,8 +2,8 @@ API keys give **server-to-server and third-party developer applications** programmatic access to the Flash public GraphQL API, without a mobile-app login (Kratos) session. They are the credential a BTCPayServer plugin, a back-office script, or a partner integration uses to call Flash on behalf of a single account. -- **[USAGE.md](./USAGE.md)** — developer guide: creating, using, listing, rotating, and revoking keys; the scope model; rate-limit, IP, and error behavior. -- **[ARCHITECTURE.md](./ARCHITECTURE.md)** — operator/maintainer guide: the gateway design, verification, scope enforcement, IP handling, rate limiting, metrics, and audit. +- **[USAGE.md](./USAGE.md)**: developer guide: creating, using, listing, rotating, and revoking keys; the scope model; rate-limit, IP, and error behavior. +- **[ARCHITECTURE.md](./ARCHITECTURE.md)**: operator/maintainer guide: the gateway design, verification, scope enforcement, IP handling, rate limiting, metrics, and audit. ## Key format @@ -11,8 +11,8 @@ API keys give **server-to-server and third-party developer applications** progra fk_{keyId}_{secret} ``` -- `keyId` — 8 lowercase hex characters. A **public** identifier that appears in the API-key list, audit logs, and metrics. It is *not* a secret. -- `secret` — 64 base64url characters, generated from 48 random bytes. +- `keyId`: 8 lowercase hex characters. A **public** identifier that appears in the API-key list, audit logs, and metrics. It is *not* a secret. +- `secret`: 64 base64url characters, generated from 48 random bytes. Only the **SHA-256 hash of the secret** is stored. The full `fk_…` string is returned exactly once, at creation (and again on rotation), and can never be retrieved afterward. If it is lost, rotate or revoke and issue a new key. @@ -28,13 +28,13 @@ Only the **SHA-256 hash of the secret** is stored. The full `fk_…` string is r ## Security properties -- **Hash-only storage** — the raw secret is never persisted; verification compares a SHA-256 hash in constant time. -- **Fine-grained scopes** — a key is limited to the FIP-07 scopes granted to it (`read:wallet`, `write:wallet`, `read:transactions`, `write:transactions`, `read:user`, `write:user`, `admin`), enforced deny-by-default. Sensitive operations (login/identity, KYC, cashout, and key management itself) are never callable by any key. -- **IP constraints** — a key may be pinned to one or more IPs/CIDR ranges; requests from other addresses are rejected (fail-closed). -- **Per-key rate limits** — each key has a requests-per-minute limit (platform default, overridable per key); exceeding it returns a `TOO_MANY_REQUESTS` GraphQL error with retry metadata. -- **Expiry** — a key may be given a lifetime; expired keys stop authenticating and report status `EXPIRED`. -- **Revocation** — revoking a key stops it being honored on the very next request. +- **Hash-only storage**: the raw secret is never persisted; verification compares a SHA-256 hash in constant time. +- **Fine-grained scopes**: a key is limited to the FIP-07 scopes granted to it (`read:wallet`, `write:wallet`, `read:transactions`, `write:transactions`, `read:user`, `write:user`, `admin`), enforced deny-by-default. Sensitive operations (login/identity, KYC, cashout, and key management itself) are never callable by any key. +- **IP constraints**: a key may be pinned to one or more IPs/CIDR ranges; requests from other addresses are rejected (fail-closed). +- **Per-key rate limits**: each key has a requests-per-minute limit (platform default, overridable per key); exceeding it returns a `TOO_MANY_REQUESTS` GraphQL error with retry metadata. +- **Expiry**: a key may be given a lifetime; expired keys stop authenticating and report status `EXPIRED`. +- **Revocation**: revoking a key stops it being honored on the very next request. ## Keys cannot manage keys -Creating, listing, rotating, and revoking API keys all require a **Kratos session** (i.e. a logged-in user at account level). A request authenticated *with an API key* is rejected from every management operation. This prevents a leaked key from minting more keys or escalating its own privileges — key management is always a human-in-the-loop action. +Creating, listing, rotating, and revoking API keys all require a **Kratos session** (i.e. a logged-in user at account level). A request authenticated *with an API key* is rejected from every management operation. This prevents a leaked key from minting more keys or escalating its own privileges; key management is always a human-in-the-loop action. diff --git a/docs/api-keys/USAGE.md b/docs/api-keys/USAGE.md index 20851a87c..c1e7e01ba 100644 --- a/docs/api-keys/USAGE.md +++ b/docs/api-keys/USAGE.md @@ -1,4 +1,4 @@ -# API Keys — Developer Guide +# API Keys: Developer Guide How to create and use a Flash API key against the public GraphQL API. @@ -6,7 +6,7 @@ How to create and use a Flash API key against the public GraphQL API. ## 1. Create a key -Key management requires a **logged-in user session** (a Kratos session — see the login flow in [`spectaql/TUTORIAL.md`](../../spectaql/TUTORIAL.md)). You cannot create a key with another API key. +Key management requires a **logged-in user session** (a Kratos session; see the login flow in [`spectaql/TUTORIAL.md`](../../spectaql/TUTORIAL.md)). You cannot create a key with another API key. ```graphql mutation ApiKeyCreate { @@ -37,7 +37,7 @@ mutation ApiKeyCreate { > **GraphQL enum names use underscores.** The `ApiKeyScope` enum values in a query are `read_wallet`, `write_wallet`, `read_transactions`, `write_transactions`, `read_user`, `write_user`, `admin`. Their canonical FIP-07 form (`read:wallet`, …) is what appears in scope error messages and on the wire to the gateway. -**Response** — the `apiKey.apiKey` field holds the full `fk_{keyId}_{secret}` string. It is returned only here. Store it in your secret manager immediately; Flash cannot show it again. `warning` restates this. +**Response**: the `apiKey.apiKey` field holds the full `fk_{keyId}_{secret}` string. It is returned only here. Store it in your secret manager immediately; Flash cannot show it again. `warning` restates this. Defaults and limits: - `scopes` defaults to `[read_user]` if omitted. @@ -55,7 +55,7 @@ curl -X POST http://localhost:4002/graphql \ -d '{"query":"{ me { defaultAccount { wallets { walletCurrency balance } } } }"}' ``` -A valid key resolves to its owning account, exactly as a session would — subject to the key's scopes. +A valid key resolves to its owning account, exactly as a session would; subject to the key's scopes. ## 3. Scopes @@ -64,21 +64,21 @@ Scopes are enforced **deny-by-default**: an API key may only reach operations ex | Scope | Grants (representative operations) | |---|---| | `read:user` | `me` and profile/account reads | -| `write:user` | profile mutations — `userUpdateUsername`, `userUpdateLanguage`, `accountUpdateDefaultWalletId`, `accountUpdateDisplayCurrency`, notification settings | -| `read:wallet` | wallet balances and fee estimates — wallet `balance`/`pendingIncomingBalance`, `onChainTxFee`, `ln*InvoiceFeeProbe`, on-chain address reads | -| `write:wallet` | money movement and invoicing — `lnInvoiceCreate`, `lnInvoicePaymentSend`, `intraLedgerPaymentSend`, `onChainPaymentSend`, `lnurlPaymentSend`, etc. | -| `read:transactions` | transaction history — `transactionDetails`, wallet `transactions`/`transactionsByAddress`, contact `transactions`/`transactionsCount`, CSV export | +| `write:user` | profile mutations, `userUpdateUsername`, `userUpdateLanguage`, `accountUpdateDefaultWalletId`, `accountUpdateDisplayCurrency`, notification settings | +| `read:wallet` | wallet balances and fee estimates, wallet `balance`/`pendingIncomingBalance`, `onChainTxFee`, `ln*InvoiceFeeProbe`, on-chain address reads | +| `write:wallet` | money movement and invoicing, `lnInvoiceCreate`, `lnInvoicePaymentSend`, `intraLedgerPaymentSend`, `onChainPaymentSend`, `lnurlPaymentSend`, etc. | +| `read:transactions` | transaction history, `transactionDetails`, wallet `transactions`/`transactionsByAddress`, contact `transactions`/`transactionsCount`, CSV export | | `write:transactions` | reserved for transaction-creating operations | | `admin` | all of the above | Two implication rules apply: -- **`write:X` implies `read:X`** — a key with `write:wallet` can also perform `read:wallet` operations. +- **`write:X` implies `read:X`**: a key with `write:wallet` can also perform `read:wallet` operations. - **`admin` implies every scope.** **Never available to API keys**, regardless of scope: login/identity (`userLogin*`, email/phone/TOTP registration and deletion), device notification tokens, KYC and Bridge fiat rails, cashout, account deletion, feedback, and **API-key management itself**. These require a user session. -Nested data is guarded too: `me` only needs `read:user`, but reaching wallet balances or transactions through it additionally requires `read:wallet` / `read:transactions` respectively — a `read:user`-only key cannot escalate to financial data. +Nested data is guarded too: `me` only needs `read:user`, but reaching wallet balances or transactions through it additionally requires `read:wallet` / `read:transactions` respectively; a `read:user`-only key cannot escalate to financial data. ## 4. Rate limiting @@ -100,7 +100,7 @@ Every key has a requests-per-minute budget (its `rateLimitPerMinute`, or the pla } ``` -Clients should branch on `errors[].extensions.code === "TOO_MANY_REQUESTS"` and back off for `retryAfterSeconds`. The public API is served through the Apollo federation router, which forwards subgraph GraphQL errors but not subgraph HTTP status codes or response headers — so the machine-readable signal lives in the error `extensions`, not in a `429` status or `X-RateLimit-*` / `Retry-After` headers. (Those headers are still set best-effort on the backend response and are visible only to callers that reach the API service directly, bypassing the router.) +Clients should branch on `errors[].extensions.code === "TOO_MANY_REQUESTS"` and back off for `retryAfterSeconds`. The public API is served through the Apollo federation router, which forwards subgraph GraphQL errors but not subgraph HTTP status codes or response headers; so the machine-readable signal lives in the error `extensions`, not in a `429` status or `X-RateLimit-*` / `Retry-After` headers. (Those headers are still set best-effort on the backend response and are visible only to callers that reach the API service directly, bypassing the router.) ## 5. IP constraints @@ -114,7 +114,7 @@ If `expiresIn` was set, the key stops authenticating once past its expiry and re All three require a **user session** (not an API key). -**List** — never returns secret material: +**List**: never returns secret material: ```graphql query ApiKeys { @@ -132,7 +132,7 @@ query ApiKeys { } ``` -**Rotate** — issues a replacement (new `keyId` + secret) carrying the **same** name, scopes, IP constraints, rate limit, and expiry, and revokes the old key. The new raw key is returned once: +**Rotate**: issues a replacement (new `keyId` + secret) carrying the **same** name, scopes, IP constraints, rate limit, and expiry, and revokes the old key. The new raw key is returned once: ```graphql mutation ApiKeyRotate { @@ -143,7 +143,7 @@ mutation ApiKeyRotate { } ``` -**Revoke** — irreversibly disables the key; verification stops honoring it on the next request: +**Revoke**: irreversibly disables the key; verification stops honoring it on the next request: ```graphql mutation ApiKeyRevoke { @@ -154,7 +154,7 @@ mutation ApiKeyRevoke { } ``` -Both rotate and revoke are scoped to the caller's own account — a key id belonging to another account reads as not-found. +Both rotate and revoke are scoped to the caller's own account; a key id belonging to another account reads as not-found. ## Error reference @@ -178,4 +178,4 @@ API keys surface two error channels. |---|---| | Missing / malformed / unknown / revoked / expired / IP-rejected key | `401` `{ "error": "invalid_api_key" }` | -A malformed, unknown, revoked, expired, or IP-rejected key produces a gateway `401` — indistinguishable to the caller (by design, to avoid leaking which keys exist). Once a key authenticates, everything else (scope denials, rate limiting, validation) comes back as a GraphQL error with an `extensions.code`, not an HTTP status — the rate-limit case is `TOO_MANY_REQUESTS` (see §4). +A malformed, unknown, revoked, expired, or IP-rejected key produces a gateway `401`; indistinguishable to the caller (by design, to avoid leaking which keys exist). Once a key authenticates, everything else (scope denials, rate limiting, validation) comes back as a GraphQL error with an `extensions.code`, not an HTTP status; the rate-limit case is `TOO_MANY_REQUESTS` (see §4). diff --git a/docs/bridge-integration/ALERTING.md b/docs/bridge-integration/ALERTING.md index 270621a8b..cb474da0e 100644 --- a/docs/bridge-integration/ALERTING.md +++ b/docs/bridge-integration/ALERTING.md @@ -10,9 +10,9 @@ destinations. | Severity | PagerDuty (page) | Slack / Mattermost (inform) | Discord (inform) | | ------------ | :--------------: | :-------------------------: | :--------------: | | **critical** | ✅ | ✅ | ✅ | -| **warning** | — | ✅ | ✅ | +| **warning** | none | ✅ | ✅ | -Delivery is best-effort and fire-and-forget — a failing or unconfigured +Delivery is best-effort and fire-and-forget; a failing or unconfigured destination never blocks or fails the webhook/request path. **A destination with no configured credential is silently skipped**, so channels can be enabled incrementally. @@ -28,13 +28,13 @@ Alerts carry a stable `dedupKey` so repeated failures do not spam on-call or cha Key classes (see `src/services/alerts/dedup-key.ts`): -- `bridge-api:5xx` / `bridge-api:timeout` / `bridge-api:network` — coarse outage keys (30 min inform TTL) -- `erpnext-audit:deposit:{transfer_id}` — per deposit audit failure (1 h inform TTL) -- `erpnext-audit:transfer-complete:{transfer_id}` / `transfer-failed:{transfer_id}` — per transfer audit failure -- `bridge-webhook:deposit:{event_id}` / `bridge-webhook:transfer:{transfer_id}:{event}` — per webhook processing error -- `ibex:crypto-receive:{tx_hash}` — per IBEX crypto receive webhook failure (1 h inform TTL) -- `ibex:reconcile:bridge-without-ibex:{tx_hash}` / `ibex:reconcile:ibex-without-bridge:{tx_hash}` — per reconciliation orphan -- `ibex:reconcile:failed:{tx_hash}` — reconciliation handler threw +- `bridge-api:5xx` / `bridge-api:timeout` / `bridge-api:network`; coarse outage keys (30 min inform TTL) +- `erpnext-audit:deposit:{transfer_id}`: per deposit audit failure (1 h inform TTL) +- `erpnext-audit:transfer-complete:{transfer_id}` / `transfer-failed:{transfer_id}`; per transfer audit failure +- `bridge-webhook:deposit:{event_id}` / `bridge-webhook:transfer:{transfer_id}:{event}`; per webhook processing error +- `ibex:crypto-receive:{tx_hash}`: per IBEX crypto receive webhook failure (1 h inform TTL) +- `ibex:reconcile:bridge-without-ibex:{tx_hash}` / `ibex:reconcile:ibex-without-bridge:{tx_hash}`; per reconciliation orphan +- `ibex:reconcile:failed:{tx_hash}`: reconciliation handler threw Inform dedup is in-process per pod; PagerDuty dedup is global to the service integration. @@ -44,7 +44,7 @@ Inform dedup is in-process per pod; PagerDuty dedup is global to the service int | ------------------------------------------------- | -------- | ----------------------------------------------------------- | | ERPNext audit-write failure (deposit + transfer) | critical | `services/bridge/webhook-server/routes/{deposit,transfer}.ts` | | Bridge webhook processing exception | critical | same routes (catch block) | -| Bridge API outage — 5xx / timeout / network | critical | `services/bridge/client.ts` | +| Bridge API outage, 5xx / timeout / network | critical | `services/bridge/client.ts` | | IBEX crypto receive webhook failure | warning | `services/ibex/webhook-server/routes/crypto-receive.ts` | | Bridge↔IBEX reconciliation orphan / failure | warning | `services/bridge/reconciliation.ts`, deposit/crypto catch | @@ -62,27 +62,27 @@ Three optional env vars, each gating one destination: ### How to get each value -**PagerDuty** — `ALERT_PAGERDUTY_ROUTING_KEY` +**PagerDuty**: `ALERT_PAGERDUTY_ROUTING_KEY` 1. PagerDuty → **Services** → pick (or create) the service that should page for Bridge. 2. **Integrations** → **Add integration** → **Events API v2**. -3. Copy the **Integration Key** — that is the routing key. +3. Copy the **Integration Key**; that is the routing key. -**Slack** — `ALERT_SLACK_WEBHOOK_URL` +**Slack**: `ALERT_SLACK_WEBHOOK_URL` 1. Create/choose a Slack app → **Incoming Webhooks** → **Activate**. 2. **Add New Webhook to Workspace** → choose the target channel. 3. Copy the URL (`https://hooks.slack.com/services/...`). - _Mattermost works too_ — it accepts the same `{ text }` payload; use its incoming-webhook URL. + _Mattermost works too_; it accepts the same `{ text }` payload; use its incoming-webhook URL. -**Discord** — `ALERT_DISCORD_WEBHOOK_URL` +**Discord**: `ALERT_DISCORD_WEBHOOK_URL` 1. Discord → target channel → **Edit Channel** → **Integrations** → **Webhooks**. 2. **New Webhook** → name it → **Copy Webhook URL**. ### Where to set them - **Local dev:** add to `.env` (and `.env.ci` for CI). -- **Staging / production:** set as environment variables / secrets in the deployment — the same place `MATTERMOST_WEBHOOK_URL` is configured. Treat all three as **secrets**. +- **Staging / production:** set as environment variables / secrets in the deployment; the same place `MATTERMOST_WEBHOOK_URL` is configured. Treat all three as **secrets**. -> If none are set, alerting is a no-op (no errors, no delivery) — useful until the channels are provisioned. +> If none are set, alerting is a no-op (no errors, no delivery); useful until the channels are provisioned. ## Verifying in staging (ENG-361 acceptance) diff --git a/docs/bridge-integration/API.md b/docs/bridge-integration/API.md index 3c919862c..b59c3dfab 100644 --- a/docs/bridge-integration/API.md +++ b/docs/bridge-integration/API.md @@ -86,7 +86,7 @@ mutation BridgeAddExternalAccount { ### `bridgeExchangePlaidPublicToken` -Exchanges the Plaid `publicToken` (from Link `onSuccess`) with Bridge. Call from the app backend path only — Flash keeps the Bridge Api-Key server-side. External accounts are created asynchronously and arrive via webhook. +Exchanges the Plaid `publicToken` (from Link `onSuccess`) with Bridge. Call from the app backend path only; Flash keeps the Bridge Api-Key server-side. External accounts are created asynchronously and arrive via webhook. **Request:** ```graphql @@ -136,7 +136,7 @@ mutation BridgeRequestWithdrawal($input: BridgeRequestWithdrawalInput!) { - `externalAccountId`: The ID of the linked bank account. **Response:** -- `id`: MongoDB withdrawal record ID — pass this to `bridgeInitiateWithdrawal` or `bridgeCancelWithdrawalRequest`. +- `id`: MongoDB withdrawal record ID; pass this to `bridgeInitiateWithdrawal` or `bridgeCancelWithdrawalRequest`. - `status`: Always `"pending"` on success. - `externalAccountId`: Linked bank account used for the withdrawal. diff --git a/docs/id-verification.md b/docs/id-verification.md index 46bcafadf..9465778d1 100644 --- a/docs/id-verification.md +++ b/docs/id-verification.md @@ -1,4 +1,4 @@ -# ID verification — Phase 0 (backend) +# ID verification: Phase 0 (backend) The backend half of Flash's ID-verification tool. Phase 0 adds structured identity evidence to the account-upgrade request, records a companion @@ -13,8 +13,8 @@ only `idDocument`, keeps working with no behaviour change. ## 1. Evidence input contract -Both upgrade mutations — `businessAccountUpgradeRequest` and -`accountCapabilityUpgradeRequest` — accept an optional `evidence` list next +Both upgrade mutations, `businessAccountUpgradeRequest` and +`accountCapabilityUpgradeRequest`, accept an optional `evidence` list next to the existing `idDocument` key: ```graphql @@ -46,7 +46,7 @@ input BusinessAccountUpgradeRequestInput { Files are uploaded exactly as today: `idDocumentUploadUrlGenerate` returns a 15-minute presigned PUT and the object key `id_documents//` -(the `/` separator is deliberate — usernames may contain `_`, so only a +(the `/` separator is deliberate; usernames may contain `_`, so only a character `UsernameRegex` forbids can unambiguously mark the boundary); the client then references that key from an evidence row. @@ -95,7 +95,7 @@ minimum supported app version. ## 2. Bridge KYC rule When the account's Bridge KYC is approved and the evidence contains a -`BRIDGE_KYC` row — including the fallback row above — the backend snapshots +`BRIDGE_KYC` row, including the fallback row above, the backend snapshots the Bridge customer via `BridgeApiClient.getCustomer` and stores **only** `id`, `status`, `updated_at` and `endorsements` (`src/services/bridge/customer-snapshot.ts`). Names, emails and addresses @@ -177,8 +177,8 @@ logs a warning carrying the kind and country only. Rows read back from ERPNext already hold the code and country name, so the retention job's rewrite round-trips unchanged. -**Non-fatal by design.** If the POST fails — an older ERPNext without the -doctype, a validation error on the ERPNext side, a network blip — the error +**Non-fatal by design.** If the POST fails (an older ERPNext without the +doctype, a validation error on the ERPNext side, a network blip), the error is logged at **warn** (with identifiers only) and the user's upgrade request succeeds anyway. The Account Upgrade Request already carries the first `ID_FRONT` key in `id_document`, so reviewers lose nothing they have today. @@ -232,7 +232,7 @@ hold only `Flash Admin` could open the request but not finish the approval, because the approval flow calls three admin fields. `Flash Admin` is now a recognised role (`src/services/frappe/Roles.ts`) and -is added — via an explicit allowlist, never a blanket widening — on exactly +is added, via an explicit allowlist, never a blanket widening, on exactly those fields (`src/servers/authorization/admin-permissions.ts`): | Field | Kind | Why the reviewer needs it | @@ -281,7 +281,7 @@ entries carry `updatedAt` (Mongoose default `Date.now`, required), so the closure timestamp *is* derivable without a schema change: `getAccountClosedAt(account)` returns the `updatedAt` of the latest `closed` entry, but only while the account's current status is still -`closed` — a reopened account restarts the clock. `locked` is not closure +`closed`; a reopened account restarts the clock. `locked` is not closure (an admin lock is reversible and the relationship continues). No new field was added to the account schema. @@ -300,7 +300,7 @@ Runs as a task in the existing cron server (`src/servers/cron.ts`, (`deleteIdDocument`, which refuses any key outside `id_documents/`) and stamp `deleted_at` on the row. Frappe has no PATCH for a child row; the job PUTs the parent's full `evidence` table with the stamped rows - (`ErpNext.updateIdVerificationEvidence`) — rows missing from that PUT + (`ErpNext.updateIdVerificationEvidence`); rows missing from that PUT would be dropped, so the whole table is always sent. Per-record failures are counted and logged, never fatal; only a listing diff --git a/docs/plans/2026-06-05-eng-406-usdt-lnurl-units.md b/docs/plans/2026-06-05-eng-406-usdt-lnurl-units.md index 661a6ff55..531fc4d91 100644 --- a/docs/plans/2026-06-05-eng-406-usdt-lnurl-units.md +++ b/docs/plans/2026-06-05-eng-406-usdt-lnurl-units.md @@ -1,21 +1,21 @@ # Fix USDT LNURL-pay Amount Units -> **ENG-406** — Fix USDT LNURL-pay amount units for Flash Lightning Address edge case +> **ENG-406**: Fix USDT LNURL-pay amount units for Flash Lightning Address edge case ## Overview -The IBEX LNURL-pay API (`POST /v2/lnurl/pay/send`) expects the `amount` field in **millisatoshis**, but the `payToLnurl` wrapper was passing `args.send.amount` directly — the wallet currency's base unit (USDT micros, USD cents, or BTC sats). +The IBEX LNURL-pay API (`POST /v2/lnurl/pay/send`) expects the `amount` field in **millisatoshis**, but the `payToLnurl` wrapper was passing `args.send.amount` directly; the wallet currency's base unit (USDT micros, USD cents, or BTC sats). ## Root Cause -`src/services/ibex/client.ts:255` — `payToLnurl` passed `args.send.amount` directly to `Ibex.payToLnurl()`. The `args.send` was of type `IbexCurrency` (`{ amount: number, currencyId: IbexCurrencyId }`), and for USDT wallets `amount` is USDT micros (~10,000 per USD cent), not millisatoshis. +`src/services/ibex/client.ts:255`: `payToLnurl` passed `args.send.amount` directly to `Ibex.payToLnurl()`. The `args.send` was of type `IbexCurrency` (`{ amount: number, currencyId: IbexCurrencyId }`), and for USDT wallets `amount` is USDT micros (~10,000 per USD cent), not millisatoshis. ## Fix -**Interface change** — `src/services/ibex/index.types.d.ts`: +**Interface change**: `src/services/ibex/index.types.d.ts`: - Replaced `send: IbexCurrency` with `amountMsat: number` on `PayLnurlArgs` -**Call-site change** — `src/services/ibex/client.ts`: +**Call-site change**: `src/services/ibex/client.ts`: - Changed `amount: args.send.amount` → `amount: args.amountMsat` ## Rationale @@ -23,11 +23,11 @@ The IBEX LNURL-pay API (`POST /v2/lnurl/pay/send`) expects the `amount` field in - The field name `amountMsat` makes the expected unit unambiguous - Conversion from wallet currency → msats requires the DealerPriceService (app layer), not available in the services layer - Callers are forced to perform explicit conversion, preventing silent unit bugs -- `PayLnurlArgs` is only used by `payToLnurl` — no other callers to break +- `PayLnurlArgs` is only used by `payToLnurl`; no other callers to break ## Remaining Work -- Wire up a GraphQL mutation that calls `payToLnurl` with proper MSAT conversion (ENG-406 follow-up) — completed in the follow-up implementation described below +- Wire up a GraphQL mutation that calls `payToLnurl` with proper MSAT conversion (ENG-406 follow-up); completed in the follow-up implementation described below - Authored by: Vandana ## Implementation Status @@ -44,11 +44,11 @@ Implemented locally on `eng-274/sandbox-e2e-plan`: Verification: -- `yarn test:unit --testPathPattern=app/payments/lnurl-pay.spec.ts` — passed. -- `yarn test:unit --testPathPattern=graphql/public/root/mutation/lnurl-payment-send.spec.ts` — passed. -- `npx prettier --check ...` on touched files — passed. -- `npx tsc --noEmit --skipLibCheck` — no ENG-406 source/test errors after fixes; still fails on pre-existing unrelated test type errors in offers, cash-wallet history resolver specs, and Bridge webhook/reconciliation specs. -- `yarn check:sdl` — wrote schemas and composed the supergraph, then failed because `src/graphql/public/schema.graphql` had expected generated changes to commit. +- `yarn test:unit --testPathPattern=app/payments/lnurl-pay.spec.ts`: passed. +- `yarn test:unit --testPathPattern=graphql/public/root/mutation/lnurl-payment-send.spec.ts`: passed. +- `npx prettier --check ...` on touched files; passed. +- `npx tsc --noEmit --skipLibCheck`: no ENG-406 source/test errors after fixes; still fails on pre-existing unrelated test type errors in offers, cash-wallet history resolver specs, and Bridge webhook/reconciliation specs. +- `yarn check:sdl`: wrote schemas and composed the supergraph, then failed because `src/graphql/public/schema.graphql` had expected generated changes to commit. --- diff --git a/docs/send-guard.md b/docs/send-guard.md index a6d92f0cd..1ae817811 100644 --- a/docs/send-guard.md +++ b/docs/send-guard.md @@ -9,13 +9,13 @@ untouched. It is **not** the only user-initiated path that moves a user's money. Cashout pays a bolt11 out of the user's own wallet, and `bridgeInitiateWithdrawal` sends the user's own USDT out through IBEX, without either one ever touching the -guard — read [Not covered by the guard at +guard; read [Not covered by the guard at all](#not-covered-by-the-guard-at-all) before concluding from this page that a rail is guarded. Every send mutation runs it before anything reaches IBEX. On every rail that accepts an `idempotencyKey`, it runs as `withPaymentIdempotency`'s `authorize` -hook — inside the lock, after the cache re-check, immediately before the send — +hook, inside the lock, after the cache re-check, immediately before the send, so a replayed key returns the cached result without spending attempt budget or being re-judged against a moved mid price. Ahead of the wrapper, a client retrying a timed-out send burned a burst point per retry and, past 10/min, got @@ -23,13 +23,13 @@ retrying a timed-out send burned a burst point per retry and, past 10/min, got The checks: -1. **attempt budget** — two Redis buckets keyed on the account, a burst bucket +1. **attempt budget**: two Redis buckets keyed on the account, a burst bucket (10/min) and a daily bucket (200/day). Every attempt costs a point, rejected ones included, so probing the amount space is bounded by the caller's own budget. -2. **amount sanity** — positive and finite. USD/USDT cents may be fractional +2. **amount sanity**: positive and finite. USD/USDT cents may be fractional (USDT settles in micros); sats must be whole. -3. **daily limit as per-transaction cap** — `amount <= dailyLimit(level)`. The +3. **daily limit as per-transaction cap**: `amount <= dailyLimit(level)`. The `intraledger` rails use the intraLedger limit; every lightning, lnurl and on-chain rail uses the withdrawal limit. Phase 1 replaces this with the remaining allowance. @@ -37,12 +37,12 @@ The checks: **"Withdrawal" here means the rail, not the destination.** A bolt11 or LN-address payment to another Flash user never leaves Flash, but every lightning and lnurl rail passes `kind: "lightning"` / `"lnurl"` and is judged - against `withdrawalLimit` — the destination is not knowable at the guard, + against `withdrawalLimit`; the destination is not knowable at the guard, which runs before the payment flow that resolves it. That is a deliberate approximation for Phase 0, and **as the ladder stands it cannot change any outcome**: every level carries equal withdrawal and intraLedger limits. - Level 1 was the one exception — Galoy shipped $1,000 external against - $2,000 internal — and it was settled on 2026-09-08 to a single $1,000 rather + Level 1 was the one exception, Galoy shipped $1,000 external against + $2,000 internal, and it was settled on 2026-09-08 to a single $1,000 rather than by raising the external cap. So an L1 user paying another Flash user $1,500 is refused either way, by their lightning invoice and by their username alike, and the rail/destination distinction is invisible. @@ -60,8 +60,8 @@ The checks: | mode | behaviour | | ---------- | ------------------------------------------------------------------------------------------------------ | -| `off` | Returns immediately. No Redis, no price lookup, no ops event, no span attribute. Pre-ENG-573 behaviour **for the guard** — but not for the level-0 default: an account with no `level` field is still capped at the level-0 limits, unconditionally, by Galoy's own checker. See [What `off` does not cover](#what-off-does-not-cover). | -| `log-only` | All checks run, every would-be rejection posts a `transfer / would-reject` ops event — the send proceeds. Includes `lnInvoicePaymentSend`'s bolt11 decode gate: an undecodable or no-amount invoice is reported and still handed to IBEX, exactly as before ENG-573. | +| `off` | Returns immediately. No Redis, no price lookup, no ops event, no span attribute. Pre-ENG-573 behaviour **for the guard**, but not for the level-0 default: an account with no `level` field is still capped at the level-0 limits, unconditionally, by Galoy's own checker. See [What `off` does not cover](#what-off-does-not-cover). | +| `log-only` | All checks run, every would-be rejection posts a `transfer / would-reject` ops event, the send proceeds. Includes `lnInvoicePaymentSend`'s bolt11 decode gate: an undecodable or no-amount invoice is reported and still handed to IBEX, exactly as before ENG-573. | | `enforce` | Rejections are real. | Anything unrecognised degrades to `log-only`. The failure mode of this switch @@ -70,7 +70,7 @@ must be "the guard does not block", never "every send is refused". `off` covers every check the guard itself runs, not just the three above (see [What `off` does not cover](#what-off-does-not-cover) for the one thing ENG-573 changed that it does not). `lnInvoicePaymentSend` also gained a bolt11 decode -gate — the guard needs the amount and the amount is inside the invoice, so the +gate; the guard needs the amount and the amount is inside the invoice, so the resolver decodes it where it previously handed the raw bolt11 straight to IBEX. `decodeInvoice` refuses anything `invoices.parsePaymentRequest` cannot parse and any invoice with no payment secret, which is a rejection class that rail never @@ -78,7 +78,7 @@ had. That gate lives inside the same `authorize` hook and answers to the same switch: on `off` the resolver does not decode at all, so an invoice IBEX would have paid is restored by the flag rather than by a deploy. -The decode gate charges the attempt budget too, exactly once per request — a +The decode gate charges the attempt budget too, exactly once per request; a request the gate handles never reaches `authorizeSend`, and vice versa. It has to: the `LnPaymentRequest` scalar is `/^ln[a-z0-9]+$/i`, so `paymentRequest: "lnx"` is a well-formed request that fails the decode, and without a charge it @@ -89,11 +89,11 @@ limit. One thing ENG-573 changed is **not** behind the switch: reading an account with no `level` field as level 0. That default lives in `effectiveAccountLevel`, -applied inside `getAccountLimits` (`src/config/yaml.ts:186`) — the config layer, +applied inside `getAccountLimits` (`src/config/yaml.ts:186`); the config layer, deliberately, so the guard, `Account.limits` and `remainingLimit` cannot disagree about those ~300 accounts. Galoy's own `AccountLimitsChecker` reads the same function (`src/app/payments/helpers.ts:185`), so it caps an unleveled -account at the level-0 limits **on every mode, `off` included** — and because +account at the level-0 limits **on every mode, `off` included**; and because the guard is not involved, there is no ops event, no `would-reject` embed and no `sendGuard.*` span attribute to explain it. @@ -102,18 +102,18 @@ still routed through `@app/payments` (the USD and amount-bearing resolvers are FLASH FORK bodies that pay IBEX directly and never reach the checker, so on those `off` really is pre-ENG-573 behaviour): -- `lnNoAmountInvoicePaymentSend` — `checkIntraledgerLimits` / +- `lnNoAmountInvoicePaymentSend`: `checkIntraledgerLimits` / `checkTradeIntraAccountLimits` at `src/app/payments/send-lightning.ts:484` when the invoice settles inside Flash, `checkWithdrawalLimits` at `:682` when it leaves. Over the cap, the caller gets Galoy's own `Cannot transfer more than $125.00 in 24 hours`. -- `lnNoAmountInvoiceFeeProbe` — the same three checks at +- `lnNoAmountInvoiceFeeProbe`: the same three checks at `src/app/payments/get-protocol-fee.ts:178`, `:187` and `:196`, so the probe refuses before the send is ever attempted. This is not a regression. Before ENG-573 that cohort indexed the level map with `undefined`, got `NaN` limits, and `paymentAmountFromNumber(NaN)` returned a -`BigIntConversionError` out of `checkLimit` — the send failed anyway, with a +`BigIntConversionError` out of `checkLimit`; the send failed anyway, with a type error instead of a limit message. But it is not switchable either: if an unleveled account has to send above the level-0 cap during an incident, `sendGuard.mode: off` will not do it. Raise that account's `level`, or raise @@ -122,17 +122,17 @@ unleveled account has to send above the level-0 cap during an incident, ## Not covered by the guard at all `off` / `log-only` / `enforce` describe the rails that *call* the guard. The -following move money without it, on every mode — no attempt budget, no +following move money without it, on every mode; no attempt budget, no per-transaction cap, and nothing in the `would-reject` sample or the `sendGuard.*` census to count: -- **Cashout / offers — a gap, not a decision.** `ValidOffer.execute()` pays a +- **Cashout / offers: an open gap.** `ValidOffer.execute()` pays a bolt11 out of the user's own wallet via `Ibex.payInvoice` (`src/app/offers/ValidOffer.ts:68`), reached from the `initiateCashout` mutation through `CashoutManager.executeCashout`. Its compensating controls - are `CashoutValidator`'s configured min/max — `cashout.minimum.amount` / + are `CashoutValidator`'s configured min/max, `cashout.minimum.amount` / `cashout.maximum.amount` in the values file, checked at - `src/app/offers/Validator.ts:29` and `:44` — plus a balance check + `src/app/offers/Validator.ts:29` and `:44`, plus a balance check (`hasSufficientBalance`, `:75`), an account-level floor (`cashout.accountLevel`), and the bank-account / ERP-party checks. So the per-transaction ceiling on this rail is one number for every level, not @@ -141,7 +141,7 @@ per-transaction cap, and nothing in the `would-reject` sample or the limits`. Raising or lowering `accountLimits` does nothing here, and `sendGuard.mode: enforce` does not cap a cashout. Phase 1 should wire this rail in next. -- **Bridge USDT withdrawal — a gap, not a decision.** `bridgeInitiateWithdrawal` +- **Bridge USDT withdrawal: an open gap.** `bridgeInitiateWithdrawal` (`src/graphql/public/root/mutation/bridge-initiate-withdrawal.ts`) → `BridgeService.initiateWithdrawal` (`src/services/bridge/index.ts:1269`) moves the user's own USDT out through `IbexClient.sendCrypto` (`:1442`). Its only @@ -154,7 +154,7 @@ per-transaction cap, and nothing in the `would-reject` sample or the min/max the way cashout has one, and `accountLimits` is never consulted. These are the largest per-transaction amounts on the platform. Phase 1 should wire this rail in alongside cashout. -- **System credits — deliberate.** Quiz rewards, referral payouts, card top-up +- **System credits: deliberate.** Quiz rewards, referral payouts, card top-up credits, operator reimbursements. They move money out of a Flash-owned funding wallet on our own instruction, so an account-scoped attempt budget and a per-account daily cap describe nothing about them (a 30-payment referral batch @@ -177,8 +177,8 @@ users), and on `enforce` they are capped at $125 per transaction. "An unleveled account is an unverified one" is an assumption, not a verified fact about those accounts. -That assumption lives in exactly one place — `effectiveAccountLevel`, applied -inside `getAccountLimits` — so the guard, `Account.limits` and `remainingLimit` +That assumption lives in exactly one place, `effectiveAccountLevel`, applied +inside `getAccountLimits`, so the guard, `Account.limits` and `remainingLimit` all read the same numbers for those accounts. Revising it is a one-line change there, not a hunt through the call sites. The price of putting it there is that `sendGuard.mode` does not reach it: see @@ -188,31 +188,31 @@ there, not a hunt through the call sites. The price of putting it there is that ## Flipping to enforce -1. Deploy with `sendGuard.mode: log-only` (the default — nothing to set). +1. Deploy with `sendGuard.mode: log-only` (the default; nothing to set). 2. Watch the ops feed for `🔁 Transfer — Would Reject` embeds for at least a full day, ideally a week (weekly payout and settlement patterns are the ones that sit near a cap). Each embed carries the account, the level, the amount and the reason in `step`. 3. Count them by `step`: - - `over-daily-limit` — real traffic the cap would have refused. If any of it + - `over-daily-limit`; real traffic the cap would have refused. If any of it is legitimate, raise the level's limit or the account's level *before* enforcing; do not enforce and then triage support tickets. A `sendGuard.kind` of `lightning` or `lnurl` says which rail was used, not where the money went, so an inside-Flash payment is counted here as though it were external (see check 3 above). With every level's two limits equal that costs nothing - — the same amount is refused either way — but it becomes a real ambiguity +, the same amount is refused either way, but it becomes a real ambiguity in this bucket the moment any level's intraLedger limit exceeds its withdrawal limit. - - `invalid-amount` — malformed client input. Should be near zero. A + - `invalid-amount`; malformed client input. Should be near zero. A send-all on an empty wallet is deliberately **not** in this bucket: `getBalanceForWallet` reads a drained or never-funded wallet as - `USDAmount.ZERO` — post-cutover the default for every migrated account's - legacy USD wallet — so `onchain-payment-send-all.ts` skips the guard when + `USDAmount.ZERO`, post-cutover the default for every migrated account's + legacy USD wallet, so `onchain-payment-send-all.ts` skips the guard when the balance rounds to zero cents and leaves the refusal to the rail, one layer down. Be clear about what that refusal is: `payOnChainByWalletId` reaches `OnchainUsdPaymentValidator`'s `checkOnchainMin`, which returns a bare `ValidationError("Amount must be greater than 0")`, and `mapError` - has no case for it beyond the catch-all — the client gets `Unexpected error + has no case for it beyond the catch-all; the client gets `Unexpected error occurred, please try again or contact support if it persists (code: ValidationError: Amount must be greater than 0)`. That is unchanged by ENG-573 and not a good message, but it is not the guard's to fix here. The @@ -220,20 +220,20 @@ there, not a hunt through the call sites. The price of putting it there is that tap would read as malformed client input, in the one bucket the runbook says should be near zero, in the sample the enforce decision is made from. - - `limits-unavailable` — Redis or the price feed failing, i.e. the guard + - `limits-unavailable`; Redis or the price feed failing, i.e. the guard itself unable to decide. Must be zero before enforcing: on `enforce` these block the send. They are also recorded as span exceptions - (`ErrorLevel.Critical` when enforcing, `Warn` in log-only) — alert on that + (`ErrorLevel.Critical` when enforcing, `Warn` in log-only); alert on that signal, because a price-pod outage past the 10-minute price cache rejects every amount-bearing lightning send and every BTC intraledger send. - `rate-limited` is **coalesced**, not silent. Silencing it would leave one - of the three checks with no observable output at all — unreadable during + of the three checks with no observable output at all; unreadable during the very rollout this mode exists for, so an operator would count a week of would-reject embeds, see no rate-limit signal by construction, flip to enforce, and hand a 30-payment payout batch twenty `TooManyRequestError`s. - `undecodable-invoice` is `lnInvoicePaymentSend` only: the bolt11 could not be decoded, or carried no amount. The guard introduced that decode, so this - is a rejection class the rail never had — count it before enforcing. Also + is a rejection class the rail never had; count it before enforcing. Also **coalesced**; count it from the span attributes below, not by tallying embeds. @@ -249,8 +249,8 @@ there, not a hunt through the call sites. The price of putting it there is that attempt budget has been charged, so the budget bounds them. **What the embed tells you.** The one that does post carries `muted: N` in - its meta — how many events of that reason were coalesced away since the last - one that posted — plus `mutedAgeS`, how many seconds ago the window that + its meta, how many events of that reason were coalesced away since the last + one that posted, plus `mutedAgeS`, how many seconds ago the window that accumulated them opened. A count is only ever cleared by being delivered: a 40-second blip that mutes 499 rejections and then goes quiet reports all 499 on the next embed of that reason, however much later, and `mutedAgeS` is what @@ -263,17 +263,17 @@ there, not a hunt through the call sites. The price of putting it there is that `sendGuard.level`, `sendGuard.error` and (when the guard got as far as an amount) `sendGuard.cents`. The guard opens that span itself rather than writing to whatever span happens to be active, and that is load-bearing for - this step: the ambient span differs by call path — on the six rails that take + this step: the ambient span differs by call path, on the six rails that take an `idempotencyKey` the guard runs inside `services.lock.lockPaymentIdempotencyKey`, and without a key, or on either - on-chain rail, inside the GraphQL resolver span — so a query scoped to one of + on-chain rail, inside the GraphQL resolver span, so a query scoped to one of those names would have counted a fraction of the traffic, and the fraction that went missing is the newer mobile clients that send idempotency keys. Both entry points share the name; `code.function` on the span says whether it was `authorizeSend` or the `gateSend` decode gate. Every attribute is a **string** except `sendGuard.cents`, which is a number - so it can be aggregated — so query `sendGuard.level = "0"`, not `= 0`, and + so it can be aggregated; so query `sendGuard.level = "0"`, not `= 0`, and read a level-0 or level-less account as the literal `"0"` rather than as an absent attribute. Range and percentile filters on `sendGuard.cents` are numeric. Count from tracing, not from Discord: the ops feed is @@ -282,16 +282,16 @@ there, not a hunt through the call sites. The price of putting it there is that dropped" summary. Read the feed, count the spans. **What to alert on.** `limits-unavailable` additionally records a span - *exception* on every occurrence, unthrottled — `ErrorLevel.Critical` when + *exception* on every occurrence, unthrottled; `ErrorLevel.Critical` when enforcing, `Warn` in log-only. That is the page-worthy signal: it means the guard cannot decide, and on `enforce` that blocks the send. 4. Set `sendGuard.mode: enforce`, `helm upgrade`, and **restart the api pods** - (see below) — the mode is read once at process start. + (see below); the mode is read once at process start. ## Rolling back Same flag: `log-only`, or `off` if the guard itself is the outage. Neither -restores sends for an account with **no `level` field** — that cap is enforced a +restores sends for an account with **no `level` field**; that cap is enforced a layer below the flag, silently, and no amount of mode-flipping lifts it. Read [What `off` does not cover](#what-off-does-not-cover) before concluding the rollback failed. @@ -311,24 +311,23 @@ kubectl rollout status deploy/ ``` Budget for a pod roll, not for an instant flag flip. If you need the guard to -stop blocking *now* and the roll is too slow, `off` costs the same restart — -there is no faster switch, which is worth knowing before the incident rather +stop blocking *now* and the roll is too slow, `off` costs the same restart; there is no faster switch, which is worth knowing before the incident rather than during it. ## Related config -- `accountLimits.*.level` — the caps. **Every level 0-3 is `required`**: a +- `accountLimits.*.level`: the caps. **Every level 0-3 is `required`**: a deployment that overrides `accountLimits` partially fails at boot rather than resolving a missing level to `NaN` and silently blocking that level's sends. An *account* with no level is separate: `getAccountLimits` resolves that to level 0 via `effectiveAccountLevel` before indexing. That resolution is in the - config layer, not the guard, so it is **unconditional** — `sendGuard.mode`, + config layer, not the guard, so it is **unconditional**; `sendGuard.mode`, `off` included, does not lift it, and Galoy's `AccountLimitsChecker` goes on enforcing the level-0 cap on `lnNoAmountInvoicePaymentSend` and `lnNoAmountInvoiceFeeProbe` regardless of mode, with none of the guard's ops-event or span output to show for it. See [What `off` does not cover](#what-off-does-not-cover). -- `rateLimits.paymentSendAttempt` / `paymentSendDailyAttempt` — the two buckets. +- `rateLimits.paymentSendAttempt` / `paymentSendDailyAttempt`; the two buckets. `blockDuration` must be **>= `duration`**: rate-limiter-flexible rewrites the key's TTL to `blockDuration` on the first breach, so a shorter block throws the counter away early and grants a fresh budget (a 3600s block on an 86400s window diff --git a/test/flash/bridge-sandbox-e2e/README.md b/test/flash/bridge-sandbox-e2e/README.md index b6dc4472a..b7f641853 100644 --- a/test/flash/bridge-sandbox-e2e/README.md +++ b/test/flash/bridge-sandbox-e2e/README.md @@ -104,14 +104,14 @@ IBEX_ENVIRONMENT=sandbox yarn test:bridge-sandbox-e2e | External account | Skipped by default; with `BRIDGE_SANDBOX_EXTERNAL_ACCOUNT_LINK_CONFIRMED=true`, `bridgeAddExternalAccount` returns `{linkToken, expiresAt}` | Requires Bridge sandbox API key/customer entitlement for Plaid link-token generation | | Deposit webhook | Injected webhook processes and persists deposit | Verify webhook secret in config.yaml | | Withdrawal error paths | Validation errors returned for invalid inputs | Check withdrawal schema deployed (ENG-348) | -| Withdrawal **success** path | ⚠️ **Not expected to pass first run** — requires real KYC-approved sandbox customer, funded wallet, and verified external account. | The full withdrawal flow only runs with `BRIDGE_SANDBOX_WITHDRAWAL_CONFIRMED=true`; error-path tests run without it | +| Withdrawal **success** path | ⚠️ **Not expected to pass first run**, requires real KYC-approved sandbox customer, funded wallet, and verified external account. | The full withdrawal flow only runs with `BRIDGE_SANDBOX_WITHDRAWAL_CONFIRMED=true`; error-path tests run without it | ### If something fails 1. Check `IBEX_ENVIRONMENT` is `sandbox` (not `production`) 2. Confirm MongoDB is running: `mongosh --eval "db.adminCommand('ping')"` 3. Run `./dev/setup.sh --webhook` again to refresh ngrok, Bridge webhook endpoints, and local public keys -4. Preflight failure → `src/services/bridge/index.ts` still has `level < 2` — apply the Task 0 fix +4. Preflight failure → `src/services/bridge/index.ts` still has `level < 2`; apply the Task 0 fix 5. KYC/VA failures → confirm the corresponding ENG issue is deployed to sandbox ## Commands diff --git a/test/flash/smoke/README.md b/test/flash/smoke/README.md index ac683adeb..806b764d0 100644 --- a/test/flash/smoke/README.md +++ b/test/flash/smoke/README.md @@ -1,7 +1,7 @@ # UAT Smoke Suite Black-box smoke tests for the integrated UAT matrix. Everything runs through -the **public GraphQL API over HTTP** — no app imports, no database access — so +the **public GraphQL API over HTTP**, no app imports, no database access, so the same suite targets the local quickstart stack, CI, or a deployed environment (TEST) after a release. @@ -27,24 +27,24 @@ yarn test:smoke ``` Set `SMOKE_EXPECT_FLAGS_OFF=false` once bridge/topup/cashout are enabled in the -target environment — the `topupEnabled` flag spec then asserts a boolean +target environment; the `topupEnabled` flag spec then asserts a boolean instead of asserting `false`. ## Two tiers -The suite is tiered because the quickstart stack **mocks IBEX** — it can't +The suite is tiered because the quickstart stack **mocks IBEX**; it can't resolve wallet balances, create Lightning invoices, register device tokens, or persist usernames, and its synthetic wallet ids don't satisfy the `WalletId` scalar. -- **CI tier (default, `SMOKE_BACKEND_FULL=false`)** — everything reachable +- **CI tier (default, `SMOKE_BACKEND_FULL=false`)**: everything reachable without a provisioned backend: API health, login/session, account+wallet identity, transaction-history shape, username-set (accepted), recipient non-resolution, payment-validation rejections, `globals.topupEnabled`, and that bridge/cashout resolvers respond structurally (no 5xx). This is what runs in CI against quickstart and catches the bulk of API-contract, auth, and resolver-crash regressions. -- **Full tier (`SMOKE_BACKEND_FULL=true`)** — balances, invoice creation, +- **Full tier (`SMOKE_BACKEND_FULL=true`)**: balances, invoice creation, onchain address, default-wallet mutation, device-token registration, username resolution, and (with `SMOKE_DOCKER_HELPERS=true` / `SMOKE_ALLOW_PAYMENTS=true`) external funding and two-account payments. Run @@ -68,23 +68,23 @@ Tier: **C** = CI (runs against quickstart mock), **F** = full backend only. | RECV-03/04/05 | F | `03-receive` | invoice create paths; expired-UI is manual | | FUND-01/02 | F | `04-funding-external` | needs lnd-outside + real balances | | EXT-01/02 | F | `04-funding-external` | needs lnd-outside; TEST external wallet is manual | -| EXT-03 | — | manual | LNURL hostname routing is env-specific | +| EXT-03 | none | manual | LNURL hostname routing is env-specific | | SEND-01/02 | F | `05-payments-internal` | intraledger both directions with memo | | SEND-03 | C | `06-validation` | unknown username does not resolve | | SEND-04/05 | C | `06-validation` | self-send, zero, too-high all rejected | | CONTACT-01 | F | `05-payments-internal` | contacts populated after payment | | WALLET-01 | F | `02-wallets-home` | default-wallet mutation (needs real WalletId) | -| WALLET-02/03 | — | manual | conversion UX + max-amount UI flow | +| WALLET-02/03 | none | manual | conversion UX + max-amount UI flow | | ONCHAIN-01 | F | `02-wallets-home` | when a BTC wallet exists | -| ONCHAIN-02 | — | manual | below-min UX validation | +| ONCHAIN-02 | none | manual | below-min UX validation | | BRIDGE-01 | C | `07-flags-bridge` | `globals.topupEnabled` matches flag state | | BRIDGE-03/05 | C | `07-flags-bridge` | resolvers respond structurally (no 5xx); WebView/UI manual | -| BRIDGE-02/04 | — | manual | WebView + order/instruction UI | +| BRIDGE-02/04 | none | manual | WebView + order/instruction UI | | NOTIF-01 | F | `07-flags-bridge` | device-token registration; push delivery is manual | -| NOTIF-02/03 | — | manual | push arrival + deep links need devices | -| SCAN-00..03 | — | manual | camera/permissions need devices | -| SETTINGS-*, MAP, REPORT, CHAT | — | manual | device UI | -| WIDGET-01, WATCH-01 | — | manual | iOS platform extensions | +| NOTIF-02/03 | none | manual | push arrival + deep links need devices | +| SCAN-00..03 | none | manual | camera/permissions need devices | +| SETTINGS-*, MAP, REPORT, CHAT | none | manual | device UI | +| WIDGET-01, WATCH-01 | none | manual | iOS platform extensions | Roughly **14 rows run in CI** (quickstart mock) and **~10 more in the full tier** against a real backend; the remainder are device-bound UI/UX rows that @@ -95,7 +95,7 @@ net; the full tier is the money-movement confidence you point at TEST. - Specs are ordered (`00-` … `07-`) and mirror the UAT phase plan; later phases depend on backend state created earlier (accounts, funding), not on - in-process state — every spec re-logs-in with the same deterministic phones. + in-process state; every spec re-logs-in with the same deterministic phones. - Quickstart logins use `UNSECURE_DEFAULT_LOGIN_CODE=000000`; deployed environments must list the smoke phones under `test_accounts` in config. - Money movement is opt-out (`SMOKE_ALLOW_PAYMENTS=false`) for shared