diff --git a/.github/workflows/build-ci.yml b/.github/workflows/build-ci.yml index 22ffe10..f807c34 100644 --- a/.github/workflows/build-ci.yml +++ b/.github/workflows/build-ci.yml @@ -22,6 +22,22 @@ jobs: strategy: matrix: node: ["22.18.0", "24.19.0"] + services: + mysql: + image: mariadb:11.8.6 + env: + MARIADB_ROOT_PASSWORD: isolated-root-password + MARIADB_DATABASE: otf_test + MARIADB_USER: otf_test + MARIADB_PASSWORD: isolated-test-password + ports: + - 3306:3306 + options: >- + --health-cmd="healthcheck.sh --connect --innodb_initialized" + --health-interval=5s --health-timeout=5s --health-retries=20 + env: + OTF_TEST_MYSQL_ISOLATED: "1" + OTF_TEST_MYSQL_URL: mysql://otf_test:isolated-test-password@127.0.0.1:3306/otf_test steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 diff --git a/README.md b/README.md index e14f0e4..f2da8d8 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,10 @@ Additionally, there are some environment variables that can optionally be specif | `HTTP_PORT` | The port that the HTTP server should listen on. | `3000` | | `WEBSOCKET_PORT` | The port that the websocket server should listen on. | `8080` | | `REQUEST_TIMEOUT_MS` | Maximum wait for a complete controller response, in milliseconds. Must be an integer from 1 to 300000; invalid values fall back to the default with a warning. | `60000` | +| `WEBSOCKET_MAX_PAYLOAD_BYTES` | Maximum incoming WebSocket message size, including all fragments and the response envelope, before or after authentication. Integer from 1 to 16777216; invalid values prevent startup. | `4194304` (4 MiB) | +| `STATS_INTERVAL_MS` | Interval for aggregate `proxy_stats` log records. Integer from 1 to 300000; invalid values prevent startup. | `10000` | +| `AUTH_CONCURRENCY` | Maximum running authentication operations. Integer from 1 to 100; invalid values prevent startup. | `10` | +| `AUTH_QUEUE_LIMIT` | Maximum authentication jobs waiting to start. Integer from 1 to 10000; invalid values prevent startup. | `500` | ## Connection and request behavior @@ -37,12 +41,28 @@ Controllers are authenticated before registration and protocol handling. A new connection using an already connected device key is rejected; it does not replace the existing owner. Authentication has a ten-second deadline; timeout rejects with close code `1011`, and late results cannot register the connection. Disconnecting -during authentication releases its deadline and wait state. MySQL queries also -have a ten-second driver inactivity limit; a lookup that completes after the +during authentication releases its deadline and wait state. MySQL queries have a +driver inactivity limit capped at ten seconds and the remaining authentication +time after connection acquisition; a lookup that completes after the socket's deadline is ignored. The existing 10-second ping interval and missed-pong policy are preserved. Policy rejections use WebSocket close code `1008`; authentication backend failures use `1011`. Rejected connections receive an `ERR:` message and a close frame, with forced termination after five seconds if closing stalls. +Oversized incoming messages close with code `1009`. This bounds each message; +it is not a limit on total memory or connection count. + +Authentication uses a bounded FIFO queue and at most one waiting/running lookup +per key. An existing open owner's duplicate is rejected before any lookup; the +owner is still rechecked after validation. Expired or disconnected waiters do not +enter the backend. A running lookup retains its concurrency slot and same-key +reservation until the backend actually finishes, even if its caller is gone. +Queue-full or same-key contention receives close code `1011` with reason +`authentication busy`. Capacity failures never use HTTP `401`/`403`. This does not +require firmware backoff support and does not change heartbeat or ownership policy. +The defaults are starting limits to validate against the deployment's database +latency and memory budget. This queue bounds authentication work, not all sockets: +there is no pre-upgrade connection cap, and rejected sockets may linger during the +five-second close fallback. Reverse-proxy/OS connection limits remain necessary. Pending requests belong to their individual controller session. A disconnected controller fails its pending requests with HTTP `408`, and requests for an absent @@ -73,6 +93,16 @@ printing their values. Malformed payloads and raw database errors are not logged. Rejection and other repeated diagnostic events have bounded output and aggregation state. +Aggregate `proxy_stats` records include cumulative counters (including suppressed +rejections), counter deltas since the previous record, current/high-water gauges, +fixed cumulative latency buckets, process memory/CPU and event-loop measurements. +Running authentication remains counted until the backend actually finishes, even +after the socket closes. Memory `arrayBuffers` overlaps `external`; do not add them +together. Admission, request and protocol warnings have separate bounded log +budgets. Aggregate metrics contain no per-token labels and have no public HTTP +endpoint. These records describe this Node process; correlate them with separate +reverse-proxy/container/host measurements to investigate incidents. + Both Docker stages use Node `22.18.0` on Alpine, pinned to the immutable image index digest in `Dockerfile`. Dependency installation uses `npm ci`. Updating the base image requires an intentional digest change and runtime validation; it no @@ -109,6 +139,21 @@ The plugin can be configured through the following environment variables: | `MYSQL_CONNECTION_URL` | The URL of the MySQL server to connect to. | `mysql://username:password@localhost:3306/otf` | | `MYSQL_TABLE` | The table in the database that stores valid device keys. | `registered_devices` | +The plugin uses an existence lookup with `LIMIT 1`; index `device_key` for the +deployment's lookup workload. The pool's connection limit follows +`AUTH_CONCURRENCY`, waiting inside the driver is disabled, and driver debug output +is disabled to prevent SQL/key leakage. These limits override URL pool/debug +options. Disconnected/expired callers are checked again after acquisition, before +SQL begins. An in-progress query is not cancelled by destroying its connection; +the driver may itself discard a connection on a fatal inactivity timeout. + +CI also runs real-driver integration tests against a disposable MariaDB. For +local integration tests, explicitly set `OTF_TEST_MYSQL_ISOLATED=1` and +`OTF_TEST_MYSQL_URL` to a disposable database named `otf_test`, then run `npm test`. +The tests create/drop their own tables and acquire table locks. Never point this +test URL at a shared or production database. Without these explicit test settings, +the integration cases are skipped and the normal suite uses fake backends. + #### EnvironmentVariableAuthenticationPlugin diff --git a/src/AuthenticationPlugin.ts b/src/AuthenticationPlugin.ts index b86bd3a..e8aa2ba 100644 --- a/src/AuthenticationPlugin.ts +++ b/src/AuthenticationPlugin.ts @@ -1,5 +1,13 @@ import { Logger } from "pino"; +export interface AuthenticationContext { + // Aborting means the caller has gone away, not that SQL was cancelled. + signal: AbortSignal; + remainingMs(): number; +} + +export interface AuthenticationInitOptions { concurrency: number; } + export abstract class AuthenticationPlugin { public constructor() {} @@ -8,11 +16,13 @@ export abstract class AuthenticationPlugin { * any requests until the promise it returns has been resolved. The default implementation returns a Promise that * will be resolved immediately. */ - public async init(logger: Logger): Promise {} + public async init(logger: Logger, options?: AuthenticationInitOptions): Promise {} + + public async close(): Promise {} /** * Returns a Promise that resolves with a boolean indicating if the specified device key is valid. If the Promise is * resolved with `false` or is rejected, access will be denied. */ - public abstract validateKey(deviceKey: string): Promise; + public abstract validateKey(deviceKey: string, context?: AuthenticationContext): Promise; } diff --git a/src/authenticationPlugins/MySQLAuthenticationPlugin.ts b/src/authenticationPlugins/MySQLAuthenticationPlugin.ts index 1109017..1100852 100644 --- a/src/authenticationPlugins/MySQLAuthenticationPlugin.ts +++ b/src/authenticationPlugins/MySQLAuthenticationPlugin.ts @@ -1,55 +1,80 @@ import { Logger } from "pino"; -import { AuthenticationPlugin } from "../AuthenticationPlugin"; +import { AuthenticationContext, AuthenticationInitOptions, AuthenticationPlugin } from "../AuthenticationPlugin"; import * as mysql from "mysql"; import { backendErrorCode } from "../diagnostics"; -import { ConfigurationError } from "../configuration"; +import { boundedInteger, ConfigurationError } from "../configuration"; /** * Validates device keys by checking if they exist in a MySQL database. */ export default class MySQLAuthenticationPlugin extends AuthenticationPlugin { private pool: mysql.Pool; - private logger: Logger; + private table: string; + private stopped?: Promise; - public async init(logger: Logger): Promise { - this.logger = logger; - return new Promise((resolve, reject) => { - // Make sure that necessary variables have been defined. - for (const key of ["MYSQL_TABLE", "MYSQL_CONNECTION_URL"] as const) { - if (!process.env[key]) { - reject(new ConfigurationError(key, "missing")); - return; - } - } + public async init(_logger: Logger, options?: AuthenticationInitOptions): Promise { + for (const key of ["MYSQL_TABLE", "MYSQL_CONNECTION_URL"] as const) { + if (!process.env[key]) throw new ConfigurationError(key, "missing"); + } + const concurrency = boundedInteger("AUTH_CONCURRENCY", + options?.concurrency ?? process.env.AUTH_CONCURRENCY, 10, 100); + this.table = process.env.MYSQL_TABLE; + try { + // Let the driver parse its supported URL options. Before any use, + // override admission/time limits so URL options cannot bypass them. + this.pool = mysql.createPool(process.env.MYSQL_CONNECTION_URL); + this.pool.config.connectionLimit = concurrency; + this.pool.config.waitForConnections = false; + this.pool.config.queueLimit = 1; + this.pool.config.acquireTimeout = 10_000; + this.pool.config.connectionConfig.connectTimeout = 10_000; + // Driver debug output prints SQL and credentials outside our logger. + this.pool.config.connectionConfig.debug = false; + } catch { + throw new ConfigurationError("MYSQL_CONNECTION_URL", "invalid"); + } + } - try { - this.pool = mysql.createPool(process.env.MYSQL_CONNECTION_URL); - } catch { - reject(new ConfigurationError("MYSQL_CONNECTION_URL", "invalid")); - return; - } - resolve(); - }); + private failure(error: unknown): Error { + // Never retain the driver's SQL-bearing error object as a cause. + return Object.assign(new Error("Authentication backend unavailable"), { code: backendErrorCode(error) }); } - public validateKey(deviceKey: string): Promise { + public validateKey(deviceKey: string, context?: AuthenticationContext): Promise { return new Promise((resolve, reject) => { - this.pool.query( - { sql: "SELECT * FROM ?? WHERE device_key = ?", timeout: 10_000 }, - [process.env.MYSQL_TABLE, deviceKey], - (err, results, fields) => { - if (err) { - // mysql errors include the formatted SQL and key. The - // forwarder rate-limits logging of this sanitized failure. - const failure = new Error("Authentication backend unavailable"); - Object.assign(failure, { code: backendErrorCode(err) }); - reject(failure); - return; - } + const stale = () => context && (context.signal.aborted || context.remainingMs() < 1); + const expired = () => this.failure({ code: "AUTH_TIMEOUT" }); + if (stale()) { reject(expired()); return; } + try { + this.pool.getConnection((error, connection) => { + if (error) { reject(this.failure(error)); return; } + let released = false; + const release = () => { + if (released) return; + released = true; + connection.release(); + }; + if (stale()) { release(); reject(expired()); return; } + const timeout = Math.min(10_000, Math.floor(context?.remainingMs() ?? 10_000)); + if (timeout < 1) { release(); reject(expired()); return; } + try { + connection.query({ sql: "SELECT 1 FROM ?? WHERE device_key = ? LIMIT 1", timeout }, + [this.table, deviceKey], (err, results) => { + release(); + if (err) reject(this.failure(err)); + else resolve(results.length !== 0); + }); + } catch (err) { release(); reject(this.failure(err)); } + }); + } catch (error) { reject(this.failure(error)); } + }); + } - resolve(results.length !== 0); - } - ); + public close(): Promise { + if (!this.stopped) this.stopped = new Promise((resolve, reject) => { + if (!this.pool) { resolve(); return; } + this.pool.end(error => error ? reject(this.failure(error)) : resolve()); }); + return this.stopped; } } diff --git a/src/authenticationQueue.ts b/src/authenticationQueue.ts new file mode 100644 index 0000000..2e17a0a --- /dev/null +++ b/src/authenticationQueue.ts @@ -0,0 +1,125 @@ +import { performance } from "perf_hooks"; +import { AuthenticationPlugin } from "./AuthenticationPlugin"; +import { Telemetry } from "./telemetry"; + +type Result = { status: "valid" | "invalid" | "timed_out" | "cancelled" | "busy" } + | { status: "error"; error: unknown }; + +interface Job { + key: string; + startedAt: number; + deadline: number; + active: boolean; + settled: boolean; + controller: AbortController; + timer: ReturnType; + detach(): void; + resolve?: (result: Result) => void; +} + +/** Bounded, FIFO admission. Client lifetime and backend lifetime are separate. */ +export class AuthenticationQueue { + private jobs = new Map(); + private waiting = new Map(); + private active = 0; + private stopping = false; + + constructor(private readonly plugin: AuthenticationPlugin, private readonly telemetry: Telemetry, + private readonly concurrency: number, private readonly queueLimit: number, + private readonly timeoutMs: number) {} + + submit(key: string, signal: AbortSignal): Promise { + if (signal.aborted || this.stopping) return Promise.resolve({ status: "cancelled" }); + if (this.jobs.has(key) || (this.active >= this.concurrency && this.waiting.size >= this.queueLimit)) { + this.telemetry.count("authentication_busy"); + return Promise.resolve({ status: "busy" }); + } + return new Promise(resolve => { + const startedAt = performance.now(); + const job: Job = { key, startedAt, deadline: startedAt + this.timeoutMs, + controller: new AbortController(), active: false, settled: false, + timer: undefined, detach: () => {}, resolve }; + const abandon = () => this.settle(job, { status: "cancelled" }); + signal.addEventListener("abort", abandon, { once: true }); + job.detach = () => signal.removeEventListener("abort", abandon); + job.timer = setTimeout(() => this.settle(job, { status: "timed_out" }), this.timeoutMs); + job.timer.unref(); + this.jobs.set(key, job); + this.waiting.set(key, job); + this.telemetry.adjust("authQueued", 1); + this.pump(); + }); + } + + private settle(job: Job, result: Result): void { + if (job.settled) return; + job.settled = true; + clearTimeout(job.timer); + job.detach(); + if (result.status === "cancelled" || result.status === "timed_out") { + this.telemetry.count(result.status === "cancelled" + ? "authentication_cancelled" : "authentication_timed_out"); + job.controller.abort(); + } + if (!job.active) { + if (this.waiting.delete(job.key)) this.telemetry.adjust("authQueued", -1); + if (this.jobs.get(job.key) === job) this.jobs.delete(job.key); + } + this.telemetry.observe("authentication", performance.now() - job.startedAt); + const resolve = job.resolve; + job.resolve = undefined; + resolve?.(result); + } + + private pump(): void { + while (!this.stopping && this.active < this.concurrency && this.waiting.size) { + const job = this.waiting.values().next().value as Job; + if (performance.now() >= job.deadline) { + this.settle(job, { status: "timed_out" }); + continue; + } + this.waiting.delete(job.key); + this.telemetry.adjust("authQueued", -1); + this.telemetry.observe("queueWait", performance.now() - job.startedAt); + job.active = true; + this.active++; + this.telemetry.adjust("authActive", 1); + // Defer invocation, then recheck: a close/deadline may have run in between. + void Promise.resolve().then(async () => { + if (job.controller.signal.aborted || performance.now() >= job.deadline) { + this.settle(job, { status: job.controller.signal.aborted ? "cancelled" : "timed_out" }); + return; + } + const lookupStarted = performance.now(); + this.telemetry.count("authentication_started"); + try { + const valid = await this.plugin.validateKey(job.key, { + signal: job.controller.signal, + remainingMs: () => Math.max(0, job.deadline - performance.now()), + }); + this.telemetry.count(valid ? "authentication_valid" : "authentication_invalid"); + this.settle(job, performance.now() >= job.deadline + ? { status: "timed_out" } : { status: valid ? "valid" : "invalid" }); + } catch (error) { + this.telemetry.count("authentication_failed"); + this.settle(job, performance.now() >= job.deadline + ? { status: "timed_out" } : { status: "error", error }); + } finally { + this.telemetry.observe("lookup", performance.now() - lookupStarted); + } + }).finally(() => { + // Do not release a slot or same-key guard just because a socket left. + job.active = false; + this.active--; + this.telemetry.adjust("authActive", -1); + if (this.jobs.get(job.key) === job) this.jobs.delete(job.key); + this.pump(); + }); + } + } + + close(): void { + this.stopping = true; + for (const job of this.jobs.values()) this.settle(job, { status: "cancelled" }); + } +} diff --git a/src/configuration.ts b/src/configuration.ts index 36bef58..c5a1cb9 100644 --- a/src/configuration.ts +++ b/src/configuration.ts @@ -1,4 +1,5 @@ -type Setting = "AUTHENTICATION_PLUGIN" | "DEVICE_KEYS" | "MYSQL_TABLE" | "MYSQL_CONNECTION_URL"; +type Setting = "AUTHENTICATION_PLUGIN" | "DEVICE_KEYS" | "MYSQL_TABLE" | "MYSQL_CONNECTION_URL" + | "WEBSOCKET_MAX_PAYLOAD_BYTES" | "STATS_INTERVAL_MS" | "AUTH_CONCURRENCY" | "AUTH_QUEUE_LIMIT"; // These messages are safe to log: they name settings, never their values. export class ConfigurationError extends Error { @@ -8,6 +9,17 @@ export class ConfigurationError extends Error { } } +// Report names, never configuration values (which may contain credentials). +export function boundedInteger(setting: Setting, value: unknown, fallback: number, maximum: number): number { + if (value === undefined) return fallback; + const parsed = typeof value === "number" ? value + : typeof value === "string" && /^\d+$/.test(value) ? Number(value) : NaN; + if (!Number.isSafeInteger(parsed) || parsed < 1 || parsed > maximum) { + throw new ConfigurationError(setting, "invalid"); + } + return parsed; +} + export function startupFailure(error: unknown) { if (error instanceof ConfigurationError) { return { event: "startup_failure", setting: error.setting, diff --git a/src/diagnostics.ts b/src/diagnostics.ts index fb4669a..9060c99 100644 --- a/src/diagnostics.ts +++ b/src/diagnostics.ts @@ -27,8 +27,10 @@ export function forwardedEndpoint(url: string): string | undefined { export class EventLimiter { private entries = new Map(); private windowStart: number; - private emitted = 0; - private suppressed = 0; + // A burst of invalid keys must not starve request/protocol diagnostics. + private budgets = { admission: { emitted: 0, suppressed: 0 }, + requests: { emitted: 0, suppressed: 0 }, protocol: { emitted: 0, suppressed: 0 }, + other: { emitted: 0, suppressed: 0 } }; constructor( private readonly windowMs = 60_000, @@ -44,10 +46,15 @@ export class EventLimiter { const now = this.now(); if (now - this.windowStart >= this.windowMs) { this.windowStart = now; - this.emitted = 0; + for (const budget of Object.values(this.budgets)) budget.emitted = 0; } - if (this.emitted >= this.maxEvents) { - this.suppressed++; + const category = ["connection_rejected", "rejection_forced_close", "rejected_connection_closed", + "pre_auth_error", "auth_closed"].includes(event) ? "admission" + : ["request_timeout", "response_write_failed"].includes(event) ? "requests" + : ["invalid_response_frame", "unknown_response_id"].includes(event) ? "protocol" : "other"; + const budget = this.budgets[category]; + if (budget.emitted >= this.maxEvents) { + budget.suppressed++; return; } const key = `${event}:${identity}`; @@ -60,10 +67,10 @@ export class EventLimiter { this.entries.delete(this.entries.keys().next().value); } this.entries.set(key, { at: now, suppressed: 0 }); - this.emitted++; + budget.emitted++; logger[level]({ ...fields, event, - suppressed: (entry?.suppressed || 0) + this.suppressed }); - this.suppressed = 0; + suppressed: (entry?.suppressed || 0) + budget.suppressed }); + budget.suppressed = 0; } } @@ -75,6 +82,7 @@ export function backendErrorCode(error: unknown): string { "PROTOCOL_CONNECTION_LOST", "PROTOCOL_SEQUENCE_TIMEOUT", "ER_ACCESS_DENIED_ERROR", "ER_BAD_DB_ERROR", "ER_NO_SUCH_TABLE", "ER_PARSE_ERROR", "ER_CON_COUNT_ERROR", "POOL_ENQUEUELIMIT", + "POOL_CONNLIMIT", "POOL_CLOSED", "AUTH_TIMEOUT", ]); return typeof code === "string" && known.has(code) ? code : "AUTH_BACKEND_ERROR"; } diff --git a/src/routes/forwarder.ts b/src/routes/forwarder.ts index 7888d5f..c0559a8 100644 --- a/src/routes/forwarder.ts +++ b/src/routes/forwarder.ts @@ -6,6 +6,9 @@ import { Logger } from "pino"; import { AuthenticationPlugin } from "../AuthenticationPlugin"; import { backendErrorCode, EventLimiter, forwardedEndpoint, tokenFingerprint } from "../diagnostics"; import { RequestIds } from "../requestIds"; +import { boundedInteger } from "../configuration"; +import { RejectionReason, Telemetry } from "../telemetry"; +import { AuthenticationQueue } from "../authenticationQueue"; /** * Checks if a given buffer contains only correct UTF-8. @@ -75,6 +78,7 @@ function isValidUTF8(buf: Buffer): boolean { export const DEFAULT_REQUEST_TIMEOUT_MS = 60_000; export const MAX_REQUEST_TIMEOUT_MS = 300_000; export const AUTHENTICATION_TIMEOUT_MS = 10_000; +export const DEFAULT_WEBSOCKET_MAX_PAYLOAD_BYTES = 4 * 1024 * 1024; export function requestTimeout(value: unknown, logger: Logger): number { if (value === undefined) return DEFAULT_REQUEST_TIMEOUT_MS; @@ -92,6 +96,10 @@ export interface ForwarderOptions { heartbeatIntervalMs?: number; rejectionTimeoutMs?: number; authenticationTimeoutMs?: number; + maxPayloadBytes?: number; + statsIntervalMs?: number; + authenticationConcurrency?: number; + authenticationQueueLimit?: number; } interface PendingResponse { @@ -125,18 +133,32 @@ export async function setupWebsockets( logger: Logger, options: ForwarderOptions = {} ) { - await authPlugin.init(logger.child({ name: "auth" })); const timeoutMs = requestTimeout(options.requestTimeoutMs ?? process.env.REQUEST_TIMEOUT_MS, logger); const heartbeatMs = options.heartbeatIntervalMs ?? 10_000; const rejectionMs = options.rejectionTimeoutMs ?? 5_000; const authenticationMs = options.authenticationTimeoutMs ?? AUTHENTICATION_TIMEOUT_MS; + const maxPayload = boundedInteger("WEBSOCKET_MAX_PAYLOAD_BYTES", + options.maxPayloadBytes ?? process.env.WEBSOCKET_MAX_PAYLOAD_BYTES, + DEFAULT_WEBSOCKET_MAX_PAYLOAD_BYTES, 16 * 1024 * 1024); + const statsIntervalMs = boundedInteger("STATS_INTERVAL_MS", + options.statsIntervalMs ?? process.env.STATS_INTERVAL_MS, 10_000, 300_000); + const authenticationConcurrency = boundedInteger("AUTH_CONCURRENCY", + options.authenticationConcurrency ?? process.env.AUTH_CONCURRENCY, 10, 100); + const authenticationQueueLimit = boundedInteger("AUTH_QUEUE_LIMIT", + options.authenticationQueueLimit ?? process.env.AUTH_QUEUE_LIMIT, 500, 10_000); + await authPlugin.init(logger.child({ name: "auth" }), { concurrency: authenticationConcurrency }); + const telemetry = new Telemetry(logger, statsIntervalMs); + const authentication = new AuthenticationQueue(authPlugin, telemetry, + authenticationConcurrency, authenticationQueueLimit, authenticationMs); const sessions = new Map(); const limiter = new EventLimiter(); let stopping = false; + let stopPromise: Promise | undefined; const server = new WebSocketServer({ host: options.host ?? "127.0.0.1", port: options.port ?? 8080, skipUTF8Validation: true, + maxPayload, }); function settle(session: Session, id: string, entry: PendingResponse, @@ -144,14 +166,23 @@ export async function setupWebsockets( if (entry.settled) return; entry.settled = true; // A delayed callback must never remove a newer entry with the same ID. - if (session.pending.get(id) === entry) session.pending.delete(id); + if (session.pending.get(id) === entry) { + session.pending.delete(id); + telemetry.adjust("pendingRequests", -1); + } clearTimeout(entry.timer); entry.detach(); const fields = { requestId: id, reason: outcome.reason, durationMs: Date.now() - entry.startedAt, endpoint: entry.endpoint, pendingRequests: session.pending.size }; session.logger.debug({ ...fields, event: "request_settled" }); + telemetry.count("requests_settled"); + telemetry.observe("request", fields.durationMs); + if (outcome.reason === "controller_response") telemetry.count("controller_responses"); + if (outcome.reason === "browser_departed") telemetry.count("browser_departures"); + if (outcome.reason === "send_failed") telemetry.count("send_failures"); if (outcome.reason === "request_timeout") { + telemetry.count("request_timeouts"); limiter.log(session.logger, "warn", "request_timeout", session.connectionId, fields); } const res = entry.res; @@ -177,6 +208,7 @@ export async function setupWebsockets( } catch { res.locals.controllerResponseForwarded = false; socket.destroy(); + telemetry.count("response_write_failures"); limiter.log(session.logger, "warn", "response_write_failed", session.connectionId); } } @@ -185,6 +217,9 @@ export async function setupWebsockets( message = "Controller disconnected.", closeCode?: number): void { if (session.closed) return; session.closed = true; + telemetry.adjust("sessions", -1); + telemetry.count("session_cleanups"); + if (reason === "heartbeat_expired") telemetry.count("heartbeat_expired"); clearInterval(session.heartbeat); if (session.onMessage) session.ws.off("message", session.onMessage); if (session.onPong) session.ws.off("pong", session.onPong); @@ -200,24 +235,32 @@ export async function setupWebsockets( } server.on("connection", (ws, req) => { + telemetry.count("connections_opened"); + telemetry.adjust("connections", 1); + telemetry.adjust("unaccepted", 1); const connectionId = randomUUID(); const connectedAt = Date.now(); let connectionLogger = logger.child({ name: "ws", connectionId, client: req.socket.remoteAddress }); let session: Session | undefined; let closed = false; + let closeObserved = false; let rejected = false; let rejectionTimer: ReturnType | undefined; - let authenticationTimer: ReturnType | undefined; - let cancelAuthentication: (() => void) | undefined; + const authenticationAbort = new AbortController(); let identity = "unspecified"; // Observe transport state before URL parsing or asynchronous auth. ws.on("close", (code) => { + if (closeObserved) return; + closeObserved = true; + telemetry.count("connections_closed"); + telemetry.adjust("connections", -1); + if (!session) telemetry.adjust("unaccepted", -1); + if (rejected) telemetry.adjust("closing", -1); closed = true; clearTimeout(rejectionTimer); - clearTimeout(authenticationTimer); - cancelAuthentication?.(); + authenticationAbort.abort(); if (session) { cleanup(session, "peer_close", undefined, code); connectionLogger.debug({ event: "connection_closed", closeCode: code, @@ -228,10 +271,14 @@ export async function setupWebsockets( } }); ws.on("error", (error) => { + if (closed) return; + telemetry.count("transport_error"); + if ((error as { code?: string }).code === "WS_ERR_UNSUPPORTED_MESSAGE_LENGTH") { + telemetry.count("oversized_message"); + } closed = true; clearTimeout(rejectionTimer); - clearTimeout(authenticationTimer); - cancelAuthentication?.(); + authenticationAbort.abort(); const fragment = error.message === "Invalid WebSocket frame: invalid opcode 0"; if (session) cleanup(session, fragment ? "fragment_error" : "transport_error", fragment ? "Controller dropped packet." : "Controller disconnected."); @@ -239,12 +286,17 @@ export async function setupWebsockets( ws.terminate(); }); - function reject(code: number, reason: string, extra: Record = {}): void { + function reject(code: number, reason: RejectionReason, extra: Record = {}): void { if (closed || rejected || ws.readyState !== WebSocket.OPEN) return; rejected = true; + telemetry.rejected(reason); + telemetry.adjust("closing", 1); + if (reason === "device key already connected") telemetry.count("duplicate_rejected"); + else if (code === 1008) telemetry.count("policy_rejected"); limiter.log(connectionLogger, "info", "connection_rejected", `${identity}:${reason}`, { reason, closeCode: code, ...extra }); rejectionTimer = setTimeout(() => { + telemetry.count("rejection_forced_close"); limiter.log(connectionLogger, "warn", "rejection_forced_close", `${identity}:${reason}`, { reason }); ws.terminate(); }, rejectionMs); @@ -267,39 +319,32 @@ export async function setupWebsockets( identity = tokenFingerprint(key); connectionLogger = connectionLogger.child({ tokenHash: identity }); + // Cheap duplicate rejection never changes ownership or enters the backend queue. + const currentOwner = sessions.get(key); + if (currentOwner && !currentOwner.closed && currentOwner.ws.readyState === WebSocket.OPEN) { + reject(1008, "device key already connected", { + ownerConnectionId: currentOwner.connectionId, ownerAgeMs: Date.now() - currentOwner.startedAt, + pendingRequests: currentOwner.pending.size, + }); + return; + } + if (currentOwner) cleanup(currentOwner, "transport_unavailable"); + async function authenticate(): Promise { if (closed || stopping || ws.readyState !== WebSocket.OPEN) return; - let valid: boolean; - let timedOut = false; - const authenticationStarted = Date.now(); - const cancelled = new Promise(resolve => { - cancelAuthentication = () => resolve(false); - authenticationTimer = setTimeout(() => { - timedOut = true; - resolve(false); - }, authenticationMs); - authenticationTimer.unref(); - }); - try { - // Promise.race also observes a late backend rejection, so it - // cannot become an unhandled rejection after timeout/close. - valid = await Promise.race([ - Promise.resolve().then(() => authPlugin.validateKey(key)), cancelled, - ]); - } - catch (error) { - reject(1011, "authentication unavailable", { errorCode: backendErrorCode(error) }); + const result = await authentication.submit(key, authenticationAbort.signal); + if (closed || stopping || ws.readyState !== WebSocket.OPEN) return; + if (result.status === "cancelled") return; + if (result.status === "busy") { reject(1011, "authentication busy"); return; } + if (result.status === "error") { + reject(1011, "authentication unavailable", { errorCode: backendErrorCode(result.error) }); return; - } finally { - clearTimeout(authenticationTimer); - cancelAuthentication = undefined; } - if (closed || stopping || ws.readyState !== WebSocket.OPEN) return; - if (timedOut || Date.now() - authenticationStarted >= authenticationMs) { + if (result.status === "timed_out") { reject(1011, "authentication timed out", { errorCode: "AUTH_TIMEOUT" }); return; } - if (!valid) { reject(1008, "invalid device key"); return; } + if (result.status === "invalid") { reject(1008, "invalid device key"); return; } // Final ownership check and claim are synchronous, with no await gap. const owner = sessions.get(key); if (owner) { @@ -313,6 +358,9 @@ export async function setupWebsockets( startedAt: connectedAt, lastPongAt: Date.now(), isAlive: true, closed: false, pending: new Map(), ids: new RequestIds() }; sessions.set(key, session); + telemetry.adjust("unaccepted", -1); + telemetry.adjust("sessions", 1); + telemetry.count("sessions_accepted"); const accepted = session; accepted.onPong = () => { if (accepted.closed) return; @@ -342,6 +390,7 @@ export async function setupWebsockets( const match = !binary && isValidUTF8(header) ? /^RES: ([0-9a-f]{4})\r$/.exec(header.toString()) : null; if (!match) { + telemetry.count("invalid_response_frame"); limiter.log(connectionLogger, "warn", "invalid_response_frame", identity, { length: buffer.length, reason: binary ? "binary" : "invalid_header" }); return; @@ -349,6 +398,7 @@ export async function setupWebsockets( const id = match[1]; const entry = accepted.pending.get(id); if (!entry) { + telemetry.count("unknown_response_id"); limiter.log(connectionLogger, "debug", "unknown_response_id", identity, { requestId: id }); return; } @@ -397,8 +447,10 @@ export async function setupWebsockets( { reason: "request_timeout", message: "Controller did not respond." }), timeoutMs); entry.timer.unref(); session.pending.set(id, entry); + telemetry.adjust("pendingRequests", 1); if (res.destroyed || !res.socket || res.socket.destroyed) { abandoned(); return; } try { + telemetry.count("forwarded_requests"); session.ws.send(`FWD: ${id}\r\n${rawRequest}`, (error) => { if (error) settle(session, id, entry, { reason: "send_failed", message: "Controller disconnected." }); }); @@ -407,11 +459,22 @@ export async function setupWebsockets( } }; - await once(server, "listening"); - return { server, forwardRequestV1, async close(): Promise { + try { await once(server, "listening"); } + catch (error) { + authentication.close(); + telemetry.close(); + await authPlugin.close?.().catch(() => {}); + throw error; + } + return { server, forwardRequestV1, stats: () => telemetry.snapshot(), close(): Promise { + if (stopPromise) return stopPromise; stopping = true; + authentication.close(); for (const session of sessions.values()) cleanup(session, "server_shutdown"); for (const client of server.clients) client.terminate(); - await new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())); + stopPromise = new Promise((resolve, reject) => + server.close(error => error ? reject(error) : resolve())) + .then(() => authPlugin.close?.()).finally(() => telemetry.close()); + return stopPromise; } }; } diff --git a/src/telemetry.ts b/src/telemetry.ts new file mode 100644 index 0000000..e34f687 --- /dev/null +++ b/src/telemetry.ts @@ -0,0 +1,118 @@ +import { monitorEventLoopDelay, performance } from "perf_hooks"; +import { Logger } from "pino"; + +const counters = { + connections_opened: 0, connections_closed: 0, sessions_accepted: 0, session_cleanups: 0, + connection_rejected: 0, duplicate_rejected: 0, policy_rejected: 0, + authentication_started: 0, authentication_valid: 0, authentication_invalid: 0, + authentication_failed: 0, authentication_timed_out: 0, authentication_cancelled: 0, + authentication_busy: 0, heartbeat_expired: 0, transport_error: 0, oversized_message: 0, + invalid_response_frame: 0, unknown_response_id: 0, forwarded_requests: 0, + requests_settled: 0, controller_responses: 0, request_timeouts: 0, + browser_departures: 0, send_failures: 0, response_write_failures: 0, + rejection_forced_close: 0, +}; +const gauges = { connections: 0, unaccepted: 0, closing: 0, sessions: 0, + authActive: 0, authQueued: 0, pendingRequests: 0 }; +const rejectionCounts = { "invalid path": 0, "missing device key": 0, "invalid device key": 0, + "device key already connected": 0, "authentication unavailable": 0, + "authentication timed out": 0, "authentication busy": 0 }; +export type RejectionReason = keyof typeof rejectionCounts; + +// Fixed buckets/labels: no samples, keys, URLs or per-device metric series. +class Histogram { + private readonly bounds = [1, 5, 10, 50, 100, 500, 1000, 5000, 10_000, 60_000]; + private counts = Array(this.bounds.length + 1).fill(0) as number[]; + private count = 0; + private sumMs = 0; + private maxMs = 0; + + observe(ms: number): void { + const value = Math.max(0, ms); + const index = this.bounds.findIndex(bound => value <= bound); + this.counts[index < 0 ? this.bounds.length : index]++; + this.count++; + this.sumMs += value; + this.maxMs = Math.max(this.maxMs, value); + } + + snapshot() { + return { boundsMs: this.bounds.slice(), buckets: this.counts.slice(), + count: this.count, sumMs: this.sumMs, maxMs: this.maxMs }; + } +} + +export class Telemetry { + private totals = { ...counters }; + private previous = { ...counters }; + private current = { ...gauges }; + private highWater = { ...gauges }; + private rejections = { ...rejectionCounts }; + private histograms = { authentication: new Histogram(), lookup: new Histogram(), + queueWait: new Histogram(), request: new Histogram() }; + private readonly loop = monitorEventLoopDelay({ resolution: 20 }); + private utilization = performance.eventLoopUtilization(); + private cpu = process.cpuUsage(); + private readonly startedAt = performance.now(); + private reportedAt = this.startedAt; + private readonly timer: ReturnType; + + constructor(private readonly logger: Logger, intervalMs = 10_000) { + this.loop.enable(); + this.timer = setInterval(() => this.report(), intervalMs); + this.timer.unref(); + } + + count(name: keyof typeof counters): void { this.totals[name]++; } + + rejected(reason: RejectionReason): void { + this.count("connection_rejected"); + this.rejections[reason]++; + } + + adjust(name: keyof typeof gauges, change: number): void { + this.current[name] += change; + this.highWater[name] = Math.max(this.highWater[name], this.current[name]); + } + + observe(name: keyof Telemetry["histograms"], ms: number): void { + this.histograms[name].observe(ms); + } + + snapshot() { + return { counters: { ...this.totals }, rejections: { ...this.rejections }, + gauges: { ...this.current }, highWater: { ...this.highWater }, + latency: { authentication: this.histograms.authentication.snapshot(), + lookup: this.histograms.lookup.snapshot(), queueWait: this.histograms.queueWait.snapshot(), + request: this.histograms.request.snapshot() } }; + } + + report(): void { + const now = performance.now(); + const cpuNow = process.cpuUsage(); + const utilizationNow = performance.eventLoopUtilization(); + const usage = process.memoryUsage(); + const delta = { ...this.totals }; + for (const name of Object.keys(delta) as (keyof typeof counters)[]) { + delta[name] -= this.previous[name]; + } + this.logger.info({ event: "proxy_stats", ...this.snapshot(), delta, + intervalMs: now - this.reportedAt, uptimeMs: now - this.startedAt, + memory: { rss: usage.rss, heapUsed: usage.heapUsed, heapTotal: usage.heapTotal, + external: usage.external, arrayBuffers: usage.arrayBuffers }, + cpu: { userMicros: cpuNow.user - this.cpu.user, systemMicros: cpuNow.system - this.cpu.system }, + eventLoop: { utilization: performance.eventLoopUtilization(utilizationNow, this.utilization).utilization, + p99Ms: this.loop.percentile(99) / 1e6, maxMs: this.loop.max / 1e6 }, + }); + this.previous = { ...this.totals }; + this.cpu = cpuNow; + this.utilization = utilizationNow; + this.reportedAt = now; + this.loop.reset(); + } + + close(): void { + clearInterval(this.timer); + this.loop.disable(); + } +} diff --git a/tests/authentication.test.cjs b/tests/authentication.test.cjs new file mode 100644 index 0000000..ff37b18 --- /dev/null +++ b/tests/authentication.test.cjs @@ -0,0 +1,143 @@ +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const { AuthenticationQueue, Telemetry, setupWebsockets } = require(process.env.OTF_TEST_BUNDLE); +const { fixture, capture, until, delay, receive, reply, id } = require('./helpers.cjs'); + +test('auth limits are validated before initializing a backend', async () => { + for (const [option, setting, values] of [ + ['authenticationConcurrency', 'AUTH_CONCURRENCY', [0, -1, 1.1, 101, 'secret-invalid']], + ['authenticationQueueLimit', 'AUTH_QUEUE_LIMIT', [0, -1, 1.1, 10001, 'secret-invalid']], + ]) { + for (const value of values) { + let initialized = false; + await assert.rejects(setupWebsockets({ init: async () => { initialized = true; } }, + capture().logger, { port: 0, [option]: value }), error => { + assert.equal(error.message, `${setting} is invalid`); + return true; + }); + assert.equal(initialized, false); + } + } +}); + +test('60 disconnected clients leave at most 10 running lookups and no waiting work', async t => { + const complete = []; + const f = await fixture(t, { auth: { init: async () => {}, + validateKey: () => new Promise(resolve => complete.push(resolve)) } }); + const clients = await Promise.all(Array.from({ length: 60 }, (_, n) => f.connect({ key: `fake-${n}` }))); + assert.equal(complete.length, 10); + assert.equal(f.stats().gauges.authActive, 10); + assert.equal(f.stats().gauges.authQueued, 50); + for (const client of clients) client.ws.close(); + await Promise.all(clients.map(client => client.closed)); + await until(() => f.stats().gauges.authQueued === 0 && f.stats().gauges.connections === 0); + assert.equal(f.stats().gauges.authActive, 10); + assert.equal(complete.length, 10); + for (const finish of complete) finish(true); + await until(() => f.stats().gauges.authActive === 0); + assert.equal(f.stats().counters.sessions_accepted, 0); + assert.equal(f.stats().counters.authentication_cancelled, 60); +}); + +test('queue-full and same-key contention reject promptly without extra backend calls', async t => { + const calls = []; + const f = await fixture(t, { authenticationConcurrency: 1, authenticationQueueLimit: 1, + auth: { init: async () => {}, validateKey: key => { calls.push(key); return new Promise(() => {}); } } }); + const running = await f.connect({ key: 'running' }); + const queued = await f.connect({ key: 'queued' }); + const duplicate = await f.connect({ key: 'running' }); + const full = await f.connect({ key: 'excess' }); + for (const client of [duplicate, full]) { + assert.deepEqual(await client.closed, { code: 1011, reason: 'authentication busy' }); + } + assert.deepEqual(calls, ['running']); + assert.equal(f.stats().gauges.authActive, 1); + assert.equal(f.stats().gauges.authQueued, 1); + assert.equal(f.stats().counters.authentication_busy, 2); + assert.equal(f.stats().gauges.sessions, 0); + running.ws.close(); + queued.ws.close(); +}); + +test('expired waiters never enter the plugin and running slots survive socket deadlines', async t => { + let finish; + const calls = []; + const f = await fixture(t, { authenticationConcurrency: 1, authenticationTimeoutMs: 70, + auth: { init: async () => {}, validateKey: key => { + calls.push(key); return new Promise(resolve => { finish = resolve; }); + } } }); + const running = await f.connect({ key: 'running' }); + const queued = await f.connect({ key: 'queued' }); + assert.equal((await running.closed).reason, 'authentication timed out'); + assert.equal((await queued.closed).reason, 'authentication timed out'); + assert.deepEqual(calls, ['running']); + assert.equal(f.stats().gauges.authActive, 1); + assert.equal(f.stats().gauges.authQueued, 0); + finish(true); + await until(() => f.stats().gauges.authActive === 0); + assert.equal(f.stats().counters.sessions_accepted, 0); +}); + +test('backend outage and busy queue do not prevent an established owner forwarding', async t => { + let lookups = 0; + const f = await fixture(t, { authenticationConcurrency: 1, authenticationQueueLimit: 1, + auth: { init: async () => {}, validateKey: async () => { + if (++lookups === 1) return true; + return new Promise(() => {}); + } } }); + const owner = await f.connect(); + await owner.accepted(); + await f.connect({ key: 'hung-backend' }); + await f.connect({ key: 'queued' }); + const overflow = await f.connect({ key: 'overflow' }); + assert.equal((await overflow.closed).reason, 'authentication busy'); + const duplicate = await f.connect(); + assert.equal((await duplicate.closed).reason, 'device key already connected'); + assert.equal(lookups, 2); + const incoming = receive(owner.ws); + const pending = f.request(); + reply(owner.ws, id(await incoming), 'STILL HEALTHY'); + assert.equal((await pending.response).body.toString(), 'STILL HEALTHY'); +}); + +test('queue wait consumes the same deadline and FIFO jobs get only remaining time', async t => { + let finish; + let remaining; + const f = await fixture(t, { authenticationConcurrency: 1, authenticationTimeoutMs: 300, + auth: { init: async () => {}, validateKey: (key, context) => { + if (key === 'first') return new Promise(resolve => { finish = resolve; }); + remaining = context.remainingMs(); + assert.equal(context.signal.aborted, false); + return Promise.resolve(true); + } } }); + const first = await f.connect({ key: 'first' }); + const second = await f.connect({ key: 'second' }); + await delay(60); + finish(true); + await first.accepted(); + await second.accepted(); + assert.ok(remaining > 0 && remaining <= 250, String(remaining)); + assert.equal(f.stats().latency.queueWait.count, 2); +}); + +test('a delayed timer cannot let an expired queue entry start SQL', async () => { + const logs = capture(); + const telemetry = new Telemetry(logs.logger); + let finish; + const calls = []; + const queue = new AuthenticationQueue({ validateKey: key => { + calls.push(key); return new Promise(resolve => { finish = resolve; }); + } }, telemetry, 1, 2, 25); + try { + const first = queue.submit('first', new AbortController().signal); + await until(() => finish); + const second = queue.submit('second', new AbortController().signal); + const end = performance.now() + 40; + while (performance.now() < end) { /* delay timers deliberately */ } + finish(true); + assert.equal((await first).status, 'timed_out'); + assert.equal((await second).status, 'timed_out'); + assert.deepEqual(calls, ['first']); + await until(() => telemetry.snapshot().gauges.authActive === 0); + } finally { queue.close(); telemetry.close(); } +}); diff --git a/tests/exports.ts b/tests/exports.ts index 027e6e4..98179ef 100644 --- a/tests/exports.ts +++ b/tests/exports.ts @@ -1,7 +1,10 @@ -export { setupWebsockets, requestTimeout, DEFAULT_REQUEST_TIMEOUT_MS } from "../src/routes/forwarder"; +export { setupWebsockets, requestTimeout, DEFAULT_REQUEST_TIMEOUT_MS, + DEFAULT_WEBSOCKET_MAX_PAYLOAD_BYTES } from "../src/routes/forwarder"; export { createHttpApp } from "../src/httpApp"; export { RequestIds } from "../src/requestIds"; export { EventLimiter, tokenFingerprint, backendErrorCode } from "../src/diagnostics"; export { default as MySQLAuthenticationPlugin } from "../src/authenticationPlugins/MySQLAuthenticationPlugin"; export { ConfigurationError, startupFailure } from "../src/configuration"; export { forwardedEndpoint } from "../src/diagnostics"; +export { Telemetry } from "../src/telemetry"; +export { AuthenticationQueue } from "../src/authenticationQueue"; diff --git a/tests/helpers.cjs b/tests/helpers.cjs index 9927744..36b35ec 100644 --- a/tests/helpers.cjs +++ b/tests/helpers.cjs @@ -3,6 +3,7 @@ const { once } = require('node:events'); const http = require('node:http'); const WebSocket = require('ws'); const pino = require('pino'); +const { randomUUID } = require('node:crypto'); const { setupWebsockets, createHttpApp } = require(process.env.OTF_TEST_BUNDLE); const token = 'local-only-fake-token'; @@ -31,6 +32,12 @@ async function fixture(t, options = {}) { const server = createHttpApp(forward, logs.logger).listen(0, '127.0.0.1'); await once(server, 'listening'); const clients = new Set(); + const connecting = new Map(); + proxy.server.on('connection', (socket, req) => { + const tag = req.headers['x-otf-test-connection']; + const resolve = connecting.get(tag); + if (resolve) { connecting.delete(tag); resolve(socket); } + }); t.after(async () => { for (const client of clients) client.terminate(); server.closeAllConnections(); @@ -40,13 +47,15 @@ async function fixture(t, options = {}) { }); async function connect({ key = token, path = '/socket/v1', query, ...wsOptions } = {}) { const acceptedBefore = logs.records().filter(r => r.event === 'session_accepted').length; - const serverConnection = once(proxy.server, 'connection'); - const ws = new WebSocket(`ws://127.0.0.1:${proxy.server.address().port}${path}${query ?? `?deviceKey=${key}`}`, wsOptions); + const tag = randomUUID(); + const serverConnection = new Promise(resolve => connecting.set(tag, resolve)); + const ws = new WebSocket(`ws://127.0.0.1:${proxy.server.address().port}${path}${query ?? `?deviceKey=${key}`}`, + { ...wsOptions, headers: { ...wsOptions.headers, 'x-otf-test-connection': tag } }); clients.add(ws); ws.on('error', () => {}); const closed = new Promise(resolve => ws.once('close', (code, reason) => resolve({ code, reason: reason.toString() }))); await once(ws, 'open'); - const [socket] = await serverConnection; + const socket = await serverConnection; return { ws, socket, closed, accepted: () => until(() => logs.records().filter(r => r.event === 'session_accepted').length > acceptedBefore) }; } diff --git a/tests/lifecycle.test.cjs b/tests/lifecycle.test.cjs index fce349f..23fd691 100644 --- a/tests/lifecycle.test.cjs +++ b/tests/lifecycle.test.cjs @@ -11,18 +11,19 @@ test('simultaneous delayed authentication accepts only one owner and gates proto const second = await f.connect(); assert.equal(first.socket.listenerCount('message'), 0); assert.equal(second.socket.listenerCount('message'), 0); - validations[1](true); - await second.accepted(); - const incoming = receive(second.ws); + assert.deepEqual(await second.closed, { code: 1011, reason: 'authentication busy' }); + assert.equal(validations.length, 1); + validations[0](true); + await first.accepted(); + const incoming = receive(first.ws); const pending = f.request(); const requestId = id(await incoming); // The authenticated-but-not-yet-accepted connection cannot answer for its owner. - reply(first.ws, requestId, 'FORGED'); + // The rejected contender cannot install a response handler. + assert.equal(second.socket.listenerCount('message'), 0); await delay(10); assert.equal(f.logs.records().filter(r => r.event === 'request_settled').length, 0); - validations[0](true); - assert.equal((await first.closed).code, 1008); - reply(second.ws, requestId, 'OWNER'); + reply(first.ws, requestId, 'OWNER'); assert.equal((await pending.response).body.toString(), 'OWNER'); assert.equal(f.logs.records().filter(r => r.event === 'session_accepted').length, 1); }); @@ -85,7 +86,7 @@ test('invalid authenticated duplicate cannot displace the owner', async t => { await owner.accepted(); valid = false; const duplicate = await f.connect(); - assert.equal((await duplicate.closed).reason, 'invalid device key'); + assert.equal((await duplicate.closed).reason, 'device key already connected'); assert.equal(owner.ws.readyState, WebSocket.OPEN); }); @@ -101,6 +102,7 @@ test('old close, error and heartbeat callbacks cannot affect replacement request await old.accepted(); const lateClose = old.socket.listeners('close')[0]; const lateError = old.socket.listeners('error')[0]; + const oldCallbacks = callbacks.slice(); const oldIncoming = receive(old.ws); const oldRequest = f.request(); await oldIncoming; @@ -114,7 +116,7 @@ test('old close, error and heartbeat callbacks cannot affect replacement request const requestId = id(await incoming); lateClose(1006); lateError(new Error('old transport error')); - callbacks[0](); + for (const callback of oldCallbacks) callback(); lateClose(1006); reply(replacement.ws, requestId, 'NEW'); assert.equal((await pending.response).body.toString(), 'NEW'); @@ -192,10 +194,14 @@ for (const outcome of ['success', 'error']) { assert.deepEqual(await old.closed, { code: 1011, reason: 'authentication timed out' }); assert.equal(old.socket.listenerCount('message'), 0); assert.equal((await f.request().response).status, 404); - const replacement = await f.connect(); - await replacement.accepted(); + const contender = await f.connect(); + assert.deepEqual(await contender.closed, { code: 1011, reason: 'authentication busy' }); + assert.equal(calls, 1); if (outcome === 'success') complete(true); else fail(new Error('late backend secret')); + await until(() => f.stats().gauges.authActive === 0); + const replacement = await f.connect(); + await replacement.accepted(); const incoming = receive(replacement.ws); const pending = f.request(); reply(replacement.ws, id(await incoming)); @@ -205,7 +211,7 @@ for (const outcome of ['success', 'error']) { }); } -test('an authenticating duplicate timing out preserves its healthy owner and pending requests', async t => { +test('a duplicate skips authentication and preserves its healthy owner and pending requests', async t => { let calls = 0; const f = await fixture(t, { authenticationTimeoutMs: 30, auth: { init: async () => {}, validateKey: () => ++calls === 1 ? Promise.resolve(true) : new Promise(() => {}), @@ -216,7 +222,8 @@ test('an authenticating duplicate timing out preserves its healthy owner and pen const pending = f.request(); const requestId = id(await incoming); const duplicate = await f.connect(); - assert.equal((await duplicate.closed).code, 1011); + assert.deepEqual(await duplicate.closed, { code: 1008, reason: 'device key already connected' }); + assert.equal(calls, 1); reply(owner.ws, requestId); assert.equal((await pending.response).status, 200); assert.equal(owner.ws.readyState, WebSocket.OPEN); @@ -236,7 +243,7 @@ test('authentication deadline timers are cancelled on acceptance and peer depart const owner = await f.connect(); await owner.accepted(); hung = true; - const abandoned = await f.connect(); + const abandoned = await f.connect({ key: 'another-fake-key' }); abandoned.ws.close(); await abandoned.closed; await delay(50); diff --git a/tests/mysql.integration.test.cjs b/tests/mysql.integration.test.cjs new file mode 100644 index 0000000..2bd09e7 --- /dev/null +++ b/tests/mysql.integration.test.cjs @@ -0,0 +1,157 @@ +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const mysql = require('mysql'); +const { MySQLAuthenticationPlugin } = require(process.env.OTF_TEST_BUNDLE); +const { fixture, capture, token, delay, until, receive, reply, id } = require('./helpers.cjs'); + +const options = { skip: process.env.OTF_TEST_MYSQL_ISOLATED !== '1' }; +let sequence = 0; + +async function database(t) { + // Never fall back to the application's real authentication configuration. + const url = process.env.OTF_TEST_MYSQL_URL; + assert.equal(new URL(url).pathname, '/otf_test', 'Use a disposable otf_test database only'); + const admin = mysql.createConnection(url); + const sql = (query, args = []) => new Promise((resolve, reject) => + admin.query(query, args, (error, rows) => error ? reject(error) : resolve(rows))); + const table = `otf_test_${process.pid}_${sequence++}`; + await sql('CREATE TABLE ?? (device_key VARCHAR(128) PRIMARY KEY)', [table]); + await sql('INSERT INTO ?? VALUES (?), (?)', [table, token, 'second-fake-token']); + t.after(async () => { + await sql('UNLOCK TABLES'); + await sql('DROP TABLE IF EXISTS ??', [table]); + await new Promise((resolve, reject) => admin.end(error => error ? reject(error) : resolve())); + }); + async function configure(action) { + const previous = { MYSQL_CONNECTION_URL: process.env.MYSQL_CONNECTION_URL, MYSQL_TABLE: process.env.MYSQL_TABLE }; + try { + process.env.MYSQL_CONNECTION_URL = url; + process.env.MYSQL_TABLE = table; + return await action(); + } finally { + for (const [key, value] of Object.entries(previous)) { + if (value === undefined) delete process.env[key]; else process.env[key] = value; + } + } + } + async function plugin(concurrency = 1) { + const instance = new MySQLAuthenticationPlugin(); + await configure(() => instance.init(capture().logger, { concurrency })); + t.after(() => instance.close()); + return instance; + } + async function proxy(extra = {}) { + const auth = new MySQLAuthenticationPlugin(); + const f = await configure(() => fixture(t, { auth, authenticationConcurrency: 1, ...extra })); + return { ...f, auth }; + } + async function blockedQuery() { + for (let n = 0; n < 100; n++) { + const rows = await sql('SHOW FULL PROCESSLIST'); + if (rows.some(row => row.Info?.includes(`SELECT 1 FROM \`${table}\``))) return; + await delay(10); + } + assert.fail('Expected the actual authentication SELECT to reach the locked table'); + } + return { sql, table, plugin, proxy, blockedQuery }; +} + +test('isolated MySQL: indexed existence lookup accepts and rejects keys', options, async t => { + const db = await database(t); + const plugin = await db.plugin(); + assert.equal(await plugin.validateKey(token), true); + assert.equal(await plugin.validateKey('unregistered-fake-token'), false); + assert.equal(plugin.pool._connectionQueue.length, 0); +}); + +test('isolated MySQL: occupied pool fails promptly instead of retaining a hidden queue', options, async t => { + const db = await database(t); + const plugin = await db.plugin(); + const reserved = await new Promise((resolve, reject) => plugin.pool.getConnection((err, c) => err ? reject(err) : resolve(c))); + try { + const start = performance.now(); + await assert.rejects(plugin.validateKey(token), error => error.code === 'POOL_CONNLIMIT'); + assert.ok(performance.now() - start < 1000); + assert.equal(plugin.pool._connectionQueue.length, 0); + } finally { reserved.release(); } + assert.equal(await plugin.validateKey(token), true); +}); + +test('isolated MySQL: abort during actual connection acquisition prevents SQL', options, async t => { + const db = await database(t); + const plugin = await db.plugin(); + const abort = new AbortController(); + let queries = 0; + plugin.pool.on('connection', connection => { + const original = connection.query; + connection.query = function (...args) { queries++; return original.apply(this, args); }; + }); + const abandon = () => abort.abort(); + plugin.pool.on('acquire', abandon); + await assert.rejects(plugin.validateKey(token, { signal: abort.signal, remainingMs: () => 1000 }), + error => error.code === 'AUTH_TIMEOUT'); + assert.equal(queries, 0); + assert.equal(plugin.pool._freeConnections.length, 1); + plugin.pool.off('acquire', abandon); + assert.equal(await plugin.validateKey(token), true); + assert.equal(queries, 1); +}); + +test('isolated MySQL: disconnected running SQL retains its slot and abandoned waiter never runs', options, async t => { + const db = await database(t); + const f = await db.proxy({ authenticationTimeoutMs: 2000 }); + await db.sql('LOCK TABLES ?? WRITE', [db.table]); + const running = await f.connect(); + await db.blockedQuery(); + const queued = await f.connect({ key: 'second-fake-token' }); + assert.equal(f.stats().gauges.authQueued, 1); + running.ws.close(); + queued.ws.close(); + await Promise.all([running.closed, queued.closed]); + await until(() => f.stats().gauges.authQueued === 0); + assert.equal(f.stats().gauges.authActive, 1); + assert.equal(f.stats().counters.authentication_started, 1); + await db.sql('UNLOCK TABLES'); + await until(() => f.stats().gauges.authActive === 0); + assert.equal(f.stats().counters.authentication_valid, 1); + assert.equal(f.stats().counters.sessions_accepted, 0); + const next = await f.connect(); + await next.accepted(); + assert.equal(f.stats().counters.authentication_started, 2); +}); + +test('isolated MySQL: real query inactivity timeout releases the slot and recovery succeeds', options, async t => { + const db = await database(t); + const f = await db.proxy({ authenticationTimeoutMs: 200 }); + await db.sql('LOCK TABLES ?? WRITE', [db.table]); + const start = performance.now(); + const running = await f.connect(); + await db.blockedQuery(); + assert.equal((await running.closed).code, 1011); + await until(() => f.stats().gauges.authActive === 0); + const elapsed = performance.now() - start; + assert.ok(elapsed >= 150 && elapsed < 1500, String(elapsed)); + assert.equal(f.auth.pool._connectionQueue.length, 0); + assert.equal(f.stats().counters.authentication_failed, 1); + await db.sql('UNLOCK TABLES'); + const recovered = await f.connect(); + await recovered.accepted(); + assert.equal(f.stats().gauges.sessions, 1); +}); + +test('isolated MySQL: backend errors are sanitized and established forwarding survives', options, async t => { + const db = await database(t); + const f = await db.proxy(); + const owner = await f.connect(); + await owner.accepted(); + f.auth.table = 'otf_test_missing_table'; + const rejected = await f.connect({ key: 'second-fake-token' }); + assert.deepEqual(await rejected.closed, { code: 1011, reason: 'authentication unavailable' }); + const incoming = receive(owner.ws); + const request = f.request(); + reply(owner.ws, id(await incoming), 'HEALTHY'); + assert.equal((await request.response).body.toString(), 'HEALTHY'); + assert.equal(f.logs.text().includes(token), false); + assert.equal(f.logs.text().includes('SELECT 1'), false); + assert.equal(f.logs.text().includes('sqlMessage'), false); +}); diff --git a/tests/mysql.test.cjs b/tests/mysql.test.cjs new file mode 100644 index 0000000..38dda47 --- /dev/null +++ b/tests/mysql.test.cjs @@ -0,0 +1,81 @@ +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const { MySQLAuthenticationPlugin } = require(process.env.OTF_TEST_BUNDLE); +const { capture, token } = require('./helpers.cjs'); + +test('MySQL pool limits override URL options and driver SQL debugging is disabled', async () => { + const previous = { MYSQL_CONNECTION_URL: process.env.MYSQL_CONNECTION_URL, MYSQL_TABLE: process.env.MYSQL_TABLE }; + const plugin = new MySQLAuthenticationPlugin(); + try { + process.env.MYSQL_CONNECTION_URL = 'mysql://fake:fake@localhost/otf_test?connectionLimit=999&queueLimit=0&waitForConnections=true&debug=true'; + process.env.MYSQL_TABLE = 'fake_devices'; + await plugin.init(capture().logger, { concurrency: 3 }); + assert.equal(plugin.pool.config.connectionLimit, 3); + assert.equal(plugin.pool.config.waitForConnections, false); + assert.equal(plugin.pool.config.queueLimit, 1); + assert.equal(plugin.pool.config.connectionConfig.debug, false); + assert.equal(plugin.pool.config.acquireTimeout, 10_000); + } finally { + await plugin.close(); + for (const [name, value] of Object.entries(previous)) { + if (value === undefined) delete process.env[name]; else process.env[name] = value; + } + } +}); + +test('caller disappearance during acquisition skips SQL and releases the connection once', async () => { + const plugin = new MySQLAuthenticationPlugin(); + let acquired; + let releases = 0; + const abort = new AbortController(); + plugin.pool = { getConnection: callback => { acquired = callback; } }; + const operation = plugin.validateKey(token, { signal: abort.signal, remainingMs: () => 500 }); + abort.abort(); + acquired(null, { release: () => releases++, query: () => assert.fail('abandoned lookup ran SQL') }); + await assert.rejects(operation, error => error.code === 'AUTH_TIMEOUT'); + assert.equal(releases, 1); +}); + +test('driver query timeout uses remaining time and release waits for query completion', async () => { + const plugin = new MySQLAuthenticationPlugin(); + plugin.table = 'fake_devices'; + let remaining = 200; + let acquired; + let completed; + let releases = 0; + plugin.pool = { getConnection: callback => { acquired = callback; } }; + const operation = plugin.validateKey(token, { signal: new AbortController().signal, remainingMs: () => remaining }); + remaining = 47.9; + acquired(null, { release: () => releases++, query: (options, args, callback) => { + assert.equal(options.sql, 'SELECT 1 FROM ?? WHERE device_key = ? LIMIT 1'); + assert.equal(options.timeout, 47); + assert.deepEqual(args, ['fake_devices', token]); + completed = callback; + } }); + assert.equal(releases, 0); + completed(null, [{}]); + assert.equal(await operation, true); + assert.equal(releases, 1); +}); + +test('deadline after acquisition and synchronous driver errors never leak SQL or start stale work', async () => { + for (const stale of [true, false]) { + const plugin = new MySQLAuthenticationPlugin(); + let acquired; + let remaining = 100; + let releases = 0; + plugin.pool = { getConnection: callback => { acquired = callback; } }; + const operation = plugin.validateKey(token, { signal: new AbortController().signal, remainingMs: () => remaining }); + if (stale) remaining = 0; + acquired(null, { release: () => releases++, query: () => { + assert.equal(stale, false); + throw Object.assign(new Error(token), { code: 'ER_PARSE_ERROR', sql: token }); + } }); + await assert.rejects(operation, error => { + assert.equal(error.code, stale ? 'AUTH_TIMEOUT' : 'ER_PARSE_ERROR'); + assert.equal(JSON.stringify(error).includes(token), false); + return true; + }); + assert.equal(releases, 1); + } +}); diff --git a/tests/payload.test.cjs b/tests/payload.test.cjs new file mode 100644 index 0000000..60933f4 --- /dev/null +++ b/tests/payload.test.cjs @@ -0,0 +1,90 @@ +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const { fixture, receive, id, capture, until } = require('./helpers.cjs'); +const { setupWebsockets, DEFAULT_WEBSOCKET_MAX_PAYLOAD_BYTES } = require(process.env.OTF_TEST_BUNDLE); + +test('incoming message cap defaults to 4 MiB and rejects invalid configuration without values', async t => { + const f = await fixture(t); + assert.equal(f.server.options.maxPayload, 4 * 1024 * 1024); + assert.equal(DEFAULT_WEBSOCKET_MAX_PAYLOAD_BYTES, f.server.options.maxPayload); + for (const value of [0, -1, 0.5, Infinity, 16 * 1024 * 1024 + 1, 'secret-invalid-value']) { + const logs = capture(); + await assert.rejects(setupWebsockets({ init: async () => {} }, logs.logger, + { port: 0, maxPayloadBytes: value }), error => { + assert.equal(error.message, 'WEBSOCKET_MAX_PAYLOAD_BYTES is invalid'); + assert.equal(JSON.stringify(error).includes('secret-invalid-value'), false); + return true; + }); + } +}); + +for (const size of [511, 512, 513]) { + test(`message size ${size} against a 512-byte cap is checked across fragments`, async t => { + const f = await fixture(t, { maxPayloadBytes: 512 }); + const controller = await f.connect(); + await controller.accepted(); + const incoming = receive(controller.ws); + const pending = f.request(); + const requestId = id(await incoming); + // Equal-width body lengths keep the complete envelope length predictable. + const prefix = `RES: ${requestId}\r\nHTTP/1.1 200 OK\r\nContent-Length: 000\r\n\r\n`; + const body = 'x'.repeat(size - Buffer.byteLength(prefix)); + const frame = Buffer.from(prefix.replace('000', String(body.length)) + body); + assert.equal(frame.length, size); + controller.ws.send(frame.subarray(0, 256), { binary: false, fin: false }); + controller.ws.send(frame.subarray(256), { binary: false, fin: true }); + const result = await pending.response; + if (size <= 512) { + assert.equal(result.status, 200); + assert.equal(result.body.toString(), body); + assert.equal(controller.ws.readyState, 1); + } else { + assert.equal((await controller.closed).code, 1009); + assert.equal(result.status, 408); + } + }); +} + +test('oversized fragmented messages are rejected before delayed authentication completes', async t => { + let complete; + const f = await fixture(t, { maxPayloadBytes: 512, auth: { + init: async () => {}, validateKey: () => new Promise(resolve => { complete = resolve; }), + } }); + const controller = await f.connect(); + await until(() => complete); + controller.ws.send('x'.repeat(300), { fin: false }); + controller.ws.send('x'.repeat(300), { fin: true }); + assert.equal((await controller.closed).code, 1009); + complete(true); + assert.equal((await f.request().response).status, 404); + assert.equal(f.logs.records().some(r => r.event === 'session_accepted'), false); +}); + +for (const offset of [-1, 0, 1]) { + test(`default 4 MiB message boundary ${offset >= 0 ? '+' : ''}${offset} works across fragments`, async t => { + const f = await fixture(t, { requestTimeoutMs: 5000 }); + const controller = await f.connect(); + await controller.accepted(); + const incoming = receive(controller.ws); + const pending = f.request(); + const requestId = id(await incoming); + const size = DEFAULT_WEBSOCKET_MAX_PAYLOAD_BYTES + offset; + let prefix = `RES: ${requestId}\r\nHTTP/1.1 200 OK\r\nContent-Length: 0000000\r\n\r\n`; + const bodyLength = size - Buffer.byteLength(prefix); + prefix = prefix.replace('0000000', String(bodyLength)); + const frame = Buffer.alloc(size, 0x78); + frame.write(prefix); + const middle = Math.floor(size / 2); + controller.ws.send(frame.subarray(0, middle), { binary: false, fin: false }); + controller.ws.send(frame.subarray(middle), { binary: false, fin: true }); + const response = await pending.response; + if (offset <= 0) { + assert.equal(response.status, 200); + assert.equal(response.body.length, bodyLength); + assert.ok(response.body.every(byte => byte === 0x78)); + } else { + assert.equal((await controller.closed).code, 1009); + assert.equal(response.status, 408); + } + }); +} diff --git a/tests/telemetry.test.cjs b/tests/telemetry.test.cjs new file mode 100644 index 0000000..6be069d --- /dev/null +++ b/tests/telemetry.test.cjs @@ -0,0 +1,103 @@ +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const { Telemetry, EventLimiter } = require(process.env.OTF_TEST_BUNDLE); +const { fixture, capture, token, until, delay, receive, reply, id } = require('./helpers.cjs'); + +test('aggregate counters and fixed histograms survive suppression and report interval deltas', () => { + const logs = capture(); + const telemetry = new Telemetry(logs.logger); + try { + for (let n = 0; n < 10_000; n++) { + telemetry.rejected('invalid device key'); + telemetry.observe('lookup', n); + } + telemetry.adjust('authQueued', 5); + telemetry.adjust('authQueued', -5); + telemetry.report(); + telemetry.report(); + const [first, second] = logs.records(); + assert.equal(first.delta.connection_rejected, 10_000); + assert.equal(first.rejections['invalid device key'], 10_000); + assert.equal(second.delta.connection_rejected, 0); + assert.equal(second.counters.connection_rejected, 10_000); + assert.equal(first.highWater.authQueued, 5); + assert.equal(first.gauges.authQueued, 0); + assert.equal(first.latency.lookup.buckets.length, 11); + assert.equal(first.latency.lookup.buckets.reduce((a, b) => a + b), 10_000); + assert.equal(typeof first.memory.external, 'number'); + assert.equal(typeof first.eventLoop.p99Ms, 'number'); + const snapshot = telemetry.snapshot(); + snapshot.counters.connection_rejected = -1; + snapshot.latency.lookup.buckets.fill(-1); + assert.equal(telemetry.snapshot().counters.connection_rejected, 10_000); + assert.ok(telemetry.snapshot().latency.lookup.buckets.every(n => n >= 0)); + assert.equal(logs.text().includes(token), false); + } finally { telemetry.close(); } +}); + +test('admission log bursts cannot consume the protocol or request warning budgets', () => { + const logs = capture(); + const limiter = new EventLimiter(); + for (let n = 0; n < 1000; n++) limiter.log(logs.logger, 'info', 'connection_rejected', String(n)); + limiter.log(logs.logger, 'warn', 'request_timeout', 'one'); + limiter.log(logs.logger, 'warn', 'invalid_response_frame', 'one'); + assert.equal(logs.records().length, 102); + assert.equal(logs.records().at(-2).event, 'request_timeout'); + assert.equal(logs.records().at(-1).event, 'invalid_response_frame'); + assert.ok(limiter.entries.size <= 256); +}); + +test('socket/session/pending gauges settle once and duplicate counters include suppressed events', async t => { + const f = await fixture(t); + const controller = await f.connect(); + await controller.accepted(); + for (let n = 0; n < 20; n++) { + const duplicate = await f.connect(); + assert.equal((await duplicate.closed).code, 1008); + } + await until(() => f.stats().gauges.connections === 1); + assert.equal(f.stats().counters.duplicate_rejected, 20); + assert.equal(f.stats().rejections['device key already connected'], 20); + assert.equal(f.logs.records().filter(r => r.event === 'connection_rejected').length, 1); + const incoming = receive(controller.ws); + const pending = f.request(); + const requestId = id(await incoming); + assert.equal(f.stats().gauges.pendingRequests, 1); + reply(controller.ws, requestId); + assert.equal((await pending.response).status, 200); + assert.equal(f.stats().gauges.pendingRequests, 0); + const lateClose = controller.socket.listeners('close')[0]; + controller.ws.close(); + await controller.closed; + await until(() => f.stats().gauges.connections === 0); + lateClose(1006); + assert.deepEqual(f.stats().gauges, { connections: 0, unaccepted: 0, closing: 0, sessions: 0, + authActive: 0, authQueued: 0, pendingRequests: 0 }); + assert.equal(f.stats().counters.session_cleanups, 1); + assert.equal(f.stats().counters.requests_settled, 1); +}); + +test('disconnected authentication remains counted until actual backend completion', async t => { + let finish; + const f = await fixture(t, { auth: { init: async () => {}, + validateKey: () => new Promise(resolve => { finish = resolve; }) } }); + const client = await f.connect(); + await until(() => finish); + client.ws.close(); + await client.closed; + await until(() => f.stats().gauges.connections === 0); + assert.equal(f.stats().gauges.authActive, 1); + finish(true); + await until(() => f.stats().gauges.authActive === 0); + assert.equal(f.stats().counters.sessions_accepted, 0); +}); + +test('periodic reports stop when the proxy closes', async t => { + const f = await fixture(t, { statsIntervalMs: 10 }); + await until(() => f.logs.records().some(r => r.event === 'proxy_stats')); + // The fixture closes again during teardown; close() must be idempotent. + await f.close(); + const before = f.logs.records().filter(r => r.event === 'proxy_stats').length; + await delay(40); + assert.equal(f.logs.records().filter(r => r.event === 'proxy_stats').length, before); +}); diff --git a/tests/utilities.test.cjs b/tests/utilities.test.cjs index facc23e..97e8710 100644 --- a/tests/utilities.test.cjs +++ b/tests/utilities.test.cjs @@ -58,12 +58,13 @@ test('MySQL backend errors reject with a sanitized code instead of invalid-key f const logs = capture(); const plugin = new MySQLAuthenticationPlugin(); plugin.logger = logs.logger; - plugin.pool = { query: (_sql, args, callback) => { + const connection = { release: () => {}, query: (_sql, args, callback) => { assert.equal(args[1], token); callback(Object.assign(new Error(`query contained ${token}`), { code: 'ER_PARSE_ERROR', sql: `SELECT secret '${token}'`, sqlMessage: token, })); } }; + plugin.pool = { getConnection: callback => callback(null, connection) }; await assert.rejects(plugin.validateKey(token), error => { assert.equal(error.code, 'ER_PARSE_ERROR'); assert.equal(error.message, 'Authentication backend unavailable'); @@ -72,8 +73,8 @@ test('MySQL backend errors reject with a sanitized code instead of invalid-key f return true; }); assert.equal(logs.text().includes(token), false); - plugin.pool = { query: (_sql, _args, callback) => callback(null, []) }; + connection.query = (_sql, _args, callback) => callback(null, []); assert.equal(await plugin.validateKey(token), false); - plugin.pool = { query: (_sql, _args, callback) => callback(null, [{}]) }; + connection.query = (_sql, _args, callback) => callback(null, [{}]); assert.equal(await plugin.validateKey(token), true); });