Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/workflows/build-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,22 @@ jobs:
strategy:
matrix:
node: ["22.18.0", "24.19.0"]
services:
mysql:
image: mariadb:11.8.6
env:
MARIADB_ROOT_PASSWORD: isolated-root-password
MARIADB_DATABASE: otf_test
MARIADB_USER: otf_test
MARIADB_PASSWORD: isolated-test-password
ports:
- 3306:3306
options: >-
--health-cmd="healthcheck.sh --connect --innodb_initialized"
--health-interval=5s --health-timeout=5s --health-retries=20
env:
OTF_TEST_MYSQL_ISOLATED: "1"
OTF_TEST_MYSQL_URL: mysql://otf_test:isolated-test-password@127.0.0.1:3306/otf_test
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
Expand Down
49 changes: 47 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,19 +30,39 @@ Additionally, there are some environment variables that can optionally be specif
| `HTTP_PORT` | The port that the HTTP server should listen on. | `3000` |
| `WEBSOCKET_PORT` | The port that the websocket server should listen on. | `8080` |
| `REQUEST_TIMEOUT_MS` | Maximum wait for a complete controller response, in milliseconds. Must be an integer from 1 to 300000; invalid values fall back to the default with a warning. | `60000` |
| `WEBSOCKET_MAX_PAYLOAD_BYTES` | Maximum incoming WebSocket message size, including all fragments and the response envelope, before or after authentication. Integer from 1 to 16777216; invalid values prevent startup. | `4194304` (4 MiB) |
| `STATS_INTERVAL_MS` | Interval for aggregate `proxy_stats` log records. Integer from 1 to 300000; invalid values prevent startup. | `10000` |
| `AUTH_CONCURRENCY` | Maximum running authentication operations. Integer from 1 to 100; invalid values prevent startup. | `10` |
| `AUTH_QUEUE_LIMIT` | Maximum authentication jobs waiting to start. Integer from 1 to 10000; invalid values prevent startup. | `500` |

## Connection and request behavior

Controllers are authenticated before registration and protocol handling. A new
connection using an already connected device key is rejected; it does not replace
the existing owner. Authentication has a ten-second deadline; timeout rejects with
close code `1011`, and late results cannot register the connection. Disconnecting
during authentication releases its deadline and wait state. MySQL queries also
have a ten-second driver inactivity limit; a lookup that completes after the
during authentication releases its deadline and wait state. MySQL queries have a
driver inactivity limit capped at ten seconds and the remaining authentication
time after connection acquisition; a lookup that completes after the
socket's deadline is ignored. The existing 10-second ping interval and missed-pong policy
are preserved. Policy rejections use WebSocket close code `1008`; authentication
backend failures use `1011`. Rejected connections receive an `ERR:` message and a
close frame, with forced termination after five seconds if closing stalls.
Oversized incoming messages close with code `1009`. This bounds each message;
it is not a limit on total memory or connection count.

Authentication uses a bounded FIFO queue and at most one waiting/running lookup
per key. An existing open owner's duplicate is rejected before any lookup; the
owner is still rechecked after validation. Expired or disconnected waiters do not
enter the backend. A running lookup retains its concurrency slot and same-key
reservation until the backend actually finishes, even if its caller is gone.
Queue-full or same-key contention receives close code `1011` with reason
`authentication busy`. Capacity failures never use HTTP `401`/`403`. This does not
require firmware backoff support and does not change heartbeat or ownership policy.
The defaults are starting limits to validate against the deployment's database
latency and memory budget. This queue bounds authentication work, not all sockets:
there is no pre-upgrade connection cap, and rejected sockets may linger during the
five-second close fallback. Reverse-proxy/OS connection limits remain necessary.

Pending requests belong to their individual controller session. A disconnected
controller fails its pending requests with HTTP `408`, and requests for an absent
Expand Down Expand Up @@ -73,6 +93,16 @@ printing their values. Malformed
payloads and raw database errors are not logged. Rejection and other repeated
diagnostic events have bounded output and aggregation state.

Aggregate `proxy_stats` records include cumulative counters (including suppressed
rejections), counter deltas since the previous record, current/high-water gauges,
fixed cumulative latency buckets, process memory/CPU and event-loop measurements.
Running authentication remains counted until the backend actually finishes, even
after the socket closes. Memory `arrayBuffers` overlaps `external`; do not add them
together. Admission, request and protocol warnings have separate bounded log
budgets. Aggregate metrics contain no per-token labels and have no public HTTP
endpoint. These records describe this Node process; correlate them with separate
reverse-proxy/container/host measurements to investigate incidents.

Both Docker stages use Node `22.18.0` on Alpine, pinned to the immutable image
index digest in `Dockerfile`. Dependency installation uses `npm ci`. Updating the
base image requires an intentional digest change and runtime validation; it no
Expand Down Expand Up @@ -109,6 +139,21 @@ The plugin can be configured through the following environment variables:
| `MYSQL_CONNECTION_URL` | The URL of the MySQL server to connect to. | `mysql://username:password@localhost:3306/otf` |
| `MYSQL_TABLE` | The table in the database that stores valid device keys. | `registered_devices` |

The plugin uses an existence lookup with `LIMIT 1`; index `device_key` for the
deployment's lookup workload. The pool's connection limit follows
`AUTH_CONCURRENCY`, waiting inside the driver is disabled, and driver debug output
is disabled to prevent SQL/key leakage. These limits override URL pool/debug
options. Disconnected/expired callers are checked again after acquisition, before
SQL begins. An in-progress query is not cancelled by destroying its connection;
the driver may itself discard a connection on a fatal inactivity timeout.

CI also runs real-driver integration tests against a disposable MariaDB. For
local integration tests, explicitly set `OTF_TEST_MYSQL_ISOLATED=1` and
`OTF_TEST_MYSQL_URL` to a disposable database named `otf_test`, then run `npm test`.
The tests create/drop their own tables and acquire table locks. Never point this
test URL at a shared or production database. Without these explicit test settings,
the integration cases are skipped and the normal suite uses fake backends.


#### EnvironmentVariableAuthenticationPlugin

Expand Down
14 changes: 12 additions & 2 deletions src/AuthenticationPlugin.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
import { Logger } from "pino";

export interface AuthenticationContext {
// Aborting means the caller has gone away, not that SQL was cancelled.
signal: AbortSignal;
remainingMs(): number;
}

export interface AuthenticationInitOptions { concurrency: number; }

export abstract class AuthenticationPlugin {
public constructor() {}

Expand All @@ -8,11 +16,13 @@ export abstract class AuthenticationPlugin {
* any requests until the promise it returns has been resolved. The default implementation returns a Promise that
* will be resolved immediately.
*/
public async init(logger: Logger): Promise<void> {}
public async init(logger: Logger, options?: AuthenticationInitOptions): Promise<void> {}

public async close(): Promise<void> {}

/**
* Returns a Promise that resolves with a boolean indicating if the specified device key is valid. If the Promise is
* resolved with `false` or is rejected, access will be denied.
*/
public abstract validateKey(deviceKey: string): Promise<boolean>;
public abstract validateKey(deviceKey: string, context?: AuthenticationContext): Promise<boolean>;
}
99 changes: 62 additions & 37 deletions src/authenticationPlugins/MySQLAuthenticationPlugin.ts
Original file line number Diff line number Diff line change
@@ -1,55 +1,80 @@
import { Logger } from "pino";
import { AuthenticationPlugin } from "../AuthenticationPlugin";
import { AuthenticationContext, AuthenticationInitOptions, AuthenticationPlugin } from "../AuthenticationPlugin";
import * as mysql from "mysql";
import { backendErrorCode } from "../diagnostics";
import { ConfigurationError } from "../configuration";
import { boundedInteger, ConfigurationError } from "../configuration";

/**
* Validates device keys by checking if they exist in a MySQL database.
*/
export default class MySQLAuthenticationPlugin extends AuthenticationPlugin {
private pool: mysql.Pool;
private logger: Logger;
private table: string;
private stopped?: Promise<void>;

public async init(logger: Logger): Promise<void> {
this.logger = logger;
return new Promise((resolve, reject) => {
// Make sure that necessary variables have been defined.
for (const key of ["MYSQL_TABLE", "MYSQL_CONNECTION_URL"] as const) {
if (!process.env[key]) {
reject(new ConfigurationError(key, "missing"));
return;
}
}
public async init(_logger: Logger, options?: AuthenticationInitOptions): Promise<void> {
for (const key of ["MYSQL_TABLE", "MYSQL_CONNECTION_URL"] as const) {
if (!process.env[key]) throw new ConfigurationError(key, "missing");
}
const concurrency = boundedInteger("AUTH_CONCURRENCY",
options?.concurrency ?? process.env.AUTH_CONCURRENCY, 10, 100);
this.table = process.env.MYSQL_TABLE;
try {
// Let the driver parse its supported URL options. Before any use,
// override admission/time limits so URL options cannot bypass them.
this.pool = mysql.createPool(process.env.MYSQL_CONNECTION_URL);
this.pool.config.connectionLimit = concurrency;
this.pool.config.waitForConnections = false;
this.pool.config.queueLimit = 1;
this.pool.config.acquireTimeout = 10_000;
this.pool.config.connectionConfig.connectTimeout = 10_000;
// Driver debug output prints SQL and credentials outside our logger.
this.pool.config.connectionConfig.debug = false;
} catch {
throw new ConfigurationError("MYSQL_CONNECTION_URL", "invalid");
}
}

try {
this.pool = mysql.createPool(process.env.MYSQL_CONNECTION_URL);
} catch {
reject(new ConfigurationError("MYSQL_CONNECTION_URL", "invalid"));
return;
}
resolve();
});
private failure(error: unknown): Error {
// Never retain the driver's SQL-bearing error object as a cause.
return Object.assign(new Error("Authentication backend unavailable"), { code: backendErrorCode(error) });
}

public validateKey(deviceKey: string): Promise<boolean> {
public validateKey(deviceKey: string, context?: AuthenticationContext): Promise<boolean> {
return new Promise<boolean>((resolve, reject) => {
this.pool.query(
{ sql: "SELECT * FROM ?? WHERE device_key = ?", timeout: 10_000 },
[process.env.MYSQL_TABLE, deviceKey],
(err, results, fields) => {
if (err) {
// mysql errors include the formatted SQL and key. The
// forwarder rate-limits logging of this sanitized failure.
const failure = new Error("Authentication backend unavailable");
Object.assign(failure, { code: backendErrorCode(err) });
reject(failure);
return;
}
const stale = () => context && (context.signal.aborted || context.remainingMs() < 1);
const expired = () => this.failure({ code: "AUTH_TIMEOUT" });
if (stale()) { reject(expired()); return; }
try {
this.pool.getConnection((error, connection) => {
if (error) { reject(this.failure(error)); return; }
let released = false;
const release = () => {
if (released) return;
released = true;
connection.release();
};
if (stale()) { release(); reject(expired()); return; }
const timeout = Math.min(10_000, Math.floor(context?.remainingMs() ?? 10_000));
if (timeout < 1) { release(); reject(expired()); return; }
try {
connection.query({ sql: "SELECT 1 FROM ?? WHERE device_key = ? LIMIT 1", timeout },
[this.table, deviceKey], (err, results) => {
release();
if (err) reject(this.failure(err));
else resolve(results.length !== 0);
});
} catch (err) { release(); reject(this.failure(err)); }
});
} catch (error) { reject(this.failure(error)); }
});
}

resolve(results.length !== 0);
}
);
public close(): Promise<void> {
if (!this.stopped) this.stopped = new Promise((resolve, reject) => {
if (!this.pool) { resolve(); return; }
this.pool.end(error => error ? reject(this.failure(error)) : resolve());
});
return this.stopped;
}
}
125 changes: 125 additions & 0 deletions src/authenticationQueue.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
import { performance } from "perf_hooks";
import { AuthenticationPlugin } from "./AuthenticationPlugin";
import { Telemetry } from "./telemetry";

type Result = { status: "valid" | "invalid" | "timed_out" | "cancelled" | "busy" }
| { status: "error"; error: unknown };

interface Job {
key: string;
startedAt: number;
deadline: number;
active: boolean;
settled: boolean;
controller: AbortController;
timer: ReturnType<typeof setTimeout>;
detach(): void;
resolve?: (result: Result) => void;
}

/** Bounded, FIFO admission. Client lifetime and backend lifetime are separate. */
export class AuthenticationQueue {
private jobs = new Map<string, Job>();
private waiting = new Map<string, Job>();
private active = 0;
private stopping = false;

constructor(private readonly plugin: AuthenticationPlugin, private readonly telemetry: Telemetry,
private readonly concurrency: number, private readonly queueLimit: number,
private readonly timeoutMs: number) {}

submit(key: string, signal: AbortSignal): Promise<Result> {
if (signal.aborted || this.stopping) return Promise.resolve({ status: "cancelled" });
if (this.jobs.has(key) || (this.active >= this.concurrency && this.waiting.size >= this.queueLimit)) {
this.telemetry.count("authentication_busy");
return Promise.resolve({ status: "busy" });
}
return new Promise(resolve => {
const startedAt = performance.now();
const job: Job = { key, startedAt, deadline: startedAt + this.timeoutMs,
controller: new AbortController(), active: false, settled: false,
timer: undefined, detach: () => {}, resolve };
const abandon = () => this.settle(job, { status: "cancelled" });
signal.addEventListener("abort", abandon, { once: true });
job.detach = () => signal.removeEventListener("abort", abandon);
job.timer = setTimeout(() => this.settle(job, { status: "timed_out" }), this.timeoutMs);
job.timer.unref();
this.jobs.set(key, job);
this.waiting.set(key, job);
this.telemetry.adjust("authQueued", 1);
this.pump();
});
}

private settle(job: Job, result: Result): void {
if (job.settled) return;
job.settled = true;
clearTimeout(job.timer);
job.detach();
if (result.status === "cancelled" || result.status === "timed_out") {
this.telemetry.count(result.status === "cancelled"
? "authentication_cancelled" : "authentication_timed_out");
job.controller.abort();
}
if (!job.active) {
if (this.waiting.delete(job.key)) this.telemetry.adjust("authQueued", -1);
if (this.jobs.get(job.key) === job) this.jobs.delete(job.key);
}
this.telemetry.observe("authentication", performance.now() - job.startedAt);
const resolve = job.resolve;
job.resolve = undefined;
resolve?.(result);
}

private pump(): void {
while (!this.stopping && this.active < this.concurrency && this.waiting.size) {
const job = this.waiting.values().next().value as Job;
if (performance.now() >= job.deadline) {
this.settle(job, { status: "timed_out" });
continue;
}
this.waiting.delete(job.key);
this.telemetry.adjust("authQueued", -1);
this.telemetry.observe("queueWait", performance.now() - job.startedAt);
job.active = true;
this.active++;
this.telemetry.adjust("authActive", 1);
// Defer invocation, then recheck: a close/deadline may have run in between.
void Promise.resolve().then(async () => {
if (job.controller.signal.aborted || performance.now() >= job.deadline) {
this.settle(job, { status: job.controller.signal.aborted ? "cancelled" : "timed_out" });
return;
}
const lookupStarted = performance.now();
this.telemetry.count("authentication_started");
try {
const valid = await this.plugin.validateKey(job.key, {
signal: job.controller.signal,
remainingMs: () => Math.max(0, job.deadline - performance.now()),
});
this.telemetry.count(valid ? "authentication_valid" : "authentication_invalid");
this.settle(job, performance.now() >= job.deadline
? { status: "timed_out" } : { status: valid ? "valid" : "invalid" });
} catch (error) {
this.telemetry.count("authentication_failed");
this.settle(job, performance.now() >= job.deadline
? { status: "timed_out" } : { status: "error", error });
} finally {
this.telemetry.observe("lookup", performance.now() - lookupStarted);
}
}).finally(() => {
// Do not release a slot or same-key guard just because a socket left.
job.active = false;
this.active--;
this.telemetry.adjust("authActive", -1);
if (this.jobs.get(job.key) === job) this.jobs.delete(job.key);
this.pump();
});
}
}

close(): void {
this.stopping = true;
for (const job of this.jobs.values()) this.settle(job, { status: "cancelled" });
}
}
Loading
Loading