diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index e917ba2b..ce742c22 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -274,6 +274,17 @@ only. Key hooks: `dropForeignKeyStatement`, `dropIndexStatement`, `dropTriggerSt `dropFunctionStatement`, `dropProcedureStatement`. Full hook map + fallback behavior + per-dialect gotchas live in `packages/sql/src/providers/DIALECTS.md` (tracked). +Safe column change is a sibling registry, not a `SqlDialect` hook: +`planColumnChange` (`modules/utilities/column-change.ts`) finds impact and +blockers from the schema Fox already loaded; each family writes steps in +`providers//*.column-change.ts` (`column-change.registry.ts`). Engines not +listed answer "not yet" rather than SQL that would fail there. The Utils panel +computes the plan in the browser (`@foxschema/ui-shared`); Fox runs none of it. +A rename alone is in-place; a type change adds, copies in batches, switches +over, and drops last. PostgreSQL / YugabyteDB / CockroachDB, MySQL / MariaDB / +TiDB, and SQL Server / Azure SQL are in v1. SQLite needs a table rebuild; +Redshift's `ALTER TABLE` cannot express the steps. + Version-aware DDL: `SchemaProvider.detectVersion?()` → stored in Zustand as `targetServerVersion` → flows into `SchemaMapping` → dialect drop hooks use it. Oracle pre-23c uses PL/SQL exception blocks; DB2 (all versions) uses SQL PL `CONTINUE HANDLER FOR SQLSTATE '42704'`. @@ -395,6 +406,7 @@ Constraints: 2. Register in `provider-settings.ts`, `adapter-registry.ts`, `provider-registry.ts`, `modules/dialect/registry.ts` (and `modules/access/user-sql.registry.ts` / `modules/access/access-sql.registry.ts` when the engine has account or GRANT SQL). Backup/restore commands: `.backup.ts` in `modules/utilities/backup.registry.ts`. A `history` query, where the engine records its backups, lets the panel list them; a server-side SQL backup can be run from the panel, a restore never is. + Safe column change: `.column-change.ts` in `modules/utilities/column-change.registry.ts`. Omit the engine to answer "not yet"; do not guess SQL. SQLite is a table rebuild (later); Redshift is not in v1. 3. Add the dialect name to the `Dialect` union **and** the `DIALECTS` array in `packages/sql/src/providers/provider-settings.ts` (`dialect-registry.test.ts` fails until they match `PROVIDER_SETTINGS`). Nothing in `apps/web` needs diff --git a/docs/CODE_MAP.md b/docs/CODE_MAP.md index 3986f786..9b0b8a3e 100644 --- a/docs/CODE_MAP.md +++ b/docs/CODE_MAP.md @@ -17,11 +17,18 @@ packages/db Database drivers and the connection runtime: connect factory, pooling, circuit breaker. Depends on sql. Redis keys-as-tables live in `providers/redis/redis.keyspace.ts` (prefix tables vs collection tables; bounded scans). + MongoDB (`providers/mongodb/mongodb.adapter.ts`) shows + `_id` as text, matches an ObjectId hex a SELECT showed, + and projects only named columns (Mongo would otherwise + add `_id` to every find). packages/shared Contracts the frontend, server and CLI must agree on: permission names, error codes, wire message shapes, - the COMMUNITY_NAV registry (`nav.ts`), and the - shared data-migrate cap (`DATA_MIGRATE_MAX_OPS`). + the COMMUNITY_NAV registry (`nav.ts`), the + shared data-migrate cap (`DATA_MIGRATE_MAX_OPS`), + GET `/api/system/services` (`service-status.ts`), + and the connection-name rule (`connection-name.ts`: + 120 characters, no control characters). Browser-safe. packages/workflow-contract Browser-safe control-plane types for the workflow engine @@ -77,6 +84,8 @@ flags (`workflow`, `enterprise.channels`). interfaces/ The shared vocabulary: TableSchema, TableDiff, MigrationStep. providers/ One folder per dialect — settings, SqlDialect, and Access SQL (`*.user-sql.ts`, `*.access-sql.ts`). Backup/restore commands are `*.backup.ts` (registered in `modules/utilities/backup.registry.ts`). + Safe column-change steps are `*.column-change.ts` (registered in + `modules/utilities/column-change.registry.ts`). `dialectFamily()` in `provider-settings.ts` maps wire-compatible relatives onto mysql / postgres / sqlserver. cores/ Connection strings, and shaping catalog rows into TableSchema. @@ -93,7 +102,7 @@ modules/ One folder per domain, named to match the frontend feature | `lokee-weave` | Content-addressed schema versioning and revert | | `sql-editor` | FoxScript parsing, code cells, SELECT aliasing, the fail-closed SQL subset Redis and MongoDB run (`sql-subset.ts`) | | `access` | Permission intent, effective access, GRANT/REVOKE and account DDL (facades; emitters live in `providers/`) | -| `utilities` | DBA queries: server insights, index fragmentation (`isSafeIndexFragmentationCustomSql`: one read-only SELECT; Db2 defrag is `providers/db2/db2.index-fragmentation.ts`), backup/restore command builders | +| `utilities` | DBA queries: server insights, index fragmentation (`isSafeIndexFragmentationCustomSql`: one read-only SELECT; Db2 defrag is `providers/db2/db2.index-fragmentation.ts`), backup/restore command builders, SQL Tracer, safe column change (`planColumnChange` in `column-change.ts`; per-family SQL in `providers//*.column-change.ts`, registered in `column-change.registry.ts` — engines not listed answer "not yet") | `dialect` and `sql-text` are the foundations: the other folders build on them, never the reverse. @@ -151,12 +160,12 @@ database/ The metadata store and its migrations. | Folder | What it covers | |---|---| | `access` | Database permission inspection and DBA utilities. Index-fragmentation custom SQL is gated read-only in `index-fragmentation.service.ts` (same tokenizer as `/sql/execute`). | -| `system` | Health and version, updates, app info, the metadata-database probe, activity | +| `system` | Health and version, updates, app info, the metadata-database probe, activity, Home Services (`GET /api/system/services` → `readServiceStatus`: API always `running`; engine `running` / `not_configured` / `unreachable` from the workflow `/health` probe). Status only — start/stop is later. | | `admin` | Users and roles, secrets, cloud credentials, policy | | `backup` | Per-user backup defaults (`GET`/`PUT /api/backup-settings`). Commands themselves are built in `@foxschema/sql` (`modules/utilities/backup.ts` + `providers//*.backup.ts`); the panel runs only a server-side SQL backup, after confirmation, and lists the backups the server recorded (`backupHistoryQuery`); restores are never run. | | `auth` | Sign-in, setup, password and SSO routes (the services are `platform/identity`) | | `compare` | Schema comparison | -| `connections` | Routes for saved database connections, driver check/install and connection tests; the store is `platform/connections` (`authMethod` / `domain` on encrypted `ConnectionOptions`; `none` for Redis/MongoDB with no auth; NTLM is adapter-side) | +| `connections` | Routes for saved database connections, driver check/install and connection tests; the store is `platform/connections` (`authMethod` / `domain` on encrypted `ConnectionOptions`; `none` for Redis/MongoDB with no auth; NTLM is adapter-side). Create and update share `checkConnectionName` (`packages/shared`): free text, 120 characters, no control characters — a narrower form-only rule used to rewrite saved names on edit. | | `data-migrate` | Moving data between databases. The route and the editor share `DATA_MIGRATE_MAX_OPS` (2,000) from `@foxschema/shared`. | | `files` | File uploads, querying an uploaded file, the import machinery (parsers, column detection, the parse worker pool) and the database-file picker | | `git` | Git repositories migrations are committed to: bare local copies through isomorphic-git, branches, fetch / pull / push, and the guarded HTTP client that refuses private addresses. Plan: [plans/2026-10-02-migrations-in-git.md](plans/2026-10-02-migrations-in-git.md) | @@ -164,6 +173,7 @@ database/ The metadata store and its migrations. | `migration` | Applying DDL migrations, and their run history | | `schema` | Reading a schema | | `sql-editor` | SQL editor, code cells, sandboxed execution. `sql-execute.service.ts` (`runStatements`) applies the saved schema (`adapter.setCurrentSchema`) then optionally `BEGIN`s one transaction for the batch; paging wrap is `sql-page-wrap.service.ts` (`wrapSqlForOrderedPage` for a compare window: outer ORDER BY, `limit + 1`). | +| `sql-trace` | SQL Tracer sessions: start / events / stop. Live reads the engine's session list; Fox activity is statements this install ran on a saved connection (`platform/sql-activity`). Redacted; 15 min / 10,000 statements. | | `users` | Profile, preferences, first-run wizard | | `workspaces` | Shared workspaces: create, switch, members, invites, public or private (`/api/workspaces`). Which workspace a request acts in is `platform/workspaces`. A personal workspace's invites and memberships are capped at the account's role (`workspace-role-ceiling.test.ts`). Admins list every workspace (`GET /all`) and open one only by joining. | | `workflow` | Workflow engine settings and health; the engine proxy (an allowlist of engine routes, each behind a `workflow.*` permission); saved-connection grants; and the token-guarded internal routes the engine calls to resolve a granted connection and read its settings. Save, run, and steering (cancel / resume / answer a paused run) refuse a connection the caller cannot use in this workspace (`WorkflowConnectionGrants.notUsableBy`). Runbook: [workflow.md](workflow.md). | @@ -188,7 +198,9 @@ A feature uses the layers it needs; it does not create empty ones. ## Frontend — `apps/web/src/frontend` ``` -app/ The application shell and settings screens. +app/ The application shell and settings screens. Home's Services card + is `app/shell/ServiceStatusCard.tsx` (`GET /api/system/services` + via `shared/api/servicesApi.ts`); it does not start or stop anything. features/ The view registry: `featureRegistry.ts` (each view's rail button, who may see it, where someone goes who may not, what it adds to the top toolbar, @@ -255,8 +267,8 @@ the page-epoch guard, bookmarks and recents, SQL variables. | `lokee-weave` | Schema history graph and version compare (Snapshots workspace) | | `migrations` | Migration run history (Applies on the rail) | | `schema-diff` | Diff rendering shared by compare and history | -| `sql-editor` | SQL editor, its store (`oneTransaction`, `selectOnlyConnection` so the schema browser's connection is the one Run hits), results grid, data peek, the shared Monaco setup (`monaco.ts`) (Index/Clone/Query files live in `utilities`) | -| `utilities` | Own workspace: clone table, index management, backup & restore commands, server insights, query files, DB users & grants | +| `sql-editor` | SQL editor, its store (`oneTransaction`, `selectOnlyConnection` so the schema browser's connection is the one Run hits; opening a sample bookmark keeps checked Destinations), results grid, data peek, the shared Monaco setup (`monaco.ts`) (Index/Clone/Query files live in `utilities`). The run rail's **In [n]** selects only that statement. | +| `utilities` | Own workspace: clone table, Change Column (`ColumnChangePanel` → `planColumnChange` in the browser; nothing runs), index management, backup & restore commands, server insights, SQL Tracer, query files, DB users & grants | | `workflow` | Workflow designer (canvas, inspector, triggers, SQL and script editors), runs (`RunsPanel` cancel / answer a paused human gate), variables, credentials and engine settings — through the engine proxy, plus linking saved connections to workflows | | `workspaces` | The rail's workspace button (`WorkspaceRailButton`) and the menu it opens: switch, invite, join public, members, settings. Admin → Workspaces lists every workspace; opening one is joining. Editor localStorage is keyed by workspace (`shared/lib/editorWorkspaceStorage.ts`). | @@ -276,6 +288,8 @@ the page-epoch guard, bookmarks and recents, SQL variables. | Calling an API endpoint | use `api` from `@/shared/api/client` — never `fetch` directly | | SQL for permissions or accounts | `packages/sql/src/modules/access/` (facade) + `packages/sql/src/providers//*.user-sql.ts` / `*.access-sql.ts`. Db2 OS-user docker steps: `buildDb2OsUserInstructions`. | | Backup / restore commands for an engine | `packages/sql/src/providers//*.backup.ts` registered in `modules/utilities/backup.registry.ts`. Saved folder/format defaults: `packages/server/src/features/backup/`. | +| Safe column-change SQL for an engine | `packages/sql/src/providers//*.column-change.ts` registered in `modules/utilities/column-change.registry.ts`. The facade (`planColumnChange`) finds impact and blockers; omit the engine to answer "not yet" rather than guessed SQL. The panel is `apps/web/.../ColumnChangePanel.tsx`; no server route. | +| Home Services / engine reachability | `packages/server/src/features/system/service-status.ts` (`GET /api/system/services`). Types in `packages/shared/src/service-status.ts`. The card is `apps/web/.../app/shell/ServiceStatusCard.tsx`. | | A dialect capability the app must branch on | `packages/sql/src/modules/capabilities/` (`dialect-features.ts`) and `packages/sql/src/modules/dialect/` | | Guard or cross-cutting HTTP concern | `packages/server/src/platform/` | | How a dialect lexes SQL (permission gates, splitter) | `packages/sql/src/modules/sql-text/sql-readings.ts` + `sql-tokens.ts`. Do not add a second scanner. | diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 79819524..32975cef 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -167,7 +167,8 @@ docker compose -f docker-compose.app.yml --profile workflow up -d --build `WORKFLOW_ENGINE_ENDPOINT` is the unsaved default only. An endpoint saved in Workflow settings still wins. Leave the profile off and the app runs as Fox -Schema alone. +Schema alone. Home → **Services** (`GET /api/system/services`) shows whether +this API can reach the engine; it does not start it. | Variable | Set on | Default | Purpose | |----------|--------|---------|---------| diff --git a/docs/FEATURE_MAP.md b/docs/FEATURE_MAP.md index 7ec23175..e035a935 100644 --- a/docs/FEATURE_MAP.md +++ b/docs/FEATURE_MAP.md @@ -12,10 +12,10 @@ an interactive page (open it in a browser), and [the test and fix plan](plans/2026-10-25-feature-and-dialect-test-plan.md) sets the order every feature and dialect is tested in. -Measured on `main` at `b12bcd5607` (2026-10-31): 35 features, 18 server -feature modules, 158 HTTP routes (pinned by +Measured on `main` at `66575674` (2026-11-02): 35 features, 18 server +feature modules, 159 HTTP routes (pinned by `packages/server/src/api/http-contract.test.ts`), 14 frontend features, -16 database engines, 28 metadata tables (31 migrations), 57 e2e specs in 35 +16 database engines, 28 metadata tables (31 migrations), 58 e2e specs in 35 groups, ~550 unit-test files. Re-measure with the commands in [Keeping this page true](#keeping-this-page-true). @@ -38,7 +38,7 @@ flowchart LR end subgraph server["Fox server — packages/server (Fastify)"] PLAT["platform
identity · RBAC · workspaces
connections · guards"] - FEAT["18 feature modules
158 routes"] + FEAT["18 feature modules
159 routes"] end WFS["Workflow server :8081
apps/workflow-server
+ packages/workflow-engine"] META[("Metadata store
SQLite · 28 tables")] @@ -95,7 +95,7 @@ flowchart LR PLATF --> p3[platform.workspaces
Workspaces] PLATF --> p4[platform.connections
Connections & drivers] PLATF --> p5[platform.secrets
App secrets] - PLATF --> p6[platform.system
Shell, health, updates] + PLATF --> p6[platform.system
Shell, health, services] SCH --> s1[schema.compare
Compare & deploy] SCH --> s2[schema.applies
Migration history] @@ -149,7 +149,7 @@ in its folders and its e2e specs (`apps/e2e/src/tests`). | `platform.workspaces` | Workspaces — rail button and menu, create, switch, archive, restore, settings, members and workspace roles (a personal workspace lends no more than its account holds), invites (accept / decline, 14 days), public discover / join, shared workspace scripts | Rail button under the Fox mark | `/api/workspaces` (20) | server `workspaces`, `platform/workspaces`; web `workspaces` | `workspace.members`, `workspace.settings`, `workspace.create` | `workspaces`, `workspace_members`, `workspace_invites`, `workspace_scripts` | server 5, web 3; e2e `workspaces`, `workspaces-sharing-limits` | | `platform.connections` | Connections & drivers — credential manager, connection form (password / Windows / LDAP / No password on Redis and MongoDB, SSL, pool, SQLite / DuckDB file picker), schema list, connection test, driver check and Install driver, cloud provider credentials | Creds (rail), connection modal | `/api/connections` (4), `/api/connection/test`, `/api/db/test`, `/api/driver` (2) | server `connections`, `platform/connections`, `internal/driver-install`; web `connections` | per workspace role; `admin.drivers` to install | `connections`, `cloud_provider_credentials` | server 1, web 1 | | `platform.secrets` | App secrets — vault entries, cloud secret providers, resolving a secret | Creds → secrets, editor Secrets panel | `/api/app-secrets` (9) | server `admin` (app-secrets mount) | `secrets.view/create/edit/delete` | `app_secrets`, `cloud_provider_credentials` | — | -| `platform.system` | Shell, health, updates — activity rail, top toolbar, command palette, home, backend-offline banner, activity, app info, config, update check / apply, email and update settings, send feedback | App shell, Prefs | `/api/health`, `/api/config`, `/api/app-info`, `/api/activity`, `/api/updates` (2) | server `system`; web `app/shell`, `app/settings`, `feedback` | — | `app_settings` | server 1, web 2; e2e `smoke`, `lazy-loading`, `send-feedback`, `auto-error-catch` | +| `platform.system` | Shell, health, updates — activity rail, top toolbar, command palette, home, **Services** card (API / Web UI / Workflow engine status; no start/stop yet), backend-offline banner, activity, app info, config, update check / apply, email and update settings, send feedback | App shell, Prefs, Home | `/api/health`, `/api/config`, `/api/app-info`, `/api/activity`, `/api/updates` (2), `/api/system/services` | server `system` (`service-status.ts`); web `app/shell` (`ServiceStatusCard`), `app/settings`, `feedback` | — | `app_settings` | server 2, web 3; e2e `smoke`, `lazy-loading`, `send-feedback`, `auto-error-catch` | ### Schema @@ -548,11 +548,13 @@ the one table every screen asks. "—" has a reason string the UI shows. | SQLite | ✓ | — | — | ✓ | ✓ | ✓ | `dialects/sqlite` | | DuckDB | ✓ | — | — | ✓ | ✓ | ✓ | `dialects/duckdb` | | ClickHouse | ✓ | — | ✓ | ✓ | — | — | `dialects/clickhouse` | -| Redis | — | — | — | — | ✓ (hash keys; lists, sets, sorted sets, streams and JSON read as tables) | — | — | -| MongoDB | — | — | — | — | ✓ | — | — | +| Redis | — | — | — | — | ✓ (hash keys; lists, sets, sorted sets, streams and JSON read as tables) | — | `redis-editor` | +| MongoDB | — | — | — | — | ✓ | — | `mongodb-sql` | Redis and MongoDB run a fail-closed SQL subset in the editor (single-table -SELECT / INSERT / UPDATE / DELETE with `column = value` predicates). +SELECT / INSERT / UPDATE / DELETE with `column = value` predicates). A MongoDB +SELECT returns only the columns you name; `_id` is text, and a write that uses +that value matches the ObjectId a SELECT showed. ## Tests and known gaps @@ -564,22 +566,20 @@ SELECT / INSERT / UPDATE / DELETE with `column = value` predicates). | Unit — web | `apps/web` | 179 (sql-editor 59, lokee-weave 14, access 12, workflow 12, …) | | Unit — workflow | `packages/workflow-engine`, `apps/workflow-server` | 87 + 19 | | Unit — CLI | `apps/cli` | 14 | -| E2E (Playwright) | `apps/e2e/src/tests` | 56 spec files (`npm run test:e2e:all`) | +| E2E (Playwright) | `apps/e2e/src/tests` | 58 spec files (`npm run test:e2e:all`) | -Known gaps and open work (2026-10-26, after #557 – #561) — each belongs on the planning board: +Known gaps and open work (2026-11-02, after #575 – #589) — each belongs on the planning board: | Feature | Gap | Kind | |---|---|---| | `schema.snapshots` | Revert scope e2e ("reverts only the ticked object") recorded as failing — re-check | bug? | | `editor.code` | `code-cell-execute.service.test.ts` timeout test flakes under load (4 s budget includes worker cold start) | test | | `engines.dialects` | YugabyteDB container stuck on a stale master list (needs a fresh volume) | env | -| `access.inspector` | Oracle `FOX_READER` role not listed: seed account lacks dictionary access | test | -| `editor.run` | Opening a sample bookmark replaces the checked Destinations with none | bug | | `platform.auth` | Manual walk: SSO with real providers, launch session, email verification with real mail (automated part green 2026-10-25) | manual | | `platform.workspaces` | Workspace ceiling caps by rank; a custom role mapping is not intersected with the account's permissions | improvement | -| `platform.connections` | Redis / MongoDB: an explicit "No password" sign-in choice (a requirepass-only Redis saved without its password is not prompted) | improvement | -| `platform.admin` | Read-only as a permission: routes declare `reads`, roles lack `app.write` (the Demo role uses a URL allow-list today) | improvement | -| `editor.datacompare` | Compare 2,000 rows with paging (done, #559/#560); SQL tracer Phase 1 (done, see `utilities.trace`) | feature | +| `platform.connections` | A requirepass-only Redis saved without its password is not prompted (the **No password** login method is shipped) | improvement | +| `platform.system` | Home → Services is status only; start/stop the Workflow engine (`admin.services`) is later | feature | +| `utilities.columns` | SQLite (table rebuild), Oracle, Db2, Redshift, ClickHouse not in v1; primary-key / inbound-FK / identity columns refused; running the steps from the app is v2 | feature | ## Planning work diff --git a/docs/README.md b/docs/README.md index f5ff5f53..634a71c1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -73,8 +73,8 @@ Finished or in-progress design notes. Do not treat them as the map of the code. | Excel and Sheets import | [plans/excel-and-sheets-import.md](plans/excel-and-sheets-import.md) | | Feature review test plan | [plans/2026-10-06-review-features-test-plan.md](plans/2026-10-06-review-features-test-plan.md) | | Feature and dialect test plan, and its first results | [plans/2026-10-25-feature-and-dialect-test-plan.md](plans/2026-10-25-feature-and-dialect-test-plan.md), [plans/2026-10-26-feature-and-dialect-test-results.md](plans/2026-10-26-feature-and-dialect-test-results.md) | -| Service status on Home, with start and stop for the Workflow engine (design) | [plans/2026-11-01-service-status.md](plans/2026-11-01-service-status.md) | -| Safe column change: plan and SQL for review (design) | [plans/2026-11-01-safe-column-change.md](plans/2026-11-01-safe-column-change.md) | +| Service status on Home (status shipped; start/stop later) | [plans/2026-11-01-service-status.md](plans/2026-11-01-service-status.md) | +| Safe column change: plan and SQL for review (planner + Utils screen shipped) | [plans/2026-11-01-safe-column-change.md](plans/2026-11-01-safe-column-change.md) | ## Releases diff --git a/docs/feature-map.html b/docs/feature-map.html index 8bd70dc0..95abb87e 100644 --- a/docs/feature-map.html +++ b/docs/feature-map.html @@ -307,7 +307,7 @@

System

end subgraph server["Fox server — packages/server (Fastify)"] PLAT["platform<br/>identity · RBAC · workspaces<br/>connections · guards"] - FEAT["18 feature modules<br/>158 routes"] + FEAT["18 feature modules<br/>159 routes"] end WFS["Workflow server :8081<br/>apps/workflow-server<br/>+ packages/workflow-engine"] META[("Metadata store<br/>SQLite · 28 tables")] @@ -666,7 +666,7 @@

End-to-end suites