Skip to content

feat(plugins): add external-to-internal data replication plugin (#72) - #360

Open
andreikharytonenka-lab wants to merge 1 commit into
outerbase:mainfrom
andreikharytonenka-lab:feat/replication-plugin-72
Open

andreikharytonenka-lab wants to merge 1 commit into
outerbase:mainfrom
andreikharytonenka-lab:feat/replication-plugin-72

Conversation

@andreikharytonenka-lab

Copy link
Copy Markdown

/claim #72
Closes #72

Summary

Implements a pull-based External-to-Internal Data Replication Plugin (DataSyncPlugin & DatabaseSyncSource adapters) that replicates schema and data from an external database (PostgreSQL / Supabase, MySQL, Turso, Cloudflare D1, StarbaseDB, Cloudflare Hyperdrive, or Outerbase API) into StarbaseDB's internal Cloudflare Durable Object SQLite database so StarbaseDB can act as an ultra-low-latency edge replica.


Architecture & Key Design Decisions

1. Decoupled DataSyncPlugin & DatabaseSyncSource Abstraction

Following @Brayden's architecture guidance, DataSyncPlugin is decoupled from dialect-specific introspection and accepts a DatabaseSyncSource implementation (PostgresSyncSource, MySQLSyncSource, SQLiteSyncSource), while also supporting auto-selection from dataSource.external.dialect:

import { DataSyncPlugin, PostgresSyncSource } from '@outerbase/starbasedb/plugins'

const syncStore = new PostgresSyncSource({
    dialect: 'postgresql',
})

const plugins = [
    new DataSyncPlugin(syncStore, {
        sync_interval: 60,
        tables: ['users', 'person'],
    }),
] satisfies StarbasePlugin[]

2. True External vs. Internal Query Routing

  • External Queries (DatabaseSyncSource.executeExternalQuery): Executes information_schema / pragma_table_info schema discovery, connection checks, and incremental batch pulls against the external data source via StarbaseDB's external query pipeline (executeExternalQuery / @outerbase/sdk / Outerbase API / Cloudflare Hyperdrive postgres client).
  • Internal Queries (dataSource.rpc.executeQuery): Creates replicated SQLite tables, upserts sanitized rows (INSERT OR REPLACE INTO), and updates sync watermarks in the internal Durable Object SQLite database.
  • Result Normalization & SQLite Value Sanitization: Seamlessly normalizes both @outerbase/sdk object-array responses (Record<string, any>[]) and tabular responses ({ rows: any[][], columns: string[] }), and sanitizes values (Date -> ISO-8601 string, boolean -> 1 | 0, JSON/Array -> serialized JSON string, bigint -> safe integer/string, undefined -> null) prior to writing to Durable Object SqlStorage.

3. Replicated Table Naming & Outerbase Studio Compatibility

  • Internal Plugin Metadata Table: Uses the tmp_ prefix exclusively for internal plugin metadata (tmp_data_sync_metadata) so users can distinguish Starbase system metadata from their own tables.
  • User Replicated Tables:
    • Default / public schema tables (e.g. 'users', 'person', 'public.products') are created in SQLite root as users, person, products — without a tmp_ prefix and without literal dots in SQLite table names (which break Outerbase Studio's table viewer).
    • Non-default schema tables (e.g. 'analytics.events') are created in SQLite as analytics_events (or an explicit target_table override).
  • Smart beforeQuery Hook:
    • Automatically omits public. prefixes (SELECT * FROM public.users / SELECT * FROM "public"."users" -> SELECT * FROM users) and maps custom schema prefixes (SELECT * FROM analytics.events -> SELECT * FROM analytics_events) when querying the internal SQLite data source.
    • Preserves single-quoted string literals and never touches SQLite catalog/introspection queries (sqlite_master, pragma_table_info, PRAGMA).

4. Flexible Incremental Cursors & Tables Without created_at or id

  • Supports custom per-table tracking columns (cursor_column, timestamp_column, id_column, primary_key, batch_size).
  • Auto-Adaptation for Tables Without created_at or id: When a table is configured by string name (such as tables: ['users', 'person'] where person only has first_name and last_name), validateTableStructure and syncTable inspect the discovered columns and gracefully fall back to full-table batch replication (SELECT * FROM public.person LIMIT 1000) instead of failing validation.

5. Request Lifecycle & Monitoring Endpoints

  • Lazily initializes tmp_data_sync_metadata and triggers initial/interval-based sync inside the Hono request middleware once c.get('dataSource') is bound (using executionContext.waitUntil for non-blocking background interval syncs).
  • Exposes monitoring and control routes:
    • GET /sync-status & GET /sync/status — Current sync configuration and per-table watermarks/errors
    • POST /sync & POST /sync/trigger — Trigger on-demand sync for all tables or a specific { "table": "users" }
    • GET /sync-data & GET /sync/data — Preview replicated rows and metadata

Verification & Testing

  • Added 46 unit and integration tests across plugins/postgres-sync/index.test.ts (19 tests) and plugins/data-sync/index.test.ts (27 tests):
    • Connection validation, schema discovery, data type mapping, default normalization, structure validation, and incremental queries for PostgresSyncSource, MySQLSyncSource, and SQLiteSyncSource
    • End-to-end replication for tables with standard created_at/id, custom cursor_column, and tables without created_at or id (person)
    • beforeQuery SQL schema rewriting and string literal preservation
    • HTTP monitoring and manual sync trigger endpoints
  • Verified formatting with Prettier (npx prettier --check plugins/data-sync plugins/postgres-sync dist/plugins.ts).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replicate data from external source to internal source with a Plugin

1 participant