Skip to content

FEE-890 Create a generic upsert route that accepts all root CrossApp types (1/2) - #1621

Open
sid597 wants to merge 10 commits into
eng-2005-zod-validators-for-the-cross-app-typesfrom
fee-890-crossapp-upsert-steps
Open

sid597 wants to merge 10 commits into
eng-2005-zod-validators-for-the-cross-app-typesfrom
fee-890-crossapp-upsert-steps

Conversation

@sid597

@sid597 sid597 commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Reviewer brief

FEE-890 adds one route that takes the five root CrossApp types for a space and writes them with one upsert_concepts call. It ships as two PRs, stacked on #1619 (ENG-2005, the validators):

  1. This PR. packages/database only: the request type and its validator, and the pure steps the route runs.
  2. FEE-890 Create a generic upsert route that accepts all root CrossApp types (2/2) #1622. The route in apps/website, its unit tests and the bearer-token client. It is the only caller of this PR's exports.

Why two and not one: together they are 527 changed lines excluding tests (386 here, 141 in #1622). The split follows the package boundary: shared code first, then the route that uses it.

  • Result: packages/database/src/lib/crossAppUpsert.ts holds the steps, each a pure function with unit tests:
    • parseUpsertData checks the body with the ENG-2005 validators and returns { success: false, issues: [{ path, message }] } on failure.
    • schemaReferences and findMissingSchemas find the schemas the body references but does not contain.
    • toOrderedEntries converts the items and puts each one after the items it depends on.
    • mapUpsertResults maps the row ids upsert_concepts returns back to local ids. -1 means any unique violation, a name or a Content clash, so its reason is unique_conflict; -2 is error.
  • Review focus: Default, can change in a later PR. A schema the body references but does not contain is looked up in the space, and the request is refused only when the schema is in neither. Roam publish sends only the schemas that are not synced yet (publishNodesToGroups.ts:391-409), so refusing every incomplete body would make each publish resend all schemas. Marc-Antoine asked for the lookup on FEE-890 (2026-10-23). Michael asked to reject instead on #1237, then approved building on the defaults (FEE-890, 2026-11-05).
  • Review focus: An item's own rid must name the space the route writes to. _local_concept_to_db_concept takes the space from the rid over the one the route passes, so an item with another space's rid would be written there. RIDs used as references (a relation's ends, a slot value) are not checked.
  • Review focus: metadata on schemas and assets on nodes are refused, because no converter writes them and accepting them would drop them without notice. UNWRITTEN_FIELDS lists them; removing an entry accepts that field again.
  • Review focus: Inline content with a scale other than document is refused. upsert_concepts stores every inline content as a document (its content_as_document default), so another scale would be changed without notice. No current caller sends another scale.
  • Review focus: A node's nodeType or a relation's relationType written as a RID is refused. upsert_concepts would store such a row with a NULL schema_id and no error when the RID names nothing, and the schema check looks up local ids only. A RID in a slot value or a triple end is still accepted: unresolved, it fails its row with error. A triple's relation written as a RID is looked up like a local id and gets 422 when missing. No current caller sends a RID type.
  • Review focus: An empty localId is refused by the validators. upsert_concepts turns an empty source_local_id into NULL, so the upsert never matches an existing row and every call would insert a new one.
  • Review focus: An item without modifiedAt is written with last_modified equal to its createdAt. modifiedAt is optional in the contract, but Concept.last_modified is NOT NULL, so such an item got HTTP 200 and was not stored. A live run of the route found this. Existing callers always set modifiedAt, so their output is unchanged. An update should carry modifiedAt: without it, the row's last_modified goes back to createdAt, and Obsidian reads last_modified to detect changes.
  • Review focus: The dependency-ordering helper has one copy here, used only by the route. It walks with its own stack, so a long chain of references can't overflow the call stack. The Roam and Obsidian copies in conceptConversion.ts are untouched and still recurse. They go when the apps move onto the route; those tickets don't exist yet.
  • Risk or follow-up, none ticketed yet:
    • A node typed by a relation type schema passes the schema check.
    • A schema in the body that fails to write leaves the nodes that use it stored with a NULL schema_id, and those nodes count as upserted.
    • The local part of an item's rid is not compared with its localId.
    • A rid inside a node's inline content is accepted and not written.
    • An inline embedding the database can't store (any model other than openai_text_embedding_3_small_1536 with 1,536 values) is accepted, dropped by upsert_content with a warning, and reported as upserted. Whether the route covers embeddings is FEE-890's open decision 5.
    • Items that reference each other, or an item whose slot names its own local id, can't be ordered. When neither row exists yet, both come back in failed with error and nothing is stored for them; when one already exists, both are written (read from the SQL, not run).

Verification

  • pnpm ci:validate passes: check-types 8/8 tasks, test:unit 6/6 tasks (database: 217 tests).
  • Mutation check: disabling the rid check, the unwritten-field check, the union flattening, the RangeError catch, the ordering or the result mapping each fails a test. On the previous head, the tests for RID types, a 10,000-node chain and unique_conflict each fail.

Loom video

fee-890-terminal-v2-proof.mp4

One video covers both FEE-890 PRs.

Setup: a local build of #1622's branch at 37a4277b. Later commits refuse a non-document scale, answer 502 for an auth server error, refuse dates outside years 1 to 9999 and RID types, order entries with an explicit stack, rename name_conflict to unique_conflict and bound the schema lookup; unit tests cover those, not the video. The route ran under next dev from the PR worktree, against an isolated local Supabase stack started from this branch's migrations, with edge functions served so create-space could make two new spaces. Every call is curl, and every case reads Postgres back on screen.

  • 0:01 Setup: branch head, local stack, two new spaces A and B, each signed in as its anonymous account.
  • 0:12 One call writes the whole graph: nine items, none with modifiedAt. The cited node is written before the node citing it, nodes and the relation have their schema, and the triple carries the relation type's label and complement.
  • 0:29 Every row's last_modified equals its created.
  • 0:36 The same node sent again with a later modifiedAt updates the same row, and its last_modified moves forward. No new row is added.
  • 0:46 A triple missing a field, an item whose rid names another space, and a node with assets each get 400 naming the field. Nothing is written.
  • 0:58 No token, or a token the auth server rejects, gets 401. Nothing is written.
  • 1:08 Another space's token gets 403 here, and nothing is written. The same token works on its own space.
  • 1:18 A node type in neither the body nor the space gets 422 listing the missing id. A type that is only in the database works, and the node lands with that schema.
  • 1:32 A node reusing an existing name comes back in failed as name_conflict (now unique_conflict), while the other node in the call is written.
  • 1:43 An empty body gets 200 and writes nothing. The CORS preflight from roamresearch.com gets the allow-origin header.

Scope check

  • Ran $scope-check against FEE-890 and the final diff.
  • Scope beyond Done When: Refusing schema metadata, node assets, a non-document scale and RID node or relation types is narrower than "all aspects of upserting"; the Reviewer brief gives the reasons. Whether FEE-890 ships without these is open with sid; no Decision: comment on FEE-890 records it yet.

Standards check

  • Ran $dg-pr-adherence-check against the final diff and PR metadata.

Local delegated full review

  • Ran a comprehensive review of the entire final diff in a subagent with a fresh context. Use $dg-delegated-full-review when no other full-review workflow is available.
  • Ran Review Lab (DiscourseGraphs/review-lab at 01b07f3) on 85223e99: one P3 finding (F5), listed under Risk; nothing blocks review. Review Lab page

@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
discourse-graph Ready Ready Preview Nov 16, 2026 5:54am UTC

Request Review

@linear-code

linear-code Bot commented Nov 13, 2026

Copy link
Copy Markdown

FEE-890

@supabase

supabase Bot commented Nov 13, 2026

Copy link
Copy Markdown

This pull request has been ignored for the connected project zytfjzqyijgagqxrzbmz because there are no changes detected in packages/database/supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.


Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

Concept.last_modified and Content.last_modified are NOT NULL, so an item
the contract accepts without modifiedAt failed in upsert_concepts. An item
never modified was last modified when it was created.
…reatedAt

Only an item with createdAt and no modifiedAt should change. Roam's publish
tests mock relations and triples with neither date, and those threw.
@sid597
sid597 marked this pull request as ready for review November 16, 2026 16:38
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T16:45:13.418768Z 85223e9 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment on lines +281 to +283
const related = byId.get(id);
if (related !== undefined && !processed.has(id))
stack.push(enter(related));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Same-space RID dependencies written too late

When a slot references a later new item by same-space RID, orderByDependency leaves the referring item first. byId contains only local IDs, so the RID never matches. The referring item fails despite its dependency being included in the upsert.

Learn more

The ordering helper indexes entries by local ID but compares reference strings without resolving RIDs. parseUpsertData accepts RIDs in slots and triple endpoints. The database reference resolver resolves those RIDs to stored concepts, not pending input entries. An unresolved scalar reference produces a NULL reference value in _local_concept_to_db_concept, causing the row to fail before its dependency is inserted. This affects node slots, schema slots, and triple endpoints when their referenced entry comes later.

Example: In an initially empty space, send nodes evidence-1 followed by source-1, with the first node's source slot set to https://roamresearch.com/#/app/graph-a/source-1. Ordering returns evidence-1, source-1; replacing the RID with source-1 correctly returns source-1, evidence-1. With the RID, the first node fails although the request contains its source.

Recommended fix: Supply the target space URL to dependency ordering and normalize same-space RID references to their local IDs before looking up entries. Preserve cross-space RIDs as external dependencies. Add ordering tests for same-space RID references with and without an explicit RID on the referenced item.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 85223e99e8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +189 to +193
const schemaIds = new Set([
...relationTypeIds,
...(data.nodeSchemas ?? []).map((s) => s.localId),
...(data.relationTripleSchemas ?? []).map((s) => s.localId),
]);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate schema references against their expected kind

When a node's nodeType matches a relationTypeSchemas entry, this set treats the ID as a valid schema reference regardless of kind. schemaReferences then omits it, so findMissingSchemas cannot reject it and toOrderedEntries writes the wrong schema ID into the node's schema_id, persisting an invalid graph. Track the expected schema kind for each reference and reject mismatches.

Useful? React with 👍 / 👎.

Comment on lines +93 to +95
if (
item.rid !== undefined &&
ridToSpaceUriAndLocalId(item.rid).spaceUri !== spaceUrl

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject or preserve RIDs on inline content

If content.direct, directAndDescription, or full includes its own rid—permitted by the inline content validator—this check validates only the top-level item's rid. inlineCrossAppContentToDbContent does not map the nested RID to space_url, so the upsert stores that content in the enclosing node's space instead of the space it declares. Reject nested RIDs until supported, or preserve their space when converting.

Useful? React with 👍 / 👎.

This branch was successfully deployed

1 active deployment
Preview — 85223e99 Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant