feat: use jdv as the data viewer backend, including Arrow IPC stream support

[eitsupi]

PoC for #1860. Not intended to merge until jdv is on CRAN.

Requires the current development version of jdv. jdv is unreleased, so older jdv versions are not supported.

Summary

When the jdv package is installed, View() on a data frame, matrix, list or Arrow data is served by jdv instead of the data viewer code in sess, in both R terminals and Interactive. Interactive inline tables and lists are also served by jdv. The existing AG Grid and List Viewer webviews are reused; the R-side data handling moves to jdv.

• Transport. The extension host listens on a per-session Unix socket or Windows named pipe (src/jdvTransport.ts) and implements the host side of the jdv transport: hello validation (protocol version, producer, opaque session id, required limits), ack credit, wakes only when the producer asks for them, at most 32 live requests, and length-prefixed Arrow bodies. On the first View(), sess gets the address with request_client("jdv/address") and opens a persistent connection with jdv::jdv_connect(address), so jdv never uses a shared discovery file or options(jdv.address). The listener is private to the user (mode 0600, random name) and its address is only given to that session.
• sess. View() registers the object with jdv::jdv_register(mode = "paged"); jdv announces the view to the host itself with a view/open notification.
• Extension. src/jdvAdapter.ts (VS Code-independent, unit tested) validates view descriptors, converts AG Grid sort/filter models to the jdv query AST, jdv column pages to grid rows, and List Viewer index paths to jdv typed paths. src/jdvViewer.ts manages the panels per session. Grid pages request only the displayed columns (jdv cols), so hidden columns of a wide data frame cost nothing.
• Interactive. Explicit View() in an Interactive cell opens the Data Viewer panel through a jdv listener owned by that notebook (see below). Inline previews use jdv handles.
• Arrow. Arrow tables, record batch readers, nanoarrow array streams and Polars data frames are viewed as forward-only Arrow streams, without converting them to data frames (see below).
• Fallback. Without jdv (or nanoarrow for Arrow data), with options(sess.dataviewer = "sess"), or when jdv is already connected elsewhere or used by an embedding host, the current sess viewer is used.

Why jdv

The goal is separation of responsibilities, not speed (see benchmark below).

• A self-contained, dependency-free data backend. jdv is a C-based package with no R package dependencies and its own written protocol and query specification (?jdv_spec, ?jdv_query_spec). Data viewing no longer has to live in sess or depend on jsonlite, and the same backend can be tested and used outside vscode-R.
• R semantics for sorting and filtering. For example, an unordered factor sorts by its levels in jdv. Missing values come from the column's missingness mask, never from display text, and NaN is kept distinct from NA in the display.
• Safer object browsing. Tree navigation never forces promises (e.g. lazy-loaded bindings), never evaluates active bindings, and never calls classed [[, $ or names methods. It detects cycles and reports unreadable children individually instead of failing the whole list.
• Bounded work. Page rows, response size, and request size are limited, and oversized pages are shortened instead of producing unbounded JSON.
• Live views. On R >= 4.6, jdv can follow the variable passed to View() and rebuild the view when it is reassigned.
• Arrow without conversion. Arrow tables, record batch readers and Polars data frames are read through the Arrow C stream interface and sent as Arrow IPC. Nothing is collected into an R data frame, and the first rows show before the stream is fully read.

Following variables, Refresh and Follow

• On R >= 4.6, View(df) passes the binding of df (found with jdv::jdv_binding(), without evaluating anything) so that df <- transform(df, ...) updates the open view in place. Updates from tracking or Refresh do not bring the tab to the front; an explicit View() does (jdv's view/open.reason).
• New setting r.session.viewers.trackVariables (default true), passed to sess::connect(track_variables = ). options(sess.dataviewer.track = TRUE/FALSE) in R overrides it in both directions. Arrow sources are never tracked.
• The grid and List Viewer toolbars get a Refresh button (view/refresh, when the view supports it) and, for tracked views, a Follow toggle (view/track) that pauses or resumes following. The toggle reflects view/tracking notifications (paused, or lost when the variable is removed).
• Failed tracked updates (view/update_failed) are reported with a warning.
• sess's development version is bumped to 3.2.9000.9000 for the connect() change.

Opening cells

Cells of columns that jdv marks as drillable (e.g. list columns) get an open button in the grid. It sends object/open with the cell's original source_row, so it opens the right cell after sorting and filtering. The child opens as its own grid or List Viewer panel, titled by jdv (e.g. df[2, "l"]).

Interactive

• View(). For an Interactive notebook, the extension host creates a jdv listener keyed by the notebook and its R generation, and answers the jdv/address client request from the agent. The listener is closed on detach, stop, restart and host dispose. Explicit View() now opens the Data Viewer panel instead of an inline preview.
• Inline tables and lists. Each table or list output holds one registry-free jdv::jdv_handle(mode = "paged") in R. Handles take no viewer slot, send no notifications and do not evict open views. The first page (20 rows, or 200 list children, at most 256 KiB) is stored in the cell output as before, so saving and exporting are unchanged. Paging, sorting, filtering and list exploration go through jdv_handle_request(). At most 64 handles are kept per R process (oldest closed first); clearing a completed cell, stopping the notebook or clearing the runtime closes them.
• Open Data viewer. Opening a truncated table (or a table inside a list) registers the value with jdv and shows it in the Data Viewer panel, titled Cell N: ....
• The sess dataview path is still used for inline outputs without jdv, with options(sess.dataviewer = "sess"), for Arrow/Polars values, and for values jdv cannot represent. Outputs saved before this change stay static, and live interaction after a restart shows the existing "previous R process" message.

Arrow data

• Cache in the extension host. The stream is forward-only, so received IPC pages are kept per view generation (64 MiB per view, 256 MiB in total, oldest first). Rows evicted from the cache are shown as no longer retained. This keeps scrolled-away rows and webview reloads working without re-reading R.
• Decoding in the extension host. Pages are decoded with apache-arrow and formatted from the physical buffers (exact 64-bit integers, decimals, and timestamps with time zones; formatting ported from jdv's reference viewer). The webview keeps the existing AG Grid infinite row model and receives JSON rows for the requested block, with an unknown row count until the stream ends. Sorting and filtering are disabled. Nested values show a summary (<list: 2 items>), and unsupported columns show jdv's reason.

Decoding in the extension host rather than in the webview keeps the webview and its message protocol almost unchanged, keeps a single cache next to the data, and leaves the Arrow data where a future extension-side query engine or nested-cell inspection could use it. Measured cost: about 14 ms to decode and format a 2,000-row, 5-column page.

Behavior changes

• Display strings are formatted by R per page, so a column can show 3.0 on one page and 3 on another.
• Row numbers are jdv's row labels (row names when present, otherwise the original row position).
• Filters offer only the operators jdv defines for each column type: text filters for character and categorical (factor) columns, number filters (also for difftime/hms durations, in the column's units), date filters, and "Is NA / Is not NA" for every column. List columns and other classes cannot be filtered. Text columns also offer "Is empty / Is not empty", kept separate from "Is NA / Is not NA" (combine them with OR to match either); AG Grid's "Blank / Not blank" are not offered. Number ranges are inclusive.
• View(x$a$b) opens the value of x$a$b as its own root, instead of opening x navigated to a/b. Repeated View() reuses a tab per title, not per root object. Child views are titled by jdv (x$a, x[[3]], df[12, "col"]).
• The List Viewer also shows attributes such as names as children.
• When the sess runtime stops (detach or disconnect), sess closes the jdv connection it opened, and with it the views.
• In Interactive, explicit View() opens a panel instead of an inline preview, and inline table previews reference the original object instead of a copy of its first 1,000 rows (the printed text is still taken when the cell runs).

Benchmark (informational)

1e6-row, 8-column data frame, one 500-row page, R 4.5.2 on Linux, elapsed seconds (measured with an earlier relay; the query work in R is the same):

Operation	sess	jdv
Register + first page	0.037	0.070
Sort numeric desc	0.192	0.207
Sort character + factor	0.074	0.130
Filter text contains	0.224	0.399
Filter number range + sort integer64	0.063	0.313

Both are interactive-speed for this size; jdv is somewhat slower in these cases.

Testing

• src/test/node/jdvAdapter.test.ts: descriptor validation (reason, tracking, capabilities), query and column conversion (including display-only and duration columns), row labels, tree paths, cell object/open paths.
• src/test/node/jdvArrow.test.ts: value formatting, the page cache, and the transport against a fake producer (hello, wakes, ack, Arrow bodies, view/track and view/tracking, errors, disconnect). Runs on Linux and Windows (named pipe) in CI.
• src/test/node/jdvTable.test.ts, interactiveJdvAddress.test.ts, interactiveRuntime.test.ts: Interactive handle tables (page, sort, filter, reconnect, close), list exploration, opening in the Data Viewer, the per-notebook jdv listener, and View() from a real R cell.
• src/test/suite/dataViewer.test.ts: jdv column definitions.
• sess/inst/tinytest/test-jdv.R and test-interactive-handles.R (skipped when jdv is not installed): View() through a fake TCP host, tracking (R >= 4.6), Arrow, release, and Interactive handles. Existing sess viewer tests pin options(sess.dataviewer = "sess").

Not included

• Sorting/filtering Arrow data, and inspecting nested Arrow cells.
• Snapshot views (all data sent at once) and client-side sorting.
• Removal of the sess viewer code (still the fallback without jdv).
• Applying a changed trackVariables setting to already attached sessions.
• Refresh on a List Viewer returns to the root.
• Manual testing in VS Code of the Follow toggle, cell open buttons and Interactive inline tables.

Follow-up

• Viewport columns for wide tables. With every column displayed, each 500-row page still builds all columns in R. For 686k rows x 150 columns this is about 230 ms in R and 1.9 MB of JSON per page, versus about 40 ms for 25 columns; the time scales with the number of columns. Requesting only the columns AG Grid renders plus a margin (getAllDisplayedVirtualColumns(), virtualColumnsChanged) and reloading the visible blocks when the view scrolls past them can be done in vscode-R alone, reusing the cache purge used when a hidden column is shown. Filling missing columns into cached rows instead of reloading would avoid the brief reload but needs per-block column tracking, so it will be considered only if the simple version flickers noticeably.

[renkun-ken]

Reviewed 5599b4ff7204bc451626686e7139122cb23a571b. I found two issues in the Interactive jdv path, both reproduced with the real agent/R backend: shortened pages can skip rows, and opening a nested data frame can dispatch a custom [[ method that jdv's tree browser deliberately avoids. Details and reproduction inputs are inline.

Validation used R 4.6.1, the PR's built dist/resources/sess installed in an isolated library, development jdv at 2cd06199364baa547d952366ea77df178ee4c86e, and nanoarrow 0.9.0. Build, TypeScript compilation, lint, 35 focused Node tests, 12 Data Viewer script tests, five existing real-R jdv integration tests, 68 assertions in the two new sess test files, and source-identity checks passed. An additional real-R Arrow IPC test decoded numeric, string, logical, date, timestamp, and missing values successfully. The two additional regression reproductions failed as described inline.

All five GitHub checks are green. I did not manually exercise the VS Code webviews.

[renkun-ken]

Follow-up review of 55799dfeddb76add0b78243c625044b4298f7f8c.

The previous nested-list extraction finding is fixed: the original custom-[[ reproduction now opens successfully through the real Interactive agent, and the new root/child handle tests pass. The on-demand paging loop also now returns IDs 1–100 and 101–150 without gaps when the combined initial preview fits the event limit.

Two issues remain in the new changes, detailed inline:

• Refilling the initial preview can exceed the 2 MiB rich-event limit. The original 100,000-character-cell reproduction now emits only a truncation event, losing the entire inline table.
• The new handle-registration test leaves an embedded jdv session behind, so running it followed by test-jdv.R produces an order-dependent test failure.

Validation used R 4.6.1, this revision's built sess in an isolated library, jdv 660703e (including jdv_handle_register()), and nanoarrow 0.9.0. Build, TypeScript compilation, lint, 54 focused Node/viewer tests, six existing real-R jdv integration tests, source-identity checks, and additional nested-list, contiguous-paging, and Arrow IPC checks passed. The two R files pass separately (51 and 50 assertions), but together produce 100 passes and one failure. The original wide-preview regression fails through the real agent/R path as described inline.

All five GitHub checks are green; the current Ubuntu log confirms both jdv R test files are skipped because jdv is not installed. I did not manually exercise the VS Code webviews.

[eitsupi]

I believe I have addressed all of the points.

[renkun-ken]

Reviewed de07a676c7af729ba8e30ea99a3091b34d885048. All of my previously reported findings are resolved, and I found no remaining actionable issues in the latest changes. LGTM.

• The original 100,000-character-cell input now produces a bounded table display instead of a truncation event. Real-agent tests retrieve all 150 rows without gaps at both 20- and 100-row page sizes. Browser checks also confirm automatic completion of partial/empty previews, correct Next/Reset behavior, and no requests for disconnected, stopped, or archived outputs.
• The handle and View R test files now pass together: 134 assertions, with no leaked embedded jdv session.
• The nested classed-list reproduction still passes. I also checked the Arrow numeric-string preservation, row-count hint scoping, and a real Polars stream publishing its known row total while continuing to page through Arrow IPC.

Validation: build, TypeScript compilation, lint, 69 focused Node/viewer tests, eight existing real-R jdv integration tests, an additional real-Polars integration check, 135 browser-renderer assertions, and source-identity checks all passed. Native tests used this revision's built sess in an isolated library with R 4.6.1, jdv 660703e, and nanoarrow 0.9.0. Renderer interaction was exercised in the browser harness. All five GitHub checks are green on the reviewed head.

[eitsupi]

@renkun-ken Thanks for the approval.
After it, I pushed a few follow-up commits to address the wide data frame scrolling performance reported in #1860:

• Fetch only displayed columns: grid pages now request only the visible columns via jdv's cols, so hidden columns of a wide data frame no longer cost anything. Showing a column again refetches the rendered blocks in place.
• Faster frame receiving: JdvPeer no longer re-copies its buffer for every chunk, which was quadratic for large responses.
• jdv protocol update: missing values are now read from jdv's sparse na_rows positions instead of the dense na array.

Fetching only the columns in the visible horizontal range, which would help when all columns of a wide table are shown, is left as a follow-up (noted in the PR description).