Skip to content

Serve lazy Datasets over Arrow Flight SQL #256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alxmrs wants to merge 11 commits into
base: main
Choose a base branch
from
+2,057 −121
Open

Serve lazy Datasets over Arrow Flight SQL #256

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
4163116
Serve lazy Datasets over Arrow Flight SQL
alxmrs Sep 26, 2026
c7f66b2
Bound FlightSQLServer shutdown with a timeout
alxmrs Sep 26, 2026
960e653
Serve plain Arrow Flight path descriptors alongside Flight SQL
alxmrs Sep 26, 2026
5d4a02c
Test Spark and ClickHouse as Flight clients; document both
alxmrs Sep 26, 2026
ee8eae1
Skip the arrowFlight test unless ClickHouse is a server
alxmrs Sep 27, 2026
653b461
Drop a needless clone of a Copy type (clippy)
alxmrs Sep 27, 2026
b75d5c9
Return a bool from FlightSQLServer.is_running (mypy)
alxmrs Sep 27, 2026
2a3f2fb
Keep a served table's name exactly as registered
alxmrs Sep 28, 2026
ba6a8b7
Run the ADBC contract and ERA5 queries against the Flight SQL server
alxmrs Sep 28, 2026
1db5efa
Find mixed-case tables by a plain Flight path; surface server errors
alxmrs Sep 29, 2026
0f4a116
Bound the server's query memory; register mixed Datasets all or none
alxmrs Sep 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 17 additions & 3 deletions .github/workflows/adbc-databases.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@
# real database servers in service containers, one job per database so
# each pulls only its own image and a failure names its database. The
# main CI job runs the contract on the in-process backends (SQLite,
# DuckDB, chDB, DataFusion); the "in-process" job here adds their
# ARCO-ERA5 queries.
# DuckDB, chDB, DataFusion, and xarray-sql's own Flight SQL server); the
# in-process job here adds their ARCO-ERA5 queries. The ClickHouse job
# also has ClickHouse query that Flight SQL server over Arrow Flight.
name: adbc databases

on:
Expand All @@ -13,12 +14,15 @@ on:
paths:
- "xarray_sql/backends/adbc.py"
- "xarray_sql/backends/_adbc_dialects.py"
- "xarray_sql/backends/flight.py"
- "src/flight.rs"
- "xarray_sql/ds.py"
- "xarray_sql/lazyscan.py"
- "xarray_sql/roundtrip.py"
- "tests/_adbc.py"
- "tests/test_adbc_backend.py"
- "tests/test_adbc_era5_integration.py"
- "tests/test_flight_sql.py"
- ".github/workflows/adbc-databases.yml"
workflow_dispatch:

Expand All @@ -35,7 +39,7 @@ jobs:
fail-fast: false
matrix:
include:
- backend: sqlite,duckdb,chdb,datafusion
- backend: sqlite,duckdb,chdb,datafusion,served
- backend: postgresql
image: postgres:18
port: 5432
Expand Down Expand Up @@ -72,6 +76,8 @@ jobs:
image: ${{ matrix.image }}
ports:
- ${{ matrix.port || 1 }}:${{ matrix.port || 1 }}
# Lets ClickHouse reach a Flight SQL server the tests start here.
options: --add-host=host.docker.internal:host-gateway
env:
POSTGRES_PASSWORD: xql
MYSQL_ROOT_PASSWORD: xql
Expand Down Expand Up @@ -155,6 +161,14 @@ jobs:
if: matrix.uri
run: uv run --no-project pytest -v -rs tests/test_adbc_backend.py

- name: Query xarray-sql's Flight SQL server from ClickHouse
if: matrix.backend == 'clickhouse'
run: >-
uv run --no-project pytest -v -rs tests/test_flight_sql.py
-k clickhouse
env:
XARRAY_SQL_TEST_CLICKHOUSE_FLIGHT_HOST: host.docker.internal

- name: Run realistic ARCO-ERA5 queries
# Reads a regional subset anonymously from the public bucket.
run: >-
Expand Down
Loading
Loading