Repository navigation
0.1.8: Iceberg virtual-edge sources (DuckDB) — Glue + S3 Tables, proven live - #9
Merged
Merged
Conversation
Virtual edges can now resolve against an Apache Iceberg table in a lakehouse, not just an operational SQL store. Set source_type="iceberg" with a pyiceberg `catalog` config + `table` (namespace.table); pyiceberg resolves the current metadata pointer (or a pinned `snapshot_id` for time-travel), DuckDB scans it. The table is registered as a named DuckDB view, so the binding's query is written against an ordinary table name — identical to the SQLite/Postgres case, and resolve()'s parameterized bind path is unchanged. Catalog values written "env:VAR" resolve from the environment, keeping creds out of the graph (same discipline as dsn_env). Scope: format-version 2 (where the ecosystem still lives); the reader is version-agnostic and will pick up v3 reads transparently as DuckDB/pyiceberg gain them. pyiceberg can't yet *write* v3 (apache/iceberg-python#1551), so v3 is read-only-when-available and untested by fixtures here. - kgrdbms/iceberg.py: catalog→DuckDB view opener, env: indirection - virtual.py: source_type/catalog/table/snapshot_id on VirtualEdge; _open_source() dispatch; driver-agnostic row→dict (duckdb yields tuples) - mcp_server.py: kg_virtual_edge_add/list carry the iceberg fields - tests/test_iceberg.py: real SqlCatalog v2 table, end-to-end resolve, env indirection, snapshot pinning, roundtrip (skips without the extra) - pyproject: [iceberg] extra (duckdb, pyiceberg, pyarrow); version 0.1.8 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Iceberg source now reaches s3://-backed warehouses, not just local files. When the resolved metadata location is on s3, open_source loads DuckDB's httpfs/aws extensions and a credential-chain secret automatically, so iceberg_scan reads the managed storage with the host's AWS identity (sigv4 included) — predicate pushdown intact. Verified end-to-end against live AWS S3 Tables: a HOLDS virtual edge bound to the SEC form13f.infotable resolves an institutional manager's holdings (CIK -> issuers) straight out of the lakehouse on traversal, zero rows copied into the graph. AWS Glue (type=glue) rides the identical path. - iceberg.py: _prepare_s3() (httpfs/aws + credential_chain secret, region from catalog hint or AWS env); s3:// detection in open_source - tests/test_iceberg.py: env-gated live S3 Tables integration test (KG_ICEBERG_S3TABLES_ARN), skipped in CI - pyproject: iceberg extra gains glue+s3fs+boto3 for cloud catalogs - README: Glue / S3 Tables / SQL catalog config table Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ne edge A binding mounted exactly one table, so its query couldn't join. Add `tables` (a list, alongside the primary `table`): every entry is resolved via pyiceberg and registered as a DuckDB view by leaf name on the same connection, so one edge's query can JOIN across them freely. One S3 credential setup serves all views. `snapshot_id` is rejected with multiple tables (a snapshot pins a single table's version). Proven live: a HOLDS edge joining S3 Tables form13f.infotable ⨝ cusip_lookup resolves a manager's holdings with real issuer names + sector, on traversal, zero copy. - virtual.py: VirtualEdge.tables + iceberg_tables() (dedup, order-preserving); _open_source passes the full set; persisted in to/from_node - iceberg.py: open_source mounts a list of tables as views; single-setup S3 prep - mcp_server.py: kg_virtual_edge_add/list carry `tables` - tests: local multi-table JOIN, snapshot-pin-rejects-multiple Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Widen the missing-extra handlers from ModuleNotFoundError to ImportError so a blocked/partial install of duckdb/pyiceberg still surfaces the clear "install the iceberg extra" hint instead of a raw traceback. Add a regression test that blocks duckdb/pyiceberg/pyarrow at the import finder and proves: kgrdbms imports, a SQLite virtual edge resolves, and only an iceberg binding fails — with the install hint. Locks in the promise that the heavy deps are import-lazy and confined to the [iceberg] extra (core dependencies stay []). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Virtual edges can now resolve against Apache Iceberg tables in a lakehouse, not just an operational SQL store. Set
source_type="iceberg"on a binding with a pyicebergcatalogconfig +table; pyiceberg resolves the current metadata pointer (or a pinnedsnapshot_idfor time-travel), DuckDB scans it. The table is mounted as a named DuckDB view, so the binding'squeryis written against an ordinary table name — byte-for-byte identical to the SQLite/Postgres path, andresolve()'s bind logic is unchanged.What's in it
kgrdbms/iceberg.py): catalog → DuckDB view opener;env:VARindirection keeps catalog creds out of the graph (same discipline asdsn_env).type=glue) and S3 Tables (managed Iceberg, REST sigv4) ride the same seam. When resolved metadata iss3://, the opener auto-loads DuckDB'shttpfs/awsextensions + a credential-chain secret, so it reads managed storage with the host's AWS identity (predicate pushdown intact).tables=[...]) from one catalog and JOIN across them in a single edge (e.g. holdings ⨝ name-lookup).snapshot_idis rejected with multiple tables.Proven live
End-to-end against AWS S3 Tables (
form13flakehouse): aHOLDSedge bound toinfotable(⨝cusip_lookupfor issuer names) resolves an institutional manager's holdings — CIK → issuers, with real names + sector — straight out of the lake on traversal, zero rows copied into the graph. Gated integration test:KG_ICEBERG_S3TABLES_ARN=… pytest -k s3tables_live(skipped in CI).Scope
Format-version 2 — pyiceberg can't yet write v3 (iceberg-python#1551); the reader is version-agnostic and will pick up v3 reads transparently when the toolchain lands them.
Tests
122 passed, 9 skipped (local Iceberg via
SqlCatalogv2 + multi-table join + snapshot pinning; AWS path env-gated). New extra:pip install 'knowledge-graph-rdbms[iceberg]'.🤖 Generated with Claude Code