Add jdbc-v2: read DatabaseMetaData with SHOW and DESCRIBE statements

CHANGELOG.md

@@ -31,6 +31,11 @@
 
 - **[r2dbc]** Pre `1.0.0` is not supported anymore because R2DBC API reached stable `1.0.0` version.  
 
+- **[jdbc-v2]** `DatabaseMetaData#getTables` now returns `null` in `REMARKS` (table comment) and `TYPE_SCHEM` by
+  default, because metadata is read with `SHOW` statements (see the `jdbc_metadata_use_show_statements` entry in New
+  Features). Set `jdbc_metadata_use_show_statements=false` to get the table comments.
+  (https://github.com/ClickHouse/clickhouse-java/issues/2907)
+
 ### New Features
 
 - **[client-v2,jdbc-v2]** Added ZSTD compression support for Block compression stream. Previously only
@@ -177,6 +182,15 @@ now required to work with server. (https://github.com/ClickHouse/clickhouse-java
   as `(INDEX, VALUE)` pairs where each `VALUE` is the tuple. (https://github.com/ClickHouse/clickhouse-java/issues/2477)
 - **[client-v2, jdbc-v2]** Added logging on previously-silent error and diagnostic paths (no functional or
   public-API change). (https://github.com/ClickHouse/clickhouse-java/issues/2969)
+- **[jdbc-v2]** `DatabaseMetaData#getSchemas`, `#getTables` and `#getColumns` now read metadata with `SHOW DATABASES`,
+  `SHOW TABLES` and `DESCRIBE TABLE` instead of the `system.databases`, `system.tables` and `system.columns` tables.
+  The server does not show some tables in the system tables by default (for example, tables of `DataLakeCatalog`
+  databases), so tools that use `DatabaseMetaData` did not find them. The new driver property
+  `jdbc_metadata_use_show_statements` (default `true`) selects the implementation; set it to `false` to use the system
+  tables. With the `SHOW` statements, `getTables()` returns `null` in `REMARKS` (table comment) and `TYPE_SCHEM`;
+  `getColumns()` sends one `DESCRIBE TABLE` query for each table that matches and skips a table that was dropped
+  meanwhile, that the user cannot describe, or whose data lake metadata cannot be read. All other columns and values
+  are the same. (https://github.com/ClickHouse/clickhouse-java/issues/2907)
 
 ### Bug Fixes 
 

docs/features.md

@@ -88,6 +88,7 @@ Compatibility-sensitive traits:
 - Binary string reads: `ResultSet#getBytes(int|String)` and `ResultSet#getBinaryStream(int|String)` return the raw bytes of a `String`/`FixedString` column. Combined with the `binary_string_support` connection property, non-UTF-8/binary content stored in `String` columns round-trips byte-for-byte; `NULL` values report `null` with `wasNull()` set. `ResultSet#getObject(...)` never exposes the internal `StringValue` holder for these columns: `getObject(column, byte[].class)` returns the raw bytes, while `getObject(column, Object.class)` and the no-type `getObject(column)` overloads return a decoded `String`.
 - Result-set metadata: Exposes JDBC `ResultSetMetaData` backed by ClickHouse column schema.
 - Database metadata: Implements JDBC `DatabaseMetaData` for ClickHouse catalogs, schemas, tables, columns, and related capability reporting.
+- Metadata with `SHOW` statements: By default `DatabaseMetaData#getSchemas`, `#getTables` and `#getColumns` use `SHOW DATABASES`, `SHOW FULL TABLES` and `DESCRIBE TABLE`, so they also return tables that the server does not show in the system tables by default (for example, tables of `DataLakeCatalog` databases). The driver property `jdbc_metadata_use_show_statements=false` selects the `system.databases`, `system.tables` and `system.columns` queries.
 - Parameter metadata: Reports prepared-statement parameter counts.
 - Type mapping and conversions: Maps ClickHouse types to JDBC types and Java classes, including date/time handling and `java.time` support.
 - QBit type mapping: For ClickHouse `25.10+`, JDBC exposes the experimental `QBit(element_type, dimension)` type as `ARRAY`, returning the vector as a `java.sql.Array` of the element type from `getObject()`/`getArray()`. Supported element types are `BFloat16` and `Float32` (both `java.lang.Float`) and `Float64` (`java.lang.Double`). The `allow_experimental_qbit_type` server setting is required to create a `QBit` column.
@@ -103,6 +104,7 @@ Compatibility-sensitive traits:
 Compatibility-sensitive traits:
 
 - Prepared statements are client-side SQL rendering, not server-side prepared statements. Changes to literal encoding or placeholder parsing are externally visible behavior.
+- `DatabaseMetaData#getSchemas`, `#getTables` and `#getColumns` return the same result-set columns and values with and without `jdbc_metadata_use_show_statements`, with two exceptions for the `SHOW` statements: `getTables()` returns `null` in `REMARKS` and `TYPE_SCHEM`, because `SHOW TABLES` returns no table comment and no database engine. `getColumns()` with the `SHOW` statements sends one `DESCRIBE TABLE` query for each table that matches, and skips a database or table that fails with `UNKNOWN_TABLE`, `UNKNOWN_DATABASE`, `ACCESS_DENIED` or `DATALAKE_DATABASE_ERROR` (for example, a table with column-level grants only, which `system.columns` shows with the granted columns).
 - JDBC `cluster_name` is compatibility-sensitive for cancellation behavior: when configured, `Statement.cancel()` issues `KILL QUERY ON CLUSTER '<name>'`, and when omitted it falls back to a local `KILL QUERY`.
 - String parameters are escaped with backslash-based escaping: backslashes are doubled and single quotes are backslash-escaped before values are wrapped in single quotes.
 - `?` placeholder detection is SQL-aware and should not treat question marks inside quoted strings, quoted identifiers, comments, casts, or similar syntax as bind parameters.

docs/integration-jdbc.md

@@ -833,6 +833,7 @@ Key JDBC-specific properties (see [`DriverProperties`](../jdbc-v2/src/main/java/
 | `jdbc_type_mappings` | — | Custom ClickHouse → Java type overrides |
 | `default_query_settings` | — | Default settings for all queries |
 | `jdbc_metrics_recorder` | — | Custom `MetricsRecorder` implementation class name |
+| `jdbc_metadata_use_show_statements` | `true` | `DatabaseMetaData` schemas, tables and columns via `SHOW` / `DESCRIBE` instead of `system.*` tables |
 
 
 ## Observability & Monitoring

jdbc-v2/src/main/java/com/clickhouse/jdbc/DriverProperties.java

@@ -160,6 +160,17 @@ public enum DriverProperties {
      */
     METRICS_RECORDER("jdbc_metrics_recorder", null),
 
+    /**
+     * Makes {@link java.sql.DatabaseMetaData#getSchemas()}, {@link java.sql.DatabaseMetaData#getSchemas(String, String)},
+     * {@link java.sql.DatabaseMetaData#getTables} and {@link java.sql.DatabaseMetaData#getColumns} read metadata with
+     * {@code SHOW DATABASES}, {@code SHOW TABLES} and {@code DESCRIBE TABLE} instead of {@code system.databases},
+     * {@code system.tables} and {@code system.columns}. These statements also return tables that the system tables
+     * hide by default, for example tables of {@code DataLakeCatalog} databases. With these statements
+     * {@code getTables()} returns {@code null} in {@code REMARKS} and {@code TYPE_SCHEM}.
+     * Default: true
+     */
+    METADATA_USE_SHOW_STATEMENTS("jdbc_metadata_use_show_statements", String.valueOf(Boolean.TRUE)),
+
     ;
 
 

jdbc-v2/src/main/java/com/clickhouse/jdbc/internal/DetachedResultSet.java

@@ -1,7 +1,9 @@
 package com.clickhouse.jdbc.internal;
 
+import com.clickhouse.data.ClickHouseColumn;
 import com.clickhouse.jdbc.JdbcV2Wrapper;
 import com.clickhouse.jdbc.ResultSetImpl;
+import com.clickhouse.jdbc.metadata.ResultSetMetaDataImpl;
 
 import java.io.InputStream;
 import java.io.Reader;
@@ -97,6 +99,20 @@ public static DetachedResultSet createFromResultSet(ResultSet resultSet, Calenda
         return new DetachedResultSet(records, metaData, defaultCalendar);
     }
 
+    /**
+     * Creates a result set from records built by the driver.
+     *
+     * @param records - rows, each maps a column name to its value
+     * @param columns - columns of the result set
+     * @param defaultCalendar - calendar used for date and time values
+     * @return result set with the records
+     * @throws SQLException - if the result set cannot be created
+     */
+    public static DetachedResultSet createFromRecords(List<Map<String, Object>> records, List<ClickHouseColumn> columns,
+                                                      Calendar defaultCalendar) throws SQLException {
+        return new DetachedResultSet(records, new ResultSetMetaDataImpl(columns, "", "", "", null, null), defaultCalendar);
+    }
+
     @Override
     public boolean next() throws SQLException {
         ensureOpen();

jdbc-v2/src/main/java/com/clickhouse/jdbc/metadata/DatabaseMetaDataImpl.java

@@ -78,6 +78,10 @@ public String getTypeName() {
 
     private String jdbcUrl;
 
+    private final boolean useShowStatements;
+
+    private final ShowStatementsMetaData showStatementsMetaData;
+
     /**
      * Creates an instance of DatabaseMetaData for the given connection.
      *
@@ -93,6 +97,8 @@ public DatabaseMetaDataImpl(ConnectionImpl connection, boolean useCatalogs, Stri
         this.useCatalogs = useCatalogs;
         this.catalogPlaceholder = useCatalogs ? "'local' " : "''";
         this.jdbcUrl = url;
+        this.useShowStatements = connection.getJdbcConfig().isFlagSet(DriverProperties.METADATA_USE_SHOW_STATEMENTS);
+        this.showStatementsMetaData = new ShowStatementsMetaData(connection);
     }
 
     private Statement createStatement() throws SQLException {
@@ -104,6 +110,10 @@ private Statement createStatement() throws SQLException {
     }
 
     private PreparedStatement prepareStatement(String sql) throws SQLException {
+        return prepareStatement(connection, sql);
+    }
+
+    static PreparedStatement prepareStatement(ConnectionImpl connection, String sql) throws SQLException {
         PreparedStatement stmt = connection.prepareStatement(sql);
         if (stmt instanceof StatementImpl) {
             ((StatementImpl) stmt).getLocalSettings().setFormat(ClickHouseFormat.RowBinaryWithNamesAndTypes);
@@ -955,9 +965,9 @@ public static boolean isSystemTableEngine(String engine) {
         return engine != null && (engine.startsWith("System") || engine.startsWith("Async"));
     }
 
-    private static final String TABLE_TYPE_COL_IN_GET_TABLES = "TABLE_TYPE";
+    static final String TABLE_TYPE_COL_IN_GET_TABLES = "TABLE_TYPE";
 
-    private static final Consumer<Map<String, Object>> TABLE_TYPE_MUTATOR = row -> {
+    static final Consumer<Map<String, Object>> TABLE_TYPE_MUTATOR = row -> {
         String engine = (String) row.get(TABLE_TYPE_COL_IN_GET_TABLES);
 
         String tableType;
@@ -987,8 +997,16 @@ public ResultSet getTables(String catalog, String schemaPattern, String tableNam
         // TODO: when switch between catalog and schema is implemented, then TABLE_SCHEMA and TABLE_CAT should be populated accordingly
         // TODO: handle useCatalogs == true and return schema catalog name
 
-        // Get engines that map to the requested table types
         Set<String> requestedTypes = (types == null || types.length == 0) ? TABLE_TYPES : Arrays.stream(types).collect(Collectors.toSet())  ;
+        if (useShowStatements) {
+            return showStatementsMetaData.getTables(schemaPattern, tableNamePattern, requestedTypes);
+        } else {
+            return getTablesImpl(schemaPattern, tableNamePattern, requestedTypes);
+        }
+    }
+
+    private ResultSet getTablesImpl(String schemaPattern, String tableNamePattern, Set<String> requestedTypes) throws SQLException {
+        // Get engines that map to the requested table types
         Set<String> engines = getEnginesForTableTypes(requestedTypes);
 
         // Build engine filter conditions
@@ -1060,6 +1078,14 @@ public ResultSet getTables(String catalog, String schemaPattern, String tableNam
     @Override
     public ResultSet getSchemas() throws SQLException {
         // TODO: handle useCatalogs == true and return schema catalog name
+        if (useShowStatements) {
+            return showStatementsMetaData.getSchemas(null);
+        } else {
+            return getSchemasImpl();
+        }
+    }
+
+    private ResultSet getSchemasImpl() throws SQLException {
         try {
             return createStatement().executeQuery("SELECT name AS TABLE_SCHEM, " + catalogPlaceholder + " AS TABLE_CATALOG FROM system.databases ORDER BY name");
         } catch (Exception e) {
@@ -1102,9 +1128,17 @@ public ResultSet getTableTypes() throws SQLException {
     }
 
     @Override
-    @SuppressWarnings({"squid:S2095", "squid:S2077"})
     public ResultSet getColumns(String catalog, String schemaPattern, String tableNamePattern, String columnNamePattern) throws SQLException {
         // TODO: handle useCatalogs == true and return schema catalog name
+        if (useShowStatements) {
+            return showStatementsMetaData.getColumns(schemaPattern, tableNamePattern, columnNamePattern);
+        } else {
+            return getColumnsImpl(schemaPattern, tableNamePattern, columnNamePattern);
+        }
+    }
+
+    @SuppressWarnings({"squid:S2095", "squid:S2077"})
+    private ResultSet getColumnsImpl(String schemaPattern, String tableNamePattern, String columnNamePattern) throws SQLException {
         final String sql = "SELECT " +
                 catalogPlaceholder + " AS TABLE_CAT, " +
                 "database AS TABLE_SCHEM, " +
@@ -1156,7 +1190,7 @@ private static String generateSqlTypeSizes(String columnName) {
     }
 
 
-    private static final Consumer<Map<String, Object>> DATA_TYPE_VALUE_FUNCTION = row -> {
+    static final Consumer<Map<String, Object>> DATA_TYPE_VALUE_FUNCTION = row -> {
         String typeName = (String) row.get("TYPE_NAME");
         SQLType type = JdbcUtils.CLICKHOUSE_TYPE_NAME_TO_SQL_TYPE_MAP.get(typeName);
         if (type == null) {
@@ -1814,6 +1848,14 @@ public RowIdLifetime getRowIdLifetime() throws SQLException {
     @Override
     public ResultSet getSchemas(String catalog, String schemaPattern) throws SQLException {
         // TODO: handle useCatalogs == true and return schema catalog name
+        if (useShowStatements) {
+            return showStatementsMetaData.getSchemas(schemaPattern);
+        } else {
+            return getSchemasImpl(schemaPattern);
+        }
+    }
+
+    private ResultSet getSchemasImpl(String schemaPattern) throws SQLException {
         String sql = "SELECT name AS TABLE_SCHEM, " + catalogPlaceholder + " AS TABLE_CATALOG FROM system.databases " +
                 "WHERE name LIKE ?";
         try (PreparedStatement stmt = prepareStatement(sql)) {