[client-v2, jdbc-v2] JSONEachRow: a query that fails mid-stream on ClickHouse 25.8 returns the {"exception": ...} row as data and raises no error #3220

Description

On ClickHouse 25.8, when a JSONEachRow query fails after the server has started the response, the server writes the error as one more JSON row, {"exception": "Code: 395. DB::Exception: ..."}, and ends the body normally. JSONEachRowFormatReader returns that row as an ordinary data row. The caller gets a truncated result plus one row in which every column is null, and no exception. The query failure is lost.

This affects client-v2 (JSONEachRowFormatReader) and jdbc-v2 (FORMAT JSONEachRow with jdbc_json_parser_factory), with both JacksonJsonParserFactory and GsonJsonParserFactory.

On 26.3 and later the same code throws, because those servers end the body with the tagged __exception__ frame and cut the connection:

25.8 is in the CI matrix and is the CH_VERSION of nightly.yml / release.yml.

Response compression changes when this occurs. client-v2 asks for compressed responses by default (compress=1), and the server then holds the start of the output in a buffer. If the whole failed response fits in that buffer, the server returns HTTP 500 and the client throws correctly. The bug occurs when the output before the error is larger than that buffer. In the repro below the output before the error is approximately 1.6 MB compressed (HTTP 200). With toString(number) in place of randomPrintableASCII(10) it is 881 KB compressed and the server returns HTTP 500. So small results are safe, and large streamed results are affected.

Steps to reproduce

1. Start clickhouse/clickhouse-server:25.8 (25.8.33.6). Use default server settings.
2. Run the query below with format JSONEachRow, through client-v2 JSONEachRowFormatReader or through jdbc-v2 with FORMAT JSONEachRow.
3. Read all rows. No exception occurs. The last row is the server error, returned as data.

Server side, with no client involved (curl, default settings, HTTP 200, curl exit code 0). The last two lines of the body:

{"number":94999,"s":"_D:]Rf&$TL","t":0}
{"exception": "Code: 395. DB::Exception: Value passed to 'throwIf' function is non-zero: while executing 'FUNCTION throwIf(equals(__table1.number, 100000_UInt32) :: 1) -> throwIf(equals(__table1.number, 100000_UInt32)) UInt8 : 0'. (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 25.8.33.6 (official build))"}

The server writes this row because http_write_exception_in_output_format defaults to 1 ("Write exception in output format to produce valid output. Works with JSON and XML formats."). The query failed: system.query_log has type = ExceptionWhileProcessing, exception_code = 395.

Error Log or Exception StackTrace

No exception is thrown. That is the problem.

25.8.33.6, client-v2, JacksonJsonParserFactory:
  rows returned = 100001, no exception
  last row      = {exception=Code: 395. DB::Exception: Value passed to 'throwIf' function is non-zero: ...}
  on last row   : hasValue("number") = false, getString("s") = null,
                  getLong("number") throws NullValueException: Column number has null value and it cannot be cast to long
25.8.33.6, client-v2, GsonJsonParserFactory: same (last row = {exception=Code: 395. ...}, no exception)

25.8.33.6, jdbc-v2, Jackson and Gson:
  rs.next() returned true 100001 times, no SQLException
  last row: getLong("number") = 0, wasNull() = true, getString("s") = null

26.3.28.5 and 26.9.1.1629, same code:
  client-v2: java.lang.RuntimeException: Failed to read next JSON row
  jdbc-v2:   java.sql.SQLException: Failed to read next JSON row

Expected Behaviour

The read fails with the server error (ServerException in client-v2, SQLException in jdbc-v2), as it does on 26.3+ and for RowBinaryWithNamesAndTypes on 25.8 (on 25.8 the binary reader throws ClientException: Failed to read value for column s on the same query). The error row is never returned as data.

Code Example

String sql = "SELECT number, randomPrintableASCII(10) AS s, throwIf(number = 100000) AS t"
        + " FROM numbers(200000) SETTINGS max_block_size = 1000";

// client-v2
try (QueryResponse r = client.query(sql, new QuerySettings().setFormat(ClickHouseFormat.JSONEachRow)).get();
     ClickHouseTextFormatReader reader = new JSONEachRowFormatReader(
             new JacksonJsonParserFactory().createJsonParser(r.getInputStream()))) {
    Map<String, Object> row, last = null;
    long n = 0;
    while ((row = reader.next()) != null) { n++; last = row; }
    System.out.println(n + " rows, last = " + last); // 100001 rows, last = {exception=Code: 395. ...}
}

// jdbc-v2, property jdbc_json_parser_factory = com.clickhouse.client.api.data_formats.JacksonJsonParserFactory
try (Statement st = conn.createStatement();
     ResultSet rs = st.executeQuery(sql + " FORMAT JSONEachRow")) {
    while (rs.next()) {
        // last iteration: rs.getLong("number") == 0, rs.wasNull() == true
    }
} // no SQLException

Root cause

• client-v2/src/main/java/com/clickhouse/client/api/data_formats/JSONEachRowFormatReader.java:92-125 (next() / readNextRow()) returns every object that the JsonParser produces. Nothing checks for the server's error object.
• JacksonJsonParserFactory.java:35-44 and GsonJsonParserFactory.java:101-112 parse the error object as a normal Map, because it is valid JSON.
• JSONEachRowFormatReader.java:44-57 infers the schema from the first row. The error row has none of the schema's keys, so readValue / getString return null and primitive getters throw NullValueException. In jdbc-v2 the getters return 0 / null with wasNull() == true.
• jdbc-v2 uses the same reader: jdbc-v2/src/main/java/com/clickhouse/jdbc/StatementImpl.java:323.

Open PR #3097 (issue #2702) handles the tagged __exception__ frame of newer servers. It does not change JSONEachRowFormatReader, so it does not fix this case.

Suggested fix

• In JSONEachRowFormatReader, when a parsed row has exactly one key, exception, with a string value, and the inferred schema is not exactly one column named exception, treat it as a server error. Raise a ServerException (code parsed from the Code: NNN. prefix) through the existing deferred-error path in next(). Then the rows before it are still delivered, and the error surfaces on the next call. jdbc-v2 then reports it as SQLException, as on 26.3+.
• Contrast cases that must keep their current behavior:
  • a real result column named exception: SELECT 'boom' AS exception FORMAT JSONEachRow returns {"exception":"boom"}, which has the same shape after parsing;
  • a schema that has an exception column among other columns (every row then has all keys).
• Alternative considered: send http_write_exception_in_output_format=0 on JSONEachRow requests. On 25.8 this makes the server cut the body (verified: curl exit 18), and the reader already throws on that. But it adds a setting to the request, which a readonly=1 profile can reject, and jdbc-v2 selects the format from the SQL FORMAT clause, so it does not know the format when it sends the request. The jdbc-v2 test ResultSetImplTest.testCloseNotFullyReadResultSet in PR [client-v2, jdbc-v2] Do not fail closing a partly read response that the server ended early #3217 puts this setting in its query to avoid this behavior on 25.8.

Configuration

Client Configuration

new Client.Builder()
        .addEndpoint("http://localhost:8123")
        .setUsername("default")
        .setPassword("...")
        .build(); // all other options default (response compression on)

Environment

• Cloud
• Client version: main @ 5392b64 (0.12.0-rc1-SNAPSHOT), client-v2 and jdbc-v2
• Language version: Java 17.0.20.1
• OS: Linux (Docker)

ClickHouse Server

• ClickHouse Server version: 25.8.33.6 (affected); 26.3.28.5, 26.8.1.2041, 26.9.1.1629 (not affected, see table)
• ClickHouse Server non-default settings, if any: none
• CREATE TABLE statements for tables involved: none (numbers())
• Sample data for all these tables: none

Found by automated analysis (Polyglot AI) while working on #3203. Verified against live servers (versions above) with client-v2 and jdbc-v2 built from main, not by code inspection only.