From cd780c2eb023cb518914de525d626df302a58580 Mon Sep 17 00:00:00 2001 From: "Diego Nieto (lesandie)" Date: Thu, 29 Oct 2026 09:40:58 +0100 Subject: [PATCH 1/6] Add KB article: unblock a stuck DDL queue via system.zookeeper Covers allow_zookeeper_write, writable columns, the znode layout of a distributed DDL task (finished/, shards//executed), name escaping, and two recipes (host-level finished, shard-level executed) with verification and safety notes. Tested on 26.10 single-node. Co-Authored-By: Claude Sonnet 5.5 --- ...y-kb-unblock-ddl-queue-system-zookeeper.md | 146 ++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md diff --git a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md new file mode 100644 index 0000000000..dd66ba2b45 --- /dev/null +++ b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md @@ -0,0 +1,146 @@ +--- +title: "Unblocking a stuck DDL queue by writing to system.zookeeper" +linkTitle: "Unblock the DDL queue with system.zookeeper" +description: > + Mark a stuck ON CLUSTER task as finished with INSERT INTO system.zookeeper, without touching the queue by hand in Keeper. +keywords: + - distributed_ddl_queue + - allow_zookeeper_write + - ON CLUSTER stuck + - UNFINISHED +--- + +A distributed DDL task (`ON CLUSTER`) that can never finish blocks every task queued behind it on the affected host, because each host's `DDLWorker` runs the queue in order. The usual symptom is `Unexpected error (N times in a row) ... maximum retries exceeded (UNFINISHED)` in the server log and a growing list of `Inactive` rows in `system.distributed_ddl_queue`. + +If you have already established that the task's effect is applied (or is no longer wanted), the cleanest unblock is to write the flag the worker is waiting for. `system.zookeeper` is writable for exactly this. It is safer than `rmr` on the queue: you change one znode, and only on the entries you select. + +{{}} +flowchart TD + A["Oldest entry with status != Finished
(system.distributed_ddl_queue)"] --> B{"Is its effect applied
or no longer needed?"} + B -- "No" --> X["Fix the cause first
(this article does not apply)"] + B -- "Yes" --> C{"Any shards/<shard>/tries_to_execute > 3
and no shards/<shard>/executed?"} + C -- "Yes (replicated-table ALTER)" --> D["Recipe B: create shards/<shard>/executed
then Recipe A if rows stay Inactive"] + C -- "No" --> E["Recipe A: create finished/<host:port>
for each non-Finished host"] + D --> V["Verify in system.distributed_ddl_queue"] + E --> V +{{}} + +## Prerequisites + +- `allow_zookeeper_write` is a **server config** key, not a user setting. Without it, `INSERT` fails with `Prohibit writing to system.zookeeper, unless config allow_zookeeper_write as true` (`BAD_ARGUMENTS`). Put it in `config.d/`, and remove it when you are done: + +```xml + + true + +``` + +- The user needs the `INSERT` grant on `system.zookeeper`. +- Reading with `path LIKE ...` needs `SETTINGS allow_unrestricted_reads_from_keeper = 1`. An exact `path = '...'` filter does not. + +## What an INSERT does + +Only `name`, `value`, `path` and `zookeeperName` are writable; every other column is `MATERIALIZED` (`INSERT` returns `ILLEGAL_COLUMN`). + +```sql +SELECT name, type, default_kind FROM system.columns +WHERE database = 'system' AND table = 'zookeeper' ORDER BY position; +``` + +- Each row creates the znode `/`, or sets its value if it exists (no version check). Missing parent znodes are created too. A typo in `path` therefore creates bogus nodes instead of failing. +- All rows of one `INSERT` go to Keeper as one transaction per Keeper cluster. +- There is no delete. Removing a znode needs `clickhouse-keeper-client` or `zkCli`. +- `name` cannot contain `/`. + +## The znodes of a DDL task + +{{}} +graph LR + Q["/clickhouse/task_queue/ddl/query-0000000004"] --> ACT["active/"] + Q --> FIN["finished/"] + Q --> SH["shards/"] + FIN --> F1["localhost:9000 = 0 (status OK)"] + SH --> S1["localhost:9000 (shard node)"] + S1 --> T["tries_to_execute = 1"] + S1 --> EX["executed = localhost:9000"] +{{}} + +Path shown is the default `distributed_ddl.path`; check yours in `system.server_settings`. + +- `finished/:`: one child per host that completed the task. Value `0\n` means success (exception code 0). The name is the host name escaped with `escapeForFileName` (everything except `[A-Za-z0-9_]` becomes `%XX`: `-` is `%2D`, `.` is `%2E`). +- `shards//executed`: for `ALTER` on replicated tables only one replica per shard executes; the others wait for this node. `tries_to_execute` counts attempts by the executor. The value of `executed` is informational; only its existence matters. +- The `` node name is **not** escaped: it is the shard's replica names, sorted and joined with `,` (for example `host-a:9000,host-b:9000`). Its `executed` value **is** escaped. + +Because of this, copy names from `system.zookeeper` or from `system.distributed_ddl_queue` instead of typing them. + +## Recipe A: mark a task finished on the hosts that did not finish it + +Find the oldest non-finished entry: + +```sql +SELECT entry, host, port, status, exception_code, query +FROM system.distributed_ddl_queue +WHERE status != 'Finished' +ORDER BY entry, host LIMIT 20; +``` + +Create the `finished` znode for those hosts, taking the entry id from the view (never type a new `query-NNN` id): + +```sql +INSERT INTO system.zookeeper (name, path, value) +SELECT concat(replaceAll(replaceAll(host, '-', '%2D'), '.', '%2E'), ':', toString(port)), + concat('/clickhouse/task_queue/ddl/', entry, '/finished'), + '0\n' +FROM system.distributed_ddl_queue +WHERE entry = 'query-0000000004' AND status != 'Finished'; +``` + +The `replaceAll` pair is exact for DNS names and IPv4 addresses. For IPv6 literals, copy the real name from `finished/` of another entry. + +## Recipe B: set the shard `executed` flag + +Use it when the shard node has `tries_to_execute` above 3 and no `executed` child: + +```sql +SELECT name, value FROM system.zookeeper +WHERE path = '/clickhouse/task_queue/ddl/query-0000000004/shards/'; +``` + +```sql +INSERT INTO system.zookeeper (name, path, value) +SELECT 'executed', concat(path, '/', name), + replaceAll(replaceAll(splitByChar(',', name)[1], '-', '%2D'), '.', '%2E') +FROM system.zookeeper +WHERE path = '/clickhouse/task_queue/ddl/query-0000000004/shards' + AND name = ''; +``` + +The waiting replicas then treat the task as executed and finish it themselves. If a row stays `Inactive`, follow with Recipe A. + +## Verify + +```sql +SELECT host, status, exception_code FROM system.distributed_ddl_queue +WHERE entry = 'query-0000000004'; +``` + +Then count what is still blocked: `SELECT host, status, count() FROM system.distributed_ddl_queue WHERE status != 'Finished' GROUP BY host, status`. + +## Before you run it + +- **Finished is not executed.** Nothing runs the DDL. Check its effect first (for example `system.columns` or `system.data_skipping_indices` on every replica), and re-run the DDL by hand if it did not apply. +- **The backlog replays.** The next entries start executing in order as soon as the head is finished. Pause pipelines that issue DDL, and read the queued statements first. +- **A wrong name does nothing useful.** With an unescaped host (`ghost-host.example.com:9000`) the original row stays `Inactive` and the view shows an extra phantom host row. Remove such a node with `clickhouse-keeper-client`. +- **A malformed entry breaks reads.** Rewriting an entry's `value` (for example the `hosts:` list) with the wrong format makes `system.distributed_ddl_queue` fail with `CANNOT_PARSE_QUOTED_STRING`. +- Entries older than `task_max_lifetime` (default 7 days) or more than `max_tasks_in_queue` (default 1000) behind the newest one are purged by the cleanup. That also unblocks the head, but it silently drops the purged entries without running them. Do not wait for it. + +## Version notes + +- The recipes were tested on 26.10 with a single-node `default` cluster, which covers the znode layout, escaping and status transitions. +- The "retry the same task forever" behaviour comes from tasks whose retries are exhausted ending as `UNFINISHED`. It is fixed upstream in 26.8 ([PR 108505](https://github.com/ClickHouse/ClickHouse/pull/108505)), so 26.10 does not reproduce it. We have seen it on Altinity 26.3 builds. +- Recipe A has been used on a 3-replica 26.3 cluster and drained the queue. Recipe B follows the worker's source code and has been verified at the znode level only. + +## Related resources + +- [There are N unfinished hosts (0 of them are currently active)](/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/there-are-n-unfinished-hosts-0-of-them-are-currently-active/) +- [system.zookeeper](https://clickhouse.com/docs/operations/system-tables/zookeeper) From b73cce6bf29dbf7747fd0eb918f2cb20962e2188 Mon Sep 17 00:00:00 2001 From: "Diego Nieto (lesandie)" Date: Thu, 29 Oct 2026 09:47:33 +0100 Subject: [PATCH 2/6] Tighten wording in DDL queue unblock article Co-Authored-By: Claude Sonnet 5.5 --- ...nity-kb-unblock-ddl-queue-system-zookeeper.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md index dd66ba2b45..33469fcae6 100644 --- a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md +++ b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md @@ -10,9 +10,9 @@ keywords: - UNFINISHED --- -A distributed DDL task (`ON CLUSTER`) that can never finish blocks every task queued behind it on the affected host, because each host's `DDLWorker` runs the queue in order. The usual symptom is `Unexpected error (N times in a row) ... maximum retries exceeded (UNFINISHED)` in the server log and a growing list of `Inactive` rows in `system.distributed_ddl_queue`. +A distributed DDL task (`ON CLUSTER`) that never finishes blocks every task queued behind it on that host, because each host's `DDLWorker` runs the queue in order. You see `Unexpected error (N times in a row) ... maximum retries exceeded (UNFINISHED)` in the server log and a growing list of `Inactive` rows in `system.distributed_ddl_queue`. -If you have already established that the task's effect is applied (or is no longer wanted), the cleanest unblock is to write the flag the worker is waiting for. `system.zookeeper` is writable for exactly this. It is safer than `rmr` on the queue: you change one znode, and only on the entries you select. +If the task's effect is already applied (or you no longer want it), unblock the queue by writing the flag the worker waits for. `INSERT INTO system.zookeeper` does this. It is safer than `rmr` on the queue: you write one znode per host and leave the other entries alone. {{}} flowchart TD @@ -65,13 +65,13 @@ graph LR S1 --> EX["executed = localhost:9000"] {{}} -Path shown is the default `distributed_ddl.path`; check yours in `system.server_settings`. +The diagram uses the default `distributed_ddl.path`. Check yours in `system.server_settings`. - `finished/:`: one child per host that completed the task. Value `0\n` means success (exception code 0). The name is the host name escaped with `escapeForFileName` (everything except `[A-Za-z0-9_]` becomes `%XX`: `-` is `%2D`, `.` is `%2E`). - `shards//executed`: for `ALTER` on replicated tables only one replica per shard executes; the others wait for this node. `tries_to_execute` counts attempts by the executor. The value of `executed` is informational; only its existence matters. - The `` node name is **not** escaped: it is the shard's replica names, sorted and joined with `,` (for example `host-a:9000,host-b:9000`). Its `executed` value **is** escaped. -Because of this, copy names from `system.zookeeper` or from `system.distributed_ddl_queue` instead of typing them. +Copy names from `system.zookeeper` or `system.distributed_ddl_queue` instead of typing them. ## Recipe A: mark a task finished on the hosts that did not finish it @@ -130,14 +130,14 @@ Then count what is still blocked: `SELECT host, status, count() FROM system.dist - **Finished is not executed.** Nothing runs the DDL. Check its effect first (for example `system.columns` or `system.data_skipping_indices` on every replica), and re-run the DDL by hand if it did not apply. - **The backlog replays.** The next entries start executing in order as soon as the head is finished. Pause pipelines that issue DDL, and read the queued statements first. -- **A wrong name does nothing useful.** With an unescaped host (`ghost-host.example.com:9000`) the original row stays `Inactive` and the view shows an extra phantom host row. Remove such a node with `clickhouse-keeper-client`. -- **A malformed entry breaks reads.** Rewriting an entry's `value` (for example the `hosts:` list) with the wrong format makes `system.distributed_ddl_queue` fail with `CANNOT_PARSE_QUOTED_STRING`. -- Entries older than `task_max_lifetime` (default 7 days) or more than `max_tasks_in_queue` (default 1000) behind the newest one are purged by the cleanup. That also unblocks the head, but it silently drops the purged entries without running them. Do not wait for it. +- **A wrong name doesn't unblock the task.** With an unescaped host (`ghost-host.example.com:9000`) the original row stays `Inactive` and the view shows an extra phantom host row. Remove the stray node with `clickhouse-keeper-client`. +- **A malformed entry breaks reads.** If you rewrite an entry's `value` (for example the `hosts:` list) in the wrong format, `system.distributed_ddl_queue` fails with `CANNOT_PARSE_QUOTED_STRING`. +- **Don't wait for the cleanup.** It purges entries older than `task_max_lifetime` (default 7 days) or more than `max_tasks_in_queue` (default 1000) behind the newest one. That unblocks the head too, but it silently drops the purged entries without running them. ## Version notes - The recipes were tested on 26.10 with a single-node `default` cluster, which covers the znode layout, escaping and status transitions. -- The "retry the same task forever" behaviour comes from tasks whose retries are exhausted ending as `UNFINISHED`. It is fixed upstream in 26.8 ([PR 108505](https://github.com/ClickHouse/ClickHouse/pull/108505)), so 26.10 does not reproduce it. We have seen it on Altinity 26.3 builds. +- When a task exhausts its retries, it ends as `UNFINISHED` and the worker retries it forever. Upstream fixed this in 26.8 ([PR 108505](https://github.com/ClickHouse/ClickHouse/pull/108505)), so 26.10 does not reproduce it. We have seen it on Altinity 26.3 builds. - Recipe A has been used on a 3-replica 26.3 cluster and drained the queue. Recipe B follows the worker's source code and has been verified at the znode level only. ## Related resources From 4f3708ceac63ac3abffe2d64abf846064b8129cb Mon Sep 17 00:00:00 2001 From: "Diego Nieto (lesandie)" Date: Thu, 29 Oct 2026 09:52:15 +0100 Subject: [PATCH 3/6] DDL queue unblock article: add introduction version and no-restart note Co-Authored-By: Claude Sonnet 5.5 --- .../altinity-kb-unblock-ddl-queue-system-zookeeper.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md index 33469fcae6..8f512c7eb4 100644 --- a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md +++ b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md @@ -14,6 +14,8 @@ A distributed DDL task (`ON CLUSTER`) that never finishes blocks every task queu If the task's effect is already applied (or you no longer want it), unblock the queue by writing the flag the worker waits for. `INSERT INTO system.zookeeper` does this. It is safer than `rmr` on the queue: you write one znode per host and leave the other entries alone. +It also avoids a restart. `DDLWorker` keeps the queue position and its unfinished tasks in memory, so after you delete entries with `rmr` it never learns about it, and you normally have to restart ClickHouse. The `INSERT` writes the same `finished` and `executed` znodes the worker writes itself, and the worker checks for them each time it retries a task. The queue then moves on with no restart. A 26.3 cluster drained its queue this way without a pod restart. + {{}} flowchart TD A["Oldest entry with status != Finished
(system.distributed_ddl_queue)"] --> B{"Is its effect applied
or no longer needed?"} @@ -35,6 +37,7 @@ flowchart TD ``` +- `INSERT INTO system.zookeeper` and its `allow_zookeeper_write` gate arrived in 22.6 (upstream commits from 2022-05-27 and 2022-06-01) and were backported to the 22.3 LTS from 22.3.16. - The user needs the `INSERT` grant on `system.zookeeper`. - Reading with `path LIKE ...` needs `SETTINGS allow_unrestricted_reads_from_keeper = 1`. An exact `path = '...'` filter does not. From 3f41cfa0534c1c6ccd6805b5d76a06bb2f7d48f6 Mon Sep 17 00:00:00 2001 From: "Diego Nieto (lesandie)" Date: Thu, 29 Oct 2026 10:22:22 +0100 Subject: [PATCH 4/6] DDL queue unblock article: scope the rmr/restart note to what was tested Co-Authored-By: Claude Sonnet 5.5 --- .../altinity-kb-unblock-ddl-queue-system-zookeeper.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md index 8f512c7eb4..dcc0860011 100644 --- a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md +++ b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md @@ -14,7 +14,9 @@ A distributed DDL task (`ON CLUSTER`) that never finishes blocks every task queu If the task's effect is already applied (or you no longer want it), unblock the queue by writing the flag the worker waits for. `INSERT INTO system.zookeeper` does this. It is safer than `rmr` on the queue: you write one znode per host and leave the other entries alone. -It also avoids a restart. `DDLWorker` keeps the queue position and its unfinished tasks in memory, so after you delete entries with `rmr` it never learns about it, and you normally have to restart ClickHouse. The `INSERT` writes the same `finished` and `executed` znodes the worker writes itself, and the worker checks for them each time it retries a task. The queue then moves on with no restart. A 26.3 cluster drained its queue this way without a pod restart. +It also needs no restart. The `INSERT` writes the same `finished` and `executed` znodes the worker writes itself, and the worker checks for them each time it retries a task. A 26.3 cluster drained its queue this way without a pod restart. + +`DDLWorker` keeps its queue position and unfinished tasks in memory, so deleting a task it is still retrying may leave it out of sync with Keeper. On 26.10, `rmr` on a finished entry and on the whole queue path both worked without a restart. We did not test `rmr` on a task the worker is retrying, or on older builds. {{}} flowchart TD From 4172521f993373321ddc788a6aeba06c653df83d Mon Sep 17 00:00:00 2001 From: "Diego Nieto (lesandie)" Date: Thu, 29 Oct 2026 10:25:25 +0100 Subject: [PATCH 5/6] DDL queue unblock article: state default queue path and Keeper assumption Co-Authored-By: Claude Sonnet 5.5 --- .../altinity-kb-unblock-ddl-queue-system-zookeeper.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md index dcc0860011..1f525dffa5 100644 --- a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md +++ b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md @@ -40,6 +40,7 @@ flowchart TD ``` - `INSERT INTO system.zookeeper` and its `allow_zookeeper_write` gate arrived in 22.6 (upstream commits from 2022-05-27 and 2022-06-01) and were backported to the 22.3 LTS from 22.3.16. +- The recipes use the default queue path `/clickhouse/task_queue/ddl` and the default Keeper. If `distributed_ddl.path` differs (check `system.server_settings`), replace the path in every query. If your queue is on an auxiliary Keeper, also set the `zookeeperName` column and filter on it when you read. - The user needs the `INSERT` grant on `system.zookeeper`. - Reading with `path LIKE ...` needs `SETTINGS allow_unrestricted_reads_from_keeper = 1`. An exact `path = '...'` filter does not. From 303c2786dde0c1322e8efd44027e904c7c0f82f4 Mon Sep 17 00:00:00 2001 From: "Diego Nieto (lesandie)" Date: Thu, 29 Oct 2026 10:29:17 +0100 Subject: [PATCH 6/6] DDL queue unblock article: show how to confirm the finished/ name format Co-Authored-By: Claude Sonnet 5.5 --- .../altinity-kb-unblock-ddl-queue-system-zookeeper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md index 1f525dffa5..6665b7fe38 100644 --- a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md +++ b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-ddlworker/altinity-kb-unblock-ddl-queue-system-zookeeper.md @@ -101,7 +101,7 @@ FROM system.distributed_ddl_queue WHERE entry = 'query-0000000004' AND status != 'Finished'; ``` -The `replaceAll` pair is exact for DNS names and IPv4 addresses. For IPv6 literals, copy the real name from `finished/` of another entry. +The `replaceAll` pair is exact for DNS names and IPv4 addresses. To confirm the format on your cluster, list the children of `finished/` on any finished entry: they are escaped (`host%2Da%2Eexample%2Ecom:9000`), and your new names must match them. For IPv6 literals, copy the real name from there. ## Recipe B: set the shard `executed` flag