Understand the lifecycle of a cluster

Whenever a cluster starts running a workload, it moves through a sequence of stages before its results are fully up to date. Knowing which stage a cluster is in tells you whether it is making progress or is stuck.

A cluster enters this sequence when you create it, resize it, or raise its replication factor, and whenever a replica restarts, including during Materialize Cloud’s routine maintenance and after an out-of-memory event.

Stages

No single column reports these stages. A replica’s status is only online or offline, so it answers the provisioning question and nothing else. Hydration and lag are separate signals, and you need both: a hydrated object has processed the snapshot of its inputs, but not the updates that arrived while it was doing so, and a low lag on an unhydrated object does not mean its results are available.

Two states interrupt the sequence: a cluster with no replicas never leaves provisioning, and a replica that goes offline restarts and re-enters the sequence from the top.

The queries below monitor this cluster. Substitute your own object names.

CREATE CLUSTER lifecycle_demo SIZE '25cc';

CREATE SOURCE auction_load IN CLUSTER lifecycle_demo
  FROM LOAD GENERATOR AUCTION (TICK INTERVAL '1s') FOR ALL TABLES;

CREATE MATERIALIZED VIEW bids_by_auction IN CLUSTER lifecycle_demo AS
  SELECT auction_id, count(*) AS bids, max(amount) AS max_bid
  FROM bids
  GROUP BY auction_id;

CREATE INDEX bids_by_auction_idx IN CLUSTER lifecycle_demo
  ON bids_by_auction (auction_id);

Provisioning

Replicas are scheduled and brought online. To monitor progress, check that each replica reports online in mz_cluster_replica_statuses:

SELECT c.name AS cluster, r.name AS replica, r.size, st.status, st.reason
FROM mz_internal.mz_cluster_replica_statuses st
JOIN mz_catalog.mz_cluster_replicas r ON r.id = st.replica_id
JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id
WHERE c.name = 'lifecycle_demo'
ORDER BY r.name;

    cluster     | replica | size | status | reason
----------------+---------+------+--------+--------
 lifecycle_demo | r1      | 25cc | online |
(1 row)

status is online or offline. reason is NULL while a replica is online, and otherwise reports initializing or oom-killed.

A resize passes through this stage without downtime: Materialize provisions replicas at the target size and hydrates them before retiring the old ones, so this query reports replicas at both sizes until the cutover. The new replicas are new objects with new IDs and names, so a r1 that becomes r2 is a resize, not a restart. See Monitoring a resize.

No compute

A cluster with a replication factor of 0 has no replicas, so it never leaves provisioning. The query above returns no rows at all rather than an offline status, which is worth knowing if you alert on it. Queries routed to the cluster fail immediately:

ERROR:  CLUSTER "lifecycle_demo" has no replicas available to service request
HINT:  Use ALTER CLUSTER to adjust the replication factor of the cluster.

A query run from a different cluster against an object maintained by this one does not error. It blocks until the object’s frontier advances, which it never will.

To find clusters in this state:

SELECT name, replication_factor
FROM mz_catalog.mz_clusters
WHERE replication_factor = 0;

Scaling a cluster to zero between scheduled runs saves compute, but it is not free: its objects hold back compaction of their inputs, so the cluster faces more work, and a higher memory peak, when it comes back. Prefer dropping and recreating such clusters over parking them at zero for long stretches.

Hydrating

Each replica reconstructs its in-memory state by reading from Materialize’s storage layer (see hydration). mz_hydration_statuses reports the flag for every object a cluster maintains, sources and sinks included:

SELECT o.name AS object, o.type, h.hydrated
FROM mz_internal.mz_hydration_statuses h
JOIN mz_objects o ON o.id = h.object_id
JOIN mz_catalog.mz_cluster_replicas r ON r.id = h.replica_id
JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id
WHERE c.name = 'lifecycle_demo' AND o.id LIKE 'u%'
ORDER BY o.name;

       object        |       type        | hydrated
---------------------+-------------------+----------
 accounts            | source            | t
 auction_load        | source            | t
 auctions            | source            | t
 bids                | source            | t
 bids_by_auction     | materialized-view | t
 bids_by_auction_idx | index             | t
 organizations       | source            | t
 users               | source            | t
(8 rows)

The o.id LIKE 'u%' filter excludes the built-in introspection indexes that every replica carries. The join to mz_cluster_replicas is load-bearing: mz_hydration_statuses keeps rows for replicas that no longer exist, and without the join a retired replica’s stale hydrated values contradict the live ones.

Sources hydrate more slowly than indexes and materialized views, often by a minute or more on a fresh replica, so an all-object table like the one above reads mixed for a while. To narrow the view to indexes and materialized views, and to see how long each took, mz_compute_hydration_statuses adds a hydration_time column:

SELECT o.name AS object, o.type, r.name AS replica, ch.hydrated, ch.hydration_time
FROM mz_internal.mz_compute_hydration_statuses ch
JOIN mz_objects o ON o.id = ch.object_id
JOIN mz_catalog.mz_cluster_replicas r ON r.id = ch.replica_id
JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id
WHERE c.name = 'lifecycle_demo' AND o.id LIKE 'u%'
ORDER BY o.name;

       object        |       type        | replica | hydrated | hydration_time
---------------------+-------------------+---------+----------+-----------------
 bids_by_auction     | materialized-view | r1      | t        | 00:00:00.00011
 bids_by_auction_idx | index             | r1      | t        | 00:00:00.000019
(2 rows)

Judge a replica by its objects collectively rather than one at a time. An object that finishes early reports a freshness number while its neighbours are still hydrating and competing for CPU, so that number looks bad and alerting on it produces false positives. Treat a replica as ready only once every object on it is hydrated:

SELECT r.name AS replica, bool_and(h.hydrated) AS replica_hydrated
FROM mz_internal.mz_hydration_statuses h
JOIN mz_objects o ON o.id = h.object_id
JOIN mz_catalog.mz_cluster_replicas r ON r.id = h.replica_id
JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id
WHERE c.name = 'lifecycle_demo' AND o.id LIKE 'u%'
GROUP BY r.name;

A cluster with no replicas returns no rows here, not false.

Hydration is per replica, so adding a replica or resizing a cluster hydrates only the new replicas while the existing ones keep serving. It is also the memory peak of a cluster’s life, and that peak is higher than steady-state usage, so a cluster that runs comfortably for weeks can still fail to come back after a restart. Size clusters against the peak rather than the steady state, and consider autoscaling to a larger size for the duration.

Catching up

Once hydrated, the cluster processes the backlog of updates that accumulated while it was unavailable, so its lag starts high and comes down. mz_wallclock_global_lag_recent_history records how far each object trails wallclock time, binned by minute over the last 24 hours:

SELECT o.name AS object, w.occurred_at, w.lag
FROM mz_internal.mz_wallclock_global_lag_recent_history w
JOIN mz_objects o ON o.id = w.object_id
WHERE o.name = 'bids_by_auction'
ORDER BY w.occurred_at DESC
LIMIT 8;

     object      |      occurred_at       |   lag
-----------------+------------------------+----------
 bids_by_auction | 2026-09-15 17:33:00+00 | 00:00:01
 bids_by_auction | 2026-09-15 17:32:00+00 | 00:00:01
 bids_by_auction | 2026-09-15 17:31:00+00 | 00:00:01
 bids_by_auction | 2026-09-15 17:30:00+00 | 00:00:01
 bids_by_auction | 2026-09-15 17:29:00+00 | 00:00:01
 bids_by_auction | 2026-09-15 17:28:00+00 | 00:01:42
 bids_by_auction | 2026-09-15 17:27:00+00 | 00:00:42
 bids_by_auction | 2026-09-15 17:26:00+00 | 00:00:01
(8 rows)

The cluster lost its replica just after 17:26. Lag climbs to 42 seconds, then to 1 minute 42 seconds, then drops back to a second once the replacement replica hydrates and works through the backlog. That shape, a climb followed by a return to baseline, is what catching up looks like.

Steady state

The cluster has caught up and its lag holds low and roughly constant, typically a few seconds, as in the rows from 17:29 onward above. A lag that instead climbs steadily, at about one minute per minute, means the cluster has stopped making progress.

To attribute lag to a specific input, mz_materialization_lag reports each object’s distance from its direct inputs and from the sources and tables at the root of its dependency graph:

SELECT o.name AS object, l.local_lag, l.global_lag,
       si.name AS slowest_local_input, sg.name AS slowest_global_input
FROM mz_internal.mz_materialization_lag l
JOIN mz_objects o ON o.id = l.object_id
JOIN mz_objects si ON si.id = l.slowest_local_input_id
JOIN mz_objects sg ON sg.id = l.slowest_global_input_id
WHERE o.name IN ('bids_by_auction', 'bids_by_auction_idx')
ORDER BY o.name;

       object        | local_lag | global_lag | slowest_local_input | slowest_global_input
---------------------+-----------+------------+---------------------+----------------------
 bids_by_auction     | 00:00:00  | 00:00:00   | bids                | bids
 bids_by_auction_idx | 00:00:00  | 00:00:00   | bids_by_auction     | bids
(2 rows)

The index trails its direct input, the materialized view, and the view trails the bids source, which is the root input for both.

See Troubleshooting freshness to diagnose a cluster that is not progressing, and Monitor freshness to track lag over time.

Offline replicas

A replica that becomes unavailable reports offline, then restarts and re-enters the lifecycle at provisioning, rehydrating everything it hosts. The cluster’s lag grows until it catches up again.

A point-in-time status misses a replica that died and recovered between two readings, so read mz_cluster_replica_status_history rather than polling the current status:

SELECT h.replica_id, rh.replica_name, h.status, h.reason, h.occurred_at
FROM mz_internal.mz_cluster_replica_status_history h
JOIN mz_internal.mz_cluster_replica_history rh ON rh.replica_id = h.replica_id
WHERE rh.cluster_name = 'lifecycle_demo'
ORDER BY h.occurred_at DESC
LIMIT 20;

Resolve replica names through mz_cluster_replica_history, not mz_cluster_replicas. A replica that was retired or replaced is gone from mz_cluster_replicas, so joining against it drops exactly the events a history query is for.

Reading the history:

• A single offline followed by online on the same replica_id is a restart, and is routine.
• The same pair on a new replica_id is a replica being created. A resize or a blue/green swap changes which replicas back a cluster, so key on replica_id, not on the cluster or replica name.
• A repeating offline with reason oom-killed is a crash loop: the replica is too small for its workload, and each restart triggers a rehydration that exhausts memory again. See Check for OOM crash loops.
• reason is best-effort and is sometimes empty even for an out-of-memory kill, so treat repeated restarts as the signal rather than the reason string.

Related pages

• Clusters
• Hydration
• Autoscaling
• Troubleshooting freshness
• Understand the lifecycle of a source