# Clusters Guidance for configuring and operating Materialize clusters. Clusters provide the compute resources for running dataflows in Materialize. - Learn about [clusters](/fundamentals/concepts/clusters/). - Follow the [operational guidelines](/clusters/operational-guidelines/). - Speed up hydration with [autoscaling](/clusters/autoscaling/). - Understand [system clusters](/clusters/system-clusters/). - Troubleshoot a [spike in cluster CPU](/clusters/cpu-troubleshooting/). --- ## Autoscaling for hydration > **Public Preview:** This feature is in public preview. When you create an index, materialized view, or Kafka upsert source, or when a cluster restarts, the cluster must [hydrate](/fundamentals/concepts/hydration/) the affected objects before they can serve results. Hydration reads the input data and rebuilds in-memory state, and its speed scales with the cluster [size](/sql/create-cluster/#available-sizes). The `AUTO SCALING STRATEGY (ON HYDRATION)` option lets a cluster **automatically provision an extra burst replica at the configured `HYDRATION SIZE` while it has un-hydrated objects**. This speeds up hydration without manually scaling the cluster up before hydration and back down afterward. The steady-size replicas continue hydrating in parallel, and once one of them catches up with the burst, the burst replica lingers for the `LINGER DURATION` and is then removed. The burst replica is an ordinary cluster replica, billed only for the time it is provisioned. See [Usage & billing](/materialize-cloud/billing/) for details. `AUTO SCALING STRATEGY (ON HYDRATION)` is particularly useful for [blue/green deployments](/manage/blue-green/), where a new cluster must hydrate before the cutover. It is only available on **managed clusters**, and cannot be combined with a cluster `SCHEDULE` other than the default `MANUAL`. For example, the following cluster can provision a burst replica of size `800cc`: ```mzsql CREATE CLUSTER fast_start ( SIZE = '100cc', AUTO SCALING STRATEGY = ( ON HYDRATION ( HYDRATION SIZE = '800cc', LINGER DURATION = '15s' ) ) ); ``` You can specify the following options: Option | Description -------|------------ `HYDRATION SIZE` | The [size](/sql/create-cluster/#available-sizes) of the burst replica provisioned while the cluster has un-hydrated objects. Must differ from the cluster's steady `SIZE`. Choose a larger size to speed up hydration. `LINGER DURATION` | Optional. How long the burst replica lingers after a steady-size replica catches up, before it is removed. Default: `0s`. Provisioning the burst replica requires enough compute capacity to run it. In Materialize Self-Managed, this means your Kubernetes cluster must have enough spare resources (for example, available nodes) to schedule the burst replica. The burst is best-effort and never blocks the cluster: if the burst replica cannot be provisioned, the steady-size replicas still come up and hydrate as usual, as long as there are enough resources for them. To remove the autoscaling strategy from a cluster, use `ALTER CLUSTER ... RESET (AUTO SCALING STRATEGY)` or set an empty strategy with `AUTO SCALING STRATEGY = ()`. You can inspect the configured strategy and any in-flight burst in the [`mz_internal.mz_cluster_auto_scaling_strategies`](/sql/system-catalog/mz_internal/#mz_cluster_auto_scaling_strategies) catalog view. ## Configure autoscaling on an existing cluster You can also add, change, or remove the strategy after the cluster already exists: ```mzsql ALTER CLUSTER fast_start SET ( AUTO SCALING STRATEGY = ( ON HYDRATION (HYDRATION SIZE = '800cc', LINGER DURATION = '15s') ) ); ``` ```mzsql ALTER CLUSTER fast_start RESET (AUTO SCALING STRATEGY); ``` For the full option reference, see the `AUTO SCALING STRATEGY` option on [`CREATE CLUSTER`](/sql/create-cluster/#autoscaling) and [`ALTER CLUSTER`](/sql/alter-cluster/#speed-up-hydration-by-autoscaling-to-a-larger-size). ## Monitor an autoscaling event To check whether a burst is currently running and what it's configured to do, query [`mz_internal.mz_cluster_auto_scaling_strategies`](/sql/system-catalog/mz_internal/#mz_cluster_auto_scaling_strategies): ```mzsql SELECT c.name AS cluster, s.strategy->'on_hydration'->>'hydration_size' AS hydration_size, s.state->'burst'->>'burst_size' AS inflight_burst_size FROM mz_internal.mz_cluster_auto_scaling_strategies AS s JOIN mz_clusters AS c ON c.id = s.cluster_id; ``` `inflight_burst_size` is `NULL` when no burst is running, and reports the burst replica's size while one is up. [`SHOW CLUSTERS`](/sql/show-clusters/) also summarizes an in-flight burst in its `activity` column, and lists the burst replica itself, at its larger size, in `replicas`: ```nofmt name | replicas | activity -------------+------------------------+----------------------------- fast_start | r1 (100cc), r2 (800cc) | hydration burst at 800cc ``` To check whether the objects driving the burst have finished hydrating on the steady-size replicas, query [`mz_internal.mz_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_hydration_statuses) for the cluster's steady-size replica IDs. Once every object is hydrated on a steady-size replica, the burst replica lingers for the configured `LINGER DURATION` and is then removed. The [audit log](/sql/system-catalog/mz_catalog/#mz_audit_events) records when a burst replica is created and dropped, so you can look back at past autoscaling events after the fact. ## If the steady-size replica never hydrates The burst replica has no maximum lifetime: if a steady-size replica never catches up (for example, it's undersized and runs out of memory during hydration), the burst replica keeps serving indefinitely at the configured `HYDRATION SIZE`, billed for as long as it runs. This is deliberate: the burst replica may be the only thing keeping the cluster able to serve results, so Materialize does not tear it down just because it has been up a while. If you find yourself in this situation, [resize the cluster](/sql/alter-cluster/#resizing) to a `SIZE` that can hydrate the objects at steady state. The burst replica keeps serving throughout the resize, so there's no gap in availability while you find the right size. ## If capacity is unavailable Provisioning the burst replica requires enough compute capacity to run it. In **Materialize Self-Managed**, this means your Kubernetes cluster needs enough spare resources (for example, available nodes) to schedule the burst replica's pods. If it doesn't, Materialize keeps retrying automatically; no action is required on your part, though the burst can't speed up hydration until capacity frees up. Either way, the burst is best-effort and never blocks the cluster: the steady-size replicas come up and hydrate as usual, independently of whether the burst replica could be provisioned, as long as there are enough resources for them. If you plan to lean on autoscaling for hydration in a capacity- constrained Self-Managed deployment, make sure your node pools have enough spare capacity for the burst replicas you expect to run; see [Resize node pools](/self-managed-deployments/deployment-guidelines/resize-node-pools/). ## Limitations - Autoscaling for hydration only speeds up the initial hydration of indexes, materialized views, and Kafka upsert sources. It does not help with ongoing freshness or staleness, and it has no effect on [single-replica sources](/fundamentals/concepts/hydration/#objects-and-hydration) (PostgreSQL, MySQL, and SQL Server sources), which stay pinned to one replica regardless of the strategy. - `AUTO SCALING STRATEGY` cannot be combined with a `SCHEDULE` other than the default `MANUAL`, and is only available on managed clusters. - Autoscaling for hydration does not apply to the new generation during a Materialize Self-Managed version upgrade. Until it's promoted, the new generation runs read-only and can't provision a burst replica, so it hydrates only at its configured steady `SIZE`. See [Rollout strategies](/self-managed-deployments/upgrading/#rollout-strategies). ## Related pages - [Hydration](/fundamentals/concepts/hydration/) - [`CREATE CLUSTER`](/sql/create-cluster/#autoscaling) - [`ALTER CLUSTER`](/sql/alter-cluster/#speed-up-hydration-by-autoscaling-to-a-larger-size) - [`mz_internal.mz_cluster_auto_scaling_strategies`](/sql/system-catalog/mz_internal/#mz_cluster_auto_scaling_strategies) - [Blue/green deployments](/developer-tools/dbt/blue-green-deployments/) --- ## Cluster CPU troubleshooting A cluster replica's CPU goes to the dataflows that maintain the cluster's indexes, materialized views, and sinks, and to the ad-hoc queries served from it. A spike means one of those started doing more work, or that the same work stopped being spread evenly across the replica's workers. ## Common causes | Cause | Check | | ----- | ----- | | **Worker skew**: one worker does far more work than its peers, so the replica saturates a core while the rest idle. | [Check for worker skew](#check-for-worker-skew) | | **An expensive object**: one dataflow dominates the cluster's CPU. | [Find the objects consuming CPU](#find-the-objects-consuming-cpu) | | **Hydration**: a replica restart, cluster resize, or DDL forced objects to rebuild from their inputs. | [Check for recent hydration](#check-for-recent-hydration) | | **Ad-hoc query load**: `SELECT`s served by the cluster compete with its maintenance work. | [Check ad-hoc query load](#check-ad-hoc-query-load) | | **Upstream volume**: more data is arriving from sources, so there is more incremental work to do. | [Check upstream data volume](#check-upstream-data-volume) | | **Memory pressure**: a replica that is paging to disk burns CPU on I/O rather than on your dataflows. | [Rule out memory pressure](#rule-out-memory-pressure) | | **An undersized cluster**: the workload is spread evenly and nothing has changed; there is simply not enough compute. | [Rule out general cluster overload](#rule-out-general-cluster-overload) | ## Confirm the spike window and scope Before diagnosing a cause, establish when the spike started and which replicas it affected, replacing `` with the name of your cluster: ```mzsql SELECT u.occurred_at, r.name AS replica_name, u.cpu_percent FROM mz_internal.mz_cluster_replica_utilization_history u JOIN mz_catalog.mz_cluster_replicas r ON u.replica_id = r.id JOIN mz_catalog.mz_clusters c ON r.cluster_id = c.id WHERE c.name = '' AND u.occurred_at > now() - INTERVAL '6 hours' ORDER BY u.occurred_at DESC; ``` Every replica of a cluster maintains the same objects, so: - If **all replicas** spike together, the workload itself changed. Continue with the checks below. - If **one replica** spikes, that replica is doing something its peers are not, most often rehydrating after a restart. See [Check for recent hydration](#check-for-recent-hydration). - If the spike **repeats on a fixed cadence**, or CPU stays high while the cluster's inputs are idle, suspect a [temporal filter](/transform-data/optimization/#improve-performance-when-using-temporal-filters). A window whose boundary moves with `mz_now()` retracts rows as they age out, so it generates work with no upstream input at all. > **Note:** `cpu_percent` is a percentage of the replica's *total* allocation across all of > its cores, so it averages over workers. A replica whose work is concentrated on > one worker can peg a core while reporting a modest `cpu_percent`. Degraded > freshness with unremarkable CPU is a strong signal of [worker > skew](#check-for-worker-skew), not a reason to stop looking. ## Check for worker skew Materialize distributes work across a replica's workers by hashing keys, such as join keys and `GROUP BY` keys. If one key value accounts for a disproportionate share of rows, the worker responsible for that value does disproportionate work while its peers idle. To check for skew across an entire cluster, connect to it and run [`EXPLAIN ANALYZE CLUSTER CPU WITH SKEW`](/sql/explain-analyze/#explain-analyze--with-skew): ```mzsql SET CLUSTER TO ; EXPLAIN ANALYZE CLUSTER CPU WITH SKEW; ``` > **Note:** `EXPLAIN ANALYZE` and the `mz_introspection` relations read logging data that > each replica collects about itself. On a cluster with more than one replica, a > query against them fails unless you also target a replica with `SET > cluster_replica = ;`. `RESET cluster_replica` clears the > targeting, which otherwise applies to every subsequent query in the session. The output reports each dataflow's CPU time per worker against the average across workers, alongside the `global_id` of the underlying index or materialized view. A ratio near `1` means a worker is doing a roughly average share of the work; a ratio far above `1` on one worker points to skew in that object. > **Important:** `max_operator_cpu_ratio` is the maximum across all of a dataflow's operators, so > a high ratio can come from an operator that contributes almost nothing to the > dataflow's CPU time. Only act on a row whose `total_elapsed` is also > significant. Two further caveats: - The numbers accumulate from the moment each dataflow was created, so a short burst of skew in a long-running dataflow is diluted by its history. - On a cluster with many objects the output can run to thousands of rows. Use [`EXPLAIN ANALYZE ... AS SQL`](/sql/explain-analyze/#explain-analyze--as-sql) to get the underlying query, then filter and sort it yourself. ### Localize the skew to an operator Once you've identified a skewed object by its `global_id`, drill into it to find the specific operator responsible: ```mzsql EXPLAIN ANALYZE CPU WITH SKEW FOR MATERIALIZED VIEW ; ``` (Use `FOR INDEX ` for an index.) The operator with the highest ratio, most often a `Join` or `Reduce`, is where the skew originates. ### Find the hot key A skewed `Join` or `Reduce` operator is almost always caused by a **hot key**: one value in the join or `GROUP BY` column(s) accounts for far more rows than the rest. Confirm it by counting rows per value on the suspect column: ```mzsql SELECT , count(*) AS num_rows FROM GROUP BY ORDER BY num_rows DESC LIMIT 20; ``` A single value with an outsized count relative to the rest confirms the hot key. A common trigger is an upstream change that collapses a once-diverse column to a single value or to `NULL`. See [Is work distributed equally across workers?](/transform-data/dataflow-troubleshooting/#is-work-distributed-equally-across-workers) for other causes of skew, such as cross joins and `ORDER BY`/`LIMIT`/`OFFSET` queries. To resolve skew, restructure the query so the hot key's rows aren't concentrated on a single worker, for example by pre-aggregating or filtering rows before the join. If the skew can't be eliminated, size the cluster for the busiest worker's load rather than the average, since the other workers won't absorb it. ## Find the objects consuming CPU If the work is spread evenly across workers, find which objects the CPU is going to: ```mzsql SET CLUSTER TO ; EXPLAIN ANALYZE CLUSTER CPU; ``` The output is sorted by `total_elapsed`, so the objects at the top are the cluster's most expensive dataflows. Drill into one with `EXPLAIN ANALYZE CPU FOR MATERIALIZED VIEW ` to see its operators. > **Note:** `total_elapsed` accumulates from the moment the dataflow was created, so a > long-lived object can outrank one that is expensive *right now*. To see which > operators are busy at this moment, use > [`mz_compute_operator_durations_histogram`](/transform-data/dataflow-troubleshooting/#debugging-expensive-dataflows-and-operators). To resolve, [optimize the expensive object](/transform-data/optimization/), move it to its own cluster, or size the cluster up with [`ALTER CLUSTER ... SET (SIZE = '')`](/sql/alter-cluster/). Cross joins and joins without a suitable index are the usual culprits. ## Check for recent hydration Hydration is CPU-bound: the replica reprocesses each object's inputs from scratch. Any event that drops a replica's in-memory state re-runs that work, and hydrating a new object on a busy cluster can saturate it outright. To check whether a hydration episode overlaps the spike window: ```mzsql SELECT r.name AS replica_name, h.started_at, h.finished_at, h.object_count FROM mz_internal.mz_replica_hydration_history h JOIN mz_catalog.mz_clusters c ON h.cluster_id = c.id LEFT JOIN mz_catalog.mz_cluster_replicas r ON h.replica_id = r.id WHERE c.name = '' ORDER BY h.started_at DESC LIMIT 10; ``` To check what is still hydrating right now, including sources and sinks: ```mzsql SELECT o.name, h.replica_id FROM mz_internal.mz_hydration_statuses h JOIN mz_catalog.mz_objects o ON h.object_id = o.id JOIN mz_catalog.mz_clusters c ON o.cluster_id = c.id WHERE c.name = '' AND NOT coalesce(h.hydrated, false); ``` Hydration resolves itself, so no action is needed unless it keeps recurring. If it does, find what is triggering it: - **Replica restarts**, including [OOM crash loops](/transform-data/freshness-troubleshooting/#check-for-oom-crash-loops), which rehydrate the whole cluster on every restart. - **DDL or deploy activity**. Creating, altering, or dropping an object hydrates it on whichever cluster it lives on, which is why new objects belong in a [blue/green deployment](/developer-tools/dbt/blue-green-deployments/) rather than on a live production cluster. See [Check for DDL or deploy activity](/transform-data/freshness-troubleshooting/#check-for-ddl-or-deploy-activity). - **Cluster resizes**, which hydrate the new replicas before dropping the old ones. To shorten hydration without paying for a larger cluster continuously, see [autoscaling](/clusters/autoscaling/). ## Check ad-hoc query load A cluster serves `SELECT`s from the same replicas that maintain its indexes and materialized views, so query traffic and maintenance work compete for the same CPU. A burst of queries, or a query that can't be answered from an index, shows up as a cluster-wide spike. Client-side restart loops are a frequent cause: each reconnect resubmits the same queries, and each one spins up a temporary dataflow. Introspection relations report on the replica you are connected to, so run the following against the affected cluster: ```mzsql SET CLUSTER TO ; SELECT object_id, type, count(*) AS active_peeks FROM mz_introspection.mz_active_peeks GROUP BY object_id, type ORDER BY active_peeks DESC; ``` This is a point-in-time snapshot of in-flight reads; sample it repeatedly during a spike. For the window after the fact, query [`mz_internal.mz_recent_activity_log`](/sql/system-catalog/mz_internal/#mz_recent_activity_log) instead. To resolve, serve queries from a cluster separate from the one maintaining the objects, as described in the [operational guidelines](/clusters/operational-guidelines/#three-tier-architecture). ## Check upstream data volume A dataflow's steady-state CPU is proportional to the rate of change flowing through it, not to the size of its inputs. A cluster that was comfortable can saturate when an upstream system starts writing faster, with no change on the Materialize side. If the cluster hosts sources, sample their counters twice a minute apart and compare: ```mzsql SELECT s.name, ss.replica_id, ss.messages_received, ss.updates_committed FROM mz_internal.mz_source_statistics ss JOIN mz_catalog.mz_sources s ON ss.id = s.id JOIN mz_catalog.mz_clusters c ON s.cluster_id = c.id WHERE c.name = ''; ``` These counters are best-effort and only meaningful as rates; see [counter metrics](/sql/system-catalog/mz_internal/#counter-metrics). If the rate is far above what the workload was sized for, either size the cluster up or reduce the volume upstream. If the sources live on a different cluster from the objects that spiked, check that cluster too: downstream compute inherits its inputs' change rate. See [Check source ingestion](/transform-data/freshness-troubleshooting/#check-source-ingestion). ## Rule out memory pressure A replica that is close to its memory limit spills data to disk, and the resulting paging registers as CPU time that isn't doing any of your work. Check the other columns of [`mz_cluster_replica_utilization`](/sql/system-catalog/mz_internal/#mz_cluster_replica_utilization) for the same window: ```mzsql SELECT r.name AS replica_name, u.cpu_percent, u.memory_percent, u.disk_percent, u.swap_percent FROM mz_internal.mz_cluster_replica_utilization u JOIN mz_catalog.mz_cluster_replicas r ON u.replica_id = r.id JOIN mz_catalog.mz_clusters c ON r.cluster_id = c.id WHERE c.name = ''; ``` High `memory_percent` alongside rising `disk_percent` or `swap_percent` means you are looking at a memory problem wearing a CPU costume. Resolve it by sizing the cluster up or by reducing the memory footprint of its objects. ## Rule out general cluster overload If CPU is spread evenly across workers with no recent hydration, DDL, or upstream change, the cluster is undersized for its workload. To measure how much headroom is left, subscribe to the time workers spend idle: ```mzsql SET CLUSTER TO ; SUBSCRIBE ( SELECT sum(slept_for_ns * count) / 1e9 AS idle_seconds FROM mz_introspection.mz_scheduling_parks_histogram ); ``` Each update reports the total seconds the replica's workers have spent idle. Over a window of `T` seconds a fully idle replica accrues `T × `; as a rule of thumb, a replica with healthy headroom stays above 10% of that. A size's worker count is `processes * workers` from [`mz_catalog.mz_cluster_replica_sizes`](/sql/system-catalog/mz_catalog/#mz_cluster_replica_sizes). Note that this aggregates across workers, so it will not reveal skew. To resolve, size the cluster up with [`ALTER CLUSTER ... SET (SIZE = '')`](/sql/alter-cluster/) or move objects to another cluster. See [Check cluster health](/transform-data/freshness-troubleshooting/#check-cluster-health) for the corresponding freshness symptoms, and the [operational guidelines](/clusters/operational-guidelines/) for how to lay out clusters. --- ## M.1 to cc size mapping The following table provides a general mapping between cc and M.1 cluster sizes: **cc to M.1:** | cc Size | M.1 Size | | --- | --- | | 25cc | M.1-nano | | 50cc | M.1-nano | | 100cc | M.1-micro | | 200cc | M.1-xsmall | | 300cc | M.1-small | | 400cc | M.1-small | | 600cc | M.1-medium | | 800cc | M.1-large or M.1-medium | | 1200cc | M.1-1.5xlarge | | 1600cc | M.1-2xlarge or M.1-1.5xlarge | | 3200cc | M.1-8xlarge or M.1-4xlarge or M.1-3xlarge | | 6400cc | M.1-16xlarge | | 128C | M.1-32xlarge | | 256C | M.1-64xlarge | | 512C | M.1-128xlarge | **M.1 to cc:** | M.1 Size | cc Size | | --- | --- | | M.1-nano | 25cc | | M.1-nano | 50cc | | M.1-micro | 100cc | | M.1-xsmall | 200cc | | M.1-small | 300cc or 400cc | | M.1-medium | 600cc or 800cc | | M.1-large | 800cc | | M.1-1.5xlarge | 1200cc or 1600cc | | M.1-2xlarge | 1600cc | | M.1-3xlarge | 3200cc | | M.1-4xlarge | 3200cc | | M.1-8xlarge | 3200cc | | M.1-16xlarge | 6400cc | | M.1-32xlarge | 128C | | M.1-64xlarge | 256C | | M.1-128xlarge | 512C | Some sizes have multiple mappings. When converting between cc and M.1 sizing, we recommend choosing the larger mapping size first. --- ## Operational guidelines The following provides some general guidelines for production. ## Clusters ### Production clusters for production workloads only Use production cluster(s) for production workloads only. That is, avoid using production cluster(s) to run development workloads or non-production tasks. ### Three-tier architecture

In production, use a three-tier architecture, if feasible.

Image of the 3-tier architecture: Source cluster(s), Compute/Transform cluster(s), Serving cluster(s)

A three-tier architecture consists of:

| Tier | Description | | --- | --- | | Source cluster(s) |

A dedicated cluster(s) for sources.

In addition, for upsert sources:

  • Consider separating upsert sources from your other sources. Upsert sources have higher resource requirements (since, for upsert sources, Materialize maintains each key and associated last value for the key as well as to perform deduplication). As such, if possible, use a separate source cluster for upsert sources.

  • Consider using a larger cluster size during snapshotting for upsert sources. Once the snapshotting operation is complete, you can downsize the cluster to align with the steady-state ingestion.

| | Compute/Transform cluster(s) |

A dedicated cluster(s) for compute/transformation:

  • Materialized views to persist, in durable storage, the results that will be served. Results of materialized views are available across all clusters.

    > **Tip:** If you are using stacked views (i.e., views whose definition depends > on other views) to reduce SQL complexity, generally, only the topmost > view (i.e., the view whose results will be served) should be a > materialized view. The underlying views that do not serve results do not > need to be materialized.

  • Indexes, only as needed, to make transformation fast (such as possibly indexes on join keys).

    > **Tip:** From the compute/transformation clusters, do not create indexes on the > materialized views for the purposes of serving the view results. > Instead, use the [serving cluster(s)](#tier-serving-clusters) when > creating indexes to serve the results.
| | Serving cluster(s) | A dedicated cluster(s) for serving queries, including indexes on the materialized views. Indexes are local to the cluster in which they are created. |

Benefits of a three-tier architecture include:

#### Alternatives If a three-tier architecture is infeasible or unnecessary due to low volume or a non-production setup, a two cluster or a single cluster architecture may suffice. See [Appendix: Alternative cluster architectures](/clusters/operational-guidelines/appendix-alternative-cluster-architectures/) for details. ## Sources ### Scheduling If possible, schedule creating new sources during off-peak hours to mitigate the impact of snapshotting on both the upstream system and the Materialize cluster. ### Separate cluster(s) for sources In production, if possible, use a dedicated cluster for [sources](/fundamentals/concepts/sources/); i.e., avoid putting sources on the same cluster that hosts compute objects, sinks, and/or serves queries. In addition, for upsert sources: - Consider separating upsert sources from your other sources. Upsert sources have higher resource requirements (since, for upsert sources, Materialize maintains each key and associated last value for the key as well as to perform deduplication). As such, if possible, use a separate source cluster for upsert sources. - Consider using a larger cluster size during snapshotting for upsert sources. Once the snapshotting operation is complete, you can downsize the cluster to align with the steady-state ingestion. See also [Production cluster architecture](#three-tier-architecture). ## Sinks ### Separate sinks from sources To allow for [blue/green deployment](/developer-tools/dbt/blue-green-deployments/), avoid putting sinks on the same cluster that hosts sources . See also [Cluster architecture](#three-tier-architecture). ## Snapshotting considerations For upsert sources, snapshotting is a resource-intensive operation that can require a significant amount of CPU and memory. ## Hydration considerations When sizing a cluster, budget for hydration memory on top of the steady-state cost. The table below summarizes, per object type, when each object hydrates and the memory it uses. For more on hydration, including strategies to reduce its impact, see [Hydration](/fundamentals/concepts/hydration/). | Object | Hydration behavior | | --- | --- | | Materialized views | - **When**: Hydrates on creation and on every replica (re)start or cluster resize. - **What**: Rebuilds the dataflow's operator state: the arrangements that joins, aggregations, and similar operators keep to update results incrementally. Note: A materialized view's result lives in durable storage, so it rebuilds only this maintenance state, not the result. - **Memory Use**: Scales with the view's definition, which it holds at steady state, plus a transient output buffer up to twice the output size: the current output plus a read-back of the previously persisted output. On first creation, since there is no previous output, the buffer is a single output size. | | Indexes | - **When**: Hydrates on creation and on every replica (re)start or cluster resize. - **What**: Rebuilds the arranged (indexed) data it keeps in memory to serve reads, plus any operator arrangements its dataflow maintains (for joins, aggregations, and similar). - **Memory Use**: Its memory is proportional to the indexed data plus those arrangements, and is held for as long as the index exists. | | Kafka upsert sources and associated read-only tables/subsources | - **When**: On replica (re)start or cluster resize. These sources do not hydrate on creation; instead, on creation, their indexes are built as part of [snapshotting](/fundamentals/concepts/snapshotting/). - **What**: Rebuilds the table's or subsource's internal upsert index from storage. - **Memory Use**: The index holds the latest value per key, so its memory scales with the source's key space. On standard cluster sizes it can spill to disk when the key space exceeds memory. | | Append-only Kafka sources and CDC database sources (PostgreSQL, MySQL, SQL Server), and their read-only tables/subsources | - **When**: On replica (re)start or cluster resize, marked hydrated as soon as the dataflow starts. - **What**: Effectively nothing. These sources keep no internal index to rebuild and resume from their persisted position, so hydration is a no-op. - **Memory Use**: Negligible, since there is no index to hold. | | Webhook sources | Not applicable. A webhook source is not maintained by a dataflow. It receives data pushed over HTTP and writes the data directly to storage, so it does not hydrate. | | Sinks | - **When**: If created `WITH (SNAPSHOT = true)` (the default), hydrates: - On creation, when the sink first emits its input snapshot. - On a replica (re)start, but only if the sink restarted before recording any progress: it then re-reads the whole input snapshot, and any data already written to the external system is discarded, but the memory cost still occurs. An established sink resumes from its recorded progress without re-reading the snapshot. - **What**: Loads a full copy of its input snapshot into the arrangement that feeds the sink before it can emit. - **Memory Use**: Peaks at roughly a full copy of the input snapshot, then decreases as the snapshot is written out. Negligible on a restart of an established sink. At steady state, a sink retains little in memory. | | Subscriptions | - **When**: On creation and, while it remains active, on every replica (re)start: the dataflow is re-installed on the (re)started replica and the subscription resumes. A subscription that targets a specific replica instead ends with an error when that replica restarts. A subscription ends with its session and is not reported in `mz_hydration_statuses`. - **What**: Rebuilds the dataflow when it starts. - **Memory Use**: Scales with the dataflow, held while the subscription runs. | ## Role-based access control (RBAC) **Cloud:** ### Cloud ##### Follow the principle of least privilege Role-based access control in Materialize should follow the principle of least privilege. Grant only the minimum access necessary for users and service accounts to perform their duties. ##### Restrict the assignment of **Organization Admin** role {{% include-headless "/headless/rbac-cloud/org-admin-recommendation" %}} ##### Restrict the granting of `CREATEROLE` privilege {{% include-headless "/headless/rbac-cloud/createrole-consideration" %}} ##### Use Reusable Roles for Privilege Assignment {{% include-headless "/headless/rbac-cloud/use-resusable-roles" %}} See also [Manage database roles](/security/access-control/manage-roles/). ##### Audit for unused roles and privileges. {{% include-headless "/headless/rbac-cloud/audit-remove-roles" %}} See also [Show roles in system](/security/cloud/access-control/manage-roles/#show-roles-in-system) and [Drop a role](/security/cloud/access-control/manage-roles/#drop-a-role) for more information. **Self-Managed:** ### Self-Managed ##### Follow the principle of least privilege Role-based access control in Materialize should follow the principle of least privilege. Grant only the minimum access necessary for users and service accounts to perform their duties. ##### Restrict the granting of `CREATEROLE` privilege {{% include-headless "/headless/rbac-sm/createrole-consideration" %}} ##### Use Reusable Roles for Privilege Assignment {{% include-headless "/headless/rbac-sm/use-resusable-roles" %}} See also [Manage database roles](/security/self-managed/access-control/manage-roles/). ##### Audit for unused roles and privileges. {{% include-headless "/headless/rbac-sm/audit-remove-roles" %}} See also [Show roles in system](/security/self-managed/access-control/manage-roles/#show-roles-in-system) and [Drop a role](/security/self-managed/access-control/manage-roles/#drop-a-role) for more information. --- ## System clusters ## Overview When you enable a Materialize region, various [system clusters](/sql/system-clusters/) are pre-installed to improve the user experience as well as support system administration tasks. ### `quickstart` cluster A cluster named `quickstart` with a size of `25cc` and a replication factor of `1` will be pre-installed in every environment. You can modify or drop this cluster at any time. > **Note:** The default value for the `cluster` session parameter is `quickstart`. > This cluster functions as a default option, pre-created for your convenience. > It allows you to quickly start running queries without needing to configure a cluster first. > If the `quickstart` cluster is dropped, you must run [`SET cluster`](/sql/select/#ad-hoc-queries) > to choose a valid cluster in order to run `SELECT` queries. A _superuser_ (i.e. `Organization Admin`) > can also run [`ALTER SYSTEM SET cluster`](/sql/alter-system-set) to change the > default value. ### `mz_catalog_server` system cluster A system cluster named `mz_catalog_server` will be pre-installed in every environment. This cluster has several indexes installed to speed up `SHOW` commands and queries using the system catalog. To take advantage of these indexes, Materialize will automatically re-route `SHOW` commands and queries using system catalog objects to the `mz_catalog_server` system cluster. You can disable this behavior in your session via the `auto_route_catalog_queries` [configuration parameter](/sql/show/#other-configuration-parameters). The following characteristics apply to the `mz_catalog_server` cluster: * You are **not billed** for this cluster. * You cannot create objects in this cluster. * You cannot drop this cluster. * You can run `SELECT` or `SUBSCRIBE` queries in this cluster as long as you only reference objects in the [system catalog](/sql/system-catalog/). ### `mz_probe` system cluster A system cluster named `mz_probe` will be pre-installed in every environment. This cluster is used for internal uptime monitoring. The following characteristics apply to the `mz_probe` cluster: * You are **not billed** for this cluster. * You cannot create objects in this cluster. * You cannot drop this cluster. * You cannot run `SELECT` or `SUBSCRIBE` queries in this cluster. ### `mz_support` system cluster A system cluster named `mz_support` will be pre-installed in every environment. This cluster is used for internal support tasks. The following characteristics apply to the `mz_support` cluster: * You are **not billed** for this cluster. * You cannot create objects in this cluster. * You cannot drop this cluster. * You cannot run `SELECT` or `SUBSCRIBE` queries in this cluster. ### `mz_system` system cluster A system cluster named `mz_system` will be pre-installed in every environment. This cluster is used for internal system jobs. The following characteristics apply to the `mz_system` cluster: * You are **not billed** for this cluster. * You cannot create objects in this cluster. * You cannot drop this cluster. * You cannot run `SELECT` or `SUBSCRIBE` queries in this cluster. ## Related pages - [`CREATE CLUSTER`](/sql/create-cluster) - [`SHOW CLUSTER`](/sql/show-clusters) - [`DROP CLUSTER`](/sql/drop-cluster)