CREATE CLUSTER

CREATE CLUSTER creates a new cluster.

Syntax

CREATE CLUSTER <cluster_name> (
    SIZE = <text>
    [, REPLICATION FACTOR = <int>]
    [, MANAGED = <bool>]
    [, AUTO SCALING STRATEGY = (
        ON HYDRATION (
            HYDRATION SIZE = <text>
            [, LINGER DURATION = <interval>]
        )
    )]
);

Details

Initial state

Each Materialize region initially contains a pre-installed cluster named quickstart with a size of 25cc and a replication factor of 1. You can drop or alter this cluster to suit your needs.

Choosing a cluster

When performing an operation that requires a cluster, you must specify which cluster you want to use. Not explicitly naming a cluster uses your session’s active cluster.

To show your session’s active cluster, use the SHOW command:

SHOW cluster;

To switch your session’s active cluster, use the SET command:

SET cluster = other_cluster;

Resource isolation

Clusters provide resource isolation. Each cluster provisions a dedicated pool of CPU, memory, and, optionally, scratch disk space.

All workloads on a given cluster will compete for access to these compute resources. However, workloads on different clusters are strictly isolated from one another. A given workload has access only to the CPU, memory, and scratch disk of the cluster that it is running on.

Clusters are commonly used to isolate different classes of workloads. For example, you could place your development workloads in a cluster named dev and your production workloads in a cluster named prod.

Available sizes

The SIZE option determines the amount of compute resources available to the cluster.

See also:

• cc to M.1 size mapping.

• Materialize service consumption table.

• Blog:Scaling Beyond Memory: How Materialize Uses Swap for Larger Workloads.

Cluster resizing

You can change the size of a cluster to respond to changes in your workload using ALTER CLUSTER.

As of v26.35, resizing is graceful and incurs no downtime: Materialize provisions new replicas at the target size, waits for them to hydrate, then retires the old ones. See Monitoring a resize.

In versions before v26.35, resizing could incur downtime, and zero-downtime resizing required the WAIT UNTIL READY option.

See the reference documentation for ALTER CLUSTER for more details on cluster resizing.

Autoscaling

When you create an index, materialized view, or Kafka upsert source, or when a cluster restarts, the cluster must hydrate the affected objects before they can serve results. Hydration reads the input data and rebuilds in-memory state, and its speed scales with the cluster size.

The AUTO SCALING STRATEGY (ON HYDRATION) option lets a cluster automatically provision an extra burst replica at the configured HYDRATION SIZE while it has un-hydrated objects. This speeds up hydration without manually scaling the cluster up before hydration and back down afterward. The steady-size replicas continue hydrating in parallel, and once one of them catches up with the burst, the burst replica lingers for the LINGER DURATION and is then removed. The burst replica is an ordinary cluster replica, billed only for the time it is provisioned. See Usage & billing for details.

AUTO SCALING STRATEGY (ON HYDRATION) is particularly useful for blue/green deployments, where a new cluster must hydrate before the cutover. It is only available on managed clusters, and cannot be combined with a cluster SCHEDULE other than the default MANUAL.

For example, the following cluster can provision a burst replica of size 800cc:

CREATE CLUSTER fast_start (
    SIZE = '100cc',
    AUTO SCALING STRATEGY = (
        ON HYDRATION (
            HYDRATION SIZE = '800cc',
            LINGER DURATION = '15s'
        )
    )
);

You can specify the following options:

Provisioning the burst replica requires enough compute capacity to run it. In Materialize Self-Managed, this means your Kubernetes cluster must have enough spare resources (for example, available nodes) to schedule the burst replica.

The burst is best-effort and never blocks the cluster: if the burst replica cannot be provisioned, the steady-size replicas still come up and hydrate as usual, as long as there are enough resources for them.

To remove the autoscaling strategy from a cluster, use ALTER CLUSTER ... RESET (AUTO SCALING STRATEGY) or set an empty strategy with AUTO SCALING STRATEGY = ().

You can inspect the configured strategy and any in-flight burst in the mz_internal.mz_cluster_auto_scaling_strategies catalog view.

Replication factor

The REPLICATION FACTOR option determines the number of replicas provisioned for the cluster. Each replica of the cluster provisions a new pool of compute resources to perform exactly the same computations on exactly the same data.

Provisioning more than one replica improves fault tolerance. Clusters with multiple replicas can tolerate failures of the underlying hardware that cause a replica to become unreachable. As long as one replica of the cluster remains available, the cluster can continue to maintain dataflows and serve queries.

Materialize makes the following guarantees when provisioning replicas:

• Replicas of a given cluster are never provisioned on the same underlying hardware.
• Replicas of a given cluster are spread as evenly as possible across the underlying cloud provider’s availability zones.

Materialize automatically assigns names to replicas like r1, r2, etc. You can view information about individual replicas in the console and the system catalog, but you cannot directly modify individual replicas.

You can pause a cluster’s work by specifying a replication factor of 0. Doing so removes all replicas of the cluster. Any indexes, materialized views, sources, and sinks on the cluster will cease to make progress, and any queries directed to the cluster will block. You can later resume the cluster’s work by using ALTER CLUSTER to set a nonzero replication factor.

Credit usage

Each replica of the cluster consumes credits at a rate determined by the cluster’s size:

Credit usage is measured at a one second granularity. For a given replica, credit usage begins when a CREATE CLUSTER or ALTER CLUSTER statement provisions the replica and ends when an ALTER CLUSTER or DROP CLUSTER statement deprovisions the replica.

A cluster with a replication factor of zero uses no credits.

As an example, consider the following sequence of events:

Cluster c will have consumed 0.4 credits in total:

• Replica c.r1 was provisioned from 3:45:00 to 3:47:15, consuming 0.3 credits.
• Replica c.r2 was provisioned from 3:45:00 to 3:45:45, consuming 0.1 credits.

Known limitations

Clusters have several known limitations:

• When a cluster using legacy cc size of 3200cc or larger uses multiple replicas, those replicas are not guaranteed to be spread evenly across the underlying cloud provider’s availability zones.

Examples

Basic

Create a cluster with two 200cc replicas:

CREATE CLUSTER c1 (SIZE = '200cc', REPLICATION FACTOR = 2);

Empty

Create a cluster with no replicas:

CREATE CLUSTER c1 (SIZE '100cc', REPLICATION FACTOR = 0);

You can later add replicas to this cluster with ALTER CLUSTER.

Privileges

The privileges required to execute this statement are:

• CREATECLUSTER privileges on the system.

See also

• ALTER CLUSTER
• DROP CLUSTER