Resize node pools

1. Declare a second node pool with the new VM type

Add a new nodepool module instance alongside the existing one, keeping the old pool unchanged. Copy the existing configuration, then change:

• The prefix, so the pool gets a distinct name.
• The machine_type.
• For a swap-enabled pool, a distinct disk_setup_name (e.g. disk-setup-xl). It names the disk-setup namespace and daemonset, which otherwise collide with the old pool’s.
• The local_ssd_count, if the new machine type bundles a different number of local SSDs (c4a-highmem-16-lssd has 4, for example).

Keep the same labels and taints as the existing pool so Materialize pods are eligible to schedule on it. For example:

module "materialize_nodepool" {
  # ... existing pool config, unchanged ...
  machine_type = "c4a-highmem-8-lssd"
}

module "materialize_nodepool_xl" {
  # ... copy of the existing config, with a new prefix ...
  machine_type    = "c4a-highmem-16-lssd"
  local_ssd_count = 4
  disk_setup_name = "disk-setup-xl"
}

Run terraform init to pick up the new module instance, then apply:

terraform init
terraform apply

Both pools now exist. Materialize pods have not yet been scheduled on the new pool.

2. Taint the old pool so no new pods schedule on it

Add a decommission taint to the old pool’s node_taints:

node_taints = [
  # ... existing taints ...
  {
    key    = "materialize.cloud/decommissioned"
    value  = "true"
    effect = "NO_SCHEDULE"
  }
]

Taints update in place (no pool replacement) on the provider versions the modules require. Running pods are not evicted, but no new pods schedule to the old pool, and the cluster autoscaler will not scale it up for pending pods, since they don’t tolerate the taint. Use a taint key the Materialize pods don’t tolerate (not materialize.cloud/workload or kubernetes.io/arch).

Apply:

terraform apply

3. Roll out the Materialize instance

With the old pool tainted, a forced rollout lands the new generation of pods on the new pool.

The Materialize spec itself is unchanged (the node move happens at the Kubernetes cluster level and not in the Materialize CR), so you need to force the rollout.

module "materialize_instance" {
  # ...
  rollout_strategy = "WaitUntilReady"  # default
  force_rollout    = "00000000-0000-0000-0000-000000000002"  # any new UUID
}

module "materialize_instance" {
  # ...
  rollout_strategy = "WaitUntilReady"  # default
  request_rollout  = "00000000-0000-0000-0000-000000000002"  # any new UUID
  force_rollout    = "00000000-0000-0000-0000-000000000002"  # same UUID
}

Apply:

terraform apply

See Rollout behavior for what to expect during the rollout. Verify the new environmentd and clusterd pods are only scheduled onto the new pool.

4. Remove the old pool

Once the rollout has completed, the old pool’s nodes have no Materialize workloads on them. Remove the old nodepool module instance from your Terraform configuration and apply:

terraform apply

The destroy step now succeeds because the pool has no running workloads.

5. Optional: rename the new pool back

If you want the pool to keep the original name (for example because other Terraform or kubectl tooling references it), repeat these steps with a third pool that carries the original name. Otherwise, accept the new name and update any references.

1. Declare a second node pool with the new VM type

Add a new nodepool module instance alongside the existing one, keeping the old pool unchanged. Copy the existing configuration, then change:

• The prefix, so the pool gets a distinct name. The AKS node pool name is the prefix with dashes removed, truncated to 12 characters, so the new prefix must differ from the old one within those characters. A suffix appended to a long prefix is silently truncated away and the apply fails because the pool name already exists.
• The vm_size.
• For a swap-enabled pool, a distinct disk_setup_name (e.g. disk-setup-xl). It names the disk-setup namespace and daemonset, which otherwise collide with the old pool’s.

Keep the same labels and taints as the existing pool so Materialize pods are eligible to schedule on it. For example:

module "materialize_nodepool" {
  # ... existing pool config, unchanged ...
  prefix  = "mzpool"
  vm_size = "Standard_E4pds_v6"
}

module "materialize_nodepool_xl" {
  # ... copy of the existing config ...
  prefix          = "mzpoolxl"
  vm_size         = "Standard_E8pds_v6"
  disk_setup_name = "disk-setup-xl"
}

Run terraform init to pick up the new module instance, then apply:

terraform init
terraform apply

Both pools now exist. Materialize pods have not yet been scheduled on the new pool.

2. Taint the old pool so no new pods schedule on it

Add a decommission taint to the old pool’s node_taints:

node_taints = [
  # ... existing taints ...
  {
    key    = "materialize.cloud/decommissioned"
    value  = "true"
    effect = "NO_SCHEDULE"
  }
]

Taints update in place (no pool replacement) on the provider versions the modules require. Running pods are not evicted, but no new pods schedule to the old pool, and the cluster autoscaler will not scale it up for pending pods, since they don’t tolerate the taint. Use a taint key the Materialize pods don’t tolerate (not materialize.cloud/workload or kubernetes.io/arch).

Apply:

terraform apply

3. Roll out the Materialize instance

With the old pool tainted, a forced rollout lands the new generation of pods on the new pool.

The Materialize spec itself is unchanged (the node move happens at the Kubernetes cluster level and not in the Materialize CR), so you need to force the rollout.

module "materialize_instance" {
  # ...
  rollout_strategy = "WaitUntilReady"  # default
  force_rollout    = "00000000-0000-0000-0000-000000000002"  # any new UUID
}

module "materialize_instance" {
  # ...
  rollout_strategy = "WaitUntilReady"  # default
  request_rollout  = "00000000-0000-0000-0000-000000000002"  # any new UUID
  force_rollout    = "00000000-0000-0000-0000-000000000002"  # same UUID
}

Apply:

terraform apply

See Rollout behavior for what to expect during the rollout. Verify the new environmentd and clusterd pods are only scheduled onto the new pool.

4. Remove the old pool

Once the rollout has completed, the old pool’s nodes have no Materialize workloads on them. Remove the old nodepool module instance from your Terraform configuration and apply:

terraform apply

The destroy step now succeeds because the pool has no running workloads.

5. Optional: rename the new pool back

If you want the pool to keep the original name (for example because other Terraform or kubectl tooling references it), repeat these steps with a third pool that carries the original name. Otherwise, accept the new name and update any references.

The AWS modules provision Materialize nodes with Karpenter, so there is no second pool to create. Karpenter provisions new-spec nodes on demand.

If you have disabled Karpenter and run Materialize on a static EKS node group, follow the blue-green pattern from the GCP and Azure tabs instead, changing instance_types on a second node group module instance.

1. Update the instance types

Change instance_types on the Materialize karpenter-ec2nodeclass and karpenter-nodepool module instances and apply:

module "ec2nodeclass_materialize" {
  # ...
  instance_types = ["r7gd.4xlarge"]
}

module "nodepool_materialize" {
  # ...
  instance_types = ["r7gd.4xlarge"]
}

terraform apply

Karpenter marks the existing nodes as drifted but does not drain them: the environmentd and clusterd pods carry the karpenter.sh/do-not-disrupt annotation, which blocks voluntary disruption while they run. This relies on the nodepool’s expire_after being Never, see Karpenter node expiry.

2. Cordon the drifted nodes

Cordon the existing Materialize nodes so the rollout’s new pods cannot schedule onto them and instead trigger Karpenter to provision nodes with the new instance types:

# The nodepool name is the `name` input of the karpenter-nodepool module
# ("materialize" in the examples).
for node in $(kubectl get nodes -l karpenter.sh/nodepool=materialize -o name); do
  kubectl cordon "$node"
done

Nodes that Karpenter provisions after this point are not cordoned.

3. Roll out the Materialize instance

The Materialize spec itself is unchanged (the node move happens at the Kubernetes cluster level and not in the Materialize CR), so you need to force the rollout.

module "materialize_instance" {
  # ...
  rollout_strategy = "WaitUntilReady"  # default
  force_rollout    = "00000000-0000-0000-0000-000000000002"  # any new UUID
}

module "materialize_instance" {
  # ...
  rollout_strategy = "WaitUntilReady"  # default
  request_rollout  = "00000000-0000-0000-0000-000000000002"  # any new UUID
  force_rollout    = "00000000-0000-0000-0000-000000000002"  # same UUID
}

Apply:

terraform apply

Karpenter provisions new-spec nodes for the pending pods of the new generation. See Rollout behavior for what to expect during the rollout. Verify the new environmentd and clusterd pods are only scheduled onto the new nodes.

4. Verify the old nodes are removed

Once the rollout has completed and the old generation’s pods are gone, the cordoned nodes are empty and Karpenter consolidates them away (the modules configure consolidationPolicy: WhenEmpty). Confirm they disappear:

kubectl get nodes -l karpenter.sh/nodepool=materialize

Set forceRollout to a new UUID:

kubectl patch materialize <instance-name> \
  -n <materialize-instance-namespace> \
  --type='merge' \
  -p "{\"spec\": {\"forceRollout\": \"$(uuidgen)\"}}"

Set both requestRollout and forceRollout to the same new UUID:

UUID="$(uuidgen)"
kubectl patch materialize <instance-name> \
  -n <materialize-instance-namespace> \
  --type='merge' \
  -p "{\"spec\": {\"requestRollout\": \"$UUID\", \"forceRollout\": \"$UUID\"}}"