# Upgrade on AWS
Upgrade Materialize on AWS using the Terraform module.
The following tutorial upgrades your Materialize deployment running on AWS
Elastic Kubernetes Service (EKS). The tutorial assumes you have installed the
example on [Install on
AWS](/self-managed-deployments/installation/install-on-aws/).

## Upgrade guidelines

<p>When upgrading:</p>
<ul>
<li>
<p><strong>Always</strong> check the <a href="/materialize/38323/self-managed-deployments/upgrading/version-notes/" >version-specific upgrade
notes</a> for your target
version:</p>
<ul>
<li><a href="/materialize/38323/self-managed-deployments/upgrading/version-notes/#upgrading-to-v2630-and-later-versions" >Upgrading to <code>v26.30</code> and later versions</a></li>
<li><a href="/materialize/38323/self-managed-deployments/upgrading/version-notes/#upgrading-to-v261-and-later-versions" >Upgrading to <code>v26.1</code> and later versions</a></li>
<li><a href="/materialize/38323/self-managed-deployments/upgrading/version-notes/#upgrading-to-v260" >Upgrading to <code>v26.0</code></a></li>
<li><a href="/materialize/38323/self-managed-deployments/upgrading/version-notes/#upgrading-between-minor-versions-less-than-v26" >Upgrading between minor versions less than <code>v26</code></a></li>
</ul>
</li>
<li>
**Always** upgrade the Materialize Operator **before**
upgrading the Materialize instances.

</li>
</ul>


> **Note:** For major version upgrades, you can **only** upgrade **one** major version
> at a time. For example, upgrades from **v26**.1.0 to **v27**.3.0 is
> permitted but **v26**.1.0 to **v28**.0.0 is not.


> **Note:** Downgrading is not supported.


## Prerequisites

### Required Tools

- [Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform)
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
- [kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html)

## Upgrade process

> **Important:** The following procedure performs a rolling upgrade, where both the old and new Materialize instances are running before the old instances are removed. When performing a rolling upgrade, ensure you have enough resources to support having both the old and new Materialize instances running.


### Step 1: Update the Materialize Terraform Modules source version

Update each module's `source` to point to the desired release tag, substituting
`<RELEASE_TAG>` in the code block below with your tag version:

> **Important:** The following code block is not comprehensive. Only the core modules and their
> dependency chain are shown below.
> If your configuration includes additional modules (networking, storage,
> database, node pools, etc.) provided by Materialize, **update those to the same
> release tag as well**.


```hcl
module "eks" {
  source = "github.com/MaterializeInc/materialize-terraform-self-managed//aws/modules/eks?ref=<RELEASE_TAG>"
  # ... your existing configuration ...
}

module "cert_manager" {
  source = "github.com/MaterializeInc/materialize-terraform-self-managed//kubernetes/modules/cert-manager?ref=<RELEASE_TAG>"
  # ... your existing configuration ...

  # Your configuration may have additional dependencies here.
  depends_on = [module.eks]
}

module "operator" {
  source = "github.com/MaterializeInc/materialize-terraform-self-managed//aws/modules/operator?ref=<RELEASE_TAG>"
  # ... your existing configuration ...

  # Your configuration may have additional dependencies here.
  depends_on = [module.cert_manager]
}

module "materialize_instance" {
  source = "github.com/MaterializeInc/materialize-terraform-self-managed//kubernetes/modules/materialize-instance?ref=<RELEASE_TAG>"
  # ... your existing configuration ...

  # Your configuration may have additional dependencies here.
  depends_on = [module.operator]
}

# Update the source of any additional Materialize-provided modules to the same release tag
```

### Step 2: Explicitly request rollout if using v1alpha1

<p><code>v1alpha1</code> is the default CRD version for the Materialize Helm
chart. The Terraform modules default to <code>v1</code> starting in v4.0.0.
With <code>v1alpha1</code>, instance rollouts require manually rotating a
UUID.</p>

> **Important:** Starting in Terraform module version v4.0.0, `crd_version` defaults to
> `v1`. If your instance uses `v1alpha1` and you are upgrading to module
> version v4.0.0 or greater, set `crd_version = "v1alpha1"` explicitly to
> stay on `v1alpha1`. Otherwise, applying migrates the instance to `v1` and
> triggers a rollout. See [Adopting the v1
> CRD](/self-managed-deployments/upgrading/adopting-the-v1-crd/).



To check the CRD version of the Materialize manifest that was applied, run
the following:

```sh
terraform state show 'module.materialize_instance.kubectl_manifest.materialize_instance' \
  | grep -iE 'api_?version|kind'
```


- If you are using `v1`, skip to the [Apply the updated Terraform
  step](#step-3-apply-the-updated-terraform).
- **If you are using `v1alpha1`**, you need to update your `terraform.tfvars`
file to set the `request_rollout` variable to a new UUID value, substituting
the example value in the code block below with a UUID you generate (for
example, with `uuidgen`):

```hcl
# ...
# ...
request_rollout = "DBB4FCEC-1837-44F6-9CF2-3894678DD8D5" # ONLY for v1alpha1
```


### Step 3: Apply the updated Terraform

1. Initialize the Terraform directory to download the required providers
    and modules:

    ```bash
    terraform init
    ```

1. Review the execution plan before applying. In particular, check for any
  resources Terraform plans to destroy and recreate (shown as `-/+` in the
  plan), especially stateful resources such as your cluster, storage, and
  database:

    ```bash
    terraform plan
    ```

1. After reviewing the plan, apply the Terraform configuration.

    ```bash
    terraform apply
    ```


### Step 4: Verify the upgrade

Configure `kubectl` to connect to your EKS cluster, replacing `<your-region>`
with the region of your cluster (found in your `terraform.tfvars`; e.g.,
`us-east-1`):

```bash
# aws eks update-kubeconfig --name <your-eks-cluster-name> --region <your-region>
aws eks update-kubeconfig --name $(terraform output -raw eks_cluster_name) --region <your-region>
```

> **Note:** `terraform apply` returns once the Materialize custom resource is updated.
> The Operator then rolls out the new generation asynchronously, so the new
> `environmentd` pods may take a few minutes to become ready.

<ol>
<li>
<p>Check the status of the <code>materialize</code> namespace:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">kubectl -n materialize get all
</span></span></code></pre></div></li>
<li>
<p>Check the status of the <code>materialize-environment</code> namespace:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">kubectl -n materialize-environment get all
</span></span></code></pre></div></li>
<li>
<p>Once a new <code>environmentd</code> is up (may take a few minutes), confirm the
running <code>environmentd</code> version matches the version you upgraded to. Check
the <code>Image</code> field in the pod description.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">kubectl -n materialize-environment describe pod -l <span class="nv">app</span><span class="o">=</span>environmentd
</span></span></code></pre></div></li>
</ol>
<p>If you run into an error during the upgrade, refer to the
<a href="/materialize/38323/self-managed-deployments/troubleshooting/" >Troubleshooting</a>.</p>


## Enable the monitoring stack

The Terraform modules can install a monitoring stack — Grafana, Thanos, Loki,
Grafana Alloy, and Alertmanager — alongside your deployment, with the
Materialize dashboards pre-installed. You can turn it on during an upgrade, in
the same `terraform apply` as the version bump.

The stack below arrived in **v10.0.0** of the Materialize Terraform Modules,
replacing the earlier single Prometheus and Grafana. **v10.1.0** then added
durable state for Grafana and a load balancer to reach it on.

> **Warning:** Starting with **v12.0.0** of the Materialize Terraform Modules,
> `enable_observability` defaults to `true`. Bumping `ref=<RELEASE_TAG>` to
> v12.0.0 or later therefore installs the whole stack, and its billable
> supporting resources, on a deployment that never set the variable. Set
> `enable_observability = false` in the same change if you do not want it.


> **Warning:** `kubernetes/modules/prometheus` and `kubernetes/modules/grafana` were **removed**
> in v10.0.0, not deprecated in place. If your configuration references either
> directly, that reference breaks — pin the previous major until you have
> migrated.
> If you were running the old stack, upgrading **destroys** its Helm releases and
> PersistentVolumeClaims. Up to 15 days of local Prometheus data goes with them,
> along with anything hand-created in the old Grafana. There is no backfill. See
> [How to upgrade from previous versions of the Materialize Terraform
> Modules](/manage/monitor/self-managed/grafana/#how-to-upgrade-from-previous-versions-of-the-materialize-terraform-modules).


### If you use the example configuration

Nothing is required starting with v12.0.0 of the Materialize Terraform
Modules, where the variable defaults to `true`. To be explicit, or on an
earlier release, set the following in your `terraform.tfvars`:

```hcl
enable_observability = true
```

### If you instantiate the modules yourself

1. Add the `alekc/kubectl` provider to your `versions.tf`. The monitoring module
   uses it for the `TargetGroupBinding` that attaches the Grafana load balancer
   to the Grafana Service:

   ```hcl
   kubectl = {
     source  = "alekc/kubectl"
     version = "2.4.1"
   }
   ```

1. Add the `monitoring` module, using the same release tag as the rest of your
   modules:

   ```hcl
   module "monitoring" {
     source = "github.com/MaterializeInc/materialize-terraform-self-managed//aws/modules/monitoring?ref=<RELEASE_TAG>"

     name_prefix = var.name_prefix
     region      = var.aws_region

     namespace = "monitoring"
     # The operator module already creates this namespace.
     create_namespace = false

     oidc_provider_arn       = module.eks.oidc_provider_arn
     cluster_oidc_issuer_url = module.eks.cluster_oidc_issuer_url

     storage_class = module.ebs_csi_driver.storage_class_name

     materialize_instance_namespace = "materialize-environment"
     materialize_operator_namespace = "materialize"

     # Grafana's own state. Omit to leave Grafana on SQLite.
     grafana_database = {
       vpc_id                    = module.networking.vpc_id
       subnet_ids                = module.networking.private_subnet_ids
       cluster_name              = module.eks.cluster_name
       cluster_security_group_id = module.eks.cluster_security_group_id
       node_security_group_id    = module.eks.node_security_group_id
     }

     # Reach Grafana without port forwarding. Omit to keep it on ClusterIP.
     grafana_load_balancer = {
       vpc_id                 = module.networking.vpc_id
       subnet_ids             = module.networking.private_subnet_ids
       node_security_group_id = module.eks.node_security_group_id
       ingress_cidr_blocks    = var.ingress_cidr_blocks
     }

     depends_on = [module.operator]
   }
   ```

1. Turn on the operator's scrape annotations so its pods are collected:

   ```hcl
   module "operator" {
     # ...
     helm_values = {
       observability = {
         enabled = true
         prometheus = {
           scrapeAnnotations = {
             enabled = true
           }
         }
       }
     }
   }
   ```

### What this creates

Applying the above adds S3 buckets for metrics and logs, and — from
Materialize Terraform Modules v10.1.0 — a `db.t4g.micro` RDS instance for
Grafana's own state and an internal NLB to reach Grafana on. The database and
the load balancer are both billable.

> **Warning:** The Grafana load balancer terminates no TLS, and Grafana has no identity
> provider until you configure one. Keep it internal until both are addressed. A
> public load balancer whose allowlist is still `0.0.0.0/0` is refused at plan
> time for Grafana specifically.


> **Note:** The monitoring stack runs several components: Loki, Thanos, Grafana,
> Alertmanager, kube-state-metrics, and two Alloy roles. Your generic node pool
> may need to grow before the apply can schedule all of them.


For accessing Grafana, pointing the stack at a database you already run, sizing
profiles, and retention, see
[Grafana](/manage/monitor/self-managed/grafana/). For what the stack stores and
the backends it can forward to, see [How logs and metrics are
stored](/manage/monitor/self-managed/storage/).

## See also

- [Materialize Operator
  Configuration](/self-managed-deployments/operator-configuration/)
- [Materialize CRD Field
  Descriptions](/self-managed-deployments/materialize-crd-field-descriptions/)
- [Troubleshooting](/self-managed-deployments/troubleshooting/)