Install on GCP

Materialize provides a set of modular Terraform modules that can be used to deploy all services required for Materialize to run on Google Cloud. The module is intended to provide a simple set of examples on how to deploy Materialize. It can be used as is or modules can be taken from the example and integrated with existing DevOps tooling.

Self-managed Materialize requires: a Kubernetes (v1.31+) cluster; PostgreSQL as a metadata database; blob storage; and a license key. The example on this page deploys a complete Materialize environment on GCP using the modular Terraform setup from this repository.

What Gets Created

This example provisions the following infrastructure:

Networking

Compute

Database

Storage

Kubernetes Add-ons

Materialize

Prerequisites

GCP Account Requirements

A Google account with permission to:

• Enable Google Cloud APIs/services on for your project.
• Create:
  • GKE clusters
  • Cloud SQL instances
  • Cloud Storage buckets
  • VPC networks and networking resources
  • Service accounts and IAM bindings

Required Tools

• Terraform
• gcloud CLI
• kubectl
• Helm 3.2.0+
• kubectl gke plugin

License Key

Getting started: Simple example

Step 1: Set Up the Environment

1. Open a terminal window.

2. Clone the Materialize Terraform repository and go to the gcp/examples/simple directory.

   git clone https://github.com/MaterializeInc/materialize-terraform-self-managed.git
   cd materialize-terraform-self-managed/gcp/examples/simple
   

3. Authenticate to GCP with your user account.

   gcloud auth login
   

4. Find the list of GCP projects:

   gcloud projects list
   

5. Set your active GCP project, substitute with your <PROJECT_ID>.

   gcloud config set project <PROJECT_ID>
   

6. Enable the following APIs for your project:

   gcloud services enable container.googleapis.com               # For creating Kubernetes clusters
   gcloud services enable compute.googleapis.com                 # For creating GKE nodes and other compute resources
   gcloud services enable sqladmin.googleapis.com                # For creating databases
   gcloud services enable cloudresourcemanager.googleapis.com    # For managing GCP resources
   gcloud services enable servicenetworking.googleapis.com       # For private network connections
   gcloud services enable iamcredentials.googleapis.com          # For security and authentication
   gcloud services enable iam.googleapis.com                     # For managing IAM service accounts and policies
   gcloud services enable storage.googleapis.com                 # For Cloud Storage buckets
   

7. Authenticate application default credentials for Terraform

   gcloud auth application-default login
   

Step 2: Configure Terraform Variables

1. Create a terraform.tfvars file and specify the following variables:

   Variable	Description
   project_id	Set to your GCP project ID.
   name_prefix	Set a prefix for all resource names (e.g., simple-demo) as well as your release name for the Operator
   region	Set the GCP region for the deployment (e.g., us-central1).
   license_key	Set to your Materialize license key.
   labels	Set to the labels to apply to resources.

   project_id  = "my-gcp-project"
   name_prefix = "simple-demo"
   region      = "us-central1"
   license_key = "your-materialize-license-key"
   labels = {
     environment = "demo"
     created_by  = "terraform"
   }
   

Step 3: Apply the Terraform

1. Initialize the Terraform directory to download the required providers and modules:

   terraform init
   

2. Apply the Terraform configuration to create the infrastructure.

   • To deploy with the default internal NLB for Materialize access:

   terraform apply
   

   • To deploy with public NLB for Materialize access:

   terraform apply -var="internal=false"
   

   If you are satisfied with the planned changes, type yes when prompted to proceed.

3. From the output, you will need the following field(s) to connect:

   • console_load_balancer_ip for the Materialize Console
   • balancerd_load_balancer_ip to connect PostgreSQL-compatible clients/drivers.

   terraform output -raw <field_name>
   

   💡 Tip: Your shell may show an ending marker (such as %) because the output did not end with a newline. Do not include the marker when using the value.

4. Configure kubectl to connect to your GKE cluster, replacing:

   • <your-gke-cluster-name> with your cluster name; i.e., the gke_cluster_name in the Terraform output. For the sample example, your cluster name has the form <name_prefix>-gke; e.g., simple-demo-gke

   • <your-region> with your cluster location; i.e., the gke_cluster_location in the Terraform output. Your region can also be found in your terraform.tfvars file.

   • <your-project-id> with your GCP project ID.

   # gcloud container clusters get-credentials <your-gke-cluster-name> --region <your-region> --project <your-project-id>
   gcloud container clusters get-credentials $(terraform output -raw gke_cluster_name) \
    --region $(terraform output -raw gke_cluster_location) \
    --project <your-project-id>
   

Step 4. Optional. Verify the status of your deployment

1. Check the status of your deployment:

   To check the status of the Materialize operator, which runs in the materialize namespace:

   kubectl -n materialize get all
   

   To check the status of the Materialize instance, which runs in the materialize-environment namespace:

   kubectl -n materialize-environment get all
   

   If you run into an error during deployment, refer to the Troubleshooting.

Step 5: Connect to Materialize

Connect using the Materialize Console

Using the console_load_balancer_ip from the Terraform output, you can connect to Materialize via the Materialize Console.

To connect to the Materialize Console, open a browser to https://<console_load_balancer_ip>:8080, substituting your <console_load_balancer_ip>.

From the terminal, you can type:

open "https://$(terraform output -raw console_load_balancer_ip):8080/materialize"

Connect using the psql

Using the balancerd_load_balancer_ip value from the Terraform output, you can connect to Materialize via PostgreSQL-compatible clients/drivers, such as psql:

psql "postgres://$(terraform output -raw balancerd_load_balancer_ip):6875/materialize"

Customizing Your Deployment

You can customize each module independently.

• For details on the Terraform modules, see both the top level and GCP specific modules.

• For details on recommended instance sizing and configuration, see the GCP deployment guide.

See also:

• Materialize Operator Configuration
• Materialize CRD Field Descriptions

Cleanup

To delete the whole sample infrastructure and deployment (including the Materialize operator and Materialize instances and data), run from the Terraform directory:

terraform destroy

When prompted to proceed, type yes to confirm the deletion.

See Also

• Materialize Operator Configuration
• Troubleshooting