Google Cloud Monitoring

This guide walks you through the steps required to monitor the performance and overall health of your Materialize region using Google Cloud Monitoring ⧉. Self-Managed Materialize pushes metrics to Cloud Monitoring from the monitoring stack the Materialize Terraform modules install.

This destination is GCP only, and it is the one destination that needs cloud resources the monitoring chart cannot create for itself. That is why it is enabled with a flat variable rather than a configuration block: Terraform has to know at plan time whether to create the service account and the Workload Identity binding that authenticate it.

How it works

The stack collects metrics and logs before any destination is involved. For the collection pipeline and where that data is stored by default, see How logs and metrics are stored.

Cloud Monitoring is an additive destination. It receives its own filtered copy of the metrics, and the bundled Thanos, Grafana, and Alertmanager keep working as before.

Authentication is ambient, not a credential you supply. The module creates a Google service account, grants it roles/monitoring.metricWriter, and binds the gateway’s in-cluster ServiceAccount to it through Workload Identity. The exporter then authenticates with Application Default Credentials.

Instructions

Before you begin

Ensure you have:

• A Materialize deployment created with the Materialize Terraform modules, with the monitoring stack enabled. See Step 1.

• Terraform ⧉ installed.

• kubectl ⧉ installed and configured to connect to your cluster.

You also need:

• A deployment on GCP, created with the GCP Terraform modules. Workload Identity must be enabled on the cluster, which the gke module already sets.

• Permission to create a service account and IAM bindings in the project.

Step 1. Enable observability

The Materialize Terraform Modules take an enable_observability variable. Starting with v12.0.0 it defaults to true, so a fresh apply installs the monitoring stack without any configuration, and bumping ref=<tag> to v12.0.0 or later installs it on a deployment that never set the variable.

1. To confirm the setting, or to change it, set it explicitly in your terraform.tfvars:

   enable_observability = true    # default starting with Materialize Terraform Modules v12.0.0
   

2. Apply the configuration:

   terraform apply
   

   The apply creates the object storage and cloud identities for metrics and logs, and installs the stack into the monitoring namespace.

Step 2. Configure the Cloud Monitoring destination

1. On the monitoring module block of your Terraform, set:

   module "monitoring" {
     # ...
   
     enable_google_cloud_metrics         = true
     google_cloud_metrics_min_importance = "recommended"
   }
   

   Variable	Default	Purpose
   enable_google_cloud_metrics	false	Turns the destination on, and creates the service account and Workload Identity binding it authenticates with.
   google_cloud_metrics_min_importance	recommended	Which metrics to send. See How to control which metrics Cloud Monitoring receives.
   google_cloud_metrics_prefix	workload.googleapis.com/mzmon	The metric name prefix in Cloud Monitoring.

2. Apply the configuration:

   terraform apply
   

Step 3. Confirm metrics are arriving

1. Check that the gateway picked up the new configuration and is healthy:

   kubectl -n monitoring rollout status deployment/alloy-gateway
   

2. Query the receiving backend for recent samples of a metric you expect, such as mz_dataflow_wallclock_lag_seconds.

In Cloud Monitoring, use Metrics explorer and filter to the prefix, which is workload.googleapis.com/mzmon unless you changed it.

Step 4. Build alerts

Build Cloud Monitoring alerting policies ⧉ from the metrics and thresholds in Alerting.

The monitoring stack also ships Alertmanager rules that evaluate against the bundled Thanos. Decide which system owns which alerts rather than running both against the same thresholds.

How to control which metrics Cloud Monitoring receives

Cloud Monitoring bills per custom metric and per sample, so the tier you pick sets the bill.

Every metric the stack collects carries an importance tier, and each destination keeps only the metrics at or above a floor you choose. The tiers below run from most to least important, and the floor is cumulative: it keeps that tier and every tier above it.

The tiers are shared across the stack, so a tier selected in Terraform means the same set of metrics as the same tier selected in Helm. For the membership of each tier, see List of metrics ⧉. For the metrics Materialize recommends dashboarding and alerting on, see essential metrics, and for everything it exposes, the appendix of all metrics.

google_cloud_metrics_min_importance defaults to recommended, which covers the metrics the dashboards and alerts use. all is a diagnostic setting, not a steady state.

Instructions when using Helm

If you install the materialize-monitoring chart directly rather than through the Terraform modules, you create the identity and binding yourself, then point the chart at it.

1. Create a Google service account, grant it roles/monitoring.metricWriter on the project, and bind the gateway’s ServiceAccount to it with roles/iam.workloadIdentityUser.

2. Enable the exporter and annotate the gateway ServiceAccount so the binding applies:

   alloy-gateway:
     serviceAccount:
       annotations:
         iam.gke.io/gcp-service-account: <gsa>@<project>.iam.gserviceaccount.com
   
   pipeline:
     metrics:
       gateway:
         destination:
           otel:
             enabled: true
             googleCloudExporter:
               enabled: true
               minMetricImportance: recommended
   

There is no Secret to create, because the exporter authenticates with the ambient identity. Cloud Monitoring supports gzip compression only.

For the full value reference, see Metrics > Storing ⧉.

See also

• How logs and metrics are stored, for the bundled stores and the other backends you can send metrics and logs to.

• OpenTelemetry, for OTLP destinations, which can also carry logs.

• Alerting, for the metrics and thresholds to alert on.