# Self-Managed Monitor the performance of your Self-Managed Materialize region with Datadog and Grafana. This section covers monitoring and alerting for Self-Managed Materialize. ### Monitoring You can monitor the performance and overall health of your Self-Managed Materialize. To help you get started, the following guides are available: - [Grafana](/manage/monitor/self-managed/grafana/) - [Datadog using Prometheus SQL Exporter](/manage/monitor/self-managed/datadog/) ### Alerting After setting up a monitoring tool, you can configure alert rules. Alert rules send a notification when a metric surpasses a threshold. This will help you prevent operational incidents. For alert rules guidelines, see [Alerting](/manage/monitor/self-managed/alerting/). --- ## Alerting After setting up a monitoring tool, it is important to configure alert rules. Alert rules send a notification when a metric surpasses a threshold. This will help you prevent operational incidents. This page describes which metrics and thresholds to build as a starting point. For more details on how to set up alert rules in Datadog or Grafana, refer to: * [Datadog monitors](https://docs.datadoghq.com/monitors/) * [Grafana alerts](https://grafana.com/docs/grafana/latest/alerting/fundamentals/) ## Thresholds Alert rules tend to have two threshold levels, and we are going to define them as follows: * **Warning:** represents a call to attention to a symptom with high chances to develop into an issue. * **Alert:** represents an active issue that requires immediate action. For each threshold level, use the following table as a guide to set up your own alert rules: Metric | Warning | Alert | Description -- | -- | -- | -- CPU | 85% | 100% | Average CPU usage for a cluster in the last *15 minutes*. Memory | 80% | 90% | Average memory usage for a cluster in the last *15 minutes*. Source status | - | On Change | Source status change in the last *1 minute*. Cluster status | - | On Change | Cluster replica status change in the last *1 minute*. Freshness | > 5s | > 1m | Average [lag behind an input](/reference/system-catalog/mz_internal/#mz_materialization_lag) in the last *15 minutes*. ### Custom Thresholds For the following table, replace the two variables, _X_ and _Y_, by your organization and use case: Metric | Warning | Alert | Description -- | -- | -- | -- Latency | Avg > X | Avg > Y | Average latency in the last *15 minutes*. Where X and Y are the expected latencies in milliseconds. Credits | Consumption rate increase by X% | Consumption rate increase by Y% | Average credit consumption in the last *60 minutes*. --- ## Datadog using Prometheus SQL Exporter This guide walks you through the steps required to monitor the performance and overall health of your Materialize region using [Datadog](https://www.datadoghq.com/). ## Before you begin To make Materialize metadata available to Datadog, you must configure and run the following additional services: * A Prometheus SQL Exporter. * A Datadog Agent configured with an [OpenMetrics check](https://docs.datadoghq.com/integrations/openmetrics/). ## Step 1. Set up a Prometheus SQL Exporter To export metrics from Materialize and expose them in a format that Datadog can consume, you need to configure and run a Prometheus SQL Exporter. This service will run SQL queries against Materialize at specified intervals, and export the resulting metrics to a Prometheus endpoint. We recommend using [`justwatchcom/sql_exporter`](https://github.com/justwatchcom/sql_exporter), which has been tried and tested in production environments. 1. In the host that will run the Prometheus SQL Exporter, create a configuration file (`config.yml`) to hold the Exporter configuration. > **Tip:** You can use [this sample > `config.yml.example`](https://github.com/MaterializeInc/materialize-monitoring/blob/main/legacy/sql_exporter/config.yml) > as guidance to bootstrap your monitoring with some key Materialize metrics > and indicators. 1. In the configuration file, define the connection to your Materialize region under `connections` using the credentials provided in the [Materialize Console](/console/). > **Note:** You must escape the special `@` character in `USER` for a successful > connection. Example: instead of `name@email.com`, use `name%40email.com`. **Filename:** config.yml ```yaml --- jobs: - name: "materialize" # Interval between the runs of the job interval: '1m' # Materialize connection string connections: - "postgres://:@:6875/materialize?application_name=mz_datadog_integration&sslmode=require" ... ``` To specify different configurations for different sets of metrics, like a different `interval`, use additional jobs with a dedicated connection. ```yaml ... - name: "materialize" interval: '1h' connections: - "postgres://:@:6875/materialize?application_name=mz_datadog_integration&sslmode=require" ... ``` 1. Then, configure the `queries` that the Prometheus SQL Exporter should run at the specified `interval`. Take [these considerations](#considerations) into account when exporting metrics from Materialize. ```yaml ... queries: # Prefixed with sql_ and used as the metric name. - name: "replica_memory_usage" # Required option of the Prometheus default registry. Currently NOT # used by the Prometheus server. help: "Replica memory usage" # Array of columns used as additional labels. All lables should # be of type text. labels: - "replica_name" - "cluster_id" # Array of columns used as metric values. All values should be # of type float. values: - "memory_percent" # The SQL query that is run unalterted for each job. query: | SELECT name::text AS replica_name, cluster_id::text AS cluster_id, memory_percent::float AS memory_percent FROM mz_cluster_replicas r JOIN mz_internal.mz_cluster_replica_utilization u ON r.id=u.replica_id; ``` 1. Once you are done with the Prometheus SQL Exporter configuration, follow the intructions in the [`sql_exporter` repository](https://github.com/justwatchcom/sql_exporter#getting-started) to run the service using the configuration file from the previous step. ## Step 2. Set up a Datadog Agent To scrape the metrics available in the Prometheus SQL Exporter endpoint, you must then set up a [Datadog Agent](https://docs.datadoghq.com/agent/) check configured to scrape the OpenMetrics format. 1. Follow the [instructions to install and run a Datadog Agent](https://docs.datadoghq.com/agent/) in your host. 1. To configure an [OpenMetrics check](https://docs.datadoghq.com/integrations/openmetrics/) for the Datadog Agent installed in the previous step, edit the `openmetrics.d/conf.yaml` file at the root of the installation directory. **Filename**: openmetrics.d/conf.yaml ```yaml init_config: timeout: 50 instances: - openmetrics_endpoint: /metrics/ # The namespace to prepend to all metrics. namespace: "materialize" metrics: [.*] ``` **Tip:** see [this sample](https://github.com/MaterializeInc/demos/blob/main/integrations/datadog/datadog/conf.d/openmetrics.yaml) for all available configuration options. For more details on how to configure, run and troubleshoot Datadog Agents, see the [Datadog documentation](https://docs.datadoghq.com/getting_started/agent/). ## Step 3. Build a monitoring dashboard With the Prometheus SQL Exporter running SQL queries againt your Materialize region and exporting the results as metrics, and the Datadog Agent routing these metrics to your Datadog account, you're ready to build a monitoring dashboard! **Tip:** use [this sample](https://github.com/MaterializeInc/demos/blob/main/integrations/datadog/dashboard.json) to bootstrap a new dashboard with the key Materialize metrics and indicators defined in the sample `config.yml`. 1. **Log in** to your Datadog account. 1. Navigate to **Dashboards**, and select **New Dashboard**. 1. To use the sample dashboard, navigate to ⚙️ in the upper right corner, and select **Import dashboard JSON**. Copy and paste the contents of the provided sample `.json` file.
Template Datadog monitoring dashboard ## Considerations Before adding a custom query, make sure to consider the following: 1. The label set cannot repeat across rows within the results of the same query. 2. Columns must not contain `NULL` values. 3. Value columns must be of type `float`. 4. The Datadog agent is subject to a limit of 2000 metrics. 5. Queries can impact cluster performance. --- ## Grafana The [Materialize Terraform modules](/self-managed-deployments/) can deploy a monitoring stack alongside your Materialize deployment. Enabling it installs: | Component | Purpose | |-----------|---------| | **Grafana** | Dashboards and query UI, with the Materialize dashboards pre-installed. | | **Thanos** | Metrics storage backed by object storage, with a Prometheus-compatible query endpoint. | | **Loki** | Log storage backed by object storage. | | **Grafana Alloy** | Collection of metrics and logs from Materialize and from the cluster. | | **Alertmanager** | Alert routing. | The stack comes from the [`materialize-monitoring` ⧉](https://github.com/MaterializeInc/materialize-monitoring) charts. The Terraform modules also create the object storage and the cloud identities the stack needs, so you do not have to configure scrape targets, data sources, or dashboards yourself. ## Before you begin Ensure you have: - A Materialize deployment created with the [Materialize Terraform modules](/self-managed-deployments/). - [Terraform ⧉](https://developer.hashicorp.com/terraform/install) installed. - [kubectl ⧉](https://kubernetes.io/docs/tasks/tools/) installed and configured to connect to your cluster. ## Step 1. Enable observability The `simple` and `enterprise` examples for each cloud take an `enable_observability` variable, which defaults to `false` in `simple` and `true` in `enterprise`. 1. In your `terraform.tfvars`, set: ```hcl enable_observability = true ``` 1. Apply the configuration: ```bash terraform apply ``` The apply creates the object storage and cloud identities for metrics and logs, and installs the stack into the `monitoring` namespace. > **Note:** The monitoring stack runs several components: Loki, Thanos, Grafana, > Alertmanager, kube-state-metrics, and two Alloy roles. Your generic node pool > may need to grow before the first apply can schedule all of them. If you instantiate the modules in your own Terraform rather than using an example, add the `monitoring` module for your cloud (see the [Terraform installation guide ⧉](https://materializeinc.github.io/materialize-monitoring/getting-started/terraform/)), and turn on the operator's scrape annotations so its pods are collected: ```hcl module "operator" { # ... helm_values = { observability = { enabled = true prometheus = { scrapeAnnotations = { enabled = true } } } } } ``` ## Step 2. Access Grafana Grafana is deployed as a `ClusterIP` service in the `monitoring` namespace, so reaching it means port forwarding. 1. Retrieve the `admin` password from the Terraform output: ```bash terraform output -raw grafana_admin_password ``` > **Tip:** Your shell may show an ending marker (such as `%`) because the output did not > end with a newline. Do not include the marker when using the value. 1. Forward a local port to the Grafana service: ```bash kubectl -n monitoring port-forward svc/grafana 3000:80 ``` > **Warning:** Port forwarding is for testing purposes only. For production environments, > expose Grafana through your own ingress and configure authentication for it. 1. Open [http://localhost:3000](http://localhost:3000) in a browser and log in as `admin` with the password from the first step. ## Step 3. Open the Materialize dashboards The dashboards and their data sources are installed by grafana-operator from the released chart, so they track the chart version rather than a copy you maintain. To confirm they landed: ```bash kubectl -n monitoring get grafanamanifest,grafanadatasource ``` > **Note:** Helm returns once the operator's Deployment is ready. Pushing the dashboards > into Grafana happens afterwards and can fail on its own, so check these > resources rather than the Helm release status. ![Image of Grafana](/images/self-managed/grafana-monitoring-success.png) For the list of dashboards and what each one covers, see [Grafana dashboards ⧉](https://materializeinc.github.io/materialize-monitoring/dashboards/grafana/importing/). ## Connect existing tooling If you already run Grafana, or want to point other tools at the collected data, the examples output the query endpoints: | Output | Endpoint | |--------|----------| | `metrics_url` | Thanos Query. Prometheus-API-compatible, so anything that speaks to a Prometheus server works against it. | | `logs_url` | Loki read endpoint. | ```bash terraform output -raw metrics_url ``` ## Advanced configuration The monitoring modules expose further options, including sizing profiles, retention, node placement, and raw Helm value overrides. For these, and for installing the stack without the Materialize Terraform modules, see: - [Terraform installation guide ⧉](https://materializeinc.github.io/materialize-monitoring/getting-started/terraform/), for the full set of module variables. - [Helm installation guide ⧉](https://materializeinc.github.io/materialize-monitoring/getting-started/helm/), for installing the stack with Helm rather than Terraform. - [Production best practices ⧉](https://materializeinc.github.io/materialize-monitoring/operating/production-best-practices/), for the throughput envelope each sizing profile assumes and what to scale when metric/logging queries feel slow. ## Alerting The stack includes Alertmanager. For the metrics and thresholds to start from, see [Alerting](/manage/monitor/self-managed/alerting/).