# Self-Managed Deployments Learn about the key components and architecture of self-managed Materialize deployments. > For the complete documentation index, see [llms.txt](https://preview.materialize.com/materialize/40269/llms.txt). ## Overview Whereas Materialize Cloud gives you a fully managed service for Materialize, Self-Managed Materialize allows you to deploy Materialize in your own infrastructure. Self-Managed Materialize deployments on Kubernetes consist of several layers of components that work together to provide a fully functional database environment. Understanding these components and how they interact is essential for deploying, managing, and troubleshooting your Self-Managed Materialize. ## Getting started To help you get started, the following installation guides are available: ### Install using Helm Commands | Guide | Description | | ------------- | -------| | [Install locally on Kind](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/installation/install-on-local-kind/index.md) | Uses standard Helm commands to deploy Materialize to a Kind cluster in Docker. ### Install using Terraform Modules > **Note:** We recommend pinning your module sources to specific tags to avoid unexpected breaking > changes in future versions. > We recommend updating your module source tags when updating Materialize versions, > taking care to follow any instructions in the release notes. Materialize provides [**Terraform modules**](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main?tab=readme-ov-file#materialize-self-managed-terraform-modules), which provides concrete examples and an opinionated model for deploying Materialize. | Module | Description | | --- | --- | | Amazon Web Services (AWS) | An example Terraform module for deploying Materialize on AWS. See Install on AWS for detailed instructions usage. | | Azure | An example Terraform module for deploying Materialize on Azure. See Install on Azure for detailed instructions usage. | | Google Cloud Platform (GCP) | An example Terraform module for deploying Materialize on GCP. See Install on GCP for detailed instructions usage. | ## Architecture layers A Self-Managed Materialize deployment is organized into the following layers: Layer | Component | Description ------|-----------|------------ **Infrastructure** | [Helm Chart](#helm-chart) | Package manager component that bootstraps the Kubernetes deployment **Orchestration** | [Materialize Operator](#materialize-operator) | Kubernetes operator that manages Materialize instances **Database** | [Materialize Instance](#materialize-instance) | The Materialize database instance itself **Compute** | [Clusters and Replicas](#clusters-and-replicas) | Isolated compute resources for workloads ### Helm chart The Helm chart is the entry point for deploying Materialize in a self-managed Kubernetes environment. It defines and deploys the Materialize Operator as part of the Helm package management workflow. The Helm repository is hosted at `https://materializeinc.github.io/materialize`. ```sh helm repo add materialize https://materializeinc.github.io/materialize ``` #### What gets installed When you install the Materialize Helm Chart (e.g., `helm install materialize materialize/materialize-operator ...`), it: - Deploys the **Materialize Operator** as a Kubernetes deployment. - Creates necessary cluster-wide resources (CRDs, RBAC roles, service accounts). - Configures operator settings and permissions. Once installed, the **Materialize Operator** handles the deployment and management of Materialize instances. ### Materialize Operator The Materialize Operator (implemented as `orchestratord`) is a Kubernetes operator that automates the deployment and lifecycle management of Materialize instances. It implements the [Kubernetes operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) to extend Kubernetes with domain-specific knowledge about Materialize. #### Managed resources The operator watches for Materialize custom resources and creates/manages all the Kubernetes resources required to run a Materialize instance, including: - **Namespaces**: Isolated Kubernetes namespaces for each instance - **Services**: Network services for connecting to Materialize - **Network Policies**: Network isolation and security rules - **Certificates**: TLS certificates for secure connections - **ConfigMaps and Secrets**: Configuration and sensitive data - **Deployments**: These support the `balancerd` and `console` pod used as the ingress layer for Materialize. - **StatefulSets**: `environmentd` and `clusterd` which are the database control plane and compute resources respectively. #### Configuration For configuration options for the Materialize Operator, see the [Materialize Operator Configuration page](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/operator-configuration/index.md). ### Materialize Instance A Materialize instance is the actual database that you connect to and interact with. Each instance is an isolated Materialize deployment (deployed via a Kubernetes Custom Resource) with its own data, configuration, and compute resources. #### Components When you create a Materialize instance, the operator deploys three core components as Kubernetes resources: - **balancerd**: A pgwire and HTTP proxy that routes all Materialize client connections to `environmentd` for handling. `balancerd` is deployed as a Kubernetes Deployment. - **environmentd**: The main database control plane, deployed as a StatefulSet. **`environmentd`** runs as a Kubernetes pod and is the primary component of a Materialize instance. It houses the control plane and contains: - **Adapter**: The SQL interface that handles client connections, query parsing, and planning. - **Storage Controller**: Maintains durable metadata for storage. - **Compute Controller**: Orchestrates compute resources and manages system state. On startup, `environmentd` will create several built-in clusters. - **console**: Web-based administration interface, deployed as a Deployment. #### Instance responsibilities A Materialize instance manages: - **SQL objects**: Sources, views, materialized views, indexes, sinks - **Schemas and databases**: Logical organization of objects - **User connections**: SQL client connections and authentication - **Catalog metadata**: System information about all objects and configuration - **Compute orchestration**: Coordination of work across clusters and replicas #### Deploying with the operator To deploy Materialize instances with the operator, create and apply Materialize custom resource definitions(CRDs). For a full list of fields available for the Materialize CR, see [Materialize CRD Field Descriptions](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/materialize-crd-field-descriptions/index.md). **v1alpha1:**

v1alpha1 is the default CRD version for the Materialize Helm chart. The Terraform modules default to v1 starting in v4.0.0. With v1alpha1, instance rollouts require manually rotating a UUID.

```yaml apiVersion: materialize.cloud/v1alpha1 kind: Materialize metadata: name: 12345678-1234-1234-1234-123456789012 namespace: materialize-environment spec: environmentdImageRef: materialize/environmentd:v26.47.1 # ... additional fields omitted for brevity ``` **v1:**

The v1 CRD is available starting in v26.30. It is opt-in for the Helm chart and the default for the Terraform modules starting in v4.0.0. See Adopting the v1 CRD to enable it.

```yaml apiVersion: materialize.cloud/v1 kind: Materialize metadata: name: 12345678-1234-1234-1234-123456789012 namespace: materialize-environment spec: environmentdImageRef: materialize/environmentd:v26.47.1 # ... additional fields omitted for brevity ``` When you first apply the Materialize custom resource, the operator automatically creates all required Kubernetes resources. #### Modifying the custom resource **v1alpha1:**

v1alpha1 is the default CRD version for the Materialize Helm chart. The Terraform modules default to v1 starting in v4.0.0. With v1alpha1, instance rollouts require manually rotating a UUID.

To modify a custom resource, update the CRD with your changes, including the `requestRollout` field with a new UUID value. When you apply the CRD, the operator will roll out the changes. > **Note:** If you do not specify a new `requestRollout` UUID, the operator > watches for updates but does not roll out the changes. **v1:**

The v1 CRD is available starting in v26.30. It is opt-in for the Helm chart and the default for the Terraform modules starting in v4.0.0. See Adopting the v1 CRD to enable it.

To modify a custom resource, update the CRD with your changes. When you apply the CRD, the operator will roll out the changes. For a full list of fields available for the Materialize CR, see [Materialize CRD Field Descriptions](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/materialize-crd-field-descriptions/index.md). See also: - [Upgrade Overview](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/upgrading/index.md) #### Connecting to an instance Once deployed, you interact with a Materialize instance through the Materialize Console or standard PostgreSQL-compatible tools and drivers: ```bash # Connect with psql psql "postgres://materialize@:6875/materialize" ``` Once connected, you can issue SQL commands to create sources, define views, run queries, and manage the database: ```sql -- Create a source CREATE SOURCE my_source FROM KAFKA ...; -- Create a materialized view CREATE MATERIALIZED VIEW my_view AS SELECT ... FROM my_source ...; -- Query the view SELECT * FROM my_view; ``` ### Clusters and Replicas Clusters are isolated pools of compute resources that execute workloads in Materialize. They provide resource isolation and fault tolerance for your data processing pipelines. For a comprehensive overview of clusters in Materialize, see the [Clusters concept page](https://preview.materialize.com/materialize/40269/markdown-docs/fundamentals/concepts/clusters/index.md). #### Cluster architecture - **Clusters**: Logical groupings of compute resources dedicated to specific workloads (sources, sinks, indexes, materialized views, queries) - **Replicas**: Physical instantiations of a cluster's compute resources, deployed as Kubernetes StatefulSets Each replica contains identical compute resources and processes the same data independently, providing fault tolerance and high availability. #### Kubernetes resources When you create a cluster with one or more replicas in Materialize, the instance coordinates with the operator to create: - One or more **StatefulSet** resources (one per replica) - **Pods** within each StatefulSet that execute the actual compute workload - **Persistent volumes** (if configured) for scratch disk space For example: ```sql -- Create a cluster with 2 replicas CREATE CLUSTER my_cluster SIZE = '100cc', REPLICATION FACTOR = 2; ``` This creates two separate StatefulSets in Kubernetes, each running compute processes. #### Managing clusters You interact with clusters primarily through SQL: ```sql -- Create a cluster CREATE CLUSTER ingest_cluster SIZE = '50cc', REPLICATION FACTOR = 1; -- Use the previous cluster for a source CREATE SOURCE my_source IN CLUSTER ingest_cluster FROM KAFKA ...; -- Create a cluster for materialized views CREATE CLUSTER compute_cluster SIZE = '100cc', REPLICATION FACTOR = 2; -- Use the previous cluster for a materialized view CREATE MATERIALIZED VIEW my_view IN CLUSTER compute_cluster AS SELECT ... FROM my_source ...; -- Resize a cluster ALTER CLUSTER compute_cluster SET (SIZE = '200cc'); ``` Materialize handles the underlying Kubernetes resource creation and management automatically. ### Workflow The following outlines the workflow process, summarizing how the various components work together: 1. **Install the Helm chart**: This deploys the Materialize Operator to your Kubernetes cluster. 1. **Create a Materialize instance**: Apply a Materialize custom resource. The operator detects this and creates all necessary Kubernetes resources, including the `environmentd`, `balancerd`, and `console` pods. 1. **Connect to the instance**: Use the Materialize Console on port 8080 to connect to the `console` service endpoint or SQL client on port 6875 to connect to the `balancerd` service endpoint. If authentication is enabled, you must first connect to the Materialize Console and set up users. 1. **Create clusters**: Issue SQL commands to create clusters. Materialize coordinates with the operator to provision StatefulSets for replicas. 1. **Run your workloads**: Create sources, materialized views, indexes, and sinks on your clusters. ## Related pages - [Installation guides](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/installation/index.md) - [Materialize Operator Configuration](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/operator-configuration/index.md) - [Materialize CRD Field Descriptions](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/materialize-crd-field-descriptions/index.md) - [Operational guidelines](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/deployment-guidelines/index.md) - [Clusters concept page](https://preview.materialize.com/materialize/40269/markdown-docs/fundamentals/concepts/clusters/index.md) - [Materialize architecture overview](https://preview.materialize.com/materialize/40269/markdown-docs/fundamentals/concepts/index.md) ## Pages in this section - [Appendix](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/appendix/index.md) - [Configure single sign-on](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/sso/index.md): Choose between OIDC and advanced single sign-on (SSO) for Self-Managed Materialize. - [Configuring System Parameters](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/configuration-system-parameters/index.md): How to configure system parameters for Materialize using a Kubernetes ConfigMap - [Deployment guidelines](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/deployment-guidelines/index.md) - [FAQ](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/faq/index.md): Frequently asked questions about self-managed deployments. - [Installation](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/installation/index.md): Installation guides for Self-Managed Materialize. - [Materialize CRD Field Descriptions](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/materialize-crd-field-descriptions/index.md): Reference page on Materialize CRD Fields - [Materialize Operator Configuration](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/operator-configuration/index.md): Configuration reference for the Materialize Operator Helm chart - [Query History](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/query-history/index.md): How query history and statement logging are configured in self-managed Materialize deployments - [Self-managed release versions](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/release-versions/index.md) - [Troubleshooting](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/troubleshooting/index.md) - [Upgrading](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/upgrading/index.md): Upgrading Self-Managed Materialize. - [Usage (Self-Managed)](https://preview.materialize.com/materialize/40269/markdown-docs/self-managed-deployments/usage/index.md): Overview of the resource usage for Self-Managed Materialize.