Apache Iceberg

• An AWS account with permissions to create and manage IAM policies and roles.
• An AWS S3 Table bucket in your AWS account. The S3 Table bucket must be in the same AWS region as your Materialize deployment.
• A namespace in the AWS S3 Table bucket. For details on creating namespaces, see AWS S3 documentation: Creating a namespace.

Google Cloud documents the Lakehouse/BigLake setup process here. The parts you’ll need:

• A Google Cloud project with the BigLake API enabled.
• A Google Cloud Storage bucket to serve as the Iceberg warehouse.
• A Lakehouse runtime catalog backed by your warehouse bucket.
  • NOTE: Materialize uses a service account key, not catalog-vended credentials to write Iceberg data files. So you may configure your catalog with either “End-user credentials” or “Credential vending mode”.
• A namespace in the BigLake catalog.

Step 1. Set up permissions in AWS

In AWS, set up permissions to allow Materialize to write data files to the object storage backing your Iceberg catalog. This tutorial uses an IAM policy and IAM role to grant the required permissions. We strongly recommend using role assumption-based authentication to manage access.

Step 1A. Create an IAM policy

Create an IAM policy that allows full access to your S3 Tables API.Replace <S3 table bucket ARN> with the ARN of your S3 table bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3tables:*",
            "Resource": [
                "<S3 table bucket ARN>",
                "<S3 table bucket ARN>/table/*"
            ]
        }
    ]
}

Step 1B. Create an IAM role

Create an IAM role that Materialize can assume.

1. For the Trusted entity type, specify Custom trust policy with the following:

   • Principal: The example uses the Materialize Cloud IAM principal. For self-managed deployments and the Emulator, the principal will differ.
   • ExternalId: "PENDING" is a placeholder and will be updated after creating the AWS connection in Materialize.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::664411391173:role/MaterializeConnection"
               },
               "Action": "sts:AssumeRole",
               "Condition": {
                   "StringEquals": {
                       "sts:ExternalId": "PENDING"
                   }
               }
           }
       ]
   }
   

2. For permissions, add the IAM policy created earlier to grant access to the S3 Tables.

Once you have created the IAM role, copy the role ARN from the AWS console. You’ll use the ARN in the next step.

Step 2. Create an AWS connection in Materialize

In Materialize, create an AWS connection to authenticate with the object storage:

1. Use CREATE CONNECTION ... TO AWS to create an AWS connection, replacing:

   • <IAM role ARN> with your IAM role ARN from step 1
   • <region> with your AWS region (e.g., us-east-1):

   CREATE CONNECTION aws_connection TO AWS (
       ASSUME ROLE ARN = '<IAM role ARN>',
       REGION = '<region>'
   );
   

   For more details on AWS connection options, see CREATE CONNECTION.

2. Fetch the external_id for your connection, replacing <IAM role ARN> with your IAM role ARN:

   SELECT external_id
   FROM mz_internal.mz_aws_connections awsc
   JOIN mz_connections c ON awsc.id = c.id
   WHERE c.name = 'aws_connection'
   AND awsc.assume_role_arn = '<iam-role-arn>';
   

   You will use the external_id to update the IAM role in the next step.

Step 3. Update the IAM role in AWS

Once you have the external_id, update the trust policy for the IAM role created in step 1. Replace "PENDING" with your external ID value. Your IAM trust policy should look like the following (but with your external ID value):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::664411391173:role/MaterializeConnection"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "mz_1234abcd-5678-efgh-9012-ijklmnopqrst_u123"
                }
            }
        }
    ]
}

Step 4. Create an Iceberg catalog connection in Materialize

In Materialize, create an Iceberg catalog connection for the Iceberg sink to use. To create, use CREATE CONNECTION ... TO ICEBERG CATALOG, replacing:

• <region> with your AWS region (e.g., us-east-1) and
• <S3 table bucket ARN> with your AWS S3 Table bucket ARN.

The command uses the AWS connection you created earlier.

CREATE CONNECTION iceberg_catalog_connection TO ICEBERG CATALOG (
    CATALOG TYPE = 's3tablesrest',
    URL = 'https://s3tables.<region>.amazonaws.com/iceberg',
    WAREHOUSE = '<S3 table bucket ARN>',
    AWS CONNECTION = aws_connection
);

Step 1. Set up permissions in GCP

Materialize authenticates to BigLake as a Google Cloud service account you own.

1. Create the service account.
2. Grant the service account these roles on your project:
   • biglake.editor (BigLake Editor)
   • serviceusage.serviceUsageConsumer (Service Usage Consumer)
3. Grant the service account this role on your Iceberg warehouse bucket:
   • storage.objectUser (Storage Object User)
4. Create a service account key.

Step 2. Create a GCP connection in Materialize

In Materialize, create a GCP connection that holds the service account key.

1. Base64-encode the JSON key file. For example, in a shell:

   base64 < key.json
   

   The decode(..., 'base64') form below lets you paste the encoded value into SQL without worrying about embedded newlines or quotes in the JSON.

2. Store the key in a Materialize secret:

   CREATE SECRET gcp_service_account_key
     AS decode('<base64-encoded service account key JSON>', 'base64');
   

3. Create the GCP connection. See CREATE CONNECTION for more on GCP connection options.

   CREATE CONNECTION gcp_connection TO GCP (
       SERVICE ACCOUNT KEY = SECRET gcp_service_account_key
   );
   

Step 3. Create an Iceberg catalog connection in Materialize

In Materialize, create an Iceberg catalog connection for the Iceberg sink to use. Use CREATE CONNECTION ... TO ICEBERG CATALOG, replacing <bucket> with your GCS bucket name.

The command uses the GCP connection you created earlier.

CREATE CONNECTION iceberg_catalog_connection TO ICEBERG CATALOG (
    CATALOG TYPE = 'rest',
    URL = 'https://biglake.googleapis.com/iceberg/v1/restcatalog',
    WAREHOUSE = 'gs://<bucket>',
    GCP CONNECTION = gcp_connection
);