Home  /  Concepts

Snapshotting

View as Markdown

Snapshotting is the initial sync of a table’s data. It reads from the upstream system and writes the data into Materialize’s storage. The initial snapshot is committed to storage atomically, with all records assigned the same ingestion timestamp.

When snapshotting occurs

When snapshotting occurs depends on the syntax.

Snapshot duration

Snapshotting reads from the upstream system, so its duration depends on the volume of upstream data and the size of the source’s cluster. For upsert sources, snapshotting can be especially resource-intensive (compared to append-only), and large upsert sources can take hours to snapshot.

Queries during snapshotting

Queries on a table that is snapshotting are blocked until its snapshot completes.

  • With the legacy CREATE syntax:

    • None of the tables created as part of CREATE SOURCE ... FOR ... are queryable until they have all finished snapshotting.

    • When altering a source to add a new table (ALTER SOURCE ... ADD SUBSOURCE), only the new table snapshots. The source’s other tables remain queryable. However, ingestion for these tables is temporarily blocked, so they stop advancing until the snapshot completes.

  • With the source-versioning CREATE TABLE FROM SOURCE syntax:

    • None of the tables created within a transaction block are queryable until all their snapshots complete.

    • When you create new tables from a source that already has tables, only the new tables snapshot. The source’s existing tables remain queryable. However, ingestion for the existing tables is temporarily blocked, so they stop advancing until the snapshots for the new tables complete.

Impact on upstream system

Snapshotting has the following upstream impacts:

  • Read load. Snapshotting puts read, CPU, and network load on the upstream system, proportional to the data volume.

  • Change-log retention for CDC database sources. When ingesting data from CDC database sources (PostgreSQL, MySQL, SQL Server), the upstream system must retain its change-log data until Materialize consumes it. During the initial snapshot, changes accumulate from the source’s starting position until the snapshot completes and Materialize has consumed the accumulated changes. A stalled or long-running snapshot can therefore increase disk usage on the upstream database.

Back to top ↑