Warehouse loading

A nightly load reads the same rows thirty times.

Feed Snowflake, BigQuery, Redshift, Databricks, or ClickHouse from your production database continuously, so the numbers an analyst reads are minutes old rather than a day old.

30x rows reread per month A nightly full load reads every row again, whether or not it changed.
1x rows read with a tail A snapshot runs once. After that the warehouse receives only what changed.
24h staleness a nightly load buys Every dashboard answers yesterday’s question until the next load lands.
Where the multiple comes from
Batch
Stream

The batch rereads the same block every night. The stream reads it once and then reads only what changed.

What a new column does
Column added
Propagates to the target on the next change that carries it.
Column renamed
Follows the compatibility mode you choose, so downstream queries do not break unannounced.
Type widened
Applied to the target table before the first row that needs it lands.
Connectors for this

None of these 5 connectors ship today. Track dates on the roadmap.

Snowflake · roadmap BigQuery · roadmap Redshift · roadmap Databricks · roadmap ClickHouse · roadmap
Different problem · one runtime

This page is orders.yaml.

orders.yaml YAML 
name: orders
connectors:
  - name: orders-db
    type: source/postgres
    host: "${POSTGRES_HOST}"
    port: "${POSTGRES_PORT}"
    database: "${POSTGRES_DATABASE}"
    user: "${POSTGRES_USER}"
    publication: "${POSTGRES_PUBLICATION}"
    slot_name: "${POSTGRES_SLOT_NAME}"
    secret_refs:
      - { field: password, ref: postgres-password }
  - name: snowflake-out
    type: sink/snowflake
    # required config: /connectors/snowflake
routes:
  - from: orders-db
    to: [snowflake-out]
zipline pipeline apply -f orders.yaml

Not every connector in this pair ships yet. The spec is what it will be · check the roadmap for dates.

All seven use cases Runtime handbook