Search index sync

Find the record you renamed two seconds ago.

Push every row change into Elasticsearch or OpenSearch as it happens, so search results match the database instead of trailing a reindex job.

ashgrove ceramics
3 results · reindexed on write, not on a schedule
Ashgrove Ceramics, Studio
doc 88213 · updated 2 seconds ago
Ashgrove Kiln Supply
doc 41902 · updated 6 days ago
Ashgrove Pottery, the old name
doc 88213 · removed from the index
Three things a nightly reindex misses
The rename
A user renames a record and cannot find it by its new name until tomorrow. The row changed; the index did not.
The delete
Incremental reindexing by timestamp cannot see a deleted row, because the row is gone. It stays searchable forever.
The cost
A full reindex is expensive enough that teams run it nightly, which is how a search box ends up a day behind the database.
Connectors for this

Elasticsearch and OpenSearch are both on the roadmap, so there is no search sink to point you at yet. The SDK is how teams build one in the meantime, and the change stream feeding it is the same one every other sink reads.

Build a sink with the SDK See the roadmap
Different problem · one runtime

This page is orders.yaml.

orders.yaml YAML 
name: orders
connectors:
  - name: orders-db
    type: source/postgres
    host: "${POSTGRES_HOST}"
    port: "${POSTGRES_PORT}"
    database: "${POSTGRES_DATABASE}"
    user: "${POSTGRES_USER}"
    publication: "${POSTGRES_PUBLICATION}"
    slot_name: "${POSTGRES_SLOT_NAME}"
    secret_refs:
      - { field: password, ref: postgres-password }
  - name: elasticsearch-out
    type: sink/elasticsearch
    # required config: /connectors/elasticsearch
routes:
  - from: orders-db
    to: [elasticsearch-out]
zipline pipeline apply -f orders.yaml

Not every connector in this pair ships yet. The spec is what it will be · check the roadmap for dates.

All seven use cases Runtime handbook