An agent reasoning over a warehouse snapshot from last night will confidently tell you about an order that shipped this morning. Stream the changes instead.
Retrieval quality gets treated as a ranking problem, and sometimes it is. The more common failure is duller: the store the agent reads was loaded on a schedule, and the world moved since. No amount of reranking recovers a row that is not there.
A deleted record that never left the store is worse than a missing one. The agent finds it, believes it, and cites it. Streaming the delete is the same work as streaming the insert, which is why it costs nothing to get right.
Change what you store and replay the stream to fill it, rather than waiting for the next load to catch up. The shape of the store stops being a decision you have to get right the first time.
None of these 3 stores ship yet, and the dates live on the roadmap.
name: orders
connectors:
- name: orders-db
type: source/postgres
host: "${POSTGRES_HOST}"
port: "${POSTGRES_PORT}"
database: "${POSTGRES_DATABASE}"
user: "${POSTGRES_USER}"
publication: "${POSTGRES_PUBLICATION}"
slot_name: "${POSTGRES_SLOT_NAME}"
secret_refs:
- { field: password, ref: postgres-password }
- name: clickhouse-out
type: sink/clickhouse
# required config: /connectors/clickhouse
routes:
- from: orders-db
to: [clickhouse-out]
Not every connector in this pair ships yet. The spec is what it will be · check the roadmap for dates.