Skip to content

Event Spine Integration and Ingest Job events

This ADR documents the integration of NATS JetStream as the fleet's async event spine, wiring it to the backend-core RAG ingest jobs lifecycle.

Status

proposed

Decision

  1. Broker & Client: Use NATS JetStream as the inter-service messaging plane. We use the official nats-py client library.
  2. Subjects & Streams:
  3. Define the ingest stream mapping to the wildcard subject ingest.*.
  4. Publish three CloudEvent types under the ingest stream:
    • ingest.queued.v1 (subject: ingest.queued): Emitted when a job enters the async indexing queue.
    • ingest.done.v1 (subject: ingest.done): Emitted when a job finishes indexing text/images.
    • ingest.failed.v1 (subject: ingest.failed): Emitted when a job exhausts all retry attempts.
  5. Envelope:
  6. Adhere to the CloudEvents v1.0 standard envelope, sharing the same schema structure as the LLM FinOps and evaluation trace events (specversion, id, type, source, subject, time, datacontenttype, data).
  7. Idempotency & DBOS Step Boundary:
  8. In production, we run under DBOS durable workflows. The JetStream publish calls should act as idempotent operations within the workflow context (DBOS @DBOS.step).
  9. For local development where DBOS is not present, we use standard async loop task scheduling.
  10. Configuration Guard:
  11. Event publishing is guarded by EVENTS_ENABLED configuration setting. It defaults to false until explicitly enabled via the environment.