Skip to content

ARC-ADR-055 — Hybrid Object Query System

One line: How the Universal Data Adapter (UDA) orchestrates semantic vector retrieval, structural graph context expansion, and relational/axiomatic filtering into a single, enmeshed Object Model read-seam (IHyperElement) for agent swarms.

Context and Problem Statement

The untool.ai platform is built on an agent-army-first architecture where the unified system ontology is materialized three ways: Knowledge Graph (ArcadeDB), Vector Store (pgvector/Chroma), and the Object Model (IHyperElement).

Currently, our search implementation is decoupled and flat: - The KnowledgeStore (in store.py) executes a simple vector similarity search on flat Chunk elements. - It lacks any graph-context expansion (e.g., fetching a chunk's author, sibling repo, related decisions, or project boundaries). - Agents require context-rich, entity-resolved information (such as a Document carrying its parent Repository, or Evidence carrying its governedByDecision reference).

If RAG (Retrieval-Augmented Generation) is kept as a decoupled silo, agents lose the structural and semantic relationships defined in the ontology, leading to context loss and higher hallucination rates. We need a unified search read-seam that queries the enmeshed Object Model directly.

Decision Drivers

  • Enmeshed RAG: Knowledge assets (Document, Evidence, SourcePill) must be first-class citizens in the ontology and the generated C#/Python object models.
  • Unified Read-Seam: Agents must never call databases or vector indexes directly; they invoke a single Search(QueryString) pattern via the Universal Data Adapter (UDA).
  • Capability-Neutral Integration: The query system must orchestrate queries across heterogeneous backends (VectorCapable stores, GraphCapable engines, and relational databases) without hard-coding specific products.
  • Security & Scope Isolation: Search results must be filtered dynamically based on the calling principal's security clearaces and project bounds (as defined in ARC-ADR-013).

Considered Options

Option A — Siloed Multi-Query Search (Client-Side Orchestrated)

The agent or API controller queries Chroma for vectors, queries ArcadeDB for relationships, and PostgreSQL for metadata/permissions, performing the merging and hydration in application code.

  • Pros:
  • Simple connector implementations.
  • No execution planning logic required inside the UDA.
  • Cons:
  • High latency (multiple round-trips from the client to separate databases).
  • Logic duplication across Python and C# spoke repositories.
  • Lack of a standardized Common Data Model (CDM) Arrow representation for hybrid queries.

Option B — Standardized Single-Database Strategy (Bypass UDA)

Force all RAG knowledge, vector embeddings, and graph relations into a single multi-model database engine (e.g., pgvector in PostgreSQL, or ArcadeDB vector indexes) and bypass the UDA connector separation.

  • Pros:
  • Single database boundary simplifies transactions.
  • High performance due to in-process indexing and filtering.
  • Cons:
  • Limits enterprise connector flexibility; prevents search over external data lakes (e.g., BigQuery, Snowflake) or distributed graph nodes.
  • Violates the congruence-first principle (ARC-ADR-041/ARC-ADR-042) by forcing model structures to match database capabilities.

Option C — UDA-Orchestrated Hybrid Search (Chosen)

The UDA acts as a query router and execution planner, orchestrating a three-phase retrieval pipeline and combining results into hydrated Object Model entities (IHyperElement) using Reciprocal Rank Fusion (RRF).

  • Pros:
  • Single Read-Seam: Standardizes Search(QueryString) across both C# and Python runtimes.
  • Decoupled Engine Topology: Reuses capability mixins (VectorCapable for embeddings, GraphCapable for relationships).
  • Entity Resolved Result Sets: Hydrates raw chunks into fully resolved graph sub-networks before returning them.
  • Cons:
  • High implementation complexity within the UDA query planner.
  • Additional compute overhead for Reciprocal Rank Fusion (RRF) aggregation.

Decision

Adopt Option C. Implement the Hybrid Object Query System as a native query-planning capability of the Universal Data Adapter.

1. Unified Search Contract

Every UDA connection registry exposes a unified Search method:

class SearchRequest(BaseModel):
    query: str
    limit: int = 5
    clearance_scope: list[str] = Field(default_factory=list)
    context_filters: dict[str, Any] = Field(default_factory=dict)

class SearchResult(BaseModel):
    element_id: str
    element_type: str  # "Document" | "Evidence" | "SourcePill"
    score: float
    properties: dict[str, Any]
    context_nodes: list[dict[str, Any]]

2. Three-Phase Execution Pipeline

The UDA planner coordinates retrieval across three distinct phases:

                  ┌──────────────────────────────┐
                  │      Search(QueryString)     │
                  └──────────────┬───────────────┘
                                 │
                     Phase 1: Dense Semantic
                                 ▼
                  ┌──────────────────────────────┐
                  │   Vector-Capable Connector   │
                  │   ( pgvector / ChromaDB )    │
                  └──────────────┬───────────────┘
                                 │
                   Retrieves Top-N Candidate RIDs
                                 ▼
                     Phase 2: Graph Expansion
                                 ▼
                  ┌──────────────────────────────┐
                  │   Graph-Capable Connector    │
                  │        ( ArcadeDB )          │
                  └──────────────┬───────────────┘
                                 │
                   Hydrates Context / Metadata
                                 ▼
                    Phase 3: Relational Filter
                                 ▼
                  ┌──────────────────────────────┐
                  │  Axiomatic Security & RRF    │
                  │       Rank Aggregator        │
                  └──────────────┬───────────────┘
                                 │
                                 ▼
                     Hydrated IHyperElements

Phase 1: Dense Vector Retrieval (Semantic)

  1. The UDA computes the query embedding JIT using a shared local embedder (OpenVINO/local API).
  2. It queries the configured VectorCapable connector for the top $N$ candidate identifiers (RIDs or UUIDs) based on cosine similarity: $$\text{Score}_{\text{vector}} = \cos(\mathbf{q}, \mathbf{d})$$

Phase 2: Graph Context Expansion (Structural)

  1. Taking the top $N$ candidate identifiers, the UDA queries the GraphCapable connector (ArcadeDB) to execute a traversal (depth = 1..2).
  2. Traversal hydrates the surrounding ontology nodes:
  3. Document ── partOf ──▶ SystemComponent (Repository / Epic)
  4. Evidence ── provesRelation ──▶ agent-teaming / crosstalk-bridge
  5. Document ── governedBy ──▶ Decision

Phase 3: Relational / Axiomatic Filtering & RRF Scoring

  1. Clearance Filtering: The UDA filters out any entities whose dataSensitivity or scope properties exceed the caller's clearance_scope (e.g. bypassing proprietary code nodes if the agent is running in an open-source workspace).
  2. Version Decay: Appoints a decaying weight to older document states based on the Hybrid Logical Clock timestamp (ARC-ADR-038).
  3. Reciprocal Rank Fusion (RRF): Ranks the hydrated nodes by combining vector similarity ranks and graph centrality ranks: $$RRF(d) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$ Where $M$ is the retrieval phases (semantic ranking, graph centrality degree ranking), $r_m(d)$ is the position of document $d$ in phase $m$, and $k$ is the smoothing constant (default: 60).

3. Emitted Object Hydration

The combined Arrow record batch is converted directly into typed domain classes (Document, Evidence, SourcePill) using middle-core/backend-core generated code projections, making search results instantly traversable via dot-notation:

// Example agent usage in middle-core
var results = await uda.Search("agent collaboration patterns", limit: 3);
foreach (var doc in results.OfType<Document>()) {
    Console.WriteLine($"Found doc: {doc.Name} in Repo: {doc.Repository.Name}");
}

Consequences

  • + Congruence-First Execution: Graph models remain decoupled from how vector engines index bytes.
  • + Reduced Hallucination: Agents receive complete graph context networks, not just disjoint text chunks.
  • + Single Security Gate: Authentication and scope isolation are enforced once at the UDA layer rather than duplicated across DB connectors.
  • − Query Latency: Multi-phase execution adds overhead (mitigated by parallelizing Phase 1 & Phase 2 where feasible, and caching results via ARC-ADR-012).
  • − Engine Complexity: Requires the UDA engine to maintain a unified execution planner and dependency resolver.

Alternatives Considered

  • Siloed Search: Rejected as it shifts the integration burden to agent application developers, leading to divergent schemas and split-brain query behaviors.
  • Graph-Only RAG (No Vectors): Standardizing purely on graph keyword matching (like full-text indexes in Neo4j/ArcadeDB). Rejected because it fails to capture semantic meaning across cross-modal queries (e.g., matching a text prompt to an image chunk).