Paid-Acquisition RAG + Agent — Design & Post-Cutover Deploy Plan

Created: 2026-07-17 · Owner: Marita · Status: DESIGN ONLY this pass. Nothing deployed. The netcup datacenter box is mid-cutover — do NOT deploy there yet. This document is the build spec for after cutover.

Design doctrine (from the estate operating rules): "The LLM builds it. Math runs it." The LLM is a carpenter that writes deterministic code once and acts as a thin, grounded judgment node; deterministic code does the repeated work forever. Deterministic-first, LLM as a node, minimize spend, zero silent errors (verify + failure-pings), draft-and-approve on anything that touches a live system. LLM only via the Max bridge (http://100.95.59.64:9876/run, model sonnet) — never a metered API key. Read-only on live ad data.


Part A — The hybrid RAG

A1. Why hybrid (not pure vector)

Paid-acquisition questions split into three retrieval shapes, so retrieval must be three-headed:

  1. Conceptual / best-practice ("how does LinkedIn CAPI dedupe with the Insight Tag?") → semantic embeddings over the knowledge pack + canon.
  2. Exact-term / doc-lookup ("utm_medium value for Reddit", "Original Source drilldown", error strings, property names) → keyword / BM25 over the same corpus (embeddings miss exact tokens).
  3. Numeric / structured ("LinkedIn CPC this month", "cost-per-opportunity by channel", "benchmark for B2B CPL") → structured store (tables), not prose retrieval.

A single vector index answers #1 well, #2 poorly, #3 not at all. Hence semantic + keyword + structured, fused by a reranker.

A2. Corpus & chunking

A3. Provenance & freshness (hard requirements)

A4. Retrieval + reranking pipeline

  1. Query → intent classifier (deterministic rules + light LLM) routes to {conceptual, exact, numeric, mixed}.
  2. Run semantic (embeddings, top-k≈20) and keyword/BM25 (top-k≈20) in parallel; if numeric, also query the structured store.
  3. Fuse candidate sets (Reciprocal Rank Fusion) → rerank (cross-encoder or LLM-as-reranker via bridge) to top-5–8.
  4. Apply freshness + confidence + client-scope filters (never leak one client's data into another's answer — per-client isolation is enforced in code, not prompt).
  5. Assemble a grounded context block (chunks + citations) → hand to the agent's judgment node.

A5. Structured data layer

A small SQLite/Postgres store holds: canon registry (source, date, review_by, sub_area), benchmark tables (channel → metric → value → date → source), and per-client ad-metric snapshots (read-only imports; see Agent). The agent computes metrics from this store deterministically; the LLM never does arithmetic on money.

A6. Eval set (real questions — the accuracy contract)

The RAG+agent is graded against these before it is trusted. Each has a known-good, cited answer. (Full set lives with the code; seed below.)

  1. "Does zero LinkedIn platform conversions mean LinkedIn isn't converting?" → No; measurement artifact; cite CAPI/secondary-conversion + bounce sources.
  2. "Where do Google's GA4 form-fills get lost before HubSpot?" → source-mapping + lifecycle-clobber + UTM-not-persisted; cite HubSpot source docs.
  3. "Correct utm_medium for LinkedIn paid vs Google search vs Reddit?" → paid-social / cpc / paid-social; cite UTM standard.
  4. "Is LinkedIn ~$94 ABX CPC ever justified?" → yes when ACV + close-rate support it; cite B2B benchmark + LTV:CAC.
  5. "What must be true before we kill a channel?" → the D6 decision gate.
  6. "How should ~$40k/mo split across Google/LinkedIn/Reddit for B2B industrial?" → full-funnel staged allocation, gated on tracking; cite allocation source.
  7. "What's a current B2B CPL benchmark and is MSAI's fair given ACV?" → cite dated benchmark.
  8. "Server-side vs browser tracking — why does it matter here?" → ad-blocker/ITP/consent durability; cite server-side GTM/CAPI. Scoring: (a) correct decision/direction, (b) every claim cited with in-date source, (c) numbers pulled from structured store not hallucinated, (d) refuses/faces-flag when data insufficient.

Part B — The specialized Paid-Acquisition agent

Thin LLM judgment wrapped in thick deterministic scaffolding. The agent is a pipeline of deterministic nodes with two small LLM nodes (retrieve-rerank assist, and final grounded synthesis).

B1. Deterministic scaffolding (the "math runs it" core)

  1. Data pull (read-only): connectors to ad data already flowing to the estate — Windsor.ai (ad-platform spend/clicks/impressions/conversions), GA4, HubSpot (contacts/deals/source), and the ad platforms' reporting where available. Read-only, no writes, no live-account mutation. Pulls land in the structured store as dated snapshots.
  2. Metric computation: deterministic functions compute CPC, CPL, CTR, cost-per-opportunity, cost-per-SQL, CAC, MER, ROAS, blended CPC, click→session→contact reconciliation ratios. All arithmetic is code; the LLM never computes money.
  3. Audit checks (rule engine): encodes the diagnosis playbook as deterministic checks, e.g.:
    • conversions set to secondary/observational? → flag "zero-conversion artifact risk"
    • CAPI / server-side present? → flag durability gap
    • UTMs on live final URLs match the standard? (string diff) → flag stripped/missing
    • paid contacts landing as Offline/Direct in HubSpot? → flag source-map break
    • GA4 sessions ≪ platform clicks (order-of-magnitude)? → flag pre-tag bounce / classification break
    • landing-page load > 2.5s? → flag bounce-before-tag
    • three-way reconciliation gap beyond tolerance? → flag Each check emits a finding with severity + the canon citation for the fix.
  4. Benchmark join: each computed metric is joined to the dated benchmark table → "above/below benchmark, source X (date)."

B2. Thin LLM judgment (grounded + cited)

The LLM node receives: the deterministic findings, the computed metrics + benchmarks, and the RAG grounded-context block. It only:

B3. Operating patterns (non-negotiable)

B4. Agent I/O contract


Part C — Post-cutover deploy plan

Do not deploy until the netcup datacenter box cutover is complete. Sequence:

  1. Now (this pass, done): knowledge pack + canon + decisions + this design live as files on the NAS (Z:\Knowledge\Playbooks\PaidAcquisition\). No services, no live-data writes. First real audit (MSAI) produced from existing read-only data.
  2. Cutover-gated preflight: confirm netcup cutover done + the concurrency/reliability rules in force (max 2 concurrent CT110-touching agents; sweep orphans). Do not add any cron outside the estate gate.
  3. Host decision: primary target = the netcup unified datacenter box (all agents/RAG/browsers) once stable; interim fallback = Optiplex CT110 (engine room, always-on) if the RAG is needed before netcup is fully settled. Never the driftbymar VPS (client-facing/lean only).
  4. Build order: a. Stand up the structured store + canon registry (seed from these files). b. Ingest KB + canon → embeddings + BM25 index; wire freshness job (weekly, America/Mexico_City). c. Implement metric functions + rule-engine checks + benchmark join (deterministic, unit-tested against the MSAI audit numbers as fixtures). d. Wire read-only connectors (Windsor/GA4/HubSpot) as dated-snapshot pulls; verify read-only. e. Wire the LLM judgment node to the Max bridge (sonnet); no metered keys. f. Run the eval set; must pass the accuracy contract before any client use.
  5. Surface: results flow into the client dashboard (MSAI native dashboard on CT110 / tailnet) — never a bare port link; a tile, not a wall. Draft outputs route to approval.
  6. Guardrails carried in: read-only on live ad data; draft-and-approve; per-client isolation; failure-pings on every pull; LEARNINGS feedback loop enabled.

Explicitly out of scope this pass: any deployment, any live-account access beyond the already-flowing read-only data, any netcup-box change.