Paid-Acquisition RAG + Agent — Design & Post-Cutover Deploy Plan
Created: 2026-07-17 · Owner: Marita · Status: DESIGN ONLY this pass. Nothing deployed. The netcup datacenter box is mid-cutover — do NOT deploy there yet. This document is the build spec for after cutover.
Design doctrine (from the estate operating rules): "The LLM builds it. Math runs it." The LLM is a carpenter that writes deterministic code once and acts as a thin, grounded judgment node; deterministic code does the repeated work forever. Deterministic-first, LLM as a node, minimize spend, zero silent errors (verify + failure-pings), draft-and-approve on anything that touches a live system. LLM only via the Max bridge (http://100.95.59.64:9876/run, model sonnet) — never a metered API key. Read-only on live ad data.
Part A — The hybrid RAG
A1. Why hybrid (not pure vector)
Paid-acquisition questions split into three retrieval shapes, so retrieval must be three-headed:
- Conceptual / best-practice ("how does LinkedIn CAPI dedupe with the Insight Tag?") → semantic embeddings over the knowledge pack + canon.
- Exact-term / doc-lookup ("utm_medium value for Reddit", "Original Source drilldown", error strings, property names) → keyword / BM25 over the same corpus (embeddings miss exact tokens).
- Numeric / structured ("LinkedIn CPC this month", "cost-per-opportunity by channel", "benchmark for B2B CPL") → structured store (tables), not prose retrieval.
A single vector index answers #1 well, #2 poorly, #3 not at all. Hence semantic + keyword + structured, fused by a reranker.
A2. Corpus & chunking
- Sources:
PaidAcquisition/KB/*.md(knowledge pack),01-CANON.md(canon),00-DECISIONS.md, and client audit docs (e.g. MSAIPAID-ACQ-AUDIT.md). Canonical platform docs are linked + summarized, not scraped wholesale (freshness + copyright). - Chunking: by markdown heading (H2/H3), ~300–800 tokens, with parent-doc + heading breadcrumb kept on each chunk. Small enough to rank, large enough to carry a complete claim + its citation.
- Every chunk carries metadata:
source_url,publisher,published_or_updated_date,sub_area(platform / measurement / strategy / benchmark / creative),b2b_or_ecom,client_scope(global vs a specific client),ingested_at,confidence.
A3. Provenance & freshness (hard requirements)
- Provenance tags on every chunk → every answer cites source URL + date. No citation, no claim (mirrors the knowledge-pack rule).
- Freshness pipeline: each canon source has a
review_bydate (platform docs = 90 days; benchmark reports = at publisher cadence; strategy = 180 days). A deterministic weekly job flags chunks pastreview_by, re-fetches the source, diffs it, and pings if it changed materially (e.g., a platform deprecates an API). Stale-but-unreviewed chunks are down-weighted at retrieval and labeled "verify — may be outdated" in answers. - Recency-aware ranking: on tied relevance, newer wins; a 2024 platform-API doc is demoted vs a 2026 one for the same claim.
A4. Retrieval + reranking pipeline
- Query → intent classifier (deterministic rules + light LLM) routes to {conceptual, exact, numeric, mixed}.
- Run semantic (embeddings, top-k≈20) and keyword/BM25 (top-k≈20) in parallel; if numeric, also query the structured store.
- Fuse candidate sets (Reciprocal Rank Fusion) → rerank (cross-encoder or LLM-as-reranker via bridge) to top-5–8.
- Apply freshness + confidence + client-scope filters (never leak one client's data into another's answer — per-client isolation is enforced in code, not prompt).
- Assemble a grounded context block (chunks + citations) → hand to the agent's judgment node.
A5. Structured data layer
A small SQLite/Postgres store holds: canon registry (source, date, review_by, sub_area), benchmark tables (channel → metric → value → date → source), and per-client ad-metric snapshots (read-only imports; see Agent). The agent computes metrics from this store deterministically; the LLM never does arithmetic on money.
A6. Eval set (real questions — the accuracy contract)
The RAG+agent is graded against these before it is trusted. Each has a known-good, cited answer. (Full set lives with the code; seed below.)
- "Does zero LinkedIn platform conversions mean LinkedIn isn't converting?" → No; measurement artifact; cite CAPI/secondary-conversion + bounce sources.
- "Where do Google's GA4 form-fills get lost before HubSpot?" → source-mapping + lifecycle-clobber + UTM-not-persisted; cite HubSpot source docs.
- "Correct utm_medium for LinkedIn paid vs Google search vs Reddit?" → paid-social / cpc / paid-social; cite UTM standard.
- "Is LinkedIn ~$94 ABX CPC ever justified?" → yes when ACV + close-rate support it; cite B2B benchmark + LTV:CAC.
- "What must be true before we kill a channel?" → the D6 decision gate.
- "How should ~$40k/mo split across Google/LinkedIn/Reddit for B2B industrial?" → full-funnel staged allocation, gated on tracking; cite allocation source.
- "What's a current B2B CPL benchmark and is MSAI's fair given ACV?" → cite dated benchmark.
- "Server-side vs browser tracking — why does it matter here?" → ad-blocker/ITP/consent durability; cite server-side GTM/CAPI. Scoring: (a) correct decision/direction, (b) every claim cited with in-date source, (c) numbers pulled from structured store not hallucinated, (d) refuses/faces-flag when data insufficient.
Part B — The specialized Paid-Acquisition agent
Thin LLM judgment wrapped in thick deterministic scaffolding. The agent is a pipeline of deterministic nodes with two small LLM nodes (retrieve-rerank assist, and final grounded synthesis).
B1. Deterministic scaffolding (the "math runs it" core)
- Data pull (read-only): connectors to ad data already flowing to the estate — Windsor.ai (ad-platform spend/clicks/impressions/conversions), GA4, HubSpot (contacts/deals/source), and the ad platforms' reporting where available. Read-only, no writes, no live-account mutation. Pulls land in the structured store as dated snapshots.
- Metric computation: deterministic functions compute CPC, CPL, CTR, cost-per-opportunity, cost-per-SQL, CAC, MER, ROAS, blended CPC, click→session→contact reconciliation ratios. All arithmetic is code; the LLM never computes money.
- Audit checks (rule engine): encodes the diagnosis playbook as deterministic checks, e.g.:
- conversions set to secondary/observational? → flag "zero-conversion artifact risk"
- CAPI / server-side present? → flag durability gap
- UTMs on live final URLs match the standard? (string diff) → flag stripped/missing
- paid contacts landing as Offline/Direct in HubSpot? → flag source-map break
- GA4 sessions ≪ platform clicks (order-of-magnitude)? → flag pre-tag bounce / classification break
- landing-page load > 2.5s? → flag bounce-before-tag
- three-way reconciliation gap beyond tolerance? → flag Each check emits a finding with severity + the canon citation for the fix.
- Benchmark join: each computed metric is joined to the dated benchmark table → "above/below benchmark, source X (date)."
B2. Thin LLM judgment (grounded + cited)
The LLM node receives: the deterministic findings, the computed metrics + benchmarks, and the RAG grounded-context block. It only:
- explains findings in plain language,
- prioritizes/sequences fixes,
- writes the recommendation narrative,
- and must cite every external claim to a source+date from the context block. It may not introduce a number that isn't in the structured store, nor a claim without a citation. If inputs are insufficient, it says so and lists what's missing (no guessing).
B3. Operating patterns (non-negotiable)
- Draft-and-approve: every output is a draft for human approval; the agent never changes a live ad account, HubSpot config, or budget. Read-only in, draft out.
- Verify-and-explain: every recommendation carries (a) the evidence it's built on, (b) the citation, (c) the acceptance test that would confirm the fix (e.g., "one live test conversion lands in HubSpot with correct source"). Zero silent errors: if a data pull fails or returns implausible values, it fails loud (failure-ping) rather than emitting a confident wrong number.
- Result-feedback loop: after a recommendation is acted on (human marks it done), the agent re-pulls the affected metrics on the next snapshot and reports actual vs expected; validated learnings are appended to
LEARNINGS.md(Tried / Result / Reuse), and proven patterns get promoted into the knowledge pack and the rule engine. This is how the system compounds. - Per-client isolation: connectors, credentials, and the structured store are scoped per client; the RAG filters by
client_scope; no cross-client leakage.
B4. Agent I/O contract
- In: a decision from
00-DECISIONS.md(D1–D7) + client scope + a date range. - Out: a cited draft answer = {findings (severity-ranked), computed metrics vs benchmarks, recommendation, acceptance tests, open data gaps}. Rendered visual-first (status tiles / funnel / channel table), not a text wall, per house style.
Part C — Post-cutover deploy plan
Do not deploy until the netcup datacenter box cutover is complete. Sequence:
- Now (this pass, done): knowledge pack + canon + decisions + this design live as files on the NAS (
Z:\Knowledge\Playbooks\PaidAcquisition\). No services, no live-data writes. First real audit (MSAI) produced from existing read-only data. - Cutover-gated preflight: confirm netcup cutover done + the concurrency/reliability rules in force (max 2 concurrent CT110-touching agents; sweep orphans). Do not add any cron outside the estate gate.
- Host decision: primary target = the netcup unified datacenter box (all agents/RAG/browsers) once stable; interim fallback = Optiplex CT110 (engine room, always-on) if the RAG is needed before netcup is fully settled. Never the driftbymar VPS (client-facing/lean only).
- Build order: a. Stand up the structured store + canon registry (seed from these files). b. Ingest KB + canon → embeddings + BM25 index; wire freshness job (weekly, America/Mexico_City). c. Implement metric functions + rule-engine checks + benchmark join (deterministic, unit-tested against the MSAI audit numbers as fixtures). d. Wire read-only connectors (Windsor/GA4/HubSpot) as dated-snapshot pulls; verify read-only. e. Wire the LLM judgment node to the Max bridge (sonnet); no metered keys. f. Run the eval set; must pass the accuracy contract before any client use.
- Surface: results flow into the client dashboard (MSAI native dashboard on CT110 / tailnet) — never a bare port link; a tile, not a wall. Draft outputs route to approval.
- Guardrails carried in: read-only on live ad data; draft-and-approve; per-client isolation; failure-pings on every pull; LEARNINGS feedback loop enabled.
Explicitly out of scope this pass: any deployment, any live-account access beyond the already-flowing read-only data, any netcup-box change.