Ephemeral Architecture

Recorded 2026-07-20. The two-model split architecture for Align's paid product. Companion to Deployment (tiers), Trust (invariants), and prep/DIALOG-QUALITY.md (the measured baselines this builds on).

1 · The problem this solvesCopy Anchor Link

The COMPLIANT plane split (Deployment §5b) keeps the artefact, RAG and graph inside the customer boundary with a Knowledge Resident, while an amnesiac Moderator runs the room. Measured cost of that split (prep/DIALOG-QUALITY.md, 5×): team-referential questions collapse from 100% → 50% — because the Resident never sees the team chat, so "does the document support Priya's proposal?" fails. Self-contained and answer-anaphoric questions were unaffected; the deficit was isolated entirely to references that point at what a teammate said.

2 · The hypothesisCopy Anchor Link

Move reference-resolution to the plane where the conversation lives. The Moderator already sees the team chat, so it should resolve team-references into a self-contained, document-blind query before it crosses to the Resident — the Resident then only ever receives clean factual questions.

Secondary hypothesis: a per-room ephemeral key can do double duty — authenticate the cross-plane channel (fixing the "KXP is unauthenticated loopback" gap) and seal the Moderator's in-room working memory so that destroying the key at room exit crypto-shreds the room's entire working set. This lets us relax strict amnesia (Moderator may hold artefact-derived facts during the room) in exchange for a provable "nothing survives room exit" guarantee.

Falsifiable prediction: acc_team moves 50% → toward baseline (100%) while self and anaphora hold; a crypto-shred check passes.

3 · Reference links + what Align incorporatedCopy Anchor Link

The design is grounded in four established lines of work. Each entry records what was adopted into the implementation and what was not (kept candid — several papers we cite for the idea, not the full method).

Query rewriting / de-referencing (the accuracy mechanism)

• Ma et al. 2023, Query Rewriting for Retrieval-Augmented LLMs — arXiv:2305.14283 — https://arxiv.org/abs/2305.14283
  • ✅ Incorporated: the core Rewrite→Retrieve→Read shape — decontextualize the utterance into a standalone query before retrieval. This is exactly reference-resolver.js.
  • ✅ Adapted (our addition): we place the rewriter on the Moderator plane (where team chat lives), not co-located with the retriever — the paper assumes one agent sees everything; our split forces the rewrite to the only plane that holds the referent. We also constrain it to resolve to substance not label, and to fire only on team-attributable references, never bare pronouns (those are left to the Resident's own memory) — both are our findings, not in the paper.
  • ❌ Not used: the paper's trainable rewriter (RL-tuned T5). Ours is a zero-shot prompt on the Moderator model — good enough at n=5, retrainable later.
• DMQR-RAG (diverse multi-query rewriting) — https://arxiv.org/html/2411.13154v1
  • ❌ Not used: we emit a single standalone query, not a diverse set. Noted as a future lever if recall becomes the bottleneck.

Crypto-shredding / self-destructing data (the ephemerality mechanism)

• NIST SP 800-88 Rev.1 — cryptographic erase as purge — https://www.goworkwize.com/blog/what-is-nist-800-88-guide
  • ✅ Incorporated: the exact definition we implement — destroy the key, ciphertext is unrecoverable. ephemeral-key.js#destroy zeroizes (Buffer.fill(0)) then drops K_room; the sealed fact cache is then permanently unreadable. Proven by the ephemeral suite's crypto-shred check (positive-controlled).
• Crypto-shredding — https://en.wikipedia.org/wiki/Crypto-shredding
  • ✅ Incorporated: per-room key = per-tenant crypto boundary; AES-256-GCM seal for the working set.
• Geambasu et al., Vanish, USENIX Security 2009 — self-destructing data via time-decaying DHT key shares.
  • ✅ Incorporated (concept): "data that self-destructs when its key dies." ❌ Not used (mechanism): no DHT / Shamir key-splitting / time-decay — our key lives exactly as long as the room and is destroyed deterministically at exit, not by timeout erosion.
• EphPub, Toward Robust Ephemeral Publishing — arXiv:1003.5510 — https://arxiv.org/pdf/1003.5510 — cited for the ephemeral-publishing lineage; mechanism not used.
• Forward secrecy (Signal Double Ratchet, TLS 1.3 ephemeral).
  • ✅ Incorporated (property): per-room forward secrecy — a compromised key from one room reveals nothing about another (each K_room is independent randomBytes(32); cross-room MAC verification fails by test).
  • ❌ Not used (mechanism): no ratchet / per-message key evolution — one K_room per room, not per message.

Small+large model collaboration (the business/architecture split)

• Chen et al. 2023, FrugalGPT — arXiv:2305.05176 — https://arxiv.org/abs/2305.05176
  • ✅ Incorporated (principle): heavy intelligence (multi-turn, reference resolution, conversion-driving) on the Moderator; a commodity lite Resident does grounding — the split is a two-tier cascade, and the RESIDENT_MODEL seam lets the Resident be a genuinely small model (measured: Bonsai-8B-Q2K as Resident, prep/README.md).
  • ❌ Not used: FrugalGPT's learned confidence-based router (escalate to a bigger model on low confidence). Our routing is architectural (plane role), not confidence-scored — a candidate future addition.
• Survey on Collaborating Small & Large LMs — https://arxiv.org/pdf/2510.13890 — cited for the taxonomy; positions our split as "role-based collaboration."

LLM agent memory (the Moderator's in-room working memory)

• MemGPT — arXiv:2310.08560 — OS-style context paging.
  • ✅ Incorporated (principle): bounded, session-scoped working memory the Moderator manages (the K_room-sealed fact cache) rather than unbounded context. ❌ Not used: the full virtual-memory paging hierarchy — our cache is a flat sealed key→fact map, capped and shredded at exit, not a paged main/external tiering.
• A-Mem, Agentic Memory for LLM Agents — arXiv:2502.12110 — https://arxiv.org/pdf/2502.12110 — cited for the agent-memory framing; structured/linked memory not adopted (out of scope for a minutes-long room).

Prior-art parallel (what the incumbents do)

• Slack AI escrow-VPC + ephemeral generation ("point-in-time responses not stored on disk") — https://slack.engineering/how-we-built-slack-ai-to-be-secure-and-private/
  • ✅ Matched: ephemeral, non-persisted responses.
  • ✅ Went further: Slack still sends full document text to the model in its VPC; Align keeps the raw artefact fully on-prem with the Resident and sends only grounded facts across the boundary — a stronger posture than "brokered VPC + no-retention promise."
• M365 Copilot grounding inside the service boundary — https://learn.microsoft.com/en-us/microsoft-365/copilot/microsoft-365-copilot-architecture
  • ✅ Matched: access-scoped grounding. ❌ Behind them: identity-grade access control (Entra/Purview) — Align's boundary is the room, not enterprise identity (on the Deployment §Part-1(d) gap list).

4 · The architectureCopy Anchor Link

CUSTOMER BOUNDARY (on-prem)                 MODERATOR PLANE (paid; may be Align-managed)
─────────────────────────────               ─────────────────────────────────────────
LITE RESIDENT                               SMART MODERATOR (the paid value)
  parse · lite-RAG · graph                    room orchestration + value dials
  grounds a STANDALONE question               multi-turn team dialog memory
  → returns grounded facts                    REFERENCE RESOLUTION (rewrite→standalone)
  ~stateless per query                        K_room-sealed fact cache
        ▲                                             │
        │   KXP, HMAC-SHA256 signed +                 │
        └──── payload under K_room ───────────────────┘
              K_room minted by Moderator, adopted by Resident at ingest,
              destroyed by BOTH at room exit  → crypto-shred

Only the resolved standalone question crosses to the Resident — never raw team chat. Only grounded facts come back — never the raw artefact. The Moderator is document-blind: it can be Align-hosted because it never sees the customer's contract.

5 · ImplementationCopy Anchor Link

Flag: TRUST_EPHEMERAL — a MODE layered on TRUST_COMPLIANT, excluded from TRUST_ALL (a blanket must never silently change topology). Modular, one owner per responsibility (the ownership boundary is deliberately relaxed here, so each relaxation lives behind a single seam):

Flow (compliant @h, ephemeral on): resolve reference → check sealed cache → signed KXP ask of the standalone question → seal the answer into the cache → relay for display (still kxp:true-tagged, still amnesia-stubbed in the transcript/archive). Room end → keyring.destroy(room) on both planes.

Key exchange (prototype): Moderator mints K_room, hands it to the Resident once at ingest over loopback (X-Room-Key); both sign/verify thereafter. Production hardening = TLS + a real KEX; the demonstrable properties here are message authentication and crypto-shred.

6 · Findings (5×, llama3.2 Moderator + Resident, MLflow dialog-quality runs)Copy Anchor Link

Hypothesis confirmed. The context-asymmetry gap is fully closed: team 50% → 100% (±0), anaphora held at 100%, self unchanged. Overall 80% → 97% — statistically indistinguishable from single-plane (overlapping error bars). Cost: ~100 ms p50 for the rewrite hop, on team-referential turns only.

Two bugs the harness caught before ship (why we measure, not assume):

1. The resolver first resolved Priya's label ("Falcon extension") not her substance ("extend grace to 90 days") — fixed by prompting for the concrete thing.
2. It intercepted bare-pronoun anaphora the Resident already handled correctly — fixed by triggering only on team-attributable references, never bare pronouns. (This is the key design lesson: the Moderator resolves what the Resident CANNOT see — team chat — and leaves answer-anaphora to the Resident's own dialog memory.)

Trust properties proven (test/harness/ephemeral-test.js, 11/11): HMAC signs/verifies genuine messages; tampered and cross-room MACs rejected; sealed fact readable while keyed, UNRECOVERABLE after key destroy (crypto-shred, positive-controlled); flag semantics (own flag, not in TRUST_ALL).

Full gate at time of writing: 513/513 across 21 suites.

7 · The invariant shift, and the product payoffCopy Anchor Link

Amnesia relaxes from "the Moderator never holds artefact content" to "the Moderator holds artefact-derived facts only within the room, sealed, and nothing survives room exit." That is a different, tunable point on the privacy↔accuracy dial — and it unlocks the paid-product business model:

• Align hosts the smart Moderator (proprietary conversion engine, value dials) — safe to host because it is document-blind, seeing only grounded facts, never the raw contract.
• The customer runs only the commodity lite-Resident on-prem.
• The value dials could expose the privacy↔accuracy choice as an explicit privacy dial across tiers (strict amnesia · ephemeral · managed-remote).

8 · Candid residual risksCopy Anchor Link

• Rewrite can mis-resolve ("Priya's proposal" → wrong thing) and propagate a bad query — measured, low at n=5, but real; keep it in the dialog-quality harness.
• Crypto-shred is only as good as key hygiene — K_room must never touch disk/swap and must be zeroized (it is, Buffer.fill(0)); production needs a memory-locking review.
• Managed-remote Moderator means grounded facts (not raw docs) transit Align infra transiently — must be disclosed; the strict on-prem-Moderator tier stays available for buyers who won't accept even that.