Organic traffic at any scale.

For multi-location brands losing organic traffic to AI search: an autonomous content pipeline that generates and publishes product-detail pages at any scale — 10 stores or 10,000, zero manual work. Built by TNG Shopper. 11 enterprise retailers across 5 countries · ~10.5M PDPs per month · $0.0006 each. Live trace below.

—

PDPs per month

~10.5M product-location pages

—

Cost per PDP

Self-hosted Gemma 4 MoE on A100

—

Enterprise tenants

—

Countries

—

Surfaces

—

Deterministic pass

Intentional fail-closed gate

The Problem

Content-gen automations are unreliable at scale.

Every content generation pipeline in the market requires human-in-the-loop.

Hallucinations & Drift

Models invent facts and hallucinate references over time, making unsupervised outputs dangerously unreliable for enterprise clients.

Format Instability

LLMs struggle to adhere to strict schema structures (JSON/Markdown) consistently, breaking downstream parsing and rendering logic.

Cost & Latency Spikes

Relying entirely on generative inference for every pipeline step leads to massive API costs and unpredictable execution times at volume.

Architecture Walkthrough

Reading the trace.

Anonymized sample from a recent pipeline run. Scroll to walk through what each line means.

Beat 1 of 4

Every node fires through DEMAS.

Six gates, every block, fail-closed. The lines marked demas are the JIT audit layer intercepting at every node boundary.

Beat 2 of 4

Node 5 is the only inference call.

~11 seconds of generation, ~1 millisecond of validation. The probabilistic surface is small and pinned at one node — every other node is deterministic.

Beat 3 of 4

Failures don’t propagate.

The line marked FAIL halts downstream. Firestore captures the trace; the orchestrator retries with a corrected prompt. Nothing below threshold reaches production.

Beat 4 of 4

~73.5M of these lines per full run.

Eleven tenants, seven nodes, ~10.5M product-location pages — a full run writes about 73.5M telemetry events to Langfuse. The pipeline is silent everywhere else.

Trace log · sample

Multi-tenant 11 enterprise retailers · 5 countries · 234 surfaces
MoE · Multi-LoRA Gemma 4 26B-A4B-it · ~4B active per token · per-tenant LoRA
O-R-A-V validation Observe · Reason · Act · Validate · 68.9% pass by design
DEMAS boundary Deterministic check at every node · fail-closed · JIT retry · Firestore requeue
Data flywheel Every run becomes training data · DPO · RLAIF

Fig. 1 · Pipeline DAG · 2026.05

Every downstream node reasons about geography. Node 1 locks down locale facts — ISO codes, timezone, jurisdiction-specific data — before any of them runs. Without this step, taxonomy mapping, synonym expansion, and content generation drift on stale or invented context.

Locale Resolver validates ISO-3166 codes and timezone offsets and rejects malformed location data. If the gate passes, Grounded Search fetches factual data from verified .gov sources per locale via Gemini. No model call is made until the deterministic resolver returns clean.

Locale → Gate → Web Search → Context

Tenant data lands in inconsistent shapes — partial fields, free-form taxonomies, mixed product types. Node 2 maps everything to a canonical schema so every downstream node sees the same shape.

Schema Validator (Pydantic + AST) enforces strict JSON schema and fails on mismatch. Past the gate, a text classifier maps to internal taxonomy and a vision model extracts context from product images. The two outputs join on the canonical record.

Input → Schema → Classification → Record

Node 3 produces the keyword surface area — locale-aware synonyms and landing page name variants — that Node 4 will filter against search volume. Over-generation is the design; the next node has the volume signal to drop the noise.

Dedup Filter runs exact-match and fuzzy deduplication (SimHash + Levenshtein) on the synonym set before any classifier runs. The LPN Generator then produces locale-tuned variants using a text classification model keyed to the tenant’s product dictionary.

Terms → Dedup → Classification → Set

Node 4 is a hard gate. It drops sub-threshold keywords before they reach Node 5 — where ~80% of run cost lives. Every keyword that survives this node will be generated against; everything else stops here.

Volume Fetcher queries the Google Ads API for keyword search volume per synonym per location and stages results in BigQuery. Keyword Ranker then scores keywords by commercial value per city using NumPy and custom ranking math. No LLM is invoked in this node.

Set → API Query → Math Ranking → Qualified

This is the only sustained inference path in the pipeline. ~80% of run cost lives in this ~11-second window. Every other node is either deterministic Python or a short-call classifier — the probabilistic surface is one node wide and pinned here on purpose.

Template Selector (Jinja2 + constraint set) enforces structural bounds before generation: length, heading count, content block schema. If the template gate passes, the Content Generator produces 10 SEO-optimized content blocks per product-location pair on Gemma 4 MoE with per-tenant LoRA adapters.

Multi-LoRA serving keeps adapter switching at inference time cheap. Tenant isolation runs through the adapter binding, not through separate model weights. Detailed serving topology in the inline deep-dive below.

Qualified → Template → LoRA → 10 Blocks

Inline deep-dive Mixture-of-Experts · Multi-LoRA serving engine

Base model: Gemma 4 26B-A4B-it, a Mixture-of-Experts transformer. The naming convention is literal — 26B total parameters, A4B means ~4B parameters fire per forward pass. A learned router selects 4 experts of 128 per token; the other 124 stay cold. The serving cost matches a dense 4B model; the capacity matches a dense 26B model.

That sparsity is what makes Node 5 affordable at production volume. At a dense 26B equivalent, the unit economics break; at ~4B active, they hold. The full cost derivation lives in the Economics section.

Multi-LoRA serving handles tenant specialization. The base MoE weights stay frozen and shared; per-tenant LoRA adapters layer on top, one per client_id. 11 enterprise retailers across 5 countries means 11 adapters bound at inference time, each carrying tuned weights from previous successful runs of that tenant’s 18 pipeline prompts. Adapter swap is a sparse matrix load, not a model reload — one server, many tenants, no cross-contamination.

1 instance · 11 swappable adapters · sparse matrix load · no cross-contamination

The isolation stack the engine enforces: tenant (adapter_id binding), prompt (scoped system cache), task (DAG node boundaries), KV cache (PagedAttention). Each layer is a separate guarantee — tenant data cannot leak across adapters, and the deterministic Nodes 4 + 6 either side of this one re-validate the contract anyway.

Pure deterministic Python — the final structural defense before output ships. Zero LLM calls. The probabilistic surface ended at Node 5; Node 6 enforces the contracts on what came out.

Format Check runs O(1) checks (regex + blocklist): format, word count, forbidden patterns. Past that gate, the O-R-A-V engine observes diagnostics, reasons about severity, acts with corrections, and validates the final output. Each step is a rule, not a model.

Outcomes are PASS, RETRY, or FAIL. Fail-closed: below-threshold blocks halt downstream, Firestore captures the trace, the orchestrator retries with a corrected prompt. The O-R-A-V mechanism and the boundary-level DEMAS contract are documented inline below.

Draft → Format → O-R-A-V → PASS / RETRY / FAIL

Inline deep-dive O-R-A-V Engine · Observe / Reason / Act / Validate

O-R-A-V is a control-flow pattern, not a model. Node 6 executes it across every content block produced by Node 5. Zero LLM calls — pure rule-based defense-in-depth. The four axes are sequential gates, not parallel scores:

O · Observe — scan diagnostics across the block. Template variable leaks, internal label leaks, transactional language blocklist hits, schema deviations, character-length bounds, empty fields.
R · Reason — assign a severity tier to each diagnostic. A leaked Jinja token is a fatal; a borderline word count is a corrective; a stylistic deviation is a soft signal.
A · Act — apply deterministic corrections where the severity allows it (strip a leak, truncate to bound, retry the block with a corrected prompt). The act layer never re-prompts Node 5 with raw failure noise — only with the diagnostic the rule named.
V · Validate — final gate. The block either ships, retries, or falls through to the failure tier. There is no probabilistic appeal.

Production rate: 68.9% pass across 11 tenants. The mechanism is what this section documents; the Flywheel section explains why the threshold sits there.

Boundary callout DEMAS · Deterministic Evaluation & Monitoring Audit System

DEMAS is not a node — it is the dashed perimeter around the production DAG. It intercepts at every node boundary, not just at Node 6, validating data integrity and schema compliance in real time. Node 6 is where the contract is most visible because the validator and DEMAS share the same hand-off: when a block clears O-R-A-V, DEMAS has already cleared its boundary check.

Fail-closed: rejected outputs never propagate downstream. The boundary halts on a violation; the orchestrator routes the failure based on tier. Block-level failures take ~2× JIT retry with a corrected prompt against the same DEMAS contract. Product-level failures route to Firestore for requeue on the next CRON cycle — the run does not block on a single product’s recovery.

Routing tools: Pydantic + schema for the gate, Langfuse + Firestore for the failure trace. The full failure flow is the audit surface that makes the multi-tenant pipeline auditable across 234 managed websites without per-tenant operator intervention.

JIT consensus audit. Node 7 validates per-block content quality before commit — the qualitative complement to Node 6’s structural checks. Currently in R&D in Langfuse; not yet in the production code repo, surfaced here for completeness.

Quality Threshold guards SLM evaluation by enforcing minimum O-R-A-V scores upstream — nothing reaches the judge that the deterministic validator already rejected. SLM-as-Judge (Gemma 4 SLM) then runs language-agnostic content quality assessment inline with Node 5’s output.

Content → Threshold → SLM Audit → Accept/Reject

Every production run lands in a 3-tier dataset. The spread between quality and failure tiers becomes DPO preference data. The same machinery that gates the production loop generates the training signal.

Fig. 2 · Self-improvement loop · failures are labelled negatives

The 68.9% pass rate at the O-R-A-V/DEMAS boundary is the engine of this loop. The thresholds are tuned so the failure tier stays large enough to feed DPO with a real chosen/rejected spread. A 99% pass rate would starve the loop; 50% would burn cost. 68.9% is the production setpoint. Failures aren’t waste — they’re the labelled half of the training signal, generated by the same deterministic gates that fail the production DAG closed. No human-in-the-loop annotation; the gates do the labelling.

Unit Economics

$0.0006 per page. Here’s the math.

The Writer node (Node 5, the only sustained inference path) runs on a self-hosted Gemma 4 26B-A4B MoE on a single A100 80GB. Fixed monthly VM cost ÷ pages served. Below: cost derivation, then the volume that produces it.

$5.06/hr × 730 hrs/mo = ~$3,694/month (a2-ultragpu-1g · 1× A100 80GB)
~$3,694/month ÷ ~6.16M PDPs/month = ~$0.0006/PDP

11 tenants × ~45K products × ~21 locations = ~10.5M PDPs per full run
~10.5M PDPs × 7 DAG nodes = ~73.5M ops per full run

80%

Deterministic compute

Near-zero marginal cost

Per-token API fees

Self-hosted Gemma 4 MoE

~20%

Actual LLM calls

City + client caching

Organic traffic at any scale.

Content-gen automations are unreliable at scale.

Hallucinations & Drift

Format Instability

Cost & Latency Spikes

Reading the trace.

Every node fires through DEMAS.

Node 5 is the only inference call.

Failures don’t propagate.

~73.5M of these lines per full run.

Node 1 · City DNA · Context Injection

Node 2 · Normalizer · Data Cleansing

Node 3 · Synonyms · Expansion

Node 4 · SV Gate · Volume Filter

Node 5 · Writer · Generation

Node 6 · Validator · Deterministic QA

Node 7 · Evaluator · DEMAS JIT

Every run becomes training data.

$0.0006 per page. Here’s the math.