System-1 Non-Generative Decision Models: Eliminating Output Token Costs and High-Throughput Stream Routing Architecture

Executive Summary & Architectural Comparative Matrix

In large-scale production architectures handling tens of thousands of event streams daily—such as high-throughput news radars, customer communication brokers, or transaction audit logs—deploying general-purpose autoregressive Large Language Models (Generative LLMs / System-2) for classification, routing, and guardrail enforcement introduces two prohibitive bottlenecks: steep output token costs ($0.40 to $2.50 per 1M tokens) and unacceptable latency (1,500ms to 4,000ms caused by token-by-token sequential decoding).

The emergence of System-1 Non-Generative Decision Models represents a fundamental paradigm shift: rather than generating open-ended text, the model executes a Single Forward Pass over the joint context vector, returning a calibrated probability vector across three standardized primitives:

1. noul: Conditional binary probability $P(\text{True} \mid \text{State}) \in [0.0, 1.0]$.
2. choice: Multi-class probability distribution over target routing labels paired with confidence metrics.
3. score: Ordinal rubric ranking across a continuous or discrete scale.

With input pricing as low as $0.042 / 1M tokens, $0 output token costs, and sub-200ms latency (70ms – 200ms), System-1 models unlock up to a 95% reduction in operational expenditure. However, naive adoption—specifically invoking granular 1-to-1 API calls for each individual event—plunges systems into the Context Re-injection Flooding Trap, incinerating millions of redundant tokens on repeated static system prompts.

Below is an architectural comparative matrix evaluating the three primary approaches:

flowchart TD
    subgraph NaiveTrap["❌ Granular 1-1 Anti-Pattern: Context Re-Injection Flooding Trap"]
        R1["Item 1 (50 tokens)"] -->|"Repeats 2,000t Context"| J1["Decision API (2,050 tokens)"]
        R2["Item 2 (50 tokens)"] -->|"Repeats 2,000t Context"| J2["Decision API (2,050 tokens)"]
        Rn["Item 3,700 (50 tokens)"] -->|"Repeats 2,000t Context"| Jn["Decision API (2,050 tokens)"]
        note1["💥 7,400,000 tokens wasted daily solely re-transmitting static system instructions!"]
    end

    subgraph Tier1Standard["✅ Tier-1 Production: 3-Stage Decision Pipeline"]
        IN["Raw Ingestion Stream (Feeds, Webhooks, Inquiries)"] --> T1["Stage 1: Zero-Cost Local Dedup (MinHash / TF-IDF) + Valkey Cache TTL 4d"]
        T1 -->|"Eliminates 45% duplicates (Zero API Cost)"| T2["Stage 2: Token-Budgeted Dynamic Batch Builder (<= 64k Tokens)"]
        T2 -->|"Packs 80-120 items into single payload"| JEV["System-1 Decision Engine (Single Forward Pass ~200ms)"]
        JEV --> T3["Stage 3: Two-Pass Gate Architecture"]
        T3 -->|"Bucket 1: Static Template"| OUT1["Instant Cached Reply (0ms AI)"]
        T3 -->|"Bucket 2: Deep Synthesis"| LLM["System-2 Generative LLM"]
        LLM --> GUARD["System-1 Post-Guardrail (~0.2s: Brand & pricing audit)"]
        GUARD -->|"Pass >= 0.85"| OUT2["Dispatched / Published"]
        T3 -->|"Bucket 3: High Risk"| ESC["Human Escalation Queue"]
    end

1. Naive Baseline & The Context Re-Injection Flooding Trap

The Mathematics of Context Flooding in Granular Setups

In standard classification or scoring pipelines (such as evaluating technical depth or categorizing customer inquiries), engineering teams construct comprehensive system instruction files (system_context.json) defining:

• Target persona, audience criteria, and editorial guidelines.
• Ordinal rubric definitions spanning 1 to 5 stars.
• Explicit exclusionary and high-priority entity rules.

Such structured context schemas typically range between 1,500 and 2,500 tokens. Under the naive 1-by-1 baseline, whenever an incoming record is processed from a feed or message queue: $$\text{Request}_i = \text{SystemContext} \ (2,000 \ \text{tokens}) + \text{Payload}_i \ (50 \ \text{tokens})$$

For a modest daily workload of $N = 3,700$ records: $$\text{Total Tokens Ingested} = 3,700 \times (2,000 + 50) = 7,585,000 \ \text{tokens/day}$$

The effective data payload ratio is remarkably trivial: $$\text{Payload Efficiency Ratio} = \frac{3,700 \times 50}{7,585,000} \approx 2.44%$$

More than 97.5% of the operational token expenditure is squandered repeatedly transmitting identical static instructions over the wire. Furthermore, expressing system guidelines in verbose natural language forces Byte-Pair Encoding (BPE) tokenizers to fragment subwords aggressively, introducing an additional 30%–40% token inflation compared to concise English technical prose.

2. Scale Bottlenecks & Failure Modes

Attempting to scale granular 1-to-1 decision calls to enterprise throughput triggers severe systemic failure modes:

1. Socket Starvation & HTTP Handshake Overhead:
   • Spawning thousands of individual HTTPS connections per minute saturates OS ephemeral socket tables (EADDRNOTAVAIL), introducing significant TCP/TLS handshake latency and wasteful network transit.
2. Cache Obliviousness & Duplicated Computation:
   • Ingested feeds and social feeds display extreme semantic clustering (a single event is syndicated across dozens of publications). Pushing duplicated content through decision endpoints duplicates compute without adding information.
3. Structured Output Degeneracy Under High Concurrency:
   • Forcing Generative LLMs to emit strict JSON schemas under heavy concurrency leads to a non-zero rate of syntax serialization failures (SyntaxError: Unexpected token in JSON), necessitating expensive retry loops.

3. Tier-1 Production Architecture: The 3-Stage Pipeline

To overcome token flooding while maximizing classification precision, Tier-1 engineering architecture decouples ingestion into a structured three-tier pipeline:

Stage 1: Zero-Cost Local Deduplication & Multi-Day Cache

• Incoming text streams are tokenized into n-grams and hashed via 64-bit SimHash or TF-IDF representations in worker memory.
• If an item exhibits $\ge 85%$ similarity (low Hamming distance) against an existing entry processed within the last 4 days (cached in an in-memory Valkey bitset store), the cached evaluation is returned immediately with zero API calls and zero network latency.

Stage 2: Token-Budgeted Dynamic Batch Packing

• Remaining unique items are packed via a Greedy Bin-Packing algorithm into consolidated batches bounded strictly by a $64,000$-token threshold.
• The system instructions are rewritten in terse, unambiguous technical English, condensing static context overhead from 2,500 tokens down to fewer than 800 tokens.
• Across 3,700 daily records, total network requests collapse from 3,700 down to fewer than 20 batches.

Stage 3: Two-Pass Gate Architecture

• Pass 1 (Pre-Router): The System-1 model routes each item into one of three execution paths:
  1. Static Template: Frequent queries or routine lookups $\rightarrow$ Dispatched immediately from deterministic templates at $0 cost.
  2. Generative Synthesis: Nuanced, high-entropy requests $\rightarrow$ Awakens expensive System-2 LLMs for bespoke drafting.
  3. Human Escalation: High-liability disputes or severe anomalies $\rightarrow$ Enqueued for administrative audit.
• Pass 2 (Post-Check Guardrail): Following text generation by System-2, a sub-200ms System-1 verification pass inspects the generated output:
  • Audits factual pricing, inventory boundaries, and SLA statements.
  • Verifies compliance with brand guidelines.
  • When confidence $P(\text{Compliant}) < 0.85$, the response is automatically held for human-in-the-loop sign-off.

// Production Architecture: Token-Budgeted Dynamic Batch Packer
export interface ScoredCandidate {
  id: string;
  body: string;
  estimatedTokens: number;
}

export interface DecisionBatchPayload {
  contextSchema: string;
  records: Array<{ id: string; text: string }>;
}

export class DecisionBatchPacker {
  private readonly maxBatchCapacity: number = 60_000;
  private readonly staticContextWeight: number = 750; // Compact English technical rubric

  public buildOptimalBatches(candidates: ScoredCandidate[]): DecisionBatchPayload[] {
    const batches: DecisionBatchPayload[] = [];
    let currentBucket: ScoredCandidate[] = [];
    let currentTokenLoad = this.staticContextWeight;

    for (const item of candidates) {
      if (currentTokenLoad + item.estimatedTokens > this.maxBatchCapacity && currentBucket.length > 0) {
        batches.push(this.serializePayload(currentBucket));
        currentBucket = [];
        currentTokenLoad = this.staticContextWeight;
      }
      currentBucket.push(item);
      currentTokenLoad += item.estimatedTokens;
    }

    if (currentBucket.length > 0) {
      batches.push(this.serializePayload(currentBucket));
    }

    return batches;
  }

  private serializePayload(items: ScoredCandidate[]): DecisionBatchPayload {
    return {
      contextSchema: "Categorize input streams into: [template_reply, deep_synthesis, human_escalation].",
      records: items.map(record => ({ id: record.id, text: record.body })),
    };
  }
}

4. Production Benchmarks & Quantitative Impact

The table below summarizes performance across an empirical test suite of $3,700$ simulated stream events:

5. Architectural Takeaway & Engineering Philosophy

System-1 Non-Generative Decision Models deliver profound speed and cost advantages by bypassing autoregressive GPU decoding entirely. Yet, the efficacy of specialized primitives is entirely dependent on rigorous systems engineering:

“Do not deploy general-purpose heavy artillery for deterministic routing, and when operating precision sniper tools, never haul an entire empty context trailer behind every bullet.”

By coupling zero-cost local hash deduplication, token-budgeted dynamic batch bin-packing, and a two-pass gate, modern architectures achieve sub-100ms routing and bulletproof guardrails at near-zero marginal cost.

🔬

Related Architecture Monographs

More in AI Engineering & System Architecture ↗

Database & Storage

Oct 30, 2026•14 min read

Hot-Key Lock Contention Forensics: Write Coalescing, Valkey Write-Behind & 16-Slot Sharded Counter

Dissecting relational database lock exhaustion under hot-key write concurrency, architecting a 3-tier write-coalescing pipeline cutting 99.9% of SQL writes, and deploying a 16-slot hash counter reducing database CPU from 100% to 2%.

🎭Chronicles:Ep 140

High-Concurrency Backend & Edge

Oct 17, 2026•14 min read

Eliminating the Real-Time State Glitch Triad: Confirmed Mutation Ledger, SingleFlight Epoch Guard, and Anti-304 Routing

A deep dive into distributed UI state synchronization: neutralizing stale HTTP 304 cache overwrites, in-flight SingleFlight cache race conditions, and double-tap idempotency anomalies in high-frequency PWA workflows.

🎭Chronicles:Ep 124

Cloud, DevOps & Observability

Oct 16, 2026•14 min read

Deconstructing Deterministic Tick Simulation Engines & Capacity Planning via Little's Law: From Write-Wall Traps to Cold Cache Stampedes and Slot Exhaustion

Dissecting the internal mechanics of deterministic discrete-event distributed system simulators and applying Little's Law (L = λW) for capacity planning: eliminating connection slot exhaustion, worker backpressure stalls, and thundering herd failures.

🎭Chronicles:GC Ep 1, GC Ep 2, GC Ep 3, GC Ep 4, GC Ep 5, GC Ep 6

Security, Auth & Access Control

Oct 15, 2026•14 min read

Multi-Temporal Tenancy & The 4-Layer Timezone Shield (Duolingo-Grade Architecture): Navigating Asia/Saigon ICU Traps, Cloudflare IP Spoofing, and Rolling Timezone Bucket Sweeps

Architectural analysis of global multi-timezone systems (Multi-Temporal Tenancy): Resolving Node.js ICU timezone traps, defeating Cloudflare IP header spoofing, and implementing O(1) rolling timezone bucket sweeps to eliminate midnight database thundering herds.

🎭Chronicles:Ep 87, Ep 128, Ep 129, Ep 130, Ep 131

Author & Systems Architect•Da Nang, Vietnam

Duong Phuc Khanh (Dương Phúc Khánh · Kido)

Senior Software Engineer & Systems Architect with 9+ years of experience in high-concurrency distributed architectures, Google Core Web Vitals optimization (0ms TBT, CLS 0.000), and author of the 148+ episode engineering war-story series Kido Chronicles.

khanhdp.com/en ↗•Biography & Career•Google Knowledge Graph•Wikidata Q141718572•ORCID