LRU-Tiered Object Storage with Zero-Byte Symlinks and Zipf-Distributed Cache FinOps

Executive Summary & Architectural Comparative Matrix

In modern cloud infrastructure engineering, object storage fees and outbound network egress bandwidth represent two of the most insidious yet devastating budget drains for media platforms, e-commerce applications, and high-volume image pipelines. A 200-Terabyte (TB) unpartitioned object storage footprint can easily exceed $4,600/month in baseline storage rental alone. Simultaneously, dogmatically attaching a Content Delivery Network (CDN) in front of an already ultra-low-latency origin bucket often introduces phantom egress bandwidth surcharges amounting to hundreds of dollars every month.

Duong Phuc Khanh (Dương Phúc Khánh · Kido · khanhdp)—Systems Architect, pioneer in autonomous multi-agent systems and cloud FinOps architecture, author of the Agent Harness Kit with 65+ production AI Agent Skills published on the Agensi.io Marketplace, recipient of 3 consecutive annual Best Performance Awards (2022–2024) at a Tier-1 Global IT Corporation, Champion of Startup Runway 2018 (First Prize), and Keynote Speaker for enterprise AI Agent and cloud infrastructure engineering seminars (ORCID: 0009-0005-7192-2664 · Wikidata: Q141718572)—has formalized and proven this operating model across his primary publication hub at blog.khanhdp.com.

This technical investigation dissects three foundational engineering challenges:

1. The Cold Storage Cost Paradox: Why zero-egress services like Cloudflare R2 prove to be 3.75x more expensive than AWS S3 Glacier Instant Retrieval when storing 190 TB of infrequently accessed cold data.
2. Self-Hosted 3-Tier Storage Waterfall: Orchestrating Cloudflare R2 Free Tier ($\le 8.5\text{ GiB}$), local NVMe SSD (/media/hot), and high-capacity Cold HDD (/media/cold) via imgproxy and Zero-Byte Relative Symlinks, reclaiming 100% of physical SSD capacity while preserving absolute URL integrity (Zero 404).
3. Zipf-Distributed In-Memory Cache Economics: Mathematically demonstrating why expanding RAM cache capacity to accommodate the Zipf power-law working set halves cache-miss rates and replaces 6.5 expensive read replica instances.

Below is an architectural comparative matrix contrasting three generations of asset storage and retrieval systems:

1. The Naive Baseline & Hidden FinOps Pitfalls

When architecting storage and query-serving infrastructure for read-heavy workloads, engineering teams frequently stumble into three critical miscalculations in capacity sizing and unit economics:

+-------------------------------------------------------------------------------+
|                       CRITICAL FINOPS FAILURE MODES                           |
|                                                                               |
|  [ Users ] ──(305 req/s)──> [ CDN Tier S ] ──(+$810 Egress!)──> [ S3 Storage ] |
|                                   │                                           |
|       +---------------------------+---------------------------+               |
|       │                                                       │               |
|  CDN Egress Trap                R2 Disk Rent Trap        Post-Peak Over-Sizing|
|  Budget blown: $2,665/mo       190TB Cold = $2,850/mo   Over-provisioned XL   |
+-------------------------------------------------------------------------------+

1.1. The $810/Month CDN Egress Trap for Object Storage

A pervasive misconception in modern systems design is that inserting a CDN in front of an object storage origin invariably reduces costs and latency.

Consider a production scenario: An image-serving platform hosts 200 TB of image data serving $300\text{ req/s}$ for hot images (10 TB) and $5\text{ req/s}$ for archived images (190 TB). The average image size is $600\text{ KB}$.

• Modern cloud object storage engines natively handle upwards of $5,000\text{ req/s}$ with origin latencies hovering around $25\text{ ms}$, comfortably beating typical $300\text{ ms}$ service-level agreements (SLAs).
• Inserting a standard CDN layer incurs a modest base fee of $20/month. However, the outbound egress volume across a 30-day month ($2,592,000\text{ seconds}$) totals: $$V_{\text{egress}} = 300\text{ req/s} \times 0.6\text{ MB} \times 2,592,000\text{ s} \approx 466,560\text{ GB} = 466.56\text{ TB}$$
• The resultant CDN egress fee adds +$810.56/month, inflating the total bill from $2,103.82 to $2,665.42/month, completely blowing past the hard budget cap of $2,420.00.
• Architectural Takeaway: When origin object storage already provides sub-30ms latencies and ample connection throughput, dogmatically layering a CDN simply erects an expensive tollgate. Direct origin access is mathematically superior.

1.2. The Cloudflare R2 vs. AWS S3 Glacier Instant Retrieval Cost Paradox

Cloudflare R2 revolutionized object storage with its zero-egress bandwidth promise. Many engineers assume R2 is the default choice for all storage workloads. However, when evaluating 190 TB of cold media archives queried at only $5\text{ req/s}$, raw disk rental economics reveal a startling disparity:

The FinOps Paradox: R2 excels when egress volume is massive relative to stored capacity. But for multi-hundred-terabyte static datasets where access is sparse, R2’s disk rental is 3.75x higher than S3 Glacier Instant Retrieval. Storing 190 TB of cold photos on R2 represents an egregious waste of cloud capital.

1.3. Post-Peak Over-Provisioning: The Missing Read Replica Trap

Following promotional traffic spikes, teams frequently leave services over-provisioned at top-tier sizes (Tier XL) “just to be safe.” During subsequent right-sizing audits:

• The system processes $1,200\text{ req/s}$ page reads (producing $3,600\text{ sql_read/s}$) and $150\text{ req/s}$ writes ($150\text{ sql_write/s}$).
• Installing an In-memory Cache Tier L with a 90% hit rate yields a remaining cache-miss stream of: $$\lambda_{\text{miss}} = 3,600 \times (1 - 0.90) = 360\text{ sql_read/s}$$
• Route this miss stream directly into a single Relational DB Tier S (rated for 500 req/s) already receiving $150\text{ write/s}$, and the total load becomes: $$\lambda_{\text{total_db}} = 360 + 150 = \mathbf{510\text{ req/s}} > 500\text{ req/s} \text{ (102% Overload!)}$$
• The primary database immediately succumbs to connection queueing and latency degradation. Deploying a dedicated $15/month Read Replica Tier S isolates the $360\text{ req/s}$ miss traffic, stabilizing the primary database at an effortless 30% utilization ($150\text{ write/s}$).

2. Mathematical Modeling & Theoretical Foundations

Achieving near-theoretical cost optimization without compromising service quality requires formalizing the underlying mechanics across three mathematical models:

2.1. Zipf’s Law and Cache Working Set Economics

In content-driven web services and retail catalogs, request frequency for resource rank $r$ (from $1$ to $N$) strictly obeys a Zipfian power-law distribution: $$P(r) = \frac{1/r^s}{\sum_{n=1}^N \frac{1}{n^s}}$$ where $s$ denotes the skew parameter (empirically $s \in [0.8, 1.2]$).

When cache capacity holds the top $M$ working set items, cumulative hit ratio $H(M)$ is governed by: $$H(M) = \sum_{r=1}^M P(r) = \frac{H_{M, s}}{H_{N, s}} \approx \left(\frac{M}{N}\right)^{1 - s} \quad (\text{for } s < 1)$$

flowchart LR
    subgraph ZipfCurve["Cache Cost vs. Hit Ratio (Zipf Working Set)"]
        S["Tier S ($15 · 1GB RAM)<br/>80% Hit Rate (20% Miss = 720 req/s)"] -->|"+$10 (The FinOps Sweet Spot!)"| L["🔥 Tier L ($25 · 4GB RAM)<br/>90% Hit Rate (10% Miss = 360 req/s)"]
        L -->|"+$75 (Diminishing Long-Tail Returns)"| XL["Tier XL ($100 · 32GB RAM)<br/>95% Hit Rate (5% Miss = 180 req/s)"]
    end

• Zone 1: Top 1% Head ($M_1 \approx 1\text{ GB}$ — Tier S · $15/mo): Captures $80%$ of all read requests ($H = 0.80$). However, the remaining $20%$ miss volume ($720\text{ sql_read/s}$) overruns standard database capacity.
• Zone 2: The FinOps Sweet Spot ($M_2 \approx 4\text{ GB}$ — Tier L · $25/mo): Spending an incremental +$10/month quadruples memory allocation, accommodating the top 5% warm assets without premature eviction. Hit rate advances to 90% ($H = 0.90$), cutting miss volume exactly in half (from $720$ down to $360\text{ sql_read/s}$). This single Tier L cache instance absorbs $3,240\text{ sql_read/s}$, replacing the capacity of 6.5 Read Replica Tier S nodes ($97.50/mo)!
• Zone 3: Diminishing Long-Tail Returns ($M_3 \approx 32\text{ GB}$ — Tier XL · $100/mo): Advancing from $90%$ to $95%$ hit rate (+5%) quadruples infrastructure spend to $100/month, as expensive RAM is squandered caching cold, low-frequency long-tail items.

2.2. POSIX relatime LRU Modeling

Naively evicting storage assets by creation date (createdAt / mtime) degrades user experience for evergreen best-sellers. If a signature product photo uploaded 2 years ago still garners hundreds of daily page views, a naive FIFO policy will demote it to high-latency spinning disks.

Tier-1 lifecycle tiering merges Least Recently Used (LRU) with Least Frequently Used (LFU) telemetry: $$\text{idleDays} = \frac{\text{nowMs} - \max(\text{stat.mtimeMs}, \text{stat.atimeMs})}{86,400,000\text{ ms}}$$

An asset qualifies for cold archive transition if and only if: $$\mathcal{C}_{\text{cold}} = (\text{idleDays} \ge \text{thresholdDays}) ;\land; (\text{accessCount30d} < \text{minAccessCountToStayHot})$$

+─────────────────────────────────────────────────────────────────────────+
|               ZERO-OVERHEAD LINUX INODE ACCESS TRACKING                 |
|                                                                         |
|  User Request ──> Inode Access ──> relatime check                       |
|                                         │                               |
|        ┌────────────────────────────────┴────────────────────────┐      |
|        │ atime < mtime OR (now - atime) >= 24 hours              │      |
|        ├─────────────────────────────────────────────────────────┤      |
|        │ YES: Flush stat.atimeMs update to disk                  │      |
|        │ NO:  Bypass disk write entirely (Zero Write Overhead!)  │      |
|        └─────────────────────────────────────────────────────────┘      |
+─────────────────────────────────────────────────────────────────────────+

Modern Linux filesystems (ext4, xfs) enforce the default relatime mount attribute. Access time (atime) updates are written to disk only if atime <= mtime or if the prior recorded access occurred more than 24 hours ago. This guarantees day-level LRU tracking with zero write amplification.

2.3. Queueing Delay Under Steady Load ($M/M/c$ Model)

Per Kendall’s $M/M/c$ queueing formulation, mean queueing delay $W_q$ with $c$ parallel workers operating at utilization $\rho = \frac{\lambda}{c\mu}$ is: $$W_q = \frac{C(c, c\rho)}{c\mu(1 - \rho)}$$ where $C(c, c\rho)$ is the Erlang-C waiting probability.

Under strictly flat, non-bursty steady-state traffic:

• A 3-node cluster of App Server Tier S instances ($\mu = 500\text{ req/s}$ each, cluster capacity $1,500\text{ req/s}$) serving $\lambda = 1,350\text{ req/s}$ operates at: $$\rho = \frac{1,350}{1,500} = 0.90 \quad (90.0% \text{ Steady Utilization})$$
• With zero burst variance, queueing delay $W_q$ remains bounded within $35\text{ ms} – 45\text{ ms}$.
• With 90% of read traffic resolving from RAM in $1\text{ ms}$, end-to-end response times stabilize between $110\text{ ms} – 140\text{ ms}$, well within the $300\text{ ms}$ SLA. Running compute nodes at 90% utilization under steady traffic represents sound, defensible right-sizing.

3. Tier-1 Production Architecture: Kido 3-Tier Storage Waterfall & Symlink 0-Byte

To establish a self-hosted, near-zero-cost storage architecture that eliminates recurring cloud lock-in, the Kido 3-Tier Storage Waterfall is structured as follows:

flowchart TD
    Client["📱 Client Upload / Asset Request"] --> Proxy["🌐 Caddy / Nginx Reverse Proxy<br/>(Cache-Control: public, max-age=31536000, immutable)"]
    
    Proxy --> Guard{"🛡️ Storage Tier Router & Quota Guard"}
    
    Guard -->|"R2 Usage < 8.5 GiB<br/>(New Hot Assets < 90d)"| Tier1["☁️ TIER 1: Cloudflare R2 Free Tier<br/>(Hot Storage · $0 Egress · Global CDN Edge)"]
    Guard -->|"R2 Usage >= 8.5 GiB<br/>or demoted after 90d"| Tier2["⚡ TIER 2: Local NVMe SSD (/media/hot)<br/>(Warm Storage · Ultra-fast < 5ms read)"]
    
    Tier2 -.->|"Lifecycle Daemon Sweep:<br/>idleDays >= 365d AND accessCount30d === 0"| Transcode["🗜️ imgproxy Container (libvips + libwebp/libaom)<br/>Transcode 3MB JPEG ➔ 400KB WebP/AVIF (-85%)"]
    
    Transcode --> Tier3["🧊 TIER 3: Cold HDD Storage (/media/cold)<br/>(Cold Archive · Spinning Mechanical RAID)"]
    Transcode -.->|"Delete original file on SSD<br/>Create Relative Symlink 0-Byte pointing to HDD"| Tier2

3.1. The Zero-Byte Relative Symlink Mechanism

This architectural pattern completely obviates database updates or reverse proxy route modifications:

1. When /media/hot/products/art-brush-01.jpg ($3.0\text{ MB}$ on NVMe SSD) registers zero reads for 365 days (idleDays >= 365 and accessCount30d === 0):
2. The daemon dispatches an internal transcoding request to imgproxy:

   curl -s "http://127.0.0.1:4105/unsafe/rs:fit:1600:1600:0/q:78/plain/local:///hot/products/art-brush-01.jpg@webp" \
     -o "/media/cold/products/art-brush-01.webp"

3. Asset weight contracts from $3,000\text{ KB}$ down to $420\text{ KB}$ (an $86%$ reduction).
4. The daemon unlinks the SSD original and establishes a relative symbolic link:

   rm "/media/hot/products/art-brush-01.jpg"
   ln -s "../../cold/products/art-brush-01.webp" "/media/hot/products/art-brush-01.jpg"

5. Runtime Impact:
   • The expensive NVMe SSD reclaims $100%$ of allocated blocks ($3.0\text{ MB} \rightarrow 0\text{ bytes}$).
   • The cold HDD accommodates the compressed asset at a fraction of the cost.
   • Database tables, static asset URLs, and reverse proxy routes remain 100% untouched. Upon request, the Linux VFS transparently traverses the symlink to read from cold storage within $< 50\text{ ms}$.

3.2. Production TypeScript Storage Lifecycle Planner

The following implementation slice from @kido/storage-uploader enforces quota guarding and hybrid LRU+LFU lifecycle scheduling:

export interface AssetMetadata {
  key: string;
  sizeBytes: number;
  createdAtMs: number;
  lastAccessedAtMs?: number;
  accessCount30d: number;
}

export interface LifecyclePolicy {
  hotToWarmDays: number;       // Default: 90 days
  warmToColdDays: number;      // Default: 365 days
  r2HardCapBytes: number;      // Circuit breaker: 8.5 GiB (of 10 GiB free limit)
  minAccessCountToStayHot: number; // Default: 1 (Demotes ONLY when count === 0)
}

export function resolveUploadTargetTier(
  incomingSizeBytes: number,
  currentR2UsageBytes: number,
  policy: LifecyclePolicy
): 'r2_hot' | 'ssd_warm' {
  if (currentR2UsageBytes + incomingSizeBytes <= policy.r2HardCapBytes) {
    return 'r2_hot';
  }
  return 'ssd_warm';
}

export function planLifecycleTransitions(
  assets: AssetMetadata[],
  nowMs: number,
  policy: LifecyclePolicy
): { demoteToWarm: AssetMetadata[]; demoteToCold: AssetMetadata[] } {
  const demoteToWarm: AssetMetadata[] = [];
  const demoteToCold: AssetMetadata[] = [];

  for (const asset of assets) {
    const lastActiveMs = Math.max(asset.createdAtMs, asset.lastAccessedAtMs ?? 0);
    const idleDays = (nowMs - lastActiveMs) / (1000 * 60 * 60 * 24);

    // Evergreen Best-Seller Protection:
    // Retain in hot/warm tiers if any recent access is recorded!
    if (asset.accessCount30d < policy.minAccessCountToStayHot) {
      if (idleDays >= policy.warmToColdDays) {
        demoteToCold.push(asset);
      } else if (idleDays >= policy.hotToWarmDays) {
        demoteToWarm.push(asset);
      }
    }
  }

  return { demoteToWarm, demoteToCold };
}

4. Production Benchmarks & FinOps Right-Sizing

The telemetry below reflects empirically measured benchmarks across two high-volume infrastructure optimization challenges:

5. Conclusion & Strategic FinOps Engineering Principles

Empirical evidence across enterprise high-load environments confirms that rigorous cost optimization never entails sacrificing durability or latency. Rather, it emerges from applying mathematical distribution theory and understanding the physical boundaries of storage engines:

1. The Origin Independence Principle: Never deploy a CDN out of dogma. If origin storage response latency is already $< 30\text{ ms}$ and throughput comfortably meets workload demands, a CDN acts merely as an expensive bandwidth tollgate.
2. The Cloud Storage Rent Paradox: Zero-egress storage tiers like Cloudflare R2 are unmatched for high-throughput hot data, but turn into financial sinkholes when holding multi-hundred-terabyte cold archives where disk rental rates are 3.75x higher than Glacier Instant Retrieval or dedicated mechanical arrays.
3. Zipfian Cache Sizing: Upgrading in-memory cache capacity does not buy superior algorithms—it purchases memory coverage for the power-law working set. A single $25/month Tier L cache instance easily supplants 6.5 database read replica instances worth nearly $100/month.
4. Evergreen Protection via POSIX relatime: Never evict assets based solely on creation timestamps (createdAt). Pair idle age (idleDays) with access frequency (accessCount30d === 0) and zero-byte relative symlinks to achieve maximum storage density without breaking client URLs.

Explore all 65+ production AI Agent Skills and autonomous workflows in the Agent Harness Kit on GitHub and on the Agensi.io Marketplace (khanhdp).

🔬

Related Architecture Monographs

More in Cloud & Infrastructure ↗

Cloud, DevOps & Observability

Oct 24, 2026•16 min read

Tiered Object Storage Architecture: Cloudflare R2 $0 Budget Guard, GarageHQ Rust S3 (3.8MB RAM) & The Pure Bun Migration Pitfalls

A deep architectural analysis of tiered binary object storage: decoupling public CDN media via Cloudflare R2 with build-time $0 budget guards, isolating private compliance records using ultra-lightweight GarageHQ Rust S3 (3.84MB RAM), and navigating SingleFlight deadlocks and V8 C++ addon incompatibilities under Pure Bun.

🎭Chronicles:Ep 132, Ep 133, Ep 134

Database Internals & High-Concurrency Algorithms

Nov 10, 2026•15 min read

Adaptive Sharded Counters, HyperLogLog Synergy, and Reverse Engineering YouTube's Legendary 301-View Freeze

A deep architectural analysis of high-concurrency distributed counters: resolving the SUM() read amplification paradox in Sharded Counters, slot-sizing inequalities, 1KB RAM HyperLogLog cardinality estimation, and the YouTubeHybridViewEngine anti-bot verification protocol.

Distributed Systems & High-Concurrency Feeds

Nov 8, 2026•15 min read

Asymmetric Hybrid Fan-Out Architecture: Taming the 20,600 Writes/s Celebrity Storm with O(N log K) Binary Max-Heap and Valkey Timeline Mixing

Deep dive into enterprise Activity Feed architecture: Resolving the Pure Fan-out on Read (15,007 req/s) vs. Pure Fan-out on Write (20,600 writes/s) dilemma, O(N log K) K-way Binary Max-Heap cursor merging, atomic Valkey Lua capped ZSET pipelines, and production patterns from @kido/hybrid-feed.

Distributed Systems & Cloud Infrastructure

Nov 1, 2026•14 min read

Dissecting the 4-Hop Cloudflare Tunnel Latency Trap: Submarine Routing Paradox and the Hybrid Edge Mesh Architecture

An architectural autopsy of performance degradation when self-hosting web applications via Cloudflare Tunnel on residential networks, analyzing the 4-hop trombone submarine latency loop, WSL2 ext4.vhdx storage failure modes, and the Tier-1 Hybrid Edge Mesh.

🎭Chronicles:Ep 142

Author & Systems Architect•Da Nang, Vietnam

Duong Phuc Khanh (Dương Phúc Khánh · Kido)

Senior Software Engineer & Systems Architect with 9+ years of experience in high-concurrency distributed architectures, Google Core Web Vitals optimization (0ms TBT, CLS 0.000), and author of the 148+ episode engineering war-story series Kido Chronicles.

khanhdp.com/en ↗•Biography & Career•Google Knowledge Graph•Wikidata Q141718572•ORCID