Full production suite / August 2026
Waltier object-storage benchmark
Four S3-compatible services, the same 200 GiB payload matrix, and one durability-first WAL implementation. The results separate remote performance from the host’s RAM and NVMe caches.
Abstract
S3 Express One Zone is the unambiguous performance leader in this evaluation. General-purpose S3 is the strongest all-around choice when single-AZ placement is not acceptable. Tigris remains a credible value-oriented option: slower, especially at the tail, but fast enough to sustain Waltier’s CAS-based design. Cloudflare R2 passed every correctness gate, yet its write serialization and follower latency leave it outside the competitive set for this workload.
Scope matters. This is one full run per provider from one 3 Gbit/s host path. It compares the tested endpoint, region, account, and date; it does not establish universal vendor performance or availability. S3 Express stores data in one Availability Zone.
| Rank | Provider | Index | Assessment |
|---|
The relative index is deliberately narrow: it aggregates durable commits, WAL pressure, cold reads, and follower behavior from this run. It excludes price and every host-local hot or warm cache rate. The raw comparisons below remain the primary evidence; the index is a compact reading aid, not a universal vendor grade.
Cross-provider comparison
The services do not differ by one constant multiplier. S3 Express changes the latency regime for metadata and small objects; general-purpose S3 closes much of the gap for large objects; Tigris remains workable at a lower performance tier; and R2’s strongest large-read results do not carry into conditional-write or follower workloads.
S3 Express is the decisive leader. This phase is request-latency bound, so proximity and the directory-bucket request path matter more than bandwidth.
The field compresses at 16 MiB because transfer time becomes a larger part of every acknowledgment. S3 and S3 Express are effectively peers on aggregate logical throughput.
Every provider benefits from independent CAS chains, but the absolute operating range still differs sharply. Relative speedup cannot substitute for low single-request latency.
Cold bulk reads are the most competitive part of the result. Followers expose the opposite extreme: repeated authoritative metadata refreshes amplify tail latency.
Run identity and coverage
All four runs used the full schema-4 profile and benchmark commit fdd0e92b8914f9eda5c0e5c18a0eb7105cbef5e0. Each produced 12,044 acknowledged payloads at 4 KiB, 1 MiB, and 16 MiB, for 36,132 acknowledged and verified payloads and approximately 200 GiB of logical data.
| Provider | Region | Run ID | Wall time | Remote bytes | Correctness |
|---|
Every provider returned exactly one CAS create winner and one CAS update winner, converged all follower phases, verified every acknowledged payload key and size, reported zero orphan payloads, and closed its compaction accounting with one intentionally superseded race candidate. Correctness therefore does not distinguish the ranking: every service met the contract, and the report ranks the cost and latency of meeting it.
Durable payload commits
A successful operation includes an immutable payload PUT and an accepted WAL reference. Local cache population happens only after both remote facts are established. The panels separate payload sizes and metrics so small-object request rate, large-object bandwidth, and latency remain legible.
The 4 KiB phase is the cleanest request-overhead test. Payload bytes are negligible, yet each logical acknowledgment still requires immutable-object durability and a conditional WAL update. The resulting spread is therefore mostly a spread in the remote control path.
At 16 MiB, S3 and S3 Express converge near the same logical bandwidth while Tigris and R2 move closer. This does not erase the metadata result; it shows when object transfer begins to dominate the fixed cost of the WAL protocol.
WAL sharding
One WAL is one serialized ETag-CAS chain. Sharding creates independent chains and raises aggregate throughput without creating cross-shard atomicity. S3 Express reaches the knee early; S3 continues scaling through 64 shards, while Tigris and R2 remain bounded by longer remote request latency.
Absolute throughput and relative scaling answer different questions. S3 Express extracts less incremental gain after 32 shards because it is already above one thousand operations per second. S3’s 64-shard result continues climbing and is the strongest general-purpose result.
Sharding is not a correctness shortcut. Each object retains its own total order and ETag fence; no figure claims an atomic order spanning all shards. Applications must choose the partition key that makes shard-local ordering meaningful.
Cold remote reads
Cold phases use zero-capacity payload caches and therefore exercise the object store directly. S3 Express dominates small-object reads and the writer-side large-object path. General-purpose S3 is consistently second. Tigris and R2 are closer on bulk reads than their WAL results suggest, but their tails remain materially longer.
The role-level comparison matters because the writer child runs alone, while reader head and replicas run concurrently. A provider that holds its per-role rate under the concurrent group is behaving differently from one whose aggregate host path becomes the constraint.
Logical MiB/s is payload size multiplied by completed reads. It is not packet capture throughput and includes the normal Waltier digest verification and process path. The fixed 3 Gbit/s internet channel is a ceiling for the host, not a promise that every role can reach it simultaneously.
What “warm” measures
Warm is not one disk benchmark. The 4 KiB cohort is only about 47 MiB and fits in every RAM cache. The 1 MiB cohort is 11.76 GiB: it fits the writer’s 20 GiB RAM allocation, but not a reader’s 4 GiB allocation. The 16 MiB cohort is about 188 GiB and forces every role through RAM promotion and NVMe. The last panel is therefore the strongest measure of the intended warm storage tier.
The provider labels in warm panels identify the run that produced the host-local observation. They should not be read as remote object-store throughput.
The apparent provider differences in 4 KiB warm throughput are run-to-run host variance: that complete cohort is resident in memory. Treating those values as object-store scores would be a category error.
The 16 MiB cohort is the useful operational case. It is larger than every configured cache, so steady reads cycle through bounded RAM and NVMe and exercise promotion, verification, eviction, and local concurrency together.
Live followers
Follower lag is measured from writer acknowledgment to first application. Steady-state histograms exclude the declared 10-second hint-loss and 20-second follower-pause windows. Hints are wakeups only; periodic conditional GET remains authoritative. S3 Express’s hint-assisted lag is in a different class, while R2’s longer request latency propagates directly into convergence.
Hint-assisted followers are not trusting an in-memory notification. A hint only advances the next refresh; every follower still fetches accepted WAL state from the provider. This keeps missed or coalesced hints safe and makes the poll-only curve a real fallback baseline.
The 250 ms comparison shows whether hints buy useful latency without an aggressive poll interval. On S3 Express and S3 they do. On a high-latency metadata path, the remote refresh itself dominates enough that wakeup policy cannot recover the lost ground.
Provider findings
These sections expose the role-level and per-second detail behind the shared comparisons. The history figures use independent linear panels and recorded measurement seconds only.
AWS S3 Express One Zone
The performance-first choice. It leads the durable, WAL, small-object read, and hint-assisted follower measurements by large margins. That advantage is architectural: directory buckets place data in one Availability Zone, so this is not a drop-in durability-domain substitute for general-purpose S3.
Its main limitation is not visible in an ops/s chart. A directory bucket is explicitly zonal, so a production design must decide whether the application already owns cross-AZ replication or can accept that placement boundary. Within that boundary, the results show a qualitatively different metadata path rather than a modest optimization.
AWS S3
The all-around leader among multi-AZ services tested. It scales WAL traffic cleanly, posts competitive large-object throughput, and keeps follower tails controlled without S3 Express’s placement constraint.
S3’s strongest case is balance. It does not match Express on request-heavy phases, but its 64-shard WAL result and cold large-object reads are strong enough that the general-purpose durability model does not imply a slow WAL. Its weakness in the cost model is public internet egress, not request pricing.
Tigris
The premium-economy result: it preserves the conditional-write semantics Waltier needs and completes the suite correctly, while accepting lower request throughput and longer tails in exchange for simpler, egress-free economics.
Tigris remains inside the useful design space: WAL sharding continues to add throughput, cold reads are credible, and every follower converges. The report’s price envelope is where that lower tier becomes strategically relevant, because 250 TB of public egress carries no transfer charge under the cited standard pricing.
Cloudflare R2
R2 is not ready for this particular sustained CAS-heavy WAL pattern on the tested path. It passes correctness and reaches isolated parity on some large-object reads, but WAL serialization, durable small writes, and follower tails make it a poor fit for the design measured here.
The distinction is workload-specific, not a claim that R2 cannot store the data. The service completes the full correctness suite and its large-object reads can approach Tigris or S3 in isolated cells. The failure is economic performance for frequent conditional metadata mutation: cheap capacity cannot compensate for the observed control-path latency.
Timeline limitation
The schema records per-second samples inside measured phases, but it does not timestamp preconditioning, warmup, follower phases, or gaps between phases. The history figures therefore retain phase-local seconds and suite order. They are not stretched to the exact 44–65 minute wall times, and no continuous trace is fabricated.
Method and interpretation
- All runs use the same host, binary commit, 16 dataset writers, payload counts, cache allocations, and timing windows.
- Durable throughput counts only payload-plus-WAL acknowledgments.
- Logical MiB/s is object size multiplied by successful operations, not measured wire bytes.
- The payload read matrix runs after writes stop and is not a simultaneous mixed workload.
- Writer-head read children finish before reader-head and two replicas run concurrently.
- All rankings are relative to this dataset and exclude price and host-local cache rates.
- One run per provider does not quantify run-to-run variance.
- See the environment, pricing model, and chart data for audit detail.