TexBench

Key-value stores are widely adopted as the storage engine for modern applications as they offer high throughput for writes, support for heterogeneous workloads, and easy tunability. Given the large number of key-value stores available and how widely their performance varies with workload characteristics, finding the suitable data store and tuning for a specific workload and performance target often entails extensive benchmarking and analysis. State-of-the-art key-value benchmarks, such as YCSB, db_bench, and KVBench, however, are unable to capture several key characteristics of modern application workloads, such as dynamically shifting workload characteristics, data with varied degrees of sortedness, or application-specific data formats. Further, existing benchmarking tools do not provide a unified interface to benchmark and compare multiple databases against the same workload. We present TexBench, a unified key-value benchmarking suite that enables benchmarking key-value stores against dynamically shifting and production-like workloads and comparing their performance side by side. TexBench is built on top of Tectonic, a highly configurable, Rust-based key-value workload generator that can generate multi-phased shifting workloads, supports a rich set of operations and operation-specific distributions, variable data sortedness, and custom data formats. TexBench’s unified framework also enables its users to perform an apples-to-apples comparison of multiple databases against the same workload, and compare the benchmarking results readily within a single interface. Lastly, we augment TexBench with an LLM core that allows users to describe a workload in natural language, have that translated into a custom key-value workload, and benchmark and compare multiple databases against it in parallel.

Key-value stores are widely adopted to serve modern heterogeneous workloads, yet selecting the best key-value store for a given workload and target performance remains an incredibly complex and time-consuming task, even for expert DBAs. In practice, this entails extensive, large-scale benchmarking and analysis involving dozens of candidate key-value stores. Comparing databases using existing key-value benchmarks, such as YCSB, KVBench, db_bench, and Tectonic, has several challenges: they require substantial effort to set up the benchmarking environment and connect the databases, are unable to capture the dynamic and complex nature of real-world workloads, or rely on fixed, templated benchmark workloads with limited customization options. We introduce TexBench, a unified key-value benchmarking suite that abstracts the complexities of workload configuration, benchmark setup, and database connections behind a unified interface, enabling users to evaluate multiple key-value stores against production-like workloads and visually compare their performance side by side. TexBench uses Tectonic under the hood to emulate the dynamic and complex properties of real-world workloads, and integrates a large language model (LLM) plugin that captures benchmarking specifications expressed in natural language and translates them into a JSON format interpretable by Tectonic. TexBench currently supports four key-value stores – ScyllaDB, RocksDB, Cassandra, and Redis – and performs up to 16.7x faster in terms of workload generation compared to raw LLM prompting.

Key-value stores are the backbone of many modern SQL- and NoSQL-based data systems, serving a variety of real-world applications. Despite their widespread adoption, existing key-value benchmarks fall short across multiple dimensions when accurately replicating complex and dynamic real-world workloads. In this paper, we introduce Tectonic, a Rust-based, highly configurable, and resource-efficient key-value workload generator designed to model the temporal, structural, and dynamic properties of real-world workloads. Tectonic offers fine-grained control over data access patterns, configurable composite key generation, dynamic workload generation, and generation of workloads with user-specified data sortedness — at 2x higher throughput and up to 84% lower memory footprint than the state-of-the-art.