Test Case</th>	Legacy Slot Migration</th>	Atomic Slot Migration</th>	Speedup</th> </tr>
No Load: 3 to 4 shards</td>	1m42.089s</td>	0m10.723s</td>	9.52x</td> </tr>
No Load: 4 to 3 shards</td>	1m20.270s</td>	0m9.507s</td>	8.44x</td> </tr>
Heavy Load: 3 to 4 shards</td>	2m27.276s</td>	0m30.995s</td>	4.75x</td> </tr>
Heavy Load: 4 to 3 shards</td>	2m5.328s</td>	0m27.105s</td>	4.62x</td> </tr> </table> The main culprit here is unnecessary network round trips (RTTs) in legacy slot migration. Each slot requires:</p> 2 RTTs to call `SETSLOT</code> and begin the migration</li>` Each batch of keys in a slot requires: 1 RTT for CLUSTER GETKEYSINSLOT</code></li> 1 RTT for MIGRATE</code></li> 1 RTT for the actual migration of the key batch from source to target node</li> </ul> </li> 2 RTTs to call SETSLOT</code> and end the migration</li> </ul> Those round trip times add up. For this test case where we have:</p> 4096 slots to move</li> 160 keys per slot</li> valkey-cli</code>'s default batch size of 10</li> </ul> We need:</p> </p> Even with a 300 microsecond round trip time, legacy slot migration spends over a minute just waiting for those 212,992 round trips.</p> By removing this overhead, atomic slot migration is now only bounded by the speed that one node can push data to another, achieving much faster end-to-end latency.</p> Client Impact: How Would Applications Respond?</h3> The experiment measured the throughput of a simulated valkey-py</a> workload with 1:10 set/get ratio while doing each scaling event. Over three trial throughput averages are shown below.</p> </p> Despite atomic slot migration causing a more acute throughput hit, you can see the recovery of the client application is much faster due to far fewer topology changes and an overall lower end-to-end latency. Each topology change needs to be handled by the Valkey client, so the quicker the topology changes are made, the sooner the impact ends. By collapsing the topology changes and performing atomic handover, atomic slot migration leads to less client impact overall than legacy slot migration.</p> Under the Hood: State Machines and Control Commands</h2> To coordinate the complex dance between the two nodes, a new internal command, CLUSTER SYNCSLOTS</code></a>, is introduced. This command orchestrates the state machine with the following sub-commands:</p> CLUSTER SYNCSLOTS ESTABLISH SOURCE <source-node-id> NAME <unique-migration-name> SLOTSRANGE <start> <end> ...</code> Informs the target node of an in progress slot migration and begins tracking the current connection as a slot migration link.</li> </ul> </li> CLUSTER SYNCSLOTS SNAPSHOT-EOF</code> Used as a marker to inform the target the full snapshot of the hash slot contents have been sent.</li> </ul> </li> CLUSTER SYNCSLOTS REQUEST-PAUSE</code> Informs the source node that the target has received all of the snapshot and is ready to proceed.</li> </ul> </li> CLUSTER SYNCSLOTS PAUSED</code> Used as a marker to inform the target no more mutations should occur as the source has paused mutations.</li> </ul> </li> CLUSTER SYNCSLOTS REQUEST-FAILOVER</code> Informs the source node that the target is fully caught up and ready to take over the hash slots.</li> </ul> </li> CLUSTER SYNCSLOTS FAILOVER-GRANTED</code> Informs the target node that the source node is still paused and takeover can be safely performed.</li> </ul> </li> CLUSTER SYNCSLOTS FINISH</code> Inform the replica of the target node that a migration is completed (or failed).</li> </ul> </li> CLUSTER SYNCSLOTS CAPA</code> Reserved command allowing capability negotiation.</li> </ul> </li> </ul> The diagram below shows how CLUSTER SYNCSLOTS</code> is used internally to drive a slot migration from start to finish:</p> </p> Get Started Today!</h2> This new Atomic slot migration is a massive step forward for Valkey cluster management. It provides a faster, more reliable, and overall easier mechanism for resharding your data.</p> So, go download Valkey 9.0</a> and try Atomic Slot Migration for yourself! A huge thank you to everyone in the community who contributed to the design and implementation.</p> Valkey 9.0: innovation, features, and improvements. 2025-10-21T00:00:01+00:00 For Valkey's second major release, Valkey 9.0 brings innovation, long-requested features, and improvements to classic features updated for today’s workloads. Read on to find out all the team packed into this release.</p> Atomic Slot Migrations</h2> Atomic slot migrations fundamentally changes how Valkey migrates data from node-to-node inside the cluster. Prior to Valkey 9.0, data migrated in the cluster key-by-key. This approach works for most situations, but corner cases can lead to degraded performance, operational headaches, and, at worst, blocked node migrations and lost data.</p> Key-by-key migration uses a move-then-delete sequence. Performance issues arise when a client tries to access a key during a partially migrated state: if the migration hasn’t completed, the client may not know if the key resides on a the original node or the new node and leading to a condition that has more network hops and additional processing. Worse, in a multi-key operation, if one key resides in the original node and another in the new node, Valkey cannot properly execute the command, so it requires the client to retry the request until the data resides on a single node, leading to a mini-outage where Valkey still has the data but it is inaccessible until the migration is complete for the affected data. Finally, in a situation where Valkey is attempting to migrate a very large key (such as collections in data types like sorted sets, sets, or lists) from one node to another, the entire key may be too large to be accepted by the target node’s input buffer leading to a blocked migration that needs manual intervention. To unblock the migration you need to increase the input buffer limit or end up losing data either through forcing the slot assignment or deleting the key.</p> In Valkey, keys are bundled into one of 16,384 ‘slots’ and each node takes one or more slots. In Valkey 9.0 instead of being key-by-key, Valkey migrates entire slots at a time, atomically moving the slot from one node to another using the AOF format. AOF can send individual items in a collection instead of the whole key. Consequently, this prevents large collections from causing latency spikes when they are being processed during migration. The new atomic slot migration doesn’t migrate keys directly, instead, the move-then-delete sequence is on the entire slot; the original node still retains all the keys and data until the entire slot migration is complete avoiding the pre-Valkey 9.0 issues with redirects or retries. For more information, check outthe video from our recent Keyspace conference talk recording about Valkey 9.0</a> and look for an upcoming deep dive on atomic slot migrations.</p> Hash Field Expiration</h2> The hash data type allows you to neatly tie together data with multiple fields under one key. But because all this data lives attached to a single key, the expiry was, until Valkey 9.0, all-or-nothing: you couldn't expire fields individually. For users who needed some</em> of the data to expire, the limitation made users look to awkward hacks with multiple keys, compounding the complexity and increasing the memory footprint of the data. Valkey 9.0 now address this gap by adding the following commands:</p> HEXPIRE</a></li> HEXPIREAT</a></li> HEXPIRETIME</a></li> HGETEX</a></li> HPERSIST</a></li> HPEXPIRE</a></li> HPEXPIREAT</a></li> HPEXPIRETIME</a></li> HPTTL</a></li> HSETEX</a></li> HTTL</a></li> </ul> You can read more about how hash field expiration works in Ran Shidlansik's deep dive</a> on the subject.</p> Numbered Databases in cluster mode</h2> Numbered databases allow you to separate data and avoid key name clashes: each database contains keys unique to that database. This is an old feature dating back to the very first version of the preceding project. However, before Valkey 9.0, numbered databases were severely limited with cluster mode being restricted to having a single database (db 0). Without cluster support, numbered databases were discouraged as using them limited you to never scaling beyond a single node.</p> Based on user feedback and reconsideration by the team, Valkey 9.0 breaks from the preceding project and adds full support for numbered databases in cluster mode. Numbered databases have a whole host of use cases and some very handy clustering characteristics, find out more in the feature write up</a>.</p> Much, much, more</h2> Valkey 9.0 brings numerous small changes and improvements:</p> 1 Billion Requests/Second with Large Clusters</a></strong>: Improvements in the resilience of large clusters, enabling scaling to 2,000 nodes and achieving over 1 billion requests per second,</li> Pipeline Memory Prefetch</a></strong>: Memory prefetching when pipelining, yielding up to 40% higher throughput,</li> Un-deprecation</a></strong>: In a similar vein to numbered databases, the Valkey project re-evaluated 25 previously deprecated commands and, based on the stance of API backward compatibility, restored the usage recommendation for these commands.</li> Zero copy Responses</a></strong>: Large requests avoid internal memory copying, yielding up to 20% higher throughput,</li> Multipath TCP</a></strong>: Adds Multipath TCP support which can reduce latency by 25%.</li> SIMD for BITCOUNT and HyperLogLog</a></strong>: optimizations that yield up to a 200% higher throughput,</li> By Polygon for Geospatial Indices</a></strong>: query location by a specified polygon,</li> Conditional Delete</a></strong>: Adds the DELIFEQ</a> command that deletes the key if the value is equal to a specified value,</li> CLIENT LIST Filtering</a></strong>: options to filter CLIENT LIST</a> using flags, name, idle, library name/version, database, IP, and capabilities.</li> </ul> And, perhaps, most importantly, a new, whimsical LOLWUT</a> generative art piece especially for version 9.</p> Get it today</h2> Valkey 9.0 was built by the collaborative efforts of dozens of contributors. Make sure and grab Valkey 9.0 today as a binary, container</a>, or build it from source</a> and watch for it in your favourite Linux distribution. Feel free to post a question on our GitHub discussions or Slack and if you find a bug, make sure and tell the team as an issue.</p> Scaling a Valkey Cluster to 1 Billion Request per Second 2025-10-20T00:00:00+00:00 The upcoming Valkey 9.0 release brings major improvements in the resilience of large clusters, enabling scaling to 2,000 nodes and achieving over 1 billion requests per second, all while ensuring bounded recovery time. In this blog post, we provide an overview of how the Valkey clustering system works, along with the architectural improvements and rigorous testing that made this level of scale possible.</p> Valkey’s standalone configuration is a single server setup with optional replicas for availability, but all writes flow to one primary. It is one process, one dataset, zero coordination, blazing fast and simple to operate when a single machine’s CPU, memory, and NIC can carry the load. However, Valkey at-scale moves past single node limits.</p> Valkey’s cluster mode shards the keyspace into 16,384 hash slots</strong> and spreads them across multiple primaries with replicas for redundancy. Clients are cluster aware: they route commands directly to the node owning the slot and follow redirections during resharding or failover. The result is horizontal scalability, balanced throughput, and built in fault tolerance without a central coordinator. Behind the scenes, the cluster bus keeps this distributed system coherent, coordinating membership, gossip, and failover. </p> Cluster Bus Overview</h2> Under the hood, nodes coordinate over the cluster bus a persistent TCP mesh with a lightweight, gossip-based protocol. It handles MEET</code> (discovery), PING</code>/PONG</code> heartbeats with piggybacked cluster topology, quorum-based FAIL</code> decisions, replica promotion elections, and epoch-based conflict resolution so the cluster converges cleanly after partitions. Since membership, health, and slot ownership flow through gossip, it allows Valkey to scale to large node counts while remaining resilient to node and network failures.</p> Membership Discovery and Information Dissemination (Gossip)</h3> Nodes join a Valkey cluster by sending a MEET</code> message to any existing member (a seed). From there, the cluster bus, a mesh of persistent TCP links which spreads membership and topology using lightweight gossip of peers piggybacked on periodic PING/PONG heartbeats. This peer-to-peer exchange quickly converges every node in the cluster and which slots each primary owns, without any central coordinator. </p> Cluster Bus: Failure Detection, Failover, and Epoch Update</h3> When a node misses heartbeats beyond the configurable node-timeout, peers mark it potentially failed (PFAIL</code> - suspect). If a quorum of primaries observes the timeout, they declare FAIL</code> and initiate failover: the replica(s) of the failed primary becomes candidate to take over the ownership of the shard and send vote requests to all the primaries. Primaries respond and the winner is promoted to serve the affected slots. Clients that hit the old owner receive MOVED/ASK</code> and transparently retry against the new primary.</p> </p> Major improvements to scale Cluster Bus in Valkey</h2> Across multiple major/minor versions, the community has improved cluster stability a lot and have made multiple changes to handle various failure scenarios to heal the system in a steady manner and avoid any manual intervention. Here are few of the interesting improvements:</p> Multiple primary failures</strong> Failover operation across cluster is serialized. Hence, only one shard can undergo failover at a given point in time. When there are multiple primary outages across the cluster, it used to lead to collision of vote request from replica candidates of impacted shards. Due to the collision, the votes would get split and the cluster won’t reach to consensus of promoting a replica in a given cycle. With a large number of primaries failing, the cluster won’t heal automatically and further an administrator needs to manually intervene. This problem was tackled by Binbin Zhu</a> in Valkey 8.1 by introducing a ranking mechanism</a> to each shard based on lexicographic order of the shard ID. With the ranking algorithm, the shard with highest ranking would go sooner and the one with lower ranking would add additional delay to start the election for the given shard. This helped to get a consistent recovery time in a multiple primary outage.</li> Reconnection attempt storm to unavailable nodes</strong> Profiling revealed that with multiple node failures, a large chunk of compute goes into attempting to reconnect with the already failed nodes. Each node attempts to reconnect to all the failed nodes every 100ms. This lead to significant compute usage when there are hundreds of failed nodes in the cluster. In order to prevent the cluster from getting overwhelmed, Sarthak Aggarwal</a> implemented a throttling mechanism</a> which allows enough reconnect attempts within a configured cluster node timeout, while ensuring the server node is not overwhelmed.</li> Optimized failure report tracking</strong> Profiling also revealed that when hundreds of nodes fail simultaneously, the surviving nodes spend a significant amount of time processing and cleaning up redundant failure reports. For example, after 499 out of 2000 nodes were killed, the remaining 1501 nodes continued to gossip about each failed node and exchange reports, even after those nodes had already been marked as failed. Seungmin Lee</a> optimized the addition/removal of failure report</a> by using a radix tree to store the failure report time information rounded to every second and group multiple report together. This helps with cleaning up expired failure report efficiently as well. Further optimizations were also made to avoid duplicate failure report processing and save CPU cycles.</li> Pub/Sub System</strong> Cluster bus system is also used for pub/sub operations</a> to provide a simplified interface for a client to connect to any node to publish data and subscribers connected on any node would receive the data. The data is transported via the cluster bus. This is quite an interesting usage of the cluster bus. However, the metadata overhead of each packet is roughly 2 KB which is quite large for small pub/sub messages. The observation was the packet header was large due to the slot ownership information (16384 bits = 2048 bytes). And that information was irrelevant for a pub/sub message. Hence, Roshan Khatri</a> introduced a light weight message header</a> (~30 bytes) to be used for efficient message transfer across nodes. This allowed pub/sub to scale better with large clusters. Valkey also has a sharded pub/sub system which keeps the data traffic of shard channels to a given shard which is an major improvement over the global pub/sub system in cluster mode. This was also onboarded to the light weight message header.</li> </ul> Valkey 9.0 has plenty of other improvements to increase the overall stability of the clustering system. All these enhancements allowed us to scale to 2,000 nodes with bounded recovery time during network partitions and below we have documented the benchmark setup and the throughput we were able to attain.</p> Benchmarking</h2> In order to scale the Valkey cluster to 1 billion requests per second (RPS) for write command, we decided to choose a SET type command to accurately reflect the scale. We have seen previous experiments where a single instance was able to achieve more than 1 million RPS</a> so the goal was to reach 1 billion RPS with a 2,000 node cluster, where each shard has 1 primary and 1 replica. A replica is added to each shard for better availability.</p> Hardware Configuration</strong> For this experiment, Valkey cluster was deployed on AWS r7g.2xlarge</code> instance type, which is a memory optimized instance, featuring 8 cores and 64 GB memory on an ARM-based (aarch64</code>) architecture. In order to generate enough traffic across all the slots, we used 750 instances of AWS c7g.16xlarge</code>.</p> System Configuration</strong> Note: The core assignments used in this guide are examples. Optimal core selection may vary depending on your specific system configuration and workload.</p> Each of the Valkey server nodes had 8 cores, so we decided to pin 2 cores to ensure interrupt affinity.</p> # identify the network interface, for this run, it was ens5</span></span> IFACE</span>=</span>$(</span>ip</span> route</span> \|</span> awk</span> '/default/ {print $5; exit}'</span>)</span></span> </span> # expose two combined Rx/Tx queues</span></span> sudo</span> ethtool</span> -L</span> "</span>$IFACE</span>" combined</span> 2</span></span> </span> # grab exactly the two IRQ numbers that belong to those queues</span></span> IRQS</span>=</span>($(</span>grep</span> -w</span> "</span>$IFACE</span>" /proc/interrupts</span> \|</span> awk</span> '{gsub(":","",$1); print $1}'</span>))</span></span> IRQ0</span>=</span>${IRQS[0]}</span></span> IRQ1</span>=</span>${IRQS[1]}</span></span> </span> # pin queue-0 interrupt to CPU0</span></span> echo 0</span> \|</span> sudo</span> tee /proc/irq/</span>$IRQ0</span>/smp_affinity_list</span> # bind IRQ0 → CPU 0</span></span> echo 1</span> \|</span> sudo</span> tee /proc/irq/</span>$IRQ1</span>/smp_affinity_list</span> # bind IRQ1 → CPU 1</span></span> sudo</span> systemctl stop irqbalance</span></span></code></pre> We increased the default limit of file descriptors of the instance so that the node is able to handle connections from the Valkey cluster nodes and clients.</p> # increase max conn</span></span> ulimit -n 1048544</span></span></code></pre> With 6 cores left on the node, we decided to assign all the remaining cores to the valkey-server</code>, so that the 6 io-threads</code> (including the main thread</code>) can leverage the most from the instance.</p> # pin 6 cores to valkey-server</span></span> CPUSET</span>=</span>2-7</span></span> sudo</span> cset shield</span> --cpu=</span>$CPUSET</span> --kthread=on</span></span> sudo</span> cset shield</span> --exec</span> taskset</span> -- -c</span> $CPUSET</span> ./valkey-server valkey-cluster.conf</span> --daemonize</span> yes</span></span></code></pre> Server Configuration</strong> We launched each Valkey process with minimal changes to the default configurations.</p> cluster-enabled yes # To enable cluster mode</span></span> cluster-config-file nodes.conf # To persiste the topology information</span></span> cluster-require-full-coverage no # To prioritize availability</span></span> cluster-allow-reads-when-down yes # To allow reads and partial operations to continue even under slot loss</span></span> save "" # Disable periodic snapshots</span></span> io-threads 6 # Allow offloading io operations to separate io-threads</span></span> maxmemory 50gb # Limit maximum memory utilization by the process</span></span></code></pre> Benchmark Configurations</strong> We ran the valkey-benchmark</code> from 750 client instances to generate the required traffic. We used the following parameters for each of those instances.</p> Number of requests (-n): 100M</code></li> Number of clients (-c): 1000</code></li> Test (-t): SET</code></li> Data Size (-d): 512</code></li> Threads: 20</code></li> </ul> </p> Overall observation has been that the throughput grows almost linearly with the overall count of primaries and cluster bus has a minimal overhead with the default node timeout.</p> Recovery</h3> In the same environment, we tested the recovery time of the cluster when multiple primary nodes go down. We picked up to 50% of the primaries and killed them and let the automatic failover process to heal the write outage for a given shard. To simulate primary node failures, we sent SIGKILL</code> to hard-stop the Valkey process, and let the replica take over through automated failovers. The recovery time measured here is from the moment any of the nodes discovered a primary in a PFAIL</code> state until the cluster reported ok and all slots were covered again.</p> </p> Closing thoughts</h2> With all these improvements made in Valkey, a cluster can now scale to 1 billion RPS using 2,000 nodes which is quite a remarkable feat. However, there is plenty of room to improve further. The steady state CPU utilization overhead from the cluster bus message transfer/processing can be reduced further by incorporating the SWIM protocol</a> or moving the cluster bus messsage processing off the main thread into an independent separate thread. The failover logic can be made smarter as well by incorporating the AZ placement of nodes. We would also like to introduce more observability metrics/logs into the system for better manageability. All of these are being linked under the Support Large Cluster</a> issue. Feel free to check it out and add in your suggestions.</p> How Valkey 8.1 Handles 50 Million Sorted Set Inserts 2025-10-02T00:00:01+00:00 When you run infrastructure at scale, the smallest efficiencies compound into massive savings. Sorted sets (ZSETs) are the backing data structure for far more than leaderboards. They're used for time‑ordered feeds, priority queues, recommendation rankings and more. Each entry carries per‑item overhead; when you're inserting tens of millions of items, those bytes accumulate into gigabytes. The latest Valkey 8.1 release introduces a redesigned hash table</a> and other optimizations that promise lower memory usage and higher throughput. In this post, we put Valkey 8.1 under pressure by benchmarking it against Valkey 8.0, inserting 50 million members into a sorted set and measuring memory consumption and throughput along the way.</p> About Momento and Raider.IO</h2> Momento builds enterprise‑grade real‑time data infrastructure. Our flagship service, Momento Cache</a>, allows developers to focus on innovation rather than maintaining complex back‑ends. Behind the curtains, Valkey is one of the primary storage engines we operate. Its memory efficiency and predictable performance make it an ideal choice for mission‑critical workloads. By building on Valkey, we can deliver caching experiences that are fast, reliable, and cost‑effective.</p> One such workload is Raider.IO</a>, the most popular companion site for World of Warcraft players. Raider.IO maintains leaderboards that track millions of characters and guilds, updating instantly whenever a player clears a dungeon or raid. That translates into hundreds of millions of sorted set operations every day. In this environment, downtime or latency isn't acceptable. Memory efficiency and throughput directly shape the player experience. Running Valkey at hyperscale for workloads like Raider.IO means that improvements in Valkey 8.1 immediately impact both infrastructure cost and the ability to absorb peak traffic without breaking a sweat.</p> Benchmark Setup</h2> To provide a fair comparison, both Valkey 8.0 and Valkey 8.1.1 were run on the same hardware:</p> AWS c8g.2xl instance (Graviton4, 8 vCPU, 16 GB RAM)</li> Identical settings for persistence, I/O threads and memory limits</li> 50 million members inserted into a ZSET via pipelined ZADD commands (score=i, member=m:{i})</li> 250,000 item batch size per pipeline</li> Metrics collected every 1,000,000 inserts (flush between runs)</li> </ul> The benchmark code is open sourced</a> and straightforward to reproduce: it connects to both servers, flushes the test key, performs batched inserts, and records used_memory</code>, used_memory_rss</code>, total elapsed time, and throughput after each million inserts. This repeatability mirrors the ethos of Valkey's community - every optimization is measurable, and anyone can verify the results.</p> Memory Usage – 8.1 vs 8.0</h2> Valkey 8.1's redesigned dictionary structure cuts roughly 20–30 bytes per key</a>, and those savings compound with scale. The figure below charts used_memory</code> during the benchmark for both versions:</p> </p> At 1 million inserts, Valkey 8.0 used ~95 MB while Valkey 8.1 used ~81 MB. As the ZSET grew, the gap widened. By 10 million inserts, 8.0 consumed 1.06 GB versus 0.77 GB for 8.1 - a 27% reduction</strong>. At the end of the run (50 million inserts), 8.1 used 3.77 GB compared to 4.83 GB on 8.0, saving 1.06 GB (≈22 %).</p> These numbers align with the release notes. Valkey's 8.1 announcement highlights lower per‑key overheads and improved data structure handling; Linuxiac notes that 8.1's architectural changes can reduce memory footprints by approximately 20 bytes per KV pair</a>, with each pair normally consuming 100 bytes (a 20% reduction!). Our results also confirm a similar improvement on large ZSET workloads.</p> Throughput and Total Time</h2> Memory isn't the only metric that matters. Throughput determines how quickly you can fill or update a set. Although Valkey 8.0 started strong with about 661k inserts per second at the 1M mark, its throughput gradually declined as the data set grew. Valkey 8.1 maintained higher and more consistent throughput, finishing the benchmark at 573k/s compared with 532k/s for 8.0. Overall time to insert 50 million members dropped from 94 seconds on 8.0 to 87 seconds on 8.1, a 7% improvement.</p> The smoother throughput curve is just as important as the raw peak. Spiky performance complicates capacity planning, especially when running close to memory limits. By contrast, Valkey 8.1's combination of improved hash‑table implementation and I/O thread optimizations delivers predictable scaling.</p> Why These Gains Matter</h2> For Raider.IO and similar workloads, these improvements translate directly into better economics and user experience:</p> Lower infrastructure cost - Saving over a gigabyte per 50 M members means fewer nodes are needed to support the same dataset. At hyperscale, this compounds into significant cost reductions.</li> More headroom during peaks - Efficiency gains provide a buffer during seasonal events like game expansions or esports tournaments. When leaderboards surge, there's less risk of hitting memory ceilings.</li> Predictable scaling - A smoother memory growth curve and stable throughput make it easier to forecast when capacity needs to be added and to automate scaling policies.</li> </ul> These benefits don't just apply to leaderboards. Time‑ordered feeds (such as activity streams), job scheduling queues, recommendation rankings, rate‑limiting windows and search scoring all rely on sorted sets. In every case, per‑entry overhead multiplied by millions of entries determines whether you can do more with less. As described in the Linux Foundation press release</a>, upgrading to Valkey 8.1 can reduce memory footprints for common workloads by up to 20%, helping enterprises scale while keeping costs in check.</p> Closing Thoughts</h2> As an operator who lives and breathes real‑time data systems, I'm continually amazed by the pace of innovation in the Valkey community. We didn't tune any configuration knobs to achieve these results - Valkey 8.1's efficiency is built in, and the improvements materialized instantly once we upgraded. On a 50 million‑entry benchmark, the new release used 27% less memory while delivering about 8% higher throughput, and completed the workload seven seconds faster than its predecessor. Those deltas may seem small in isolation, but at hyperscale they compound into transformative savings and more resilient services.</p> If you're running sorted set heavy workloads - whether leaderboards, feeds, queues or scoring engines - I encourage you to upgrade to Valkey 8.1 and run this benchmark yourself. Interested in how Valkey 8.1 stacked up to its competitors? So were we</a>!</p> Our code is open source</a> and available to run on your own hardware. I think you'll be pleasantly surprised by how much headroom you gain and how effortlessly Valkey handles pressure. The power of open source and community‑driven engineering continues to shine through in every release.</p> Introducing Hash Field Expirations 2025-09-30T00:00:00+00:00 One of the great strengths of Valkey has always been its built-in ability to expire keys. This simple but powerful mechanism lets developers keep their datasets fresh, automatically clear caches, or enforce session lifetimes without additional logic. But there has always been one limitation: expiration worked at the level of whole keys. If you stored multiple fields in a hash, you could set a TTL for the hash itself, but you couldn’t say “this field should live for 10 seconds while another field should stick around for 10 minutes.”</p> That limitation forced developers into awkward choices.</p> Imagine you’re running a feature flag system. You want to store all of a customer’s feature toggles inside one hash:</p> Some flags are short-lived experiments, meant to turn off automatically after a few seconds or minutes.</p> </li> Others are long-term rollouts, where the toggle might remain valid for days or weeks.</p> </li> </ul> If you only had key-level expirations, you were forced to compromise. You could put all flags for a customer into a single hash, but then they would all share one expiration time. Or, you could store each flag in its own key so that each could expire independently — but this would explode the number of keys your system has to manage, hurting memory efficiency and increasing overhead for operations.</p> The inability to set different TTLs per field meant developers either built complex cleanup processes outside of Valkey, or gave up flexibility in how expirations were handled.</p> Finding the Right Expiration Model</h2> At first glance, adding field expirations might sound like a simple matter of storing timestamps per field. In order to understand the problem, it’s important to understand how expiration is normally handled in Valkey today.</p> Valkey uses two complementary mechanisms to reclaim expired keys:</p> Lazy expiration:</strong> A key is deleted only when accessed. If you try to read or write an expired key, Valkey notices it has passed its TTL and deletes it immediately. This is cheap, but untouched keys would linger indefinitely and waste memory.</p> </li> Active expiration:</strong> A background cron job runs 10 times per second, sampling a small set of keys with expiration. Expired keys are deleted until a time budget is reached, ensuring memory is reclaimed proactively without introducing latency spikes.</p> </li> </ul> For hash fields, we made a key design choice: we did not implement lazy expiration. Adding expiration checks to every HGET</code></a>, HSET</code></a>, or HDEL</code></a> would have complicated the hot path of hash commands and risked performance regressions. Instead, we extended the active expiration job so that it can now also scan field-level expiration buckets, alongside top-level keys.</p> This approach keeps expiration logic unified, predictable, and free from new latency cliffs. But it also introduced a challenge: how do we efficiently track and clean up expired fields inside hashes that may contain thousands (or millions) of entries, when only a small subset are volatile — and an even smaller fraction are expired at any given time? The challenge is balancing three conflicting goals:</p> Keep memory usage small.</strong> Hashes can grow to millions of fields, so metadata overhead must be minimal.</p> </li> Maintain fast lookups and updates.</strong> Most hash commands run in O(1)</code> time, and expiration tracking can’t change that.</p> </li> Reclaim memory efficiently.</strong> The active expiration job is time-bounded, so we need to minimize wasted CPU cycles spent scanning unexpired fields.</p> </li> </ul> Approaches Considered</h3> "We explored several ways to solve this problem:</p> 1. Secondary hashtable per field expiration</h4> This seemed simple: build a secondary hashtable in each Hash object which maps only volatile fields.</p> </p> This is actually the way volatile generic keys are being tracked. Per each database, Valkey manage a secondary map mapping volatile keys. During active expiration process the existing job scan the secondary map and each key found to pass it's assigned TTL is expired and it's memory is reclaimed. The problem with this design option is the potential inefficiency introduced while scanning many items which should not expire.</p> 2. Radix Tree-based index</h4> Using a radix tree to hold field names plus expirations provides sorted access for free.</p> </p> This is also not a new concept in Valkey. In fact this exact structure is being used to manage client connections blocked on key operations like BLPOP</code></a> and XREAD</code></a>.</p> But the memory overhead per node was high. During experiments we measured more than 54 bytes overhead per each hash field when using this type of index.</p> 3. Global sorted structure</h4> We wanted to have the ability to efficiently scan over fields which are already expired. A good way to achieve this is by using a sorted index. Using a radix tree was possible, but the memory overhead was high. Instead we could use a more lightweight data structure like a skip list, which memory consumption is more bounded and is not governed by the data atrophy.</p> </p> However, This would give O(log N)</code> access to the index which did not work well with our target to keep hash operations constant time complexity.</p> Coarse Buckets with Adaptive Encoding</h3> Instead of tracking every field’s expiration individually, we designed a coarse bucket system. Each field’s timestamp is mapped into a time bucket, represented by a shared “end timestamp.”</p> </p> This solution introduces a semi-sorted data structure which we named 'vset' (stands for "volatile set"). The volatile set manages buckets in different time window resolutions and adaptive encodings. Buckets can split if too many expirations cluster in one interval. This adaptability keeps the number of buckets small while ensuring they’re fine-grained enough for efficient cleanup.</p> When a new field with TTL is added, the volatile set either places it into an existing bucket or creates a new one. If a bucket grows too large, it is narrowed into smaller time windows to keep expiration scans efficient. For example: scanning 1,000 items in a single 10-second bucket may result in mostly misses, while spreading them across 10 smaller buckets avoids wasted work. Why not always use a very small time window (e.g few milliseconds) for each bucket? The reason is memory. Recall that the buckets are managed in a Radix-Tree which introduces a high overhead per each item. When many buckets are used, the memory overhead can be high and we might end up with the same memory overhead introduced by using a Radix-Tree index.</p> The volatile set also uses different bucket encoding based on the number of items in the bucket. A bucket with only a single element would require just a single pointer size of bytes. A bucket of small amount of items would be encoded as a vector of item pointers, and when the bucket contains many items we will use the Valkey hashtable</a> structure to map the relevant items. This way, we are leveraging existing data structures which are highly optimized for modern CPUs.</p> A hash object that contains volatile fields now also carries a secondary volatile set index. At the database level, we maintain a global map of hashes with volatile fields. The active expiration cron job scans both regular keys and these hashes, but only iterates over volatile set buckets whose end time has passed. This ensures that CPU time is spent only on fields that are truly ready to expire.</p> Benchmarking our solution</h2> Validating the new design meant benchmarking across several dimensions: memory overhead, command performance, and expiration efficiency.</p> Memory Overhead</h3> We first measured the per-field memory overhead when setting TTLs. The raw expiration time itself requires a constant 8 bytes (though this could be reduced in the future by storing only a delta relative to a reference timestamp, such as server start time). On top of that, extra memory is needed for tracking within the volatile set.</p> The actual overhead depends on both how many fields have expirations and how spread out their expiration times are. This is because the bucket encoding chosen by the volatile set adapts to the data distribution. In practice, the overhead ranged between 16 and 29 bytes per field.</p> </p> The higher end of this range primarily affects small hashes, where compact encodings like listpack are avoided when volatile fields are present.</p> Command Performance</h3> Next, we benchmarked common hash commands both with and without field expirations. The results showed no measurable performance regression and throughput remained stable when expirations were added.</p> </p> We also benchmarked the new expiration-aware commands (e.g., HSETEX</code>), confirming that their performance is on par with traditional hash operations.</p> </p> Active Expiration Efficiency</h3> The design goal of the volatile set was to enable efficient background deletion of expired fields. To test this, we preloaded 10 million fields with TTLs. We distributed these fields across varying numbers of hash objects to see how object size influences expiration. During the load phase, we disabled the expiration job using the DEBUG</code></a>, then re-enabled it once all fields had expired.</p> The following chart shows the time it took the expiration cron job to complete the full deletion of all the 10M fields.</p> </p> The results revealed that expiration time depends not just on the number of fields, but also on how they are distributed across objects. Smaller hashes tend to fit into CPU caches, so random field deletions remain cache-friendly. Very large hashes, however, cannot fit entirely in cache, which means more expensive memory lookups during expiration.</p> Another important factor is CPU utilization. The active expiration job is deliberately CPU-bounded and designed to use no more than ~25% of available CPU time (unless configured to work more aggressively), preventing it from overwhelming the system. The chart above shows that average CPU usage was consistently kept under this cap, even when expiring millions of fields, ensuring predictable tail latency.</p> We also simulated a more realistic scenario: expiring data during continuous ingestion.</em> Using the 'valkey-benchmark' executing the new HSETEX</code></a> command, we continuously inserted 10 million fields with 10-second TTLs, while the active expiration job ran in the background.</p> This setup maintained a constant pool of fields at different stages of their lifecycle — some fresh, some nearing expiration, some ready to be reclaimed. We then tracked memory usage over a 5-minute period.</p> </p> The results aligned with our theoretical expectation:</p> Memory = (Injection Throughput) x (AVG TTL) x (AVG Item memory)</span></span></code></pre> In our experiment:</p> Injection rate: 300K commands/sec</p> </li> Average TTL: 10 seconds</p> </li> Base item size: ~61 bytes</p> </li> Additional expiration overhead: ~19 bytes</p> </li> </ul> This yields an expected memory footprint of ~230MB:</p> 300K × 10 × 80B = 230MB</span></span></code></pre> The observed memory matched this estimate closely, demonstrating that the active expiration mechanism is able to keep up with load and prevent memory from spiking unexpectedly.</p> Benchmark Takeaways</h2> The benchmarks demonstrate that field-level expirations can be added to Valkey without compromising memory efficiency, or latency. The memory overhead remains modest and predictable, command throughput is unaffected, and the shared active expiration job efficiently reclaims memory even under heavy ingestion workloads. Together, these results validate that the coarse-bucket design with adaptive encoding delivers the right balance of efficiency, scalability, and correctness, while preserving Valkey’s reputation for high performance and low latency.</p> Using Hash Field Expiration in Valkey</h2> Valkey 9.0 introduces a new API for hash field expirations, fully compatible with the Redis 8.0 API. Any existing client library supporting this API can be used. In this example, we will use the latest valkey-glide</a> 2.1.0 release, which also supports the new hash field expiration commands.</p> Let's start with a simple example.</p> First, we create a client connecting to a local Valkey server:</p> from</span> glide_sync</span> import</span> GlideClientConfiguration, NodeAddress, GlideClient</span></span> addresses</span> =</span> [NodeAddress(</span>"localhost"</span>,</span> 6379</span>)]</span></span> config</span> =</span> GlideClientConfiguration(addresses,</span> request_timeout</span>=</span>500</span>)</span> # 500ms timeout</span></span> client</span> =</span> GlideClient.create(config)</span></span></code></pre> Next, let's create a new hash object to store some random user data:</p> client.hset(</span>"User1"</span>, {</span>"name"</span>:</span> "Ran"</span>,</span> "age"</span>:</span> 'old'</span>,</span> "password"</span>:</span> "1234"</span>})</span></span> ></span> 3</span></span></code></pre> Suppose we want the user’s password to be available only for a specific timeframe (e.g., 60 seconds). With the new hash field expiration feature, we can now do that:</p> client.hexpire(</span>"User1"</span>,</span> 60</span>, [</span>"password"</span>])</span></span> ></span> [</span>1</span>]</span></span></code></pre> Note that in this example, we only set a TTL for a single field, but the command can accept multiple fields at once.</p> We can check the remaining TTL for the password field:</p> client.httl(</span>"User1"</span>, [</span>"password"</span>])</span></span> ></span> [</span>46</span>]</span></span></code></pre> The reply shows the remaining time in seconds for the field to live. We can also query the absolute expiration time:</p> client.hexpiretime(</span>"User1"</span>, [</span>"password"</span>])</span></span> ></span> [</span>1757499351</span>]</span></span></code></pre> This is a Unix timestamp, which we can easily convert to human-readable time:</p> date</span> -d</span> @1757499351</span></span> ></span> Wed</span> Sep</span> 10</span> 10:15:51 UTC</span> 2025</span></span></code></pre> After 60 seconds, if we read the hash:</p> client.hgetall(</span>"User1"</span>)</span></span> ></span> {</span>b</span>'name'</span>:</span> b</span>'Ran'</span>,</span> b</span>'age'</span>:</span> b</span>'old'</span>}</span></span></code></pre> Notice that the password field is no longer present, since it has expired.</p> Another practical use case is managing a collection of links where each field represents a URL along with its metadata:</p> links:valkey:blogs -> {</span></span> "https://valkey.io/blog/valkey-is-a-key-value-store" -></span></span> {votes: 15, category: "tech"},</span></span> "https://valkey.io/blog/valkey-supports-different-ai-workloads" -></span></span> {votes: 42, category: "ai"},</span></span> "https://myblog.com/blog/how-to-write-good-valkey-blog" -></span></span> {votes: 7, category: "blog"}</span></span> }</span></span></code></pre> Here’s how hash field expirations help:</p> Each link can have a TTL representing its “relevance window.” If a link hasn’t been accessed within a certain period, it automatically expires.</li> Every time a user queries or updates a link, its TTL can be refreshed, keeping active links alive while letting inactive ones naturally fall off.</li> Expired links are removed automatically, so you don’t need extra cleanup logic and can focus on active links.</li> </ul> Using Valkey’s HSETEX</a> and HGETEX</a>, we can update a field’s TTL whenever it’s accessed or modified:</p> # Set or update link metadata with a 30-day TTL</span></span> client.hsetex(</span></span> 'links:user:42'</span>,</span></span> {</span></span> 'https://valkey.io/blog/valkey-is-a-key-value-store'</span>:</span></span> '{"clicks":15,"category":"tech"}'</span></span> },</span></span> expiry</span>=</span>ExpirySet(ExpiryType.</span>MILLSEC</span>,</span> 30</span></span>24</span></span>3600</span>))</span></span> </span> # Retrieve a link and refresh its TTL</span></span> link_data</span> =</span> client.hgetex(</span></span> 'links:user:42'</span>,</span></span> {</span>'https://valkey.io/blog/valkey-is-a-key-value-store'</span>},</span></span> expiry</span>=</span>ExpirySet(ExpiryType.</span>MILLSEC</span>,</span> 30</span></span>24</span></span>3600</span>)</span></span> )</span></span></code></pre>What’s Next?</h2> What we’ve shipped so far is just the first step: the ability to set and control time-to-live at the hash field level. But we’re not stopping here. Future work will focus on two main areas: reducing memory overhead and improving performance. For example, we plan to support compressed encodings for hashes with volatile fields, and to leverage modern CPU features such as memory prefetching and SIMD instructions to speed up operations.</p> Another critical area for improvement is the active expiration job. Today, all volatile data in Valkey is tracked in unsorted maps (hashtables). The background job must repeatedly scan these unordered sets, which means wasting CPU cycles on entries that aren’t close to expiration. By introducing structured tracking—through the volatile set or even alternative approaches like Hierarchal Time Wheels</a> —we can significantly reduce wasted work and make expiration more efficient at scale.</p> Numbered Databases in Valkey 9.0 2025-09-25T00:00:00+00:00 If you explore Valkey’s documentation you might run across a feature called ‘numbered databases’ which allows you to separate the keyspace into (by default) 16 different databases. Digging into this feature reveals tantalizing ways to avoid key prefixing, house different workloads together on Valkey, and even perform patterns that are otherwise clunky. However, if you’ve done more research outside of the documentation on numbered databases you find advice like “don’t use them,” “they don’t scale,” and “they’re a bad idea.” Well, the forthcoming Valkey 9.0 changes many things with numbered databases and you’ll see in this post that advice definitely needs some updating.</p> Today, a common way to conceptualize Valkey is that your keys represent a unique name for pointers to data in memory across a cluster of nodes. So, key foo</code> is unique and deterministically linked to a specific node and on that node there is a memory address where the value resides. However, this misses one important detail: the database number. The reality is that key names belong to a specific numbered database and aren’t unique</em> on a given instance of Valkey. To put this another way, Valkey can have the key foo</code> as many times as there are numbered databases with each one pointing to different data.</p> </p> Historically, before Valkey’s preceding project had the ability to cluster (before version 3.0.0), using multiple numbered databases was fully supported. However, when clustering was implemented numbered databases weren’t included: a cluster had one database (DB 0) that spanned across the entire cluster. In a world where using numbered databases in your application locked you into never going to a cluster, the early advice made sense, however Valkey 9.0 adds the ability to have numbered databases on a cluster, changing everything about that advice.</p> Why bring this feature to cluster mode in Valkey 9.0? In the intervening years since numbered databases were left out of the cluster spec, users have found a number of very handy patterns that were unfortunately limited without numbered databases. However, numbered databases aren't a panacea: read on to find out when you should stick to DB 0 and when to go for a numbered database.</p> Clustering and numbered databases</h2> As mentioned earlier, the key name dictates where the key lives in the cluster and this doesn’t change for numbered databases. As a refresher, Valkey clustering</a> takes the key name as a string, runs that through a CRC-16 function, then does a modulo of 16,384 which determines the ‘slot’. Each of these slots belong to a node in the cluster. For a numbered database, each slot contains all the numbered databases, so, carrying forward the idea that you can have the same key name in each database on a single instance, in Valkey 9.0 you can have the same key multiple times in a given slot, each in their own database. In other words, numbered databases do not affect clustering: the key is still the determinate factor in calculating slots.</p> You can see this directly with CLUSTER KEYSLOT</code></a> and SELECT</code></a>. Take the following example:</p> > SELECT 0</span></span> OK</span></span> > SET somekey hi</span></span> OK</span></span> > CLUSTER KEYSLOT somekey</span></span> (integer) 11058</span></span> > SELECT 5</span></span> OK</span></span> > SET somekey hello</span></span> OK</span></span> > CLUSTER KEYSLOT somekey</span></span> (integer) 11058</span></span></code></pre> Here the key somekey</code> was set to two different values: ‘hi’ on database 0, and ‘hello’ on database 5, with SELECT</code> altering the selected database. CLUSTER KEYSLOT</code> was called twice: each one yielding the same slot number meaning this key will be assigned to the same node in the cluster, no matter which database is selected.</p> Limitations</h2> Numbered databases do not change the properties of how a Valkey cluster works. The keyspace of each database is still split amongst all the nodes. As a consequence, operations that need to span the entire keyspace will to be run on each node:</p> FLUSHDB</code></a> will flush the keys in the current database on the connected node</em></li> SCAN</code></a> will iteratively return keys in the current database on the connected node.</em></li> DBSIZE</code></a> will return the number of keys in the current database on the connected node.</em></li> </ul> You get the picture: if a command previously said it did something for the entire database, in cluster mode it really means for the connected node’s portion of the database. These are especially important to understand if you're planning to move an application built for non-clustered, numbered databases to a cluster.</p> Additionally, numbered databases do not provide any form of resource isolation. It's tempting to point a bunch of applications to a single Valkey cluster with each application taking its own database. While this certainly can</em> work, it works without resource isolation and this setup can suffer from classic noisy neighbour problems: busy applications will affect the others using the same cluster. If you're worried about resource sharing, most of the time you'll be better off just having distinct clusters for each application instead of numbered databases.</p> Where to use numbered databases</h2> If you take away one thing from this blog it should be this: numbered databases are a form of namespacing</strong>.</p> The most straight forward use case of numbered databases is when you need to separate your data logically and you can tolerate the effects of resource sharing (see above). This might be something like keeping customer data separated from one another or combining applications on to a single cluster when resources are unlikely to be an issue. In a similar manner, multiple databases are a useful debugging tool. When you’re building an application it can be difficult to see what happens inside Valkey when you make a change. If you have multiple databases you can run your original code on one database and the changed version on a different database then you can more easily see changes in how data looks in Valkey by just the connection swapping between databases. This is also a useful pattern during a migration when you want your old data to stick around while your new data is populated.</p> An entirely different use of numbered databases is related to the MOVE</code></a> command. MOVE</code> allows you to change a key from one database to another without copying the data,</em> meaning it uses very little resources and it’s an O(1)</code> operation. This allows a couple of things: 1) you can effectively make data inaccessible from a database whilst keeping it on the same cluster node, and 2) you can replace a complex key atomically.</p> Looking at the first use of MOVE</code>: imagine you have some content, maybe a user submitted post, that is stored at the key “mycontent.” At some point this content gets flagged as needing to be reviewed. You might not want to delete</em> this content but you also don’t want it accessible. In this case you take the content and MOVE</code> it to a different database. Once it’s reviewed (or edited!) you can MOVE</code> it back. Never does the data actually leave the node nor get copied. Whilst your use-case might not be the same as this, the pattern illustrates an operation useful in a variety of contexts.</p> </p> The second use of MOVE</code> is similar to the first except it revolves around complex keys that contain many elements. Take, for example, a large sorted set. It’s not unusual for these keys to be quite large because the key contains thousands or millions of smaller elements. If you need to replace this key with one that contains different elements then you’d have to first DEL</code> the original key, then ZADD</code> thousands of elements. Doing this in a transaction would be expensive and monopolize the node, and without a transaction, it would reveal a (potentially undesirable) partial state. Instead, with multiple databases, you build up new a complex key on a different database (under no urgency) and do a much simpler MULTI</code> / EXEC</code> transaction of of DEL</code> followed by MOVE</code> to atomically make the new complex key available to the database being used to serve commands without the chance of a partial state.</p> </p> This is really a form of temporal namespacing: you're taking advantage of the namespace to have two active keys under the same name both of which can be read and manipulated. You can apply this pattern to other cases: anyplace you want to make a key inaccessible for a period of time.</p> How is key prefixing different than numbered databases?</strong> Some of the patterns in this blog can be achieved without numbered databases and instead with key prefixing (e.g. app0:...</code>, app1:...</code>). Key prefixing has a few downsides when compared to numbered databases:</p> Prefixes add up at scale. Millions of keys each with repeated prefixes over the span of the whole cluster means less RAM for data. While each database does have a memory overhead it is linear relative to the number of databases used, not keys. For a deeper look at how numbered database memory overhead works, including using 10 million databases, read the comments on valkey-io/valkey#1609</a>.</li> Databases are transparent to your application. To support prefixing, either your application or your client library has to be able to interpolate the prefix into each key. With numbered databases, the changes can be as simple as a number in the connection URI, not to each key name.</li> Databases avoid pattern-based iterations. Having all your keys in DB 0 then needing to iterate over the entire keyspace to affect a specific pattern is expensive and complicated. If your keys are separated into databases, this both subsets that iteration and enables some database wide commands (i.e. FLUSHDB</code> instead of trying to delete keys by a pattern).</li> </ul> </blockquote> Gotchas and future work</h2> Aside from the aforementioned lack of resource isolation, numbered databases in clustered Valkey have a few places that you need to look out for today. First, right now numbered databases don’t have a lot of dedicated metrics, so it can be hard to gain insights into per-database resource usage. Second, the ACL system doesn’t currently address numbered databases, so you can’t meaningfully restrict access to databases (at time of writing, there is an open pull request that addresses this</a>). Unlike in the previous project, databases are an active area of work for the Valkey project and the database abstraction holds potential as a way to implement new features.</p> Finally, while numbered databases are well supported in client libraries there are rough edges:</p> Some client libraries artificially restrict usage to a single database in cluster mode,</li> Pooled clients may naïvely manage the selected database, so a client returned to the pool after running SELECT</code> might retain the database number in subsequent usage. A similar situation is possible for multiplexed clients.</li> </ul> In general, watch out for three assumptions: 1) the selected database is always 0, 2) there is only one database in cluster mode, and 3) if there are multiple databases in use it isn’t in cluster mode. None of these are true in Valkey 9.0.</p> With this in mind, Valkey 9.0 gives you the ability to divvy up that keyspace into nice neat numbered databases and spread them out over a whatever cluster you have. So, get rid of that old, outdated advice and start using them, seeing how they scale, and what a good idea number databases actually are for Valkey 9.0.</p> Valkey: An Investment in Open Source 2025-09-16T00:00:00+00:00 Valkey was founded just over a year ago to keep high-performance key/value storage in the open source community: free from vendor lock-in and restrictive licenses. Backed by contributors like AWS, Google Cloud, Ericsson, Oracle, Alibaba, Huawei, Tencent, Percona, Aiven, Heroku, Verizon, Chainguard, and Canonical, the project shows how “free” in open source depends on investment: time, talent, and ongoing financial support.</p> Built by Builders, Backed by the Linux Foundation</h3> Valkey was created on March 28, 2024 and has solidified itself as the open-source high-performance key/value datastore that supports a variety of workloads such as caching, message queues, and can act as a primary database. Valkey is backed by the Linux Foundation, a neutral organization founded in 2000 that supports developers and technologists to scale and manage open source projects. Valkey operates under an open governance model focused on growing community contributions and adoption.</p> One Year In: Growth and Momentum</h3> In just a year’s time the project has celebrated two major releases, an increase in corporate participants from 22 to 47, is defining, innovating, and executing on its roadmap, and is growing in its adoption. Users and the community have embraced the project and they are firm in their commitment to continue to improve Valkey for the benefit of all end users.</p> New Valkey Features & Use Cases</h2> Solving Practical Problems with JSON and Bloom Filters</h3> Two new data types, JSON and Bloom filters, expand what developers can do with Valkey, especially in distributed systems where speed and structure are equally important. These additions reduce complexity in application logic and improve how data is handled at the edge. “Adding JSON and Bloom filters to Valkey is about giving developers practical tools to solve real-world problems in distributed systems,” says Madelyn Olson, Valkey co-maintainer.</p> JSON support allows developers to work with rich, structured data natively, instead of relying on custom serialization or extra middleware. “JSON lets you work with rich, structured data directly in Valkey, which simplifies development and reduces glue code,” Olson explains.</p> Valkey Bloom filters</a>, on the other hand, offer a compact way to perform fast checks. “Bloom filters are ideal when you need to make fast, memory-efficient existence checks, whether you're trying to catch fraud or avoid unnecessary backend calls in a high-traffic service,” adds Olson.</p> As Olson put it, the goal is simple: “I’m really excited to see what problems our users will be able to solve with these new data types.”</p> Valkey Search: Speed and Scale for AI Workloads</h3> Google contributed a new module, Valkey Search</a>, which brings vector similarity search to Valkey. It’s fast, delivering single-digit millisecond latency and built to handle billions of vectors while maintaining over 99% recall.</p> Developers can run both approximate nearest neighbor (ANN) searches (via HNSW) and exact k-nearest neighbor (KNN) searches. It supports indexing with either Hash or Valkey-JSON</a> data types. While Valkey Search currently focuses on vector search, future goals would extend Valkey into a broader search engine with full-text support and more indexing options.</p> Performance, Reliability, and Security</h3> Over the past year, contributors from companies like Ericsson, Oracle, and Percona have focused on making Valkey faster, safer, and more enterprise-ready. The following improvements showcase how Valkey is evolving to meet scaled production needs without compromising for any of its users.</p> Multithreading Improvements</h4> Recent updates to Valkey’s internals improve how the system handles work across multiple threads, cutting down on lock contention and better using multi-core processors. This update significantly improves throughput under high concurrency, making the project more attractive for large-scale production use. Ericsson, one of Valkey’s core contributors, is already seeing real value in the project’s performance work.</p> Viktor Söderqvist, Ericsson Software Technology Engineer & Valkey co-maintainer, points out the core efficiency as a key reason Valkey is becoming more production-ready. “To get the most out of the CPUs and memory in terms of storage efficiency and speed, the recent hashtable redesign lately improved with SIMD techniques as well as the recent and ongoing improvements in multithreading and memory batch-prefetching techniques are exciting areas of improvement.”</p> SIMD-Accelerated Hashtable Redesign</h4> The project’s core hashtable was reengineered to leverage SIMD instructions, allowing Valkey to process key lookups more efficiently by handling multiple operations in parallel. These low-level optimizations translate to faster response times in latency-sensitive environments.</p> Batch Memory Prefetching</h4> New support for batch memory prefetching helps reduce cache misses by anticipating access patterns and loading data proactively. The result is smoother performance and more consistent behavior under heavy or sequential access workloads.</p> LDAP Integration</h4> LDAP integration brings centralized authentication and access control to Valkey, making it easier to deploy in enterprise environments with existing identity infrastructure. This feature addresses a common adoption barrier in security-conscious and compliance-driven organizations.</p> “A lot of our customers already rely on LDAP to manage access across their infrastructure, so bringing that to Valkey just made sense,” says Vadim Tkachenko, Co-Founder of Percona. “It’s one of those features that removes friction. You get auditability, group-based permissions, and it works with what teams already have in place.”</p> Rust Module</h4> Oracle is contributing a Rust-based module SDK to Valkey, aimed at improving safety and performance for low-level extensions. By adopting Rust’s strong guarantees around memory and concurrency, the project opens the door to safer, more maintainable systems integration, especially for production environments under load.</p> The Real Investment Behind Open Source Success</h3> The investment in open source and choosing to employ contributors to the project clearly demonstrates that a key aspect in technological innovation is for companies at varying levels, enterprise to mom-and-pop shops, to succeed they need to work together and that means investing in the resources that can move projects forward - the human capital of open source, the coders, the writers, the governance, and the community behind the ‘free’ moniker that we frequently associate with open source.</p> What’s Happening With Valkey</h3> On August 15, Valkey dropped the first release candidate for 9.0</a>. This testing release previews the new capabilities around atomic slot migrations, hash field expirations, and numbered databases in cluster mode alongside scads of performance enhancements and bug fixes. Additional release candidates are to follow with general availability of 9.0 in early autumn 2025.</p> Parallel to the testing phase for 9.0, two events happened last month in Amsterdam:</p> Open Source Summit Europe - Aug 25–27, 2025</a>: Three-day conference for open source developers, technologists, and leaders featuring keynotes, sessions, and community networking.</li> Valkey User Conference: Keyspace - August 28, 2025</a>: A one-day conference for developers, SREs, and DevOps teams with sessions, lightning talks, and workshops.</li> </ul> What you'll see at Keyspace 2025 2025-08-12T01:01:01+00:00

Beyond Vectors: Introducing Full-Text Search and Aggregations to Valkey

2026-03-17T01:00:00+00:00

Valkey Search now lets you search across text, tag, numeric, and vector attributes in a single query, and analyze results with server-side aggregations at the low latency you expect from Valkey. Valkey Search enables searching terabytes of data with latency as low as microseconds, providing a flexible foundation for querying across a range of use cases, from powering in-app search experiences to recommendation systems and analyzing data to support in-app analytics and reporting dashboards.

Until now, Valkey Search focused on vector similarity, enabling a wide range of workloads such as semantic search and AI workloads (see launch blog</a>). But if you needed to filter your data by numeric attributes such as price ranges, match on exact attributes such as color or size, or search within text attributes such as product reviews, you had to build that yourself.

With Valkey Search 1.2, that changes. In this post, we'll walk through what's new, show how it works, and explore the use cases these capabilities unlock.

Searching Your Valkey Data</h2>

Valkey Search uses indexes to organize your data by searchable attributes, so queries can find matching keys without scanning every document. To get started with search, you first define an index over the attributes you want to search. Valkey Search supports four query types — full-text, tag, numeric range, and vector similarity, which you can use independently or combine with boolean operators. For example, consider an e-commerce retailer Acme.inc using Valkey to store their product catalog who creates an index like this:

FT.CREATE product_index ON HASH SCHEMA</span></span>
  description TEXT</span></span>
  color TAG</span></span>
  manufacturer TAG</span></span>
  price NUMERIC SORTABLE</span></span>
  rating NUMERIC SORTABLE</span></span>
  review vector HNSW 6 TYPE FLOAT32 DIM 768 DISTANCE_METRIC COSINE</span></span></code></pre>
This creates an index called product_index</code> over hash keys, indexing description</code> as a full-text searchable attribute, color</code> and manufacturer</code> as exact-match tags, price</code> and rating</code> as sortable numeric attributes, and review</code> as a vector attribute for similarity search. With this index in place, you can now query across all four types:</p>
Full-text Search:</strong> Full-text search (FTS) helps you retrieve relevant documents by matching words and phrases within larger text when you do not have an exact identifier or a structured attribute value.
You can use full-text queries to retrieve documents by matching keywords, phrases, or patterns such as prefix, suffix, wildcard, and fuzzy queries (typo-tolerant), anywhere in your document index.
You can use FTS to express constraints like "user reviews mentioning headphones with active noise cancellation and</strong> long battery life, but not</strong> weighing more than 3 ounces."
Hence, full-text search is a powerful capability for workloads that search unstructured data rather than exact tags.
Full-text search queries shine in use cases such as e-commerce and catalog searches to help users find products instantly across large inventories.
The prefix/suffix pattern matching provided by full-text search makes it ideal for powering instant suggestions as users type, enabling discovery in search bars and product catalogs.
Finally, fuzzy matching capabilities allow you to accurately match documents despite spelling variations, typos, or inconsistent formats to support typo-tolerant retrieval.
For example, when a shopper on Acme.inc types "noise cancelling earphones" in the search bar, the app can search across descriptions to retrieve relevant products:</p>
FT.SEARCH product_index "noise canc*"</span></span></code></pre>
This returns matching products with all their indexed attributes:</p>
1) (integer) 2</span></span>
2) "product:1001"</span></span>
3) 1) "description"</span></span>
   2) "Wireless earphones with active noise cancelling and 30-hour battery life"</span></span>
   3) "color"</span></span>
   ...</span></span>
4) "product:1042"</span></span>
   ...</span></span></code></pre>
Tag Search:</strong> Tag queries filter documents by checking attribute values for exact matches against the provided value. Tag attributes are best for structured categorical data where queries rely on exact value matching.
You can treat tag as a specialized alternative to text for attributes that you want to match as discrete values such as userIDs, categories, and status flags.
This makes tag a great fit to express conditions like "category is earphones and</strong> size is small, but color is not</strong> red."
Tag filtering shines in use cases such as streaming and gaming platforms to retrieve metadata by exact identifiers such as user ID, region, or genre to power real-time stats and fast in-app filtering.
It is also well suited for session and user management, where you need to quickly locate active sessions and entitlements by exact identifiers such as session ID, device properties, or tenant ID across millions of documents.
For example, when a shopper on Acme.inc filters for earphones manufactured by "shopnow" that are "white" in color, the app can query the tag index:</p>
FT.SEARCH product_index "@manufacturer:{shopnow} @color:{white}"</span></span></code></pre>
Numeric Range Queries:</strong> Range search retrieves documents by filtering and sorting over numeric and time attributes using comparisons such as greater than, less than, and between. This makes it ideal for applications that query attributes such as scores, price bands, time windows, inventory thresholds, distances, and timestamps.
Valkey Search supports range-based queries with microsecond latencies, making it a strong fit for real-time leaderboards where you rank and retrieve content by numeric metrics such as scores, downloads, or engagement with updates reflected immediately.
Range queries are also well suited for financial transactions or time series data, enabling ultra-fast lookups by amounts, fees, date ranges, and risk scores to power customer-facing applications, personalization, or real-time monitoring.
For example, when a shopper on Acme.inc sets the price slider to $50–$100 and filters by 4-star and above, the app can query the numeric index:</p>
FT.SEARCH product_index "@price:[50 100] @rating:[4 +inf]"</span></span></code></pre>
Hybrid Queries:</strong> Valkey Search lets you combine full-text, tag, numeric range, and vector similarity clauses in a single query so you can retrieve relevant results.
Hybrid queries are ideal for applications that need to query across multiple attribute types in one request. For example, a product discovery query might combine vector similarity over reviews for semantic meaning, a tag filter for category and availability, a numeric range for price, and full-text search over product titles — all in a single request.
This eliminates the need to stitch together multiple queries or round-trips, making it a natural fit for recommendation systems, media platforms, and real-time operational workflows.
Combining our examples from above, when a shopper on Acme.inc searches for "noise cancelling earphones" and filters for products in "white" color, under $150, that are good for running, the app can combine all of this into a single query:</p>
FT.SEARCH product_index "noise cancelling earphones @color:{white} @price:[0 150] =>[KNN 5 @review $vector]" PARAMS 2 vector 'VECTOR_REPRESENTATION'</span></span></code></pre>
Here, the query is searching through vector representations of product reviews to find “earphones that are good for running”.</p>
Transform Your Valkey Data with Aggregations</h2>
Aggregations help you analyze and summarize the results of a search query, instead of returning a raw list of matching documents.
You can use GROUPBY</code> to form groups on any indexed attribute such as category, brand, region, and time, apply REDUCE</code> functions such as COUNT</code>, SUM</code>, and AVG</code> to compute per-group statistics, and use APPLY</code> to create computed attributes on the fly.
You can then refine and shape the output with post-aggregation FILTER</code>, SORTBY</code>, and LIMIT</code>, chaining stages together to build multi-step workflows in a single query.
This makes aggregations a strong fit for low-latency lightweight analytics directly on indexed Valkey data, without the need to export large result sets to the application layer. Some applications of aggregations include:</p>

Faceted navigation and filtering: Power dynamic filtering UIs using aggregations to compute real-time counts over the current result set (for example by category, brand, price band, rating, or availability), enabling users to narrow down search results with instant feedback on available options. You can also group and count items by attribute such as genre, tags, language, or creator to power structured browsing and category-level summaries across large catalogs.</li>
Real-time statistics: Use aggregations to compute grouped rankings for powering in-app flows such as trending items by recent engagement, category leaders by revenue, or top performers by region.</li>
Analytics and reporting: Generate live counts and summary metrics to power operational dashboards, or generate on-demand reports for analysis and decision making without hitting backend databases.
For example, Acme.inc can populate the count of earphones available from “shopnow” in each price band on the webpage using:</li>
</ol>
FT.AGGREGATE product_index "@manufacturer:{shopnow}"</span></span>
APPLY "floor(@price / 50) * 50" AS price_band</span></span>
GROUPBY 1 @price_band</span></span>
REDUCE COUNT 0 AS product_count</span></span>
SORTBY 2 @price_band ASC</span></span></code></pre>
This filters for products from the "shopnow" manufacturer, uses APPLY</code> to bucket each product's price into $50 bands (0–49, 50–99, 100–149, etc.), groups by those bands, counts products in each, and sorts by price band ascending — giving Acme.inc a price distribution histogram for that manufacturer's products.</p>
Under the Hood</h2>
Real-time Search with Multi-threading</h3>
Valkey Search uses indexes to organize keys by the contents of their searchable attributes, ensuring reads remain fast and efficient as data grows.
When you add, update, or delete an indexed key, the module receives the mutation event, extracts indexed attributes, queues the indexing work, and updates the index before acknowledging the write.
Valkey Search is multi-threaded, so you can maximize ingestion throughput by using multiple parallel connections to saturate the index update process without pipelining on a single connection.
Background worker threads process index updates, and the client receives a response only after the update is committed to the index.</p>
Read-after-write Consistency</h3>
For workloads that require strict read-after-write behavior, you can configure your client to route search queries to primaries.
A write only completes after its index updates are applied, so any search sent to the same primary after the write returns will see that change.
If your application can tolerate some staleness, you can offload reads to replicas.
On replicas, search is eventually consistent because replication and index maintenance are asynchronous, and each node maintains its own local indexes.
Multi-key updates wrapped in a MULTI/EXEC transaction or Lua script are also atomically visible to search.
Valkey Search exposes the index only after all attribute updates in the batch are applied.
Separately, in cluster mode, read-after-write consistency applies per primary of the shard.
If you write a key to a primary, searches sent to that primary see the change after the write returns, but a fan-out query spanning multiple shards does not have a single global transactional view.</p>
Scale to Terabytes of Data</h3>
Valkey Search includes built-in support for cluster mode, enabling you to scale to terabytes of data without requiring application or client code changes.
In cluster mode, Valkey Search creates indexes that span multiple shards by maintaining a separate index on each node for the keys belonging to that node's slot range. When you create, update, or drop an index on any primary, Valkey Search propagates that change to all nodes.
You can scale read throughput by distributing Search queries evenly across primary nodes so that no single node becomes a bottleneck, or by routing some Search queries to replicas.
You can increase throughput by scaling to instances with more vCPUs, allowing multithreading to scale throughput linearly for both querying and ingesting, or by adding replicas to increase query throughput.</p>
For queries, the coordinating node that receives the query request, packages a query plan and sends it to every shard (to run on either primaries or replicas).
A node within each shard performs the search and fetches its matching keys, then returns results for the coordinator to merge. Because the fan-out and merge logic exists on every cluster node, any node can coordinate a query. For mutations, the primary owning the slot handles updates: when a key is added, updated, or deleted, only that primary updates its index directly, and the change replicates to its replicas.</p>
Getting Started</h2>
Valkey Search 1.2 extends search to text, tag, and numeric attribute types and adds result aggregation capabilities such as filtering, sorting, grouping, and computing metrics. Whether you're building cutting-edge AI applications, latency-sensitive search experiences, or integrating search into existing systems, we invite you to try it out.
To get started with Valkey Search, visit the Valkey Search GitHub repository</a> and the Valkey Bundle on Docker Hub</a>. The official Valkey Bundle image provides the fastest path to running Valkey with preloaded modules, including Valkey Search, so you can begin building search and aggregation workflows without manual module setup. You can connect using official Valkey client libraries such as Valkey GLIDE, valkey-py, valkey-go, and valkey-java, as well as other Redis-compatible clients. Valkey Search is available under the BSD-3-Clause license. You can learn more about Valkey Search through our documentation</a>.</p>
Get Involved: Join the Valkey Search community</a>, file issues, open pull requests, or suggest improvements. We welcome contributions of all kinds - code, documentation, testing, and feedback. Your involvement helps make Valkey Search better for everyone.</p>

Note:</strong> As of March 13, 2026, if you want to use Valkey Search 1.2 features on docker, use the current valkey/valkey-bundle:unstable image.</p>
</blockquote>

Resharding, Reimagined: Introducing Atomic Slot Migration

2025-10-29T00:00:00+00:00

Managing the topology of a distributed database is one of the most critical and challenging tasks for any operator. For a high-performance system like Valkey, moving data slots between nodes —a process known as resharding— needs to be fast, reliable, and easy.

Clustered Valkey has historically supported resharding through a process known as slot migration, where one or more of the 16,384 slots is moved from one shard to another. This slot migration process has historically led to many operational headaches. To address this, Valkey 9.0 introduced a powerful new feature that fundamentally improves this process: Atomic Slot Migration.

Atomic Slot Migration includes many benefits that makes resharding painless. This includes:

a simpler, one-shot command interface supporting multiple slot ranges in a single migration,</li>
built-in cancellation support and automated rollback on failure,</li>
improved large key handling,</li>
up to 9x faster slot migrations,</li>

greatly reduced client impact during migrations.</li> </ul>
Let's dive into how it works and what it means for you.
Background: The Legacy Slot Migration Process</h2>
Prior to Valkey 9.0, a slot migration from a source node to a target node was performed through the following steps:

Send <target></code>: CLUSTER SETSLOT <slot> IMPORTING <source></code></li>
Send <source></code>: CLUSTER SETSLOT <slot> MIGRATING <target></code></li>
Send <source></code>: CLUSTER GETKEYSINSLOT <slot> <count></code></li>
Send <source></code>: MIGRATE ...</code> for each key in the result of step 3</li>
Repeat 3 & 4 until no keys are left in the slot on <source></code></li>
Send <target></code>: CLUSTER SETSLOT <slot> NODE <target></code></li>
Send <source></code>: CLUSTER SETSLOT <slot> NODE <target></code></li> </ol> This was subject to the following problems:

Higher latency for client operations: All client writes and reads to keys in the migrating hash slot were subject to redirections through a special -ASK</code> error response, which required re-execution of the command on the target node. Redirected responses meant unexpected latency spikes during migrations.</li>
Multi-key operation unavailability: Commands like MGET</code> and MSET</code> which supply multiple keys could not always be served by a single node when slots were migrating. When this happened, clients would receive error responses and were expected to retry.</li>
Problems with large keys/collections: Since the migration was performed one key at a time, large keys (e.g. collections with many elements) needed to be sent as a single command. Serialization of a large key required a large contiguous memory chunk on the source node and import of that payload required a similar large memory chunk and a large CPU burst on the target node. In some cases, the memory consumption was enough to trigger out-of-memory conditions on either side, or the CPU burst could be large enough to cause a failover on the target shard due to health probes not being served.</li>
Slot migration latency: The overall latency of the slot migration was bounded by how quickly the operator could send the CLUSTER GETKEYSINSLOT</code> and MIGRATE</code> commands. Each batch of keys required a full round-trip-time between the operator's machine and the cluster, leaving a lot of waiting time that could be used to do data migration.</li>
Lack of resilience to failure: If a failure condition was encountered, for example, the hash slot will not fit on the target node, undoing the slot migration is not well supported and requires replaying the listed steps in reverse. In some cases, the hash slot may have grown while the migration was underway, and may not fit on either the source or target node.</li> </ul> The Core Idea: Migration via Replication</h2> At its heart, the new atomic slot migration process closely resembles the concepts of replication and failover which serve as the backbone for high availability within Valkey. When atomic slot migration is requested, data is asynchronously sent from the old owner (the source node) to the new owner (the target node). Once all data is transferred and the target is completely caught up, the ownership is atomically transferred to the target node. This gets logically broken down into three phases: Snapshot Phase: The source node first sends a point-in-time snapshot of all the data in the migrating slots to the target node. The snapshot is done asynchronously through a child process, allowing the parent process to continue serving requests. The snapshot is formatted as a stream of commands which the target node and its replica can consume verbatim.</li> Streaming Phase: While the snapshot is ongoing, the source node will keep track of all new mutations made to the migrating slots. Once the snapshotting completes, the source node streams all incremental changes for those slots to the target.</li> Finalization Phase: Once the stream of changes has been sent to the target, the source node briefly pauses mutations. Only once the target has fully processed these changes does it acquire ownership of the migrating slots and broadcast ownership to the cluster. When the source node discovers this, it knows it can delete the contents of those slots and redirect any paused clients to the new owner atomically.</li> </ol> Why is this better?</h2> By replicating the slot contents and atomically transferring ownership, Atomic Slot Migration provides many desirable properties over the previous mechanism: Clients are unaware: Since the entire hash slot is replicated before any cleanup is done on the source node, clients are completely unaware of the slot migration, and no longer need to follow ASK</code> redirections and retry errors for multi-key operations.</li> Keys no longer need to be atomically moved: Collections are moved as chunks of elements that are replayed as commands, preventing the reliability problems previously encountered when dumping and restoring a large collection.</li> A migration can easily be rolled back on cancellation or failure: Since Valkey places the hash slots in a staging area, they are easily wipe them independently of the rest of the database. Since this state is not broadcasted to the cluster, ending the migration is a straight-forward process of cleaning up the staging area and marking the migration as cancelled. Many failures, like out-of-memory, failover, or network partition can be handled completely by the engine.</li> Greatly lowered slot migration latency: Valkey is highly-optimized for replication. By batching the slot migrations and using this replication-like process, the end-to-end migration latency can lower by as much as 9x when compared to legacy slot migration through valkey-cli</code>.</li> </ul> How to use Atomic Slot Migration</h2> A new family of CLUSTER</code> commands gives you full control over the migration lifecycle. CLUSTER MIGRATESLOTS SLOTSRANGE start-slot end-slot NODE node-id</code></a> This command kicks off the migration. You can specify one or more slot ranges and the target node ID to begin pushing data. Multiple migrations can be queued in one command by repeating the SLOTSRANGE and NODE arguments.</li> </ul> </li> CLUSTER CANCELSLOTMIGRATIONS</code></a> Use this command to safely cancel all ongoing slot migrations originating from the node.</li> </ul> </li> CLUSTER GETSLOTMIGRATIONS</code></a> This gives you an observable log of recent and active migrations, allowing you to monitor the status, duration, and outcome of each job. Slot migration jobs are stored in memory, allowing for simple programmatic access and error handling.</li> </ul> </li> </ul> Legacy vs. Atomic: Head-to-Head Results</h2> Head-to-head experiments show the improvement provided by atomic slot migration. Test Setup</h3> To make things reproducible, the test setup is outlined below: Valkey cluster nodes are c4-standard-8</code> GCE VMs spread across GCP’s us-central1 region running Valkey 9.0.0</li> Client machine is a separate c4-standard-8</code> GCE VM in us-central1-f</li> Rebalancing is accomplished with the valkey-cli --cluster rebalance</code> command, with all parameters defaulted. The only exception is during scale in, where --cluster-weight</code> is used to set the weights to only allocate to 3 shards.</li> The cluster is filled with 40 GB of data consisting of 16 KB string valued keys</li> </ul> Slot Migration Latency: Who's Faster?</h3> The experiment had two tests: one with no load and one with heavy read/write load. The heavy load is simulated using memtier-benchmark</a> with a 1:10 set/get ratio on the client machine specified above. Test Case</th> Legacy Slot Migration</th> Atomic Slot Migration</th> Speedup</th> </tr> No Load: 3 to 4 shards</td> 1m42.089s</td> 0m10.723s</td> 9.52x</td> </tr> No Load: 4 to 3 shards</td> 1m20.270s</td> 0m9.507s</td> 8.44x</td> </tr> Heavy Load: 3 to 4 shards</td> 2m27.276s</td> 0m30.995s</td> 4.75x</td> </tr> Heavy Load: 4 to 3 shards</td> 2m5.328s</td> 0m27.105s</td> 4.62x</td> </tr> </table> The main culprit here is unnecessary network round trips (RTTs) in legacy slot migration. Each slot requires: 2 RTTs to call SETSLOT</code> and begin the migration</li> Each batch of keys in a slot requires: 1 RTT for CLUSTER GETKEYSINSLOT</code></li> 1 RTT for MIGRATE</code></li> 1 RTT for the actual migration of the key batch from source to target node</li> </ul> </li> 2 RTTs to call SETSLOT</code> and end the migration</li> </ul> Those round trip times add up. For this test case where we have: 4096 slots to move</li> 160 keys per slot</li> valkey-cli</code>'s default batch size of 10</li> </ul> We need: Even with a 300 microsecond round trip time, legacy slot migration spends over a minute just waiting for those 212,992 round trips. By removing this overhead, atomic slot migration is now only bounded by the speed that one node can push data to another, achieving much faster end-to-end latency. Client Impact: How Would Applications Respond?</h3> The experiment measured the throughput of a simulated valkey-py</a> workload with 1:10 set/get ratio while doing each scaling event. Over three trial throughput averages are shown below. Despite atomic slot migration causing a more acute throughput hit, you can see the recovery of the client application is much faster due to far fewer topology changes and an overall lower end-to-end latency. Each topology change needs to be handled by the Valkey client, so the quicker the topology changes are made, the sooner the impact ends. By collapsing the topology changes and performing atomic handover, atomic slot migration leads to less client impact overall than legacy slot migration. Under the Hood: State Machines and Control Commands</h2> To coordinate the complex dance between the two nodes, a new internal command, CLUSTER SYNCSLOTS</code></a>, is introduced. This command orchestrates the state machine with the following sub-commands: CLUSTER SYNCSLOTS ESTABLISH SOURCE <source-node-id> NAME <unique-migration-name> SLOTSRANGE <start> <end> ...</code> Informs the target node of an in progress slot migration and begins tracking the current connection as a slot migration link.</li> </ul> </li> CLUSTER SYNCSLOTS SNAPSHOT-EOF</code> Used as a marker to inform the target the full snapshot of the hash slot contents have been sent.</li> </ul> </li> CLUSTER SYNCSLOTS REQUEST-PAUSE</code> Informs the source node that the target has received all of the snapshot and is ready to proceed.</li> </ul> </li> CLUSTER SYNCSLOTS PAUSED</code> Used as a marker to inform the target no more mutations should occur as the source has paused mutations.</li> </ul> </li> CLUSTER SYNCSLOTS REQUEST-FAILOVER</code> Informs the source node that the target is fully caught up and ready to take over the hash slots.</li> </ul> </li> CLUSTER SYNCSLOTS FAILOVER-GRANTED</code> Informs the target node that the source node is still paused and takeover can be safely performed.</li> </ul> </li> CLUSTER SYNCSLOTS FINISH</code> Inform the replica of the target node that a migration is completed (or failed).</li> </ul> </li> CLUSTER SYNCSLOTS CAPA</code> Reserved command allowing capability negotiation.</li> </ul> </li> </ul> The diagram below shows how CLUSTER SYNCSLOTS</code> is used internally to drive a slot migration from start to finish: Get Started Today!</h2> This new Atomic slot migration is a massive step forward for Valkey cluster management. It provides a faster, more reliable, and overall easier mechanism for resharding your data. So, go download Valkey 9.0</a> and try Atomic Slot Migration for yourself! A huge thank you to everyone in the community who contributed to the design and implementation. Valkey 9.0: innovation, features, and improvements. 2025-10-21T00:00:01+00:00 For Valkey's second major release, Valkey 9.0 brings innovation, long-requested features, and improvements to classic features updated for today’s workloads. Read on to find out all the team packed into this release. Atomic Slot Migrations</h2> Atomic slot migrations fundamentally changes how Valkey migrates data from node-to-node inside the cluster. Prior to Valkey 9.0, data migrated in the cluster key-by-key. This approach works for most situations, but corner cases can lead to degraded performance, operational headaches, and, at worst, blocked node migrations and lost data. Key-by-key migration uses a move-then-delete sequence. Performance issues arise when a client tries to access a key during a partially migrated state: if the migration hasn’t completed, the client may not know if the key resides on a the original node or the new node and leading to a condition that has more network hops and additional processing. Worse, in a multi-key operation, if one key resides in the original node and another in the new node, Valkey cannot properly execute the command, so it requires the client to retry the request until the data resides on a single node, leading to a mini-outage where Valkey still has the data but it is inaccessible until the migration is complete for the affected data. Finally, in a situation where Valkey is attempting to migrate a very large key (such as collections in data types like sorted sets, sets, or lists) from one node to another, the entire key may be too large to be accepted by the target node’s input buffer leading to a blocked migration that needs manual intervention. To unblock the migration you need to increase the input buffer limit or end up losing data either through forcing the slot assignment or deleting the key. In Valkey, keys are bundled into one of 16,384 ‘slots’ and each node takes one or more slots. In Valkey 9.0 instead of being key-by-key, Valkey migrates entire slots at a time, atomically moving the slot from one node to another using the AOF format. AOF can send individual items in a collection instead of the whole key. Consequently, this prevents large collections from causing latency spikes when they are being processed during migration. The new atomic slot migration doesn’t migrate keys directly, instead, the move-then-delete sequence is on the entire slot; the original node still retains all the keys and data until the entire slot migration is complete avoiding the pre-Valkey 9.0 issues with redirects or retries. For more information, check outthe video from our recent Keyspace conference talk recording about Valkey 9.0</a> and look for an upcoming deep dive on atomic slot migrations. Hash Field Expiration</h2> The hash data type allows you to neatly tie together data with multiple fields under one key. But because all this data lives attached to a single key, the expiry was, until Valkey 9.0, all-or-nothing: you couldn't expire fields individually. For users who needed some of the data to expire, the limitation made users look to awkward hacks with multiple keys, compounding the complexity and increasing the memory footprint of the data. Valkey 9.0 now address this gap by adding the following commands: HEXPIRE</a></li> HEXPIREAT</a></li> HEXPIRETIME</a></li> HGETEX</a></li> HPERSIST</a></li> HPEXPIRE</a></li> HPEXPIREAT</a></li> HPEXPIRETIME</a></li> HPTTL</a></li> HSETEX</a></li> HTTL</a></li> </ul> You can read more about how hash field expiration works in Ran Shidlansik's deep dive</a> on the subject. Numbered Databases in cluster mode</h2> Numbered databases allow you to separate data and avoid key name clashes: each database contains keys unique to that database. This is an old feature dating back to the very first version of the preceding project. However, before Valkey 9.0, numbered databases were severely limited with cluster mode being restricted to having a single database (db 0). Without cluster support, numbered databases were discouraged as using them limited you to never scaling beyond a single node. Based on user feedback and reconsideration by the team, Valkey 9.0 breaks from the preceding project and adds full support for numbered databases in cluster mode. Numbered databases have a whole host of use cases and some very handy clustering characteristics, find out more in the feature write up</a>. Much, much, more</h2> Valkey 9.0 brings numerous small changes and improvements: 1 Billion Requests/Second with Large Clusters</a>: Improvements in the resilience of large clusters, enabling scaling to 2,000 nodes and achieving over 1 billion requests per second,</li> Pipeline Memory Prefetch</a>: Memory prefetching when pipelining, yielding up to 40% higher throughput,</li> Un-deprecation</a>: In a similar vein to numbered databases, the Valkey project re-evaluated 25 previously deprecated commands and, based on the stance of API backward compatibility, restored the usage recommendation for these commands.</li> Zero copy Responses</a>: Large requests avoid internal memory copying, yielding up to 20% higher throughput,</li> Multipath TCP</a>: Adds Multipath TCP support which can reduce latency by 25%.</li> SIMD for BITCOUNT and HyperLogLog</a>: optimizations that yield up to a 200% higher throughput,</li> By Polygon for Geospatial Indices</a>: query location by a specified polygon,</li> Conditional Delete</a>: Adds the DELIFEQ</a> command that deletes the key if the value is equal to a specified value,</li> CLIENT LIST Filtering</a>: options to filter CLIENT LIST</a> using flags, name, idle, library name/version, database, IP, and capabilities.</li> </ul> And, perhaps, most importantly, a new, whimsical LOLWUT</a> generative art piece especially for version 9. Get it today</h2> Valkey 9.0 was built by the collaborative efforts of dozens of contributors. Make sure and grab Valkey 9.0 today as a binary, container</a>, or build it from source</a> and watch for it in your favourite Linux distribution. Feel free to post a question on our GitHub discussions or Slack and if you find a bug, make sure and tell the team as an issue. Scaling a Valkey Cluster to 1 Billion Request per Second 2025-10-20T00:00:00+00:00 The upcoming Valkey 9.0 release brings major improvements in the resilience of large clusters, enabling scaling to 2,000 nodes and achieving over 1 billion requests per second, all while ensuring bounded recovery time. In this blog post, we provide an overview of how the Valkey clustering system works, along with the architectural improvements and rigorous testing that made this level of scale possible. Valkey’s standalone configuration is a single server setup with optional replicas for availability, but all writes flow to one primary. It is one process, one dataset, zero coordination, blazing fast and simple to operate when a single machine’s CPU, memory, and NIC can carry the load. However, Valkey at-scale moves past single node limits. Valkey’s cluster mode shards the keyspace into 16,384 hash slots and spreads them across multiple primaries with replicas for redundancy. Clients are cluster aware: they route commands directly to the node owning the slot and follow redirections during resharding or failover. The result is horizontal scalability, balanced throughput, and built in fault tolerance without a central coordinator. Behind the scenes, the cluster bus keeps this distributed system coherent, coordinating membership, gossip, and failover. Cluster Bus Overview</h2> Under the hood, nodes coordinate over the cluster bus a persistent TCP mesh with a lightweight, gossip-based protocol. It handles MEET</code> (discovery), PING</code>/PONG</code> heartbeats with piggybacked cluster topology, quorum-based FAIL</code> decisions, replica promotion elections, and epoch-based conflict resolution so the cluster converges cleanly after partitions. Since membership, health, and slot ownership flow through gossip, it allows Valkey to scale to large node counts while remaining resilient to node and network failures. Membership Discovery and Information Dissemination (Gossip)</h3> Nodes join a Valkey cluster by sending a MEET</code> message to any existing member (a seed). From there, the cluster bus, a mesh of persistent TCP links which spreads membership and topology using lightweight gossip of peers piggybacked on periodic PING/PONG heartbeats. This peer-to-peer exchange quickly converges every node in the cluster and which slots each primary owns, without any central coordinator. Cluster Bus: Failure Detection, Failover, and Epoch Update</h3> When a node misses heartbeats beyond the configurable node-timeout, peers mark it potentially failed (PFAIL</code> - suspect). If a quorum of primaries observes the timeout, they declare FAIL</code> and initiate failover: the replica(s) of the failed primary becomes candidate to take over the ownership of the shard and send vote requests to all the primaries. Primaries respond and the winner is promoted to serve the affected slots. Clients that hit the old owner receive MOVED/ASK</code> and transparently retry against the new primary. Major improvements to scale Cluster Bus in Valkey</h2> Across multiple major/minor versions, the community has improved cluster stability a lot and have made multiple changes to handle various failure scenarios to heal the system in a steady manner and avoid any manual intervention. Here are few of the interesting improvements: Multiple primary failures Failover operation across cluster is serialized. Hence, only one shard can undergo failover at a given point in time. When there are multiple primary outages across the cluster, it used to lead to collision of vote request from replica candidates of impacted shards. Due to the collision, the votes would get split and the cluster won’t reach to consensus of promoting a replica in a given cycle. With a large number of primaries failing, the cluster won’t heal automatically and further an administrator needs to manually intervene. This problem was tackled by Binbin Zhu</a> in Valkey 8.1 by introducing a ranking mechanism</a> to each shard based on lexicographic order of the shard ID. With the ranking algorithm, the shard with highest ranking would go sooner and the one with lower ranking would add additional delay to start the election for the given shard. This helped to get a consistent recovery time in a multiple primary outage.</li> Reconnection attempt storm to unavailable nodes Profiling revealed that with multiple node failures, a large chunk of compute goes into attempting to reconnect with the already failed nodes. Each node attempts to reconnect to all the failed nodes every 100ms. This lead to significant compute usage when there are hundreds of failed nodes in the cluster. In order to prevent the cluster from getting overwhelmed, Sarthak Aggarwal</a> implemented a throttling mechanism</a> which allows enough reconnect attempts within a configured cluster node timeout, while ensuring the server node is not overwhelmed.</li> Optimized failure report tracking Profiling also revealed that when hundreds of nodes fail simultaneously, the surviving nodes spend a significant amount of time processing and cleaning up redundant failure reports. For example, after 499 out of 2000 nodes were killed, the remaining 1501 nodes continued to gossip about each failed node and exchange reports, even after those nodes had already been marked as failed. Seungmin Lee</a> optimized the addition/removal of failure report</a> by using a radix tree to store the failure report time information rounded to every second and group multiple report together. This helps with cleaning up expired failure report efficiently as well. Further optimizations were also made to avoid duplicate failure report processing and save CPU cycles.</li> Pub/Sub System Cluster bus system is also used for pub/sub operations</a> to provide a simplified interface for a client to connect to any node to publish data and subscribers connected on any node would receive the data. The data is transported via the cluster bus. This is quite an interesting usage of the cluster bus. However, the metadata overhead of each packet is roughly 2 KB which is quite large for small pub/sub messages. The observation was the packet header was large due to the slot ownership information (16384 bits = 2048 bytes). And that information was irrelevant for a pub/sub message. Hence, Roshan Khatri</a> introduced a light weight message header</a> (~30 bytes) to be used for efficient message transfer across nodes. This allowed pub/sub to scale better with large clusters. Valkey also has a sharded pub/sub system which keeps the data traffic of shard channels to a given shard which is an major improvement over the global pub/sub system in cluster mode. This was also onboarded to the light weight message header.</li> </ul> Valkey 9.0 has plenty of other improvements to increase the overall stability of the clustering system. All these enhancements allowed us to scale to 2,000 nodes with bounded recovery time during network partitions and below we have documented the benchmark setup and the throughput we were able to attain. Benchmarking</h2> In order to scale the Valkey cluster to 1 billion requests per second (RPS) for write command, we decided to choose a SET type command to accurately reflect the scale. We have seen previous experiments where a single instance was able to achieve more than 1 million RPS</a> so the goal was to reach 1 billion RPS with a 2,000 node cluster, where each shard has 1 primary and 1 replica. A replica is added to each shard for better availability. Hardware Configuration For this experiment, Valkey cluster was deployed on AWS r7g.2xlarge</code> instance type, which is a memory optimized instance, featuring 8 cores and 64 GB memory on an ARM-based (aarch64</code>) architecture. In order to generate enough traffic across all the slots, we used 750 instances of AWS c7g.16xlarge</code>. System Configuration Note: The core assignments used in this guide are examples. Optimal core selection may vary depending on your specific system configuration and workload. Each of the Valkey server nodes had 8 cores, so we decided to pin 2 cores to ensure interrupt affinity. # identify the network interface, for this run, it was ens5 IFACE=$(ip route | awk '/default/ {print $5; exit}') # expose two combined Rx/Tx queues sudo ethtool -L "$IFACE" combined 2 # grab exactly the two IRQ numbers that belong to those queues IRQS=($(grep -w "$IFACE" /proc/interrupts | awk '{gsub(":","",$1); print $1}')) IRQ0=${IRQS[0]} IRQ1=${IRQS[1]} # pin queue-0 interrupt to CPU0 echo 0 | sudo tee /proc/irq/$IRQ0/smp_affinity_list # bind IRQ0 → CPU 0 echo 1 | sudo tee /proc/irq/$IRQ1/smp_affinity_list # bind IRQ1 → CPU 1 sudo systemctl stop irqbalance</code></pre> We increased the default limit of file descriptors of the instance so that the node is able to handle connections from the Valkey cluster nodes and clients. # increase max conn ulimit -n 1048544</code></pre> With 6 cores left on the node, we decided to assign all the remaining cores to the valkey-server</code>, so that the 6 io-threads</code> (including the main thread</code>) can leverage the most from the instance. # pin 6 cores to valkey-server CPUSET=2-7 sudo cset shield --cpu=$CPUSET --kthread=on sudo cset shield --exec taskset -- -c $CPUSET ./valkey-server valkey-cluster.conf --daemonize yes</code></pre> Server Configuration We launched each Valkey process with minimal changes to the default configurations. cluster-enabled yes # To enable cluster mode cluster-config-file nodes.conf # To persiste the topology information cluster-require-full-coverage no # To prioritize availability cluster-allow-reads-when-down yes # To allow reads and partial operations to continue even under slot loss save "" # Disable periodic snapshots io-threads 6 # Allow offloading io operations to separate io-threads maxmemory 50gb # Limit maximum memory utilization by the process</code></pre> Benchmark Configurations We ran the valkey-benchmark</code> from 750 client instances to generate the required traffic. We used the following parameters for each of those instances. Number of requests (-n): 100M</code></li> Number of clients (-c): 1000</code></li> Test (-t): SET</code></li> Data Size (-d): 512</code></li> Threads: 20</code></li> </ul> Overall observation has been that the throughput grows almost linearly with the overall count of primaries and cluster bus has a minimal overhead with the default node timeout. Recovery</h3> In the same environment, we tested the recovery time of the cluster when multiple primary nodes go down. We picked up to 50% of the primaries and killed them and let the automatic failover process to heal the write outage for a given shard. To simulate primary node failures, we sent SIGKILL</code> to hard-stop the Valkey process, and let the replica take over through automated failovers. The recovery time measured here is from the moment any of the nodes discovered a primary in a PFAIL</code> state until the cluster reported ok and all slots were covered again. Closing thoughts</h2> With all these improvements made in Valkey, a cluster can now scale to 1 billion RPS using 2,000 nodes which is quite a remarkable feat. However, there is plenty of room to improve further. The steady state CPU utilization overhead from the cluster bus message transfer/processing can be reduced further by incorporating the SWIM protocol</a> or moving the cluster bus messsage processing off the main thread into an independent separate thread. The failover logic can be made smarter as well by incorporating the AZ placement of nodes. We would also like to introduce more observability metrics/logs into the system for better manageability. All of these are being linked under the Support Large Cluster</a> issue. Feel free to check it out and add in your suggestions. How Valkey 8.1 Handles 50 Million Sorted Set Inserts 2025-10-02T00:00:01+00:00 When you run infrastructure at scale, the smallest efficiencies compound into massive savings. Sorted sets (ZSETs) are the backing data structure for far more than leaderboards. They're used for time‑ordered feeds, priority queues, recommendation rankings and more. Each entry carries per‑item overhead; when you're inserting tens of millions of items, those bytes accumulate into gigabytes. The latest Valkey 8.1 release introduces a redesigned hash table</a> and other optimizations that promise lower memory usage and higher throughput. In this post, we put Valkey 8.1 under pressure by benchmarking it against Valkey 8.0, inserting 50 million members into a sorted set and measuring memory consumption and throughput along the way. About Momento and Raider.IO</h2> Momento builds enterprise‑grade real‑time data infrastructure. Our flagship service, Momento Cache</a>, allows developers to focus on innovation rather than maintaining complex back‑ends. Behind the curtains, Valkey is one of the primary storage engines we operate. Its memory efficiency and predictable performance make it an ideal choice for mission‑critical workloads. By building on Valkey, we can deliver caching experiences that are fast, reliable, and cost‑effective. One such workload is Raider.IO</a>, the most popular companion site for World of Warcraft players. Raider.IO maintains leaderboards that track millions of characters and guilds, updating instantly whenever a player clears a dungeon or raid. That translates into hundreds of millions of sorted set operations every day. In this environment, downtime or latency isn't acceptable. Memory efficiency and throughput directly shape the player experience. Running Valkey at hyperscale for workloads like Raider.IO means that improvements in Valkey 8.1 immediately impact both infrastructure cost and the ability to absorb peak traffic without breaking a sweat. Benchmark Setup</h2> To provide a fair comparison, both Valkey 8.0 and Valkey 8.1.1 were run on the same hardware: AWS c8g.2xl instance (Graviton4, 8 vCPU, 16 GB RAM)</li> Identical settings for persistence, I/O threads and memory limits</li> 50 million members inserted into a ZSET via pipelined ZADD commands (score=i, member=m:{i})</li> 250,000 item batch size per pipeline</li> Metrics collected every 1,000,000 inserts (flush between runs)</li> </ul> The benchmark code is open sourced</a> and straightforward to reproduce: it connects to both servers, flushes the test key, performs batched inserts, and records used_memory</code>, used_memory_rss</code>, total elapsed time, and throughput after each million inserts. This repeatability mirrors the ethos of Valkey's community - every optimization is measurable, and anyone can verify the results. Memory Usage – 8.1 vs 8.0</h2> Valkey 8.1's redesigned dictionary structure cuts roughly 20–30 bytes per key</a>, and those savings compound with scale. The figure below charts used_memory</code> during the benchmark for both versions: At 1 million inserts, Valkey 8.0 used ~95 MB while Valkey 8.1 used ~81 MB. As the ZSET grew, the gap widened. By 10 million inserts, 8.0 consumed 1.06 GB versus 0.77 GB for 8.1 - a 27% reduction. At the end of the run (50 million inserts), 8.1 used 3.77 GB compared to 4.83 GB on 8.0, saving 1.06 GB (≈22 %). These numbers align with the release notes. Valkey's 8.1 announcement highlights lower per‑key overheads and improved data structure handling; Linuxiac notes that 8.1's architectural changes can reduce memory footprints by approximately 20 bytes per KV pair</a>, with each pair normally consuming 100 bytes (a 20% reduction!). Our results also confirm a similar improvement on large ZSET workloads. Throughput and Total Time</h2> Memory isn't the only metric that matters. Throughput determines how quickly you can fill or update a set. Although Valkey 8.0 started strong with about 661k inserts per second at the 1M mark, its throughput gradually declined as the data set grew. Valkey 8.1 maintained higher and more consistent throughput, finishing the benchmark at 573k/s compared with 532k/s for 8.0. Overall time to insert 50 million members dropped from 94 seconds on 8.0 to 87 seconds on 8.1, a 7% improvement. The smoother throughput curve is just as important as the raw peak. Spiky performance complicates capacity planning, especially when running close to memory limits. By contrast, Valkey 8.1's combination of improved hash‑table implementation and I/O thread optimizations delivers predictable scaling. Why These Gains Matter</h2> For Raider.IO and similar workloads, these improvements translate directly into better economics and user experience: Lower infrastructure cost - Saving over a gigabyte per 50 M members means fewer nodes are needed to support the same dataset. At hyperscale, this compounds into significant cost reductions.</li> More headroom during peaks - Efficiency gains provide a buffer during seasonal events like game expansions or esports tournaments. When leaderboards surge, there's less risk of hitting memory ceilings.</li> Predictable scaling - A smoother memory growth curve and stable throughput make it easier to forecast when capacity needs to be added and to automate scaling policies.</li> </ul> These benefits don't just apply to leaderboards. Time‑ordered feeds (such as activity streams), job scheduling queues, recommendation rankings, rate‑limiting windows and search scoring all rely on sorted sets. In every case, per‑entry overhead multiplied by millions of entries determines whether you can do more with less. As described in the Linux Foundation press release</a>, upgrading to Valkey 8.1 can reduce memory footprints for common workloads by up to 20%, helping enterprises scale while keeping costs in check. Closing Thoughts</h2> As an operator who lives and breathes real‑time data systems, I'm continually amazed by the pace of innovation in the Valkey community. We didn't tune any configuration knobs to achieve these results - Valkey 8.1's efficiency is built in, and the improvements materialized instantly once we upgraded. On a 50 million‑entry benchmark, the new release used 27% less memory while delivering about 8% higher throughput, and completed the workload seven seconds faster than its predecessor. Those deltas may seem small in isolation, but at hyperscale they compound into transformative savings and more resilient services. If you're running sorted set heavy workloads - whether leaderboards, feeds, queues or scoring engines - I encourage you to upgrade to Valkey 8.1 and run this benchmark yourself. Interested in how Valkey 8.1 stacked up to its competitors? So were we</a>! Our code is open source</a> and available to run on your own hardware. I think you'll be pleasantly surprised by how much headroom you gain and how effortlessly Valkey handles pressure. The power of open source and community‑driven engineering continues to shine through in every release. Introducing Hash Field Expirations 2025-09-30T00:00:00+00:00 One of the great strengths of Valkey has always been its built-in ability to expire keys. This simple but powerful mechanism lets developers keep their datasets fresh, automatically clear caches, or enforce session lifetimes without additional logic. But there has always been one limitation: expiration worked at the level of whole keys. If you stored multiple fields in a hash, you could set a TTL for the hash itself, but you couldn’t say “this field should live for 10 seconds while another field should stick around for 10 minutes.” That limitation forced developers into awkward choices. Imagine you’re running a feature flag system. You want to store all of a customer’s feature toggles inside one hash: Some flags are short-lived experiments, meant to turn off automatically after a few seconds or minutes. </li> Others are long-term rollouts, where the toggle might remain valid for days or weeks. </li> </ul> If you only had key-level expirations, you were forced to compromise. You could put all flags for a customer into a single hash, but then they would all share one expiration time. Or, you could store each flag in its own key so that each could expire independently — but this would explode the number of keys your system has to manage, hurting memory efficiency and increasing overhead for operations. The inability to set different TTLs per field meant developers either built complex cleanup processes outside of Valkey, or gave up flexibility in how expirations were handled. Finding the Right Expiration Model</h2> At first glance, adding field expirations might sound like a simple matter of storing timestamps per field. In order to understand the problem, it’s important to understand how expiration is normally handled in Valkey today. Valkey uses two complementary mechanisms to reclaim expired keys: Lazy expiration: A key is deleted only when accessed. If you try to read or write an expired key, Valkey notices it has passed its TTL and deletes it immediately. This is cheap, but untouched keys would linger indefinitely and waste memory. </li> Active expiration: A background cron job runs 10 times per second, sampling a small set of keys with expiration. Expired keys are deleted until a time budget is reached, ensuring memory is reclaimed proactively without introducing latency spikes. </li> </ul> For hash fields, we made a key design choice: we did not implement lazy expiration. Adding expiration checks to every HGET</code></a>, HSET</code></a>, or HDEL</code></a> would have complicated the hot path of hash commands and risked performance regressions. Instead, we extended the active expiration job so that it can now also scan field-level expiration buckets, alongside top-level keys. This approach keeps expiration logic unified, predictable, and free from new latency cliffs. But it also introduced a challenge: how do we efficiently track and clean up expired fields inside hashes that may contain thousands (or millions) of entries, when only a small subset are volatile — and an even smaller fraction are expired at any given time? The challenge is balancing three conflicting goals: Keep memory usage small. Hashes can grow to millions of fields, so metadata overhead must be minimal. </li> Maintain fast lookups and updates. Most hash commands run in O(1)</code> time, and expiration tracking can’t change that. </li> Reclaim memory efficiently. The active expiration job is time-bounded, so we need to minimize wasted CPU cycles spent scanning unexpired fields. </li> </ul> Approaches Considered</h3> "We explored several ways to solve this problem: 1. Secondary hashtable per field expiration</h4> This seemed simple: build a secondary hashtable in each Hash object which maps only volatile fields. This is actually the way volatile generic keys are being tracked. Per each database, Valkey manage a secondary map mapping volatile keys. During active expiration process the existing job scan the secondary map and each key found to pass it's assigned TTL is expired and it's memory is reclaimed. The problem with this design option is the potential inefficiency introduced while scanning many items which should not expire. 2. Radix Tree-based index</h4> Using a radix tree to hold field names plus expirations provides sorted access for free. This is also not a new concept in Valkey. In fact this exact structure is being used to manage client connections blocked on key operations like BLPOP</code></a> and XREAD</code></a>. But the memory overhead per node was high. During experiments we measured more than 54 bytes overhead per each hash field when using this type of index. 3. Global sorted structure</h4> We wanted to have the ability to efficiently scan over fields which are already expired. A good way to achieve this is by using a sorted index. Using a radix tree was possible, but the memory overhead was high. Instead we could use a more lightweight data structure like a skip list, which memory consumption is more bounded and is not governed by the data atrophy. However, This would give O(log N)</code> access to the index which did not work well with our target to keep hash operations constant time complexity. Coarse Buckets with Adaptive Encoding</h3> Instead of tracking every field’s expiration individually, we designed a coarse bucket system. Each field’s timestamp is mapped into a time bucket, represented by a shared “end timestamp.” This solution introduces a semi-sorted data structure which we named 'vset' (stands for "volatile set"). The volatile set manages buckets in different time window resolutions and adaptive encodings. Buckets can split if too many expirations cluster in one interval. This adaptability keeps the number of buckets small while ensuring they’re fine-grained enough for efficient cleanup. When a new field with TTL is added, the volatile set either places it into an existing bucket or creates a new one. If a bucket grows too large, it is narrowed into smaller time windows to keep expiration scans efficient. For example: scanning 1,000 items in a single 10-second bucket may result in mostly misses, while spreading them across 10 smaller buckets avoids wasted work. Why not always use a very small time window (e.g few milliseconds) for each bucket? The reason is memory. Recall that the buckets are managed in a Radix-Tree which introduces a high overhead per each item. When many buckets are used, the memory overhead can be high and we might end up with the same memory overhead introduced by using a Radix-Tree index. The volatile set also uses different bucket encoding based on the number of items in the bucket. A bucket with only a single element would require just a single pointer size of bytes. A bucket of small amount of items would be encoded as a vector of item pointers, and when the bucket contains many items we will use the Valkey hashtable</a> structure to map the relevant items. This way, we are leveraging existing data structures which are highly optimized for modern CPUs. A hash object that contains volatile fields now also carries a secondary volatile set index. At the database level, we maintain a global map of hashes with volatile fields. The active expiration cron job scans both regular keys and these hashes, but only iterates over volatile set buckets whose end time has passed. This ensures that CPU time is spent only on fields that are truly ready to expire. Benchmarking our solution</h2> Validating the new design meant benchmarking across several dimensions: memory overhead, command performance, and expiration efficiency. Memory Overhead</h3> We first measured the per-field memory overhead when setting TTLs. The raw expiration time itself requires a constant 8 bytes (though this could be reduced in the future by storing only a delta relative to a reference timestamp, such as server start time). On top of that, extra memory is needed for tracking within the volatile set. The actual overhead depends on both how many fields have expirations and how spread out their expiration times are. This is because the bucket encoding chosen by the volatile set adapts to the data distribution. In practice, the overhead ranged between 16 and 29 bytes per field. The higher end of this range primarily affects small hashes, where compact encodings like listpack are avoided when volatile fields are present. Command Performance</h3> Next, we benchmarked common hash commands both with and without field expirations. The results showed no measurable performance regression and throughput remained stable when expirations were added. We also benchmarked the new expiration-aware commands (e.g., HSETEX</code>), confirming that their performance is on par with traditional hash operations. Active Expiration Efficiency</h3> The design goal of the volatile set was to enable efficient background deletion of expired fields. To test this, we preloaded 10 million fields with TTLs. We distributed these fields across varying numbers of hash objects to see how object size influences expiration. During the load phase, we disabled the expiration job using the DEBUG</code></a>, then re-enabled it once all fields had expired. The following chart shows the time it took the expiration cron job to complete the full deletion of all the 10M fields. The results revealed that expiration time depends not just on the number of fields, but also on how they are distributed across objects. Smaller hashes tend to fit into CPU caches, so random field deletions remain cache-friendly. Very large hashes, however, cannot fit entirely in cache, which means more expensive memory lookups during expiration. Another important factor is CPU utilization. The active expiration job is deliberately CPU-bounded and designed to use no more than ~25% of available CPU time (unless configured to work more aggressively), preventing it from overwhelming the system. The chart above shows that average CPU usage was consistently kept under this cap, even when expiring millions of fields, ensuring predictable tail latency. We also simulated a more realistic scenario: expiring data during continuous ingestion. Using the 'valkey-benchmark' executing the new HSETEX</code></a> command, we continuously inserted 10 million fields with 10-second TTLs, while the active expiration job ran in the background. This setup maintained a constant pool of fields at different stages of their lifecycle — some fresh, some nearing expiration, some ready to be reclaimed. We then tracked memory usage over a 5-minute period. The results aligned with our theoretical expectation: Memory = (Injection Throughput) x (AVG TTL) x (AVG Item memory)</code></pre> In our experiment: Injection rate: 300K commands/sec </li> Average TTL: 10 seconds </li> Base item size: ~61 bytes </li> Additional expiration overhead: ~19 bytes </li> </ul> This yields an expected memory footprint of ~230MB: 300K × 10 × 80B = 230MB</code></pre> The observed memory matched this estimate closely, demonstrating that the active expiration mechanism is able to keep up with load and prevent memory from spiking unexpectedly. Benchmark Takeaways</h2> The benchmarks demonstrate that field-level expirations can be added to Valkey without compromising memory efficiency, or latency. The memory overhead remains modest and predictable, command throughput is unaffected, and the shared active expiration job efficiently reclaims memory even under heavy ingestion workloads. Together, these results validate that the coarse-bucket design with adaptive encoding delivers the right balance of efficiency, scalability, and correctness, while preserving Valkey’s reputation for high performance and low latency. Using Hash Field Expiration in Valkey</h2> Valkey 9.0 introduces a new API for hash field expirations, fully compatible with the Redis 8.0 API. Any existing client library supporting this API can be used. In this example, we will use the latest valkey-glide</a> 2.1.0 release, which also supports the new hash field expiration commands. Let's start with a simple example. First, we create a client connecting to a local Valkey server: from glide_sync import GlideClientConfiguration, NodeAddress, GlideClient addresses = [NodeAddress("localhost", 6379)] config = GlideClientConfiguration(addresses, request_timeout=500) # 500ms timeout client = GlideClient.create(config)</code></pre> Next, let's create a new hash object to store some random user data: client.hset("User1", {"name": "Ran", "age": 'old', "password": "1234"}) > 3</code></pre> Suppose we want the user’s password to be available only for a specific timeframe (e.g., 60 seconds). With the new hash field expiration feature, we can now do that: client.hexpire("User1", 60, ["password"]) > [1]</code></pre> Note that in this example, we only set a TTL for a single field, but the command can accept multiple fields at once. We can check the remaining TTL for the password field: client.httl("User1", ["password"]) > [46]</code></pre> The reply shows the remaining time in seconds for the field to live. We can also query the absolute expiration time: client.hexpiretime("User1", ["password"]) > [1757499351]</code></pre> This is a Unix timestamp, which we can easily convert to human-readable time: date -d @1757499351 > Wed Sep 10 10:15:51 UTC 2025</code></pre> After 60 seconds, if we read the hash: client.hgetall("User1") > {b'name': b'Ran', b'age': b'old'}</code></pre> Notice that the password field is no longer present, since it has expired. Another practical use case is managing a collection of links where each field represents a URL along with its metadata: links:valkey:blogs -> { "https://valkey.io/blog/valkey-is-a-key-value-store" -> {votes: 15, category: "tech"}, "https://valkey.io/blog/valkey-supports-different-ai-workloads" -> {votes: 42, category: "ai"}, "https://myblog.com/blog/how-to-write-good-valkey-blog" -> {votes: 7, category: "blog"} }</code></pre> Here’s how hash field expirations help: Each link can have a TTL representing its “relevance window.” If a link hasn’t been accessed within a certain period, it automatically expires.</li> Every time a user queries or updates a link, its TTL can be refreshed, keeping active links alive while letting inactive ones naturally fall off.</li> Expired links are removed automatically, so you don’t need extra cleanup logic and can focus on active links.</li> </ul> Using Valkey’s HSETEX</a> and HGETEX</a>, we can update a field’s TTL whenever it’s accessed or modified: # Set or update link metadata with a 30-day TTL client.hsetex( 'links:user:42', { 'https://valkey.io/blog/valkey-is-a-key-value-store': '{"clicks":15,"category":"tech"}' }, expiry=ExpirySet(ExpiryType.MILLSEC, 30*24*3600)) # Retrieve a link and refresh its TTL link_data = client.hgetex( 'links:user:42', {'https://valkey.io/blog/valkey-is-a-key-value-store'}, expiry=ExpirySet(ExpiryType.MILLSEC, 30*24*3600) )</code></pre>What’s Next?</h2> What we’ve shipped so far is just the first step: the ability to set and control time-to-live at the hash field level. But we’re not stopping here. Future work will focus on two main areas: reducing memory overhead and improving performance. For example, we plan to support compressed encodings for hashes with volatile fields, and to leverage modern CPU features such as memory prefetching and SIMD instructions to speed up operations. Another critical area for improvement is the active expiration job. Today, all volatile data in Valkey is tracked in unsorted maps (hashtables). The background job must repeatedly scan these unordered sets, which means wasting CPU cycles on entries that aren’t close to expiration. By introducing structured tracking—through the volatile set or even alternative approaches like Hierarchal Time Wheels</a> —we can significantly reduce wasted work and make expiration more efficient at scale. Numbered Databases in Valkey 9.0 2025-09-25T00:00:00+00:00 If you explore Valkey’s documentation you might run across a feature called ‘numbered databases’ which allows you to separate the keyspace into (by default) 16 different databases. Digging into this feature reveals tantalizing ways to avoid key prefixing, house different workloads together on Valkey, and even perform patterns that are otherwise clunky. However, if you’ve done more research outside of the documentation on numbered databases you find advice like “don’t use them,” “they don’t scale,” and “they’re a bad idea.” Well, the forthcoming Valkey 9.0 changes many things with numbered databases and you’ll see in this post that advice definitely needs some updating. Today, a common way to conceptualize Valkey is that your keys represent a unique name for pointers to data in memory across a cluster of nodes. So, key foo</code> is unique and deterministically linked to a specific node and on that node there is a memory address where the value resides. However, this misses one important detail: the database number. The reality is that key names belong to a specific numbered database and aren’t unique on a given instance of Valkey. To put this another way, Valkey can have the key foo</code> as many times as there are numbered databases with each one pointing to different data. Historically, before Valkey’s preceding project had the ability to cluster (before version 3.0.0), using multiple numbered databases was fully supported. However, when clustering was implemented numbered databases weren’t included: a cluster had one database (DB 0) that spanned across the entire cluster. In a world where using numbered databases in your application locked you into never going to a cluster, the early advice made sense, however Valkey 9.0 adds the ability to have numbered databases on a cluster, changing everything about that advice. Why bring this feature to cluster mode in Valkey 9.0? In the intervening years since numbered databases were left out of the cluster spec, users have found a number of very handy patterns that were unfortunately limited without numbered databases. However, numbered databases aren't a panacea: read on to find out when you should stick to DB 0 and when to go for a numbered database. Clustering and numbered databases</h2> As mentioned earlier, the key name dictates where the key lives in the cluster and this doesn’t change for numbered databases. As a refresher, Valkey clustering</a> takes the key name as a string, runs that through a CRC-16 function, then does a modulo of 16,384 which determines the ‘slot’. Each of these slots belong to a node in the cluster. For a numbered database, each slot contains all the numbered databases, so, carrying forward the idea that you can have the same key name in each database on a single instance, in Valkey 9.0 you can have the same key multiple times in a given slot, each in their own database. In other words, numbered databases do not affect clustering: the key is still the determinate factor in calculating slots. You can see this directly with CLUSTER KEYSLOT</code></a> and SELECT</code></a>. Take the following example: > SELECT 0 OK > SET somekey hi OK > CLUSTER KEYSLOT somekey (integer) 11058 > SELECT 5 OK > SET somekey hello OK > CLUSTER KEYSLOT somekey (integer) 11058</code></pre> Here the key somekey</code> was set to two different values: ‘hi’ on database 0, and ‘hello’ on database 5, with SELECT</code> altering the selected database. CLUSTER KEYSLOT</code> was called twice: each one yielding the same slot number meaning this key will be assigned to the same node in the cluster, no matter which database is selected. Limitations</h2> Numbered databases do not change the properties of how a Valkey cluster works. The keyspace of each database is still split amongst all the nodes. As a consequence, operations that need to span the entire keyspace will to be run on each node: FLUSHDB</code></a> will flush the keys in the current database on the connected node</li> SCAN</code></a> will iteratively return keys in the current database on the connected node.</li> DBSIZE</code></a> will return the number of keys in the current database on the connected node.</li> </ul> You get the picture: if a command previously said it did something for the entire database, in cluster mode it really means for the connected node’s portion of the database. These are especially important to understand if you're planning to move an application built for non-clustered, numbered databases to a cluster. Additionally, numbered databases do not provide any form of resource isolation. It's tempting to point a bunch of applications to a single Valkey cluster with each application taking its own database. While this certainly can work, it works without resource isolation and this setup can suffer from classic noisy neighbour problems: busy applications will affect the others using the same cluster. If you're worried about resource sharing, most of the time you'll be better off just having distinct clusters for each application instead of numbered databases. Where to use numbered databases</h2> If you take away one thing from this blog it should be this: numbered databases are a form of namespacing. The most straight forward use case of numbered databases is when you need to separate your data logically and you can tolerate the effects of resource sharing (see above). This might be something like keeping customer data separated from one another or combining applications on to a single cluster when resources are unlikely to be an issue. In a similar manner, multiple databases are a useful debugging tool. When you’re building an application it can be difficult to see what happens inside Valkey when you make a change. If you have multiple databases you can run your original code on one database and the changed version on a different database then you can more easily see changes in how data looks in Valkey by just the connection swapping between databases. This is also a useful pattern during a migration when you want your old data to stick around while your new data is populated. An entirely different use of numbered databases is related to the MOVE</code></a> command. MOVE</code> allows you to change a key from one database to another without copying the data, meaning it uses very little resources and it’s an O(1)</code> operation. This allows a couple of things: 1) you can effectively make data inaccessible from a database whilst keeping it on the same cluster node, and 2) you can replace a complex key atomically. Looking at the first use of MOVE</code>: imagine you have some content, maybe a user submitted post, that is stored at the key “mycontent.” At some point this content gets flagged as needing to be reviewed. You might not want to delete this content but you also don’t want it accessible. In this case you take the content and MOVE</code> it to a different database. Once it’s reviewed (or edited!) you can MOVE</code> it back. Never does the data actually leave the node nor get copied. Whilst your use-case might not be the same as this, the pattern illustrates an operation useful in a variety of contexts. The second use of MOVE</code> is similar to the first except it revolves around complex keys that contain many elements. Take, for example, a large sorted set. It’s not unusual for these keys to be quite large because the key contains thousands or millions of smaller elements. If you need to replace this key with one that contains different elements then you’d have to first DEL</code> the original key, then ZADD</code> thousands of elements. Doing this in a transaction would be expensive and monopolize the node, and without a transaction, it would reveal a (potentially undesirable) partial state. Instead, with multiple databases, you build up new a complex key on a different database (under no urgency) and do a much simpler MULTI</code> / EXEC</code> transaction of of DEL</code> followed by MOVE</code> to atomically make the new complex key available to the database being used to serve commands without the chance of a partial state. This is really a form of temporal namespacing: you're taking advantage of the namespace to have two active keys under the same name both of which can be read and manipulated. You can apply this pattern to other cases: anyplace you want to make a key inaccessible for a period of time. How is key prefixing different than numbered databases? Some of the patterns in this blog can be achieved without numbered databases and instead with key prefixing (e.g. app0:...</code>, app1:...</code>). Key prefixing has a few downsides when compared to numbered databases: Prefixes add up at scale. Millions of keys each with repeated prefixes over the span of the whole cluster means less RAM for data. While each database does have a memory overhead it is linear relative to the number of databases used, not keys. For a deeper look at how numbered database memory overhead works, including using 10 million databases, read the comments on valkey-io/valkey#1609</a>.</li> Databases are transparent to your application. To support prefixing, either your application or your client library has to be able to interpolate the prefix into each key. With numbered databases, the changes can be as simple as a number in the connection URI, not to each key name.</li> Databases avoid pattern-based iterations. Having all your keys in DB 0 then needing to iterate over the entire keyspace to affect a specific pattern is expensive and complicated. If your keys are separated into databases, this both subsets that iteration and enables some database wide commands (i.e. FLUSHDB</code> instead of trying to delete keys by a pattern).</li> </ul> </blockquote> Gotchas and future work</h2> Aside from the aforementioned lack of resource isolation, numbered databases in clustered Valkey have a few places that you need to look out for today. First, right now numbered databases don’t have a lot of dedicated metrics, so it can be hard to gain insights into per-database resource usage. Second, the ACL system doesn’t currently address numbered databases, so you can’t meaningfully restrict access to databases (at time of writing, there is an open pull request that addresses this</a>). Unlike in the previous project, databases are an active area of work for the Valkey project and the database abstraction holds potential as a way to implement new features. Finally, while numbered databases are well supported in client libraries there are rough edges: Some client libraries artificially restrict usage to a single database in cluster mode,</li> Pooled clients may naïvely manage the selected database, so a client returned to the pool after running SELECT</code> might retain the database number in subsequent usage. A similar situation is possible for multiplexed clients.</li> </ul> In general, watch out for three assumptions: 1) the selected database is always 0, 2) there is only one database in cluster mode, and 3) if there are multiple databases in use it isn’t in cluster mode. None of these are true in Valkey 9.0. With this in mind, Valkey 9.0 gives you the ability to divvy up that keyspace into nice neat numbered databases and spread them out over a whatever cluster you have. So, get rid of that old, outdated advice and start using them, seeing how they scale, and what a good idea number databases actually are for Valkey 9.0. Valkey: An Investment in Open Source 2025-09-16T00:00:00+00:00 Valkey was founded just over a year ago to keep high-performance key/value storage in the open source community: free from vendor lock-in and restrictive licenses. Backed by contributors like AWS, Google Cloud, Ericsson, Oracle, Alibaba, Huawei, Tencent, Percona, Aiven, Heroku, Verizon, Chainguard, and Canonical, the project shows how “free” in open source depends on investment: time, talent, and ongoing financial support. Built by Builders, Backed by the Linux Foundation</h3> Valkey was created on March 28, 2024 and has solidified itself as the open-source high-performance key/value datastore that supports a variety of workloads such as caching, message queues, and can act as a primary database. Valkey is backed by the Linux Foundation, a neutral organization founded in 2000 that supports developers and technologists to scale and manage open source projects. Valkey operates under an open governance model focused on growing community contributions and adoption. One Year In: Growth and Momentum</h3> In just a year’s time the project has celebrated two major releases, an increase in corporate participants from 22 to 47, is defining, innovating, and executing on its roadmap, and is growing in its adoption. Users and the community have embraced the project and they are firm in their commitment to continue to improve Valkey for the benefit of all end users. New Valkey Features & Use Cases</h2> Solving Practical Problems with JSON and Bloom Filters</h3> Two new data types, JSON and Bloom filters, expand what developers can do with Valkey, especially in distributed systems where speed and structure are equally important. These additions reduce complexity in application logic and improve how data is handled at the edge. “Adding JSON and Bloom filters to Valkey is about giving developers practical tools to solve real-world problems in distributed systems,” says Madelyn Olson, Valkey co-maintainer. JSON support allows developers to work with rich, structured data natively, instead of relying on custom serialization or extra middleware. “JSON lets you work with rich, structured data directly in Valkey, which simplifies development and reduces glue code,” Olson explains. Valkey Bloom filters</a>, on the other hand, offer a compact way to perform fast checks. “Bloom filters are ideal when you need to make fast, memory-efficient existence checks, whether you're trying to catch fraud or avoid unnecessary backend calls in a high-traffic service,” adds Olson. As Olson put it, the goal is simple: “I’m really excited to see what problems our users will be able to solve with these new data types.” Valkey Search: Speed and Scale for AI Workloads</h3> Google contributed a new module, Valkey Search</a>, which brings vector similarity search to Valkey. It’s fast, delivering single-digit millisecond latency and built to handle billions of vectors while maintaining over 99% recall. Developers can run both approximate nearest neighbor (ANN) searches (via HNSW) and exact k-nearest neighbor (KNN) searches. It supports indexing with either Hash or Valkey-JSON</a> data types. While Valkey Search currently focuses on vector search, future goals would extend Valkey into a broader search engine with full-text support and more indexing options. Performance, Reliability, and Security</h3> Over the past year, contributors from companies like Ericsson, Oracle, and Percona have focused on making Valkey faster, safer, and more enterprise-ready. The following improvements showcase how Valkey is evolving to meet scaled production needs without compromising for any of its users. Multithreading Improvements</h4> Recent updates to Valkey’s internals improve how the system handles work across multiple threads, cutting down on lock contention and better using multi-core processors. This update significantly improves throughput under high concurrency, making the project more attractive for large-scale production use. Ericsson, one of Valkey’s core contributors, is already seeing real value in the project’s performance work. Viktor Söderqvist, Ericsson Software Technology Engineer & Valkey co-maintainer, points out the core efficiency as a key reason Valkey is becoming more production-ready. “To get the most out of the CPUs and memory in terms of storage efficiency and speed, the recent hashtable redesign lately improved with SIMD techniques as well as the recent and ongoing improvements in multithreading and memory batch-prefetching techniques are exciting areas of improvement.” SIMD-Accelerated Hashtable Redesign</h4> The project’s core hashtable was reengineered to leverage SIMD instructions, allowing Valkey to process key lookups more efficiently by handling multiple operations in parallel. These low-level optimizations translate to faster response times in latency-sensitive environments. Batch Memory Prefetching</h4> New support for batch memory prefetching helps reduce cache misses by anticipating access patterns and loading data proactively. The result is smoother performance and more consistent behavior under heavy or sequential access workloads. LDAP Integration</h4> LDAP integration brings centralized authentication and access control to Valkey, making it easier to deploy in enterprise environments with existing identity infrastructure. This feature addresses a common adoption barrier in security-conscious and compliance-driven organizations. “A lot of our customers already rely on LDAP to manage access across their infrastructure, so bringing that to Valkey just made sense,” says Vadim Tkachenko, Co-Founder of Percona. “It’s one of those features that removes friction. You get auditability, group-based permissions, and it works with what teams already have in place.” Rust Module</h4> Oracle is contributing a Rust-based module SDK to Valkey, aimed at improving safety and performance for low-level extensions. By adopting Rust’s strong guarantees around memory and concurrency, the project opens the door to safer, more maintainable systems integration, especially for production environments under load. The Real Investment Behind Open Source Success</h3> The investment in open source and choosing to employ contributors to the project clearly demonstrates that a key aspect in technological innovation is for companies at varying levels, enterprise to mom-and-pop shops, to succeed they need to work together and that means investing in the resources that can move projects forward - the human capital of open source, the coders, the writers, the governance, and the community behind the ‘free’ moniker that we frequently associate with open source. What’s Happening With Valkey</h3> On August 15, Valkey dropped the first release candidate for 9.0</a>. This testing release previews the new capabilities around atomic slot migrations, hash field expirations, and numbered databases in cluster mode alongside scads of performance enhancements and bug fixes. Additional release candidates are to follow with general availability of 9.0 in early autumn 2025. Parallel to the testing phase for 9.0, two events happened last month in Amsterdam: Open Source Summit Europe - Aug 25–27, 2025</a>: Three-day conference for open source developers, technologists, and leaders featuring keynotes, sessions, and community networking.</li> Valkey User Conference: Keyspace - August 28, 2025</a>: A one-day conference for developers, SREs, and DevOps teams with sessions, lightning talks, and workshops.</li> </ul> What you'll see at Keyspace 2025 2025-08-12T01:01:01+00:00

Community Growth</h2>
In 2025, the Valkey project had 346 active contributors which are individuals who performed activities such as commits, issues, or pull requests during the selected time period. (Source: LFX analytics</a>)</p>

Where to next for the Valkey community</h2>
In 2026, Valkey will be on the road again, with appearances planned at the Unlocked conference</a> and Laracon India 2026</a> from January 31st - February 1st, with even more events to be announced as the year unfolds.</p>

Thank You</h2>
Thank you to everyone who contributed code, tested releases, filed issues, shared feedback, or adopted Valkey in production. Your continued involvement strengthens and advances the Valkey project.</p>

Legacy vs. Atomic: Head-to-Head Results</h2>
Head-to-head experiments show the improvement provided by atomic slot migration.</p>

Valkey 9.0: innovation, features, and improvements.

Scaling a Valkey Cluster to 1 Billion Request per Second

How Valkey 8.1 Handles 50 Million Sorted Set Inserts

Introducing Hash Field Expirations

Numbered Databases in Valkey 9.0

Valkey: An Investment in Open Source

New Valkey Features & Use Cases</h2>

Batch Memory Prefetching</h4>
New support for batch memory prefetching helps reduce cache misses by anticipating access patterns and loading data proactively. The result is smoother performance and more consistent behavior under heavy or sequential access workloads.</p>

What you'll see at Keyspace 2025

Valkey

Announcing Spring Data Valkey: A New Season for High-Performance Spring Applications

Value-add for Real-World Use Cases</h2>
Let's explore how the new capabilities introduced through Valkey GLIDE integration with Spring Data Valkey solve real-world problems in real-world use cases.</p>

What Valkey's new primitives tell us about the tools we need

Beyond Vectors: Introducing Full-Text Search and Aggregations to Valkey

Introducing Valkey Admin: Visual Cluster Management for Valkey

Operational Lessons from Large-Scale Valkey Deployments

Don’t Miss What’s Next in Valkey — Subscribe Now

Valkey 2025 Year-End Review: Reflecting on Progress and Looking Ahead

Valkey Helm: The new way to deploy Valkey on Kubernetes

How to Properly Secure Your Valkey Deployment

Resharding, Reimagined: Introducing Atomic Slot Migration

Valkey

Announcing Spring Data Valkey: A New Season for High-Performance Spring Applications

Value-add for Real-World Use Cases</h2> Let's explore how the new capabilities introduced through Valkey GLIDE integration with Spring Data Valkey solve real-world problems in real-world use cases.</p>

What Valkey's new primitives tell us about the tools we need

What's coming in 9.1 makes this more urgent</h2> Two PRs currently targeting 9.1 illustrate where Valkey is headed.</p>

Beyond Vectors: Introducing Full-Text Search and Aggregations to Valkey

Introducing Valkey Admin: Visual Cluster Management for Valkey

Operational Lessons from Large-Scale Valkey Deployments

Don’t Miss What’s Next in Valkey — Subscribe Now

Valkey 2025 Year-End Review: Reflecting on Progress and Looking Ahead

2025 Key Milestones</h2> This past year brought meaningful progress across the Valkey ecosystem. Some highlights:</p>

2026 Valkey Community Goals</h2> Looking ahead into 2026, the Valkey Project and its community will focus on:</p>

Thank You</h2> Thank you to everyone who contributed code, tested releases, filed issues, shared feedback, or adopted Valkey in production. Your continued involvement strengthens and advances the Valkey project.</p>

Get Involved</h2> To learn more and explore ways to get involved in the community:</p>

Valkey Helm: The new way to deploy Valkey on Kubernetes

Step 2: Deploy a new Valkey server</h3> Choose an instance name for the new deployment. It must be different from the current instance to avoid overwriting resources.</p>

Step 3: Enable replication</h3> Replicate data from current instance and ensure it is replicating:</p>

Step 4: Enter maintenance window</h3> Pause all clients connecting to the Valkey server deployed using Bitnami's chart. The failover process will pause client writes, ensure changes are replicated, and promote the new instance to primary:</p>

How to Properly Secure Your Valkey Deployment

Your Security Checklist</h2> Here's a good starting point:</p>

Resharding, Reimagined: Introducing Atomic Slot Migration

Legacy vs. Atomic: Head-to-Head Results</h2> Head-to-head experiments show the improvement provided by atomic slot migration.</p>

Valkey 9.0: innovation, features, and improvements.

Scaling a Valkey Cluster to 1 Billion Request per Second

How Valkey 8.1 Handles 50 Million Sorted Set Inserts

Why These Gains Matter</h2> For Raider.IO and similar workloads, these improvements translate directly into better economics and user experience:</p>

Introducing Hash Field Expirations

2. Radix Tree-based index</h4> Using a radix tree to hold field names plus expirations provides sorted access for free.</p> </p>

Numbered Databases in Valkey 9.0

Valkey: An Investment in Open Source

New Valkey Features & Use Cases</h2>

Batch Memory Prefetching</h4> New support for batch memory prefetching helps reduce cache misses by anticipating access patterns and loading data proactively. The result is smoother performance and more consistent behavior under heavy or sequential access workloads.</p>

What you'll see at Keyspace 2025

Value-add for Real-World Use Cases</h2>
Let's explore how the new capabilities introduced through Valkey GLIDE integration with Spring Data Valkey solve real-world problems in real-world use cases.</p>

Thank You</h2>
Thank you to everyone who contributed code, tested releases, filed issues, shared feedback, or adopted Valkey in production. Your continued involvement strengthens and advances the Valkey project.</p>

Legacy vs. Atomic: Head-to-Head Results</h2>
Head-to-head experiments show the improvement provided by atomic slot migration.</p>

Batch Memory Prefetching</h4>
New support for batch memory prefetching helps reduce cache misses by anticipating access patterns and loading data proactively. The result is smoother performance and more consistent behavior under heavy or sequential access workloads.</p>