How Discord indexes trillions of messages
Search can look like a secondary feature: a person enters a term and gets results. At trillion-message scale, it is a standalone data system with its own queues, indexers, routing, clusters and migration process.
Vicki Niu's Discord Engineering article describes how Discord moved from a design that served billions of messages well to infrastructure that can operate at a much larger scale. The lesson is not simply to use Elasticsearch or Kubernetes. It is to constrain failure domains, preserve messages while downstream systems fail, and make every bulk request reach only the destination it belongs to.
The first design was right for its first problem
In 2017, Discord sharded search data by guild or DM. Keeping a guild's messages together made queries fast and avoided one enormous global index. Messages were lazily indexed because not every person uses search, while workers took chunks from a queue to benefit from Elasticsearch bulk indexing.
That was a sensible early optimisation: fewer clusters, simpler operation and less overhead per indexed message. As scale grew, however, the same choices put too many failure modes on the same path.
Four places where scale broke the model
The Redis queue could drop messages
Redis initially handled the realtime indexing queue. When Elasticsearch lagged, particularly after node failures, the queue grew. Redis CPU could saturate, making the queue itself a failure source that dropped messages waiting to be indexed.
A slow downstream dependency does not stay downstream when the queue cannot safely absorb backlog.
Bulk requests fanned out too widely
Diagram fitted to its frame
Workers took a batch from the queue and sent it to Elasticsearch, but messages in one batch could belong to many indices and nodes. A 50-message batch could fan out to 50 nodes.
If one operation failed, the whole bulk request was treated as failed and every message was re-enqueued, including those unrelated to the failed node. Discord illustrates the effect with a 100-node cluster and evenly distributed batches of 50 messages: a single failed node gives a batch roughly a 40% chance of containing at least one message for that node. One local failure can therefore create a large retry rate.
Very large clusters made coordination expensive
As message volume grew, the direct response was to add nodes and indices. That also widened fan-out. Cluster state grew, master nodes had more coordination work and a failure somewhere in the cluster became more likely.
Discord observed master-node OOM events. The result was not isolated coordination trouble: indexing failed, backlog grew, queries slowed and timeouts followed. Adding capacity is not the same as scaling operability.
Safe upgrades became impractical
The old system was too sensitive to a single unavailable node. Rolling restarts and upgrades became difficult. With clusters above 200 nodes and terabytes of data, draining nodes safely took too long.
The consequence was operational debt: old operating-system and Elasticsearch versions remained in service. During Log4Shell remediation, Discord had to take search fully offline to restart every Elasticsearch node with the required configuration. A system that cannot tolerate maintenance accumulates risk.
Smaller clusters, grouped into cells
Discord deployed Elasticsearch on Kubernetes through the Elastic Kubernetes Operator. The more consequential change was moving from a few very large clusters to many smaller clusters, logically grouped into cells.
A cell is not one Elasticsearch cluster. It is a higher-level grouping of smaller clusters.
Diagram fitted to its frame
This reduces cluster-state overhead and narrows the blast radius of a failure. Discord also added indices so each could remain within the recommendation of roughly 200 million messages and 50 GB, rather than allowing the index for a large guild to grow toward Lucene's document limit.
Within a cluster, roles were separated:
- Master-eligible nodes coordinate the cluster.
- Ingest nodes preprocess and route data; because they hold no data, they can scale for traffic spikes.
- Data nodes devote heap to indexing and query work.
These roles use suitable machine types and node pools. Replicas are spread across zones to tolerate zonal failure. The important idea is not only splitting a cluster, but matching resource types to the work each role must perform.
The queue must outlast downstream trouble
Discord replaced Redis with PubSub for realtime indexing. The essential properties were guaranteed message delivery and tolerance for large backlog. Elasticsearch can slow down or fail; indexing can be delayed; messages must not disappear because the downstream index is unhealthy.
A queue is not merely a buffer for the healthy path. It is an isolation boundary between consumer failures and incoming data.
Bulk only messages with the same destination
This directly addresses bulk fan-out. Discord still uses bulk indexing for performance, but workers no longer put arbitrary queued messages into one request.
A message router reads from PubSub and derives a Destination for every message: the Elasticsearch cluster and index where it belongs.
Diagram fitted to its frame
The router creates a channel and a Tokio task per Destination. That task collects only messages for that destination before it sends a bulk request.
As a result, one bulk operation talks to one Elasticsearch index and the relevant node. If that node fails, impact is limited to messages that really belong there. Messages for other destinations continue instead of all being retried together.
The principle applies beyond search: batching works best when a batch respects placement. Combining items to reduce overhead while ignoring their partition turns a local fault into a batch-wide fault.
Large guilds no longer have to fit in one index
Discord's original shard key was the guild. It made queries convenient, but an exceptionally large guild could drive a Lucene index to its limit of about two billion documents. Once the limit was reached, indexing failed. The short-term response had been to work with the Safety team to identify and delete guilds created only to generate spam messages, but that was not a solution for legitimate growing communities.
A cell architecture introduces an abstraction above the index. Discord can map a large guild across more indices or clusters while allowing the search layer to know where it must query. Separating logical ownership from physical placement gives the system a growth path before a partition hits a hard limit.
Migration is not a single cutover
Moving search from the old infrastructure to cells is a data-correctness problem as well as a deployment project. Historical data needs backfill while realtime indexing continues for new messages.
Diagram fitted to its frame
The team needs validation that data arrives at the correct destination before query traffic moves.
The sequence matters: make the write path reliable, prove that data is present and queryable, then cut over. Reversing it risks serving search from new infrastructure with incomplete history or incomplete realtime data.
What to take to another system
Not every system needs Elasticsearch cells or PubSub. The decisions behind Discord's architecture provide a useful checklist when a pipeline outgrows its initial design:
- Define the failure domain of a batch. Does one item failure retry every item?
- Make queues preserve data and tolerate backlog when consumers slow down.
- Size clusters for operational limits, not just aggregate CPU or storage.
- Separate node roles when coordination, ingest and data have different resource needs.
- Put the partition key into routing and batching, not only into the data model.
- Treat migration as backfill, realtime writes, validation and then cutover.
- Design for upgrades, restarts and node failures from the beginning. They are part of capacity.
Conclusion
When data is small, a simpler architecture is usually the right architecture. At trillions of messages, the question is no longer whether a cluster is large enough. It is how far one fault can spread, whether backlog can lose data, whether batches combine unrelated work, and whether the system can be upgraded without taking service down.
Discord addressed those questions with smaller clusters, a cell layer, a durable queue and destination-aware bulk routing. It is not a fixed recipe. It is a way to scale by limiting the spread of failure while keeping the data path correct even when part of the system is unhealthy.
Sources and further reading
- Vicki Niu, How Discord Indexes Trillions of Messages (opens in a new tab), Discord Engineering, April 24, 2025.