Design Twitter (X) — posting, following, and timeline

Question

Accepted Answer

Twitter (X) is one of the most cited system design examples because it concentrates several of the hardest distributed systems problems — real-time fan-out at celebrity scale , timeline assembly with sub-second latency , and write-heavy social graph traversal — into a single product. The system handles billions of timeline reads per day while ingesting thousands of tweets per second, with a 99:1 read-to-write ratio that shapes every architectural decision. Functional and Non-Functional Requirements Functional requirements: post a tweet (up to 280 characters plus optional media), follow/unfollow users, read the home timeline (posts from followed users, reverse chronological or ranked), read a user's own timeline, retweet and reply, like, search by keyword or hashtag, and trending topics. Non-functional requirements: home timeline reads under 200ms at p99, eventual consistency between "tweet posted" and "visible to followers" (seconds, not milliseconds), 99.99% availability, and graceful handling of accounts with 100M+ followers without causing system-wide write storms. Tweet Write Path A tweet is stored in a sharded NoSQL store (Twitter originally used Manhattan, a proprietary Cassandra-like system) partitioned by tweet ID. Tweet IDs are Snowflake IDs : 64-bit integers composed of a millisecond timestamp, a datacenter ID, a machine ID, and a per-machine sequence number. Snowflake IDs are globally unique without central coordination, sortable by creation time (enabling time-range queries without a secondary index), and support roughly 4,096 tweets per millisecond per machine. Once persisted, the tweet service publishes a tweet-created event to Kafka , triggering the fan-out pipeline. The Home Timeline: Hybrid Fan-Out This is the core architectural problem. Two naive approaches fail at scale: Fan-out on write (pure push) — when Alice tweets, a worker looks up all of Alice's followers and inserts the tweet ID into each follower's home timeline cache. Reads are a single Redis lookup. For a user with 100M followers, one tweet triggers 100M write operations — unacceptable write amplification that can take minutes to complete while reads are already incoming. Fan-out on read (pure pull) — at read time, fetch the recent tweets of everyone the user follows and merge them. No fan-out work at write time. For a user following 2,000 accounts, assembling the feed requires 2,000 lookups plus a merge sort — at hundreds of thousands of concurrent requests, this does not scale. Hybrid model (Twitter's actual approach) — for users with fewer than ~500K followers, fan-out on write: push the tweet ID into each follower's home timeline, stored as a Redis sorted set keyed by user_id with score = tweet timestamp. For celebrity accounts exceeding that threshold, do not pre-fan-out — instead, at read time the timeline service fetches the precomputed feed from Redis and merges in a small live pull of recent tweets from the celebrities the viewing user follows. Inactive users (not logged in for 30+ days) are also excluded from fan-out to save write work. Timeline Read Path The home timeline service reads the top N tweet IDs from Redis ( O(log n) sorted-set range query), executes a parallel batch lookup to the tweet store to hydrate the IDs into full tweet objects, optionally re-ranks the results using a relevance model (engagement prediction, relationship strength, content type), and merges in any celebrity tweets fetched live. The full round trip — Redis read, parallel tweet hydration, ranking, serialization — completes in under 100ms for the vast majority of requests. The tweet store itself keeps a cache of recent and hot tweets in front of the persistent store, ensuring that the hydration step is mostly served from memory. Key trade-off — precomputed timelines vs live assembly: Precomputed Redis timelines give sub-10ms read latency and handle Twitter's 99:1 read-to-write ratio elegantly, but at the cost of storage (keeping 500–800 tweet IDs per active user in RAM) and write latency for normal fan-out. Live assembly is flexible and always fresh, but collapses under read load. The hybrid model exploits the observation that celebrities produce very few tweets relative to their follower count, making a live pull cheap; it is the write fan-out that is catastrophically expensive at celebrity scale, not the read pull. Supporting Systems Social graph — follow relationships are stored in a sharded MySQL adjacency list or a graph-native store like FlockDB, sharded by user_id . Fan-out workers query this to get the follower list for each new tweet. Search — tweets are indexed in a real-time inverted index (Twitter built Earlybird, built on Lucene) within seconds of posting. The search service queries the inverted index and returns results ordered by recency and relevance. Trending topics — a streaming pipeline ( Kafka → Flink or Heron) counts hashtag occurrences in sliding time windows (e.g., 5-minute and 1-hour windows) and ranks by velocity