Splitting a database into smaller pieces (shards) distributed across multiple servers to handle more data and traffic.
Sharding splits a large database into smaller pieces (shards) stored on different servers. Each shard contains a subset of the data.
Think of it like splitting a phone book - A-M on one book, N-Z on another. Easier to search smaller books.
Single Database Limits: One database can only handle so much data and traffic before slowing down or running out of storage.
Horizontal Scaling: Sharding lets you add more servers to handle growth instead of making one server bigger.
Decide how to split data (by user ID, location, date). Store each subset on a different server. Applications know which shard to query.
Example: User IDs 1-1000 on Shard A, 1001-2000 on Shard B, 2001-3000 on Shard C.
Range-Based: Split by ranges (IDs 1-1000, 1001-2000). Simple but can create uneven distribution.
Hash-Based: Hash user ID to determine shard. Even distribution but hard to add shards later.
Geographic: Store user data in region closest to them. Great for global apps.
Directory-Based: Lookup table maps keys to shards. Flexible but adds complexity.
Handle More Data: Each shard holds less data, stays fast.
Handle More Traffic: Distribute queries across multiple databases.
Geographic Performance: Store data near users for lower latency.
Fault Isolation: One shard fails, others keep working.
Complexity: Applications must know which shard to query.
Cross-Shard Queries: Joining data across shards is slow and complicated.
Rebalancing: Adding shards means moving data around.
Transactions: Distributed transactions across shards are complex.
Large Data: Database too big for one server.
High Traffic: Single database cannot handle query load.
Global Users: Need data close to users worldwide.
Not Yet: Most applications never need sharding. Vertical scaling and read replicas solve most problems.
Vertical Scaling: Bigger server with more RAM/CPU (simpler, limited).
Read Replicas: Multiple read-only copies of database (easier than sharding).
Caching: Redis/Memcached reduce database load (simpler first step).
Better Queries: Optimize slow queries before sharding.
Instagram: Shards user data across thousands of databases.
Discord: Shards by server ID - each Discord server on a specific shard.
Uber: Geographic sharding - trip data stored near where trip occurred.
Most companies use managed databases (AWS Aurora, Google Cloud Spanner) that handle sharding automatically. Manual sharding requires careful planning.
Sharding is a last resort for scaling databases. Exhaust simpler options first - caching, read replicas, query optimization, bigger servers.
When you truly need sharding, plan carefully. The complexity is real, but the scalability benefits are worth it for truly massive applications.
