How much scale your product needs

Most web products need less scale than their team expects. What they do need is measurement and a database in good shape. Caching or a queue goes where the measurements point, and more servers or a different platform come after that, if they are still needed.

An application that slows down as users grow usually has a few expensive spots: a query without an index, an endpoint that calls the database a hundred times, or work done inside the request that could happen later. You can fix those without adding a single server.

Measure first

If you don't measure, you don't know what is slow. You are guessing, and a wrong guess about scaling turns into expensive work.

Make sure you can see response times per endpoint. Look at p95 and p99, because the average hides exactly the slow requests your users notice. Put the error rate and the time per database query next to them. An APM tool or tracing in your own stack is enough for this.

Then write down what fast enough means. For example: 95 percent of API requests within 300 milliseconds, at the traffic you expect a year from now. With a target like that you can tell whether you have a problem, and when you are done.

Before a big campaign or launch, run a load test against an environment that resembles production. You will see what breaks first, and usually it is one component.

The database is usually the bottleneck

When a web application is slow, the cause is usually the database itself or the way the code talks to it. Typical causes are missing indexes, N+1 queries from the ORM, queries that load a whole table to filter ten rows in code, and too many open connections.

In Postgres, start with pg_stat_statements. The extension tracks how often each query runs and how much time it takes in total. You enable it through shared_preload_libraries (this needs a restart) followed by CREATE EXTENSION. Sort by total time and you have your work list. For the worst queries, EXPLAIN ANALYZE shows what the planner is doing.

My advice is to fix the five queries with the highest total time before you change anything else. That often does more than an extra server would.

After that you can give the database more resources. A larger database server is usually cheaper than the engineering time for a complicated workaround, and a read replica helps when you read far more than you write. Why Postgres covers most products is the subject of Postgres as the default database.

Cache what is expensive and rarely changes

Caching comes after the database. A cache hides a slow query, but the query stays slow. Every cache also brings a new class of bug: stale data.

Start at the edge. Cache public pages and static files with HTTP headers and a CDN, which takes very little code. Then add an application cache, Redis for example, for results that are expensive to compute and the same for many users.

For each cache, agree on how long a value stays valid and what invalidates it. Put the hit rate on your dashboard. A cache with a low hit rate uses memory and makes debugging harder without making anything faster.

Move slow work out of the request

Anything the user does not need to see right away can go to a queue: sending email, generating PDFs, processing images, webhooks to third parties, imports. The request puts a job on the queue and responds immediately, and a worker does the rest.

That has two advantages. The queue absorbs peaks, and the workers process them at their own pace. When an external service is briefly down, the worker also retries later instead of your user getting an error.

To start, you usually don't need a separate message broker. A table in Postgres works as a queue, using SELECT ... FOR UPDATE SKIP LOCKED so multiple workers don't block each other. The Postgres documentation mentions this use case itself. Do make jobs idempotent, send failed jobs somewhere a person will see them, and measure the length of the queue.

Prepare to scale without scaling

A few things are cheap to set up now and expensive to change later. Do them early:

• Keep app servers stateless: sessions in the database or Redis, uploads in object storage, nothing on the local disk.
• Put configuration in environment variables, so a second instance can start without manual steps.
• Write migrations that don't lock tables for long.
• Run background work through the queue, even if there is only one worker today.

With that in place, running two or three instances behind a load balancer is a configuration change on the day you need it.

If you introduce microservices, Kubernetes, sharding or multiple regions before you need them, the system gets harder to debug and operate. For a team of a few developers, the main cost is speed. When that step does make sense is covered in monolith or microservices and do you need Kubernetes.

When you do need to scale

These signals deserve attention. The database runs at high CPU all the time, even after the worst queries are fixed. The queue backlog grows every day. p95 rises with the number of users while the code has not changed. Or a known peak is coming, such as a campaign or a busy season.

Then scale in this order: more app servers, a bigger database, a read replica. Partitioning large tables or splitting one hot component into its own service comes after that. Each time, start with the component the measurements point to.

That is also how we approached Zibber: first review the existing setup and pin down the performance and scalability bottlenecks, then design the architecture.

The order at a glance

1. Write down what fast enough means and what peak you expect.
2. Turn on measurement: response time per endpoint with p95, and pg_stat_statements in Postgres.
3. Fix the five most expensive queries.
4. Move slow work to a queue.
5. Add caching where it helps, with a fixed lifetime and a hit rate you track.
6. Only then add servers, at the point the measurements show.

Sources

• PostgreSQL documentation: pg_stat_statements
• PostgreSQL documentation: SELECT, the locking clause and SKIP LOCKED