AI can help you get a backend feature working quickly. Then traffic picks up, a downstream call takes longer than expected, and requests start piling up.
I ask the same questions when I review my own backend code as I do when I review code AI wrote.
1. Are we opening a new connection for every request?
Opening a connection can mean looking up a hostname, completing a TCP handshake, negotiating TLS, and authenticating. Reuse HTTP clients and pool database connections to spread that cost across requests. Set keep-alive and connection lifetimes, and decide how long a request can wait for a connection. When a request is slow, measure that wait separately from the time the query took.
Before increasing the pool size, count how many connections all your service instances can open. If the database is already saturated, a bigger pool lets more queries pile up there.
2. How much does it cost to wait on I/O?
If your Java code mostly waits on I/O, virtual threads let you keep writing blocking code while making those waits cheaper. Async APIs can also handle many outstanding operations without tying up one platform thread for each.
Either approach can help you get through more I/O-bound work. CPU-bound work still needs the same compute time. Set limits on how much work can run at once so the extra concurrency stays within what your connection pool, memory, and downstream services can handle.
3. How fresh does cached data need to be?
A cache can have a high hit rate and still return data that’s too old to use. Decide how much staleness readers can tolerate, then use that to decide what to cache, how long to keep it, and when to invalidate it.
Limit the cache size to protect memory. When a popular entry expires, have concurrent requests share a single reload if you can. Plan for cache misses and outages too. If every request goes straight to the database during an outage, the extra traffic can overload it.
4. How much data do we actually need to move?
If the application needs a total, let PostgreSQL filter the rows and calculate it. The application then has less data to receive, deserialize, and hold in memory. Fetch only the columns you use, and look for places where one set-based update can save several round trips. Ordinary SQL often does the job. Consider a database function or stored procedure if it simplifies the operation or cuts enough overhead.
The database is shared, so moving expensive general-purpose computation into it can leave less capacity for other services. Compare query duration, lock time, and database CPU alongside the amount of data transferred.
5. Where are workers getting in each other’s way?
Adding workers may do little for throughput if they all wait for the same lock or compete to update the same counter or database row. Split work into independent tasks where you can, give each partition a clear owner, and use immutable inputs. If operations for a given key have to run in order, let the worker handle them one at a time while work on other keys continues in parallel.
Keep critical sections short. Move network calls outside the lock if the operation is still safe without it. Partitioning can leave one worker busy with a hot key while others sit idle, so check how evenly the work is spread as well as how much time workers spend waiting on locks.
6. What does the database do with this query?
A single slow query can account for most of a request’s latency. Look at the query plan before adding an index or buying more hardware. In PostgreSQL, EXPLAIN (ANALYZE, BUFFERS) reports actual row counts, timing, and buffer activity. Look for estimates that are way off, unnecessary sorts, and steps that process more rows than they need. When you look at the whole request, check for repeated queries too.
ANALYZE executes the statement. Use a safe test environment or run it under carefully controlled conditions, especially for writes.
A sequential scan can be the cheapest plan for a small table or a query that reads most of it. Since indexes take up space and add work to writes, judge a new index against the workload as a whole.
7. How will we know if the change helped?
Start with realistic data sizes and the mix of requests your service gets. Count successful operations per second alongside p50, p95, and p99 latency and the error rate. Higher throughput only helps if latency and errors stay within what the service can tolerate. Track CPU, memory, allocation rate, and time spent waiting on dependencies so you can explain what changed.
Let the JVM warm up, run the test until the results settle, and check that the load generator can keep up. Change one thing at a time when practical and rerun under comparable conditions. A test with a tiny dataset and a warm cache can make a change look better than it will in production. Try bursts, cold starts, and slow dependencies too.
8. What happens when requests come in faster than we can finish them?
A growing queue makes requests wait longer and can eventually eat up all your memory. Set a maximum queue size and limit how many calls can hit an expensive dependency at once. In RabbitMQ, use prefetch to limit messages in flight. If the producer can slow down, apply backpressure. Otherwise, decide when to reject or defer work.
Tight limits can leave capacity idle. Loose ones can hide overload until the queue has grown and requests have spent too much of their time waiting. Watch queue times, completion rates, and which resources fill up as you tune the limits to meet your latency budget.
9. How big should a batch be, and how long should it wait?
Bulk reads and batched writes can turn one network call per record into one call for the batch. Set a limit in items or bytes. If you’re collecting items as they arrive, send the batch when the oldest item reaches its wait limit, even if there’s room for more. Measure throughput and end-to-end latency, including the time each item waits in the batch.
Larger batches can use more memory, hold locks longer, and leave more work to recover after a failure. Decide what to retry if only part of a batch succeeds. A batch size that works for a background job may make an interactive request too slow.
10. How long should we keep working on a request?
A downstream call can hold a connection, use memory, and occupy a request slot after the caller has stopped waiting. Set connection and request timeouts and pass the caller’s deadline downstream. Each call and retry has to fit within the time left. If the caller gives up, cancel the underlying operation where supported so it can release its resources.
Retry transient failures only when a retry makes sense. Limit attempts and use backoff with jitter. The original operation may have succeeded even if you timed out waiting for it, so use idempotency or deduplication to make retries safe.
An overly short timeout can turn a slow success into a reported failure and trigger extra attempts. If several layers each retry independently, they can multiply traffic to the dependency that’s already struggling.
Where to start
Pick a request your service handles often and follow it under realistic load. Suppose it spends most of its time waiting for a connection. Change the pool size and rerun the same test. Check whether requests finish sooner, whether errors go up, and what happens to the database. Your service might look faster because the queue has moved downstream.
If AI suggests a fix, ask it what you should expect to see in the next test. That gives you something concrete to check before you ship it.
Working through a similar challenge?
Tell me what you’re up against. I can help.