Optimizing web server performance at scale (+ my experience)

As your website is getting more and more visitors, it needs to handle the scale accordingly. If your web infrastructure doesn’t scale well, visitors would experience slowness or the website not being available at all, which can lead them to simply leave the website for a competing one.

There are many techniques used to improve web server performance at scale, such as upgrading the hardware or hosting, replacing server software with more efficient one, or fine-tuning existing one.

This post will show you these web server performance optimization techniques. It is intentionally focused on low-level optimizations (hardware, kernel parameters, web server software), however it also mentions some higher-level application considerations.

But what about “Do things that don’t scale”?

Yes, let’s not overengineer things where a simple way exists for zero-user sites.

Many websites (which are low-traffic) probably do not need these low-level optimizations. Their owners just need to do marketing and find users. They are fine running on low-cost infrastructure and default settings, and fine-tuning this would be just a waste of time.

However, if your website traffic is scaling up, and the infrastructure is really the bottleneck (for example when users are complaining the website is slow), not marketing or lack of traffic, this post is for you.

If you’re a shared hosting provider though, be prepared that one of the websites you host might get higher traffic. Also, in this case, infrastructure is the product (customers expect stable, working infrastructure even under some load), so you can proceed to this post too.

OK, let’s proceed to performance optimizations then.

Measure first

Everything in this post is conditional on knowing what your bottleneck is, and you cannot know that without measuring. This is the step that determines whether the other five sections are worth reading for you.

The test that started this for me is the template. I measured plain HTTP at about 41,000 requests per second and the same server over HTTPS at about 14,000, on a 1 Gb/s link carrying roughly 50 MB/s of its 125 MB/s. Three times the difference, at a third of the wire, told me the cost was per-request TLS CPU rather than bandwidth or queueing. That single measurement is why I knew in advance that backlog sizes, qdiscs, and RPS tuning were not going to help, and I was right.

So before tuning anything, answer three questions: What is the actual traffic (requests per second, response sizes, concurrent connections, cache hit ratio)? What is the latency your users experience (p50, p95, p99, not the average)? And which resource is saturated: CPU, memory bandwidth, disk I/O, network bandwidth, or packet rate?

Build a load test that resembles production before you touch the server. Use wrk for HTTP/1.1, h2load for HTTP/2 and HTTP/3, and k6 or Gatling if your traffic has realistic think times and user journeys. Use your real pages, not a synthetic 200-byte response, and get the numbers from a machine on the same network segment (ideally wired, my WiFi client had ±3–5% run-to-run variance, which was larger than every tuning delta I measured and made single before-and-after comparisons worthless). Run each configuration 2–3 times and treat anything under about 5% as noise.

Then load it until it breaks, because a server that has never been saturated is a server whose limits you are guessing at. The concurrency sweep in my test (5.5k req/s at 20 connections, 13.6k at 100, 15.6k at 200, 17.1k at 400) showed the server still scaling at 400 connections, which told me its limits were CPU and not connection handling. I also learned my own limits there: I could not test anything network-related from a WiFi client whose ceiling was 640 Mb/s.

And measure the right layer. Averages are the least useful statistic available. p50 tells you almost nothing about whether anyone is unhappy; p95 and p99 tell you who is. This came up repeatedly in my own numbers, where the PGO build’s average throughput barely moved from the non-PGO build but p99 dropped from 21 ms to 16 ms. If your monitoring only graphs the average, you are optimizing for a number your users never feel.

One quick check that catches the most common misdiagnosis: if HTTPS throughput is roughly a third of HTTP throughput on the same server, your bottleneck is likely TLS CPU, not your network. If both are near the same wire capacity, you have a bandwidth problem and should look at compression, caching, and the network card.

Hardware (for dedicated servers)

If you use a dedicated server, be sure to choose hardware recent and fast enough to handle the scale.

Upgrade hardware also has some other benefits, such as security (there were significant CPU vulnerabilities) or power efficiency as side effects, but this post focuses on improving performance.

One number is worth keeping in mind for this whole section: requests per second per core. A web server handles each request on a single core, so how much traffic you can serve is roughly the core count multiplied by how fast one core is. This is why the CPU details below matter more than the “cores” number on the product page, and why memory and the network card only start to matter once the CPU is no longer the limit.

CPU

For the CPU, you would want one fast enough to handle the scale, and fast here means fast per core.

Per-core speed comes first, core count comes second. Two cheap 8-core CPUs that boost high usually serve more traffic than one 64-core CPU that runs slow, because requests do not spread themselves evenly over the cores. AMD’s own NGINX tuning guide suggests 8 to 16 vCPU per server instance is the sweet spot, not 64.

Look at all-core speed, not boost speed. The boost number on a product page is what one or two cores get. Load every core and the clock drops, because of power and thermal limits. As an example, the AMD EPYC 9575F reaches 5.0 GHz on a single thread and about 4.9 GHz on an all-core web server and TLS workload, but falls to roughly 4.3 GHz when all 128 threads are busy. The 192-core EPYC 9965 is advertised at 2.25 GHz base and 3.7 GHz boost, and it is actually slower per core than the 128-core 9755. When you compare CPUs, compare all-core web serving numbers.

But even the all-core number assumes the chip can cool itself. A 500 W CPU in a 1U server, in a colocation rack with poor airflow, will thermally throttle long before it reaches its boost clock. Airflow is a hardware decision too, and in practice it is often the difference between sustaining boost and sitting at base frequency.

“vCPU” is not a unit of compute. Single-thread and single-core are different measurements, and vendors mix them up. On current x86 CPUs, one physical core with both of its SMT threads delivers about 30% more than that core alone, but a single thread is only about 0.65 of a core. So a cloud “8 vCPU” instance is not the same amount of compute as another provider’s “8 vCPU”. Read the footnotes of any benchmark you look at.

You still need enough cores, because workers map to cores. Many web servers take advantage of multi-threading, so threads map to CPU cores. This is especially important for CPU-intensive applications, and for cryptography too.

SMT is usually worth it on a busy server, because it gives roughly 30% more per core, but the two threads also share the cache. Measure it, do not assume it.

Mind the topology: keep one instance inside one NUMA group. On AMD EPYC the cores are grouped into chiplets (CCDs), each with its own L3 cache, and on multi-socket systems those groups are spread over sockets. Memory and PCI Express devices are attached per group too, so this whole layout is NUMA (non-uniform memory access by the way, used when there are multiple physical CPUs installed, and also between chiplets on a single CPU). If one web server instance is split across two groups, it pays extra latency on every memory access. AMD measured up to a 20% throughput difference between an NGINX instance sitting neatly inside one group and one split across two sockets. So look at the topology, and try to align your instances to it.

For TLS, the right instructions matter. For TLS and HTTPS to be faster, CPUs with vector SIMD and hardware cryptography offload would help. For x86, it can be AVX instruction sets, AES-NI, SHA extensions for certificate work, and VAES for AES-GCM on newer CPUs. For ARM it’s the ARMv8-A crypto extensions (AES, PMULL, SHA), which are built on NEON. SVE is worth mentioning only because it often is, and it is not what makes TLS fast.

But hardware crypto only helps for the ciphers you actually offer. Since clients decrypt too, offer both AES-GCM (when the client has hardware support) and ChaCha20 (when it does not), because ChaCha20 has no hardware path on most x86 CPUs.

A newer CPU also has cheaper mitigations for old CPU vulnerabilities. Server CPUs ship with the Spectre and Meltdown mitigations already enabled, and on older silicon the kernel has to insert extra instructions around returns and branches. Newer CPUs do most of this in hardware, so a five-year-old CPU pays a real and ongoing tax. This is a nice extra reason to prefer recent hardware.

Buy some headroom. If your server is at 90% CPU during peak traffic, it is also running hot, throttled, and with its caches constantly overflowing. Aim for peak traffic to use roughly half to 60% of the CPU.

Memory

Memory capacity is often the biggest performance lever in this entire post, and it is not the one people expect. A web server does not need to read a file from disk if that file is already in the page cache (the Linux kernel’s cache of recently used files). So how much of your traffic never touches a disk at all is decided by how much RAM you have. If you serve static files, proxy, cache, or run a language runtime, more RAM directly means less I/O wait and lower tail latency.

Fill all the memory channels. Server CPUs have many of them (8 or 12), and a half-populated motherboard gives you roughly half the bandwidth. Memory speed is also decided by the slowest module in the machine, so mixing speeds slows down every module. This is a common and very quiet mistake.

Memory speed matters less than capacity, but bandwidth per core starts to matter later. Once your working set no longer fits in cache, or you are proxying, or a database runs on the same box, the memory channels and their speed become the limit. Current server parts give you DDR5-6000 or better across 8 to 12 channels.

Use ECC RAM. On a server with a lot of memory, the odds of a bit flip are not tiny, and without ECC you get silently wrong answers instead of a crash.

Huge pages (THP) make the kernel use 2 MB pages instead of 4 KB ones, which reduces TLB misses. It is a small win worth measuring, but it can also make memory harder to reuse, so test it rather than assuming.

Network card (NIC)

Start with a fast enough link, then look past it. Using a high-bandwidth network card (10/25Gb) and network connectivity (many data centers use fiber optics) would help, especially for large data transfers. This is because when there are more clients requesting the same resource, data transfer gets larger accordingly.

But bandwidth is rarely the number that limits you, packet rate is. A full 10GbE link carrying minimum-size packets is about 6.6 million packets per second in each direction, and every packet costs CPU work in the network stack. One server core pushes a few million packets per second through the kernel stack, and much less once TLS is added. So there are two very different workloads here: many small responses (limited by packet rate) and few large files (limited by bandwidth). Work out which one you have, and size the card for the worse of the two.

Check the queue count, and do not ask for more than you need. The NIC splits incoming traffic into several hardware queues, each with its own interrupt. AMD recommends one queue per core. More queues than cores makes the interrupt handler keep switching between them, which makes things slower rather than faster.

Check how many PCIe lanes the card really gets, because it is easy to get this wrong. 25GbE needs at least PCIe Gen3 x8 to have enough bandwidth, and 100GbE needs Gen4 x16 or Gen5 x8. A 25G card in a slot that only negotiates Gen3 x4 shares its bus with whatever else sits on that bus and becomes bus-bound. The label on the motherboard slot is not always the real negotiated speed, so verify it with lspci -vv. On large EPYC CPUs it also matters which PCIe root complex the card hangs off, since a card on the far side of the chip reaches memory by a longer path.

Hardware offload can cut CPU a lot, and it is not cheap. Offload features like TSO, checksum offload, and GRO make the card do work in hardware instead of the CPU. Chelsio measured NGINX with 10,000 connections using 64% of the CPU on a plain NIC, 28% with TCP offload, and 7% with both TCP and TLS offload. That is a real saving, but the cards cost thousands of dollars and only start to make sense at 100GbE. Be careful with LRO (large receive offload) specifically, because it merges packets in a way that breaks RTT measurement and can cause latency spikes. And remember that with offloads enabled, tcpdump no longer shows you what actually went out on the wire, so you cannot benchmark properly with them on.

Jumbo frames (MTU 9000) are mostly not for you. The measured benefits are modest: a few percent more throughput for file transfers, and “virtually no benefit” for a web service benchmark. On top of that, a larger frame has a higher serialization delay, which can make small responses slower. For internet-facing traffic it is not available at all, because you cannot get MTU 9000 all the way to your clients. The worst part is what happens when it half works: if your first hop is 9000 and the real path is 1500, the TCP handshake completes and small responses work fine, and then large responses hang silently while the sender tries to discover the path MTU (PMTUD) and the ICMP messages never arrive. Keep 1500 on the public interface, and consider jumbo frames only on an internal or storage-facing interface where you control every single hop.

A faster link can make latency worse. A 10GbE card on a 1G uplink gains you nothing, and a saturated 10GbE link queues packets, which adds delay far larger than anything you gain from a faster CPU.

Get a second network interface for management, on its own port. Then a data interface that is saturated, misconfigured, or hung cannot stop you from getting into the machine.

And consider renting the hardware you do not buy: distance. For visitors who are far away, CDN and edge capacity usually changes your numbers more than any specification on this page. It is a hardware decision, just not one you own.

Storage

The goal is to keep storage out of the request path. If files are in the page cache and you serve them with sendfile (or X-Sendfile), the web server never waits for a disk. After that, the disk only needs to absorb writes and log writes.

If you do have to read from disk, the difference is huge. NVMe, SATA SSDs, and spinning hard disks differ by one to two orders of magnitude in IOPS, and you notice it in the 99th percentile latency rather than in the average. This is the one place where storage quality is directly visible to your users.

Use a RAID/HBA controller with a battery-backed (or supercapacitor-backed) write-back cache. It also protects you from data loss during a power cut with the write cache enabled. Supercapacitors also last way longer than batteries, and modern RAID controllers prefer supercapacitor-backed write-back caches. I remember replacing a battery-backed write-back cache (with dead battery) with a supercapacitor-backed on an old rack-mount server.

Put logs on their own disks or their own controller. A burst of log writes stalling the reads that serve requests is a very common self-inflicted outage.

Motherboard and platform

The motherboard and case limit everything above. Check them before falling in love with a CPU.

Server-grade is not the same as a desktop board with a server CPU in it. You want ECC support, a decent number of memory slots (8 or 12 per socket, not 2 or 4), dual power supplies, real thermal sensors, and a BMC/IPMI. The BMC is a small separate management processor on the board, and it is the feature people miss and the one that matters most: it lets you see the screen, read the sensors, and reboot a machine whose operating system is completely hung. “The server stopped responding” is exactly the failure you cannot fix remotely without it. HPE has iLO (Integrated Lights-Out), Dell has iDRAC (Integrated Dell Remote Access Controller), and so on.

Count PCIe lanes. Current server CPUs have between 88 and 128 of them, and the onboard NIC, the M.2 slots, the BMC, and any add-in cards all consume them.

Check the power and cooling budget. Performance per watt decides your total cost of ownership, and it also decides how many servers fit in the space and on the circuit you have. It is a constraint, not a preference.

Plan for the refresh. With a dedicated server you own the lifecycle: the warranty, the spare parts, and the replacement in three or four years. A slightly slower machine with a proper warranty is often the better purchase.

ARM or x86?

ARM is often the better deal per request. Current ARM server CPUs (AWS Graviton, Google Axion, Azure Cobalt) are often better on price per request and performance per watt, and they provide one physical core per vCPU, which scales more predictably than x86 with SMT.

The deciding question is your software, not the instruction set. If you are serving static files, terminating TLS, and proxying, ARM is likely a straightforward win. If you depend on a long tail of third-party libraries, extensions, or proprietary tooling, x86 is the safer bet. Benchmark your actual stack before deciding.

Below is a video showing the performance benchmarks between x86 (Intel and AMD) and ARM (AWS Graviton). From this video, you can see that Graviton has similar performance to AMD’s x86 CPUs, at lower cost.

How much do you actually need?

Benchmark before you buy, and benchmark the workload you really have: your real pages, TLS enabled, a realistic keepalive ratio, a realistic cache hit rate, using tools like wrk, h2load, or k6. Vendor web server numbers are almost always warm-cache static file serving, which is the easy case.

Look for the knee in the scaling curve. Throughput grows nearly linearly until cores start sharing one L3 cache, then memory bandwidth becomes the limit, then the NUMA boundary, and finally the network card’s packet rate. Where your own workload bends is where you should buy.

Price per request, not per server. Divide the monthly cost by the requests per second you measured. This is the number that lets you compare a two-socket server, a bigger single-socket one, and a CDN in front of both.

Useful commands when checking a machine: lscpu (topology and NUMA), dmidecode -t memory (which memory channels are populated), lspci -vv (real negotiated PCIe speed), cpupower frequency-info (actual clocks under load), ethtool -l and ethtool -i (NIC queues and driver), numactl --hardware (NUMA layout), and iperf3 -P 4 (real throughput with parallel streams).

Linux + hardware settings

This is where most “web server tuning” advice lives, and most of it does nothing unless you already know what your bottleneck is. I ran a set of these on a real machine and kept track of what actually changed the numbers, so this section reports the settings that helped, the ones that did not, and the ones that made things worse.

The test setup, for context: an AMD Ryzen 7 8700G (8 cores, 16 threads, 5.27 GHz max) on Ubuntu with a Realtek RTL8125 NIC at 1 Gb/s, serving a 3.4 KB file over HTTPS with Ferron, measured from a second machine with wrk and h2load. Everything below is stock Ubuntu defaults unless noted.

Know your bottleneck first

The single most useful thing I did was not tuning anything at all. Plain HTTP served about 41,000 requests per second, while the same server over HTTPS served about 14,000. Three times the difference, on a 1 Gb/s link that was carrying roughly 50 MB/s out of the 125 MB/s available.

That measurement told me where to look. Roughly a third of the wire, and a 3x gap between encrypted and unencrypted, means the cost is per-request CPU work (mostly TLS), not bandwidth and not queueing. So it is worth saying plainly: backlog sizes, qdiscs, RPS, and IRQ tuning cannot fix a per-request CPU bottleneck. I applied all of them, and they made almost no difference, because they were never the problem.

This also connects back to the hardware section. If your bottleneck is TLS, then the things that matter are per-core speed, AES-NI, and how many cores you have, not the network card.

CPU frequency governor

Set it to performance on a dedicated server, but expect a small win, not a 2x one. The common belief is that powersave slows down a server. In my test it did not: with the AMD amd-pstate-epp governor in powersave, the CPU still boosted to 2.4–4.3 GHz under load, because the driver raises the clock on demand regardless of the governor. Forcing performance and setting energy_performance_preference to performance removed one more source of throttling, and I kept it, but it moved throughput by less than the measurement noise.

It is still worth doing, because on a busy server it costs nothing and removes a variable you do not control. Just do not expect it to be your breakthrough.

echo performance | sudo tee /sys/devices/system/cpu/cpufreq/policy*/scaling_governor
echo performance | sudo tee /sys/devices/system/cpu/cpufreq/policy*/energy_performance_preference

Backlog and connection queue

Raising the backlog sizes is harmless and occasionally necessary, but it is not a performance knob. I raised net.core.netdev_max_backlog from 1000 to 5000 and net.ipv4.tcp_max_syn_backlog from 2048 to 4096, and throughput did not measurably change. The reason is that these only matter when a burst of packets or connections arrives faster than the server drains them, and my server was not in that state.

Where these settings do bite is under a connection flood, for example a bot hit or a traffic spike, where the default of 1000 queued packets fills quickly and the kernel starts dropping. Raising them costs nothing and gives you headroom for exactly that case. This is the “buy headroom” argument from the hardware section, applied to queues instead of cores.

net.core.somaxconn caps the listen() backlog, and it is the setting to check if you see connections being refused or reset at high concurrency. I did not push past 400 concurrent connections in this test, and the server was still scaling at that point, so I could not measure it. If you serve thousands of keepalive connections, verify it is not below your web server’s own listen() backlog, or the application setting is silently clamped.

TIME_WAIT is about connection churn, not concurrency. This is the one people most often get backwards. TIME_WAIT sockets are not a problem as long as the number of simultaneous connections is within your limits; they only accumulate when you open and close connections at high rate, because each closed connection sits in TIME_WAIT for a timeout period (60 seconds by default). A few thousand is normal and harmless. Tens of thousands means you are creating connections faster than they drain, and the fix is connection reuse (keepalive, HTTP/2 multiplexing, or session resumption), not a shorter timeout. Shortening net.ipv4.tcp_fin_timeout does not touch TIME_WAIT at all, which is another common misconception. One time when I was benchmarking a web server under high concurrency, I found connection establishment issues were happening until I set net.ipv4.tcp_tw_reuse to 1 (reuse all TIME_WAIT connections).

Network card settings

Offloads are usually already on, and the ring buffer is usually already at maximum, so there is often nothing to do. My NIC had TSO, GSO, and GRO enabled and its RX/TX rings at 256/256, which is the maximum the driver allows, and interrupt coalescing was not supported at all. I enabled nothing here.

RPS (Receive Packet Steering) is worth understanding precisely, because it is often applied by mistake. RPS exists to spread packet processing across cores when the NIC has fewer queues than you have cores. On my machine the NIC had a single receive queue, and enabling RPS to spread across all 16 CPUs produced no gain and added cross-CPU cache traffic, so I turned it back off. The rule I would use: if your NIC has one queue per core already, leave RPS off, because locality beats distribution. If it has one queue total and you are saturating it, RPS can help.

This is the same point as the queue count advice in the hardware section, seen from the software side: one interrupt queue per core, and no more. If your NIC has two queues and you have sixteen cores, the software cannot fix the hardware mismatch as well as a NIC with the right number of queues could.

Pinning IRQs to a specific core did nothing repeatable. I pinned the NIC’s single IRQ to one core instead of letting it spread across all of them, and throughput was unchanged. This is a technique from a decade ago that mattered when NICs had one or two queues and the kernel had no better option.

Congestion control and queueing

Do not cargo-cult BBR. I switched net.ipv4.tcp_congestion_control from cubic to bbr and it was slightly worse, because my path was a 2–3 ms LAN. BBR is designed for paths with meaningful loss and varying bandwidth, which is the opposite of a LAN. On a low-RTT network cubic was as good or better, so I reverted it. If you are serving real internet traffic over a lossy path, BBR is worth testing, but test it on your path rather than assuming it wins.

The same goes for fq versus fq_codel. I tried fq and saw no gain over the default fq_codel. fq_codel is the better default because it actively controls queueing delay, and for a web server where small responses and tail latency are what you care about, controlling the queue is the point.

What I did keep from this area was net.ipv4.tcp_fastopen set to 3 (allow both client and server Fast Open) and net.ipv4.tcp_slow_start_after_idle set to 0. The second one is the interesting one: by default, a connection that has been idle for a moment drops out of congestion-controlled fast start and has to ramp its send window back up from scratch. With keepalive connections that are idle between requests, that ramp is pure waste, so disabling it is correct. Both were small wins, as expected.

sudo sysctl -w net.ipv4.tcp_fastopen=3
sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0

Measurement was harder than the tuning

The measuring machine was part of the system, and it was the least reliable part. My client was on WiFi, where run-to-run variation was ±3–5%. That is larger than every tuning delta I measured, which means a single before-and-after pair proves nothing here.

So the methodology matters more than the settings: run each configuration 2–3 times, treat anything under about 5% as noise, and for network-path work, measure from a wired host. My 10 MB bulk transfer test hit 80 MB/s, which looked like a server limit until I realized it was the client’s WiFi ceiling, and the test was measuring the client.

And A/B test instead of trusting a suspicious metric. mpstat showed 16% average I/O wait, spiking to 65% on some cores, which made access logging an obvious suspect. Disabling it changed nothing (13.9k vs 13.8k req/s), because the writes were buffered and the real cause was the live-USB filesystem. A scary-looking number in a monitoring tool is a hypothesis, not a diagnosis.

The bigger levers I did not pull

If TLS is your bottleneck, the fix is to do less TLS, not to tune queues. The single largest untested lever in my setup was TLS session resumption, which was reported as Resumption: no in the h2load output. Enabling session resumption and OCSP stapling removes most of the expensive asymmetric cryptography from repeat visits, which is a far bigger win than anything in the lists above. This is a web server configuration change, not a kernel one, and I would test it before touching any sysctl at all.

The other things I deliberately left alone: somaxconn and file descriptor limits under high connection counts, HTTP/2 multiplexing behavior, and larger assets to properly separate CPU cost from network cost.

A note on persistence

Everything above was applied at runtime with sysctl -w and sysfs writes, so it resets on reboot. For a real server, put the sysctl values in /etc/sysctl.d/ and the governor setting in a systemd unit or rc.local, so they survive a restart. A tuning that disappears on reboot is not a tuning, it is an outage waiting for the wrong moment.

Web server software

Once the hardware and the kernel are not the bottleneck, the web server itself is. This section covers the choices that actually moved numbers for me, and the ones that turned out not to.

Start by picking an event-driven server over a thread-per-connection one. This is the single largest software decision and it is architectural. Apache httpd with mod_php or prefork gives each connection its own process or thread, so every connection costs a context switch and a scheduler decision. NGINX, Ferron, Caddy, and HAProxy are all event-driven: one thread (or one per core) handles tens of thousands of connections without switching context. On the same hardware, that difference is typically an order of magnitude on concurrent workloads, and it is not something you can tune your way into.

NGINX is the most battle-tested option, with a huge ecosystem of modules and a very large body of tuning documentation (including vendor guides from AMD and Ampere, which are good starting points). Ferron 3 is a newer server that is competitive on throughput and notably lighter on memory, and it is the one I have hands-on experience with here. Caddy is a good default when automatic HTTPS and a simple config file matter more than raw numbers. Apache is not a bad server, it is just the wrong shape for a high-concurrency workload, and if you need it, mpm_event plus PHP-FPM gets you most of the way.

The memory allocator matters more than you would think

Replacing the system allocator is one of the cheapest performance wins available, because it is a build-time or environment-time change with no architectural risk. Programs on Linux default to the system allocator (glibc malloc), which is optimized for the general case of many small programs on one machine, not for a server that allocates constantly across many threads.

I benchmarked eight allocators on Ferron 3, serving a 16 KB static file over HTTPS/HTTP/2, and separately as a reverse proxy in front of a backend. The results:

The headline is the reverse proxy case: 103k to 116k requests per second (+13%), and p99 latency cut from 110 ms to 48 ms. That is a much bigger effect than I expected, and the latency improvement matters more than the throughput. For static file serving the differences are small, which makes sense: serving a cached file is mostly sendfile, so the allocator is barely involved.

jemalloc won overall, but notice the table is not a clean sweep. hoard had the best static p99 (27.55 ms) and tcmalloc the second best, while mimalloc was the worst option tested on throughput and its “secure” mode was worse still. So do not assume a famous name wins; measure your workload.

jemalloc’s defaults use more memory than glibc, which is a real downside, and it is tunable. This configuration cut memory usage further without hurting throughput (it’s the default in Ferron 3):

export JEMALLOC_SYS_WITH_MALLOC_CONF='narenas:2,percpu_arena:phycpu,metadata_thp:auto,dirty_decay_ms:100,muzzy_decay_ms:100'

narenas:2 limits the number of independent arenas, which reduces fragmentation on a share-nothing, thread-per-core server. percpu_arena:phycpu pins each arena to a physical CPU rather than a hyperthread, which improves cache locality. metadata_thp:auto lets jemalloc use transparent huge pages for its own metadata, reducing TLB misses. The two decay_ms settings make jemalloc return memory to the OS faster, which is what actually reduced the footprint.

But an allocator cannot fix a leak, and this is worth saying because it cost me real time. While benchmarking, Ferron 3 was using more memory than Caddy and appeared to be leaking. The cause was not the allocator at all: a function wrapped a value in ManuallyDrop to prevent its destructor from running, but the value held heap allocations (a String, an Rc, an Arc). Suppressing Drop on something that owns heap memory means that memory is never freed. No allocator setting fixes this, and switching allocators would have sent me chasing the wrong thing. If memory grows under load, look for a reference cycle or a suppressed destructor before you start tuning arenas. This is worth knowing when it comes to web applications later on.

Profile-guided optimization is the best build-time win

PGO means running the program on a representative workload, recording which code paths are actually hot, and rebuilding with that knowledge. The compiler can then lay out functions, inline more aggressively, and optimize branches for the code you really run, instead of guessing.

I built Ferron 3 with and without PGO and measured with h2load over 10 seconds at 250 connections:

Throughput barely moved, but tail latency improved a lot, and for the static case p99 dropped by a quarter. That is the more valuable result: PGO makes the slow path faster, so the 99th percentile user waits less. Average throughput is the number that shows up in marketing benchmarks; p99 is what your users feel.

This only works if the profile matches your workload, which is the catch. A profile collected while serving static files will optimize for static file serving, and the resulting binary may be worse for your dynamic pages. You need a representative profile, and ideally a few different ones.

BOLT is the cheaper cousin of PGO. Binary Layout Optimization uses the same profiling idea but post-processes an already-compiled binary instead of requiring recompilation, rearranging code and adding function splitting to improve instruction cache and branch prediction behavior. You get much of PGO’s benefit for a fraction of the effort, and it composes with distributions’ prebuilt packages.

Link-time optimization is the other cheap one. LTO merges separately compiled object files so the compiler can see across the whole program. It usually helps a little, and it costs build time. -march=native (or its Rust equivalent) is worth it too, but note that the binary then only runs on CPUs with at least those instructions, which matters if you build on one machine and deploy on another.

Worker count and the io_uring question

Set the worker count to the number of physical cores, and be careful what “core” means. In NGINX, worker_processes auto sets one worker per logical CPU. On a machine with SMT, that means one worker per hyperthread, which is usually right, since the two threads share a core’s execution resources anyway. Ferron uses a “primary runtime” thread per CPU by default, same idea. The important part is that this is a starting point, not an answer: AMD’s NGINX guide found 8–16 vCPU per instance to be the sweet spot rather than the maximum, because a very large worker pool spends more time in the scheduler than in requests.

io_uring is worth testing, but it is workload-specific. epoll (or kqueue on BSD/macOS) is the traditional interface: the kernel tells you which file descriptors are ready, and you read them one at a time. io_uring is a newer Linux interface where you submit a batch of operations to a shared ring buffer and the kernel completes them, so a thread can have many reads and writes in flight without one system call each.

For serving many small responses, epoll is fine and the difference is small. io_uring tends to help with large file serving and high-concurrency reverse proxying, where there are many outstanding operations. Ferron exposes it as a runtime { io_uring } directive and falls back to epoll with a warning if the kernel does not support it, which is a reasonable way to handle the fact that io_uring needs kernel 5.6+ and is still improving. Measure both, because the win is workload-dependent and not guaranteed.

{
    runtime {
        io_uring
    }
}

File descriptor limits cause silent failures

This one bit me, and it is worth checking before anything else. While benchmarking I saw requests failing, and the cause was not the server at all: the process had run out of file descriptors. The soft limit was 1024, the default, and a web server needs one per open connection, plus a few for listeners, logs, and backend connections.

The fix is to raise the soft limit to the hard limit, before the server starts:

ulimit -n $(ulimit -n -H)

Do it in the shell that launches the server, because the limit is inherited by the child process. Setting it in your interactive shell after the server is already running does nothing. For a real deployment, set LimitNOFILE in the systemd unit, or worker_rlimit_nofile in NGINX, which raises the limit for workers without a restart.

It is also worth remembering that the connection count you advertise and the descriptor count you have are different numbers. NGINX’s worker_connections (default 512) counts all connections a worker holds, including connections to proxied backends, not just client connections. So the theoretical maximum is worker_processes × worker_connections, and it cannot exceed the descriptor limit. Both numbers need raising together, and the descriptor limit is the one that fails silently.

Libraries underneath the server

The crypto and compression libraries are where a web server spends its CPU, so replacing them is often the highest-leverage change in this whole section.

For TLS, the options are OpenSSL, BoringSSL (Google), LibreSSL, and AWS-LC (AWS, which is what BoringSSL forked from). AWS-LC is a good default for a modern server: it has hardware-accelerated AES-GCM, ChaCha20, and SHA on both x86 and ARM, it is actively maintained, and it is what most other projects have migrated to. Ferron uses Rustls with AWS-LC.

It is possible to even use kTLS (kernel TLS offload) instead of userspace cryptography to reduce context switches, but this is not as commonly used, and may be unsupported. kTLS can be also used to offload cryptographic operations to a network card.

For compression, the classic zlib is no longer the fastest option. zlib-rs is a Rust rewrite of zlib that is measurably faster while remaining API-compatible, and it is what Ferron 3 uses through the flate2 crate. If your server compresses responses (and it should, for text content), this is often a drop-in win. zlib-ng is another high-performance zlib replacement worth exploring.

And the biggest library-level win is not a library at all: session resumption. As I noted in the previous section, my h2load output reported Resumption: no, meaning every request was paying for a full TLS handshake. Session resumption and OCSP stapling remove most of the expensive asymmetric cryptography for returning clients, and they are configuration changes rather than rebuilds. If your bottleneck is TLS, do this before you touch an allocator.

Reverse proxy upstream keep-alive

Upstream keep-alive connections are a key optimization for reverse proxies, as they avoid the overhead of establishing a new connection to the upstream server for each request.

Ferron 3 has them enabled by default. NGINX can be configured to use them as well:

upstream backend {
    server 127.0.0.1:8080;
    keepalive 64; # <-- this enables keep-alive
}

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

Static file serving optimizations

For static file serving, enabling sendfile can yield higher performance (especially for large files), because it doesn’t rely on userspace buffers and reduces the number of context switches between user space and kernel mode. However, for HTTPS (except kTLS) and HTTP/2, sendfile is often not supported, so it may be unavailable. Still, it can still be useful for serving static files over plain HTTP.

Many web servers (Ferron 3, Caddy) have sendfile enabled by default (if supported), and many others (like NGINX) have it enabled in default packaged configurations.

Turning the knobs in NGINX

For reference, the NGINX directives that matter most for this kind of tuning, all from the core module:

• worker_processes auto and worker_connections: worker count and per-worker connection ceiling. The two together set your maximum concurrency, and both need to fit within your descriptor limit.
• worker_rlimit_nofile: raises the descriptor limit for workers without needing a restart. This is the fix for the problem above.
• use epoll: normally unnecessary, since NGINX picks the most efficient method available, but worth knowing if you are comparing against io_uring-based servers.
• pcre_jit on: compiles regular expressions at configuration parse time instead of interpreting them. If you use regex in location blocks or rewrite rules, this is a real and free win. It is off by default.
• accept_mutex: leave it off on modern kernels. It is only needed on systems without EPOLLEXCLUSIVE or reuseport, and the NGINX docs explicitly say there is no reason to enable it otherwise. Enabling it out of habit adds a lock where none is needed.
• timer_resolution: reduces gettimeofday() calls, which NGINX makes on every kernel event by default. Lowering it to 100ms trades a small amount of timestamp precision for fewer syscalls, which can matter at high request rates.

Web application

The web server can be perfect and your application can still be the slowest thing in the request path. This section is about moving work out of your application and, where you cannot, choosing a language that makes the work cheap.

Move the generic work to the web server

TLS termination, compression, static file serving, and caching have nothing to do with your business logic, so your application should not be doing them. Put them in the web server, where they are implemented once, tuned once, and shared by every route.

The usual shape is: web server or gateway in front, application behind. The web server terminates TLS, serves anything static directly from disk or cache, applies compression, enforces rate limits, and adds headers. Only the requests that genuinely need your code reach the application. The Ferron edge caching and NGINX proxy_pass setups are the same idea at different scales.

This is exactly what API gateways do for microservices, and understanding why is useful even if you have one server. A gateway such as Envoy, Traefik, or Kong exists to centralize TLS termination, routing, rate limiting, and authentication so that individual services do not each implement them. The argument is the same one you are making at the application level: cross-cutting concerns belong in shared infrastructure, and a service that only does its own job is easier to scale and easier to replace.

The benefit is not just CPU, it is memory and code. A TLS stack in your application is a dependency you have to keep patched, and in a language runtime it may be significantly slower than a native one. In my measurements, moving from plain HTTP to HTTPS cut throughput from ~41,000 to ~14,000 requests per second on the same server, and that cost was pure per-request crypto. If your application is not terminating TLS, that cost disappears from it entirely.

Two things to be careful about. First, if you re-encrypt to reach the backend (which is what service meshes do for defence in depth), you have not removed the TLS cost, you have moved it and added a second handshake. That is a deliberate security trade, not a free optimization. Second, do not let the gateway and the application both compress, or the response gets compressed twice for no gain. Pick one.

The compression decision is mostly about static vs dynamic

Compress static assets once at build time, and serve the precompressed file. This is the single cheapest bandwidth win on the web and it costs zero CPU per request. Compress your .html, .css, .js, .json, and .svg files to Brotli level 11 during your build, keep the .br and .gz variants next to the originals, and have the web server serve whichever one the client accepts. The expensive work happens once in CI rather than on every request.

brotli -11 -k style.css
gzip -9 -k style.css

In NGINX, brotli_static on; does this automatically: request for style.css, and if style.css.br exists next to it, the server serves that with Content-Encoding: br and never touches the CPU. Ferron supports this as well.

For dynamic responses, compression level is a real CPU cost, so pick deliberately. Brotli level 11 is roughly 15–20x slower to compress than gzip level 6, which is fine in CI and disastrous on a request path. For responses generated per request, the sensible defaults are Brotli level 4–6 or gzip level 6 for broad compatibility, or Zstd at a low level (1–6) for modern clients, which is the fastest of the three and decompresses roughly 3x faster than Brotli on the client side. That last point matters more than it looks: you compress once on the server but every client decompresses, so decode speed is a user-facing number.

Do not compress tiny responses. Below roughly 150–1,000 bytes, framing overhead can make the “compressed” response larger than the original, and you have spent CPU to find out. Every codec in this discussion has that threshold.

And keep the negotiation honest. You need Vary: Accept-Encoding on compressed responses, or your caches (browser, CDN, reverse proxy) will serve a Brotli body to a client that cannot read it. The classic version of this bug is an origin compressing while the CDN decompresses and recompresses per its own policy, burning CPU on both sides to change nothing. An afternoon of curl with different Accept-Encoding headers against your top URLs finds most of these.

Go and Rust are both good choices, for different reasons

If you are writing a new high-traffic service, Go and Rust are the two defaults, and the honest summary is that they are much closer than benchmark posts suggest. The differences that matter are not “which is faster” but tail latency predictability, memory footprint, and how long it takes your team to ship.

They are both compiled programming languages, unlike let’s say Python or JavaScript (via Node.js), and they often have higher performance characteristics than interpreted languages (no interpreter or JIT overhead).

One comparison I found worth taking seriously built the same API twice, in Go (net/http) and Rust (axum on Tokio), and tested at three load levels rather than one (here’s the source by the way):

At their current traffic the two were indistinguishable, and the honest conclusion was “do not rewrite.” The gap only opened at the load they were about to grow into, and the decisive number was memory, not latency: 290 MB versus 820 MB changed which instance size they could afford. If you take one thing from this, it is that the language choice only becomes obvious once you define which kind of pain you are optimizing against, and that a rewrite justified by performance is usually not justified until you are near the load where it matters.

Rust’s advantage is that there is no garbage collector, so the latency distribution stays flat under load. The cost is that you manage memory explicitly, and Rust does not magically make a service fast if the code clones excessively or holds locks too broadly. The usual pattern is a tighter tail and 2–4x less memory per process, at the price of a real learning curve (fighting the borrow checker…).

Go’s advantage is that it is genuinely easy, and its GC is concurrent rather than stop-the-world, so pauses stay in the sub-millisecond range on normal heaps. The cost is that GC CPU is real: Uber found it was the top CPU consumer for most of their Go services, and tuning memory ratios recovered around 70,000 cores across 30 services. Go is also often better on small-payload workloads and cold starts, which matters for serverless and scale-to-zero.

The Go GC has one main knob, and it is worth knowing. GOGC sets how much the heap may grow before a collection runs, as a percentage of the live heap. The default of 100 means the heap may double. Doubling GOGC roughly halves GC CPU at the cost of doubling memory overhead, which is why Twitch cut CPU by ~30% and p99 API latency by ~45% by simply allocating a large unused “ballast” to make collections less frequent. The mistake to avoid is tuning GOGC without a memory limit, because it is not aware of your container’s ceiling. In a container, set GOMEMLIMIT (Go 1.19+) as a backstop and leave 10–15% headroom for non-heap memory.

If you do use Go, profile before you tune. net/http/pprof and go tool pprof are built in and have minimal overhead, and most Go performance problems turn out to be allocation in the request path rather than the runtime itself. Avoid fmt.Sprintf on hot paths, and use sync.Pool for buffers.

And a caveat on the numbers you will find online. Many published Rust-vs-Go comparisons are hello-world or loop benchmarks, which measure raw CPU throughput and have almost nothing to do with an HTTP service doing database queries and JSON serialization. Expect the gap on a realistic service to be far smaller than the headline, and expect the results to disagree with each other. Measure your own workload; that is the advice of this entire post, and it applies here more than anywhere.

The video below shows the performance benchmarks comparing equivalent web applications written in Rust (axum) and Go (gnet).

The things that help in any language

Cache the response, not the computation. Rendering the same page for 1,000 concurrent requests when you could render it once is the most expensive mistake available, and it is the easiest to fix. Page caching, fragment caching for repeated header/footer/database lookups, and a CDN in front of the whole site all reduce origin work per request.

Do less work per request. This sounds obvious but it is where most application performance actually lives. The usual wins are eliminating an N+1 query pattern (one query for a page instead of 500), avoiding a network call to your own service for something you already have in memory, and not deserializing a large structure to read one field out of it.

Stream instead of buffering, for both responses and large uploads. Building a 100 MB response in memory before sending the first byte is a memory spike and a latency spike; streaming it lets the client start receiving immediately.

And keep the connection between the app and the web server cheap. If the application is on the same machine, a Unix domain socket avoids the TCP stack entirely and is measurably faster than loopback TCP. Ferron, NGINX, and Envoy all support this. It is a small thing that costs nothing.

Monitor the traffic continuously

Measuring once tells you where to start. Monitoring tells you whether you were right, and it is what catches the regression when someone changes a setting six months from now.

The handful of metrics that actually matter

Most dashboards are too broad to be useful, because a dashboard with thirty charts is a dashboard nobody reads. For a web server, these are the numbers I would put on the wall:

• Request rate (requests per second), split by status code. A sudden drop in 2xx is an outage even if nothing alerts.
• Latency percentiles: p50, p95, p99, per route. Not the average.
• Error rate: 4xx and 5xx separately. A rise in 5xx is a bug; a rise in 429 means you are rate-limiting real users.
• Active connections and the descriptor count against the limit. This is the one that fails silently, and it is how you find out you need a bigger worker_rlimit_nofile.
• CPU per core, not CPU total. A server at 60% average can have one core pinned at 100% and three idle, which is a NUMA or affinity problem, not a capacity problem.
• Memory usage and, for Go applications, GC pause time and heap size. For Ferron or other Rust servers, RSS and allocator stats matter more.
• Network throughput and packet rate, since as the hardware section explained these are different limits and you need to know which one you hit.
• TLS handshake rate and handshake failures. A high failure rate here is a bot problem or a certificate expiry, and it will eat your CPU before it shows up as errors.

The metric that ties it together is saturation over time, not instantaneous load. A server that touches 95% CPU for five minutes a day is fine. A server that averages 40% and spikes to 100% during your daily traffic peak is already failing, and the average will never tell you.

Getting the metrics out

Ferron 3 has observability built in, which is one of the nicer things about it: Prometheus, StatsD, Grafana dashboards, and OTLP export are all supported, and there is a ready-made Grafana reference dashboard.

example.com {
    observability {
        provider prometheus
        endpoint_listen 127.0.0.1:8889
    }
}

That exposes /metrics on port 8889 with OpenTelemetry semantic conventions converted to Prometheus format: counters, gauges, and histograms for request duration. Point Prometheus at it:

scrape_configs:
  - job_name: "ferron"
    static_configs:
      - targets: ["localhost:8889"]
    scrape_interval: 15s

Bind the metrics endpoint to localhost and set endpoint_auth_token. An open metrics endpoint on 0.0.0.0 is an information leak, and Ferron’s ferron doctor will flag it. If you need remote access, put it behind a reverse proxy that authenticates. The same caution applies to Ferron’s admin API: it has no encryption and no authentication by default, and an exposed POST /reload endpoint is a denial-of-service button anyone can press. Bind it to 127.0.0.1 and use SSH tunneling.

NGINX needs the stub_status module, which is not built by default, so it is the one step people miss:

./configure --with-http_stub_status_module

location = /basic_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
}

That page gives you active connections, total accepts, handled, and requests, plus the Reading/Writing/Waiting breakdown. The Reading/Waiting ratio is genuinely useful: a high Waiting count with high CPU means slow responses, while high Reading means clients are not sending fast enough. Note that handled will differ from accepts when a resource limit was hit, which makes it a free canary for descriptor exhaustion.

For proper histograms and per-route latency in NGINX, use the Prometheus exporter or the OpenTelemetry module rather than parsing stub_status. stub_status is cumulative counters only, so it cannot tell you your p99. The same allow/deny restriction applies, and the security guidance in the NGINX docs is explicit that this endpoint should not be exposed publicly.

Alerting on symptoms, not causes

The most common mistake is alerting on causes. “CPU above 80%” fires constantly, gets muted within a week, and tells you nothing actionable. Alert on symptoms (things users are experiencing) and use resource metrics for diagnosis once an alert fires.

Alerts I would actually keep, roughly in order of value:

• 5xx rate above a threshold for more than a couple of minutes. This is the one that matters. Any user-visible breakage shows up here first.
• p99 latency above your target. Not average, and pick a number your users would actually complain about.
• Request rate drop of more than some percentage versus the same period yesterday. Silent failures often look healthy: the server is fine, the data source behind it is empty, and nothing is erroring.
• Health check failures from outside your network. This catches the failure mode that internal monitoring cannot see, which is exactly the one where the server is too wedged to report its own problem.
• TLS handshake failure rate spike, which usually means a scanner or an expiring certificate.
• Disk space and inode usage on the web root and log directories. Full disks are among the most common outages there are, and among the most avoidable.
• Memory growth trend, not memory usage. A server at 70% that was at 30% last week has a leak, and you want to know a week before it dies.
• Certificate expiry, at 30, 14, and 7 days out.

Every alert needs a threshold that reflects a real user impact and a runbook. If you cannot write down what to do when it fires, it is not an alert, it is noise. And the corollary from my own experiments: if you cannot measure reliably, you cannot alert reliably, so build the measurement before the alerting.

Watch for alert fatigue by counting how often each one fires. An alert that fired 200 times last month is either broken or too sensitive, and both are worse than no alert because they train people to ignore the channel. A silent alert is a dead alert.

Logging, and the temptation to log less

Access logs are genuinely useful and genuinely expensive, and the instinct to disable them is usually a performance optimization made for the wrong reason. When I disabled Ferron’s access logging I measured no change at all (13.9k vs 13.8k req/s), because writes are buffered and the real I/O wait came from something else entirely. If you think logging is a bottleneck, A/B test it before you remove the thing you will need at 3am.

If you do keep access logs, buffer them and rotate them. Ferron buffers writes and flushes every second and on shutdown, and its access_log_rotate_size / access_log_rotate_keep directives handle rotation; the same guidance applies to NGINX. Unbounded log growth will eventually fill the disk, which is why disk usage is on the alert list above.

And if you are using a CDN, the origin access log is mostly redundant. The edge already saw every request. What you lose by not logging at the origin is the ability to see what actually reached you, which is exactly what you need when the CDN is caching something it should not be.

Conclusion

The disappointing summary of my experiments is the most useful thing I learned: most of the tuning this post covers did nothing. BBR was worse than cubic on a low-RTT network, RPS and IRQ pinning made no difference, qdisc changes were a wash, and a pile of backlog and buffer settings moved throughput by less than my measurement noise. If you had told me in advance that a day’s work would produce two settings worth keeping, I would not have believed you, and I would have been wrong to believe it.

What did work was almost never a sysctl. The changes that actually moved numbers, roughly in order of size: moving TLS out of the request path (3x), PGO at build time (p99 down 24% on static serving), a 13% proxy throughput gain and a p99 cut from 110 ms to 48 ms from swapping the memory allocator, enabling TLS session resumption, and one line of ulimit that stopped requests failing. The pattern is that architectural and build-time decisions dominate, and runtime knobs are mostly noise.

So the order of operations matters more than any individual technique. Measure, identify the saturated resource, fix that, and only then consider tuning. Skipping the first two steps is how people end up tuning a sysctl that cannot possibly help, which is what I would have done if the WiFi client had not shown me the 3x HTTP-to-HTTPS gap first.

If you only do five things, in this order:

• Measure p95 and p99, not averages, on a load test that resembles production, from a wired client, run more than once.
• Use an event-driven web server, and give it file descriptors to work with. This is the largest single architectural decision and it is not close.
• Move TLS, compression, and static files out of your application and into the web server, and enable session resumption.
• Have more RAM than you think you need, so the page cache does the work instead of your disk.
• Buy per-core speed, not core count, and leave 40–50% headroom at peak traffic.

And the last word goes back to where this started. Most of this is unnecessary. If your traffic is low, the right move is to find users, not to profile your server. Everything above earns its place only after infrastructure has genuinely become the bottleneck, and if you are not sure whether it has, the first thing to build is the monitoring, not the optimization. Buy servers by cost per request rather than cost per machine, and replace them on a schedule instead of waiting for them to become the reason your users leave.

Ah, this was a very long post indeed… 😅