Observability means being able to see the internal state of a system by looking at its external outputs. This helps you debug your web infrastructure when it fails by being able to determine the root cause, so you won’t hear it’s broken from customers first.
In this post, I’ll explain where it helps you debug web infrastructure, along with various use cases and scenarios.
Microservices
This one’s obvious. Microservices are distributed systems, and so they would need distributed observability to be able to see where and when it failed.
Also, many API gateways (single entry points to various microservices) also offer observability capabilities, which you can use to debug gateway-level errors too. And service meshes can help you observe traffic too.
Since microservices are distributed, observability here becomes critical, and distributed tracing with trace context propagation and signal correlation would help a lot.
Example of issue that observability can help resolve is a cascading failure caused by slow database query. A slow database query causes an inventory service to respond slowly, which leads order service to fail, which lead more users to click “place order”, which leads to resource exhaustion, service failure, which eventually cascades.
Observability would help a lot here, for example:
- Distributed tracing: operator would see that most of time is spent on waiting for a database
- Metrics: operator would know the HTTP error rate (500/504), spike in CPU usage, drop in successful orders and so on
- Logs: operator would know why it failed by looking at database error messages.
Automatic signal correlation (log trace IDs, metric exemplars) would also help a lot by not having to manually correlate signals during the failure (operators may be exhausted when failure occurs). For example, a slow database query span could be mapped to a log indicating the error through the trace ID.
Reverse proxying (monoliths)
Monoliths also benefit from observability, although it would be simpler here than microservices (centralized logging, metrics). Many reverse proxies also support logging and metrics, which can help for basic observability.
For example, a database error occurs, but simple uptime monitoring doesn’t answer why (it just reports the server is down). Then observability would help if database queries are instrumented. A DevOps engineer could then check the logs to see why it failed, and later on resolve the issue.
Another issue is the backend server itself going down, and reverse proxy returning 502 Bad Gateway or 503 Service Unavailable static code. If monitoring isn’t set up, they might not have known that it failed (however monitoring complements observability), however they can see when and why it failed by checking logs and metrics.
Caches (including CDNs)
Caching also benefits from observability, especially caching not being effective and stale cached data being served.
Here, cache hit/miss metrics would help a lot for checking if cache is effective. Without that, operators would guess how many requests hit or miss the cache. Also, there may be situations where stale or just plain wrong data is served to website visitors. In this case, visibility on the cache becomes critical.
For example, an application that uses a caching reverse proxy is suddenly becoming slower for customers. With observability, operator checks metrics, and notices that cache hit rate is low. Operator found the root cause: cache seems to be underutilized.
Static file serving
Yes, even static file serving benefits from observability, although much of it would be delegated to CDNs…
With observability, operators know the 404 error rate was increasing after some time, and this can be used to determine this is due to a failed deployment that ended up breaking the frontend. Also, they would know what path caused the 404 errors.
Basic monitoring alone would fail here, because:
- basic uptime checks would check the root URL, which returns 200
- automated alerts stay quiet, because there are no 5xx errors, and the server appears healthy (404 errors would be tolerated due to visitors trying to visit non-existent pages)
There’s also real user monitoring (RUM) and session replays that report errors in visitors’ browsers, allowing operators to see frontend errors, and this is especially useful for large, complex frontends.
Shameless plug?
Of course, that wouldn’t be Ferron blog post if I don’t mention Ferron web server… 😄
I’m building Ferron 3 (currently it’s beta) to be easier to debug and for less debugging friction for web infrastructure, by providing comprehensive signals and observability capabilities, such as:
- Microservices: Ferron 3 automatically injects W3C Trace Context headers (
traceparentand similar), so to allow end-to-end distributed tracing. - Monoliths: Ferron 3 includes features such as several backend metrics, such as
ferron.proxy.backends.selectedorferron.proxy.backends.unhealthy, circuit breakers (which can be configured to prevent 502/503 issues based on thresholds) and active health checking for proactively making sure the backends are responsive. - Caching: Ferron 3 includes varied caching information, such as
ferron.cache.key_fingerprintin access logs, orferron.cache.requestsmetric (which can be used for checking cache hit/miss rates). - Static file serving: Ferron 3 emits static file serving-related details in access logs and traces.
Ferron 3 also supports automatic signal correlation (trace context + metric exemplars), exponential histograms (for better tail/p90/p99 latency visibility), and more. It also can export signals to OTel collectors, Prometheus, or even traditional log files (yes, even log files there are enhanced with trace ID grep anchors).
You can try it out, starting by reading its documentation. You’ll see how Ferron’s observability capabilities help you debug web infrastructure a lot.
Conclusion
Observability helps you a lot while debugging infrastructure failures, no matter if doing basic static file serving, HTTP caching, hosting a CDN, or proxying to backend applications (microservices or monolith).
This is especially helpful, given how unpredictably web infrastructure fails (you cannot predict when it would fail, that’s also why monitoring would help).