Today, we’re releasing Ferron 3.0.0-beta.6, the sixth beta release of Ferron 3.
This release focuses on three areas:
- Closing an authentication gap on the admin API and Prometheus scrape endpoint
- A significant deepening of cache observability — this time down to the level of individual coalesced requests and eviction reasons
- Multipath TCP support for the HTTP listener
We’re also publishing the internal harness we’ve been using to battle-test Ferron 3’s observability against real failure scenarios — more on that below.
A behavior change to know about
localhost is no longer exempt from strict DNS resolution. Since beta.3 introduced STRICT_DNS-style resolution for hostname-based upstreams, localhost was treated as a special case and left alone. In beta.6, it’s resolved like any other hostname, via A and AAAA records, same as everything else. For most setups this changes nothing — localhost still resolves to 127.0.0.1 (and ::1) as expected — but if your environment has unusual /etc/hosts or resolver configuration around localhost, it’s worth a quick check after upgrading.
Locking down the admin surfaces
The admin block now supports an auth_token directive, requiring bearer token authentication on every admin API endpoint except /health. Clients authenticate with an Authorization: Bearer <token> header; /health stays open so load balancers and orchestrators can keep probing without credentials.
{
admin {
listen "127.0.0.1:8081"
auth_token "secret-admin-token"
}
}The same protection is now available for the Prometheus scrape endpoint. The observability.prometheus block gains an endpoint_auth_token directive, requiring scrapers to present a bearer token to hit /metrics.
{
observability {
provider prometheus
endpoint_listen "0.0.0.0:8889"
endpoint_auth_token "secret-metrics-token"
}
}Neither of these is enabled by default — you opt in by setting a token — but if you’re running the Admin API or Prometheus endpoint on anything other than a fully trusted internal network, this closes a real gap. An unauthenticated Admin API is a config-reload and information-disclosure surface sitting in the open; this is exactly the kind of thing that looks fine until someone finds it.
Multipath TCP support
The tcp block now supports a multipath directive, enabling Multipath TCP (MPTCP) on the HTTP listener. MPTCP lets a single TCP connection use multiple network interfaces simultaneously, which can improve both throughput and resilience to a single interface flaking out — relevant if you’re running Ferron on hosts with more than one network path (multi-homed servers, some cloud multi-NIC setups).
{
tcp {
listen "[::]:443"
multipath true
}
}If the kernel doesn’t support MPTCP, or it’s disabled, Ferron logs a warning and falls back to a standard TCP socket rather than failing to start — so enabling this is safe to try even if you’re not sure your kernel supports it.
Cache observability gets forensic
Beta.5 added zone attribution and result fields to cache metrics and logs. Beta.6 goes several levels deeper, aimed squarely at the kind of cache questions that are otherwise hard to answer without reading source: why did this specific request miss, and how much time is being spent queued behind a revalidation — down to the individual coalesced request level.
Cache key fingerprint in access logs. A new ferron.cache.key_fingerprint field on access log lines gives a truncated representation of the actual cache key used for a request — no more inferring the key from configuration and guessing why two requests you expected to share a cache entry didn’t.
Singleflight coalescing visibility. When multiple concurrent requests for the same resource are deduplicated into a single upstream fetch (singleflight), access logs now include ferron.cache.coalesced and ferron.cache.coalesce_wait_duration_ms, showing exactly how long a given request queued behind another request’s revalidation. Two new metrics, ferron.cache.coalesced_requests (counter) and ferron.cache.singleflight_active_locks (gauge), expose the same behavior in aggregate — useful for confirming that a thundering-herd scenario was actually absorbed rather than passed straight through to the backend.
Cache key components in traces. The ferron.stage.cache span now carries ferron.cache.key.uri, ferron.cache.key.method, and ferron.cache.key.evaluated_cookies. When hit rate degrades because of high-cardinality metadata — a cookie fragmenting the cache key more than expected, for instance — this turns a trace flame graph into a direct answer instead of a starting point for more digging.
Eviction reason logs. Cache evictions now emit a debug-level structured log with eviction.reason (ttl_expired or capacity_reached_lru), eviction.count, and ferron.cache.zone. This sits alongside the existing eviction metrics but adds a precise timeline — useful when a max_entries capacity is being hit more often than the aggregate numbers suggested.
Status code on the store counter. The ferron.cache.stores metric now carries an http.response.status_code attribute, so you can directly query whether error responses are entering the cache without correlating timestamps across separate metric series.
New vary_cookies directive
The cache module gains a vary_cookies directive: an explicit list of cookie names to include in cache key differentiation. Without it, any cookie present on a request can end up as part of the cache key by way of existing Vary handling, and a single high-entropy tracking or session cookie can fragment your cache into near-zero hit rate. vary_cookies lets you name exactly which cookies actually matter for cache variation and ignore the rest.
example.com {
cache {
vary_cookies "session_id"
vary_cookies "tracking_id"
}
}Watching the admin plane and Prometheus scraper watch themselves
The admin API and Prometheus endpoint now emit telemetry about their own operation. The admin API gets per-request duration and count metrics (ferron.admin.request.duration, ferron.admin.request.count) with method, path, and status code attributes, a dedicated ferron.admin.reload.count counter, and structured logs for config reloads and queries (health checks are excluded to avoid log noise). The Prometheus endpoint gets scrape duration, count, and error counters (ferron_prometheus_scrape_duration_seconds, ferron_prometheus_scrape_total, ferron_prometheus_scrape_errors_total) for monitoring scrape health directly.
Process identity in telemetry
The OTLP resource now automatically includes process.pid and process.start_time. This sounds minor but solves a specific dashboarding problem: without it, cumulative counters from two overlapping process lifetimes — say, during a graceful restart — can visually interleave in a way that makes a perfectly normal restart look like a metric anomaly. With process identity attached, observability backends can tell the two lifetimes apart.
Cross-plane traceability with control_plane
A new control_plane directive lets you embed arbitrary key-value metadata, along with static OpenTelemetry span links, directly in Ferron’s configuration at global, host, or location scope. Metadata values are automatically attached as ferron.control_plane.* attributes on every observability signal — traces, logs, metrics, and access logs — and show up as [key=value] prefixes in console and file logs. The most specific scope wins, and values support variable interpolation.
The span links are the more interesting part: they let you establish a causal relationship between a control-plane event (a deployment, a config change made by an external orchestrator) and the data-plane traces that follow it — without needing an actual parent-child span relationship between the two.
{
control_plane {
metadata {
org_id "12345"
team "platform"
}
span_links {
trace_id "0af7651916cd43dd8448eb211c80319c"
span_id "00f067aa0ba902b7"
sampled true
}
}
}Introducing the AI SRE battle-test harness
For the last several beta cycles, a good number of the observability improvements in Ferron 3 — backend identity in access logs, circuit breaker state as metric gauges and log events, the Admin API itself, configurable connect timeouts — came out of a specific internal process: running AI SRE agents against live Ferron 3 (and Traefik, for comparison) deployments, injecting realistic failures, and checking whether the agent could actually diagnose the root cause using only the observability data available.
We’ve now published that harness: ai-sre-battletest.
It’s a chaos engineering setup with a shared observability stack (OTel Collector, Tempo, Mimir, Loki, Grafana) sitting in front of pluggable reverse proxies, currently supporting Traefik and Ferron 3. Twenty-plus scenarios are included — gray failures, circuit breaker blind spots, retry amplification storms, cache stampedes, cache poisoning, cache key fragmentation, health check path mismatches, TLS certificate expiry, and more — each designed so the agent has to actually find the root cause in traces, logs, and metrics rather than pattern-match a plausible-sounding narrative.
The admin-token work in this release is a direct example of the loop this harness is built around: an unauthenticated Admin API is one of its scenarios, and it’s exactly the class of gap that’s easy to miss until something is deliberately trying to find it. A good portion of this release’s cache observability additions map the same way to the harness’s cache-focused scenarios.
If you maintain a reverse proxy, API gateway, or similar piece of infrastructure and want to stress-test its observability the same way, the harness is designed to be extended to other proxies — check the README for details on adding one.
Fixed
Circuit breaker recording fix. Recent failures weren’t always being recorded correctly in circuit breaker state, which could prevent the circuit breaker from tripping when it should have. This is now corrected — circuit breaker logic functions as expected.
Looking toward stable
Beta.6 is smaller in scope than beta.5, but the two authentication additions matter disproportionately to their size, and the cache observability work continues a pattern we intend to keep going: when something in the cache behaves unexpectedly, the answer should be a metric or log line away, not a source read.
As always, feedback, bug reports, and testing results are welcome — and if you give the battle-test harness a run against your own setup, we’d like to hear how it goes.
Full changelog
The complete changelog for Ferron 3.0.0-beta.6 is available in the release notes.
Try it
- Documentation: https://ferron.sh/docs/v3
- GitHub repo: https://github.com/ferronweb/ferron/tree/develop-3.x
Install Ferron 3 using the installer:
sudo bash -c "$(curl -fsSL https://get.ferron.sh/v3)"