Back

Ferron 3.0.0-beta.7 released

Published on:

Today, we’re releasing Ferron 3.0.0-beta.7, the seventh beta release of Ferron 3.

This release focuses on three areas:

  • A substantial upgrade to metric fidelity — native histograms, trace exemplars, and exponential bucketing across the board
  • Reverse proxy resilience against retry storms and traffic spikes, via a token-bucket retry budget and optional rate limit throttling
  • Closing the last significant “black box” in the request path: static file resolution now produces full observability signal instead of none

Breaking changes and metric renames

Beta.7 includes several changes that affect existing dashboards, alerts, and scrapers. Review these before upgrading:

  • OTLP histograms are now exponential by default. Every OTLP histogram metric — ferron.proxy.upstream.duration, ferron.proxy.ttfb, http.server.request.duration, and the rest — now uses Base2 Exponential Histograms instead of fixed linear buckets, unconditionally. This gives much better tail-latency resolution, but it’s a different wire format. If your observability backend doesn’t understand OTel exponential histograms, check its compatibility before upgrading.
  • Prometheus text endpoint content-type changed. The /metrics text endpoint now serves application/openmetrics-text; version=1.0.0 instead of text/plain; version=0.0.4. Scrapers without OpenMetrics content negotiation support may need updating.
  • Two attribute/metric renames: stage.name is now ferron.stage.name on trace spans, and the ferron.proxy.backend.excluded metric is now ferron.proxy.backends.excluded. Update any dashboards or alert rules referencing the old names.

Metrics get dramatically richer

Native histograms and exemplars

The Prometheus backend now supports OpenMetrics native histograms as an opt-in alternative to classic bucket histograms, enabled with endpoint_native_histograms true and endpoint_format protobuf. Native histograms are protobuf-only — the text format continues to expose classic buckets regardless of this setting.

observability {
    provider prometheus
    endpoint_listen "127.0.0.1:8889"
    endpoint_format protobuf
    endpoint_native_histograms true
}

Alongside this, counters and histograms now carry OpenMetrics exemplars — the trace_id and span_id of the specific request that produced a given observation — whenever trace context is available. In Grafana or Prometheus 2.23+, this means you can click directly from a spike on a metric graph into the exact trace that caused it, instead of guessing at a time window and hoping you land on the right request. Exemplars are on by default for counters; for histograms, they’re active only when native histograms are disabled, since the two are mutually exclusive.

Everything else gets a metric

A long list of smaller, targeted metrics round out this release:

  • DNS cache TTL remaining (ferron.proxy.dns.cache_ttl_remaining_seconds, aggregated as min/max/avg) and active cache entry count (ferron.proxy.dns.cache_entries), so you can alert on DNS records approaching expiration before a backend connection actually breaks.
  • Configuration content hash — a new ferron.admin.config_mtime gauge and config_file_hash/config_file_mtime fields on the Admin API’s /status endpoint expose an xxh3 hash of the loaded configuration. A single PromQL query comparing hashes across instances now tells you if any node in a cluster has drifted.
  • Connection pool limits — ferron.proxy.pool.local_limit and ferron.proxy.pool.global_limit expose configured pool ceilings directly, for comparing against actual usage.
  • Retry metrics gain method and idempotency context — ferron.proxy.retry.count and ferron.proxy.retry.final now carry http.request.method and an ferron.proxy.method_idempotent boolean (per RFC 9110 §9.2.2), so you can alert immediately on retries of non-idempotent requests — a signal worth treating very differently from a retried GET.
  • Selected backends per request — ferron.proxy.backends.selected_per_request exposes how many backends were actually contacted to fulfill a single request, surfacing traffic amplification that would otherwise be invisible in per-backend metrics.
  • DNS resolution outcome attribute — when metrics_resolved_ip is enabled, proxy metrics and access logs now carry a ferron.proxy.dns_status attribute (resolved, nxdomain, dns_error, logical_dns, or static), giving DNS-outcome visibility without the cardinality cost of labeling by individual IP.

Retry budgets: stopping a retry storm before it starts

The new retry_budget directive adds a token-bucket-based limiter on top of retry_connection. Instead of retries being unconditionally allowed, they now draw from a shared token pool that’s replenished by successful requests. Once the budget is exhausted, further retries are refused outright and the request returns 503 Service Unavailable immediately — rather than every failing request retrying and multiplying load on an already-struggling backend.

example.com {
    proxy {
        upstream http://localhost:3000
        upstream http://localhost:3001

        algorithm round_robin
        retry_connection true
        retry_budget {
            max_retry_rate 0.1
            max_tokens 10
            refill_rate 2.0
        }
    }
}

Defaults are max_retry_rate 0.1 (10%), max_tokens 10, and refill_rate 2.0 tokens/sec. Two new metrics, ferron.proxy.retry.budget_exhausted and ferron.proxy.retry.budget_tokens_available, make it possible to see the budget being consumed in real time rather than discovering it was exhausted after the fact.

This is the difference between a backend having a bad minute and a backend having a bad minute that turns into a self-inflicted denial of service.

Rate limit throttling as an alternative to rejection

The rate_limit directive gains a throttle subdirective. Instead of rejecting requests that exceed the configured rate outright, throttled requests are queued and delayed instead. For traffic that has a temporary, bursty spike rather than sustained abuse, this trades a slightly slower response for avoiding a wave of client-side errors.

api.example.com {
    proxy http://localhost:3000
    rate_limit {
        rate 50
        burst 100
        key request.header.X-Api-Key
        throttle
    }
}

DNS resolution gets quieter under load

The DNS resolver now uses a singleflight mechanism to collapse concurrent lookups for the same hostname into a single query, rather than issuing a duplicate query per waiting request. Under load — many connections opening simultaneously to a backend whose DNS entry just expired, for instance — this meaningfully reduces load on upstream DNS servers and avoids redundant resolution work.

Closing the file resolution black box

Static file resolution has, until now, been genuinely opaque from an observability standpoint: when a request failed during path resolution — a 403 Forbidden, a 400 Bad Request — there was no trace span, no log field, nothing to look at beyond the response code itself.

Beta.7 fixes this directly. File resolution now emits a dedicated ferron.pipeline.file_resolve trace span, carrying the request path, root path, outcome, and either the resolved path (on success) or the last path candidate examined (on error). These same fields now appear in access logs for resolution error paths. Directory listings get the same treatment: a ferron.static.dir_path span and access log attribute, plus contribution to the ferron.static.responses counter with distinct outcomes for listing, listing_disabled, method_not_allowed, and options.

If you’ve ever gotten a 403 from static file serving and had to reason your way through the resolution logic to figure out why, this closes exactly that gap.

Structured JSON errors

The new http-jsonerror module, enabled via json_errors true, turns 4xx/5xx error responses into structured JSON instead of an HTML page. Two output shapes are supported: RFC 9457 application/problem+json (with type, title, status, detail, and an optional trace_id field) or a simpler flat application/json format. The type URI is configurable and supports a {status} placeholder.

This is scoped like any other directive, so it’s easy to apply only where it makes sense:

{
    location /api {
        json_errors true
    }
}

For anything serving a REST API behind Ferron, this means clients get errors they can actually parse, instead of an HTML error page meant for a browser.

Smaller HTTP core additions

  • mtls.cn variable — the client certificate’s common name is now available for interpolation in routing rules and conditionals, useful for identity-based routing in mTLS setups.
  • Error page placeholders — custom error pages can now opt into {{trace.id}} and {{trace.spanid}} placeholder substitution via error_page_placeholders true, letting a custom error page surface the request’s trace ID directly to whoever’s looking at it. Note that enabling this reads the page into memory per response, bypassing the zero-copy sendfile optimization for those specific responses.

A fresh coat of paint on the default pages

The built-in error pages, directory listings, and installation landing page have all been redesigned — a modern, minimal look with system fonts, a subtle accent gradient, a responsive layout, and automatic light/dark mode, all still fully inlined with no external asset dependencies. Default error pages now also display the request’s trace ID when trace context is available, which is a small thing but a genuinely useful one: it means the debugging path from “a user sends me a screenshot of an error page” to “here’s the exact trace” no longer requires asking them anything else.

Fixed

  • Duplicate eviction metrics — eviction metrics and logs were being emitted twice on cache hits, inflating eviction counts. This is corrected.
  • Prometheus endpoint availability — the Prometheus /metrics endpoint was previously on-demand, with availability tied to the number of active connections. It’s now always available on its configured port regardless of connection state — important if your scraper was ever hitting an endpoint that wasn’t reliably up.

Looking ahead

Beta.7 caps off what’s been the throughline of the Ferron 3 beta cycle so far: making sure that when something goes wrong, the answer is in a trace, a log, or a metric — not a guess. Between the observability work across this and the last several betas, and the resilience work in the reverse proxy (circuit breakers, retry budgets, DNS-aware load balancing), the core of what beta was meant to accomplish is largely in place.

Looking ahead, Ferron 3’s focus will start shifting toward the ecosystem around the server itself — module API stabilization among other things — as the project works toward a release candidate and, eventually, a stable release.

As always, feedback, bug reports, and testing results are welcome.

Full changelog

The complete changelog for Ferron 3.0.0-beta.7 is available in the release notes.

Try it

Install Ferron 3 using the installer:

sudo bash -c "$(curl -fsSL https://get.ferron.sh/v3)"
Share this post!
X Facebook Linkedin Threads
Newer Ferron has a new look Older Ferron 3.0.0-beta.6 released