Back

Ferron 3.0.0-beta.5 released

Published on:

Today, we’re releasing Ferron 3.0.0-beta.5, the fifth beta release of Ferron 3.

This release focuses on three major areas:

  • A significantly more resilient reverse proxy, with DNS-aware load balancing and a circuit breaker that now understands latency, flapping, and recovery
  • A large expansion of observability, connecting proxy internals, TLS handshakes, and pipeline stages directly to logs, metrics, and traces
  • New protections against symlink-based traversal and hostile endpoint scanning

Where beta.4 was about fixing what existed, beta.5 is about making the reverse proxy and its observability substantially more capable.

Behavior changes to know about

Two changes affect default runtime behavior without requiring configuration syntax updates. Review these if you’re upgrading from beta.4:

  • Circuit breakers are now enabled by default for all proxy configurations. Previously, circuit_breaker had to be explicitly configured. Now every proxy configuration gets transport-failure protection automatically. To disable this, set circuit_breaker false.
  • Static upstreams with hostnames now resolve via strict DNS by default. Instead of relying on the OS resolver to pick one IP per connection, Ferron resolves A/AAAA records itself and treats each resolved IP as a distinct backend, with its own load balancing weight, circuit breaker state, and health checks. If your upstream hostname resolves to multiple IPs, this changes how traffic is distributed across them — generally for the better, but worth knowing about. IP literals, localhost, Unix sockets, and upstreams with logical_dns true are unaffected.

A more resilient reverse proxy

The circuit breaker grows up

Ferron 3 introduced circuit breakers as Ferron’s mechanism for ejecting unhealthy backends. Beta.5 makes them substantially smarter.

Latency-aware tripping. A new latency_threshold sub-directive lets a circuit breaker count slow responses as failures, not just transport errors and 5xxs. A backend that returns 200 OK every time but takes eight seconds to do it will now get ejected just like one that’s actually down.

example.com {
    proxy {
        upstream http://localhost:3000
        upstream http://localhost:3001

        circuit_breaker {
            max_fails 3
            window "5s"
            latency_threshold "0.5s"
        }
    }
}

Flapping detection. Backends that rapidly oscillate between healthy and unhealthy are disruptive in a different way than backends that are simply down — every transition generates load balancer churn and log noise. New flapping_transitions and flapping_window sub-directives (defaulting to 3 transitions within 10s) detect this pattern. Once flapping is detected, individual transition logs are suppressed in favor of a single warning, and the ferron.proxy.circuit.flapping gauge is set to 1 until the backend stabilizes.

Slow-start recovery. When a backend transitions from half-open back to closed, it’s tempting to immediately send it a full share of traffic — which is exactly how a recovering backend gets knocked back down. A new slow_start duration applies a temporary, linearly-decaying connection penalty on recovery, giving the backend room to actually warm back up.

example.com {
    proxy {
        upstream http://localhost:3000
        upstream http://localhost:3001

        circuit_breaker {
            max_fails 3
            window "5s"
            flapping_transitions 3
            slow_start "0.1s"
        }
    }
}

DNS-aware load balancing

Static upstreams configured by hostname now go through Ferron’s own strict DNS resolution (via Hickory DNS) rather than relying on the OS resolver picking a single IP. Each resolved IP becomes its own backend, with independent load balancing, circuit breaker state, and health checks — the STRICT_DNS model familiar from Envoy.

DNS results are cached in memory, with expiry derived from the DNS response TTL (falling back to 30 seconds when no TTL is present), avoiding redundant lookups and resolver lock contention on high-traffic hostnames. Custom resolver IPs can be set via dns_servers.

For SRV upstreams, DNS-provided weights now factor directly into load balancing: each backend’s effective weight is dns_weight × config_weight, so the weight sub-directive acts as a multiplier on top of what DNS already tells Ferron. SRV priority handling also gained a config-level additive offset, letting SRV-discovered backends participate in the same tiered failover model as statically configured ones (below).

Weighted load balancing, everywhere

Weighted traffic distribution was previously limited to round_robin and least_conn. In beta.5, random, two_random, and p2c_ewma all support per-upstream weight as well — all five load balancing algorithms, plus session affinity, now respect weight consistently.

Priority-based failover

Backends can now be assigned a numeric priority, with lower values indicating higher priority. Ferron sends traffic only to the highest-priority tier with available backends, falling back to the next tier only when the current one is exhausted — useful for primary/standby topologies where you don’t want standby capacity absorbing traffic under normal conditions.

example.com {
    proxy {
        upstream http://primary:8080 {
            priority 0
        }
        upstream http://secondary:8080 {
            priority 1
        }
        upstream http://standby:8080 {
            priority 2
        }
    }
}

Smaller reverse proxy improvements

  • A connection_timeout sub-directive is now available on all upstreams, bounding how long Ferron will wait to establish a TCP connection to a backend before giving up.
  • Health check initialization logs now include the HTTP method and URI being checked.
  • Circuit breaker state transition logs now include the upstream address and how long the circuit was open, making post-incident review considerably easier.

Observability goes deep

This release adds more telemetry surface area than any prior beta. The goal throughout has been the same one that’s guided Ferron 3 from the start: when something goes wrong, you shouldn’t have to guess why.

TLS gets first-class visibility

New per-host metrics cover TLS handshake duration (ferron.tls.handshake.duration), handshake count (ferron.tls.handshake.total), and active connection count (ferron.tls.connections.active), each carrying ferron.host, tls.protocol.version, and tls.cipher_suite attributes. The ferron.request root span now also carries TLS protocol and cipher suite attributes directly, so you can correlate TLS parameters with individual requests without a separate span. OCSP fetch metrics gained a per-host dimension, plus a new ferron.ocsp.stapling.hit_total counter for per-request stapling outcomes.

Diagnosing the reverse proxy without guessing

A few additions here are worth calling out specifically:

Upstream response truncation detection. A new ferron.proxy.upstream.response_truncated counter and structured warning log fire when a backend closes its connection before sending all the bytes it promised in Content-Length. This is a failure mode that’s historically been invisible: the backend returns a valid status code, and everything looks fine until a client complains about a corrupted response. The warning log includes the backend URL, bytes actually received, and the expected content length.

Load balancer selection score. A new ferron.proxy.lb.score gauge exposes the combined score used by two_random and p2c_ewma to pick a backend, including both candidate scores compared during P2C selection. Lower scores mean more preferred backends — this gives direct visibility into why the load balancer picked what it picked, instead of needing to infer it from TTFB metrics after the fact.

Configurable IP cardinality. Resolved backend IPs are now exposed as a ferron.proxy.backend_resolved_ip metric attribute, but only when explicitly enabled via metrics_resolved_ip (default false). Previously, resolved IPs were always included, which could cause metric cardinality explosions in environments with ephemeral IPs, like Kubernetes. Enabling it gives per-IP visibility when you want it; leaving it off keeps cardinality bounded around configured backend URLs.

Tracing reaches every pipeline stage

Every HTTP pipeline stage now emits its own span attributes on ferron.stage.<name> spans — cache results, rate limit decisions, authentication outcomes, upstream backend URLs, response status codes, and more, all attached to the stage that produced them. This makes flame-graph-style trace analysis possible without needing metrics export configured at all.

The reverse proxy’s stage span specifically now carries live upstream state — circuit state, flapping status, health status, consecutive failure count, active connections — directly on the span for the request that was routed through it. If a request was slow or failed, the trace tells you what the backend looked like at that exact moment.

Logs get the same context traces do

Access logs previously required correlating with traces or metrics to get full context on a request. In beta.5, every pipeline module contributes its own fields directly to the access log line: proxy requests log backend URL, connection reuse, and retry count; cache lookups log result and zone; rate limiting logs result and zone; auth modules log their outcome; static file serving logs the resolved file path; CGI/FastCGI/SCGI backends log their script path. Circuit breaker state (closed, open, half_open) is now included on every proxied request’s access log line, not just at transition time.

Structured error logs for bad requests, timeouts, TLS handshake failures, and TCP errors also now include client.address and server.address, and IP attributes have been standardized to client_ip_canonical/server_ip_canonical across logs and traces (replacing client_ip/server_ip). Custom log fields containing a . are now used as-is in OTLP logs instead of being silently rewritten with _.

Hardening against traversal and scanning

Symlink traversal protection. A new disable_symlinks directive controls symlink handling during static file serving. Set to true, symlinks are detected but not followed during path resolution, closing off escape routes to paths outside the configured webroot. On Unix, if_not_owner allows following symlinks only when they’re owned by the same user as the target, a middle ground for shared hosting environments. Defaults to false for backward compatibility.

example.com {
    root /srv/www/example
    disable_symlinks
}

# Allow symlinks only in a specific virtual host
uploads.example.com {
    root /srv/uploads
    disable_symlinks if_not_owner
}

# Allow symlinks (default, backward compatible)
legacy.example.com {
    root /srv/www/legacy
    disable_symlinks false
}

Scanning detection. The abuse_protection directive gains an error_rate_threshold sub-directive that watches for clients generating an abnormal volume of configured error responses (404, 403, etc.) within a time window. This is aimed squarely at the kind of automated scanning that probes for old plugin paths and known-vulnerable endpoints — behavior that looks nothing like a brute-force login attempt, but is just as clearly hostile.

example.com {
    abuse_protection {
        ban_duration "15m"

        error_rate_threshold {
            events 10
            window "60s"
            status_codes "404" "403"
        }
    }
}

Static file serving internals

Two related changes tighten up how Ferron handles the filesystem, independent of disable_symlinks:

  • Static file metadata is now read via the already-open file handle (fstat-style) instead of a separate path-based stat call, closing the TOCTOU (time-of-check to time-of-use) window between validating a path and reading its metadata. A per-thread file descriptor reuse pool was added alongside this for performance.
  • Path resolution no longer relies on full filesystem canonicalization, instead using component-level validation. This is both faster — canonicalization requires a filesystem round-trip per path component — and, combined with disable_symlinks, gives Ferron the same defense-in-depth path validation model used by nginx.

Small fixes

  • tls-acme and tls-http log targets are now ferron-tls-acme and ferron-tls-http, consistent with the hyphenated naming used elsewhere, instead of the previous underscore-based names.
  • The ferron.proxy.circuit.state gauge had its open and half_open values swapped (1 and 2 respectively). This is now corrected to match the documented constants — if you built dashboards or alerts against the previous values, you’ll want to double-check them against this release.

Looking toward stable

Beta.5 pushes the reverse proxy and its observability further than any release so far. The circuit breaker is no longer just a binary healthy/unhealthy signal — it understands latency, flapping, and graceful recovery. Load balancing decisions are visible, not inferred. And a truncated backend response, previously a silent failure, now shows up as a metric and a log line the moment it happens.

As always, feedback, bug reports, and testing results are welcome — especially on the two default behavior changes above, since those are the ones most likely to surprise existing deployments.

Full changelog

The complete changelog for Ferron 3.0.0-beta.5 is available in the release notes.

Try it

Install Ferron 3 using the installer:

sudo bash -c "$(curl -fsSL https://get.ferron.sh/v3)"
Share this post!
X Facebook Linkedin Threads
Newer Ferron 3.0.0-beta.6 released Older End of life: sunsetting Ferron 1.x (and story about it)