Back to all posts

Ferron 3.0.0-rc.10 released

Published on: Author: Dorian Niemiec Category: Releases

Today, we’re releasing Ferron 3.0.0-rc.10, the tenth release candidate of Ferron 3.

This release is mainly about bug fixes and observability improvements, but it also includes new features such as CPU core pinning or support for explicit Incremental for HTTP buffering.

HTTP protocol hardening

When I was reading FoxIO’s blog post about QPACK amplification attacks, I wanted to try out similar attacks on zincio-http (which Ferron 3 uses). And I found out that zincio-http seems to be affected too (similarly to several popular HTTP/3 implementations such as quic-go or Cloudflare’s quiche).

So I ended up hardening the HTTP/3 logic against such attacks (see zincio-http changelog).

We have also hardened handling of HEAD requests, so HTTP/2 and /3 clients don’t error out due to excess body (see zincio-http changelog).

New features

We added support for CPU core pinning, which could be useful for systems with asymmetric CPU cores or multiple NUMA nodes. It can be configured like this:

{
    runtime {
        cpu_affinity 0 1 2 3 # first 4 CPU cores
    }
}

Also, we added support for disabling request and response body buffering (via buffering directives) using Incremental: ?1 (RFC 10036) or X-Accel-Buffering: no (used in some NGINX setups) headers.

Observability improvements

We also shipped improvements for observability to reduce the debugging friction:

  • http.route is now a route template: previously, the ferron.request span reported http.route as the requested hostname (or "*"). It now reports the matched route template built from the configured location block names, for example /api/users/:id, and is omitted when no location block matched.
  • ferron.host added to request spans: the requested virtual host is now reported as ferron.host on the ferron.request span. It uses the same attribute name as the TLS metrics, so traces and metrics join on the same key. server.address is unchanged and still reports the bound socket address.
  • Sampling probability reported in tracestate: when Ferron starts a trace itself and trace_sampling uses a ratio, the request’s trace context now carries the OpenTelemetry ot tracestate entry (for example ot=0.1), on the propagated headers and on the exported span. Backends that extrapolate sampled traces can scale counts back up; previously a ratio sampler silently reported only the sampled fraction of traffic. No ot entry is written for always_on, always_off, parentbased_always_on, a ratio of 1.0 or 0.0, or attribute_based sampling. An incoming tracestate is passed through unchanged.
  • Active health checking metric attributes: the metrics related to active health checking now have resolved IP address attributes when metrics_resolved_ip is enabled (similarly to circuit breaker metrics).
  • Aggregate active health checking partial success metrics: added a ferron.proxy.health.partial_success metric that reports health check probe results that succeeded only for some backend servers (percentage of successes in 0.0-1.0 scale).
  • Circuit breaker and active health check status metrics for multi-address upstreams: improved overall metric clarity and reduced ambiguity when metrics_resolved_ip is disabled.
  • TLS and HTTP protocol error log improvement: when server is logging to text-format application logs (not JSON structured ones), it now logs client IP address and port for TLS and HTTP protocol error logs.

Bug fixes

HTTP server core

  • trace.id and span.id resolution fix: previously, trace.id and span.id variable interpolations resolve to a doubly hex-encoded IDs, not proper single hex-encoded values.
  • Conditional configuration validation: the server is now reporting (via ferron validate) invalid forms of if, if_not, location and handle_error directives.

Observability

  • tracestate is now exported on spans: the OTLP exporter wrote an empty trace_state on every span, discarding the W3C tracestate that Ferron parses and propagates to upstreams. The span now reports it.
  • Span trace flags follow the sampling decision: the OTLP exporter hardcoded the “sampled” flag on every span. The flag sent to the OTLP backend or collector now reflects the recorded decision.
  • Prometheus scrape error metric: previously, ferron_prometheus_scrape_errors_total metric didn’t increase at all. This has been fix to increase when an internal error with Prometheus endpoint occured.
  • Connection pool metric fix: previously, outstanding connection count measured from exported metrics (like Prometheus or OTLP) was bogus, it didn’t decrease properly due to a logic bug.

Reverse proxying

  • Strict-DNS URL resolution fix: previously, strict-DNS URL resolution dropped the upstream path prefix (like /api in http://backend.example/api). This has been fixed to preserve the upstream path prefix.

Module API changes

The EventTraceContext struct for ferron_observability has been updated to include tracestate. If your custom Ferron modules rely on the EventTraceContext struct, you may need to update it to include tracestate (you can use tracestate: None).

Ferron 3 stable coming soon

Ferron 3 is getting closer and closer to the first stable release, as I saw some even use Ferron 3 to serve production websites. The remaining release candidate phase focuses on correctness, resilience, and making Ferron easier to operate in production.

As always, I welcome feedback, bug reports, and testing results.

Still thinking about freezing Ferron 3 for stabilization…

Full changelog

You can see the complete changelog for Ferron 3.0.0-rc.10 in the release notes.

Try it

Install Ferron 3 using the installer:

sudo bash -c "$(curl -fsSL https://get.ferron.sh/v3)"