Back to all posts

Ferron 3.0.0-rc.9 released

Published on: Category: Releases

Today, we’re releasing Ferron 3.0.0-rc.9, the ninth release candidate of Ferron 3.

This release is mostly about correctness: a set of panic fixes around response headers, per-host http block inheritance, per-address active health checks, and a pile of static file and cache fixes.

I would like to emphasize one breaking change: the ferron.static.bytes_sent and ferron.static.files_served metrics are no longer emitted. They were mostly noise in the metrics output, and if you alert on them, drop those alerts before upgrading.

http block inheritance

A host block now inherits each global http directive that it does not set itself. Previously, a host block that set any http directive replaced the whole global http block, so every global setting the host did not repeat was silently dropped. A host that only set timeout lost the global protocols, h1_enable_early_hints, and all HTTP/2 and HTTP/3 settings.

This now layers the two blocks per directive, the same way directives in every other block inherit. Configurations that relied on a host block replacing the global http block will see the global values apply again for directives the host omits, so please re-check those.

No more panics from response headers

Several directives could build a response that panicked the worker task and dropped the connection, instead of answering:

  • options_allowed_methods with a value that cannot be sent as an Allow header, which broke every OPTIONS * request
  • basic_auth with a realm containing a control byte or a line break, which broke every 401 and 407 response
  • file_cache_control on the 304 and 412 precondition paths
  • mime_type containing a line break or a control byte, which broke every static file response for that extension (and could inject extra lines into multipart/byteranges bodies)

Ferron now answers without the unusable header, ignores the mapping at request time, and reports the value with ferron validate. I consider response header injection through configuration a bug class, and this release closes the paths I found.

Per-host tracing and observability fixes

  • http { trace { generate, trust_request } } and http { trace_sampling ... } were silently ignored in host blocks and were only read from the global block. Both are now resolved from the matched host block, falling back to the global block.
  • Baggage is no longer propagated from incoming to outgoing requests when trust_request false is set.
  • Multiple observability blocks inside a host block with an IP address or hostname now all take effect (only the last one did before). Wildcard host blocks already worked this way.
  • ferron_prometheus_scrape_total and ferron_prometheus_scrape_errors_total are renamed to ferron_prometheus_scrape and ferron_prometheus_scrape_errors (the old names ended up as _total_total).
  • ferron.proxy.circuit.open_total and ferron.proxy.circuit.half_open_timeouts were reported as signed values on counter metrics, a combination no observability backend supports, so both were silently dropped. open_total is now a gauge and half_open_timeouts an unsigned counter.
  • Backend scoped proxy metrics always report ferron.proxy.dns_status now, and only ferron.proxy.backend_resolved_ip stays behind metrics_resolved_ip. Previously the same backend was exported as two Prometheus series with different attribute sets, so any aggregation counted it twice.
  • Metric-specific Prometheus histogram buckets are respected instead of falling back to the defaults.
  • Access logs can now include response header fields, which is useful for seeing the headers Ferron actually sends on redirects.

Health checks and circuit breaking per address

Active health check state was keyed by the configured upstream URL, so every address behind one hostname shared a single state entry. One unreachable address marked all of them unhealthy and returned 503 for every request while healthy capacity was still available.

Health state is now tracked per resolved address, the same way circuit breaker state already was, and each address is probed on its own connection. Related fixes: the circuit breaker now transitions from half-open to closed when the HTTP pipeline timeout occurs (it stayed half-open until a process restart), and the retry budget only charges retries that can actually be attempted and enforces the configured max_retry_rate, which was accepted but not enforced.

Probe metrics also go to the sink of the host that owns the upstream, so a provider configured in a host block receives them instead of having them discarded.

Static files, ranges and conditional requests

A group of static file fixes that came out of testing with curl and various clients:

  • If-None-Match: * now gets a 304 response instead of a 200
  • If-Modified-Since and If-Unmodified-Since with malformed dates are ignored instead of returning 400
  • bytes=-0 is treated as unsatisfiable (416) instead of being ignored as invalid syntax
  • ETag and Date conditional requests are evaluated in the right order (If-None-Match wins over If-Modified-Since, If-Match over If-Unmodified-Since)
  • Range requests with precompressed files enabled no longer produce malformed partial content
  • Directory listings work on Windows instead of returning 403 (thanks to the reporter in issue #962)

Cache purge correctness

  • cache { } now enables caching, not just cache and cache { ... }
  • Purges with X-LiteSpeed-Purge: * no longer purge entries that were not created by the current host when it is configured in a block with an ambiguous hostname
  • Cache entries created by wildcard host blocks (like *.example) are no longer stored under a hostname that leaks entries across hosts
  • PURGE requests no longer miss matching entries because the request host was read after the proxy stage consumed the request
  • ferron.cache.evictions{reason="expired"} is emitted again when entries expire

Smaller things

  • map accepts interpolated string sources, so multiple values can be mapped into one value
  • All early_hints Link header values are sent, not just the last one
  • An unresolvable variable keeps the documented {{name}} placeholder
  • REMOTE_ADDR, SERVER_ADDR, and the client IP in X-Forwarded-For/Forwarded are canonicalized to standard IPv4/IPv6 form, so they can be used for access control and logging consistently
  • HTTP-01 challenge spans no longer carry attributes that are not useful
  • bcrypt password hashes, added in rc.8, were rejected by the validator and could not be used
  • protocols values are validated (invalid ones such as h4 are now reported), and h2_max_frame_size/h2_initial_window_size are range-checked
  • ferron validate and ferron doctor accept control_plane.span_links.sampled in bare boolean form, and h3_qpack_blocked_streams is documented as taking a stream count
  • Dynamic compression no longer applies to 304 Not Modified responses, and Accept-Encoding: * picks the most appropriate algorithm
  • A chunked request body keeps its Transfer-Encoding header when forwarded upstream
  • An Upgrade token inside another Connection header value no longer switches the connection to upgrade mode
  • Bare boolean forms of once (in rewrite) and replace_last_modified are effective again

Ferron 3 stable coming soon

Ferron 3 is still stabilizing as we are going closer to the first stable release. The release candidate phase focuses on correctness, resilience, and making Ferron easier to operate in production.

As always, I welcome feedback, bug reports, and testing results.

I think I would need to freeze Ferron 3 for a while for it to stabilize and to “soak test” it…

Full changelog

You can see the complete changelog for Ferron 3.0.0-rc.9 in the release notes.

Try it

Install Ferron 3 using the installer:

sudo bash -c "$(curl -fsSL https://get.ferron.sh/v3)"