# Configuration: ACME automatic TLS > Source: https://ferron.sh/docs/configuration/security/acme This page documents the ACME TLS provider (the `tls-acme` module). It automatically gets TLS certificates from ACME-compatible Certificate Authorities (CAs) such as Let's Encrypt. The provider supports both eager (startup-time) and on-demand (lazy, first-connection) certificate issuance. It offers three challenge types: - **HTTP-01**: serves a token at `/.well-known/acme-challenge/` (default) - **TLS-ALPN-01**: responds with a self-signed cert during the TLS handshake - **DNS-01**: creates a TXT record at `_acme-challenge.` Ferron caches certificates (both in-memory and file-based) and renews them automatically before expiration. Ferron enables automatic TLS via ACME by default for public hosts: ```ferron example.com { # Automatic TLS is enabled by default, no explicit TLS directive needed } ``` ## Directives ### Challenge types #### HTTP-01 (default) The simplest challenge type. The server listens on port 80 to serve `/.well-known/acme-challenge/`. ```ferron example.com { tls { provider acme challenge http-01 contact admin@example.com } } ``` **Requirements:** The server must be reachable on port 80 for the ACME CA to validate the challenge. #### TLS-ALPN-01 The server responds with a self-signed certificate when the CA connects with the `acme-tls/1` ALPN protocol. It needs no additional port. ```ferron example.com { tls { provider acme challenge tls-alpn-01 contact admin@example.com } } ``` **Requirements:** The server must be reachable on port 443. Does not support wildcard domains. #### DNS-01 (required for wildcard domains) Creates a `_acme-challenge` TXT record via a DNS provider. The only challenge type that supports wildcard certificates. ```ferron *.example.com { tls { provider acme challenge dns-01 contact admin@example.com dns { provider cloudflare api_key "EXAMPLE_API_KEY" } } } ``` **Requirements:** Configure a DNS provider module. The module supports wildcard domains. The `dns` block must specify the `provider` name and any provider-specific credentials. See [DNS providers](https://ferron.sh/docs/configuration/security/dns-providers.md) for the full list of supported providers and their directives. ### Configuration parameters | Parameter | Type | Default | Description | | ------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------- | | `provider` | `acme` | none | Must be `"acme"` | | `challenge` | `http-01`, `tls-alpn-01`, `dns-01` | `http-01` | ACME challenge type | | `contact` | `` | none | Email for ACME account | | `directory` | `` | LE Production | ACME directory URL | | `profile` | `` | none | ACME profile name (optional) | | `eab` | `"" ""` | none | External Account Binding | | `cache` | `` | `/var/cache/ferron-acme` if on Unix and writable, otherwise platform data dir | Path for file-based certificate caching | | `save` | ` [key]` | none | Save cert (and optionally key) to disk | | `post_obtain_command` | `` | none | Command to run after certificate issuance | | `no_verification` | `` | `false` | Skip ACME directory TLS verification | | `on_demand` | `` | `false` | Enable on-demand certificate issuance | | `on_demand_ask` | `` | none | Approval endpoint URL | | `on_demand_ask_auth` | `` | none | Authorization header for approval endpoint (optional) | | `on_demand_ask_no_verification` | `` | `false` | Skip TLS verification for approval endpoint | | `fallback` | `{ ... }` | none | Fallback provider block (repeatable) | **Configuration example:** ```ferron example.com { tls { provider acme challenge http-01 contact admin@example.com directory https://acme-v02.api.letsencrypt.org/directory cache /var/cache/ferron-acme save /etc/ssl/certs/example.com.pem /etc/ssl/private/example.com.pem # `post_obtain_command` arg is a script/binary name + args, separated by spaces. post_obtain_command /var/lib/post_obtain_command.sh ocsp } } ``` ## Eager mode (recommended for known domains) Eager mode gets certificates at server startup, before any client traffic arrives. This is ideal for static configurations where you know all domain names in advance. ## On-demand mode On-demand mode defers certificate issuance when doing the first TLS handshake for a hostname. This is useful for wildcard domains, multi-tenant hosting, or when you do not know the domains at startup. ```ferron *.example.com { tls { provider acme challenge dns-01 contact admin@example.com on_demand } } ``` ### On-demand approval endpoint To prevent abuse, you can configure an approval endpoint. Before issuing a certificate, Ferron sends an HTTP GET request to the endpoint with `?domain=` as a query parameter. If the response is `200`, Ferron issues the certificate. ```ferron *.example.com { tls { provider acme challenge http-01 contact admin@example.com on_demand on_demand_ask https://internal-api.example.com/check-cert } } ``` ## Certificate caching ### In-memory cache (default) Without a `cache` path, Ferron stores certificates and account data in memory. ### File-based cache Setting a `cache` path persists certificates and accounts to disk, surviving restarts: ```ferron example.com { tls { provider acme challenge http-01 contact admin@example.com cache /var/cache/ferron-acme } } ``` The cache directory structure: ```text /var/cache/ferron-acme/ ├── account_ # ACME account credentials ├── certificate_ # Certificate chain + private key (JSON) ├── challenge_http_ # Shared HTTP-01 challenge (distributed mode) ├── challenge_tls_ # Shared TLS-ALPN-01 challenge (distributed mode) └── lock_certificate_ # Provisioning lockfile (distributed mode) ``` ### Cache key derivation - **Account cache key**: hash of `contact emails + directory URL` - **Certificate cache key**: hash of `sorted domains + profile name` - **Challenge keys**: hash of the challenge `token` (`challenge_http_*`) or identifier (`challenge_tls_*`) - **Lock keys**: `lock_` + the paired `certificate_*` / `account_*` / `hostname_*` key, so `ls lock_*` groups all locks ## Distributed mode (shared filesystem) Point every Ferron instance at the same shared `cache` directory (NFS, EFS, CephFS) to run ACME behind a load balancer: ```ferron example.com { tls { provider acme challenge http-01 contact admin@example.com cache /var/cache/ferron-acme } } ``` **How it works:** - **Challenge sync**: the node creating the order publishes `challenge_http_*` / `challenge_tls_*` files (15-minute TTL). Peers serve CA validation from these files when their in-memory locks miss, so validation succeeds no matter which node the CA reaches. - **One order at a time**: before ordering, a node takes the paired `lock_certificate_*` lockfile (atomic create). Peers skip that cycle and pick up the resulting `certificate_*` file later. On-demand `hostname_*` appends take `lock_hostname_*` the same way. - **Self-healing locks**: the holder refreshes a heartbeat every 30 seconds. A lock without heartbeat for 5 minutes is treated as stale (crashed holder) and broken by the next contender, which logs a warning. Releases only delete their own lock, never a peer's. - **Herd damping**: skipped cycles return to the 10-second loop, and cycle sleeps include jitter so restarted nodes desynchronize. **Requirements and notes:** - All nodes must share one writable `cache` path with atomic create/rename semantics and roughly synchronized clocks. In-memory caching cannot coordinate. - Challenge and key material is `0600` on Unix, but any node with cache access can read it. Restrict cache access to your Ferron hosts. - DNS-01 needs no challenge files (the TXT record is already shared), meaning the provisioning lock still prevents duplicate orders. - Debug with `ls lock_*` (stuck entries show the owner `host-pid`) and the `ACME lock stale, broken` / `ACME provisioning skipped, peer holds lock` log messages. ## Certificate renewal Ferron automatically renews certificates before expiration. The renewal check runs every 10 seconds in the background. Ferron uses the ACME `renewalInfo` endpoint (RFC 9773) when available. Otherwise it falls back to a heuristic of 50% of certificate lifetime, capped at 24 hours before expiry. ## External Account Binding (EAB) Some CAs (especially enterprise/internal ACME servers) require External Account Binding. Set the key ID and HMAC secret: ```ferron example.com { tls { provider acme challenge http-01 contact admin@example.com eab my-key-id "SMq9KpHkR7z..." directory https://acme.internal.example.com/directory } } ``` The HMAC secret must be base64url-encoded (without padding). ## Fallback providers When high availability is critical, you can configure fallback ACME providers. If the primary provider fails (for example, the CA is down or unreachable), Ferron tries the next configured fallback provider when certificate provisioning fails. ```ferron example.com { tls { provider acme challenge http-01 contact admin@example.com directory https://acme-v02.api.letsencrypt.org/directory fallback { directory https://acme-staging-v02.api.letsencrypt.org/directory contact admin@example.com } fallback { directory https://other-ca.example.com/directory eab my-key-id SMq9KpHkR7z... profile my-profile } } } ``` Each `fallback` block accepts the same provider-level directives as the primary configuration: `directory`, `contact`, `eab`, and `profile`. The `fallback` block inherits the settings of the primary provider. You only need to specify the fields that differ. ### Example: primary with staging fallback A common pattern is to configure production as the primary and staging as a fallback. Use this for testing or when production is unavailable: ```ferron example.com { tls { provider acme challenge http-01 contact admin@example.com directory https://acme-v02.api.letsencrypt.org/directory fallback { directory https://acme-staging-v02.api.letsencrypt.org/directory contact admin@example.com } } } ``` ## Saving certificates to disk To persist certificates you get for use by other tools or backup: ```ferron example.com { tls { provider acme challenge http-01 contact admin@example.com save /etc/ssl/certs/example.com.pem /etc/ssl/private/example.com.pem } } ``` If you give only one path, the key path defaults to the certificate path with a `.key` extension. After Ferron gets a certificate, it writes the private key with `0600` permissions on Unix. ## Security considerations - **Private keys are never logged** or exposed in error messages. - When saved to disk, Ferron writes keys with `0600` permissions on Unix. - When using on-demand mode, always configure an `on_demand_ask` endpoint in production to prevent certificate issuance for arbitrary hostnames. ## Troubleshooting ### "ACME certificate provisioning error: ..." Certificate issuance failed. The log message includes the affected domains. Check the error message for details (DNS resolution, ACME server errors, and so on). At debug log level, you will also see per-step messages for account loading, order creation, challenge solving, and certificate installation. ### DNS-01 issues - Make sure you configure the DNS provider correctly with valid credentials. - Check that the provider has permission to create TXT records for the domain. - DNS propagation may take longer than 60 seconds for some providers. In these cases, the ACME CA will retry validation. ### Verifying certificates ```bash # Check the certificate served by Ferron echo | openssl s_client -connect example.com -servername example.com 2>/dev/null | openssl x509 -noout -subject -dates -issuer # Verify OCSP stapling openssl s_client -connect example.com -status -servername example.com /dev/null | grep -A 5 "OCSP response" ``` ## Observability The ACME background task emits log events and metrics through the configured observability pipeline: > [!note] > The ACME log events are global-only. Ferron does not emit them via per-host observability sinks. ### Logs | Level | Message | When | | ------- | ------------------------------------------------------------------------- | --------------------------------------- | | `INFO` | `ACME background task started with N configuration(s) for domains: ...` | Service initialization | | `INFO` | `On-demand certificate requested for SNI :` | On-demand certificate request received | | `INFO` | `On-demand certificate pre-loaded for SNI :` | On-demand certificate loaded from cache | | `INFO` | `ACME certificate issued for domains: ...` | Successful certificate issuance | | `INFO` | `ACME account created for directory ..., contact: ...` | New ACME account registration | | `INFO` | `Post-obtain command started for ...: ` | Post-obtain hook execution | | `WARN` | `ACME certificate provisioning error for ...: ` | Certificate issuance failure | | `WARN` | `ACME account not found on server for ..., recreating` | Account expired/removed on CA side | | `WARN` | `Post-obtain command failed for ...: ` | Post-obtain hook error | | `WARN` | `Post-obtain command has malformed quoting for ...` | Post-obtain hook malformed command | | `WARN` | `Failed to save ACME account cache: ` | Account cache write failure | | `WARN` | `Failed to save ACME certificate cache: ` | Certificate cache write failure | | `WARN` | `Broke a stale ACME provisioning lock for ...` | Stale lockfile cleared (holder crashed) | | `WARN` | `Failed to acquire ACME provisioning lock for ..., proceeding without it` | Lock I/O failure, fail-open | | `DEBUG` | `Skipping ACME provisioning for ...: a peer holds the provisioning lock` | Contended cycle skipped (herd damping) | | `DEBUG` | `ACME provisioning cycle started — checking N configurations` | Each background loop iteration | | `DEBUG` | `ACME account loaded from cache for ...` | Account reused from cache | | `DEBUG` | `ACME order created for domains: ...` | New order placed with CA | | `DEBUG` | `ACME challenge initiated for ...` | Challenge setup started | | `DEBUG` | `ACME challenge solved for ...` | Challenge ready for validation | | `DEBUG` | `DNS-01 record created for _acme-challenge., TTL ` | DNS record published | | `DEBUG` | `DNS-01 record cleanup completed for _acme-challenge.` | DNS record removed | | `DEBUG` | `Certificate installed for ..., chain length: N` | Certificate loaded into TLS config | ### Structured logs In OTLP `log_style modern`, the `summary` field acts as the log body and Ferron emits `attributes` as typed OpenTelemetry log record attributes. | Summary | Level | Attributes | | ----------------------------------- | ----- | ----------------------------------------------------------------------------------------------------------------------------------- | | ACME background task started | INFO | `ferron.acme.config_count` (int), `ferron.acme.domains` (string) | | ACME account created | INFO | `ferron.acme.directory` (string): ACME directory URL, `ferron.acme.contact` (string): account email, `ferron.acme.profile` (string) | | ACME certificate issued | INFO | `ferron.acme.domains` (string) | | ACME post-obtain command started | INFO | `ferron.acme.domains` (string) | | On-demand certificate pre-loaded | INFO | `tls.sni` (string), `tls.port` (int) | | On-demand certificate requested | INFO | `tls.sni` (string), `tls.port` (int) | | ACME account recreated | WARN | `ferron.acme.domains` (string), `ferron.acme.directory` (string) | | ACME certificate provisioning error | WARN | `ferron.acme.domains` (string), `error.message` (string) | | ACME post-obtain command malformed | WARN | `ferron.acme.domains` (string) | | ACME post-obtain command failed | WARN | `ferron.acme.domains` (string), `error.message` (string) | | ACME post-obtain command empty | WARN | `ferron.acme.domains` (string) | | ACME account cache save failed | WARN | `error.message` (string) | | ACME certificate cache save failed | WARN | `error.message` (string) | | ACME provisioning cycle started | DEBUG | `ferron.acme.config_count` (int) | | ACME account loaded from cache | DEBUG | `ferron.acme.domains` (string), `ferron.acme.provider` (string) | | ACME order created | DEBUG | `ferron.acme.domains` (string) | | ACME certificate installed | DEBUG | `ferron.acme.domains` (string), `ferron.acme.chain_length` (int) | | ACME challenge initiated | DEBUG | `ferron.acme.domains` (string), `ferron.acme.challenge_type` (string) | | ACME challenge solved | DEBUG | `ferron.acme.domains` (string), `ferron.acme.challenge_type` (string) | | ACME DNS-01 record created | DEBUG | `ferron.acme.dns_challenge_domain` (string), `ferron.acme.dns_ttl` (int) | | ACME DNS-01 record cleanup | DEBUG | `ferron.acme.dns_challenge_domain` (string) | | ACME lock stale, broken | WARN | `ferron.acme.domains` (string), `ferron.acme.lock_key` (string): stale lockfile cleared after heartbeat lease expiry | | ACME provisioning skipped | DEBUG | `ferron.acme.domains` (string): peer holds the provisioning lock; this cycle is skipped | | ACME lock acquisition failed | WARN | `ferron.acme.domains` (string), `error.message` (string): lock I/O failed, provisioning proceeds without coordination | | ACME account creation failed | WARN | `ferron.acme.domains` (string), `ferron.acme.provider` (string), `error.message` (string) | | ACME account load/create failed | ERROR | `ferron.acme.domains` (string), `error.message` (string) | | ACME order creation failed | ERROR | `ferron.acme.domains` (string), `error.message` (string), `ferron.acme.provider` (string) | | ACME authorization failed | ERROR | `ferron.acme.domains` (string), `ferron.acme.auth_status` (string), `ferron.acme.provider` (string) | | ACME challenge type unsupported | ERROR | `ferron.acme.domains` (string), `ferron.acme.challenge_type` (string), `ferron.acme.provider` (string) | | ACME identifier type unsupported | ERROR | `ferron.acme.domains` (string), `ferron.acme.identifier_type` (string), `ferron.acme.provider` (string) | | ACME challenge ready failed | ERROR | `ferron.acme.domains` (string), `error.message` (string), `ferron.acme.provider` (string) | | ACME order finalization failed | ERROR | `ferron.acme.domains` (string), `error.message` (string), `ferron.acme.provider` (string) | | ACME order invalid | ERROR | `ferron.acme.domains` (string) | | ACME order not ready | ERROR | `ferron.acme.domains` (string), `ferron.acme.order_status` (string) | | ACME finalize failed | ERROR | `ferron.acme.domains` (string), `error.message` (string), `ferron.acme.provider` (string) | | ACME certificate obtain failed | ERROR | `ferron.acme.domains` (string), `error.message` (string), `ferron.acme.provider` (string) | | Certificate issuance denied | ERROR | `tls.sni` (string): hostname blocked by ask endpoint | | Ask endpoint error | ERROR | `tls.sni` (string), `error.message` (string) | ### Metrics | Metric | Type | Attributes | Description | | --------------------------------------- | ------- | --------------------------------------------------------------------------------- | -------------------------------------------- | | `ferron.acme.certificates_issued_total` | Counter | `ferron.acme.status` (`success`, `error`), `ferron.acme.challenge_type` | Certificate issuance outcomes | | `ferron.acme.on_demand_requests_total` | Counter | None | On-demand certificate requests | | `ferron.acme.lock_acquired_total` | Counter | `ferron.acme.domains`, `ferron.acme.lock_key` | Provisioning locks acquired | | `ferron.acme.lock_contention_total` | Counter | `ferron.acme.domains`, `ferron.acme.lock_key` | Cycles skipped, peer holds the lock | | `ferron.acme.lock_breaks_total` | Counter | `ferron.acme.domains`, `ferron.acme.lock_key` | Stale locks broken (crashed holder) | | `ferron.tls.certificate_not_after` | Gauge | `ferron.host`, `ferron.tls.provider` (`acme`), `crypto.certificate.serial_number` | Certificate `notAfter` as Unix epoch seconds | ## See also - [DNS providers](https://ferron.sh/docs/configuration/security/dns-providers.md): all supported DNS-01 provider backends and their configuration - [Security and TLS](https://ferron.sh/docs/configuration/security/tls.md): cipher suites, ECDH curves, mTLS - [TLS session ticket keys](https://ferron.sh/docs/configuration/security/session-tickets.md): session resumption - [OCSP stapling](https://ferron.sh/docs/configuration/security/ocsp.md): OCSP response stapling - [HTTP host directives](https://ferron.sh/docs/configuration/server/host.md): per-host TLS configuration ## Best practices `ferron doctor` reports the following best-practice checks for directives on this page. - **`no_verification` for ACME directory**: Only disable TLS verification for the ACME directory in testing. - **`on_demand` without `on_demand_ask`**: On-demand certificate issuance without an approval endpoint allows certificate issuance for arbitrary hostnames. Configure `on_demand_ask` to approve requests. - **`on_demand_ask_no_verification`**: Only disable TLS verification for the approval endpoint on strictly internal and otherwise authenticated endpoints. - **Missing `contact`**: Without an ACME account email, the certificate authority cannot send expiry or account notices. - **Non-public domain**: Domains using non-public TLDs (`.local`, `.internal`, `.home`, `.lan`, `.test`, `.localhost`) or bare IP addresses are unlikely to be publicly resolvable. Certificate issuance via ACME will fail because the CA cannot complete domain validation. This check applies to both explicit `domains` directives and to the host block's hostname when you use `on_demand` TLS.