OpenCache Runtime Behavior

This document describes how OpenCache actually handles traffic at the edge: what it adds to requests, what it adds to responses, how it decides what to cache, how requests collapse during peak load, how it serves stale content, and what diagnostic surface area it exposes.

It is the runtime companion to docs/configExplained.jsonc. Where that file documents the shape of SiteConfig (what knobs exist and what they accept), this document explains what those knobs actually do end-to-end and what behavior is "always-on" regardless of configuration.

If you change runtime request handling — new headers, new diagnostic endpoints, new variables, new transport features — please remember to update this file alongside the code change so operators downstream of OpenCache do not have to read the nginx config to discover behavior. Field-level schema changes belong in docs/configExplained.jsonc.

Overview
Protocols & Transport
Request → Upstream Headers
Upstream → Client Headers
Request Collapsing (proxy_cache_lock)
Cache Key Construction
Cache Decisions
Stale Serving
Cache Tier Policy & Tier Rules
GeoIP Enrichment
CORS Override
Diagnostic Endpoints
Build Identity ($oc_build_id)
Logging & Observability
Failover & Retries

1. Overview

OpenCache is a TLS-terminating reverse cache that sits between end clients and upstream origins. A single node runs an nginx data plane plus side-car services for BGP routing, prefix telemetry, and config generation from a centralized Directus key/value store (KVS).

Each PoP runs the same image, but its actual behavior is composed from three layers of configuration:

Site configuration — per-resource (opencache:edgeResource:<id>): upstreams, routes, TLS, cache rules, CORS, headers. This is what configExplained.jsonc documents.
Node configuration — per-PoP (opencache:nodeConfig:<POP_ID>) and fleet-wide (opencache:nodeConfigDefaults): cache storage, network, monitoring, availability, filters, retries, ACME, and (importantly) cacheTierPolicy — the fleet-operator-injected zones and tier rules described in §9.
Runtime, always-on behavior — emitted unconditionally by the config generator: request enrichment headers, diagnostic endpoints, observability headers, HTTP/3 advertisement, build-identity surfacing. This document focuses on this layer.

                       ┌─────────────────────────────┐
                       │  Client (browser / app)     │
                       └────────────┬────────────────┘
                                    │ TLS over TCP/443
                                    │ or QUIC over UDP/443
                                    ▼
        ┌──────────────────────────────────────────────────────┐
        │  nginx (data plane)                                  │
        │  ┌────────────────────────────────────────────────┐  │
        │  │  1. TLS termination + HSTS + OCSP stapling     │  │
        │  │  2. HTTP/1.1 / HTTP/2 / HTTP/3 negotiation     │  │
        │  │  3. GeoIP enrichment (silent fallback)         │  │
        │  │  4. Build identity tagging ($oc_build_id)      │  │
        │  │  5. Diagnostic endpoints (/oc-cgi/*)           │  │
        │  │  6. Route matching (location precedence)       │  │
        │  │  7. Cache rule selection                       │  │
        │  │  8. Query normalization (njs)                  │  │
        │  │  9. Cache key composition + request collapse   │  │
        │  │ 10. proxy_pass → upstream OR cache HIT/STALE   │  │
        │  │ 11. Cache-Control scrubbing (njs)              │  │
        │  │ 12. Observability headers (Pf-*, X-Cache-*)    │  │
        │  │ 13. CORS injection / Speculation-Rules strip   │  │
        │  └────────────────────────────────────────────────┘  │
        └────────────┬──────────────────────────┬──────────────┘
                     │                          │
                     │ HTTPS                    │ HTTP API (KVS)
                     ▼                          ▼
              ┌──────────────┐         ┌────────────────────┐
              │  Upstream    │         │  Directus KVS      │
              │  origins     │         │  (sites + nodes +  │
              └──────────────┘         │  tier policy)      │
                                       └────────────────────┘

Side-car services on the same node (not on the request path):

Bird — BGP peering, host networking, advertises PoP prefixes.
prefix-monitor — polls birdc, pushes prefix telemetry + node metrics into the KVS, exposes a Prometheus surface on :9101/oc-cgi/metrics.
config-generator — pulls edgeResource + nodeConfig* from the KVS, validates, resolves defaults, applies tier policy, renders nginx and Bird config, and triggers nginx reloads.

2. Protocols & Transport

TLS

TLS termination happens directly on the cache tier — there is no upstream TLS proxy in front of OpenCache. Certificates and keys are configured per-site under tls. Defaults emitted by the generator favor modern TLS:

Default protocol set is TLS 1.2 and TLS 1.3 unless overridden by tls.protocols.
Default cipher list is the Mozilla intermediate profile.
OCSP stapling is on by default. tls.ocspStapling controls whether the resolver is queried and verification is enforced.
HSTS is emitted when tls.hsts.enabled is true, with includeSubdomains / preload flags surfaced verbatim.
tls.redirectHttp: true causes the port 80 listener for that site to issue a 301 to https://$host$request_uri.

Legacy TLS (TLS 1.0 / 1.1, retired ciphers) is reachable for sites that opt in explicitly. When such a site is present, the generator emits OpenSSL 3.x's @SECLEVEL=0 cipher prefix so OpenSSL will load legacy primitives at all.

HTTP/2

HTTP/2 is enabled for every TLS listener using the modern http2 on; server directive (nginx 1.25+), not the deprecated listen ... http2 flag. ALPN advertises h2, http/1.1.

HTTP/3 (QUIC)

HTTP/3 is enabled on every site. The generator emits:

listen 443 quic reuseport (and the IPv6 equivalent) at the server scope, using SO_REUSEPORT so a rolling update can briefly have two nginx workers bound to UDP/443 without dropping packets.
http3 on; at the server scope.
add_header Alt-Svc 'h3=":443"; ma=86400' so HTTP/2 clients are told they can upgrade to HTTP/3 on the next connection.

Clients that already speak HTTP/3 (recent Chrome, Firefox, Safari, curl --http3) will negotiate it on the first connection; older clients fall back to HTTP/2 or HTTP/1.1 over TCP. There is no protocol downgrade attack surface — QUIC is on UDP/443 only and is independent of the TCP listener.

SO_REUSEPORT and Zero-Downtime Rolling Updates

Every default_server listener (port 80, 443 TCP, 443 UDP) is emitted with reuseport. Combined with the Swarm deploy order: start-first, this allows a new task to bind to the listening ports while the old task is still serving, producing a zero-loss handover for the duration of the deploy. KVS-driven config changes (cache rules, headers, route changes) still flow through nginx -s reload via config-watcher inside the running container — the overlap mechanism is only for image / binary rollouts.

3. Request → Upstream Headers

Every cache_proxy location injects the same set of upstream headers unconditionally, regardless of site configuration. Site authors may add their own headers, but they cannot suppress these.

Default proxy headers

These mirror nginx's proxy_set_header defaults but are emitted explicitly so they survive any proxy_pass_request_headers off-style overrides:

Host — $host (the SNI-resolved hostname the client sent, not the upstream IP).
X-Real-IP — $remote_addr.
X-Forwarded-For — $proxy_add_x_forwarded_for (preserves any upstream chain).
X-Forwarded-Proto — $scheme.
X-Forwarded-Host — $host.
X-Forwarded-Port — $server_port.

OC-* enrichment headers

OpenCache adds a structured family of OC- headers so the origin can make decisions (logging, geo-routing, abuse detection) without having to re-derive information OpenCache already computed. These are always present:

OC-Connecting-IP — $remote_addr (the TCP/UDP peer that reached this cache tier).
OC-RID — $request_id (nginx-generated per-request UUID; surfaces in our logs and in Pf-Cache-Timing so a downstream operator can pivot from a client report to an origin log line).
OC-Visitor — $oc_visitor (per-edge visitor identifier used for cache variance and abuse counting; see logging section).
OC-IPCountry — $geoip_country_code.
OC-IPCity — $geoip_city.
OC-IPContinent — $geoip_continent.
OC-Region — $geoip_region.
OC-Timezone — $geoip_timezone.
OC-IPLatitude — $geoip_latitude.
OC-IPLongitude — $geoip_longitude.

When the MaxMind GeoIP database is absent or fails to load, the $geoip_* variables silently expand to the empty string. The OC- headers are still emitted; they just carry empty values. Origins should treat absence and empty identically.

`passThroughRequest` whitelist

cache.rules[].headers.passThroughRequest is an allowlist of client request headers that should reach the upstream for that rule. Anything not in the list is stripped before proxy_pass. The default is "no extra headers" — meaning only nginx's normal proxy headers plus the OC-* family above traverse the cache tier. Operators should use this to forward Authorization, Cookie, custom auth tokens, etc. on the rules that need them, and to deliberately strip them on rules that should not key the cache on user identity.

4. Upstream → Client Headers

OpenCache rewrites a significant amount of the response surface. The general philosophy is: hide cache-tier implementation, surface observability under namespaced (Pf- / OC-) prefixes, and never emit a header twice.

`more_set_headers` vs `add_header`

OpenCache uses more_set_headers from headers-more-nginx-module for almost every response header it emits. add_header is avoided because it has two hazards:

It does not apply to all status codes by default — you have to add always and remember which directives stack.
It silently appends to upstream-supplied headers, producing duplicates that break clients (Chrome and Safari both reject duplicate Set-Cookie, Cache-Control, and Vary semantics).

more_set_headers replaces any existing instance and applies to every status code. The cost is that it depends on the ngx_http_headers_more_filter_module, which is compiled into the cache image.

Observability headers

Pf-Cache-Timing — a Server-Timing-style header carrying per-stage edge timing (TLS handshake, route match, cache lookup, upstream fetch) plus the matched cacheRule id. Format is intentionally machine-parseable but human-readable. See §14 for the field list.
Pf-Served-By — the PoP that served this response ($pop_id).
PF-Pop — same PoP id, kept for legacy clients that read this name.
PF-Server — the cache node's server name. Internal DNS suffixes (.pfsdns.com, .ocs.p.foundation) are stripped from this surface so external dashboards and clients see a shorter, more presentable identifier. Internal consumers that need the full FQDN (notably the prefix-monitor's SERVER_NAME env, which is used as a KVS key) keep the unstripped form.
X-Cache-Status — nginx's $upstream_cache_status surfaced verbatim. Values are MISS, BYPASS, EXPIRED, STALE, UPDATING, REVALIDATED, HIT. Absence of this header means the request did not enter a cache_proxy location (e.g. a 301 redirect from tls.redirectHttp, a static file route, or a diagnostic endpoint).

Cache-Control emission

Per-rule expiration.* settings translate into a Cache-Control value the edge will set on the response. The actual emission path depends on expiration.control:

passthrough — leave the upstream Cache-Control alone; do not emit our own.
override — more_set_headers replaces the upstream value with the rule's computed directive (built from browserTtl, cdnTtl, immutable, additionalDirectives).
additional — keep upstream value, append rule-specified extras.

Expires is emitted only when expiration.setExpiresHeader is true; modern clients ignore it in favor of Cache-Control, but some intermediaries still honor it.

Speculation-Rules stripping

Origins fronted by Cloudflare occasionally inject a Speculation-Rules response header pointing at a Cloudflare-hosted JSON document. When OpenCache is the client-facing tier, that pointer is meaningless (the client should be told about OpenCache's prefetch behavior, not Cloudflare's). The default behavior is to strip the header via more_set_headers "Speculation-Rules: $speculation_rules_filtered" where the variable resolves to the empty string when the filter matches. The filter is configurable; the goal is not to lose customer-set rules, only to remove Cloudflare-injected ones that leaked through.

CORS-injected headers

When CORS override is enabled for a route (see §11), the upstream's Access-Control-* family is cleared with more_clear_headers before OpenCache emits its own. This prevents the double-header trap from §4 above and makes the edge's CORS configuration authoritative.

Header emission ordering

Site-level headers[] is emitted in two phases:

Add actions (action: "add", or the default unset action) emit first via more_set_headers.
Remove actions (action: "remove") emit last via more_clear_headers.

This ordering guarantees that an operator removing a header always wins on collision, even if a different entry was trying to add the same header. The validator additionally rejects two entries with the same name (RFC 7230 case-insensitive) and opposite actions at configuration time, so this collision should be rare in practice.

5. Request Collapsing (proxy_cache_lock)

The stampede problem

When a popular object expires from cache, the naive behavior is for every in-flight request to fan out to the origin in parallel — a single second of expiration can produce thousands of upstream requests, blowing up origin load and potentially saturating the link. nginx solves this with proxy_cache_lock: only the first request goes to the origin, the rest wait for it to complete and then serve from cache.

Server-scope defaults

OpenCache emits proxy_cache_lock defaults at the server scope of every site, so they apply to every cache_proxy location by inheritance:

proxy_cache_lock_timeout 15s; — how long a waiting request will wait for the leader before bypassing the lock and going to origin itself.
proxy_cache_lock_age 5s; — how long the leader has to complete before a follower is allowed to take over and become the new leader.

These values are tuned for our workload (HLS and large objects from origins that occasionally take several seconds to begin streaming). They are significantly higher than nginx's stock defaults (5s and 200ms respectively) because the stock defaults defeated request collapsing for us: any upstream slower than 200ms produced cascading fallbacks where every "follower" became a new leader and we stampeded the origin anyway.

Per-location overrides

Cache rules may override either value via cache.lockTimeout and cache.lockAge. The generator only emits an override at the location scope when the rule's value actually differs from the server-scope default — this keeps generated configs small and makes diffs more readable.

To explicitly disable request collapsing on a specific rule, set cache.lockEnabled: false. The generator will emit proxy_cache_lock off; at that location.

Interaction with `backgroundUpdate`

When cache.backgroundUpdate: true is set on a rule, an expired cache entry is served immediately to the requester while a background request refreshes the cache. The first background-update request takes the lock, and other in-flight requests for the same object continue receiving the stale entry from cache rather than waiting. This is the preferred mode for hot objects where serving slightly-stale content is acceptable.

6. Cache Key Construction

Default key

The default cache key is $scheme$request_method$host$request_uri. The generator may extend this depending on rule configuration:

Per-rule cache.key — when set, replaces the default entirely. Variables in the string are nginx variables; $args and $normalized_args are the most common.
cache.global.queryNormalization.enabled — when true, $request_uri is replaced in the default key by a derived form that uses $normalized_args (see below).

Query normalization

cache.global.queryNormalization runs entirely in njs (the implementation lives under nginx/njs/query_norm.js and is bound via js_set $normalized_args). The pipeline order is fixed:

Drop exact — dropParams[] is matched case-insensitively against the parameter name. Any matching parameter (and its value) is removed.
Drop regex — dropParamsRegex is applied to each remaining parameter name; matches are removed.
Sort — when sortParams is true, remaining parameters are sorted alphabetically by name, with ties broken by value. Duplicate names are preserved.

The point is that ?utm_source=...&utm_medium=... does not invalidate cache entries for the underlying URL, and that ?a=1&b=2 and ?b=2&a=1 produce the same key. Drop lists are typically configured at the fleet level via tier policy; drop regex is reserved for site-specific UTM-style prefixes.

Shared-zone `$resource_id:` prefix

A cache zone with shared: true is a single nginx proxy_cache_path shared across every site that references it (rather than the usual per-site sub-directory layout). To prevent key collisions between two sites that happen to use the same default cache key for the same URL, OpenCache automatically prepends $resource_id: to the cache key when emitting the location for a shared zone. The $resource_id variable is set per-site to the resource id from the KVS (opencache:edgeResource:<id>), so the actual key becomes <resource-id>:<scheme><method><host><request_uri> (or whatever the rule's custom key is, with the same prefix).

This is invisible to operators — cache.key does not need to be edited to work in a shared zone — but it shows up in X-Cache-Status debugging if you sniff the cache files on disk.

Per-rule custom keys

cache.key is a free-form nginx variable expression. Common patterns:

$scheme$request_method$host$uri$is_args$normalized_args — same as default but force query normalization for one rule that opted in even when global is off.
$host$uri — ignore query, ignore scheme. Used for static asset rules where query strings are pure cache-busters and the cdn-applied filename hash is what actually invalidates.
$host$uri$cookie_session_tier — include a specific cookie as variance. The generator does not stop you from doing this, but you must add the cookie to headers.passThroughRequest separately or it will not actually vary because the cookie never reaches the cache_proxy location.

7. Cache Decisions

The cache decision for a given request is a sequence of gates. The order is deterministic and is the same regardless of which rule matched.

Bypass vs no-store

cache.bypass is an nginx expression evaluated per request. When truthy, the cache is consulted for write but bypassed for read — the request goes to origin, and the response may still be stored. X-Cache-Status is BYPASS.
cache.noStoreConditions is an additive set of expressions. When any evaluate truthy, the response is fetched normally but not stored. This is the right knob for "fetch live but never cache pages with ?preview=1".

The two settings are independent and may both be active.

Valid responses

cache.validResponses maps status codes (or buckets 2xx, 3xx, 4xx, 5xx, any) to durations. The matching is most-specific-first: a 200 mapping wins over a 2xx mapping. The value "off" is the explicit "do not cache this status code" sentinel — useful when, say, 5xx is set to a positive duration for negative caching but a specific 503 should not be cached.

When a status code is not mentioned in validResponses and no bucket covers it, nginx falls back to its default behavior, which is to not store the response.

`minUses`

cache.minUses is the number of times nginx must see an identical cache key before it actually writes it to disk. The default emitted by the generator is 1 (write on the first request). Setting it higher is useful for very large caches where you want to avoid one-shot URLs polluting the cache, trading first-hit MISS storage for cache hit rate.

`respectUpstreamCacheControl` modes

cache.respectUpstreamCacheControl (per-rule) and cache.global.respectUpstreamCacheControl (site default) control whether upstream Cache-Control directives are honored when deciding TTL. Modes:

respect — the upstream's directives win. If origin sends Cache-Control: no-store we will not store. If origin sends s-maxage=600 we will cache for 10 minutes regardless of validResponses.
ignore — upstream directives are dropped entirely. validResponses is the only TTL input.
merge — the more conservative of upstream-derived TTL and rule-derived TTL wins, but capped by upstreamCacheControlMaxAge (see below).

Per-directive control is available via upstreamCacheControlDirectives.respect[] and .ignore[] — useful for "honor s-maxage but ignore max-age" patterns where the origin sets very short max-age for browser cache reasons but larger s-maxage for shared caches like OpenCache.

`upstreamCacheControlMaxAge` ceiling

When respectUpstreamCacheControl is merge, the upstreamCacheControlMaxAge value is a cap on how high the merged TTL can go. This is the safety knob for "trust the origin's freshness signal, but never cache anything longer than 1 hour". Site-global default lives at cache.global.upstreamCacheControlMaxAge; per-rule overrides at cache.rules[].cache.upstreamCacheControlMaxAge. The implementation lives in njs (nginx/njs/cache_control.js) and runs as part of the response filter — it parses the upstream Cache-Control, computes the effective max-age, applies the cap, and rewrites the header before it is stored.

upstreamCacheControlMaxAge has no effect under respect or ignore modes, because those modes do not perform a merge.

8. Stale Serving

cache.stale enables nginx's proxy_cache_use_stale machinery: serve a previously-cached response when the upstream cannot be reached, has returned an error, or has not yet responded. The trigger set is configured via stale.conditions:

error — TCP-level connection error.
timeout — upstream took longer than proxy_read_timeout.
invalid_header — upstream returned malformed HTTP.
updating — a refresh is currently in flight (see backgroundUpdate).
http_500, http_502, http_503, http_504, http_403, http_404, http_429 — explicit status-code triggers.

Beyond proxy_cache_use_stale, OpenCache implements two related signals via stale.whileRevalidate and stale.whileError:

whileRevalidate — emit stale-while-revalidate=<n> in the response Cache-Control so browsers and downstream caches also know they may serve stale while we refresh.
whileError — emit stale-if-error=<n> for the same reason.

stale.maxAge is the hard ceiling on how old a cached entry may be and still be served as stale. After this, the entry is treated as gone and the request goes to origin normally (and fails normally if origin is unreachable).

9. Cache Tier Policy & Tier Rules

The problem

A fleet operator wants to apply uniform cache rules across every site without editing each site's edgeResource document. For example, "every site in this fleet should cache .css and .js files for 1 hour into a shared static_assets zone." Cache tier policy is the mechanism for this.

Storage

Cache tier policy lives in the node-config KVS, not the site-config KVS:

Fleet defaults — opencache:nodeConfigDefaults.cacheTierPolicy. Applied to every site at every PoP unless overridden.
Per-PoP overrides — opencache:nodeConfig:<POP_ID>.cacheTierPolicy. Deep-merged with the fleet defaults (arrays replaced, not appended). A PoP that needs to behave differently — say, an edge PoP with less disk that needs different zone sizes — overrides here.

Policy shape:

zones[] — full CacheZone objects. Appended to every site's cache.zones[] unless that site already has a zone with the same name (in which case the site wins and the conflict is logged).
rules[] — CacheTierRule objects (see below). Appended to every site's internal cache.tierRules[] field.
knownStaticFilesDefaults — provides a fallback cache.global.knownStaticFiles for sites that did not set one, and auto-enables cacheKnownStaticFiles: true for those sites.

`CacheTierRule` shape

A tier rule is a slimmer cousin of cache.rules[]:

id — unique identifier across the tier policy.
priority — higher wins on overlap between two tier rules.
match.uriPattern — nginx regex against the URI.
match.methods — optional method allowlist.
zone — which zone to use.
ttl — fixed TTL (no validResponses map; tier rules are intentionally simpler).
cacheKey — optional custom key.
declineWhenUpstream — array of upstream Cache-Control directives that should cause this tier rule to decline caching. Values are 'no-store', 'private', 'no-cache'.
lockEnabled — override request collapse for this tier rule.
stale — override stale config for this tier rule.

Emission semantics

Tier rules are emitted as standalone location blocks after all site routes. The site's own routes always win on a same-URI collision because they were emitted first.

However, there is a subtle interaction: when match.uriPattern has an extractable literal prefix (e.g. ^/static/.*\.css$ has literal prefix /static/), the generator wraps the tier rule's regex location inside an outer location ^~ <prefix> block. nginx's location precedence rules say ^~ prefix matches beat regex locations — even regex locations from the site config. This is intentional: it lets fleet operators carve out URL spaces that take priority over any site behavior, while still allowing sites to opt out (see below) or override specific URIs by using their own ^~ prefix locations.

Opting out

A site can disable all tier policy injection by setting zonesAutoManaged: false at the top level. This is the escape hatch for sites that need full control over their cache layout — typically because the tier policy zones clash with site-specific cache semantics that cannot tolerate fleet-wide rules.

Diagnostic report

Each config-generator run writes _TIER_POLICY_REPORT.txt into the generated configs directory, listing:

Which sites had policy applied vs opted out.
Which zones were added per site and which were skipped due to name collisions.
Which rules were added.
Memory tier oversubscription (sum of memory zone maxSize vs declared MEMORY_CACHE_SIZE).
Tier rule prefix grouping — which rules ended up inside location ^~ <prefix> wrappers and which stayed as standalone regex locations.
Zone reference validation — site routes / rules that reference a missing zone, fallback zones used by the generator, and missing knownStaticFiles zones.

This file is operator-facing; it is the first place to look when a tier rule does not behave as expected.

10. GeoIP Enrichment

OpenCache loads MaxMind GeoIP2 databases (City + ASN at minimum) at nginx startup. Database loading is decoupled from server config — when the database files are absent, the server still starts, and every $geoip_* variable silently expands to the empty string. This keeps the cache tier operational in lower environments and during the brief window after a node is provisioned but before the GeoIP databases have been synced down.

Variables exposed:

$geoip_country_code, $geoip_country_name
$geoip_continent
$geoip_region, $geoip_region_name
$geoip_city
$geoip_postal_code
$geoip_latitude, $geoip_longitude
$geoip_timezone
$geoip_as_org — AS organization, used by the ASN performance dashboard.
$geoip_asn — numeric ASN.

Consumers downstream of nginx:

Upstream OC-* headers (§3) — OC-IPCountry, OC-IPCity, etc.
Access log fields used by Loki and the per-ISP / per-ASN Grafana dashboards.
Loki stream selector labels — geoip_country_code is promoted to a stream label (low cardinality, useful for quick filtering); other geoip fields stay as structured metadata (higher cardinality, queryable but not indexed).
Diagnostic endpoints — /oc-cgi/trace exposes the resolved country code as the loc= field.

11. CORS Override

cors.enabled: true on a route (or on a site, applied per applyToUris[]) causes OpenCache to take authoritative control of the CORS response surface:

The upstream's Access-Control-Allow-Origin, Access-Control-Allow-Methods, Access-Control-Allow-Headers, Access-Control-Expose-Headers, Access-Control-Allow-Credentials, and Access-Control-Max-Age are cleared via more_clear_headers before the response is built.
The edge then injects the configured CORS values via more_set_headers.

This avoids the "two Access-Control-Allow-Origin headers" failure mode that breaks browsers entirely — CORS is one of the few cases where duplicate headers are not just sloppy but actively broken.

Preflight handling

OPTIONS requests matching a CORS-enabled route are short-circuited at the edge: OpenCache emits a 204 with the configured CORS headers and never contacts the origin. This is faster (one round trip avoided) and prevents the origin from accidentally serving a non-CORS response to a preflight.

`allowCredentials` and `*` origin

When allowCredentials: true, the wildcard origin * is replaced with the actual Origin request header. This is required by the spec — wildcards and credentials cannot coexist — and is the most common single source of "works in curl but not in browser" CORS bugs. The generator validates this at configuration time; runtime substitution happens via an nginx map.

12. Diagnostic Endpoints

Three diagnostic endpoints are emitted by the generator on every site (and on the catch-all default_server for unrecognized hostnames):

/oc-cgi/health — exact-match location, used by Swarm healthchecks.
/oc-cgi/metrics — exact-match location, proxies to the prefix-monitor Prometheus surface.
/oc-cgi/trace — exact-match location, returns a Cloudflare-style key=value diagnostic blob.

The entire /oc-cgi/ prefix is reserved by a location ^~ /oc-cgi/ block — site routes cannot accidentally claim a URL under that prefix.

`/oc-cgi/health`

Returns a plaintext OK when nginx is up and the configuration is loaded.

$ curl -s https://cache.example.net/oc-cgi/health
OK

This endpoint is used by docker-stack.yml's healthcheck. A 500 here will cause Swarm to mark the task unhealthy and (with failure_action: rollback) roll back to the previous deploy during a rolling update.

`/oc-cgi/metrics`

Proxies to http://prefix-monitor:9101/oc-cgi/metrics, which exposes:

Prefix monitor counters (BGP prefix delta rate, KVS update latency, etc.).
Host resource metrics (CPU, memory, disk).
nginx-cache scrape from the local nginx status endpoint.

$ curl -s https://cache.example.net/oc-cgi/metrics | head -10
# HELP opencache_prefix_changes_total Total BGP prefix changes observed
# TYPE opencache_prefix_changes_total counter
opencache_prefix_changes_total{pop="iad",protocol="bgp4"} 1428
# HELP opencache_kvs_update_seconds Histogram of KVS update latency
# TYPE opencache_kvs_update_seconds histogram
opencache_kvs_update_seconds_bucket{le="0.005"} 4
opencache_kvs_update_seconds_bucket{le="0.01"}  18
opencache_kvs_update_seconds_bucket{le="0.025"} 312
opencache_kvs_update_seconds_bucket{le="0.05"}  486
opencache_kvs_update_seconds_bucket{le="0.1"}   501

The endpoint is reachable from the public-facing site for convenience; access control (if any) is enforced at the prefix-monitor itself, not at this proxy.

`/oc-cgi/trace`

Returns a Cloudflare-style diagnostic body in a stable key=value format, one field per line:

$ curl -s https://cache.example.net/oc-cgi/trace
fl=1.4.2
h=cache.example.net
ip=203.0.113.42
ts=1736452200.917
visit_scheme=https
uag=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko)
colo=cache-iad-01/iad
http=http/3
loc=US
tls=TLSv1.3
sni=plaintext
kex=X25519

Field semantics:

fl — build identity ($oc_build_id). See §13.
h — $host (the SNI hostname). Useful for verifying that SNI made it through proxies.
ip — $remote_addr. The IP OpenCache sees you on.
ts — $msec (epoch seconds with millisecond precision).
visit_scheme — https or http.
uag — the User-Agent request header.
colo — <server-name>/<pop-id>. The specific cache node that served the request and the PoP it belongs to.
http — Cloudflare-style protocol id: http/1.1, http/2, or http/3. This is derived from $server_protocol via the $oc_http_version map.
loc — $geoip_country_code, or empty if GeoIP is not loaded.
tls — $ssl_protocol. TLSv1.2, TLSv1.3, etc.
sni — currently the literal plaintext; reserved for future expansion.
kex — $ssl_curve. The key exchange curve negotiated for this connection.

The same body is emitted at the catch-all default_server, so you can curl the trace endpoint at a node by IP (with --resolve) even when no site hostname resolves to it.

13. Build Identity (`$oc_build_id`)

Every generated config bakes a build identity into an http-scope variable named $oc_build_id. It is set at generation time from the config-generator's package.json version, with a -dev suffix when no IMAGE_TAG environment variable was set during generation:

IMAGE_TAG=v1.4.2 at generation → $oc_build_id = "1.4.2".
IMAGE_TAG unset → $oc_build_id = "1.4.2-dev".

The variable is surfaced via two channels:

/oc-cgi/trace's fl= field, so you can curl a node and immediately see what it is running.
The access log format includes $oc_build_id, so log entries are attributable to a specific generator output. This is useful when correlating a regression with a deploy.

Build identity is not the nginx binary version — there is a separate $nginx_version for that, and $oc_build_id only tells you which config-generator output produced this nginx config. It is the right identity for cache rule and route changes; for kernel / binary diagnostics you want the image tag from docker image inspect.

14. Logging & Observability

Access log format

The default access log is a JSON-shaped format with structured fields. Among the standard nginx fields ($remote_addr, $request, $status, $body_bytes_sent, etc.), OpenCache adds:

$oc_build_id — see §13.
$pop_id, $pf_server — the PoP and cache node identifiers.
$upstream_cache_status — HIT, MISS, etc.
$log_cache_rule_id — the id of the cache rule that matched. Set per-rule via set $log_cache_rule_id "<id>"; in the rendered config. Lets you filter logs by rule.
$log_cache_key — the actual cache key used (after query normalization and shared-zone prefixing). Useful for cache debugging — you can grep access logs for a key, see every request that hit it, and confirm normalization is doing what you expect.
$log_cache_target_time — the target TTL applied to the response, in seconds. Useful for spotting rules that are caching for the wrong duration.
$request_id — the same UUID emitted as OC-RID upstream and surfaced in Pf-Cache-Timing downstream. End-to-end correlation primitive.
$geoip_country_code, $geoip_as_org, etc. — geo enrichment.
dl_bps — observed download throughput for the response, computed from bytes sent and time-to-last-byte. Surfaced as both a log field and a Prometheus histogram. Used by the per-client and per-ISP latency dashboards.

`Pf-Cache-Timing` format

The Pf-Cache-Timing response header follows the Server-Timing syntactic convention: comma-separated entries, each of the form <name>;dur=<milliseconds>;desc="<human readable>". Names include:

tls — TLS handshake duration (only on first request of a connection).
route — route matching duration.
cacheLookup — time spent consulting the cache.
upstream — time spent waiting on the origin (zero on cache hit).
total — request total at the edge.
cacheRule — not a duration; emitted as desc="<rule-id>" only, so client-side script can pick out which rule served the request.
pop, server — emitted as desc-only fields for routing diagnosis.

Older clients ignore the header entirely; modern browsers expose it in DevTools' Network panel under "Server Timing", which makes this the most operator-friendly debugging surface OpenCache offers.

Loki labels and dashboards

Logs ship to Loki with the following stream labels (low cardinality, fast filter):

service — opencache.
pop — PoP id.
node — node hostname.
geoip_country_code — promoted from structured metadata to a label.

Everything else (rule id, cache key, AS org, etc.) stays as structured metadata — queryable via | json | label_format but not indexed as a label.

Grafana dashboards built on this data include:

ASN performance — cache hit rate, latency, and throughput broken down by geoip_as_org.
Per-client / per-ISP latency — request latency and throughput histograms keyed by visitor and AS.
OpenCache fleet overview — request rate, cache hit rate, error rate, and dl_bps per PoP.

15. Failover & Retries

`nextUpstream` policy

cache.proxySettings.nextUpstreamConditions controls when nginx considers an upstream attempt failed and tries the next upstream in the configured load-balancing pool. Allowed values mirror nginx's proxy_next_upstream directive: error, timeout, invalid_header, http_500, http_502, http_503, http_504, http_403, http_404, http_429, non_idempotent, off.

The default emitted by the generator is error timeout — retry on connection errors and timeouts only, not on 5xx response codes. This is deliberately conservative; surfacing the origin's actual 5xx to the client is usually the right behavior because it lets the client decide whether to retry, rather than the cache silently masking origin problems with extra latency.

nextUpstreamTries is the per-request attempt cap (counting the initial attempt). The default is 0 — unlimited within the timeout. nextUpstreamTimeout is the wall-clock cap on the entire upstream selection process. After this, nginx gives up and either returns the most recent upstream error or, if stale.conditions matched, serves stale.

Interaction with stale serving

Failover and stale serving compose: if stale.conditions includes the matching trigger (e.g. http_502), nginx will try the next upstream first, and only fall back to stale once nextUpstreamTries / nextUpstreamTimeout is exhausted. The X-Cache-Status of a stale response after exhausted failover is STALE. The Pf-Cache-Timing will include the failed upstream attempts as separate upstream entries, which is the easiest way to spot this pattern in practice.

`non_idempotent`

non_idempotent is the explicit opt-in to retry POST, PATCH, etc. nginx's default is to refuse to retry these because the origin might have already taken effect on a write. Enable it only when your origin is designed for it (idempotency tokens, etc.) — the default of "do not retry mutating requests" is the right one for the overwhelming majority of cases.

OpenCache Runtime Behavior

OpenCache Runtime Behavior

Table of Contents

1. Overview

2. Protocols & Transport

TLS

HTTP/2

HTTP/3 (QUIC)

SO_REUSEPORT and Zero-Downtime Rolling Updates

3. Request → Upstream Headers

Default proxy headers

OC-* enrichment headers

passThroughRequest whitelist

4. Upstream → Client Headers

more_set_headers vs add_header

Observability headers

Cache-Control emission

Speculation-Rules stripping

CORS-injected headers

Header emission ordering

5. Request Collapsing (proxy_cache_lock)

The stampede problem

Server-scope defaults

Per-location overrides

Interaction with backgroundUpdate

6. Cache Key Construction

Default key

Query normalization

Shared-zone $resource_id: prefix

Per-rule custom keys

7. Cache Decisions

Bypass vs no-store

Valid responses

minUses

respectUpstreamCacheControl modes

upstreamCacheControlMaxAge ceiling

8. Stale Serving

9. Cache Tier Policy & Tier Rules

The problem

Storage

CacheTierRule shape

Emission semantics

Opting out

Diagnostic report

10. GeoIP Enrichment

11. CORS Override

Preflight handling

allowCredentials and * origin

12. Diagnostic Endpoints

/oc-cgi/health

/oc-cgi/metrics

/oc-cgi/trace

13. Build Identity ($oc_build_id)

14. Logging & Observability

Access log format

Pf-Cache-Timing format

Loki labels and dashboards

15. Failover & Retries

nextUpstream policy

Interaction with stale serving

non_idempotent

`passThroughRequest` whitelist

`more_set_headers` vs `add_header`

Interaction with `backgroundUpdate`

Shared-zone `$resource_id:` prefix

`minUses`

`respectUpstreamCacheControl` modes

`upstreamCacheControlMaxAge` ceiling

`CacheTierRule` shape

`allowCredentials` and `*` origin

`/oc-cgi/health`

`/oc-cgi/metrics`

`/oc-cgi/trace`

13. Build Identity (`$oc_build_id`)

`Pf-Cache-Timing` format

`nextUpstream` policy

`non_idempotent`