- debugging
- nginx
- cdn
- tls
Four correct decisions, one broken link
A signed confirmation link returned 403 after login and nowhere else. CDN, nginx, framework and application were each configured correctly. Together they were not.
This is a short debugging story. It has no villain. Every component was configured the way its documentation recommends, and the result was still a broken feature that only showed up after login.
The symptom
Our customer portal sends e-mails with signed links: the URL carries an expiry and an HMAC, and the application verifies both before acting. A customer clicked one, was asked to log in (expected), logged in, and got 403 Invalid signature. The same link opened in a browser that was already logged in also failed. Links generated minutes earlier had worked.
The chain
Four facts, each individually fine:
1. The portal sits behind a CDN. The CDN terminates TLS and forwards to our origin over a private path. It adds the usual headers: the client's real IP and X-Forwarded-Proto: https.
2. nginx is configured to trust the CDN's real-IP header. set_real_ip_from plus real_ip_header so that logs and rate limits see the customer's address, not the CDN's. Standard.
3. The application has a trusted-proxy list. Like most frameworks, it only honours X-Forwarded-* headers from proxies it trusts, identified by the connecting IP. Correct; otherwise anyone could forge a forwarded protocol.
4. Signed URLs include the scheme. The signature covers the full URL, https://.... Also correct; a signature that ignored the scheme would be weaker.
Now compose them. nginx's real-IP module rewrites REMOTE_ADDR to the customer's IP before the application sees the request. The application checks whether REMOTE_ADDR is a trusted proxy. The customer's IP is not. So it ignores X-Forwarded-Proto. TLS was terminated at the CDN, so the connection the application sees is plain HTTP. The application therefore believes the request URL is http://.... It recomputes the signature over http://..., compares it to the one generated over https://..., and they differ. 403.
Why it only showed after login
The authentication middleware ran before the signature middleware. An anonymous click was redirected to the login page first, which is why the first thing the customer saw was a login form rather than an error. Only after a successful login did the request reach the signature check and fail. That ordering made the bug look like a session problem, and we spent a while looking in the wrong place.
The fix
Trust the proxy, not the rewritten address. Concretely: the application's trusted-proxy configuration was changed to accept the forwarded headers for requests arriving on the private origin path, since in our topology nothing else can reach the origin. With the forwarded protocol honoured, the request URL becomes https://... again and signatures verify. We also moved the signature check in front of authentication for these routes, so an invalid link fails fast with a clear message instead of a login form.
The lesson, such as it is
None of the four pieces was wrong. Each one was the documented, recommended setting. The failure was in the interaction, and specifically in the fact that nginx's real-IP rewrite happens outside the application's field of view. When you add a layer that rewrites what a downstream component uses for trust decisions, check what that component now believes about the request. "Who sent this" and "how did it arrive" are two different questions, and after a real-IP rewrite only one of them still has the original answer.