NGINX truncating large responses? Check your proxy_temp ownership
The page loads, the status is 200, and the HTML just stops. The cause was one line in NGINX's error log.
A page on one of our environments started cutting off partway down. The status code was 200, there was no error page, and the HTML simply stopped in the middle of a section.
The browser's network panel made the size of the problem obvious. The truncated response came back at about 471 KB. The same page on production was about 1.13 MB.
The tell: it only happens to big responses
If small responses are fine and large ones get cut off, think about buffering before you think about application code.
With proxy_buffering on (the default), NGINX reads the upstream response into memory buffers (proxy_buffers, proxy_buffer_size). When a response is bigger than those buffers can hold, NGINX spools the overflow to a temporary file under proxy_temp_path and streams it to the client from there.
If NGINX can't open that temporary file, it can't hold the rest of the response. The client has already received the headers and part of the body, so you get a truncated page with a 200 status instead of an error.
The one line that explained it
The answer was in NGINX's error log:
[crit] ... open() "/usr/local/openresty/nginx/proxy_temp/..." failed (13: Permission denied) while reading upstream
NGINX tried to create a temp file and the kernel refused.
Why the permissions were wrong
On that box, the NGINX user directive had been changed at some point. The workers there ran as lucee, while production ran them as nobody. The hashed subdirectories inside proxy_temp were created back when the workers still ran as nobody, and they were still 0700 nobody:nobody.
That's an easy one to miss. At startup NGINX makes sure the top-level temp directories belong to the worker user, but it doesn't walk the hashed subdirectories (levels=1:2) already inside them. So the top-level directory looks correct, and the subdirectories one level down are locked to the old user.
# What user are the workers running as?
ps -o user=,cmd= -C nginx | grep worker
# Who owns the temp tree, all the way down?
find /usr/local/openresty/nginx/proxy_temp -maxdepth 2 -printf '%u:%g %m %p\n' | head
The fix
Give the temp trees to the worker user and reload:
WORKER_USER=lucee
for d in proxy_temp client_body_temp fastcgi_temp uwsgi_temp scgi_temp; do
[ -d "/usr/local/openresty/nginx/$d" ] && chown -R "$WORKER_USER:" "/usr/local/openresty/nginx/$d"
done
openresty -t && systemctl reload openresty
proxy_temp was the one that bit here. The others fail the same way, though. client_body_temp breaks large uploads, and fastcgi_temp breaks large PHP responses, so fix them all at once.
Adjust the paths for a stock NGINX package (usually /var/cache/nginx/...). If SELinux is enforcing and NGINX runs confined, also check the directories' labels with ls -Z.
Keeping it from coming back
- Treat the
userdirective and the temp-directory ownership as one change. If your config management setsuser, have the same role set ownership on the temp paths. - Diff environments, not just files. Staging and production had the same templates but a different worker user, and that drift is the whole bug. Worth a check across the fleet:
ansible web -m shell -a "ps -o user= -C nginx | sort | uniq -c" - Alert on
[crit]in the NGINX error log. This one wrote a clear message on every truncated request, and nobody was watching for it.
The short version
If NGINX returns 200 but large pages are cut off, search the error log for Permission denied on a *_temp path. Then compare the worker user to the owner of the hashed subdirectories, not just the top-level directory.