Varnish Debugging
I love Varnish. As far as the world of HTTP goes, in environments where you can cache it’s one of the easiest quick wins there is. However this does somewhat come at a price to debugging. Often you’ll sit back and your black box of magic doesn’t quite produce the result you were expect from it.
Normally I end up adding a sleuth of headers cache status headers to help identify where things are going wrong. The first and most useful of these to add is X-Grace to show if Varnish thinks it has served a graced object. Graced objects are those that are outside of their TTL but are delivered anyway as no backend was available to service the request.
To start with you need to add a header in VCL receive, this ensures that all requests get the header. If this wasn’t added and the request was made, if it was a cache miss no header would be appended. Each sub step from here this header gets rewritten.
sub vcl_recv {
# Set default grace header.
set req.http.X-Grace = "First Hit";
...
return (hash);
}
You could probably add this in vcl_miss if you wanted to, there will be a tiny overhead in adding it to all requests in receive, but I’ve seen no effect on loads of up to 10k qps.
To check the cache status of the object, we need to look in via VCL hit.
sub vcl_hit {
# Deliver if fresh.
if (obj.ttl >= 0s) {
set req.http.X-Grace = "Fresh Hit";
return (deliver);
}
# Check if backend is healthy.
if (std.healthy(req.backend_hint)) {
if (obj.ttl + 20s > 0s) {
set req.http.X-Grace = "Stale Hit";
return (deliver);
}
else {
return (fetch);
}
} else {
if (obj.ttl + obj.grace > 0s) {
set req.http.X-Grace = "Rotten Hit";
return (deliver);
} else {
return (fetch);
}
}
return (deliver);
}
This is fairly simple, one if statement followed by an if else statement. Initially we check if the object, if it is, we deliver it. Else, we check to see if the backend is healthy. If it is, and object is just stale, we return the object and fetch another (hooray for async fetches in v4). If the backend is not healthy, we try to deliver a stale object rather than nothing. If nothing is available we try to fetch, and will eventually serve a 503.
Finally in deliver…
sub vcl_deliver {
# Copy the header back over to the response.
set resp.http.X-Grace = req.http.X-Grace;
}
If any of this unclear, theres an official Varnish article available here.