Proxy tracking updates: Cookies, Caching, and X-Forwarded-For

We released proxy tracking back in May, and it’s been a hit. But there were a few issues brought to our attention over the months since then, which we spent the last week addressing during Portland’s latest Icepocalypse.

You will need to reinstall a fresh copy of your website’s backend proxy code to get these updates. These updates are optional, but recommended. This is just an unfortunate reality with this kind of tracking. We don’t anticipate these kinds of changes very often, and we always strive for backwards compatibility with any kind of update to code that our users install on their websites, as is the case here.

Here’s what’s changed:

First party cookie filtering

From a web browser’s perspective, proxy tracking requests are first party, which means the browser will send all of your website’s first party cookies with every request. This was obviously never our intention, and these extra cookies have always been completely ignored by our tracking servers.

Now, PHP and Nginx proxy tracking methods will override the default cookie header, and only send service-related first party cookies to our tracking servers. We’d like to offer this with Apache and Caddy methods as well, but we are not familiar enough with their syntax to do it confidently. Most people are using PHP or Nginx so far anyways, so that was our priority. If any of you are intimately familiar with Apache or Caddy syntax, we’d love your help updating them.

IP addresses and X-Forwarded-For headers

The previous default was to ignore XFF headers and only forward the connecting IP to our tracking servers. But each method also had an embedded line of code, commented out, that you could uncomment to enable XFF, if desired. Because XFF can be spoofed, we wanted to play it safe with this default.

However, with many websites these days being behind services like Cloudflare, the connecting IP is often different from the visitor’s IP. Unless you had configured “trusted proxies” properly on your web server, the only way to get the visitor’s real IP was from the XFF header.

For this reason, we have reversed it so now XFF is trusted by default, but can still easily be disabled by commenting out or changing a line or two in the proxy code.

If your server has “trusted proxies” configured (which you should absolutely do if possible), that should deal with XFF automatically so you won’t need to worry about it. Nginx allows you to also update the server’s internal $remote_addr variable based on XFF or another header from a trusted proxy, which is what we do with our own servers. If you do this though, you should disable XFF in the proxy code – especially if the header you’re using is something other than XFF.

Note: Caddy requires “trusted proxies” to be configured for it to forward along an incoming XFF, so distrusting XFF is still the default there.

PHP caching

This was just an accidental oversight on our end, but here’s what could happen: if the tracking code cache was expired and the next visitor to request the file was a bot or IP address with a bad reputation with Cloudflare, CF would serve a block or challenge page instead of the file. And that’s what would get cached by PHP. This challenge page would then be served to all visitors for the next 24 hours, instead of the normal tracking code. They wouldn’t actually see the challenge page, since it’s a background request. Instead, because the normal code wasn’t being loaded, they just wouldn’t get logged.

Now, the PHP method will now verify an HTTP “200” response code before caching the javascript tracking code on your server.

The cache path was also updated to use sys_get_temp_dir() to ensure we can always write to a cache file. Previously, it was just trying to cache in the current directory, which was hit or miss.

While researching the PHP issue, we also realized that our Cloudflare security settings were perhaps too aggressive. We wanted to lessen false positives, which we confirmed were happening. After extensive testing and log analysis, we found about 10 CF managed rules that were responsible for almost all false positives. Once those were turned off, the number of hits blocked or challenged by CF dropped by about 90%.

These were all the major things that mostly needed a backend code update to address, so we grouped them all together in one release. Again, we hope these kinds of updates are uncommon and will always try to maintain backwards compatibility.

Let us know how proxy tracking is working for you, and what else you’d like to see!