Nathan Grigg

Cloudflare and Python

In a recent post, I made a bit of an eye-roll comment about how Letterboxd rejects requests based on their user-agent header. Then today, it happened when I was making a request to my own website. I know (or thought) that I don’t do anything so ridiculous, so I set out to investigate.

I host my site these days on a Linode instance using nginx, which I was pretty sure doesn’t do anything strange based on user-agent headers. Looking at my logs showed that my request wasn’t making it to ngnix.

When Google Domains shut down, I migrated my DNS hosting to Cloudflare, so that was the next possible culprit. Cloudflare offers traffic proxying, so that all of your traffic goes to their servers, and they send requests to your servers. This isn’t something I was looking for when I switched, but they make it easy or maybe even the default.

Inspecting the actual response that I was getting in Python, I saw the message error code: 1010. (For Letterboxd, I had seen the 403 response and hadn’t looked deeper into it, but it was probably the same thing.) Some searching confirmed that this was indeed coming from Cloudflare.

If you go to “Security > Settings” in the Cloudflare dashboard, you can turn off “Browser Integrity Check,” which fixes the problem. Or just turn off proxying entirely, although that takes a little longer to take effect, because the DNS settings need time to propagate.