snork.ca: Welcome to the brown age of computing! Mostly Cloudy, -2.5°C [W: -9°C] - Precip 3 hrs

Google Are Such Dicks 2021-06-28

If you have a web site you can create a robots.txt file for it that dictates how crawlers should behave on your site. You can tell them to never look in certain places, you can tell them to not look at your site at all, and you can tell them to only access your site every so-many-seconds. There is even a web site just for describing how a robots.txt file works. Sadly, that web site doesn't mention the crawl-delay setting... the one that tells a crawler to slow the fuck down. It is mentioned at this sleazy site, however it explicitly says that Google doesn't care about your crawl-delay setting. They'll just scrape as fast as they feel like.

Seriously, even with the vast number of sites Google scrapes, I can't possibly imagine this is a feature they need to ignore. The effort required to implement this would be trivial for an organization like Google, but they insist on being douches about it. In fact, they DO offer a way to set a scrape limit... just that it requires you setup a Google account and fuck with their wankery interface to set it. That's right, they intentionally will not respect the setting you put in your robots.txt (which has domain ownership proof built in) but they will let/force you to setup an account on their unrelated servers and which works as a nice way for them to associate your web site with your other Google activities. Twats.

Please don't use Google. Not their email services, not their search engine, not their glasses. Avoid Google when you can, because using their services like there is no other option is what is killing the other options.

2021-06-28T11:27:29-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2018-08-29-why-i-like-samorost/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:30-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2018-08-29-why-i-like-samorost/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:30-04:00 66.249.70.65 [snork.ca:443] "GET /light/posts/2018-01-27-a-dog-and-a-bug/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:31-04:00 66.249.70.95 [snork.ca:443] "GET /posts/2018-01-27-a-dog-and-a-bug/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:32-04:00 66.249.70.95 [snork.ca:443] "GET /light/posts/2018-03-31-socks5-to-vpn-gateway/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:33-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2018-03-31-socks5-to-vpn-gateway/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:36-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2014-01-08-linux-clock-codes/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:36-04:00 66.249.70.95 [snork.ca:443] "GET /posts/2014-01-08-linux-clock-codes/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:39-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2013-07-28-do-not-use-hotmail-or-yahoo/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:40-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2013-07-28-do-not-use-hotmail-or-yahoo/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:43-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2011-06-03-dd-wrt-firewall-script/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:43-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2011-06-03-dd-wrt-firewall-script/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:46-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2018-06-07-and-now-for-some-gw-ducks/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:46-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2018-06-07-and-now-for-some-gw-ducks/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:49-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2017-09-21-bell-canada-are-douchebags/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:49-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2017-09-21-bell-canada-are-douchebags/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:52-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2016-10-19-cubicool-and-your-mortality/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:52-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2016-10-19-cubicool-and-your-mortality/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:55-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2015-09-05-upcycle-crappy-gps-phone-mount/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:55-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2015-09-05-upcycle-crappy-gps-phone-mount/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:58-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2020-06-11-you-thought-i-was-exaggerating/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:27:59-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2020-06-11-you-thought-i-was-exaggerating/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:01-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2017-08-17-what-to-do-about-pc-financial-cibc/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:02-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2017-08-17-what-to-do-about-pc-financial-cibc/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:04-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2020-05-13-realnews-government-gives-a-shit/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:05-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2020-05-13-realnews-government-gives-a-shit/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:08-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2016-01-16-even-more-cloudatcost-disappointment/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:08-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2016-01-16-even-more-cloudatcost-disappointment/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:11-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2017-10-06-mailchannels-hosting-your-own-email-might-be-a-good-idea/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:11-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2017-10-06-mailchannels-hosting-your-own-email-might-be-a-good-idea/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:14-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2016-04-23-making-the-switch-from-apache-to-nginx/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:15-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2016-04-23-making-the-switch-from-apache-to-nginx/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:18-04:00 66.249.70.93 [snork.ca:443] "GET /light/posts/2017-07-22-colourizing-and-menuizing-my-xfce-ssh-sessions/ HTTP/1.1" 301 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2021-06-28T11:28:18-04:00 66.249.70.93 [snork.ca:443] "GET /posts/2017-07-22-colourizing-and-menuizing-my-xfce-ssh-sessions/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Made with Notepad++ and FastStone, hosted on Devuan with nginx, without javascript, google bullshit, CDN crap, or cookies, and powered by NK shrooms.