What The DDG? 2022-06-08

Search engines like Google and DuckDuckGo are supposed to "index" web sites and then when you search for stuff they can quickly examine their indexes to barf up some answers for you. In order to index pages they basically connect to web sites like a browser would. Well, close to it anyways. Web servers [like mine] usually keep logs to see who has been visiting their web sites. Tonight I came home to find this in my logs:

2022-06-08T08:30:33-04:00 20.185.79.15 [snork.ca:80] "GET /robots.txt HTTP/1.1" 301 "http://doramodels.jimdofree.com/" "DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)"
2022-06-08T08:30:34-04:00 20.185.79.15 [snork.ca:443] "GET /robots.txt HTTP/1.1" 200 "http://doramodels.jimdofree.com/" "DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)"
2022-06-08T08:30:34-04:00 20.185.79.15 [snork.ca:80] "GET / HTTP/1.1" 301 "http://snork.ca/robots.txt" "DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)"
2022-06-08T08:30:34-04:00 20.185.79.15 [snork.ca:443] "GET / HTTP/1.1" 200 "http://snork.ca/robots.txt" "DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)"
2022-06-08T08:31:34-04:00 20.185.79.15 [snork.ca:80] "GET /posts/2022-06-01-more-fediverse/ HTTP/1.1" 301 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:31:34-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2022-06-01-more-fediverse/ HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:31:34-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "https://snork.ca/posts/2022-06-01-more-fediverse/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:31:35-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2022-06-01-more-fediverse/castrospeaks.jpg HTTP/1.1" 200 "https://snork.ca/posts/2022-06-01-more-fediverse/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:31:36-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2022-06-01-more-fediverse/ HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:31:36-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "https://snork.ca/posts/2022-06-01-more-fediverse/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:31:36-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2022-06-01-more-fediverse/castrospeaks.jpg HTTP/1.1" 200 "https://snork.ca/posts/2022-06-01-more-fediverse/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:32:16-04:00 20.185.79.15 [snork.ca:80] "GET /pages/mans/ HTTP/1.1" 301 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:32:16-04:00 20.185.79.15 [snork.ca:443] "GET /pages/mans/ HTTP/1.1" 404 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:32:17-04:00 20.185.79.15 [snork.ca:443] "GET /pages/mans/ HTTP/1.1" 404 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:08-04:00 20.185.79.15 [snork.ca:80] "GET /gitsucks/ HTTP/1.1" 301 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:08-04:00 20.185.79.15 [snork.ca:443] "GET /gitsucks/ HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:09-04:00 20.185.79.15 [snork.ca:443] "GET /gitsucks/ HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:42-04:00 20.185.79.15 [snork.ca:80] "GET /light/posts/2016-05-29-we-had-a-bad-storm/ HTTP/1.1" 301 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:43-04:00 20.185.79.15 [snork.ca:443] "GET /light/posts/2016-05-29-we-had-a-bad-storm/ HTTP/1.1" 301 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:43-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2016-05-29-we-had-a-bad-storm/ HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:43-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "https://snork.ca/posts/2016-05-29-we-had-a-bad-storm/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:43-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2016-05-29-we-had-a-bad-storm/stormcam.jpg HTTP/1.1" 200 "https://snork.ca/posts/2016-05-29-we-had-a-bad-storm/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:43-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2016-05-29-we-had-a-bad-storm/bustedplane_th.jpg HTTP/1.1" 200 "https://snork.ca/posts/2016-05-29-we-had-a-bad-storm/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:45-04:00 20.185.79.15 [snork.ca:443] "GET /light/posts/2016-05-29-we-had-a-bad-storm/ HTTP/1.1" 301 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:45-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2016-05-29-we-had-a-bad-storm/ HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:45-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "https://snork.ca/posts/2016-05-29-we-had-a-bad-storm/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:46-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2016-05-29-we-had-a-bad-storm/stormcam.jpg HTTP/1.1" 200 "https://snork.ca/posts/2016-05-29-we-had-a-bad-storm/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:33:46-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2016-05-29-we-had-a-bad-storm/bustedplane_th.jpg HTTP/1.1" 200 "https://snork.ca/posts/2016-05-29-we-had-a-bad-storm/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:35:29-04:00 20.185.79.15 [snork.ca:80] "GET /feed.xml HTTP/1.1" 301 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:35:29-04:00 20.185.79.15 [snork.ca:443] "GET /feed.xml HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:35:30-04:00 20.185.79.15 [snork.ca:443] "GET /feed.xml HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:36:18-04:00 20.185.79.15 [snork.ca:80] "GET /style.css HTTP/1.1" 301 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:36:19-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:36:19-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:36:59-04:00 20.185.79.15 [snork.ca:80] "GET /about.shtml HTTP/1.1" 301 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:00-04:00 20.185.79.15 [snork.ca:443] "GET /about.shtml HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:00-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "https://snork.ca/about.shtml" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:00-04:00 20.185.79.15 [snork.ca:443] "GET /inc/evilsnorkca_360.jpg HTTP/1.1" 200 "https://snork.ca/about.shtml" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:00-04:00 20.185.79.15 [snork.ca:443] "GET /inc/thumbsup.jpg HTTP/1.1" 200 "https://snork.ca/about.shtml" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:01-04:00 20.185.79.15 [snork.ca:443] "GET /about.shtml HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:01-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "https://snork.ca/about.shtml" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:01-04:00 20.185.79.15 [snork.ca:443] "GET /inc/evilsnorkca_360.jpg HTTP/1.1" 200 "https://snork.ca/about.shtml" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:01-04:00 20.185.79.15 [snork.ca:443] "GET /inc/thumbsup.jpg HTTP/1.1" 200 "https://snork.ca/about.shtml" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:44-04:00 20.185.79.15 [snork.ca:80] "GET / HTTP/1.1" 301 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:45-04:00 20.185.79.15 [snork.ca:443] "GET / HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:45-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "https://snork.ca/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:46-04:00 20.185.79.15 [snork.ca:443] "GET / HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:37:46-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "https://snork.ca/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:38:47-04:00 20.185.79.15 [snork.ca:80] "GET /posts/2022-05-27-activitypub-fail/ HTTP/1.1" 301 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:38:47-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2022-05-27-activitypub-fail/ HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:38:47-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "https://snork.ca/posts/2022-05-27-activitypub-fail/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:38:48-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2022-05-27-activitypub-fail/20220208_094409.jpg HTTP/1.1" 200 "https://snork.ca/posts/2022-05-27-activitypub-fail/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:38:49-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2022-05-27-activitypub-fail/ HTTP/1.1" 200 "-" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:38:49-04:00 20.185.79.15 [snork.ca:443] "GET /style.css HTTP/1.1" 200 "https://snork.ca/posts/2022-05-27-activitypub-fail/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"
2022-06-08T08:38:49-04:00 20.185.79.15 [snork.ca:443] "GET /posts/2022-05-27-activitypub-fail/20220208_094409.jpg HTTP/1.1" 200 "https://snork.ca/posts/2022-05-27-activitypub-fail/" "'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'"

So, it would appear that DDG spent about eight minutes indexing my site tonight. In that eight minutes it:

• Downloaded my robots.txt 4 times
• Lied about being referred to my site from http://doramodels.jimdofree.com/
• Ignored my https redirection 11 times
• Looked at three posts (multiple times each)
• Looked at the root of my site (list of posts) three times
• Tried to download /pages/mans/ three times even though it has been gone a long time
• Downloaded /gitsucks/ three times
• Downloaded my stylesheet 13 times
• Hit /light/posts/2016-05-29-we-had-a-bad-storm/ 4 times even the light theme is long gone
• Downloaded my feed a couple times in the same second
• Hit my about page a couple of times

... and all from some crappy Microsoft [I assume Azure] IP address with no rDNS. I guess looking legitimate isn't so important. This looks more like the traffic of a script kiddie trying to find stuff to exploit. It isn't a surprise that I am not a fan of DDG, and this kind of slackass effort doesn't help. Lately I have been using StartPage as my search engine of choice, but it has plenty of problems too... site: syntax, and the constant hassle of being accused of hammering their site when every search I do is hand typed for example. Sucks almost as bad as a sunburn from crawling around trying to get a mowing deck attached to a tractor.

Booo! :-(