Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Those stats look a little off, and I wonder if it's because the site in question is pretty new. For comparison here's a screenshot of the crawl information from my CloudFlare analytics panel (the same information as gathered by the OP): http://i.imgur.com/vrKlz.png

jgc.org has been around since 1997, daysoutnearme.com since April, 2012. I'm guessing that the other engines don't yet know about the site.



Ah ha, that's really interesting. My site hasn't been around very long, but it has been submitted to both Google & Bing for the same amount of time. Nice to know that it balances out over time, but it looks like Google pays more attention to new sites than Bing does. Or Google has more resources with which to crawl sites, of which I have no doubt.


My impression, based on eyeballing logs for my own low traffic, unimportant sites: Google collects and uses stats on how often pages change, how often they come up in searches, etc., and for static and/or low importance pages they scale crawling back. This frees up crawling resources to spend elsewhere, like on important, frequently updated pages, new sites, et al. I haven't looked closely in a couple of years, but the pattern used to be that new content would be crawled fairly agressively and after a while it would back off to an "appropriate" level.

Back then, other search engines took a good while to begin crawling my new stuff in earnest, and once they did their crawling pattern did not seem to change much over time between a (relatively) popular changing page vs. a lonely static page.


> I'm guessing that the other engines don't yet know about the site.

Didn't he say he submitted sitemaps?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: