Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

There are at least tens of millions of web sites. This site ranks 3500 on Alexa and 12k on Compete. Google knows that and prioritizes its crawlers accordingly.

Maintaining a fresh index of the top 100k sites is where search engines get their best bang for the buck.



Doesn't Google use Pagerank for crawling policy/priority?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: