Aug OCT Dec 18 2005 2006 2008 37 captures 23 Jan 2002 - 17 Jun 2013

About this capture

COLLECTED BY

Organization: Alexa Crawls

Starting in 1996, Alexa Internet has been donating their crawl data to the Internet Archive. Flowing in every day, these data are added to the Wayback Machine after an embargo period.

Collection: 32_crawl

this data is currently not publicly accessible.

SRC Research Report 173

High-Performance Web Crawling

Marc Najork and Allan Heydon

High-performance web crawlers are an important component of many web services. For example, search services use web crawlers to populate their indices, comparison shopping engines use them to collect product and pricing information from online vendors, and the Internet Archive uses them to record a history of the Internet. The design of a high-performance crawler poses many challenges, both technical and social, primarily due to the large scale of the web. The web crawler must be able to download pages at a very high rate, yet it must not overwhelm any particular web server. Moreover, it must maintain data structures far too large to fit in main memory, yet it must be able to access and update them efficiently. This chapter describes our experience building and operating such a high-performance crawler.

Back to the SRC Research Reports main page.

• PostScript compressed with UNIX compress -- 264 Kbytes
  
• PostScript compressed with GNU compress (gzip) -- 136 Kbytes
  
• PostScript compressed with ZIP -- 136 Kbytes
  
• PostScript -- 656 Kbytes
  
• Portable Document Format (PDF) -- 96 Kbytes