Why compare crawls?

Comparing different crawls can be a very useful way to evaluate how your crawls for a specific seed or group of seeds change over time, or to evaluate the effectiveness of any new scoping rules that you may have added to a collection in-between crawling periods.

How to do it

To compare the vital information between two crawls, begin by finding the crawls listed among your reports for completed crawls. Use the checkboxes to the left side of each crawl in order to indicate which crawls you would like to compare:

By default, the first crawl you check will go on the left side of the comparison, and the second crawl will go on the right. You can easily switch the order of the two crawls or adjust your selections as needed before clicking Compare Crawls:

The ensuing "Compare Crawls report" provides an overview of the major components of each crawl and the differences between them:

From this view, you may compare the number of total and new documents, volume of total and new data, number of hosts, number of blocked pages, crawling frequency and duration, and any scoping rules in place between the crawls. Click on either Crawl # at the top of the comparison to see that crawl report in full.