To find out how large a website or collection is, we always recommend that you run a test crawl. To run a test crawl, select the seeds you want to test from the list in your collection's "Seeds" tab, click the "Run Crawl" button located at the top left-hand corner of the tab, and, most importantly, remember to toggle the radio button in the ensuing dialog box to specify that this crawl's type is: "Test Crawl."
The resulting crawl will not automatically save any data, but will generate all the normal reports, crawl statistics, and enable browsing in Wayback. You can then analyze your data by seed/collection and make any necessary adjustments or save the test crawl if you are happy with the results. When you are ready to run a production crawl to capture data, change the frequency of your seeds as necessary.
Comments
0 comments
Please sign in to leave a comment.