Why modify scope or run a patch crawl from the report

Your crawl's report can help you decide how your crawl might be better scoped to capture more/better content in the future, and to identify select missing elements for immediate inclusion with your completed crawl. In previous releases, these processes were divided between modifying your collection's crawl scope and running patch crawls from Wayback QA in disparate sections of our web application. Since the release of the 5.0 version of our web application, however, you may do both directly from your reports.

How to do it

Modify scope 

You may edit scoping rules for specific hosts individually or in batch from your crawl's Hosts report. Your changes will apply to your next crawl.  

Individual host

Your may add or edit the following rules for hosts on an individual host-by-host basis:

• Block/Unblock a host
• Create/Modify a document limit
• Ignore/Obey robots.txt
• Block URLs if...

 

Existing rules in place for the given host will also be displayed.

 

Begin by selecting your host from the list by checking the box next to it, then click the "Edit Rules" button:

 

 

 

 

To add or edit rules:

1. Select the desired rule from the drop down menu
2. Toggle the blue slider to turn the rule on/off
3. Add Rule
4. Commit Changes

 

 

 

Multiple hosts 

Editing host level scoping rules for multiple hosts at the same time is also possible. This includes the ability to create or edit the following rules:

• Block/unblock a host 
• Create/Modify a document limit 
• Ignore/Obey robots.txt   

 

Check the boxes next to multiple hosts and click the "Edit Rules" button to toggle rules applied to all selected hosts:

 

 

Patch Crawl Blocked URLs 

You may also patch crawl any URLs blocked by robots.txt files directly from the Hosts report.