On this page:

• How to modify scope from your crawl report
• Individual host
• Multiple hosts
• Patch crawl blocked URLs

How to modify scope from your crawl report

You may edit scoping rules for specific hosts individually or in batch from your crawl's Hosts report. Your changes will apply to your next crawl.  

Individual host

Your may add or edit the following rules for hosts on an individual host-by-host basis:

• Block/Unblock a host
• Create/Modify a document limit
• Ignore/Obey robots.txt
• Block URLs if...

Existing rules in place for the given host will also be displayed.

 

Begin by selecting your host from the list by checking the box next to it, then click the "Edit Rules" button:

 

 

 

 

To add or edit rules:

1. Select the desired rule from the drop down menu
2. Toggle the blue slider to turn the rule on/off
3. Add Rule
4. Commit Changes

 

 

 

Multiple hosts

Editing host level scoping rules for multiple hosts at the same time is also possible. This includes the ability to create or edit the following rules:

• Block/unblock a host 
• Create/Modify a document limit 
• Ignore/Obey robots.txt   

 

Check the boxes next to multiple hosts and click the "Edit Rules" button to toggle rules applied to all selected hosts:

 

 

Patch crawl blocked URLs

You may also patch crawl any URLs blocked by robots.txt files directly from the Hosts report of production and saved test crawls. It is not possible to patch crawl unsaved test crawls.