On this page:
How to modify scope from your crawl report
You can edit scoping rules for specific hosts individually or in batch from your crawl's Hosts report. Your changes will apply to your next crawl.
Individual host
Your can add or edit the following rules for hosts on an individual host-by-host basis:
- Block/Unblock a host
- Create/Modify a document limit
- Ignore/Obey robots.txt
- Block URLs if...
Existing rules in place for the given host will also be displayed.
Begin by selecting your host from the list by checking the box next to it, then click the "Edit Rules" button:

To add or edit rules:
- Select the desired rule from the drop down menu
- Click Add Rule
Multiple hosts
Editing host level scoping rules for multiple hosts at the same time is also possible. This includes the ability to create or edit the following rules:
- Block/unblock a host
- Create/Modify a document limit
- Ignore/Obey robots.txt
Check the boxes next to multiple hosts and click the "Edit Rules" button to toggle rules applied to all selected hosts:

Patch crawl blocked URLs
You may also patch crawl any URLs blocked by robots.txt files directly from the Hosts report of production and saved test crawls. It is not possible to patch crawl unsaved test crawls.
1. Select the hosts and click the "Run Patch Crawl" button
2. Check the "Ignore Robots.txt" check box (if this feature is not enabled for your account, please request it by submitting a Support ticket) and click the "Run Crawl" button:
After your patch crawl has run, you will see a medical tool box icon display for any hosts that have been patched:
Comments
0 comments
Please sign in to leave a comment.