How to schedule crawls

Scheduling crawls is a 2-step process:

• Step 1: Assign your seeds a Frequency. You can select from among 9 preset crawl frequencies when you first add a seed or list of seeds to your collection. 
• Step 2: Schedule the crawl at that frequency.   

Step 1: How to assign a crawl frequency to your seed(s)

Every seed in your collection has a Crawl Frequency. After clicking the Add Seeds button in your collection's Seeds Tab, and entering your new seed URLs into the dialog box, you can select the frequency at which you want them to be crawled from the Frequency drop-down menu:

 

 

Standard crawl frequencies and their corresponding durations are:

Any crawl run at one of the above frequencies can also be resumed after it completes. 

How to edit a seed's crawl frequency 

To adjust the frequency for any of your seed sites, navigate to a collection's Seeds tab, check the box next to any seed(s) for which you intend to change the crawl frequency, and click the Edit Settings button:

 

In the resulting dialog box, select the frequency at which you want to crawl the selected seed(s) and click Apply to save your change(s):

 

 

 

Note: If you change your seed's frequency to one that already has a regularly scheduled crawl set up, the system will automatically include that seed in the next scheduled crawl. If the frequency to which you assign your seed does not already have a regularly scheduled crawl, you will need to schedule a crawl at that frequency in order for it to run.

 

Step 2: How to schedule your crawl

From the Crawls tab, select Scheduled Crawls, and then click the Schedule Crawl button:

Select Crawl Now to start the crawl immediately, or select the date on which you want the crawl to start:

After your crawl has begun it will run automatically at your designated frequency.

To change the default duration or add data or document limits for a scheduled crawl, click the Edit Limits button:

You can assign/edit any limits and specify the duration for this crawl in the resulting dialog box:

 

How to unschedule a crawl

To unschedule a crawl that you previously scheduled:

• From the Crawls tab, select Scheduled Crawls.
• Select the Unschedule button.

Once a crawl frequency is unscheduled, it will remain visible in the Crawl Schedule list, but the Unschedule button will be greyed out and the Next Crawl column will be blank. You will still have access to the Edit Limits and Schedule Crawl buttons, if you wish to reschedule the crawl in the future.

Outcome

Once you have assigned your seed(s) a frequency and scheduled your crawl to run at a chosen time,  it will run automatically moving forward at your designated frequency. It will continue to run unless the crawl is manually unscheduled. 

It is best practice to review your crawl reports for scheduled crawls periodically to ensure that they continue to be scoped appropriately, as production crawls can not be deleted.

Related Content

How to run, monitor, and save a test crawl

How to manually start test and one-time crawls

How to crawl new seeds immediately with InstaCrawl