Overview
You can set and adjust time limits for one-time and recurring crawls based on your needs.
Time limits for one-time crawls - Test or Production
1 or 3 day time limits are recommended for all crawls on new seeds
There are no hard and fast rules for determining how much time a crawler will need to capture a seed or group of seeds. You have the option to select time limits that range between 10 minutes to 7 days, but we recommend a 1-3 day test crawl to start. Based on the results of your 1-3 day test crawl, you can then increase or decrease the time limit as necessary for subsequent crawls.
Time limits determine the maximum time a crawl will run. It is possible for a crawl to finish before it reaches the time limit.
If a production crawl finishes due to a time limit, it can also be resumed within 7 days after it finishes. Test crawls cannot be resumed.
Time limits for recurring crawls
Recurring crawl limits can be adjusted using Edit Limits on the Crawl Frequency tab. Recurring crawls are set to the following default time limits:
|
Frequency |
Default Time Limit |
Description and time limit extension options |
|---|---|---|
|
Every 12 Hours |
12 hours |
These twice-daily crawls repeat every 12 hours. We strongly recommend running a test crawl before scheduling your seeds to this frequency, as these crawls can quickly use up large amounts of your data budget. |
|
Daily |
24 hours |
Daily crawls repeat every day and run up to 24 hours. We strongly recommend running a test crawl before scheduling your seeds to this frequency, as these crawls can quickly use up large amounts of your data budget. |
|
Weekly |
3 Days |
Weekly crawls repeat every week and run up to 3 days (72 hours) by default, but can be extended to run 5 or 7 days. |
|
Monthly |
3 Days |
Monthly Crawls repeat every month and run up to 3 days (72 hours) by default, but can be extended to run 5 or 7 days. |
|
Bimonthly |
3 Days |
Bi-Monthly Crawls repeat every two months and run for up to 3 days (72 hours) by default, but can be extended to run 5 or 7 days. |
|
Quarterly |
3 Days |
Quarterly crawls repeat every three months and run for up to 3 days (72 hours) by default, but can be extended to run 5 or 7 days. |
|
Semiannual |
5 Days |
Semiannual crawls repeat every 6 months and run for up to 5 days by default, but can be extended to run 7 days. |
|
Annual |
5 Days |
Annual crawls repeat every 12 months and run for a maximum of 5 days by default, but can be extended to run 7 days. |
|
One-Time |
3 Days |
A One-Time crawl runs exactly once and is not scheduled to repeat. These crawls run up to 3 days (72 hours) by default, but can be extended to run 5 or 7 days. |
If a scheduled crawl finishes due to a time limit, any crawl run at one of the above frequencies can also be resumed within 7 days after it finishes.
Comments
0 comments
Please sign in to leave a comment.