Overview
You can set and adjust time limits for one-time and scheduled crawls based on your needs.
Time limits for one-time crawls - Test or Production
1 or 3 day time limits are recommended for all crawls on new seeds.
There are no hard and fast rules for determining how much time a crawler will need to capture a seed or group of seeds. You have the option to select time limits that range between 10 minutes to 30 days, but we recommend a 1 or 3 day test crawl to start. Based on the results of your 1 or 3 day test crawl, you can then increase or decrease the time limit as necessary for subsequent crawls.
Time limits determine the maximum time a crawl will run. It is possible for a crawl to finish before it reaches the time limit.
If a crawl finishes due to a time limit, it can be resumed within 7 days after it finishes.
Default time limits for recurring crawls
Recurring crawl limits can be adjusted using Edit Limits on your collection's Crawl Frequency tab. Recurring crawls are set to the following default time limits:
|
Frequency |
Default Time Limit |
Description |
|---|---|---|
|
Twice daily (every 12 hours) |
12 hours |
Twice-daily crawls repeat every 12 hours. We strongly recommend running a test crawl before scheduling your seeds to this frequency, as these crawls can quickly use up large amounts of your data budget. |
|
Daily |
24 hours |
Daily crawls repeat every day and run up to 24 hours. We strongly recommend running a test crawl before scheduling your seeds to this frequency, as these crawls can quickly use up large amounts of your data budget. |
|
Weekly |
3 Days |
Weekly crawls repeat every week and have a time limit of 3 days by default, which can be edited. |
|
Monthly |
3 Days |
Monthly Crawls repeat every month and have a time limit of 3 days by default, which can be edited. |
|
Bimonthly |
3 Days |
Bi-Monthly Crawls repeat every two months and have a time limit of 3 days by default, which can be edited. |
|
Quarterly |
3 Days |
Quarterly crawls repeat every three months and have a time limit of 3 days by default, which can be edited. |
|
Semiannual |
5 Days |
Semiannual crawls repeat every 6 months and have a time limit of 5 days by default, which can be edited. |
|
Annual |
5 Days |
Annual crawls repeat every 12 months and have a time limit of 5 days by default, which can be edited. |
|
One-Time |
3 Days |
A One-Time crawl runs exactly once and is not scheduled to repeat. These crawls have a time limit of 3 days by default, which can be edited. |
If a scheduled crawl finishes due to a time limit, it can be resumed within 7 days after it finishes.
Comments
0 comments
Please sign in to leave a comment.