Overview
You can monitor any crawl while it is running to view real-time updates on URLs and data crawled. You can also stop a crawl and edit the limits of your running crawl.
On this page:
- Where to find your currently running crawls
- View reports
- See a realtime graph and a list of recently crawled URLs
- Stop a currently running crawl
- Edit the limits of a currently running crawl
Where to find your currently running crawls
To find a list of your currently running crawls:
- From the top navigation bar, select Crawls, and then select Current Crawls. You can also access crawls for a specific collection by selecting the Crawls tab and then the Current Crawls subtab within a collection.
- Select the hyperlinked Crawl ID of the crawl that you want to monitor.
From the full crawl report, you can:
View reports
A full set of reports of documents and data collected so far -- including Seeds, Hosts, and File Types reports -- are available for currently running crawls. For an overview of crawl reports and what's inside the reports, see How to read your crawl's report.
See a realtime graph and a list of recently crawled URLs
In the Crawl Overview tab, expand the Realtime Graph to illustrate how a crawl has grown over time and the proportion of new to duplicate data that it has crawled.
You can also view list of URLs currently being crawled in the Recently Crawled pane. This information can be useful if you are concerned that the crawler has hit a trap and is no longer capturing valid URLs.
Stop a currently running crawl
To manually stop a currently running crawl, go to the Crawl Overview tab, and then select Stop Crawl.
Note: It may take a few minutes for the crawl to stop and fully process its reports. The blue banner at the top of the page will inform you as these steps are completed.
Edit the limits of a currently running crawl
To edit the document, data, and/or time limit of a currently running crawl:
- From the Crawl Overview tab, select Edit Limits. Edited limits need to be greater than the amount already captured. For example, if a crawl has already captured 10,000 documents, the document limit added to the crawl will need to be greater than 10,000.
- When finished editing, select Modify Limits.
If you want to extend the time of an already completed crawl, see resuming completed crawls.
Comments
0 comments
Please sign in to leave a comment.