We just used Brozzler to crawl a site with a lot of dynamic content. It worked well, and did ultimately finish (not due to time limit). We have over 250,000 queued documents. I spot checked them in proxy mode, and the URLs appear to be valid links and also seem to be properly archived (although it is possible, that those in the archive were captured in earlier non-Brozzler crawls). I'm somewhat new to Archive-it, and am just wondering why the large number of queued documents may remain with this finished crawl.
Please sign in to leave a comment.