Determining Crawl Scope
- How Archive-It crawlers determine scope
- Modify your collection or seed scope
- Adding Seed Level scoping rules in bulk
- Limit your crawl
- Expand the scope of your crawl
- Avoid robots.txt exclusions
- Limit your crawl to archive only PDFs
- Modify crawl scope with a Regular Expression
- Identify and avoid crawler traps