Scoping Your Crawls
- How Archive-It crawlers determine scope
- Scope Rules and how to use them
- Modify your collection or seed scope
- How to add Seed level scope rules to multiple seeds at once
- How to limit your crawl
- Expand the scope of your crawl
- Robots.txt exclusions and how they can impact your web archives
- Limit your crawl to only collect PDFs
- Modify crawl scope with a Regular Expression
- How to identify and avoid crawler traps