Scoping Your Crawls
- How Archive-It crawlers determine scope
- Modify your collection or seed scope
- How to add Seed level scoping rules to multiple seeds at once
- Limit your crawl
- Expand the scope of your crawl
- Robots.txt exclusions and how they can impact your web archives
- Limit your crawl to archive only PDFs
- Modify crawl scope with a Regular Expression
- Identify and avoid crawler traps