Logo
Community Submit a request
Sign in
  1. Archive-It Help Center
  2. Archive-It
  3. Determining Crawl Scope

Determining Crawl Scope

Follow New articles New articles and comments
  • How Archive-It crawlers determine scope
  • Modify your collection or seed scope
  • Adding Seed Level scoping rules in bulk
  • Limit your crawl
  • Expand the scope of your crawl
  • Avoid robots.txt exclusions
  • Limit your crawl to archive only PDFs
  • Modify crawl scope with a Regular Expression
  • Identify and avoid crawler traps
Archive-It Help Center