Guide for new Archive-It users

Welcome to Archive-It!  This page is designed to give you an overview of how web archiving works with Archive-It, and is a curated collection of videos and articles from our Help Center.   It is highly recommended to work through the trainings on this page before jumping into your Archive-It account.  Further information can be found elsewhere in the Help Center, and can be helpful on your web archiving journey, but is not integral to starting out.

• Getting Started 
• What is web archiving?
• Navigating Archive-It - This video will help you get acquainted with the Archive-It web application, pointing out where features are located and why you might want to use them.
  

Something to keep in mind is that web archiving is not linear, but a process, so you can move through these stages as you need, even repeating some if necessary. 

• Collections
• Terms to know:
  • Collection - A group of archived web documents curated around a common theme, topic, or domain.
  • Seed - An item in Archive-It with a unique ID number. The Seed URL is the starting point for crawlers, and as well as an access point to archived content.
  • Document - Any file with a unique URL - html, image, PDF, video, etc.  These are designated separately from seeds, although seeds can be documents as well
• How seeds, documents, and collections work together
• Create and manage a collection
• Select Seed URLs
• Known web archiving challenges - While Archive-It continuously works towards improvements, some websites are not created in a way that is "archive-friendly" and can be difficult to capture or replay in their entirety. Knowing these limitations up front can help you set your expectations. 
• Metadata - there are multiple ways and places to add metadata.  Explore these options here.
  
  
• Scoping - Each Archive-It crawl has a default set of bounds to prevent the crawls from capturing the entire web.  There are multiple ways to adjust scope to capture/not capture content.
• Some sites have automatic scoping rules applied for your convenience.  
• Pre-crawl Scoping - Make sure your seeds are set up correctly before you start your crawls. In this video you'll learn tips for selecting, formatting, and administering your seed URLs before you run a crawl to help capture the data you're looking for.
• Scoping guidance for specific types of sites - Some platforms need specific scoping rules for full capture and replay.  Learn about our most up to date guidance for archiving popular social media services and other commonly used platforms.
  
  
• Crawling - Learn about the different ways crawl content.  Please note that this section includes important information on how data is added to your account.  
• Types of crawls - There are different types of crawls to best support the Archive-It workflow.  For all of these crawls, it's important to note that only New Data will be added to your account, due to our data deduplication feature.  
  • Test Crawls - Adding new seeds or scoping rules? You'll want to make sure you run a test crawl first! This video will give you tips on how and why to run test crawls in your collections.
  • One time production crawls - The alternative to a test crawl is a production crawl.  These permanent crawls automatically add data to your data budget and can not be deleted.  
  • Scheduled crawls - Once you have run a test crawl, you’re ready to schedule crawls to run on a schedule that you set.  Please note that scheduled crawls will automatically add the data to your budget and can not be deleted.  
  • PDF Only Crawls - Learn how to run a PDF Only crawl and how to access the archived PDFs
• Crawling technologies - Archive-It has different crawling technologies that can (and should!) be used to capture different kinds of content
  • What is Brozzler, and when to use it - Brozzler is our newest crawling technology, built at the Internet Archive.
• It’s important to manage your data budget as you crawl.  Only unsaved test crawls can be deleted; no other kind of crawl can be deleted.  
  
  
• Reviewing & Quality Assurance
• Getting the most from your post crawl reports - So you've run a crawl... now what? This video walks through each report to provide detail on why each post-crawl report is necessary, and the information you can glean from them.
• Understanding your Hosts Report - The Host report is a wealth of information! Find out all of the different ways you can use it to identify crawler traps, block hosts, add data limits, run patch crawls, and more.
• Quality Assurance - What can you do if your archived websites don't look quite right in Wayback?  Following these steps may help you improve the capture and replay of your Wayback pages.
  
  
• Access
• Control access to your web archive - You can control access to your web archive so that patrons, the general public, and/or your Archive-It users can or cannot view specific content that you designate.
• Public content/ archive-it.org
• Inside your account
• Other options

Now that you have an overview of Archive-It, we encourage you to jump in!  After you’ve run some test crawls, you can sign up for Office Hours and a Web Archivist would be happy to chat with you. 

And remember- web archiving is a learning experience! As you learn more about the sites that you collect, you can find and share tips here in the User Guide or Community Forum.