Archive-It's software enables partners to capture and display content hosted by popular social media services and many other commonly used platforms. These services frequently update and change the ways that they serve content, so use the guidance on this page to avoid running into any problems.
Before you crawl
Be specific with your seed URLs.
Be as specific as possible when choosing your seed URL; in other words, add only the page that you want to archive as the seed.
- DO NOT use the whole site as a seed: www.facebook.com or www.twitter.com
- DO use: http://twitter.com/internetarchive/
Double-check your seeds!
Do you need an ending / (slash) ? Please be sure to read below for specific instructions on seeds for each site. Not doing so could result in archiving millions of documents unintentionally.
Run a test crawl first
We strongly recommend a on all new seeds before performing a production (non-test) crawl. This will ensure that your seeds are configured correctly and that you won't unintentionally crawl much more content than desired at the expense of your account's data budget.
Limit your crawls
You may want to set up data and/or document limits for these sites if the test crawl shows an unusually large volume of content and you have confirmed that your seed URL is correct.
After you crawl
It is especially important to your first captures before regularly crawling your new seeds. Please look through your reports and the archived content after you run your first crawls in order to ensure that your archived content looks accurate, and that you didn't crawl more than you intended.
Sites with recommended scoping guidelines
Refer to the following recommended guidelines if you plan to archive content from these sites:
- Blogspot sites
- Flickr streams
- Instagram feeds
- Internet Archive audio and video
- Issuu hosted content
- Omeka sites
- Password protected sites
- Scribd hosted content
- SoundCloud audio
- Squarespace sites
- Tumblr sites
- Twitter feeds
- Vimeo videos
- Wikipedia pages
- Wix sites
- Wordpress sites
- YouTube videos