Data de-duplication

Our data de-duplication feature keeps duplicate data from being archived within a single collection, and therefore from counting against your subscription's data budget. If our crawler accesses a page or document that has not changed in any way since the last time it was archived in an individual collection, then that data will not count toward your Archive-It account's data budget a second time. (As long as the data hasn't changed, a page does not have to have the same URL as the last time it was crawled for the crawler to recognize it as a duplicate.) 

If a date on a document's Wayback calendar Page has an asterisk (*) next to it, then the URL's content is new or updated.  If there is no asterisk, then the URL has not changed since the previous capture:

Our standard crawlers de-duplicate at the collection level, while Brozzler crawls de-duplicate at the seed level.  De-duplication at the collection level means that crawls de-duplicate against all content saved in the collection (saved test crawls and production crawls).  De-duplication at the seed level means that each seed within a crawl de-duplicates against content previously captured via that seed.