Data de-duplication

Data de-duplication keeps duplicate data from being archived, and therefore from counting against your subscription's data budget. De-duplication happens when the crawler accesses a page or document that has not changed in any way since the last time it was archived. As long as the document hasn't changed, data from the re-visit of that document will not count toward your Archive-It account's data budget a second time. Data de-duplication is URL agnostic meaning a document does not have to have the same URL as the last time it was crawled for the crawler to recognize it as a duplicate. 

Each crawler handles data de-duplication a bit differently. The Standard crawler de-duplicates at the collection level, while Brozzler de-duplicates at the seed level.  De-duplication at the collection level means that crawls de-duplicate against all content saved in the collection (saved test crawls and production crawls).  De-duplication at the seed level means that each seed within a crawl de-duplicates against content previously captured via that seed. 

If a date on a document's Wayback calendar Page has an asterisk (*) next to it, then the URL's content is new or updated.  If there is no asterisk, then the URL has not changed since the previous capture: