Access Archive-It's Wayback index with the CDX/C API

Overview

This article explains how to use Archive-It's CDX/C API. The API can be queried by URL, collection, date range, and other attributes to discover what was archived, when it was captured, and where it is stored in WARC files. It also describes common query parameters and filtering options, allowing users to analyze archival coverage and retrieve capture metadata more efficiently than through the Wayback interface alone.

On this page:

• Background
• Why "CDX/C"?
• Use cases
• CDX records and attributes
• Basic use
• Query by collection
• Advanced use
• Filter and order fields
• Filter by date range
• Collapse results
• More advanced options

Background

Archive-It’s Wayback CDX is the index of all archived content that the Wayback browsing interface uses to lookup and serve the specific captures requested by an end-user, such as from the Wayback calendar page. The index format is known as 'CDX' and contains various fields that describe each record, sorted by URL and date. The index's server responds to GET queries and returns the plain text CDX data. The CDX server is deployed as part of the wayback.archive-it.org Wayback browsing interface and was derived from the CDX server deployed for the general archive at web.archive.org, as part of the open-source Wayback Machine software: https://github.com/internetarchive/wayback.

For more information on the general CDX file format, see http://archive.org/web/researcher/cdx_file_format.php

Why "CDX/C"?

Unlike the global Wayback index at archive.org, the CDX/C API enables querying of archived data by collection, meaning that a user may query it to discover records of captures within one of their own, another Archive-It partner’s, or all Archive-It partners’ collections.

Use cases

Using the CDX/C API to query Archive-It data is a way to discover if and to what extent web content has been archived by Archive-It partners. Partners can use the API to find if and when specific documents were archived, and to locate that data in its WARC file storage, among other things. They can also find and filter by other capture attributes to analyze the extent and nature of their collecting any specified documents or hosts.

For an example of how the CDX/C API can be used in practice, see Greg Wiedeman's guest blog post, A Sustainable, Large-Scale, Minimal Approach to Accessing Web Archives. It describes how the University of Albany, SUNY, uses the CDX/C API  to dynamically query the index for records to reference in finding aids for collections in which websites are captured on a regular and ongoing basis.

CDX records and attributes

The CDX/C is effectively a table of plain text data. Each line (“record”) indicates a crawled document. For instance, the first record for the query: https://wayback.archive-it.org/8232/timemap/cdx?url=https://twitter.com/internetarchive/ appears as:

The publicly available attributes in the CDX/C index are:

Basic use

To access the CDX/C API, select the green CDX button on the Wayback Calendar page for an archived document.

Note: The calendar page collapses captures taken within minutes of each other under a single timestamp. This is why you may see more records in CDX than captures on the Wayback calendar page.

Alternatively, you can query the CDX/C API by curl command or in a web browser. For either method, the endpoint is:  http://wayback.archive-it.org/all/timemap/cdx. The only required parameter for the CDX server is the url= parameter. For example:

http://wayback.archive-it.org/all/timemap/cdx?url=archive-it.org

This example query will return a portion of the Archive-It’s CDX index, one capture per row, for each capture of the URL "archive-it.org" that is available in Archive-It Wayback.  To customize the query, change the ending URL.

⚠️ Queries to the /all CDX endpoint are temporarily blocked due to DDoS activity. All other queries work as expected. Let us know if this impacts you.

Query by collection

To focus your queries to return results from only one specified Archive-It collection, replace ‘all’ in the endpoint’s query string with the desired collection’s collection ID number. For example:

http://wayback.archive-it.org/4399/timemap/cdx?url=archive-it.org

This example query will return only the portion of the CDX index that includes captures of archive-it.org from Archive-It collection #4399.  Note that for private collections, you must include the collection ID in order to see results.

Advanced use

Filter and order fields

It is possible to customize the fields returned from the cdx server using the fl= parameter. Simply pass in a comma separated list of the desired fields and only those fields will be returned, in the order in which you query them. For instance, the following example returns only the timestamp and filename fields for captures of archive-it.org in collection #4399: 

http://wayback.archive-it.org/4399/timemap/cdx?url=archive-it.org&fl=timestamp,filename

Filter by date range

Results can be filtered by timestamp using from= and to= parameters. The ranges are inclusive and are specified in the same 1 to 14 digit format used for Wayback captures: yyyyMMddhhmmss. For instance, the following example returns all captures of archive-it.org in the CDX index, across all collections, that were captured between 2010 and 2011: 

http://wayback.archive-it.org/all/timemap/cdx?url=archive-it.org&from=2010&to=2011

Collapse results

One useful form of filtering query results is the option to 'collapse' them based on the value of a field, or of a substring of a field. Collapsing is done on adjacent CDX lines where all captures after the first one that are duplicate are filtered out. This is useful for filtering out captures that are 'too dense' or when looking for totally unique captures.

To use collapsing, add one or more collapse=field or collapse=field:N parameters, where N is the first N characters of field. For instancem in the following example results are collapsed to show only one result per hour of capture time  (compare the first 10 digits of the yyyyMMddhhmmss timestamp field):

http://wayback.archive-it.org/4399/timemap/cdx?url=archive-it.org&collapse=timestamp:10

The calendar pages at wayback.archive-it.org use this filter by default.

More advanced options

To explore additional advanced querying options, see the general Wayback CDX API documentation at https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server#advanced-usage