Archiving Facebook

You can add Facebook pages, profiles, and/or groups to your collection in order to crawl, archive, and replay them as you would any other seed site, just so long as you remember to format and scope them according to a few simple rules.  We recommend running a test crawl after implementing any new rules.

On this page:

• How to add Facebook seeds to your collection
• Required scoping for Facebook seeds
  1. Apply seed-level data limit
  2. Ignore robots.txt
  3. Exclude segmented video files
  4. Expand scope to capture scroll
  5. Exclude non-English interface
  6. Exclude personal user profiles
• What to expect from your archived Facebook seeds
• Troubleshooting

How to add Facebook seeds to your collection

Follow our standard guidance for adding seeds to your collection, but keep the following principles in mind:

• Be Specific! Only add seeds for specific users, groups, events, etc. Do not attempt to crawl and archive all of Facebook--don't forget to add an ending slash to your Facebook URL
• Use the HTTPS version of the URL. Since Facebook serves its content exclusively from HTTPS, you can avoid potential crawl problems related to redirects by formulating your seed using "https://" instead of "http://"
• Check a seed's availability on the live web. If you must be logged into Facebook as a specific user to view a private page or group, then our crawler will need to be logged in as that user, too. Follow our directions to crawl password protected websites, if necessary

Required scoping for Facebook seeds

The proper formatting above enables our crawler to access Facebook seeds. To ensure that it logs in and captures scroll without archiving too much material from non-target areas of Facebook, always apply all six of the following scope modifications:

1. Apply seed-level data limit

Limit the scope of each Facebook seed to archive a minimum of 1GB of data. If you are unsure how to add these rules, please consult our documentation on seed-level scoping rules.

2. Ignore robots.txt

Ignore robots.txt for each Facebook seed at the seed level.



OR Add collection-level scoping rules to ignore robots exclusions on the following hosts, exactly as they appear here: 

1. www.facebook.com - in order to archive Facebook-hosted content
2. fbcdn.net - in order to archive important page styling elements
3. akamaihd.net - in order to archive important page styling elements

3. Expand scope to capture scroll

To archive content from Facebook's dynamically scrolling pages, expand the scope of your crawl at the seed level to include URLs that match the following SURT, exactly as it appears here: +http://(net,fbcdn,

4. Exclude segmented video files

Facebook frequently serves videos in segmented files and the crawler will capture these segments in addition to files that contain the entire video. Exclude segmented video files at the seed level by adding a seed level block URL rule to all Facebook seeds on all URLs that include the text: bytestart=

5. Exclude non-English interface

Limit the scope of your crawls to only archive the Facebook interface in English (this will not affect posts written in other languages), add a rule to block URLs that match the following regular expression: ^https?://..-..\.facebook\.com.* 

If adding this rule to the Collection Scope tab, enter facebook.com in the host field.  

If you prefer to archive a Facebook page/feed as it appears in a language other than English, you will need to log our crawling technology into Facebook with an account that defaults to your preferred language. You can create a "dummy" account for this purpose, then add its login credentials to each Facebook seed to which they apply.

6. Exclude personal user profiles

Prevent the capture of individual Facebook users' personal profiles from archiving by limiting your crawl to block all URLs that match the following regular expression: ^https?://www\.facebook\.com/(profile\.php|people/).*$  

If adding this rule to the Collection Scope tab, enter facebook.com in the host field.

What to expect from your archived Facebook seeds

Because of the way that our crawling technology must log in to Facebook in order to avoid being blocked from it entirely, you can anticipate the following:

• You will not be able to expand comments sections of posts in order to see any more comments than our crawling technology saw.
• Each post will display how many times that it has been “liked,” however you will not be able to browse through the specific names of Facebook users who “liked” any post.
  

Troubleshooting

It is possible to use the Wayback QA tool on saved test crawls and production crawls to capture missing content.  It is best to use this tool to run patch crawls as soon after a crawl ends as possible, because some URLs will be time sensitive, due to how Facebook serves up its content.