Policies on Crawling Social Media

Good morning, 

I'm currently part of a team working for a pilot web archiving project at the University of Michigan. One of our central struggles is whether and under what circumstances social media sites should be included in our crawls, so we have been trying to learn more about how other institutions are approaching this challenge.

Although our team feels that social media is central to documenting our selected areas: diversity in children's literature, interactive fiction and queer digital culture, and the politics of water in Michigan, some questions have been raised about legal and ethical issues related to such content - both in terms of the expectation of privacy on the part of content-creators and the contract provisions in Twitter and Facebook's terms of use (which technically do not allow crawling). 

As we try to formulate our approach, it would be immensely helpful to learn more about how other institutions are approaching these questions. Do you crawl social media? If so, do you send opt-out or opt-in messages to content creators? Do you make a distinction between crawling publicly-accessible and password-protected content? Do you consider the terms of use to be a barrier to web archiving? 

Many thanks for your perspectives!
Juli McLoone

Outreach Librarian & Curator
University of Michigan Special Collections Library
jmcloone@umich.edu