Overview
YouTube is an online video sharing service and social media platform where users can watch, like, comment, and upload their own videos. You can add YouTube videos or channels to your collection in order to crawl, archive, and replay them as you would any other seed site, just so long as you remember to format and scope them according to a few simple rules. This guide provides an overview of how to properly format, scope, and crawl YouTube seeds.
Known issues
Social media platforms like YouTube can be difficult to archive. Currently, YouTube has the following issues which we continue to actively monitor:
- ⚠️ Video watch pages may need to be opened in new browser tabs in order to replay
Social media platforms update frequently. You can find a full list of known issues for archiving various platforms on our Social media and other platforms status page.
On this page
- How to select and format your YouTube seeds
- Scoping YouTube seeds
- Running your crawl
- What to expect from archived YouTube seeds
How to select and format YouTube seeds
Watch pages
Specific videos on YouTube are hosted on a "watch" page with a URL in the following format:
- https://www.youtube.com/watch?v=XXXXXXXXXX
Follow this formatting and always use the "One Page" seed type to avoid scoping in all of youtube.com.
Channels and Users
YouTube channels are topic-specific groups of videos and related content. YouTube channel URLs can have multiple formats. For example, the University of Melbourne's channel can be accessed at https://www.youtube.com/user/unimelb/, and the Internet Archive's channel can be accessed at https://www.youtube.com/@internetarchive_/. This URL can serve as your seed URL.
However, when you wish to archive a user's YouTube channel in its entirety, we further recommend adding an additional seed URL for its "Videos" tab, which is formatted as: https://www.youtube.com/user/[user-name]/videos or https://www.youtube.com/@internetarchive_/videos. This enables our crawler to access all videos uploaded to the user's account.
The Standard seed type is best for crawling the videos linked off of a channel page, however a test crawl is strongly recommended when crawling a channel for the first time. Depending on the number of videos, these seeds can be very data heavy. Consider using the test crawl to determine how much data you should allot to these seeds then limit your crawl by adding a data limit at the seed level.
Playlists
Playlists are specific lists of videos curated by a user from among their account's and/or other videos on YouTube. The URL for each playlist can be added as seed, and is typically formatted as: https://www.youtube.com/playlist?list=XXXXXXXXXXXXXXXXXX
To archive a playlist index and all of the video watch pages to which it links:
- Format your seed like the above - do not put a trailing slash (/) at the end
- Use the "Standard" seed type
- Use the Brozzler collecting technology
Shorts
YouTube shorts are similar to video watch pages, but the videos are often under 1 minute. Their URL format is:
For archiving individual Shorts videos:
- Format your seed like the above - do not put a trailing slash (/) at the end.
- Use a "One Page" seed type.
- Crawl with Brozzler.
- Expected replay is to use the Wayback banner's media link to play back the video.
- Shorts' placeholder images often display blank.
For archiving Shorts' collection pages, the URL format is often:
- Format your seed like the above - do not put a trailing slash (/) at the end.
- Use a "Standard" seed type.
- Crawl with Brozzler.
- Expected replay is to open individual Shorts videos' pages in new tabs (right-click the videos' thumbnail and select top option), then use the Wayback banner's media link to play back the video.
Search results
YouTube search pages such as https://www.youtube.com/results?search_query="william+shakespeare" are best crawled using the One Page Plus External Links (One Page +) seed type. We strongly recommend a test crawl when crawling search pages. As with channels, you may need to add a seed level data limit in order to avoid crawling excessive additional videos.
Embedded videos
Videos that are embedded into other sites should archive successfully as long as the default scoping rules below have been applied to that site's crawl.
Scoping YouTube seeds
You can set up your crawls to archive videos from YouTube watch pages, channel pages, or embedded videos in other sites, by adding a few simple scope modifications at either the Collection or Seed level.
Default scoping rules for YouTube seeds
New YouTube seeds added to collections will have the following default scoping rules applied automatically at the seed level; older YouTube seeds can be updated by adding the below scoping rules manually or following these instructions.
To learn more, please visit Sites with automated scoping rules.
1. Robots.txt Blocks
YouTube blocks important page styling content and some video files with robots exclusions. To make sure that you are able to capture the look and feel of a YouTube page and/or any video content, you will therefore need to add the rules listed below.
If you wish to archive YouTube videos linked or hosted by any site in the course of your crawl, you must first modify your collection's crawl scope to ignore robots.txt files from the following hosts, exactly as they appear here:
- youtube.com [note: no www.] - This is one of the hosts that serves YouTube video files.
- googlevideo.com [note: no www.] - This is the primary host that serves YouTube video files.
- ytimg.com - This host serves stylesheet and JavaScript content necessary for playback.
- OR -
If you prefer to archive only those videos and video pages from the specific YouTube seed that you added to your collection: Ignore robots.txt for each seed.
2. Crawler traps
Sometimes, YouTube crawls can run into crawler traps and archive invalid URLs with seemingly endless combinations of repeating directories. This is the most common crawler trap that affects the host www.youtube.com, as in this example:
https://www.youtube.com/channel/UCrHC0hXTvYewidE9Q5AM_8w/_/im/HTTP/www/www/HTTP/HTTP/www/HTTP/HTTP/www/www/channels
This issue can be addressed in the seed scope by adding the following repeating directories regular expression to Block URL if it matches the regular expression:
^.?(/.{2,}?/).?\1.$|^.?/(.{2,}?/)\2.*$
Optional scoping rules for YouTube seeds
1. Scoping Googlevideo.com
Host level data limits apply individually to each subdomain of that host. Because googlevideo files are served from a large number of individual subdomains, adding one rule to limit the amount of Googlevideo content often isn't enough. For example, a 1GB data limit on the host googlevideo.com could result in 1GB from r5---sn-n4v7sn7s.googlevideo.com, 1GB from r3---sn-n4v7sn7s.googlevideo.com, and so on.
The easiest way to limit the amount of googlevideo content is to identify the seeds from which googlevideos are being captured, and add seed level data limits to them. The size of the data limit will vary depending on your seed, so be sure to run test crawls to make sure you're capturing what you need.
Running your crawl
Once you have finished selecting your seeds and adding recommended scoping rules, we recommend that you crawl your seeds using Brozzler. Running a test crawl is highly recommended when crawling new YouTube seeds for the first time.
What to expect from archived YouTube videos
Archived YouTube content in Wayback currently replays most reliably in Chrome. If you do not see your videos replaying as expected, check the link in Chrome before submitting a support ticket. This issue stems from browser limitations; we will update this documentation when you can expect replay parity across browsers.
If you collected YouTube using Standard crawling technology and are having trouble replaying it across browsers, consider running a new crawl using Brozzler.
For individual watch pages, the archived video should replay within the page. Any video that was collected from that page can also be replayed by selecting the play media button in the Wayback banner.
For channels and playlists with infinite scrolling, the archived page might not display content below the fold.
Pages with embedded YouTube video(s) might not replay within the page. The archived video(s) can be replayed by selecting the play media button in the Wayback banner.
Comments
0 comments
Please sign in to leave a comment.