capturing redirects from an old site

Follow

Stuart Yeates

August 21, 2016 21:45

I'm planning the harvest of a site which used to live at http://www.nzetc.org and now lives at http://nzetc.victoria.ac.nz/ There is a blanket redirect which redirects every URL on the old host to the new URL in a new host, which involves changing only the host part of the URL.

There are hundreds of links to the old host in places like wikipedia which we really don't want to break.

Is there a way to encode this without needing the crawler to crawl each page twice, once at the old URL and once at the new?

If I do need to crawl twice, what's the easiest / lowest impact method of doing that? My current plan is: (a) harvest the new site as normal (b) make a list of all URLs harvested by the crawler based on the logs (c) search and replace the host name (d) publish that somewhere the crawler can see and (d) add as a seed.  

0

• 

  Emily Chen August 25, 2016 09:04

  Hmmm are you worried about your data budget or the crawls disrupting the host site? I would probably just have the old link and new link as seeds in a single crawl and the data will be deduplicated anyway, unless I've misunderstood your concern.

  0

  Comment actions Permalink

Please sign in to leave a comment.