Important Update : Crawling Instagram


1 comment

  • Official comment
    Mary Haberle

    Please note that Instagram recently changed their robots.txt file. Now, to capture Instagram, you'll need to either ignore robots.txt for the both the hosts and at the collection level - OR - ignore robots.txt at the seed level for your Instagram seeds.

    If you were already ignoring robots.txt at the seed level, you will not need to make any changes in order to continue capturing Instagram.

    Our help center documentation on Instagram has been updated to reflect this change.

    Comment actions Permalink

Please sign in to leave a comment.