Partners can crawl, archive, and replay websites built on the popular site-hosting platform Wix. These sites frequently do not require any crawl scope modification. However, some Wix templates can result in incomplete replay or crawler traps without the rules below in place.

General scoping rules for Wix

To ensure that all necessary scripts, stylesheets, and embedded media are archived, we recommend modifying the scope of crawls of Wix-based sites to include the following rules:

Ignore Robots.txt files on the following hosts:

• frog.wix.com
• static.parastorage.com

Expand the scope of your crawl to include URLs that contain the following strings of text:

• frog.wix.com
• static.parastorage.com
• static.wixstatic.com
• sslstatic.wix.com

Troubleshooting

Sites built on the Wix platform may need to have sub-pages added as seeds in order for those sub-pages to replay.

To allow the pages to replay correctly, users will need to access sub-pages via the base domain seed (e.g. http://www.website.com). To help facilitate this we recommend setting the sub-page seeds (e.g. http://www.website.com/about) to private.

Keep in mind when you are reviewing your crawls, you should also navigate through the base domain's Wayback link to QA the sub-pages.

Avoid crawler traps

Some templates used for Wix-based sites generate a significant number of long and ultimately invalid URLs that look like the one below:

http://www.example.com/uU5dR1gpXCHX45K8aOMct11OrLtyrYJeUnw_RxaUsg.eyJpbnN0YW5jZUlkIjoiMTNkZDc1ZTQtY2E2MC00ZGJkLWU4YTUtYTg xZjMxMzIyODVjIiwic2lnbkRhdGUiOiIyMDE1LTA1LTI4VDAwOjI5OjI5LjA3OFoiLCJpcEFuZFBvcnQiOiIyMDcuMjQxLjIyNi4xMTYvNDU2NTEiLCJkZW1vTW9kZSI6ZmFsc2UsImJpVG9rZW4iOiIxZjJjNTNlYS1hMGJjLTAyYmYtNDEwYi04N2E1N2IwNWZmYmMifQ

These URLs are usually unnecessary to replay the site correctly, but can still vastly expand the amount of documents and volume of data archived, which negatively effects your subscription budget. To prevent the crawler from accessing URLs like the one above, modify the scope of your crawl to block URLs that match the following regular expression, exactly as it appears here: ^.*/[^/]{300,}$