Partners can crawl, archive, and replay websites built on the popular site-hosting platform Wix. These sites frequently do not require any crawl scope modification. However, some Wix templates can result in incomplete replay or crawler traps without the rules below in place.

General scoping rules for Wix

To ensure that all necessary scripts, stylesheets, and embedded media are archived, we recommend modifying the scope of crawls of Wix-based sites to include the following rules:

Ignore Robots.txt files on the following hosts:

• frog.wix.com
• static.parastorage.com

Expand the scope of your crawl to include URLs that contain the following strings of text:

• frog.wix.com
• static.parastorage.com
• static.wixstatic.com
• sslstatic.wix.com

Sites that are built on the Wix platform, but have a private domain (do not have Wix in the URL) may need to have subpages added as additional seeds to be fully captured.

Avoid crawler traps

Some templates used for Wix-based sites generate a significant number of long and ultimately invalid URLs that look like the one below:

http://www.example.com/uU5dR1gpXCHX45K8aOMct11OrLtyrYJeUnw_RxaUsg.eyJpbnN0YW5jZUlkIjoiMTNkZDc1ZTQtY2E2MC00ZGJkLWU4YTUtYTg xZjMxMzIyODVjIiwic2lnbkRhdGUiOiIyMDE1LTA1LTI4VDAwOjI5OjI5LjA3OFoiLCJpcEFuZFBvcnQiOiIyMDcuMjQxLjIyNi4xMTYvNDU2NTEiLCJkZW1vTW9kZSI6ZmFsc2UsImJpVG9rZW4iOiIxZjJjNTNlYS1hMGJjLTAyYmYtNDEwYi04N2E1N2IwNWZmYmMifQ

These URLs are usually unnecessary to replay the site correctly, but can still vastly expand the amount of documents and volume of data archived, which negatively effects your subscription budget. To prevent the crawler from accessing URLs like the one above, modify the scope of your crawl to block URLs that match the following regular expression, exactly as it appears here: ^.*/[^/]{300,}$