Site Crawls FAQ

A site crawl is Siteimprove’s automated scan that navigates through webpages to evaluate the site’s quality and accessibility.

Siteimprove crawls give you an automated, up‑to‑date picture of your website by scanning your pages, following links, and collecting the information needed to check quality, accessibility, and performance. These scans help you see which content is discoverable, which pages may have issues, and how your site changes over time. Understanding how crawls work and what can affect them will help you interpret your results, spot errors, and know when additional troubleshooting or support may be needed.

Siteimprove uses automated crawlers to scan pages within the website URLs that have been registered for your unit. The system typically starts at the home page or designated entry points and follows links within the same domain to discover additional pages. It focuses on publicly accessible content, meaning pages that do not require logins or restricted access. Pages blocked by security settings, firewalls, or robots.txt files will not be scanned.

For more information on how Siteimprove’s crawlers operate and how to ensure all relevant pages are included in scans, consult the Siteimprove Help Center. Additional information can be found on Siteimprove's Crawler: Frequently Asked Questions.

Most Penn State websites in Siteimprove are automatically scanned every 21 days. For the IT Accessibility Office’s list of top 30 priority sites, scans occur more frequently, every 14 days, to provide up-to-date insight into accessibility issues and progress.

Crawls in Siteimprove can fail for a variety of technical reasons that prevent the system from accessing or analyzing a page correctly. Common causes include pages that require a login, restricted permissions, blocked crawlers, broken links, server errors, timeouts, redirects, or pages that are temporarily unavailable. In some cases, security settings, firewalls, or robots.txt files may also prevent scanning.

When a crawl fails, it does not necessarily mean there is a problem with your content. It usually means Siteimprove could not reach or process the page. These issues may require support from your unit’s web developer or IT staff to resolve.

For detailed explanations of specific crawl errors and step-by-step guidance on how to address them, users are encouraged to consult the Siteimprove Help Center article on crawl errors. This resource explains what different error messages mean and how to troubleshoot them effectively.

Yes, while Penn State’s standard subscription does not allow users to manually trigger a full site crawl on demand, Siteimprove does let you request a re‑check of:

multiple pages, or
a specific page or group of pages

without waiting for the next scheduled scan.

To do this, follow Siteimprove’s instructions for re‑checking content in the Help Center.

Re‑checking content helps confirm that your fixes have been picked up and reflects updated results more quickly than waiting for the automated crawl schedule.

Yes, you can crawl a single page using the Single Page Check instructions from Siteimprove.

: You can view the date of the last completed crawl and the date of the next scheduled crawl on the Quality Assurance dashboard in Siteimprove.

Site Crawls FAQ

Legal Statements