SEO Spider

The spider is a tool that scans through internal links of sites, subfolders or URLs to check a series of technical parameters “invisible” to the naked eye that describe the structural characteristics of the domain and highlight errors and warnings, flaws to be fixed, more or less serious problems, anomalies and obstacles that can hinder the site’s performance in Google Search.

Specifically, SEOZoom’s spider is a comprehensive tool for detecting structure errors, checking the HTTP status code of pages and resources, checking that meta tags, attributes and directives are correctly set, to avoid issues that can complicate Googlebot crawling, waste crawl budget or even prevent URL indexing on Google. In addition, thanks to the integration with the other tools in the suite, you can get information about each crawled URL, immediately displaying the number of keywords, traffic and other strategic details.

It is a complex tool: it offers data and analysis that you need to interpret carefully, but it is also the means to take a step forward and really be able to improve your website.

You can start the scan from Project or directly from the section in the side menu, through the Create a new Scan button; the number of scans available for each account depends on the type of SEOZoom plan you have subscribed to, while there are no limits to the start of scans, so you can also start the same scan multiple times.

In the window that opens you will have to indicate the name to be given to the scan, decide the extension (whether or not to restrict the analysis to the folder entered) and set the URL from which to generate the analysis (usually, the domain as it is indexed, with a focus on the protocol), and possibly indicate a pattern to be excluded from the scan

On the technical side, you can also set three different aspects:

• Courtesy level: after how many seconds the spider should scan the pages of the entered site. If set to one, the spider will scan your site every second.
• Scan depth: set the maximum level of click distance from the home page at which the spider will track pages. The value entered stops the scanning when a page reaches precisely that number of clicks from the home page.
• Pages to scan: the maximum number of pages the spider can scan, which also refers to the limit resulting from your subscription plan.

If you have subscribed to a Business and Corporate plan, you can manage another option, which relates to websites that use javascript frameworks such as React or Angular, by clicking on “Enable Javascript rendering” for a scan capable of scanning these elements as well.

Once the crawl is started, SEO Spider begins browsing the indicated site, subfolder, or URL, and the status updates you in real time on the level of completion, going from not started to queued, then crawling, and finally finished.

At this point, when the crawl is finished, SEOZoom provides you with an analysis highlighting anomalies and errors detected, broken down by tab, with a complete list of URLs analyzed and their values.

But how can you use all this information and, more importantly, do you have to worry if the Spider finds so many errors and warnings in your site? From our point of view, the approach to SEO Spider must start from the assumption that the alerts, Warnings and Errors, are all technical items related to the Spider’s settings: therefore, it is good to analyze the individual sections to unearth errors and fix them, but to this you have to put side by side a broad and conscious view of the technical aspects, because not all items are a “problem” and have the same weight. For example, and just quickly, a crucial page on your site which indexing you have inadvertently blocked via robots is definitely an error, while the absence of an alt text or duplicate H2 are minor items that should not impact your ranking (or, at least, not directly and so clearly).

And so, to have a perfect site it is definitely good to limit the number of Spider alerts, but you do not necessarily need to aim for absolute zero!