Discover Sources Greater Than 15 MB For Higher Googlebot Crawling

Find Resources Bigger Than 15 MB For Better Googlebot Crawling

Googlebot is an automated and always-on internet crawling system that retains Google’s index refreshed.

The web site worldwidewebsize.com estimates Google’s index to be greater than 62 billion internet pages.

Google’s search index is “nicely over 100,000,000 gigabytes in measurement.”

Googlebot and variants (smartphones, information, pictures, and many others.) have sure constraints for the frequency of JavaScript rendering or the scale of the sources.

Google uses crawling constraints to guard its personal crawling sources and methods.

As an example, if a information web site refreshes the beneficial articles each 15 seconds, Googlebot would possibly begin to skip the continuously refreshed sections – since they received’t be related or legitimate after 15 seconds.

Years in the past, Google introduced that it doesn’t crawl or use sources greater than 15 MB.

On June 28, 2022, Google republished this blog post by stating that it doesn’t use the surplus a part of the sources after 15 MB for crawling.

To emphasise that it hardly ever occurs, Google said that the “median measurement of an HTML file is 500 instances smaller” than 15 MB.

timeline of html bytesScreenshot from the writer, August 2022

Above, HTTPArchive.org reveals the median desktop and cellular HTML file measurement. Thus, most web sites would not have the issue of the 15 MB constraint for crawling.

However, the online is an enormous and chaotic place.

Understanding the character of the 15 MB crawling restrict and methods to investigate it will be significant for SEOs.

A picture, video, or bug could cause crawling issues, and this lesser-known search engine optimisation info will help tasks defend their natural search worth.

Find Resources Bigger Than 15 MB For Better Googlebot Crawling

Is 15 MB Googlebot Crawling Restrict Solely For HTML Paperwork?


15 MB Googlebot crawling restrict is for all indexable and crawlable paperwork, together with Google Earth, Hancom Hanword (.hwp), OpenOffice textual content (.odt), and Wealthy Textual content Format (.rtf), or different Googlebot-supported file sorts.

Are Picture And Video Sizes Summed With HTML Doc?

No, each useful resource is evaluated individually by the 15 MB crawling restrict.

If the HTML doc is 14.99 MB, and the featured picture of the HTML doc is 14.99 MB once more, they each can be crawled and utilized by Googlebot.

The HTML doc’s measurement will not be summed with the sources which can be linked through HTML tags.

Does Inlined CSS, JS, Or Information URI Bloat HTML Doc Measurement?

Sure, inlined CSS, JS, or the Information URI are counted and used within the HTML doc measurement.

Thus, if the doc exceeds 15 MB as a consequence of inlined sources and instructions, it should have an effect on the particular HTML doc’s crawlability.

Does Google Cease Crawling The Useful resource If It Is Greater Than 15 MB?

No, Google crawling methods don’t cease crawling the sources which can be greater than the 15 MB restrict.

They proceed to fetch the file and use solely the smaller half than the 15 MB.

For a picture greater than 15 MB, Googlebot can chunk the picture till the 15 MB with the assistance of “content material vary.”

The Content material-Vary is a response header that helps Googlebot or different crawlers and requesters carry out partial requests.

How To Audit The Useful resource Measurement Manually?

You should utilize Google Chrome Developer Tools to audit the useful resource measurement manually.

Comply with the steps under on Google Chrome.

  • Open an internet web page doc through Google Chrome.
  • Press F12.
  • Go to the Community tab.
  • Refresh the online web page.
  • Order the sources in keeping with the Waterfall.
  • Test the measurement column on the primary row, which reveals the HTML doc’s measurement.

Beneath, you’ll be able to see an instance of a searchenginejournal.com homepage HTML doc, which is larger than 77 KB.

search engine journal homepage html resultsScreenshot by writer, August 2022

How To Audit The Useful resource Measurement Routinely And Bulk?

Use Python to audit the HTML doc measurement routinely and in bulk. Advertools and Pandas are two helpful Python Libraries to automate and scale SEO tasks.

Comply with the directions under.

  • Import Advertools and Pandas.
  • Acquire all of the URLs within the sitemap.
  • Crawl all of the URLs within the sitemap.
  • Filter the URLs with their HTML Measurement.
import advertools as adv

import pandas as pd

df = adv.sitemap_to_df("https://www.holisticseo.digital/sitemap.xml")

adv.crawl(df["loc"], output_file="output.jl", custom_settings="LOG_FILE":"output_1.log")

df = pd.read_json("output.jl", strains=True)

df[["url", "size"]].sort_values(by="measurement", ascending=False)

The code block above extracts the sitemap URLs and crawls them.

The final line of the code is just for creating an information body with a descending order based mostly on the sizes.

holisticseo.com urls and sizePicture created by writer, August 2022

You’ll be able to see the sizes of HTML paperwork as above.

The largest HTML doc on this instance is round 700 KB, which is a class web page.

So, this web site is protected for 15 MB constraints. However, we are able to examine past this.

How To Test The Sizes of CSS And JS Sources?

Puppeteer is used to examine the scale of CSS and JS Sources.

Puppeteer is a NodeJS package deal to manage Google Chrome with headless mode for browser automation and web site checks.

Most search engine optimisation execs use Lighthouse or Page Speed Insights API for his or her efficiency checks. However, with the assistance of Puppeteer, each technical side and simulation could be analyzed.

Comply with the code block under.

const puppeteer = require('puppeteer');

const XLSX = require("xlsx");

const path = require("path");

(async () => .com", "");



          const workSheetName = "Customers";

          const filePath = `./$domainName`;

          const userList = perfEntries;



          const exportPerfToExcel = (userList) => 

              const information = perfEntries.map(url => 

                  return [url.name, url.transferSize, url.encodedBodySize, url. decodedBodySize];


              const workBook = XLSX.utils.book_new();

              const workSheetData = [




              const workSheet = XLSX.utils.aoa_to_sheet(workSheetData);

              XLSX.utils.book_append_sheet(workBook, workSheet, workSheetName);

              XLSX.writeFile(workBook, path.resolve(filePath));

              return true;








Should you have no idea JavaScript or didn’t end any form of Puppeteer tutorial, it is likely to be a bit more durable so that you can perceive these code blocks. However, it’s really easy.

It mainly opens a URL, takes all of the sources, and provides their “transferSize”, “encodedSize”, and “decodedSize.”

On this instance, “decodedSize” is the scale that we have to concentrate on. Beneath, you’ll be able to see the outcome within the type of an XLS file.

Resource SizesByte sizes of the sources from the web site.

If you wish to automate these processes for each URL once more, you have to to make use of a for loop within the “await.web page.goto()” command.

In keeping with your preferences, you’ll be able to put each internet web page into a unique worksheet or connect it to the identical worksheet by appending it.


The 15 MB of Googlebot crawling constraint is a uncommon chance that may block your technical search engine optimisation processes for now, however HTTPArchive.org reveals that the median video, picture, and JavaScript sizes have elevated in the previous few years.

The median picture measurement on the desktop has exceeded 1 MB.

Timeseries of Image BytesScreenshot by writer, August 2022

The video bytes exceed 5 MB in whole.

Timeseries of video bytesScreenshot by writer, August 2022

In different phrases, every so often, these sources – or some components of those sources – is likely to be skipped by Googlebot.

Thus, you need to have the ability to management them routinely, with bulk strategies to make time and never skip.

Extra sources:

Featured Picture: BestForBest/Shutterstock

Source link

Leave A Comment



Our purpose is to build solutions that remove barriers preventing people from doing their best work.

Giza – 6Th Of October
(Sunday- Thursday)
(10am - 06 pm)

No products in the cart.

Select the fields to be shown. Others will be hidden. Drag and drop to rearrange the order.
  • Image
  • SKU
  • Rating
  • Price
  • Stock
  • Availability
  • Add to cart
  • Description
  • Content
  • Weight
  • Dimensions
  • Additional information
  • Attributes
  • Custom attributes
  • Custom fields
Click outside to hide the compare bar
Compare ×
Let's Compare! Continue shopping