Good orientation is not only a weakness of travelers, but also of a special type of computer program – web crawlers. Unfortunately, most of these, crucial for effective website indexing, are not endowed with a sense of navigation and require our assistance. This is where the sitemap, or website map, comes in. 🙂
Website positioning consists of many steps. One of the first is optimization. However, excellent site preparation is of little use if web crawlers don't reach all the pages that should be included in the index. This is where sitemaps come in. This unassuming XML file is a crucial clue that allows web crawlers to quickly discover every important element of a page, thus accelerating its indexing and appearance in search results.
Sitemap xml - protocol history
The first information about the sitemap appeared on the Google blog in June 2005.
The sitemap's primary purpose was to provide Google with information about changes to the website and increase the number of indexed pages. Looking back 15 years later, it's safe to say the mission was a success – the sitemap still serves its original purpose today.
(The rest of the article can be found below the form)
Google didn't just expand its crawling capabilities – the sitemap.xml project was licensed under Creative Commons so that other search engines could use it as well. A year later, Yahoo! and Microsoft declared their support for sitemap.xml , and in subsequent years, Ask.com and IBM joined them.
Sitemap – what exactly is it?
A sitemap is a list of URLs that we want to point to web crawlers. XML (Extensible Markup Language), in turn, is a universal markup language that allows data to be described universally, regardless of the platform and technology used by the system.
A sitemap should include all important and public addresses—home pages, information pages, category pages, product pages, etc. The sitemap file can also—and should!—include addresses for subpages not listed in the menu. XML allows for a detailed description of these, along with important data, such as the date the subpage was last modified.
This method of presenting information about what's happening on your website allows you to quickly notify web crawlers about the creation of new subpages or changes to existing ones. Providing crawlers with a list of valid domain addresses significantly speeds up indexing – the program doesn't have to search for all the elements itself, but instead uses a pre-made list.
What does a typical sitemap.xml file contain?
SItemap.xml – as indicated by the information at the top of the printscreen – does not have any display guidelines (like pages created in HTML and CSS), hence its unfriendly form for an average user.
Above are two versions of the same file – sitemaps for the klodzko.pl website. The first image shows the browser view, the second is a preview of the file's code. Let's decipher the elements it contains:
- – the first piece of information indicates the XML version that was used to create the file, the second one refers to the encoding standard used
- – this opens a tag in which all links contained in the sitemap will be placed.
- – subpage address
- – additional information containing information about the last modification of the document located at the given address.
Other information may also be found here:
- – the content of this tag should be specified in the range 0.0 to 1.0, and its task is to indicate to robots which subpages should be indexed first.
Website Map – XML or HTML
Sitemap.xml isn't the only "site map" that can appear on a website . The purpose of an HTML sitemap is to guide users, for example, by grouping links to key products. This solution is most often used by large websites:
These types of maps are created using HTML and CSS. They contain no additional information, and links are placed within tags that create lists:
Is an HTML sitemap useful? It's certainly a place where lost website users can be directed. It also provides an additional place to find links to various categories with keywords that are important to us, which can be a great help with internal linking. Creating this type of sitemap is worth considering for larger websites – for smaller ones, there's no need for it.
Sitemap for web robots – in what cases is it worth creating one?
The main purpose of a sitemap file is to indicate to web crawlers the most important elements on a page. Creating an appropriate file is necessary in several cases:
- The website has many subpages – multi-page websites can struggle with rapid indexing – it can take several weeks for the crawler to find and index all the subpages. A sitemap, highlighting key points on it, will shorten this time and help your domain appear in search results faster.
- The website is new – a fresh domain is usually not very popular among internet users and is not very well linked to, which can make it difficult for web crawlers to discover it. Adding the website to Google Search Console and then creating an XML file with a list of all the pages you want to appear in the search engine is the best way to get it listed. site in search results.
- The website has many "hidden" subpages – some sites don't have extensive menus and, for various reasons, hide certain content outside the main categories. Categories with archived posts, for example, work this way. If valuable content is also hidden in these areas, it's worth inviting web crawlers to them and allowing them to index this part of the website.
- Your website has disappeared from search results – sometimes, a website is indexed but doesn't appear in Google. These types of errors happen even to the biggest companies – sometimes, simply changing the robots tag to "noindex" is enough to remove the website from the SERPs. In such situations, the speed of re-appearance in search results will be crucial, and a sitemap (especially if it wasn't there before) will allow you to quickly return to your previous positions.
For small websites, a sitemap isn't necessary—Google can easily crawl 5-10 tabs. Larger websites, especially those updated frequently, should ensure the file is properly prepared and made available to crawlers.
Sitemap – how to make it?
Preparing a sitemap.xml file is relatively simple, but manually completing it can be time-consuming. Fortunately, most CMSs generate sitemaps themselves or allow you to install a dedicated plugin.
Note – the most popular URL for sitemaps is www.your-site.pl/sitemap.xml. However, this location and file name are not mandatory. Some automated audits will consider the absence of a sitemap at this address as an error and report the missing XML file. However, this is not a concern. The address where the sitemap is located can be specified in the robots.txt file:
How do you create a sitemap for a website without a CMS? The easiest way is to use one of the generators available online. Before adding it to your website, it's worth reviewing the generated file – the generator, just like web crawlers, may have trouble finding pages that aren't linked in the menu or elsewhere on the site. If they're not in the file, you'll need to add them manually.
The size of the sitemap.xml file is important – the maximum size of the sitemap.xml file is 50MB. You should also not exceed the number of addresses per sitemap – the limit is 50,000 records. If a domain has thousands of subpages, it's worth splitting the file into several smaller ones and creating a sitemap index. Such solutions are used by WP plugins (mentioned below) and e-commerce engines that automatically generate entire sets of sitemaps. Creating an index with smaller sitemaps will help you avoid potential sitemap limits and maintain a clear file structure.
WordPress Sitemap – how to create it?
As with most WordPress changes, there's a plugin for this! In fact, a whole host of plugins. This functionality is offered by the most popular SEO plugins , including Yoast and All-in-One SEO.
Both plugins generate several sitemaps, and the file directories are available at:
- YOAST /sitemap_index.xml
- All in One SEO /sitemap.xml
Map components can be added and removed in the administration panel in the maps and indexing sections.
What should be in the sitemap.xml file and what shouldn't?
Creating and sharing a sitemap with crawlers aims to highlight your website's strengths and encourage their inclusion in search results. With this in mind, it's important to keep your sitemap up-to-date and include information that leads directly to the pages you want to index. Avoid:
- placing addresses returning errors 404,
- placing addresses returning errors 301,302,
- placing non-canonical addresses (whose address differs from the address contained in the canonical tag).
Adding addresses like these can cause problems with Google 's proper evaluation . It's no wonder – having to go from point A to point B also frustrates us and reduces our trust in the elements we've been directed through.
It is also not a good idea to include addresses blocked for web robots (tagged with or blocked in the robots.txt file
I have sitemap.xml, what next?
Creating a sitemap.xml file and uploading it to your server isn't the end of the road for web crawlers. The sitemap also has its own dedicated space in Google Search Console.
After logging into GSC, go to the "SITEMAPS" tab, located in the left-hand menu. In the tab, add the sitemap address—pasting just the URL, without the domain address—and click "submit."
After a while, the map address will be placed in the “Submitted sitemaps” table, where you will also find detailed information about the map – date of submission, last read, map status and the number of detected addresses.
Clicking on the map address allows you to go to the next screen:
For more details, click on “View status in index”, which takes you to the STATUS tab:
Here you will find all the information regarding the indexing of specific subpages and information about the problems encountered by Googlebot.
Sitemap.xml – what else is worth remembering?
Creating a website map is not the end – it is worth updating it from time to time, especially after significant changes to the addresses or structure of the website:
- implementing an SSL certificate and switching to an https address requires changes to all addresses on the website,
- redirecting or removing individual addresses – just change their URL or remove them from the sitemap.xml file
- complete website rebuild.
Updating sitemap.xml is also one of the activities worth doing after implementing keyword optimization .
A website map like sitemap.xml may not be as colorful as its classic cousins – but its effectiveness pays off!
FAQs about sitemaps
A sitemap is a special file (usually in XML format) that contains a complete list of all the subpages on your website. It can be compared to a digital table of contents or a precise roadmap created specifically for search engine robots (e.g., Googlebot), showing them how your website is structured.
Google's crawler navigates the web, jumping from link to link. If your site is large or has weak internal linking, the crawler may miss some pages. A sitemap hands it all URLs on a plate. This speeds up the indexing process for new and updated content, which is fundamental to search engine visibility.
The difference comes down to who the file is intended for:
- XML Sitemap: This is a technical, clean-code file created exclusively for search engine robots. Users don't need to access it.
- HTML Sitemap: This is a simple, aesthetically pleasing subpage (often linked in the website footer) created for humans. It helps users quickly navigate the complex structure of a large portal or online store.
Most modern content management systems (CMS) do this automatically. On WordPress, simply install one of the popular SEO plugins (e.g., Yoast SEO or Rank Math). Shopify and PrestaShop stores usually have this feature built in. If you have a website written from scratch, you can use free online generators (e.g., xml-sitemaps.com) that will create a file for uploading to your server.
Just creating the file is half the battle—you still need to inform Google about it. To do this, log in to the free Google Search Console tool, go to the "Sitemaps" tab, and paste your sitemap's URL there (usually yourdomain.com/sitemap.xml). It's also a good idea to add a line with the sitemap's address to your robots.txt file.
If you have a simple showcase website with only five pages, Googlebot will be able to find it without a sitemap. However, a sitemap is absolutely essential for:
- Large online stores (e-commerce).
- News portals and frequently updated blogs.
- New pages that do not yet have any external links.
- Websites with a very extensive menu and deep category structure.
Summary
The above article covers the following topics:
- Sitemap.xml is a file from 2005 containing a list of URLs in XML with information about modifications and indexing priority, which speeds up the work of web crawlers.
- Sitemaps are created automatically by CMSs or plugins like Yoast, with limits of 50MB and 50000 records. They should only contain active addresses without errors or redirects.
