Do sitemaps still matter?

0:00Hi, Gritzi. Hello. My name is Martin Split from the Google Search Relations team and I welcome you all to a new episode of Search of the Record. The podcasts where we, the Google Search Relations team, are taking you behind the scenes of Google Search and hopefully have some fun along the way. Today's topic is Site Maps. We will talk a little bit about if you need one, what they are, what they aren't, the XML format, the RSS differences, HTML, maybe LLM's TXT and who knows, maybe we will look into some of your questions around Site Maps as well. And with that, I'd like to say hello to my guests today. Hi, John. Hi, Martin. How are you doing, John? How am I doing? I'm okay.

1:00I'm standing here talking into my micro-road. I think that's okay. I think that's good. Yeah, I do the same and I actually have a question for you. Okay. I get emails and people seem to be struggling with Site Maps. Okay. Yeah, they are seeing something in Search Console that makes them worried and I wonder if we should clear these things up. Do you think so? Okay. Can we do that? Awesome. Site Maps goes the long way back. It was basically how I made my way to Google. What? If you believe it or not. Really? How so? So, at some point, I forgot who it was, but someone came up with this idea of Site Maps and they shared it with the folks at Microsoft and Yahoo at the time and this came out as kind of this semi-standard thing that everyone agreed upon or at least like three companies agreed upon.

2:01And right about that time, I started doing a little bit more on the web. So I created a Windows-based Site Maps generator. And it basically crawls your website like a normal crawler and finds all the URLs and puts them in a Site Map file properly. Which at the time was kind of unique in that a lot of people didn't crawl their own website because they had no idea. They had to do that. So it ended up being, I think, reasonably popular, not like super-super popular kind of thing, but it's basically something where I got involved a little bit more with the SEO community, the early folks in SEO at the time and with general, like, people who are making websites and started helping them out with regards to all of these questions that came up from Site Maps. Because if you never crawl your website and you have this interesting PHP or whatever-based

3:02website, then you end up with a lot of URLs that can be found which don't make a lot of sense. This was before something like WordPress or any of the mainstream CMSs were kind of clean. It's like lots of weird loops where crawlers found lots of parameters and lots of stuff to clean up. So that was kind of interesting. And then someone from Google noticed and we got into talking and the rest is history. How long ago is this? 500 years ago. Yeah. But like, seriously, this must be like 20 issues. I think that's how old Site Maps are, no? I don't know. I'm afraid to look. Okay. I mean, you just joined Google yesterday. So yes, of course, all of this is fresh in you. Okay. So Site Maps are XML files and search engines use them for what exactly? The goal at the time, I think, was just to discover all of the URLs on your website,

4:06where I think like I mentioned in the beginning, websites were really messy and hard to crawl. So if the site owner gave you a list of URLs and told you like, these change on this day and these have different priorities and whatever, then that's a lot easier for a search engine to say, okay, I will just focus on these. I will still crawl to make sure I don't miss anything, but I'll try to focus on these. Okay. But with all the changes and all the improvements and all the developments we've done in crawling, do we still need them? Can't we just figure this out? Can't we just use AI? Oh, yeah, or use an AI because you know, it's 2026. Let's use an AI. I think so. So over the years, some things went away with Site Maps. Oh my gosh, this is going to be a very nostalgic episode. I didn't realize. Some things went away. So there used to be a priority field in Site Maps and you can imagine if you tell SEOs and ask them, what is your priority for your pages?

5:07Everything. Everything is number one. Yes, of course. Every URL is a maximum priority. That ended up not being very useful. There was also a field for change frequency. How often does this page change? And what do you think SEOs did? Changes all the time. Changes all the time. Super important. Always fresh. Always fresh. I mean, partially, I think it's not because they're trying to be misleading, but it's just if you have a server side dynamic website, the PHP file that runs is going to be fresh all the time anyway. So theoretically, it's correct, but it's not very useful. I think those are the two fields that we dropped. Maybe there are some other fields that I forgot in the meantime. But basically, what we primarily focus on is a URL and the date, the change date. And the change date should be like when you last made major changes on this page.

6:09If you basically give us today as a change date for all of your URLs, we will realize that it doesn't make much sense to focus on that. But this is not like a spam thing where our spam systems would say this is a bad site map. And do all of the websites need a site map still? Or is it more useful for some types of websites and not so useful for this? I think smaller websites probably don't need a site map because we can just crawl them. But I know it's really hard to make this threshold decision as a site owner. You're like, oh, I have 10 pages. That's small. I have 50 pages. Is that big? I don't know. So my recommendation usually is just turn it on because in the meantime, almost all content management systems, even the static hosting systems, they generate site maps files

7:11by default unless you turn them off. So I would generally recommend just keeping it turned on. Even if you have a small site, if the site grows over time, it's fine. If the site stays small forever, that's also fine. There's no harm in having a site map file if your site is small. Do websites with a lot of content that changes a lot like let's say news websites do they benefit from a site map specifically or is it just like any other website? Definitely. Definitely news websites. There's even a special kind of site map for news sites specifically where I think you are only supposed to list the last 1000 pages that changed on a website so that it's easier for systems who want to keep up with a news website to actually keep up. I think it's new site map. I haven't looked at it in a while, but basically for any website that has a non-trivial amount

8:13of content and if that content changes regularly, the site map file is super helpful because maybe in some category section you change something on your site. With traditional crawling, we would have to crawl all the way and find that page and be like, oh, look, I just looked at this page again and it's new. If you have an e-commerce site, that's pretty common, like the price change. By the time we notice with normal crawling, maybe that's a bit late, whereas if you can tell us, hey, this product page changed, it's a lot easier to go there directly. And we've talked about some things that we are not using anymore like chain frequency and priority, but there's also last mod so you can tell when the URL last changed. Is that still? That's a date. Yeah. Is that still something that we use or should I ignore that as well? My understanding is we want to use it, but a lot of sites get it wrong.

9:16So we have this weird love-hate relationship, I think, with it. In that, again, my understanding, maybe this has changed in the meantime, my understanding is that if we can tell that within a site map file, the dates are kind of reasonable, then we will try to take that into account. Whereas if we can't tell that the dates are reasonable, then we will focus more on the individual URLs. And then we'll be like, oh, a new URL popped up, we will process it. Oh, but I could use timestamped URLs in the site maps file to make sure that, no. So in the site map file, you're supposed to, no, it's supposed to, it's not like a law. The site map police. The site map police. The site map police. You get a ticket. The idea is that you include the URLs that you want to be canonical. So if you have URLs that are tagged with specific things that you use internally for tracking

10:17or whatever, you would kind of submit the URLs in the site map file the way that you want to have them indexed, not the way that they're kind of like temporarily on your website. So if you have a, I don't know, a static looking URL and you add a date stamp to it because you want to make sure that the newest version is always used, then we will try to index the non date stamp version as a canonical. So if you can make sure that canonical version is in the site map file, we can take that into account. My understanding is we also use site map files with regards to choosing the canonical. So if we kind of have a variety of different URLs that we discover and they're all kind of similar, but if you list one of them in a site map file, it's a little bit more likely that we will pick that as canonical. Okay, that doesn't sound too bad. Yeah. But if I want to make sure, like if I have a blog, I sometimes, some CMS is created in

11:21RSS, is that kind of the same as a site map or is it different? It's very similar. I mean, if you ask those who made the RSS standard, they will say it's very different, but with regards to focusing purely on the URL and maybe a date, it's very similar in that the RSS feed will list the URLs that recently changed. And often it has a cap, maybe I don't know, 10 or 20 of the last URLs that changed. It includes a date stamp of this page, which I don't know exactly how it's stored in RSS, probably some obscure standard, but it has a date and a time associated with it. So if we look at the RSS file, we can find those URLs, we can find those dates as well. So in a sense, an RSS file can be used as a site map file. You can also submit it in search console as a site map.

12:24It's very similar. The main difference is that it focuses on the last URLs there. So if you have, I don't know, a thousand site map files because they're kind of limited in size, but you have a lot of pages, then a system would have to look at those thousand site map files first to find out what has changed. Or if it has the RSS feed, it can just look at the RSS feed and be like, oh, these 50 pages have changed recently, I can focus on those. So they both have usefulness with regards to finding new and updated pages. Okay, but wait, so site map files do have a limit as to how much they can contain. Yes. What's the limit? Oh my gosh. Ah, I know. I'm testing you. I should look it up before I tell you, but my memory is 50,000 URLs and 50 megabytes. Yes. Something like that. And the 50 megabytes is non-compressed.

13:24You can gzip site map files as well, but you can't target a gzip size. So the size limit is based on the non-compressed site. You can submit as many site map files as you want in search console. You can link them in Europa.txt file. You can also create a so-called site map index file, which is a kind of a site map file that links to other site map files. You can only nest them once, but you can submit as many site map index files as you want. So you can kind of do crazy stuff that you probably shouldn't be doing with lots of URLs. Yeah. I think if you're at the point where you have multiple site map index files because they get too large, then you definitely have enough URLs to warn the site map in the first place. Yeah. Maybe one thing that's also kind of unique with site map files is you can submit them with robots.txt. You can tell everyone, basically where they are.

14:25You can just call them site map.xml and a lot of systems will guess and take a look. Or you can call them anything. You can call them martenscat.xml. You probably don't even need to use that XML, but that's pretty common for XML files. If you call it something unique and you don't list it in robots.txt, then you need to tell us about it. And you can do that in search console and say, like, here's my site map file. So that's kind of helpful. RSS is a bit different in that usually RSS is linked from the head of HTML pages. So if you know an HTML page, you can find the RSS feed usually or many cases, which means that connection is a little bit clear. Some people don't want their site map files to be known to other people. And that's perfectly fine. That means you would use a different file name, for example, and you would tell Google

15:26about it directly. The downside is that other systems that might want to use site map files wouldn't be able to find it. That means if you're using Bing and you assumed that Bing would figure it out too, you probably have to tell Bing as well. That also means if you care about AI training crawlers, if you want to make sure that your content is in these AI systems, then they usually don't have any kind of a console or any setup where you can submit a site map file. So either you stick to the generic naming, call it site map.xml or you focus on RSS feeds, and then these AI crawlers can sometimes find those files and process them. I don't know if they document this anywhere, but I've seen that happen in my server logs where some AI crawler accesses my site map file and they like it.

16:27I don't know. I mean, I don't know what they do with it, but they read it. They do something with it. I've also seen that for my RSS files, which again is something that most websites have. So it's easy for these AI crawlers to notice and being like, oh, here's some more content that I could use for AI training. And that's perhaps something that the site owner wants. Okay. But do I need to use an XML-based format? Because I remember there were HTML site maps at some point. Yes, HTML site maps. That I think is very confusing to the average person because you probably know what XML is and what HTML is. But a lot of people, when they open an XML file, they see those weird brackets and they're like, oh, this is HTML or very similar, right? I mean, yeah, sure. It's very similar.

17:27It looks like this weird programming stuff. And they're called the same, but an HTML site map is basically a, almost like a map of your website for users. It's not something that replaces an XML site map file. Okay. So, of course, if someone is crawling your website and they find an HTML site map file, they can crawl those links like any other link on a website. So that's helpful. But it's not the case that the HTML site map can be submitted as a site map file. It can't be processed in one-to-one the same way as a kind of XML site map file because it doesn't have this strict structure to it. It's basically a collection of links and collection of links can be any kind of thing. But that still would help crawling because from this place, we can find all the links as well. It's just not a structure, then it's not as clear and clean cut as an XML site map, I guess.

18:35Yes, absolutely. And it's something, I think, that a lot of HTML site maps, they don't include everything. So if you have an e-commerce site, you wouldn't list all of your products in an HTML site map file. You would list maybe the categories so that people can find, go to the right place, but then pick out the product individually. So it's similar, but not exactly the same. Would that need for such a structured site map as an XML file that go away with, let's say, like, inclusion of an LLM's TXT? Oh, yeah. There we have it. LLM's TXT, the new site map file for AI crawlers. So we've talked about LLM's text for a few times, and I'm sure we're going to continue talking about it for a long time. Basically, LLM's text is a markdown file, which sometimes includes links to different

19:40pieces of content. And it's almost like an HTML site map, but the hope is that an AI system uses it to understand your website a little bit better. I think the hope is bigger than the reality, but maybe that will change at some point. Who knows? But it's basically a text file that has various links to parts of your website, which is kind of similar to an HTML site map. It probably wouldn't include all of the pages on your website, but rather more the categories. And I think, like, from a functional point of view, you can't use those as a site map file in our systems because it's just not the strict format, but it could be at some point that our systems or search systems in general process markdown files. And maybe we will also find those links there and then be able to follow them. So I would consider it more as, on the one hand, it's hopeful, like currently none of

20:43this happens, but it could be the case in the future that it's more like an HTML site map file, which has a lot of links, but it's not meant to be compromised. Okay. All right. So it's not going away in favor of LLM's text. I think I mean, I don't think so. Who knows what the future brings like next week, the site maps team will send us an email and be like, Hey, LLM's text is a new site map. I don't know. I think from a practical point of view for anyone who's making websites, you should not kind of focus your energy on this hopeful state that maybe in the future, it will be different. You should focus on what actually works now, what is documented now. So it's fine if you want to play around or if your CMS makes LLM's text files automatically, but I would not rely on it. I would not treat that as a priority.

21:44I would kind of see it as like, well, who knows what will happen in five years or whatever. I don't know. Internet time is weird. Okay. Who knows what will happen, but it's also something where I think the usefulness of site map files is not going to go away completely just because of that comprehensive nature of the site map file. That makes sense. All right. Now to the question from the beginning, why does Search Console sometimes show couldn't fetch even though the site map XML is valid because you can validate that. That's the nice thing about such a structured format. So it's valid. It's publicly accessible and it's linked from the robot's TXT and yet Search Console says can't fetch sometimes. Can't fetch. Yes. We actually see this a lot in the forums and there are primarily two reasons for that. One could be with regards to host load.

22:45We've talked about that in the past, like how much Google systems are able to crawl from a website and it could be the case that like we don't have any time to crawl the site map file because we're too busy with other things. That can happen. Then we would also position that as couldn't fetch because we didn't have time to actually fetch it. The other also related to host load is kind of the crawl demand side where if our system say we don't actually have any need to crawl a lot from this website, we're just going to skip the site map file because we got enough already. The crawl demand is very often based on the perceived quality of a website and that can have a really large impact on how much we crawl in index from a website. It's not purely a technical thing. Sometimes it's that our systems assume that the overall quality of this website is not fantastic. Therefore, we're not going to spend a lot of time crawling and indexing the content.

23:47Therefore, we're not going to bother with the site map file at the moment. If we see over time that the quality of the website improves significantly, then yes, we will go off and use that site map file. Maybe we just don't want to. Okay. It's not just a technical thing from my perspective or my site's perspective. It's also, do we actually need the site map file and do we actually want the site map file as well? Yeah. Definitely. I think that answers that question. Thanks a lot for running through the entire history of site maps, basically. I think there's one thing that we didn't mention and that you can also do a Treflang in it. We had an a Treflang episode at some point. I think that's also an aspect, but I don't think I'm wondering how many people are using that. I would love to hear some comments from our listeners out there if they're using site maps to also link to different language versions or if they are using other mechanisms

24:50to do so. Cool. Yeah. There's also image and video site, my files that you can do for the images and videos. I don't know how important those are nowadays because it's easier for us to recognize images and videos on pages in the meantime, but it is something that is still supported so you can't see. Nice. We talked about HTML site maps, XML site maps. We talked about the difference between site maps and RSS feeds and we talked a little bit about this funky error and search console. I think my site map needs our met. Cool. With that, I'd like to say thank you so, so much John for being here with me and talking about site maps and also thanks a lot for all of you who are listening. Let us know in the comments below if you have any other questions regarding site maps or other formats and just leave us a like and subscribe to the podcast if you want to hear more from us.

25:51Thank you so much and have a great day. Goodbye. Bye. We've been having fun with these podcast episodes. I hope you, the listener, have found them both entertaining and insightful too. Feel free to drop us a note on LinkedIn or chat with us at one of our next events we go to. We have any thoughts? Let us know and of course do not forget to like and subscribe. Thank you so much for listening and goodbye.