Wagtail Sitemaps: Adding, Modifying and Removing Sitemap Entries

Hello and welcome to another lesson about learning Wagtail. In this lesson we're going to be learning about site maps. Now right now we do not have any site maps. Loaded up a sitemap, usually an XML file, sometimes an HTML file, is how a crawler or a scraper or A search engine like Google or Bing will actually find all of your pages. It's an easier way. Rather than having to grab all of the links and parse your HTML, they're just readily available there. So we actually need to get this installed, but before we do that, let's take a look at our URLs. py. Our URLs. py in here does not have any sign of sitemaps. Even if we do a search, there is no sign of sitemaps. Now this does actually come with Wagtail.

And all we have to do to enable this is go to base. py and in here routable. Page, that's where we left off last lesson. In this one we're going to add Wagtail Contrib sitemaps. And we also want to make sure we have Django Contrib sitemaps. So we need Django sitemaps and we also need Wagtail SiteMaps enabled. Now if we save that and we open up our URLs. py again. We can now import this quite easily. Do we have contrib in here? It does not look like we have contrib. So let's throw in from wagtail. contrib. sitemaps dot

views import sitemap and let's put it somewhere around here So this sitemap is very, very simple. All we're going to do is basically say sitemap. xml. So whenever someone hits sitemap. xml, it can go straight to sitemap. And that's it. We need to make sure this is reg X so that this can match exactly because we don't want Slight map. xmlish and we don't want it with a trailing slash, so that's all there is to it. Now I need to go and Start my server, so Python 3 manage uppy run server and when I open up my browser

and go to sitemap. xml You will see I have a sitemap of all of our pages. So I've got localhosts, I've got localhost about, I've got localhost blog, blog post one, blog post two. So in less than what five minutes we got a site map up and running and Wegta will take care of all the details. So if you have a page that's in draft mode, guess what? It does not show up in here. That's pretty beautiful. If you have a new edition of a page that's being published to a moderator, that does not show up yet. Although if that page is already published, that URL will obviously show up, just the new content won't show up. Again, I'm really just listing off goodies that comes with Wagtail CMS. But there's one that we didn't get to see in here. And if we go to localhost 8000 slash blog slash

latest This page exists too. Now this comes from our previous lesson where we were talking about routable page mix-ins. And basically creating spin-off pages from our existing Wagtail pages. Now this one is blog/slash latest and it does not exist in our sitemap. Now we're going to want to make that exist. Now believe it or not, this is actually incredibly easy to do. So let's open up our blog models And in our listing page where we have our routable page, all we want to do here is add a function called getSitMapURLs. Self request. Now essentially all this is doing is it's passing back a little bit of additional data.

From each URL and this gets called on every Wagtail page and it gives us a location and a last modified. That's it. We could add priority and stuff in there if we wanted to, but we're not going to. In this example, all we're going to do is add latest in there. So what we need to do here is we need to grab our sitemap. So sitemap super dot get sitemap URLs and we need to pass it the request. And then simply return sitemap. So at this point this is doing absolutely nothing. This is just bubbling up to the Wagtail page. Which we get from here. And then it's going to call this method on that object or that class. And it's going to say, oh, okay, you want the pages? Get them and then return them. So really at this point it's useless What we want to do is we want to add one more page in here.

We want to add our routable page in here. So let's do sitemap. append. Let's add a dictionary and we have a location in here. And that's going to be something we don't know yet. And let's add a last mod and something we also don't know yet. So what we want to do here is we want the Well technically we we we want to reverse the entire subpage, but we actually need to be able to get that full URL. So if you were with me in that last lesson, when we pass the entire Subpage for this crazy method name here. Actually we use this name. It actually just showed up with latest slash and that was it. In fact, I'm gonna ditch that because I don't want that in there anymore. Because it only gave us latest

slash, we need to figure out what the rest of this URL is. So we can do that with self. full underscore URL. plus self dot reverse subpage, which is what we did in our last lesson as well. And this is just reversing a subpage for a routable page. And we can do latest posts. And that's it. And that latest post, I just got that from the name up here. Again, if that name does not exist, if this name does not exist. It's going to default to this one, in which case you would plop that in there. Gonna undo that though because the naming is just so much better this way Last modified.

Okay, so this is a little bit trickier because this page technically does not get modified. The blog listing page does. This is the class that we're working with. That gets modified. However, the rotable page does not necessarily get modified. Now, if we really wanted to get into this, we could get the latest blog post and we could get that modified date Because this is a listing page and this listing page is only ever going to change if a blog detail page is published, we can always check to see if there is a new blog detail page, check to see when that was last published at or when the latest revision was created at. However, we're going to be a little bit lazy and I'm just going to look for the blog listing page.

So basically, what I'm saying to search engines like Google is whenever the blog listing page is updated, also count the slash latest page as updated as well. And all I do for that is self. lastpublished at or self. latest revision created at. Okay, can I make that a little bit smaller for you so you can see it on one screen? Haha! I hope that's not too small for you to see. So here we have sitemap. All we did was run super on it We appended to this, so we added a dictionary to the sitemap, and we returned the sitemap. Okay, so Open up my terminal. Life is good in the terminal. And when I refresh, watch this. Wait for it. Wait for it.

There it is. Right there. HTTP localhost slash blog slash latest. Last modified 2019. February 7th, the same day as the blog was last modified. Now today is currently the 8th, so one day ahead of that last modified date. So let's go ahead and change this. Boop boop boop boo boo we have a blog in here. Let's edit our blog and let's just create a new revision. Saving the page and oh look at that both blog and blog latest updated to today, which is February 8th, 2019.

Now, lastly, sometimes you're going to create a page where it makes sense for the CMS to have some sort of intermediary page, but you don't actually want to have anything on it. Maybe all it does is redirect and it is simply a holder for your blog. So let's say you had a blog listing page, but you didn't want that listing page to actually be visible. Well you don't want search engines hitting your website. com slash blog because well you don't have a template for it, you don't want anything to do with it, maybe all it does is redirect the first blog post. Depends on your design of your website. Regardless, to get rid of this, all you have to do is return an empty list. So if I save that and I refresh, we're going to see that both latest and blog do not show up in here anymore.

So these two here will disappear. And look at that. So we have our home page, we have about, we have blog post one, and we have blog post two, but no listing page and no latest posts page. And there's just one more thing to learn in here, so I'm actually going to undo that, but I'm going to leave a comment in here. Return. And this is going to say. uh uncomment to have no sitemap for this page. One more thing, we can also add priority sometimes search engines like a little priority. So we can say this page is a priority. And priority is a number between 0 and 1. So let's say this one is 0. 9 I'm gonna go ahead, refresh our page, and we're going to see that

our blog does not have it, but our blog latest does have it, and that's because that's exactly what we gave it. We said the priority is 0. 9, it's 90% priority, but the blog is just a regular page. So that, my developer friends, is Honestly, all there is to know about sitemaps in Wagtail. So all you have to do is enable your sitemap, and then you can overwrite your sitemaps with a function called getSitemapURLs. And you can add to it by simply running super, appending, and then returning your site map. Or if you want to get nothing, if you want this page to be completely hidden from search engines. Through a site map anyways, not through regular crawling, all you'd have to do is return an empty list. And that's all there is to know Hello, my name is Caleb Tollen.

I am the voice behind the video. If you like this video, feel free to subscribe. You can share, you can like, you can comment, you can do all sorts of things. If you want to explore this concept just a little further, you can always hit up docs. wagtail. io or check out github. com slash wagtail slash wagtail and look at the source code Or if you're feeling a little bit crazy, you can join the rest of the Wagtail community on Slack. And you can find that link on Wagtail. io.