The Django Sitemap Framework

In today’s event loop on {{ PLP }}, we’re about to discuss the importance of having a sitemap from our site that serves as one of the basic building blocks of a successful Search Engine Optimizations (SEO).

But, before anything else, let’s have a quick recap about our last discussions regarding the Django User Authentication System, in which, I’d lay out everything you need to know on how exactly it works and once you try it for your Django powered site, Nah, rest assured, you’ve got a peace of mind about the web security for your site.

What is a sitemap?

A sitemap is a list of pages or links that contained from a file called the sitemap.xml, the .xml file extension, means the Extensible Markup Language (XML) is the most preferred file format when you store all your internal links from your site. Which means, search engines like Google, Bing, Yahoo, etc. will crawl effectively your site’s internal links and possibly get indexed faster.

Who should use the sitemap?

Likewise, these are the most common and practical usage of sitemap based on my experience as well. I’m not an expert of the SEO industry, but, as a web developer, it’s one of the components and a responsibility to provide our clients or individual web projects we have to be known and searchable by the search engines out there.

Your site is too many dynamic pages: This means that, your site accepts publicly visible submitted posts either you or your user’s activity, or your site is a business directory, or classified ads site, or eCommerce sites, etc.

Your site is newly published: But wait, I recommend you to use the Google’s Search Console which is for the webmaster to manage your site’s SEO issues and you can submit a sitemap there as well so that Google can crawl it asap.

Your site is NOT a CMS type: CMS means (Content Management System), WordPress is one of the most popular CMS sites out there and my blog itself is a WordPress site I use, because, most of the SEO related things, which you can install the plugins like the Yoast SEO which takes care of your SEO including the auto-generated sitemaps. You can have a look at how the Yoast SEO generates the sitemap for my blog.

Introduction

In the same way, Django provided the high level SiteMap Framework which will auto-generate a sitemap XML file easy for all the Django powered sites out there, as a Django developer, we are grateful that the Django core developers provide us with this framework to ease out common tasks among us.

Today, I will guide you on how we can implement this Django SiteMap Framework for your Django powered site project, and give you some good techniques that I applied personally for all of my Django projects.

So, prepare your self once again and stay focus because this would be exciting and fun learning with {{ PLP }}.

Getting Started

First of all, we need to include the Django SiteMap Framework from the INSTALLED_APPS for us to use this framework. Now, open up your dev/dev/settings.py and insert the sitemap library.

1
2
3
4
5
6
# Application definition
INSTALLED_APPS = [
     ...
     'django.contrib.sitemaps',
     ...
]

Another key point, you need to ensure that you need to enable the Django Site Framework as mentioned in our previous discussions about the Django User Authentication System with reCAPTCHA v3, JQuery and AJAX Method on Step 3.

Step 1: Create the sitemap.py

Moreover, we need to create a new Python file, name it as sitemap.py and save it under the dev/myroot/views/ directory. Then, COPY exactly the code snippets below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
from django.conf import settings
from django.contrib import sitemaps
from django.urls import reverse
from django.utils.text import slugify

# Call myroot properties
from myroot.models import ContactUs


class ContactUsSitemap(sitemaps.Sitemap):
    priority = 0.5
    changefreq = 'daily'
    protocol = "https"

    def items(self):
        contacts = ContactUs.objects.filter(is_deleted=False).order_by('-id')[:250]  # latest 250 new entries
        return contacts

    def lastmod(self, item):
        mod_date = item.submitted  # Actual datetime field
        return mod_date

    def location(self, item):
        url = '/' + slugify(item.full_name) + '-' + str(item.id)  # Actual URL of the individual page
        return url


class StaticViewSitemap(sitemaps.Sitemap):
    priority = 0.5
    changefreq = 'daily'
    protocol = "https"

    def items(self):
        return ['helloworld', 'basic_crud_create', 'basic_crud_list', 'register', 'password_reset', 'login']

    def location(self, item):
        return reverse(item)

For now, we only have one dynamic content based on the user’s submitted dummy data from our live demo which is the Django Basic CRUD: Manage Table Rows using DataTables, where we use this as an example data to create a dynamic content and we ensure that search engines like Google will index each dynamically created content to be searchable from their search results.

To explain, the priority = 0.5 is the default page priority, the value range is between 0.0 to 1.0, in which the 1.0 is the top priority indicator for the search engine crawlers to prioritize indexing, but, you can’t put all as 1.0 as crawlers may still vary accordingly. The safe priority rate would be 0.5 for each page is adequate.

The changefreq possible values are, ‘always‘, ‘hourly‘, ‘daily‘, ‘weekly‘, ‘monthly‘, ‘yearly‘, and ‘never‘ in which you tell the crawlers to revisit the specific page based on the modifications frequency for that page. But, again, it’s not guaranteed that the crawler will re-crawl so frequent if your changefreq value is hourly as an example. The daily changefreq value is adequate.

Likewise, the protocol will only have 2 values either http or https as well.

Step 2: Create a new URL for the sitemap.xml

In this example, as you can see from the sitemap.py, we have created one dynamic source of URLs specifically from the ContactUs model, and then the other example we have is the Static URLs like the ‘helloworld‘, ‘basic_crud_create‘, ‘basic_crud_list‘, ‘register‘, ‘password_reset‘, and the ‘login‘ pages.

By the way, please remember that the sitemap maximum URLs must not be exceeded to 50,000 URLs or an equivalent to 50 MB file size. So, if you’re site is too large enough specifically if you have dynamic postings that you want to index it by Google, make it sure that you retrieve the latest dynamic URLs submitted to prioritize it.

Now, open up your dev/dev/urls.py and create a new URL, but, before that, import the mandatory sitemap libraries as usual on the top section of the urls.py Python script.

1
2
3
4
5
6
7
8
9
10
11
...
from django.contrib.sitemaps.views import sitemap

# Call SiteMap properties
from myroot.views.sitemap import (ContactUsSitemap, StaticViewSitemap)

sitemaps = {
    'contacts': ContactUsSitemap,
    'static': StaticViewSitemap,
}
...

And then at the bottom section of the urls.py, COPY exactly the code snippets below.

1
2
3
4
5
...
# Django auto generate sitemap.xml file
path('sitemap.xml', sitemap, {'sitemaps': sitemaps},
     name='django.contrib.sitemaps.views.sitemap'),
...

Moreover, when you have multiple dynamic URLs source, you can add a new class from the sitemap.py and correlate it with the corresponding Django Models, just COPY the entire class ContactUsSitemap(sitemaps.Sitemap):, change the class name and the fields accordingly based on the Django Models you intend to auto-generate the URLs and include it from the sitemap.xml respectively.

In addition, once you added a new class from the sitemap.py, don’t forget to include it from the urls.py, as a quick overview, include new sitemap class from the myroot.views.sitemap and then at the sitemaps = {} context, include it as well. Below is the sample for better illustration.

1
2
3
4
5
6
7
8
9
10
...
# Call SiteMap properties
from myroot.views.sitemap import (ContactUsSitemap, StaticViewSitemap, YOUR_NEW_SITEMAP_CLASS)

sitemaps = {
    'contacts': ContactUsSitemap,
    'static': StaticViewSitemap,
    'new_site_map': YOUR_NEW_SITEMAP_CLASS,
}
...

Next, upload all your new and modified files to your web server using FileZilla as usual. And don’t forget, please reload your daphne web service after all the files been uploaded successfully.

1
daphne -b 172.104.190.249 -p 8000 dev.asgi:application

Again, just change the IP address with your own IP address or even the port number which is the 8000 with your own port number.

Step 3: Try the SiteMap.xml Demo

Finally, and the most awaited part, access the sitemap.xml from your web browser for you to see how the Django SiteMap Framework works, the scripts works perfectly fine, you can try to submit new dummy data from the CRUD Create demo and check back the sitemap.xml, the new dynamic URL will be automatically listed out as one of the URLs that need to be index by the search engines.

Try now the live SiteMap.xml from your web browser.

Step 4: Submit SiteMap.xml to Google’s Search Console

I recommend you to get sign up for the Google’s Search Console where you can manage your site’s issues, traffics, search queries from the searchers on what keywords they are looking for, etc. There are so many benefits of using Google’s webmaster tools.

The screenshot below as an example to where you can submit your sitemap.xml link to Google’s Search Console.

Indeed, the Django SiteMap Framework is one of the core frameworks from the Django ecosystem itself that aims to help us Django developers to east out the burdens in building web-based applications and instead Django is there to keep supporting us all the ways during the developing stage.

You can download the source code from our GitHub repositories at https://github.com/pinoylearnpython/dev and stay tuned for any updates.

In the next event loop on {{ PLP }}.

Congratulations!, now, we all know the usage and the importance of the sitemap for our site to be visible on search engines like Google, Bing, Yahoo, etc. It’s our responsibility as web developers to have knowledge on Search Engine Optimization (SEO), so, we’re not just building a Django powered site without this sitemap component for the search engines.

Although, the crawlers are smart enough to crawl all your pages, it’s recommended for Google Web Masters to submit the site’s sitemap especially when you have too many URLs. The Django versions would be 2.1+ and the Python 3.7+ as well.

For those who’re not able to successfully launch your Django SiteMap Framework tutorial that I’ve laid it before you, or you need more clarifications, don’t worry, leave a comment below and I’m happy to help you to succeed.

See you in the next event loop on the Supervisor Installation and Configuration on Ubuntu 18.04 LTS.

That’s all, have fun learning with {{ PLP }}.

To help Filipino students to learn Python programming language with Django to enhance their capabilities in developing robust web-based applications with practical and direct to the point tutorials, step-by-step with actual information that I provided for you. Leave a comment below or email me at [email protected], thank you!