Django Memcached with Django-Cache-Memoize

Welcome back in today’s discussions here at {{ PLP }}, it’s quite some time I’m not writing here, I’m still busy at the moment developing the Ipaskil web development project and few other things as well. Anyways, it’s nice to be back.

Today, I will be sharing it with you the importance of caching expensive functions in Django using Memcached which is natively supported by Django itself. Memcached stores result in memory which is the fastest way to retrieve database query results and expensive calculations without re-hitting the same function with the same results and database all over again for a certain period of time.

In the meantime, the django-cache-memoize which is the Django utility for a memoization decorator that uses the Django cache framework. This utility will bring easy implementation of the Django caching framework.

Just a quick glance at the rear, we have previous discussions about the Django Celery with Real-time Monitoring Tasks Using Flower that provides asynchronous task scheduler and monitor each tasks in real-time.

Introduction

The power of caching expensive and memory-intensive calculations from your Python functions will bring benefits for you as a developer to your web projects in the future. I would highly recommend that from now on-wards, use caching for your web projects in Django.

In Django, the preferred way of caching technology in memory is the Memcached which is an open-source project that was used by big companies like Facebook.

So, prepare your self once again and stay focus because this would be exciting and fun learning with {{ PLP }}.

Getting Started

So, let’s get started on this, we will be using our existing Ubuntu server to install first the Memcached and the django-cache-memoize written for Python 3.7 + and with Django 2.2 +.

During this discussion, you will learn how to implement installing and setting up the django-cache-memoize library for Python and Django.

Step 1: Install Memcached

Always, remember to update your Ubuntu server before installing anything. First, execute the command below. I assume that you have full access to your Ubuntu web server for you to install anything.

1
2
sudo apt update
sudo apt install memcached

It should be straightforward installing for the above commands. Next, install this extra Memcached tools.

1
sudo apt install libmemcached-tools

If prompts with [Y/n], type in Y and press enter to continue. Next, we need to install a Memcached binding which is the python-memcached.

1
pip3 install python-memcached

We are ready to use the Memcached technology for our Django web project. Now, we need to test if the Memcached is running by executing the command below.

1
sudo netstat -plunt

The expected results after the netstat -plunt command has been executed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
(env_dev) root@localhost:~/dev# sudo netstat -plunt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      
tcp        0      0 172.104.190.249:9001    0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:5555            0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      
tcp6       0      0 :::5672                 :::*                    LISTEN      
tcp6       0      0 ::1:6379                :::*                    LISTEN      
tcp6       0      0 :::8079                 :::*                    LISTEN      
tcp6       0      0 :::80                   :::*                    LISTEN      
tcp6       0      0 :::4369                 :::*                    LISTEN      
tcp6       0      0 :::5555                 :::*                    LISTEN      
tcp6       0      0 :::22                   :::*                    LISTEN      
udp    33024      0 127.0.0.53:53           0.0.0.0:*

Great!, it’s now up and running. You don’t need to worry about the Memcached when your server has been restarted as this works as a daemon within your Ubuntu web server, meaning, it will auto-start this program when your Ubuntu server booted up.

Step 2: Install the django-cache-memoize

Next, install this django-cache-memoize library and demonstrate to you on how we can use this library to easily implement the Django caching for us.

1
pip3 install django-cache-memoize

With this library, I will simplify how to use this for your Django projects implementation. Please remember to include this library to your requirements.txt so that you won’t forget this library in your Django project as one of the dependencies.

Step 3: Adding Memcached to your settings.py

Afterward, include this Memcached to your settings.py, COPY exactly the code snippets below and paste it somewhere in your settings.py.

1
2
3
4
5
6
7
8
9
10
11
# Memcached python-memcached binding, limit it to 2 mb.
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    'OPTIONS': {
            'server_max_value_length': 1024 * 1024,
        },
    'KEY_PREFIX': 'PLP'
    }
}

The KEY_PREFIX will be your keyword to identify each key in the Memcached that keys belong to this specific project. Although, it’s not required I recommend this for your project to properly identified within the Memcached.

Step 4: Start Caching a Function

Once again, we will have a live demonstration for your understanding and how really it behaves. We will be creating a Python function and will be using the existing ContactUs model for caching the expensive queries as an example.

Caching the SELECT statement query in Django.

Open up your myroot/vmain.py and create a new Django View to display the entries from the database.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
def cache_basic_crud_list_view(request):
    """
    This is how we cache an expensive Django queries to avoid
    keep hitting your database server with the same results.
    """

    if request.method == "GET":

        # Call a cachable function.
        db_data = get_contact_us_list()

        return render(request, 'myroot/cache_basic_crud_list.html',
                      {
                          'title': 'Django Memcached Basic CRUD: Manage Table Rows using DataTables',
                          'meta_desc': 'Learn how to use the Django Memcached to display the database table rows using DataTables.',
                          'db_data': db_data
                       })

Then, we will use the myroot/common.py to separate all the common functions to organize it there. But, of course, you can have your naming and organizing style, and it’s up to you.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
from django.conf import settings
from cache_memoize import cache_memoize

# Call all myroot app properties
from myroot.models import ContactUs


@cache_memoize(3600)
def get_contact_us_list():
    """
    Function to get the contact us list, cache this function
    for 3600 seconds equivalent to 1 hr, to relieve the stress
    of the database server.
    """

    # fetch the latest 50 rows as an example.
    data = ContactUs.objects.filter(is_deleted=False).order_by('-id')[:50]
    return data

Look, this is how we use the django-cache-memoize by importing the from cache_memoize import cache_memoize and use it using the decorator on top of your function name which is the @cache_memoize(), the 3600 is the number of seconds it will expire automatically in memory.

No need for you to worry about how it will expire, the Memcached itself will manage that, if you wanted to expire it longer, you can always increase the number of seconds or leave it empty without the number of seconds to leave in memory (not recommended) until your server will be restarted. Keep in mind that the Memcached all stored cache will be emptied when your server has restarted.

Of-course, your wondering how will it cache again when my function cache expires? no worries, it will be cache in memory automatically for 3600 seconds or an equivalent to 1 hr as this example implies. So, it will be kept caching and expires again and again.

The only thing is that, the first request will be slightly slower because it will normally hit the database query to collect the rows once again and then cache. But then, when you load the 2nd time until it expires, this will be faster as it will not hit the database query as the results itself has already been in the memory and ready to be served to your page or any function usage.

Next, create a new URL router for this new Django View, open up your dev/urls.py and insert this new link below.

1
2
3
4
# Django Caching related links
    path('cache-django-basic-crud-list/',
         myroot.views.vmain.cache_basic_crud_list_view,
         name='cache_basic_crud_list'),

And then at the myroot/views/vmain.py, don’t forget to import the common.py for us to use the newly created cachable function.

1
2
3
4
...
# All the common functions to be loaded here.
from myroot.common import get_contact_us_list
...

Save all your files and get ready to upload all your new and modified files to the web server.

Test the Caching

Now, upload all the your modified files using the FileZilla as always, we would like to access our demo page at Django Memcached Basic CRUD: Manage Table Rows using DataTables and see how this page loads, even if you keep refreshing this page it will not re-hitting the database server all over again until it expires for 1 hour (3600 seconds).

In this case, in your MySQL database table called dev_contact_us, edit manually at least one row and then refresh it, when you keep refreshing it, it will not change the value of that column, that means, your caching works perfectly, congratulations!

Step 5: Invalidating a Cache

The caching technology is great and one of the very important pieces of technology to make your Django web projects to be powerful and faster without overloading our server memory resources and the database as well.

Although, the caching is a bit complicated if you are not careful about using with it. It’s like a fire that helps us a lot but it’s dangerous when using it irresponsibly.

In this case, there is always a time we wanted to clear the specific function cached results from the memory by invalidating it and replace with the latest query results. This means, that someone modified the row when you have an edit transaction from your web project. Let’s do this, prepare your self.

Create a new Django view for Editing the row.

Now, open up your myroot/common.py once again for us to add this code snippets below.

1
2
3
4
5
6
7
8
9
10
11
12
@cache_memoize(3600)
def is_contact_us_id_exist(contact_us_id):
    """
    Function to check if the contact us id existed or not.
    Cache this for 3600 seconds equivalent to 1 hr, to relieve the stress
    of the database server.
    """

    # fetch the latest 50 rows as an example.
    if ContactUs.objects.filter(id=contact_us_id, is_deleted=False).exists():
        return True
    else:
        return False

Let’s cache this into memory for 1 hour, but, of course, it’s always up to you. But I recommend not to cache longer so that the Memcached will not be retracting it’s previous cached as one of mitigation of the program, meaning, if millions or even billions of keys stored in memory, the Memcached will automatically remove caching like a FIFO kind of technique First-in First-out, but, no need for us to worry about this under the hood of Memcached.

Next, open your myroot/vmain.py and insert the code snippets below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
def cache_basic_crud_update_row_view(request, id):
    """
    Renders the Django edit form page and execute the update statement.
    But, with invalidating the cache to clear the previous cached results.
    """

    if request.method == "GET":

        # Check if the row id existed or not.
        if is_contact_us_id_exist(id):

            # Edit the form data.
            edit_data = ContactUs.objects.get(id=id, is_deleted=False)
            formEdit = Basic_CRUD_Create_Form(instance=edit_data)

            return render(request, 'myroot/cache_basic_crud_update.html',
                          {
                              'title': "Django Memcached Basic CRUD Update Statement",
                              'meta_desc': "This is the actual example for Django Memcached using the django-cache-memoize invalidation, basic crud which is the update statement.",
                              'id': id,
                              'formEdit': formEdit
                           })
        else:
            raise Http404()

    data = dict()
    if request.method == "POST":

        # Get the  form modified data
        form_edit = Basic_CRUD_Create_Form(request.POST)
        id = request.POST.get('id')

        if form_edit.is_valid():

            # Check if the row still not deleted, get it from the cache memory
            if is_contact_us_id_exist(id):

                # Get the form edit instance
                update_data = ContactUs.objects.get(id=id, is_deleted=False)

                # Now, supply the form data to an instance
                form_edit = Basic_CRUD_Create_Form(request.POST, instance=update_data)
                form_edit.save()  # Finally save the form data

                # Here, to invalidate the previous cache results.
                get_contact_us_list.invalidate()
                is_contact_us_id_exist.invalidate(id)

                # Return some json response back to the user
                msg = """ Your data has been modified successfully, thank you! """
                data = dict_alert_msg('True', 'Awesome!', msg, 'success')

            else:
                # Return some json response back to the user
                msg = """ The data has no longer existed, the update has been aborted! """
                data = dict_alert_msg('True', 'Update Failed!', msg, 'error')
        else:

            # Extract form.errors
            msg = None
            msg = [(k, v[0]) for k, v in form_edit.errors.items()]
            data = dict_alert_msg('False', 'Oops, Error', msg, 'error')

        return JsonResponse(data)

And then, import the newly created function which is the is_contact_us_id_exist from the common.py.

1
2
# All the common functions to be loaded here.
from myroot.common import (get_contact_us_list, is_contact_us_id_exist)

Afterward, create a new URL router for the new edit page from the dev/urls.py.

1
2
path('cache_basic_crud/<int:id>/change/', myroot.views.vmain.cache_basic_crud_update_row_view,
         name='cache_change_basic_crud'),

The process is all set and ready to test it out, upload all the newly created and modified files to your web server or your local development environment.

Test the new edit page to clear the previous caching.

Time to test it out on how we clear the caching by invalidating them provided by the django-cache-memoize. Open up once an again the demo page at Django Memcached Basic CRUD: Manage Table Rows using DataTables and try to edit any rows and it should clear the previously cached results for the two functions which are the following.

1
2
3
# Here, to invalidate the previous cache results.
get_contact_us_list.invalidate()
is_contact_us_id_exist.invalidate(id)

Upon updating the row data and no need for you to import the library from django-cache-memoize when using this .invalidate() function.

1
from cache_memoize import cache_memoize

Take note on this, only import that library when your are calling this @cache_memoize() as a decorator to your function that you wanted to cache the results.

Always remember, that the @cache_memoize() decorator will ONLY work when you have something to RETURN results from the function, please remember that always.

You can continue the .invalidation() process clearing the previously cache results upon DELETE and CREATE process of your application when some rows have been deleted nor creating a new row as well and I leave it to you to continue .invalidate() the cache as your exercise.

You can download the source code from our GitHub repositories at https://github.com/pinoylearnpython/dev and stay tuned for any updates.

In the next event loop on {{ PLP }}.

Congratulations! your Django web projects are now getting more powerful and faster to load, you can continue caching functions only and not the class for this to work. Now, you have the awareness of how your database servers save a lot of resources most especially your web servers memory.

Besides, your code from now on-wards should be functional programming to make it all clean, clear, concise, efficient and easy to understand of what your web project in terms of code readability and reliability.

For those who’re not able to successfully launch your Django Memcached with django-cache-memoize tutorial that I’ve laid it before you, or you need more clarifications, don’t worry, leave a comment below and I’m happy to help you to succeed.

That’s all, have fun learning with {{ PLP }}.

To help Filipino students to learn Python programming language with Django to enhance their capabilities in developing robust web-based applications with practical and direct to the point tutorials, step-by-step with actual information that I provided for you. Leave a comment below or email me at [email protected], thank you!