Using Django rest framework with auth, pagination and filtering

I will show how to create simple yet efficient APIs using Django REST framework. Starting with basic examples and progressing to more complex ones.

Project setup

To begin, we will use pipenv to set up our environment. First, we will clone the base repository and then run a few additional commands. One of the benefits of using pipenv is that it automatically creates a virtual environment for us.

git clone -b minimal https://github.com/MounirMesselmeni/restframework-tutorial.git rest_tuto
cd rest_tuto
# installing pipenv as system package
pip install pipenv
# installing project dependencies, use --python 3.6 if it's installed
pipenv install --python3
# activating virtualenv via pipenv
pipenv shell
# run migration
./manage.py migrate
# create a superuser to play with your models via admin
./manage.py createsuperuser
# run django server
./manage.py runserver

Now, you can examine the structure of the models in the courses/models.py file. Then, proceed to the admin area and add some data.

Starting the API

Now, let's create our first API endpoint and ensure it is functioning properly. To start, create a new Python module named api.

mkdir api
touch api/__init__.py
touch api/serializers.py
touch api/views.py
touch api/urls.py

Next, we will create a serializer for the Teacher model in "api/serializers.py". The serializer will instruct Django REST framework on how to present our data in various formats (such as HTML, JSON, or XML) and how to handle data parsing between these formats and Python.

from rest_framework import serializers

from courses.models import Teacher


class TeacherSerializer(serializers.ModelSerializer):
    class Meta:
        model = Teacher
        fields = [
            'id',
            'name',
            'speciality',
        ]

We will be using ModelSerializer which is quite similar to Django's ModelForm, it's a serializer for a model, that helps us to write less code and automatically generates fields based on the model fields including validation and field types.

Now, let's utilize Django REST framework viewsets. ViewSets allows us to perform CRUD operations using a single endpoint:

Create via POST requests
Read via GET requests
Update via PUT requests
Delete via DELETE requests

We can choose to disable some of them if not needed, but for now, we will use all of them.

Let's create our first viewset in api/views.py.

from rest_framework import viewsets

from . import serializers
from courses import models as courses_models


class TeacherViewSet(viewsets.ModelViewSet):
    queryset = courses_models.Teacher.objects.all()
    serializer_class = serializers.TeacherSerializer

We need to specify a queryset for the ViewSet, which will also be used for listing teachers. If we want to display only teachers with specific filters, we can make that change here.

Now, we will inform Django REST framework to publish our first endpoint or viewset. This can be done manually by adding the viewset to the urls.py file in the traditional Django way, but it's better to use Django REST framework routers for this. They are useful as they will automatically generate URLs with an ID parameter for retrieving a specific teacher, for example.

Let's add this to api.urls.py.

from django.conf.urls import url, include
from rest_framework.routers import DefaultRouter

from . import views

router = DefaultRouter()
router.register(r'teachers', views.TeacherViewSet)

urlpatterns = [
    url(r'^', include(router.urls)),
    url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework'))
]

Now we have to include api.urls inside our project main urls, rest_tuto/urls.py

from django.contrib import admin
from django.urls import path, include
import api.urls

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/', include(api.urls)),
]

The api-auth is used for API authentication, we will cover that later.

Now, you can open your browser and navigate to "localhost:8000/api". The HTML displayed is generated by Django REST framework and lists all available endpoints. You're currently viewing the HTML format, but you can also view the JSON format by adding ?format=json to the URL.

Go to the teachers API and experiment with it. Add some data using curl or the Django REST framework HTML form.

# Create teacher
curl -d '{"name": "John", "speciality": "Physics"}' -H "Content-Type: application/json" -X POST http://localhost:8000/api/teachers/

# List teachers
curl -X GET http://localhost:8000/api/teachers/

# Get one teacher
curl -X GET http://localhost:8000/api/teachers/1/

# Delete teacher
curl -X DELETE http://localhost:8000/api/teachers/1/

# Create teacher another time
curl -d '{"name": "John", "speciality": "Physics"}' -H "Content-Type: application/json" -X POST http://localhost:8000/api/teachers/

# Update one teacher
curl -d '{"name": "John2", "speciality": "Physics"}' -H "Content-Type: application/json" -X PUT http://localhost:8000/api/teachers/2/

# Create teacher with missing data
curl -d '{"speciality": "Physics"}' -H "Content-Type: application/json" -X POST http://localhost:8000/api/teachers/
# Will complain and show errors in json

Now, let's explore relationships further, based on the models we have already set up.

Custom serializer method

t can be useful for example, for calculated fields. In the case of students, we have their birthdate and we want to return their calculated age for each student. To achieve this, we can use the Django REST framework serializer method field SerializerMethodField. The default method name will be get_fieldname where fieldname is the name of your field.


class StudentSerializer(serializers.ModelSerializer):
    age = serializers.SerializerMethodField()

    def get_age(self, obj):
        today = datetime.date.today()
        return (
            today.year - obj.birth_date.year - ((today.month, today.day) < (obj.birth_date.month,
                                                                            obj.birth_date.day))
        )
    class Meta:
        model = Student
        fields = [
            'id',
            'name',
            'birth_date',
            'current_level',
            'age',
        ]

Note that obj will be an instance of Student model.

Relationships

By default, Django REST framework will display related fields as IDs. If you want to include the entire related object, you will need to specify which serializer should be used. In the case of many-to-many relationships, you must pass the serializer with "many=True"

Let's take a look at how the "CourseSerializer" should be implemented:

class CourseSerializer(serializers.ModelSerializer):
    teacher = TeacherSerializer()
    students = StudentSerializer(many=True, read_only=True)

    class Meta:
        model = Course
        fields = [
            'id',
            'name',
            'teacher',
            'level',
            'students',
        ]

We used read_only=True to tell drf to use that field only while reading data, for creation this field will be ignored, if disabled drf will raise an exception when trying to post data complaining that you should deal on your own with m2m fields in the perfom_create method.

Here an overview of how api app looks like right now, as it's time to play around with it.

api/serializers.py

import datetime
from rest_framework import serializers

from courses.models import Teacher, Student, Course


class TeacherSerializer(serializers.ModelSerializer):
    class Meta:
        model = Teacher
        fields = [
            'id',
            'name',
            'speciality',
        ]


class StudentSerializer(serializers.ModelSerializer):
    age = serializers.SerializerMethodField()

    def get_age(self, obj):
        today = datetime.date.today()
        return (
            today.year - obj.birth_date.year - ((today.month, today.day) < (obj.birth_date.month,
                                                                            obj.birth_date.day))
        )
    class Meta:
        model = Student
        fields = [
            'id',
            'name',
            'birth_date',
            'current_level',
            'age',
        ]


class CourseSerializer(serializers.ModelSerializer):
    teacher = TeacherSerializer()
    students = StudentSerializer(many=True, read_only=True)

    class Meta:
        model = Course
        fields = [
            'id',
            'name',
            'teacher',
            'level',
            'students',
        ]

api/views.py

from rest_framework import viewsets

from . import serializers
from courses import models as courses_models


class TeacherViewSet(viewsets.ModelViewSet):
    queryset = courses_models.Teacher.objects.all()
    serializer_class = serializers.TeacherSerializer


class StudentViewSet(viewsets.ModelViewSet):
    queryset = courses_models.Student.objects.all()
    serializer_class = serializers.StudentSerializer


class CourseViewSet(viewsets.ModelViewSet):
    queryset = courses_models.Course.objects.all()
    serializer_class = serializers.CourseSerializer

api/urls.py

from django.conf.urls import url, include
from rest_framework.routers import DefaultRouter

from . import views

router = DefaultRouter()
router.register(r'teachers', views.TeacherViewSet)
router.register(r'students', views.StudentViewSet)
router.register(r'courses', views.CourseViewSet)

urlpatterns = [
    url(r'^', include(router.urls)),
    url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework'))
]

Filtering

To help with quick filtering implementation, we will use the django-filter package. It's already installed, but let's add it to our INSTALLED_APPS.

INSTALLED_APPS = [
    ...
    'django_filters',
]

Now, we can configure Django REST framework to use django-filter as our global filter backend. This can also be done on a per-view level. Add the following to your settings.py file.

REST_FRAMEWORK = {
    'DEFAULT_FILTER_BACKENDS': (
        'django_filters.rest_framework.DjangoFilterBackend',
    ),
}

Create a api/filters.py where we will add filtering for the teacher endpoint

import django_filters
from django_filters import rest_framework as filters

from courses import models 

class TeacherFilter(filters.FilterSet):
    name = django_filters.CharFilter(lookup_expr='icontains')

    class Meta:
        model = models.Teacher
        fields = ['name']

The last thing we need is to specify the filter_class in the TeacherViewSet

class TeacherViewSet(viewsets.ModelViewSet):
    queryset = courses_models.Teacher.objects.all()
    serializer_class = serializers.TeacherSerializer
    filter_class = filters.TeacherFilter

The lookup_expr setting specifies to django-filter which Django ORM lookup it will use. Let's test it out using curl and see the results.

curl -X GET http://localhost:8000/api/teachers/\?name\=john

Now, we want to filter for students based on their age, we will need to use a custom method for filtering as we will calculate the right year to look for. To calculate the difference in years, we will need to install python-dateutil.

pipenv install python-dateutil

Then modify our filters.py:

import datetime
from dateutil.relativedelta import relativedelta
import django_filters
from django_filters import rest_framework as filters

from courses import models 

class TeacherFilter(filters.FilterSet):
    name = django_filters.CharFilter(lookup_expr='icontains')

    class Meta:
        model = models.Teacher
        fields = ['name']


class StudentFilter(filters.FilterSet):
    age = django_filters.NumberFilter(method='age_filter')

    def age_filter(self, queryset, name, value):
        today = datetime.date.today()
        age = (today - relativedelta(years=value)).year
        return queryset.filter(birth_date__year=age)

    class Meta:
        model = models.Student
        fields = ['age']

Let's try it out, maybe you will need to add some students, via curl, drf forms, Django admin or Django shell

curl -X GET http://localhost:8000/api/students/?age=20

Pagination

Configure pagination in settings.py, pagination can help when dealing with a lot of data, imagine we have a list of 10000 students, is will not be good to return this big list, we can return 100 per 100.

Be careful as pagination will modify the returned format:

{
    "count": 1023
    "next": "https://api.example.org/accounts/?limit=100&offset=500",
    "previous": "https://api.example.org/accounts/?limit=100&offset=300",
    "results": [
       …
    ]
}

REST_FRAMEWORK = {
    'DEFAULT_FILTER_BACKENDS': ('django_filters.rest_framework.DjangoFilterBackend',),
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.LimitOffsetPagination',
    'PAGE_SIZE': 100,
}

Authentication

It's best to use a session if you would like to use DRF build in HTML for testing and trying out the API endpoints. There are many authentication backends supported by DRF, there are also some other third-party apps that provide more backends like JWT (You can check)

We will now configure session and token authentication, let's configure DRF in settings.py

REST_FRAMEWORK = {
    'DEFAULT_FILTER_BACKENDS': ('django_filters.rest_framework.DjangoFilterBackend',),
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.LimitOffsetPagination',
    'PAGE_SIZE': 100,
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.TokenAuthentication',
    ),
}

Now we have to add DRF auth app to INSTALLED_APPS

INSTALLED_APPS = (
    ...
    'rest_framework.authtoken',
)

We can also expose an obtain token endpoint, where users can send a username/password and have their token back. To do so we have to modify api/urls.py

from django.conf.urls import url, include
from rest_framework.routers import DefaultRouter
from rest_framework.authtoken.views import obtain_auth_token

from . import views

router = DefaultRouter()
router.register(r'teachers', views.TeacherViewSet)
router.register(r'students', views.StudentViewSet)
router.register(r'courses', views.CourseViewSet)

urlpatterns = [
    url(r'^', include(router.urls)),
    url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')),
    url(r'^api-token-auth/', obtain_auth_token),
]

Now we have to run migrate to create token app tables:

./manage.py migrate

Let's get the token via the obtain token view:

curl -X POST http://localhost:8000/api/api-token-auth/ \
-H 'Content-type: application/json' \
-d '{"username": "admin", "password": "yourAdminPassword"}'

{"token":"c98d345c4eeeb5529e184526b4f3effbc8957251"}%

We can now turn on permission, we can set up permission per view/viewset level or globally via settings.py

REST_FRAMEWORK = {
    'DEFAULT_FILTER_BACKENDS': ('django_filters.rest_framework.DjangoFilterBackend',),
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.LimitOffsetPagination',
    'PAGE_SIZE': 100,
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.TokenAuthentication',
    ),
    'DEFAULT_PERMISSION_CLASSES': ('rest_framework.permissions.IsAuthenticated',),
}

We still can override permission per view level if we want. Let's see if the curl call will return a 403 status code

curl -X GET http://localhost:8000/api/teachers/\?name\=john                                                                        
{"detail":"Authentication credentials were not provided."}%

Now we should provide the token which we got as header: Authorization: Token c98d345c4eeeb5529e184526b4f3effbc8957251 Do not forget to replace c98d345c4eeeb5529e184526b4f3effbc8957251 by your local token

curl -H 'Authorization: Token c98d345c4eeeb5529e184526b4f3effbc8957251' -X GET http://localhost:8000/api/teachers/\?name\=john

{"count":4,"next":null,"previous":null,"results":[{"id":2,"name":"John2","speciality":"Physics"},{"id":3,"name":"John","speciality":"Physics"},{"id":4,"name":"John","speciality":"Physics"},{"id":5,"name":"John","speciality":"Physics"}]}%

Nested API

We will use drf-extension to add nested support. We will be able to list all courses of a teacher directly via the teacher detail endpoint /api/teacher/1/courses/ I'm not using this too often as I prefer to use flat endpoints, but sometime it may be requested and I will show how it's possible and easy.

We will install drf-extensions which have a bunch of helpers, we will need right now the nested router.

pipenv install git+https://github.com/chibisov/drf-extensions.git@master#egg=drf_extensions

PS: Until the time of writing this, drf-extension official release does not support Django 2.0. that's why we are using the master branch directly from Github. You may not need to use git link as soon as a new version will be released (> 0.3.1)

Let's modify api.urls.py

from django.conf.urls import url, include
from rest_framework_extensions.routers import ExtendedDefaultRouter
from rest_framework.authtoken.views import obtain_auth_token

from . import views

router = ExtendedDefaultRouter()
router.register(r'teachers', views.TeacherViewSet).register(
    r'courses', views.CourseViewSet, base_name='courses', parents_query_lookups=['teacher'],
)
router.register(r'students', views.StudentViewSet)

urlpatterns = [
    url(r'^', include(router.urls)),
    url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')),
    url(r'^api-token-auth/', obtain_auth_token),
]

As you see, we can chain router registrations, keep in mind this is only supported for reading, for creation you will need to do a bit more to get for example the current teacher while creating a new course.

And then api/views/py to add NestedViewSetMixin to our viewsets

from rest_framework import viewsets
from rest_framework_extensions.mixins import NestedViewSetMixin

from . import serializers
from . import filters
from courses import models as courses_models


class TeacherViewSet(NestedViewSetMixin, viewsets.ModelViewSet):
    queryset = courses_models.Teacher.objects.all()
    serializer_class = serializers.TeacherSerializer
    filter_class = filters.TeacherFilter


class StudentViewSet(NestedViewSetMixin, viewsets.ModelViewSet):
    queryset = courses_models.Student.objects.all()
    serializer_class = serializers.StudentSerializer
    filter_class = filters.StudentFilter


class CourseViewSet(NestedViewSetMixin, viewsets.ModelViewSet):
    queryset = courses_models.Course.objects.all()
    serializer_class = serializers.CourseSerializer

Now you can try /api/teachers/1/courses/

curl -H 'Authorization: Token c98d345c4eeeb5529e184526b4f3effbc8957251' -X GET http://localhost:8000/api/teachers/1/courses/

{"count":1,"next":null,"previous":null,"results":[{"id":1,"name":"c1","teacher":{"id":1,"name":"Mounir Messelmeni","speciality":"Python"},"level":2,"students":[{"id":1,"name":"John Wick","birth_date":"1990-02-14","current_level":3,"age":27},{"id":2,"name":"JN","birth_date":"2017-12-15","current_level":2,"age":0}]}]}%

Mounir's blog