Set up file uploads to S3 via Django in 10 minutes

Set up file uploads to S3 via Django in 10 minutes

I’m going to show you how to correctly configure your Django project to allow you to upload files safely and securely to an AWS S3 bucket. This is a great way of handling files in Django and as you will see in this guide, is quick and easy to set up.

Originally published by David Bruton  at  blog.theodo.com

Setting up a simple Django App

This can be skipped if you already have a project Django has a quick guide to set up a Django project here which I’ve followed to quickly set up a demo app.

Adding a photo model

Once you’ve done the basic set up in the tutorial, you can create a “Photos” model to start testing your uploads. I’ve included the photo model class I’ve used for this below so you can add the following directly into your models.py file:

from django.db import models
import uuid

class Photo(models.Model):
uuid = models.UUIDField(
primary_key=True, default=uuid.uuid4, editable=False,
)
created_at = models.DateTimeField(auto_now_add=True)
title = models.CharField(max_length=100)
photo = models.FileField()

This is just a simple Django model which can be used to store photos. The model has a Django File Field for the photo file itself as well as a title, a created at datetime field and an UUID field.

The Django admin allows file uploads using the File field on a model which means we can use this built in functionality without having to spend time setting up forms to test our uploads.

Testing default behaviour of file uploads

First we need to go to the admin site for your project and find the photos model you have defined. Make sure you have registered your model in your admin.py so that you have access to it here.

Click on “Add photo” and you will be able to give your photo object a title and choose the file we want to upload as shown below. Once you’ve done this, click Save.

If we do this, we see that the file is uploaded to the root directory of the app.

This is rarely where we want to store uploaded files, but this can be easily modified using the MEDIA_ROOT property in the settings file of your app. This defines the path at which media file uploads will be stored.

To demonstrate this we can set this value to a path within the app using the BASE_DIR which is defined automatically when generating the project with django. If you add the folowing line to your settings.pythis will store your uploads inside the project folder at the file path specified in quotes.

MEDIA_ROOT = os.path.join(BASE_DIR, 'path/to/store/my/files/')

If you now try and upload a file using the admin, we see in the root directory of my app example-django-app there is a new file path created to the photo uploaded.

However, this is still often not a sensible place to be storing uploaded files for many reasons. If your server is on AWS or another cloud service, the files will be lost when scaling and creating new server instances as they are stored directly on a server instance’s file system. This is a major problem.

Furthermore, they are not going to be easily accessible when trying to retrieve them as they are directly saved on the server, especially if you need the files to be easily accessible externally (e.g. to allow controlled access to other users or websites).

We need a better solution for storing these files somewhere which can easily be accessed to retrieve uploaded files for those with permission, is scalable for handling large numbers of uploads on your Django app and will keep files stored safely even if your server is scaling up or down.

AWS S3 Buckets

Amazon’s “Simple Storage Service” (S3), offers a secure, reliable and scalable storage system which is perfect for this use. It is simple to set up and can be used regardless of where your Django site is being hosted.

Setting up an S3 bucket and allowing the Django app access

You can create an S3 bucket easily by logging into your AWS account, going to the S3 section of the AWS console, clicking “Create bucket” and following the steps to set up. You will also need to create a user with Programmatic access to your AWS account in the “IAM” console and give it permissions for S3FullAccess for your app to be able to correctly access your bucket. AWS provide this useful walkthrough to help set this up

Once you have done this you need to set the access key and secret access key generated when you created the user in your settings.pyfile like shown below.

AWS_ACCESS_KEY_ID = 'YOUR_ACCESS_KEY_HERE'
AWS_SECRET_ACCESS_KEY = 'YOUR_SECRET_ACCESS_KEY_HERE'

Also make sure you remove the MEDIA_ROOT property in the settings file.

For your production environment, it’s best to get these values from environment variables rather than hardcoding them in your settings file.

Installing some helper libraries

We’re nearly ready to start using the S3 bucket for uploads, we just need to install 2 python libraries: boto3 and django-storages. Boto3 is the SDK that AWS provide for Python to be able to manage AWS services from within your code. We need to use it to specify the S3 bucket which your file uploads need to be directed to (you can look at the boto3 documentation here).

Django-storages is a useful library which provides storage helper methods (you can read the documentation for it here). You will need to add 'storages' into your INSTALLED_APPS of settings file too.

To install the libraries, you can simply run:

pip install boto3
pip install django-storages

inside the virtual environment your app is running in and adding these libraries to your dependencies list.

Once you’ve done this, you need to define a few more variables in your settings file:

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
AWS_STORAGE_BUCKET_NAME = 'insert-your-bucket-name-here'
AWS_S3_REGION_NAME = 'eu-west-2'

When you created your S3 bucket you will have defined a name for the bucket as well as choosing an AWS region for it to be hosted in. Insert these here.

We should be ready now to upload files directly to S3 via the Django admin upload we tried earlier. If we try the uploads again we will see the files no longer appear in the root directory of our app but are stored in the S3 bucket we set up.

We are now able to retrieve these files in S3 in the AWS console or directly in the Django admin.

Conclusion

We have been able to set up file uploads from a very basic Django up quickly and easily. You should now be able to easily set this up on future Django projects to make your handling of file uploads hassle free.

Originally published by David Bruton  at  blog.theodo.com

===================================================================

Thanks for reading :heart: If you liked this post, share it with all of your programming buddies! Follow me on Facebook | Twitter

Learn More

☞ Complete Python Bootcamp: Go from zero to hero in Python 3

☞ Python and Django Full Stack Web Developer Bootcamp

☞ Python for Time Series Data Analysis

☞ Python Programming For Beginners From Scratch

☞ Beginner’s guide on Python: Learn python from scratch! (New)

☞ Python for Beginners: Complete Python Programming

☞ Django 2.1 & Python | The Ultimate Web Development Bootcamp

☞ Python eCommerce | Build a Django eCommerce Web Application

☞ Python Django Dev To Deployment

Python Django Tutorial | Django Course

Python Django Tutorial | Django Course

🔥Intellipaat Django course: https://intellipaat.com/python-django-training/ 👉This Python Django tutorial will help you learn what is django web development &...

This Python Django tutorial will help you learn what is django web development & application, what is django and introduction to django framework, how to install django and start programming, how to create a django project and how to build django app. There is a short django project as well to master this python django framework.

Why should you watch this Django tutorial?

You can learn Django much faster than any other programming language and this Django tutorial helps you do just that. Our Django tutorial has been created with extensive inputs from the industry so that you can learn Django and apply it for real world scenarios.

Developing Restful APIs with Python, Django and Django Rest Framework

Developing Restful APIs with Python, Django and Django Rest Framework

This article is a definitive guide for starters who want to develop projects with RESTful APIs using Python, Django and Django Rest Framework.

This article is a definitive guide for starters who want to develop projects with RESTful APIs using Python, Django and Django Rest Framework.

Introduction
  • Django is a web framework written in Python
  • Python is an interpreted high-level programming language for general-purpose programming
  • API or Application Programming Interface is a set of rules and mechanisms by which one application or component interacts with the others
  • REST or Representational State Transfer is a software architecture

REST APIs

As described in a dissertion by Roy Fielding,

REST is an "architectural style' that basically exploits the existing technology and protocols of the web.
In simple definition, it is the data representation for a client in the format that is suitable for it.

Hence, RESTful + API is a commonly used terminology for the implementation of such architecture and constraints (eg. in web services).

Here is an example GET request from GitHub's API

$ curl https://api.github.com/users/joshuadeguzman

You will see an output similar to this

{
  "login": "joshuadeguzman",
  "id": 20706361,
  "node_id": "MDQ6VXNlcjIwNzA2MzYx",
  "avatar_url": "https://avatars1.githubusercontent.com/u/20706361?v=4",
  "gravatar_id": "",
  "url": "https://api.github.com/users/joshuadeguzman",
  "html_url": "https://github.com/joshuadeguzman",
  "followers_url": "https://api.github.com/users/joshuadeguzman/followers",
  "following_url": "https://api.github.com/users/joshuadeguzman/following{/other_user}",
  "gists_url": "https://api.github.com/users/joshuadeguzman/gists{/gist_id}",
  "starred_url": "https://api.github.com/users/joshuadeguzman/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/joshuadeguzman/subscriptions",
  "organizations_url": "https://api.github.com/users/joshuadeguzman/orgs",
  "repos_url": "https://api.github.com/users/joshuadeguzman/repos",
  "events_url": "https://api.github.com/users/joshuadeguzman/events{/privacy}",
  "received_events_url": "https://api.github.com/users/joshuadeguzman/received_events",
  "type": "User",
  "site_admin": false,
  "name": "Joshua de Guzman",
  "company": "@freelancer",
  "blog": "https://joshuadeguzman.me",
  "location": "Manila, PH",
  "email": null,
  "hireable": true,
  "bio": "Android Engineer at @freelancer. Building tools for humans.",
  "public_repos": 75,
  "public_gists": 2,
  "followers": 38,
  "following": 10,
  "created_at": "2016-07-28T15:19:54Z",
  "updated_at": "2019-06-16T10:26:39Z"
}

Shown above is a data set in JSON format.

JSON or JavaScript Object Notation is an open-standard file format that uses human-readable text to transmit data objects consisting of attribute–value pairs and array data types.
Other formats include XML, INI, CSV, etc. But today, JSON is widely use for its structure is intuitive, making it comfortable to read and map domain objects no matter what programming language is being used.

Python and Django

Python, according to its creator, Guido van Rossum, is a

high-level programming language, and its core design philosophy is all about code readability and a syntax which allows programmers to express concepts in a few lines of code.
Python uses english like words representation (eg. for methods, reserve keywords and control flow) that makes it easier for any beginner to jump right into it. It also features dynamic type system meaning it verifies the type safety of program at runtime. It also does automatic memory management.

print(5 + 5) # This will result to 10

Django is a high-level Python Web Framework that enables developers to deliver projects on time with clean and pragmatic design.

Its flagship features include a design for fast development, a secure and scalable product.

Quick Django Overview

Django's way of propagating changes to your database schema is by means of its migration modules.

Sample User model

from django.db import models

class User(models.Model):
    first_name = models.CharField(max_length=50)
    middle_name = models.CharField(max_length=50)
    last_name = models.CharField(max_length=50)

    def __str__(self):
        return self.name

If any changes are made on your models, run makemigrations

$ python manage.py makemigrations

Finally, you can synchronize the database with the set of models and migrations

$ python manage.py migrate

REST APIs with Django Rest Framework

DRF or Django REST Framework is a powerful and flexible toolkit for building Web APIs. It helps the developers to not reinvent the wheel by rolling out complex and solid REST API from scratch by themselves. Because when your projects become more and more complex, you will soon realise the need of using DRF or other helpful rest framework.

1. Installation & Project Setup

Create project directory

$ mkdir djangoapi

Install virtualenv via pip

A virtual environment enables a project to have additional libraries or changes in packages within its environment without disturbing global or libraries of other environments.

pip is a package management system used to install and manage software packages written in Python.

$ pip install virtualenv

To create an environment folder in your project's directory

$ cd djangoapi
$ virtualenv venv

To activate the environment

$ source venv/bin/activate

To undo these changes to your path, simply run deactivate. More on virtualenv.

Install django, djangorestframework

$ pip install django
$ pip install djangorestframework

Creating a django project

$ django-admin startproject blog

Running your project

$ python manage.py runserver

System check identified no issues (0 silenced).

You have 15 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

August 16, 2018 - 09:58:36
Django version 2.1, using settings 'blog.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

The unapplied migrations refer to the default migration files included when you start a django project.

To synchronize these migration files, simply run migrate

$ python manage.py migrate

Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying sessions.0001_initial... OK

The default database in our project is currently set to SQLite named db.sqlite3.

Creating a django project's app

$ cd blog
$ python manage.py startapp posts

The project structure should look like

$ find .
./posts
./posts/migrations
./posts/migrations/__init__.py
./posts/models.py
./posts/__init__.py
./posts/apps.py
./posts/admin.py
./posts/tests.py
./posts/views.py
./db.sqlite3
./blog
./blog/__init__.py
./blog/__pycache__
./blog/__pycache__/settings.cpython-36.pyc
./blog/__pycache__/wsgi.cpython-36.pyc
./blog/__pycache__/__init__.cpython-36.pyc
./blog/__pycache__/urls.cpython-36.pyc
./blog/settings.py
./blog/urls.py
./blog/wsgi.py
./manage.py

2. Model

Each model instance is a definitive source of the information about your data. In general, each model pertains to a single table in your database.

# djangoapi/blog/posts/models.py
from django.db import models

# Create your models here.

class Post(models.Model):
    title = models.CharField(max_length=255)
    content = models.TextField()
    is_featured = models.BooleanField(default=False)

    def __str__(self):
        return self.name

__str__ is called by the str() built-in function and by the print statement to compute the "informal" string representation of an object.
If you try running makemigrations, django won't see those changes yet.

$ No changes detected

To solve this, add your posts app to your project's installed apps.

# djangoapi/blog/blog/settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'posts' # Add it here
]

To continue with the migration of models

$ python manage.py makemigrations

Migrations for 'posts':
  posts/migrations/0001_initial.py
    - Create model Post

$ python manage.py migrate

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, posts, sessions
Running migrations:
  Applying posts.0001_initial... OK


3. Serialization

Serializers allow data structure or object state to be translated into a format that can be stored or transmitted and be reconstructed later on.

Create API's serializers.py and views.py files and isolate them like this

# posts/api
posts/api/serializers.py
posts/api/views.py

# posts/migrations
posts/migrations/

# posts
posts/admin.py
posts/apps.py
posts/models.py
posts/tests.py
posts/views.py
# posts/api/serializers.py

from ..models import Post
from rest_framework import serializers

class PostSerializer(serializers.ModelSerializer):
    class Meta:
        model = Post
        fields = ('title', 'content', 'is_featured') # if not declared, all fields of the model will be shown

In this tutorial we have used ModelSerializer, more on this.

4. Views

A view function, or view for short, is a Python function that takes a Web request and returns a Web response.

# posts/api/views.py

from ..models import Post
from . import serializers
from rest_framework import generics, status
from rest_framework.response import Response

class PostListView(generics.ListAPIView):
    queryset = Post.objects.all()
    serializer_class = serializers.PostSerializer

As seen above, ListAPIView is used for read-only endpoints to represent a collection of model instances.

In this code snippet, we use generics view methods from the rest_framework, more on this.

5. URLs

This is where we setup our routes or URL paths to our designated views in which we expect specific responses for each.

# posts/urls.py

from django.urls import path
from . import views
from .api import views

urlpatterns = [
    path('', views.PostListView.as_view(), name=None)
]

6. Finalizing Setup

Ensure that the rest_framework is added to our project's apps.

# djangoapi/blog/blog/settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework', # Add it here
    'posts'
]

7. Django Admin

Since we haven't setup our POST requests yet, we will be populating the database through django's admin panel.

To do that, create a superuser account admin with password 1234password.

$ python manage.py createsuperuser --email [email protected] --username admin

Password:
Password (again):
This password is too common.
Bypass password validation and create user anyway? [y/N]: y
Superuser created successfully.

Register the model in the admin panel.

# posts/admin.py

from django.contrib import admin
from .models import Post

# Register your models here.
admin.site.register(Post)

That's it. Visit the admin panel and update posts model's records. More on this.

8. Testing our API

$ python manage.py runserver
GET /api/v1/posts/
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
    {
        "title": "Example Post #1",
        "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.",
        "is_featured": false
    },
    {
        "title": "Example Post #2",
        "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.",
        "is_featured": true
    }
]

Great. Now it's time for us to update our views and finish the standard CRUD operations.

9. Adding more views

POST is a method used for creating (sometimes updating) a resource in the database.

# posts/api/views.py

from ..models import Post
from . import serializers
from rest_framework import generics, status
from rest_framework.response import Response

class PostCreateView(generics.CreateAPIView):
    queryset = Post.objects.all()
    serializer_class = serializers.PostSerializer

    def create(self, request, *args, **kwargs):
        super(PostCreateView, self).create(request, args, kwargs)
        response = {"status_code": status.HTTP_200_OK,
                    "message": "Successfully created",
                    "result": request.data}
        return Response(response)

Most often, we separate List and Create view classes when we want to expose a list of data set while easily preventing a certain request to POST or create a resource in the database for that specific List view.

Usecase always varies for apps, you are opt to use ListCreateAPIView or even ViewSets for combining the logic for a set of related views.

Optional: Since we want to display the data in a more systematic way, we override create method and map our inline custom response handler.

Adding more views with methods GET, PATCH, DELETE to handle a specific blog post detail.

class PostDetailView(generics.RetrieveUpdateDestroyAPIView):
    queryset = Post.objects.all()
    serializer_class = serializers.PostSerializer

    def retrieve(self, request, *args, **kwargs):
        super(PostDetailView, self).retrieve(request, args, kwargs)
        instance = self.get_object()
        serializer = self.get_serializer(instance)
        data = serializer.data
        response = {"status_code": status.HTTP_200_OK,
                    "message": "Successfully retrieved",
                    "result": data}
        return Response(response)

    def patch(self, request, *args, **kwargs):
        super(PostDetailView, self).patch(request, args, kwargs)
        instance = self.get_object()
        serializer = self.get_serializer(instance)
        data = serializer.data
        response = {"status_code": status.HTTP_200_OK,
                    "message": "Successfully updated",
                    "result": data}
        return Response(response)

    def delete(self, request, *args, **kwargs):
        super(PostDetailView, self).delete(request, args, kwargs)
        response = {"status_code": status.HTTP_200_OK,
                    "message": "Successfully deleted"}
        return Response(response)

10. Updating URLs

# posts/urls.py

from django.urls import path
from . import views
from .api import views

urlpatterns = [
    path('', views.PostListView.as_view(), name=None),
    path('create/', views.PostCreateView.as_view(), name=None),
    path('<int:pk>/', views.PostDetailView.as_view(), name=None)
]

Now you can send requests to your API via Postman, your app or do a GETrequests from your browser, examples:

POST /api/v1/posts/create/
HTTP 200 OK
Allow: POST, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "status_code": 200,
    "message": "Successfully created",
    "result": {
        "csrfmiddlewaretoken": "rnSUN3XOIghnXA0yKghnQgxg0do39xhorYene5ALw3gWGThK5MjG6YjL8VUb7v2h",
        "title": "Creating a resource",
        "content": "Howdy mate!"
    }
}
GET /api/v1/posts/1/
HTTP 200 OK
Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "status_code": 200,
    "message": "Successfully retrieved",
    "result": {
        "title": "Sample Post",
        "content": "Sample Post Content",
        "is_featured": false
    }
}

That's it. You have successfully managed to develop RESTful APIs with DRF! Cheers!

Source code

Available on GitHub.

Mobile App Development Company India | Ecommerce Web Development Company India

Mobile App Development Company India | Ecommerce Web Development Company India

Best Mobile App Development Company India, WebClues Global is one of the leading web and mobile app development company. Our team offers complete IT solutions including Cross-Platform App Development, CMS & E-Commerce, and UI/UX Design.

We are custom eCommerce Development Company working with all types of industry verticals and providing them end-to-end solutions for their eCommerce store development.

Know more about Top E-Commerce Web Development Company