This content originally appeared on DEV Community and was authored by ThePylot
Originally published in The Pylot
In this post, you'll learn how to integrate multiple databases with the Django framework and navigate incoming data using a DB router that automatically writes them to the required database.
Real-world example scenario
Usually, the majority of projects are using relational databases such as Postgres or MySQL but sometimes we also need NoSQL databases to hold extra heavy data which decrease the overload of relational databases.
Assume that your project generates tons of logs while processing some heavy tasks in a queue. These log objects must be stored in non-relational databases instead of hitting relational databases each time and extremely overloading them with huge messy log objects. I guess you spot the problem here, so let's take a look at what can we do about it...
Setting up environment
Create an empty directory named app then let's start by creating a Dockerfile that will copy our working directory and also install required dependencies for Python and Postgres.
app/Dockerfile
FROM python:3.8-slim
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y \
build-essential \
libssl-dev \
libffi-dev \
python3-dev \
build-essential \
libjpeg-dev \
zlib1g-dev \
gcc \
libc-dev \
bash \
git \
&& pip3 install --upgrade pip
ENV LIBRARY_PATH=/lib:/usr/lib
ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1
WORKDIR /app
COPY . /app
RUN pip3 --no-cache-dir install -r requirements.txt
These wheel packages will be used while installing and setting up Postgres and other services in the system. Now, we need to add requirements.txt to install the required packages for our project.
app/requirements.txt
celery==5.0.2
Django==3.1.3
git+git://github.com/thepylot/djongo.git#egg=djongo
mongoengine==0.20.0
pylint-mongoengine==0.4.0
pymongo==3.11.0
psycopg2-binary==2.8.5
redis==3.5.3
Faker
We are going to use djongo which will help to convert SQL to MongoDB query. By using djongo we can use MongoDB as a backend database for our Django project. At the time of writing this post, djongo has issues supporting Django versions above v.3+, but it can be resolved easily by just by changing the version from the package itself by forking it to your repository. There is a file named setup.py that holds configuration settings and you'll see the block named install_requires where the supported version is mentioned (Don't forget to fork it first).
setup.py
install_requires = [
'sqlparse==0.2.4',
'pymongo>=3.2.0',
'django>=2.1,<=3.0.5',
]
We just need to refactor it to fit with our current version of Django. I am using 3.1.3 so I will replace the 3.0.5 to 3.1.3 and it will look like below:
install_requires = [
'sqlparse==0.2.4',
'pymongo>=3.2.0',
'django>=2.1,<=3.1.3',
]
Once you finished, search for requirements.txt and change the Django version there as well:
requirements.txt (in djongo)
...
django>=2.0,<=3.1.3
...
That's it! Commit your changes to the forked repository and then you need to include it to your requirements.txt file but this time it will get from your forked repository like below:
git+git://github.com/YOUR_GITHUB_USERNAME/djongo.git#egg=djongo
You don't have to go through all these processes because I already included it as you can see from above (requirements.txt) so feel free to use mine if your Django version is 3.1.3 as well.
There are some other dependencies like celery where we'll use it as the queue to pass time-consuming tasks to run in the background and redis is just a message broker for celery. This topic is out of scope for this post but you can visit Dockerizing Django with Postgres, Redis and Celery to understand it more.
Now it's time to set up our services by configuring compose file. Create this file in a root level of your current directory which means it will be outside of app directory There are going to be 5 services in total:
-
mongodb- for setting up MongoDB -
postgres- for setting up PostgreSQL -
app- Django project -
celery- Queue for tasks -
redis- Message broker that required for celery
/docker-compose.yml
version: '3'
services:
mongo:
image: mongo
container_name: mongo
restart: always
env_file: .env
environment:
- MONGO_INITDB_ROOT_USERNAME=root
- MONGO_INITDB_ROOT_PASSWORD=root
- MONGO_INITDB_DATABASE=${MONGO_DB_NAME}
- MONGO_INITDB_USERNAME=${MONGO_DB_USERNAME}
- MONGO_INITDB_PASSWORD=${MONGO_DB_PASSWORD}
volumes:
- ${PWD}/_data/mongo:/data/db
- ${PWD}/docker/_mongo/fixtures:/import
- ${PWD}/docker/_mongo/scripts/init.sh:/docker-entrypoint-initdb.d/setup.sh
ports:
- 27017:27017
postgres:
container_name: postgres
image: postgres:12
restart: always
env_file: .env
environment:
- POSTGRES_DB=app_db
- POSTGRES_USER=app_db_user
- POSTGRES_PASSWORD=supersecretpassword
- POSTGRES_PORT=5432
ports:
- 5432:5432
volumes:
- ${PWD}/_data/postgres:/var/lib/postgresql/data
- ${PWD}/docker/_postgres/scripts/create_test_db.sql:/docker-entrypoint-initdb.d/docker_postgres_init.sql
redis:
image: redis:6
container_name: redis
restart: always
env_file: .env
command: redis-server --requirepass $REDIS_PASSWORD
ports:
- 6379:6379
volumes:
- ${PWD}/_data/redis:/var/lib/redis
app:
build: ./app
image: app:latest
container_name: app
restart: always
command: "python manage.py runserver 0.0.0.0:8000"
env_file: .env
volumes:
- ${PWD}/app:/app
ports:
- 8000:8000
depends_on:
- postgres
- redis
celery:
build: ./app
image: app:latest
container_name: celery
restart: always
command: [
"celery",
"-A",
"app",
"worker",
"-c",
"1",
"-l",
"INFO",
"--without-heartbeat",
"--without-gossip",
"--without-mingle",
]
env_file: .env
environment:
- DJANGO_SETTINGS_MODULE=app.settings
- DJANGO_WSGI=app.wsgi
- DEBUG=False
volumes:
- ${PWD}/app:/app
depends_on:
- postgres
- redis
networks:
default:
I will not go through this configuration in detail by assuming you already have knowledge about docker and compose files. Simply, we are pulling required images for the services and setting up main environment variables and ports to complete the configuration.
Now we also need to add .env file to fetch values of environment variables while building services:
/.env
# Mongo DB
MONGO_DB_HOST=mongo
MONGO_DB_PORT=27017
MONGO_DB_NAME=mongo_db
MONGO_DB_USERNAME=root
MONGO_DB_PASSWORD=root
MONGO_DB_URI=mongodb://root:root@mongo:27017
# PostgreSQL
POSTGRES_HOST=postgres
POSTGRES_DB=app_db
POSTGRES_USER=app_db_user
POSTGRES_PASSWORD=supersecretpassword
POSTGRES_PORT=5432
# Redis
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_PASSWORD=supersecretpassword
BROKER_URL=redis://:supersecretpassword@redis:6379/0
REDIS_CHANNEL_URL=redis://:supersecretpassword@redis:6379/1
CELERY_URL=redis://:supersecretpassword@redis:6379/0
Next, we'll create a new Django project inside our app folder:
docker-compose run app sh -c "django admin startproject app ."
The project structure should be like below:
.
├── app
│ ├── app
│ │ ├── asgi.py
│ │ ├── __init__.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py
│ ├── Dockerfile
│ ├── manage.py
│ └── requirements.txt
├── docker-compose.yml
└── .env
If you see dbsqlite in project files then you should delete it since we'll use postgres it as a relational database. You also will notice _data directory which represents the volume of MongoDB and Postgres.
Integration with PostgreSQL
We are ready to add our primary relational database which is going to be postgres. Navigate to settings.py and update DATABASES configuration like below:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': os.environ.get('POSTGRES_HOST'),
'NAME': os.environ.get('POSTGRES_NAME'),
'USER': os.environ.get('POSTGRES_USER'),
'PASSWORD': os.environ.get('POSTGRES_PASSWORD'),
'PORT': os.environ.get('POSTGRES_PORT', 5432),
}
}
The default database set to postgres and environment variables will be fetched from .env file. Sometimes, postgres is having connection issues caused by racing issues between Django and postgres. To prevent such situations we'll implement a custom command and add it to commands block in compose file. In this way, Django will wait postgres before launch.
The recommended path of holding commands is /core/management/commands/ from the official documentation of Django. So let's create an app named core then create a management/commands directory inside it.
docker-compose run sh -c "django-admin startapp core"
Then add following command to hold Django until postgres is available:
/core/management/commands/wait_for_db.py
import time
from django.db import connections
from django.db.utils import OperationalError
from django.core.management import BaseCommand
class Command(BaseCommand):
"""Django command to pause execution until db is available"""
def handle(self, *args, **options):
self.stdout.write('Waiting for database...')
db_conn = None
while not db_conn:
try:
db_conn = connections['default']
except OperationalError:
self.stdout.write('Database unavailable, waititng 1 second...')
time.sleep(1)
self.stdout.write(self.style.SUCCESS('Database available!'))
Make sure you included __init__.py into sub-directories you created. Now update app service in compose file by adding this command:
docker-compose.yml
app:
build: ./app
image: app:latest
container_name: app
restart: always
command: >
sh -c "python manage.py wait_for_db &&
python manage.py migrate &&
python manage.py runserver 0.0.0.0:8000"
env_file: .env
volumes:
- ${PWD}/app:/app
ports:
- 8000:8000
depends_on:
- postgres
- redis
Consider the command block only and you'll see we now have two more commands there.
Integration with MongoDB
Actually, the integration of MongoDB is so simple thanks to djongo which handles everything behind the scenes. Switch to settings.py again and we'll add our second database as nonrel which stands for the non-relational database.
settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': os.environ.get('POSTGRES_HOST'),
'NAME': os.environ.get('POSTGRES_NAME'),
'USER': os.environ.get('POSTGRES_USER'),
'PASSWORD': os.environ.get('POSTGRES_PASSWORD'),
'PORT': os.environ.get('POSTGRES_PORT'),
},
"nonrel": {
"ENGINE": "djongo",
"NAME": os.environ.get('MONGO_DB_NAME'),
"CLIENT": {
"host": os.environ.get('MONGO_DB_HOST'),
"port": int(os.environ.get('MONGO_DB_PORT')),
"username": os.environ.get('MONGO_DB_USERNAME'),
"password": os.environ.get('MONGO_DB_PASSWORD'),
},
'TEST': {
'MIRROR': 'default',
},
}
}
The same logic applies here as we did for default database which is postgres.
Setting up DB router
DB router which will automatically write objects to a proper database such as whenever the log object created it should navigate to mongodb instead of postgres. Setting up a DB router is simple, we just need to use router methods that Django provides and define our non-rel models to return the proper database.
Create a new directory named utils inside the core app and also add __init__.py to mark it as a python package. Then add the new file which is DB router below:
/core/utils/db_routers.py
class NonRelRouter:
"""
A router to control if database should use
primary database or non-relational one.
"""
nonrel_models = {'log'}
def db_for_read(self, model, **_hints):
if model._meta.model_name in self.nonrel_models:
return 'nonrel'
return 'default'
def db_for_write(self, model, **_hints):
if model._meta.model_name in self.nonrel_models:
return 'nonrel'
return 'default'
def allow_migrate(self, _db, _app_label, model_name=None, **_hints):
if _db == 'nonrel' or model_name in self.nonrel_models:
return False
return True
nonrel_models - We are defining the name of our models in lowercase which belongs to a non-rel database or mongodb.
db_for_read - the function name is self-explanatory so basically, it is used for reading operations which means each time we try to get records from the database it will check where it belongs and return the proper database.
db_for_write - the same logic applies here. It's used to pick a proper database for writing objects.
allow_migrate - Decided if the model needs migration. In mongodb there is no need to run migrations since it's a non-rel database.
Next, we should add extra configuration in settings.py to activate our custom router:
settings.py
...
DATABASE_ROUTERS = ['core.utils.db_routers.NonRelRouter', ]
...
Great! Our database configurations are finished and now it's time to make a few changes in Django as well before launching everything.
Setting up Celery and Redis
This part is a bit out of scope but since I want to illustrate some real-world app then those tools are always present in projects to handle heavy and time-consuming tasks. Let's add celery to our project, but it should place in our project folder alongside with settings file:
celery.py
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app.settings')
app = Celery('app')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
Basically, it will discover all tasks alongside the project and will pass them to the queue. Next, we also need to update __init__.py file inside the current directory, which is our Django project:
init.py
from .celery import app as celery_app
__all__ = ['celery_app']
Celery requires a broker URL for tasks so in this case, we will use Redis as a message broker. Open your settings file and add the following configurations:
settings.py
CELERY_TASK_TRACK_STARTED = True
CELERY_TASK_TIME_LIMIT = 30 * 60
CELERY_IGNORE_RESULT = True
CELERY_BROKER_URL = os.environ.get('CELERY_URL')
CELERYD_HIJACK_ROOT_LOGGER = False
REDIS_CHANNEL_URL = os.environ.get('REDIS_CHANNEL_URL')
Now try to run docker-compose up -d and all services should start successfully.
Adding models and celery tasks
In this section, we'll pass a new task to the celery queue and test if our DB router works properly and writes objects to the required database. Create a new directory inside the core app named models and add the following file inside it to hold MongoDB models:
core/models/mongo_models.py
from djongo import models as mongo_models
class Log(mongo_models.Model):
_id = mongo_models.ObjectIdField()
message = mongo_models.TextField(max_length=1000)
created_at = mongo_models.DateTimeField(auto_now_add=True)
updated_at = mongo_models.DateTimeField(auto_now=True)
class Meta:
_use_db = 'nonrel'
ordering = ("-created_at", )
def __str__(self):
return self.message
As you see it's a very simple Log model where it will let us know what's going on behind the scenes of internal operations of our application. Then add a model for Postgres as well:
core/models/postgres_models.py
from django.db import models
class Post(models.Model):
title = models.CharField(max_length=255)
description = models.TextField()
def __str__(self):
return self.title
We are going to generate random posts and make some of them fail in order to write error logs for failed data.
Lastly, don't forget to include __init__.py inside models directory:
models/init.py
from .postgres_models import *
from .mongo_models import *
Now we need to create a celery task that will generate tons of posts with random values by using Faker generators:
core/tasks.py
import logging
import random
from faker import Faker
from .models import Post
from celery import shared_task
from core.utils.log_handlers import LoggingHandler
logger = logging.getLogger(__name__)
logger.addHandler(LoggingHandler())
@shared_task
def create_random_posts():
fake = Faker()
number_of_posts = random.randint(5, 100)
for i in range(number_of_posts):
try:
if i % 5 == 0:
title = None
else:
title = fake.sentence()
description = fake.text()
Post.objects.create(
title = title,
description = description,
)
except Exception as exc:
logger.error("The post number %s failed due to the %s", i, exc)
By adding if statement there we are forcing some of the posts to fail in order to catch exceptions and write them to mongodb. As you noticed, we are using a custom log handler that will write the log data right away after its produced. So, create a new file named log_handlers.py inside utils folder:
core/utils/log_handlers.py
import logging
from core.models import Log
class LoggingHandler(logging.Handler):
"""Save log messages to MongoDB
"""
def emit(self, record):
Log.objects.create(
message=self.format(record),
)
Great! We are almost ready to launch.
Lastly, let's finish creating a very simple view and URL path to trigger the task from the browser and return JSON a response if the operation succeeded.
core/views.py
from django.http.response import JsonResponse
from .tasks import create_random_posts
def post_generator(request):
create_random_posts.delay()
return JsonResponse({"success": True})
urls.py
from django.contrib import admin
from django.urls import path
from core.views import post_generator
urlpatterns = [
path('admin/', admin.site.urls),
path('', post_generator)
]
Include the core app inside INSTALLED_APPS configuration in settings.py then run the migrations and we're done!
docker-compose run app sh -c "python manage.py makemigrations core"
Now, if you navigate to 127.0.0.1:8000 the task will be passed to the queue and post objects will start to generate.
Try to visit admin and you'll see log objects created successfully and by this way we're avoiding overload postgres while it handles only relational data.
Source Code:
thepylot
/
django-mongodb-postgres
Integration of multiple databases with Django framework and navigate incoming data using DB router which automatically writes them to required database.
MongoDB and Postgres integration with Django
This project includes multiple database configurations to launch Django project with Postgres and MongoDB.
Getting Started
This project works on Python 3+ and Django 3.1.3. If you want to change the Django version please follow the link below to get information about it.
Add MongoDB and PostgreSQL in Django using Docker
Run the following command to makemigrations:
docker-compose run app sh -c "python manage.py makemigrations core"
then start the containers:
docker-compose up -d
After containers launched navigate to / home path and celery will automatically receive the task.
Support 🌏
If you feel like you unlocked new skills, please share them with your friends and subscribe to the youtube channel to not miss any valuable information.
This content originally appeared on DEV Community and was authored by ThePylot
ThePylot | Sciencx (2021-12-28T19:30:35+00:00) Add MongoDB and PostgreSQL in Django using Docker. Retrieved from https://www.scien.cx/2021/12/28/add-mongodb-and-postgresql-in-django-using-docker/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.