When a Django view does heavy work synchronously — sending emails, encoding an image, hitting a slow external API — the user waits until it’s done. Celery moves that work to a separate worker process so the request returns immediately, and adds scheduled and retryable task execution on top.

What is Celery?

Celery is an asynchronous task queue based on distributed message passing. It needs a message broker (typically Redis or RabbitMQ) to hand tasks from the producer (your Django app) to the workers, and optionally a results backend to store task outcomes.

Why use Celery in Django

Django processes requests synchronously. If a request involves a time-consuming operation, the user waits for it to finish before getting a response. Celery breaks that coupling:

• Faster responses. Long-running work moves to a worker. The view returns right away — e.g., the signup view returns “registered” immediately and dispatches the verification email as a task.
• Horizontal scalability. Add more workers (same host or others) as load grows.
• Resource separation. Web processes serve requests; workers handle heavy lifting. One slow task can’t lock up the web server.
• Retries. Failed tasks can be retried automatically with configurable backoff.
• Scheduling. Celery Beat runs tasks on a cron-like schedule for periodic jobs.

Common use cases

• Sending emails (welcome, password reset, notifications)
• Image/video processing (resizing, watermarking, encoding)
• CSV/Excel import-export and report generation
• Outbound API calls and scraping
• Push/SMS/in-app notifications
• ML inference or data analysis off the request path

Integrating Celery with Django

This walkthrough uses Redis as both broker and results backend.

Step 1: Install Celery and Redis

First, you need to install Celery and the Redis client library in your Django project’s virtual environment:

pip install celery redis

Step 2: Configure Celery in Your Django Project

Create a celery.py file inside your Django project’s main application directory (the one that contains settings.py and urls.py). This file will define your Celery application instance.

your_project_name/your_project_name/celery.py:

import os
from celery import Celery

# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'your_project_name.settings')

app = Celery('your_project_name')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()


@app.task(bind=True, ignore_result=True)
def debug_task(self):
    print(f'Request: {self.request!r}')

Next, import this Celery app instance in your project’s __init__.py file to ensure it’s loaded when Django starts. This file is located in the same directory as celery.py.

your_project_name/your_project_name/__init__.py:

from .celery import app as celery_app

__all__ = ('celery_app',)

Step 3: Configure Django Settings

Add the Celery broker and backend settings to your settings.py file. Replace your_project_name with the actual name of your Django project.

your_project_name/your_project_name/settings.py:

# Celery Configuration
CELERY_BROKER_URL = 'redis://127.0.0.1:6379/0'
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Asia/Kolkata' # Or your preferred timezone

Step 4: Define Celery Tasks

Now, you can define your tasks within any of your Django applications. A common practice is to create a tasks.py file inside your app directory. For example, if you have an app named myapp:

your_project_name/myapp/tasks.py:

from celery import shared_task
import time

@shared_task
def send_welcome_email(user_email):
    print(f"Sending welcome email to {user_email}...")
    time.sleep(5)  # Simulate a long-running task
    print(f"Welcome email sent to {user_email}!")
    return True

Step 5: Call Celery Tasks

You can call these tasks from your Django views, models, or wherever appropriate. Use .delay() for simple calls or .apply_async() for more advanced options like setting a countdown or retries.

your_project_name/myapp/views.py:

from django.http import HttpResponse
from .tasks import send_welcome_email

def register_user(request):
    # ... user registration logic ...
    user_email = "new_user@example.com"
    send_welcome_email.delay(user_email) # Task sent to Celery
    return HttpResponse("User registered! Welcome email will be sent shortly.")

Step 6: Start Redis Server

Before running Celery, ensure your Redis server is running. If you don’t have Redis installed, you can typically install it via your system’s package manager:

sudo apt update
sudo apt install redis-server
sudo systemctl enable redis-server
sudo systemctl start redis-server

Step 7: Start Celery Worker

Navigate to your Django project’s root directory (where manage.py is located) and start the Celery worker:

celery -A your_project_name worker -l info

Replace your_project_name with your actual project name. -l info sets the worker log level.

Running Celery as a systemd service on Ubuntu

For production, run Celery as a systemd service so it starts on boot and can be managed via systemctl.

Step 1: Create a Celery User and Group

It’s good practice to run Celery under a dedicated user for security reasons:

sudo adduser --system --no-create-home --group celery

Step 2: Create a Celery Configuration File

Create a configuration file for your Celery worker. This file will specify the working directory, user, group, and other options. Let’s create it at /etc/default/celeryd.

sudo nano /etc/default/celeryd

Add the following content, replacing your_project_name and /path/to/your/project with your actual project details:

# Names of nodes to start
# CELERYD_NODES="w1 w2"
# Absolute path to the Django project directory
CELERYD_CHDIR="/path/to/your/project"
# Where to store the pidfile
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Where to store the logfile
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
# Log level
CELERYD_LOG_LEVEL="INFO"
# Celery user and group
CELERYD_USER="celery"
CELERYD_GROUP="celery"
# How to start Celery (e.g., -A your_project_name worker -l info)
CELERYD_OPTS="-A your_project_name worker -l info"

Create the necessary directories for pid and log files and set appropriate permissions:

sudo mkdir -p /var/run/celery
sudo chown celery:celery /var/run/celery
sudo mkdir -p /var/log/celery
sudo chown celery:celery /var/log/celery

Step 3: Create a Systemd Service File

Create a systemd service file for Celery. This file tells systemd how to manage the Celery worker process. Let’s create it at /etc/systemd/system/celery.service.

sudo nano /etc/systemd/system/celery.service

Add the following content:

[Unit]
Description=Celery Worker for your_project_name
After=network.target

[Service]
Type=forking
User=celery
Group=celery
EnvironmentFile=/etc/default/celeryd
WorkingDirectory=/path/to/your/project
ExecStart=/bin/sh -c '${CELERYD_CHDIR}/venv/bin/celery ${CELERYD_OPTS} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE}'
ExecStop=/bin/sh -c '${CELERYD_CHDIR}/venv/bin/celery multi stopwait ${CELERYD_OPTS} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE}'
ExecReload=/bin/sh -c '${CELERYD_CHDIR}/venv/bin/celery multi restart ${CELERYD_OPTS} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE}'
Restart=always

[Install]
WantedBy=multi-user.target

Note: Replace /path/to/your/project with the actual path to your Django project, and venv/bin/celery with the correct path to your Celery executable within your virtual environment. If you are not using a virtual environment, you might just use celery or the global path to it.

Step 4: Reload Systemd and Start Celery Service

After creating the service file, reload systemd to recognize the new service, then start and enable it to run on boot:

sudo systemctl daemon-reload
sudo systemctl start celery
sudo systemctl enable celery

Step 5: Check Celery Service Status

You can check the status of your Celery worker using:

sudo systemctl status celery

And view the logs:

sudo journalctl -u celery -f

References

• Real Python — Asynchronous Tasks With Django and Celery
• freeCodeCamp — How to Use Celery in Django
• Medium — Setup celery service on Django (ubuntu/debian) server