Deploy Celery workers on Linux — production setup

TL;DR

Celery workers process background tasks and scheduled jobs for Python applications. A production Celery deployment needs a message broker (Redis), Supervisor for process management, separate processes for workers and beat scheduler, log rotation, and monitoring with Flower.

Prerequisites

• Ubuntu 24.04 with a Django or Flask application already deployed
• Redis installed and running
• Your application uses Celery for background tasks

Step 1 — Install and verify Redis

sudo apt install -y redis-server
sudo systemctl enable redis-server
redis-cli ping

Configure Redis for production in /etc/redis/redis.conf:

maxmemory 256mb
maxmemory-policy allkeys-lru

Restart Redis:

sudo systemctl restart redis-server

Step 2 — Celery configuration

In your Django or Flask project, configure Celery to use Redis:

app.conf.update(
    broker_url='redis://127.0.0.1:6379/0',
    result_backend='redis://127.0.0.1:6379/1',
    task_serializer='json',
    result_serializer='json',
    accept_content=['json'],
    timezone='UTC',
    task_acks_late=True,
    worker_prefetch_multiplier=1,
    task_reject_on_worker_lost=True,
)

Setting task_acks_late=True with worker_prefetch_multiplier=1 ensures tasks are only acknowledged after completion, preventing data loss if a worker crashes mid-task.

Step 3 — Supervisor for Celery worker

sudo apt install -y supervisor

Create /etc/supervisor/conf.d/celery-worker.conf:

[program:celery-worker]
directory=/var/www/myapp
command=/var/www/myapp/venv/bin/celery -A myapp worker \
    --loglevel=info \
    --concurrency=4 \
    --max-tasks-per-child=200 \
    -Q default,high,emails
user=deploy
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
stopwaitsecs=600
redirect_stderr=true
stdout_logfile=/var/log/celery/worker.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=5

Key settings:

• --concurrency=4 sets 4 worker processes (adjust to CPU count)
• --max-tasks-per-child=200 recycles workers periodically to prevent memory leaks
• -Q default,high,emails subscribes to multiple queues
• stopwaitsecs=600 gives workers up to 10 minutes to finish current tasks on shutdown

Step 4 — Supervisor for Celery beat

Create /etc/supervisor/conf.d/celery-beat.conf:

[program:celery-beat]
directory=/var/www/myapp
command=/var/www/myapp/venv/bin/celery -A myapp beat \
    --loglevel=info \
    --schedule=/var/www/myapp/celerybeat-schedule \
    --pidfile=/var/run/celery/beat.pid
user=deploy
autostart=true
autorestart=true
numprocs=1
redirect_stderr=true
stdout_logfile=/var/log/celery/beat.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=3

Only ever run one beat instance. Multiple beat processes will duplicate scheduled tasks.

sudo mkdir -p /var/log/celery /var/run/celery
sudo chown deploy:deploy /var/log/celery /var/run/celery
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl status

Step 5 — Log rotation

Create /etc/logrotate.d/celery:

/var/log/celery/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    copytruncate
}

copytruncate is critical for Celery — it truncates the log in place rather than moving it, so Supervisor doesn't lose its file handle.

Step 6 — Concurrency tuning

Match concurrency to your workload:

• CPU-bound tasks (image processing, data crunching): set concurrency to CPU count
• I/O-bound tasks (API calls, email sending): set concurrency to 2-4x CPU count, or use gevent pool:

command=celery -A myapp worker --pool=gevent --concurrency=100

For gevent, install it first: pip install gevent.

Step 7 — Monitoring with Flower

Install and run Flower for real-time Celery monitoring:

pip install flower

Create /etc/supervisor/conf.d/celery-flower.conf:

[program:celery-flower]
directory=/var/www/myapp
command=/var/www/myapp/venv/bin/celery -A myapp flower \
    --port=5555 \
    --basic_auth=admin:secure-password \
    --broker=redis://127.0.0.1:6379/0
user=deploy
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/celery/flower.log

Protect Flower behind nginx with SSL — never expose it directly to the internet without authentication.

Graceful shutdown on deploy

When deploying new code, restart workers gracefully:

sudo supervisorctl signal HUP celery-worker:*

This tells workers to finish current tasks before restarting with the new code. For the beat scheduler, a simple restart is safe:

sudo supervisorctl restart celery-beat

With Reflex

Reflex monitors your Celery infrastructure holistically — tracking worker process health, queue depths, task success and failure rates, Redis memory usage, and beat schedule execution. It can restart crashed workers, alert on growing queue backlogs, and correlate task failures with deployment events. See How it works.