It all started on a regular Friday evening. I was enjoying my time, and a critical issue waiting to be solved. Our Celery-powered task queue was performing well—until the Redis server went down. Tasks were lost, and my inbox was filled with alerts.

I knew we needed a more resilient setup. Enter Redis Sentinel—the guardian of high availability for Redis. My mission? Set up Redis with Sentinel and integrate it with Celery to ensure our background tasks never skip a beat.

Why Redis Sentinel?

Redis Sentinel acts as a watchdog for your Redis instances. It provides automatic failover, monitoring, and notifications, ensuring that if the primary Redis server goes down, another one takes its place without disrupting the application.

For Celery, this means:

No manual intervention when a Redis instance fails.

  • Tasks continue processing without data loss.
  • A more robust and fault-tolerant system.

Let’s dive into setting this up step by step.

Step 1: Setting Up Redis with Sentinel

First, we need to set up our Redis instances. We will configure three Redis nodes—one primary and two replicas.

1. Install Redis on Each Node

Run this on all three servers:

sudo apt update && sudo apt install redis-server -y

2. Configure Redis Instances

Each Redis server should have its own configuration file.

For the primary Redis node (redis-master), edit /etc/redis/redis.conf:

port 6379

bind 0.0.0.0

protected-mode no

logfile "/var/log/redis/redis-server.log"

appendonly yes

Restart Redis:

sudo systemctl restart redis

For the replica nodes (redis-replica1 and redis-replica2), update /etc/redis/redis.conf:

port 6379
replicaof <PRIMARY_IP> 6379

Restart each replica:

sudo systemctl restart redis

3. Configure Redis Sentinel

Now, let’s set up Sentinel to monitor these instances.

On each node, create a Sentinel configuration file /etc/redis/sentinel.conf:

port 26379

sentinel monitor mymaster <PRIMARY_IP> 6379 2

sentinel down-after-milliseconds mymaster 5000

sentinel failover-timeout mymaster 10000

sentinel parallel-syncs mymaster 1

Start Sentinel:

redis-server /etc/redis/sentinel.conf --sentinel

Now, Sentinel is watching over our Redis setup. If the primary node fails, Sentinel promotes a replica automatically.

Step 2: Configuring Celery with Redis Sentinel

Now that our Redis setup is resilient, we need to update Celery to connect to Sentinel instead of a single Redis instance.

1. Install Celery with Redis Support

pip install celery[redis]

2. Configure Celery to Use Redis Sentinel

Update your celery.py:

from celery import Celery

from redis.sentinel import Sentinel

sentinel = Sentinel([('sentinel1_host', 26379), ('sentinel2_host', 26379)], socket_timeout=0.1)

redis_client = sentinel.master_for('mymaster', socket_timeout=0.1)

app = Celery(
    'tasks',
    broker='redis://mymaster/0',
    backend='redis://mymaster/0'
)

app.conf.broker_transport_options = {'sentinel_kwargs': {'socket_timeout': 0.1}}

3. Start Celery Worker

celery -A tasks worker --loglevel=info

Now, Celery is connected to Redis Sentinel. If the master Redis instance fails, it will switch over to a replica automatically.

Step 3: Testing Failover

What’s the point of all this effort if we don’t test it?

  1. Run your Celery workers and launch a task.
  2. Manually stop the primary Redis server:
    sudo systemctl stop redis
  3. Check Sentinel logs (/var/log/redis/sentinel.log)—it should promote a replica.
  4. Celery should continue processing tasks seamlessly!

If you’re running Celery in production, don’t wait for a failure to happen—set up Redis Sentinel today.