Python Celery Beginner Tutorial

Infrastructure eventually has to deal with time-consuming tasks, like processing images or sending emails, without making users wait for them to finish. Celery is a Python framework built for this. You mark a function as a task using @app.task, and Celery automatically routes that work to background processes to execute.

Why does Celery exist?

Celery was created in 2009 by Ask Solem, originally to solve a problem in the Django community: a web request that triggers slow work shouldn’t have to wait for that work to finish before responding to the user. Before Celery, teams either accepted the slow response or hand-rolled their own background-job plumbing for every project.

What makes Celery a distributed task queue is the execution boundary: the process that triggers a task does not need to run it. You define the task in code, and Celery offloads the actual execution to a separate pool of worker processes. These workers can run locally on your machine or scale out across a fleet of remote servers.

Celery has since become the de facto standard for background task processing in Python, well beyond its Django roots, used for everything from sending emails to running data pipelines. LavinMQ is one of the brokers it can use underneath to move those tasks between producers and workers.

Why Celery still needs a broker like LavinMQ

Celery doesn’t move messages between your producer and your workers itself - it needs a broker to hold tasks until a worker is free to pick them up, and to make sure a task isn’t lost if a worker dies mid-task. LavinMQ manages the queue declarations, serialization, and acknowledgements underneath Celery, so you write task functions instead of wiring up channels by hand.

This tutorial walks through setting the two up together, using image resizing as the example task.

Setup

You’ll need LavinMQ running, plus Celery:

pip install celery

Configuring Celery to use LavinMQ

Create a celeryconfig.py file:

# celeryconfig.py

broker_url = 'lavinmq://<username>:<password>@<lavinmq_host>:<lavinmq_port>/<virtual_host>'

# Recommended settings for local LavinMQ
broker_pool_limit = 1
broker_heartbeat = None
broker_connection_timeout = 30
result_backend = None
event_queue_expires = 60
worker_prefetch_multiplier = 1
worker_concurrency = 4  # adjust based on your system's capabilities

worker_prefetch_multiplier = 1 keeps a worker from grabbing more tasks than it can work on at once, so a burst of tasks gets spread across every worker you have running rather than piling up on the first one.

Defining the task

Create tasks.py:

# tasks.py

import time
from celery import Celery

app = Celery('tasks')
app.config_from_object('celeryconfig')

@app.task
def resize_image(image_name):
    print(f"[recv] Received {image_name}")
    time.sleep(5)  # simulate a slow resize
    print(f"[done] {image_name} resized!")

The @app.task decorator is doing the heavy lifting here - it turns a plain Python function into something that can be published to a queue and picked up by a worker, without you having to declare a queue or handle acknowledgements yourself.

Sending tasks - the producer

# producer.py

from tasks import resize_image

for i in range(1, 7):
    resize_image.delay(f"image-{i}")
    print(f"[queued] Queued image-{i}")

.delay() publishes the task to the broker instead of running it in this process. There’s no exchange or routing key to think about - Celery handles that based on the task’s name.

Running the worker

Open a terminal and start a worker:

celery -A tasks worker --loglevel=info

In a second terminal, run the producer:

python producer.py

You’ll see the worker pick up each resize_image task in turn. Start a second worker in a third terminal and run the producer again. The two workers will compete for tasks rather than both processing every one, so the work gets split between them.

Monitoring with Celery Flower

Flower gives you a web UI over your workers and tasks:

pip install flower
celery -A tasks flower

Visit http://localhost:5555 to see active workers, task history, and throughput.

Learning lab: what happens when a worker dies mid-task?

• Set time.sleep(60) in resize_image to simulate a slow task.
• Start one worker, then run the producer with a single task queued.
• Kill the worker (Ctrl+C) after a few seconds, before it finishes sleeping.
• Start a new worker and watch what happens.

Question to think about: does the task run again on the new worker, or is it gone? Celery acks a task once it returns, not before, so a worker that dies mid-task hasn’t acked yet. Look up Celery’s task_acks_late setting and how it interacts with worker_prefetch_multiplier - what’s the tradeoff of acking late versus acking early?