What this usually means
At the core, this means the task message was successfully pushed to the broker (Redis, RabbitMQ, or SQS) but was never consumed by a worker—or was consumed but failed before any logging. The most common causes are: (1) the worker is configured with a different app name, queue, or serializer than the producer; (2) the worker crashed due to an import error or runtime exception during task loading; (3) the broker is reachable but the worker cannot connect (e.g., wrong broker URL, TLS mismatch); (4) the task is registered under a different name (e.g., relative vs absolute import path). Because Celery swallows many errors by default—especially in the eager path—the failure is silent unless you explicitly enable debug logging.
The first ten minutes — establish facts before touching code.
- 1Check the worker process: `ps aux | grep celery` — is it running? If not, start it with `celery -A your_project worker -l debug` and watch stdout.
- 2Send a test task from a Django shell: `from your_app.tasks import my_task; my_task.delay()`. Then check the worker logs for 'Got task from broker'.
- 3Inspect the broker queue: for Redis, `redis-cli llen celery`; for RabbitMQ, `rabbitmqctl list_queues`. If the count increases but the worker doesn't consume, the worker is not connected to the same queue.
- 4Enable debug logging on the worker: run `celery -A your_project worker -l debug` and look for lines like 'connected' or 'consumer: Ready to accept tasks'. Also check for any 'Received unregistered task' warnings.
- 5Verify task registration: `celery -A your_project inspect registered` to list all tasks the worker knows. Ensure your task name appears exactly as it was sent (e.g., 'myapp.tasks.my_task' vs 'my_task').
The specific files, logs, configs, and dashboards that usually own this bug.
- searchWorker stdout/stderr — run with `-l debug` and capture all output
- searchBroker logs: Redis `/var/log/redis/redis-server.log` or RabbitMQ logs
- searchCelery logs configured via `CELERYD_LOG_FILE` or Django logging handlers
- searchFlower dashboard (if installed) for task state and worker health
- searchDjango settings file — check CELERY_BROKER_URL, CELERY_TASK_SERIALIZER, CELERY_RESULT_BACKEND
- searchTask module `__init__.py` and `apps.py` — ensure `@shared_task` is imported and the app is registered
- searchDjango shell — `from celery import current_app; current_app.amqp.queues` to verify queue definitions
Practical causes, not theory. These are the things you will actually find.
- warningWorker and producer use different broker URLs (e.g., localhost vs 127.0.0.1, or different Redis database numbers)
- warningTask imported with a relative path in `@shared_task` but sent from a different module (e.g., `tasks.my_task` vs `myapp.tasks.my_task`)
- warningWorker started with `-A proj.settings` instead of `-A proj` (pointing to the settings module, not the Celery app)
- warningBroker connection is blocked by a firewall or Redis is configured with `requirepass` and the URL lacks the password
- warningWorker process ran out of memory or hit a Python import error on startup — but the error scrolled off the terminal
- warningCelery app instance is created in a module that is not imported early enough (e.g., inside a Django `ready()` method that runs after the worker starts)
Concrete fix directions. Pick the one that matches your root cause.
- buildUse absolute import paths in `@shared_task` and always send tasks via `app.task` or `shared_task` with explicit `name` argument
- buildRun the worker with `--pidfile` and `--logfile` to capture startup errors that scroll away
- buildSet `CELERY_TASK_SERIALIZER = 'json'` and `CELERY_ACCEPT_CONTENT = ['json']` to avoid pickle mismatches
- buildUse `app.conf.task_default_queue = 'celery'` (or a consistent queue name) to ensure producer and worker target the same queue
- buildDeploy with a process manager (supervisor, systemd) that restarts the worker on crash and logs stdout/stderr
- buildIn Django settings, ensure `CELERY_BROKER_URL` is read from environment variables and is identical across all services
- buildAdd a health check task that returns a simple result to verify end-to-end execution
A fix you cannot prove is a guess. Close the loop.
- verifiedRun `celery -A proj inspect ping` — should return 'pong' from the worker
- verifiedCall a task synchronously with `my_task.apply()`. If it works but `delay()` doesn't, the broker is the issue.
- verifiedWatch the worker debug logs as you send a task — expect a line like 'Task accepted: myapp.tasks.my_task[abc123]'
- verifiedCheck the broker queue length before and after sending a task: it should increase, then decrease once the worker picks it up.
- verifiedUse `result.get(timeout=10)` in a Django shell to block until the task completes — if it hangs, the task never ran.
- verifiedDeploy a simple 'ping' task that writes a timestamp to a file. Run it via `delay()` and verify the file appears.
Things that make this bug worse or harder to find.
- warningDon't assume the task is 'lost' without checking the broker queue first — it's usually sitting there
- warningDon't use `CELERY_ALWAYS_EAGER = True` in production; it hides all broker issues
- warningDon't rely on Flower alone — it polls the result backend and can misreport state if the backend is misconfigured
- warningDon't forget that Celery tasks are serialized at send time; any change to the task function signature after sending breaks execution
- warningDon't run the worker with `--autoreload` in production — it causes subtle race conditions on code reload
- warningDon't ignore worker startup warnings about 'unregistered task' — that's the exact cause most of the time
The Silent Queue: Why Our Celery Worker Wouldn't Eat Tasks
Timeline
- 14:22Deploy new feature that adds a background email task
- 14:30Users report emails not sent; no errors in Django logs
- 14:35Check Flower: all tasks show 'PENDING' and never transition
- 14:38SSH into worker server; `ps aux | grep celery` shows worker is running
- 14:42Check supervisor logs: no errors, worker started normally
- 14:45Add debug logging to task: still nothing. Send task from shell — no worker log
- 14:50Check Redis queue: `llen celery` shows tasks piling up
- 14:52Inspect registered tasks on worker: `celery -A proj inspect registered` — only old tasks listed, not the new email task
- 14:55Discover new task is in a different module: `myapp.jobs.email_task` vs `myapp.tasks.email_task`
- 15:00Fix the import path, restart worker, tasks start executing immediately
We deployed a new feature that sends email notifications using Celery. The Django view called `email_task.delay(user_id)`. No errors in the view, but emails never arrived. I checked Flower — all tasks showed 'PENDING' and stayed there. Our worker was running under supervisor, no crashes. I went to the worker server and checked the supervisor log — nothing. Ran `celery -A proj status` — worker was alive. I tried sending a simple ping task from a Django shell — still nothing in the worker logs.
I checked Redis with `redis-cli llen celery` — the queue had tons of messages. That meant the producer was pushing, but the worker wasn't consuming. I ran `celery -A proj inspect registered` on the worker and saw only the old tasks. The new `email_task` wasn't registered. Then I realized: the `@shared_task` decorator used `from . import email_task` in `myapp/jobs.py`, but the producer imported it as `from myapp.tasks import email_task`. The task name in the message was `myapp.tasks.email_task`, but the worker only knew `myapp.jobs.email_task`. Classic import path mismatch.
I renamed the task module to `myapp/tasks.py` and updated the import. Restarted the worker with `supervisorctl restart celery`. Within seconds, the queued tasks started executing. The lesson: always use absolute imports in `@shared_task` and verify task registration after deployment. Also, never assume the worker knows about a task just because it's imported somewhere — check `inspect registered` explicitly.
Root cause
Task registered under a different Python module path than the one used to send it, causing the worker to reject the message as 'unregistered task'.
The fix
Moved the task definition to `myapp/tasks.py` and updated the import in both the producer and the task module to use the same absolute path. Restarted the worker.
The lesson
Always verify task registration with `celery inspect registered` after adding new tasks. Use consistent module paths and consider using the `name` argument in `@shared_task` to decouple the task name from the Python path.
When you call `task.delay(args)`, Celery serializes the task name, arguments, and metadata into a message and publishes it to the broker (e.g., Redis list or RabbitMQ exchange). The message includes a 'task' field containing the full dotted path to the task function (e.g., 'myapp.tasks.my_task'). The worker continuously polls the broker for new messages. When it receives one, it deserializes the message, looks up the task by that dotted path in its registry, and executes it. If the worker's registry does not contain that exact path, it logs a warning (at DEBUG level) and rejects the message (or moves it to a dead-letter queue, depending on configuration). This is the most overlooked cause of 'task not executing.'
The registry is built when the worker starts: it imports all modules listed in `CELERY_IMPORTS` or discovered via Django's `INSTALLED_APPS`. Any task decorated with `@shared_task` is added to the registry under its natural Python import path. This path can change if the module is imported differently (e.g., relative vs absolute) or if the task is defined in a submodule that isn't imported early enough. Always use explicit `name` parameter in `@shared_task` to decouple the task name from the file location, e.g., `@shared_task(name='send_email')`.
Even if `CELERY_BROKER_URL` looks correct, subtle differences can break connectivity. For Redis, ensure the database number matches (default 0). If Redis requires a password, include it in the URL: `redis://:password@host:6379/0`. For RabbitMQ, the virtual host (`/` by default) must exist and the user must have permissions on it. Also, check that the worker can resolve the broker hostname — DNS issues are common in containerized environments. Use `celery -A proj inspect ping` to verify connectivity; if it fails, the error message usually points to the broker.
Another silent failure: the broker connection is established but the worker cannot subscribe to the correct queue. By default, Celery uses a queue named 'celery'. If you changed the task queue name in Django settings (e.g., `CELERY_TASK_DEFAULT_QUEUE = 'myqueue'`), you must start the worker with `-Q myqueue` or set the same queue in the worker config. Otherwise, the worker listens on 'celery' while tasks go to 'myqueue'. Use `celery -A proj inspect active_queues` to see which queues the worker is consuming from.
When you start a Celery worker in the background (e.g., via supervisor), any error during import of tasks or initialization of the Celery app is printed to stderr but often not captured. Common examples: a syntax error in a tasks file, a missing dependency, or a database connection failure during app startup. Because the worker exits immediately after the error, it may not log a clear message in supervisor's log if stderr isn't redirected. To catch these, run the worker manually once: `celery -A proj worker -l debug` in the foreground. Look for tracebacks after 'celery.worker.bootsteps: Worker: Starting'. Also configure Celery's logging to a file via `CELERYD_LOG_FILE` or a dedicated Django logger.
Another subtlety: if you use Django's `AppConfig.ready()` to import tasks (e.g., to register signals), and that method raises an error, the entire app fails to load. The worker may show a generic 'WorkerStopped' error. Wrap `ready()` contents in a try-except and log the exception. Also, ensure that the Celery app instance is created in a module that is imported early — typically `celery.py` at the project root, which is then imported in `proj/__init__.py`.
By default, Celery uses JSON serialization for tasks. However, if the producer (Django) uses `CELERY_TASK_SERIALIZER = 'pickle'` and the worker expects JSON (or vice versa), the worker will fail to deserialize the message and log an error like 'ContentDisallowed: Refusing to deserialize untrusted content type'. This is often silent because Celery logs it at WARNING level, which may be suppressed in production. The fix is to set `CELERY_TASK_SERIALIZER = 'json'` and `CELERY_ACCEPT_CONTENT = ['json']` on both sides. If you must use pickle (e.g., for complex objects), ensure both sides agree and set `CELERY_ACCEPT_CONTENT = ['pickle', 'json']`.
Also, the result backend serializer (`CELERY_RESULT_SERIALIZER`) can cause similar issues if the worker cannot write results. If `result.get()` hangs, check that the result backend URL is correct and the serializer is supported. Tools like Flower rely on the result backend; if it's misconfigured, tasks appear stuck even though they completed.
`CELERY_ALWAYS_EAGER = True` forces all tasks to execute synchronously in the caller process, bypassing the broker entirely. This is useful for testing but catastrophic in production if left enabled — it hides broker issues and can cause performance problems. I've seen teams accidentally leave it on after a test, then wonder why their worker never receives tasks. Always check that this setting is `False` (or not set) in production. You can verify by running `celery -A proj inspect stats` and looking for `total` task counts — if the worker shows zero tasks executed despite many being sent, eager mode may be on.
Additionally, `task_always_eager` can be set per-task via the `@shared_task(eager=True)` decorator. If you have a mix of eager and non-eager tasks, the eager ones run synchronously, and the non-eager ones go to the broker — but if the worker is down, only the eager ones appear to work. This creates a confusing half-working state.
Frequently asked questions
My task shows PENDING forever even though the worker is running. What's wrong?
This usually means the task was sent to the broker but never consumed by the worker. First, check the broker queue length: for Redis, `redis-cli llen celery`. If the count is increasing, the worker is not consuming from that queue. Run `celery -A proj inspect active_queues` to see which queues the worker listens to. Also check `celery -A proj inspect registered` to ensure the task is registered. If the task isn't in the registry, the worker silently ignores it (or logs a DEBUG message).
How do I see if a task was rejected by the worker?
Run the worker with `-l debug` and look for lines like 'Received unregistered task of type myapp.tasks.my_task'. The worker logs a warning at DEBUG level when it receives an unknown task. By default, it discards the message. You can configure a dead-letter queue to capture these rejected tasks — set `CELERY_TASK_REJECT_ON_WORKER_LOST = True` and use a custom queue with proper dead-letter settings on the broker (e.g., Redis or RabbitMQ).
Can a firewall block Celery tasks?
Yes. If the worker cannot connect to the broker (e.g., Redis port 6379 or RabbitMQ ports 5672/15672), tasks will sit in the producer's outbound buffer or fail immediately with a connection error. The producer (Django) may not show an error if the broker connection is async — the message is queued locally and lost if the connection never succeeds. Always verify connectivity: `redis-cli -h <host> ping` from the worker server. Also check that security groups or iptables allow outbound traffic to the broker.
Why does my task work when I call it synchronously with `apply()` but not with `delay()`?
`apply()` executes the task in the current process (like a normal function call), bypassing the broker. `delay()` sends the task to the broker for async execution. If `apply()` works but `delay()` doesn't, the issue is definitely in the broker or worker configuration — not the task code itself. Check broker connectivity, queue names, and worker registration.
I have multiple Celery apps — can they interfere?
Yes. If you create multiple Celery app instances (e.g., one in `celery.py` and another in a test file), they each have their own registry and configuration. Tasks registered on one app won't be visible to workers started with another app. Always use a single app instance, and import it consistently. Use `from myproject.celery import app` everywhere. Also, be careful with `@shared_task` — it binds to the app instance that is imported first, which may not be the one you intend.