What this usually means
FastAPI's BackgroundTasks and Starlette's background task mechanisms rely on the ASGI lifespan protocol or request event loop. When tasks don't run, it's usually because: 1) The lifespan context never started (wrong lifespan handler), 2) The task is defined as a regular function but uses async operations without proper event loop, 3) The server configuration (e.g., gunicorn with multiple workers) kills worker before task completes, or 4) An unhandled exception in the task is silently swallowed by the ASGI server.
The first ten minutes — establish facts before touching code.
- 1Add a print() or logging statement as the very first line of your background task function. If it doesn't appear, the task was never called.
- 2Check your lifespan handler: if you use @app.on_event('startup') or lifespan context manager, ensure the background task registration is inside that block, not before it.
- 3Run with single worker: uvicorn main:app --workers 1. If it works, the issue is multi-worker concurrency (tasks run in worker that gets killed).
- 4Wrap the task call in a try/except and log the exception to stderr: sys.stderr.write(f'{e}\n'). Many servers (gunicorn) only forward stderr.
- 5Inspect the ASGI server logs for 'Task was destroyed but it is pending' or 'Exception in task' messages.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchmain.py or app.py — the lifespan handler registration
- searchGunicorn config file (gunicorn.conf.py) — timeout, workers, graceful-timeout settings
- searchUvicorn logs (stdout/stderr) — use --log-level debug
- searchApplication log file (if configured) — check for unhandled exceptions
- searchSystemd journal (if running as service): journalctl -u myapp.service -f
- searchDocker container logs: docker logs <container> --tail 50
- searchTask function itself — check if it's an async def when it should be sync, or vice versa
Practical causes, not theory. These are the things you will actually find.
- warningBackground task registered outside the lifespan context, so it's never scheduled
- warningAsync background task that does not await anything: the coroutine is never executed
- warningGunicorn's --timeout kills the worker before the task completes (default 30s)
- warningUvicorn's --limit-max-requests causes worker to restart before task finishes
- warningBackground task uses synchronous blocking I/O (e.g., requests.get) without thread pool, blocking the event loop
- warningTask raises an exception that is not caught; exception is silently dropped by the ASGI server
- warningFastAPI BackgroundTasks only work if the endpoint returns a response; if endpoint crashes, task never runs
Concrete fix directions. Pick the one that matches your root cause.
- buildMove background task registration inside the lifespan context manager (or @app.on_event('startup')).
- buildFor async tasks: ensure the function is async and you await all async calls. For sync tasks: wrap async calls in asyncio.run() (Python 3.7+) or use anyio.
- buildIncrease gunicorn timeout: --timeout 120 or set worker_class to 'uvicorn.workers.UvicornWorker' and adjust accordingly.
- buildUse @app.on_event('startup') with a background task that runs indefinitely (e.g., a consumer loop) and ensure it's spawned with asyncio.create_task().
- buildCatch all exceptions in the task and log them explicitly to a file or stderr.
- buildSwitch to Celery or a task queue for long-running background work that must survive worker restarts.
A fix you cannot prove is a guess. Close the loop.
- verifiedAdd an explicit log line at the start and end of the task: logging.info('Task started') / logging.info('Task finished').
- verifiedRun the server with --log-level trace to see ASGI lifecycle events.
- verifiedCheck the process list: ps aux | grep uvicorn; ensure worker count matches expected.
- verifiedSend a request that triggers the task and immediately check application logs for the start log.
- verifiedTest with a simple task: async def test_task(): print('hello') — if it prints, your setup is correct.
- verifiedSimulate a long task by adding asyncio.sleep(5) and verify it completes before worker timeout.
Things that make this bug worse or harder to find.
- warningDon't assume the task runs in the same request context; it runs after response is sent, so request-scoped dependencies are gone.
- warningDon't use BackgroundTasks for CPU-bound or very long tasks; use a proper task queue.
- warningDon't forget to await async functions inside the task; otherwise the coroutine is created but never executed.
- warningDon't ignore gunicorn's worker timeout; it's the #1 reason tasks vanish in production.
- warningDon't rely solely on print() for debugging; it may be buffered. Use logging with immediate flush or sys.stderr.write().
- warningDon't register the same background task multiple times (e.g., inside a loop) unless intentional.
The Missing Newsletter Job: A FastAPI Background Task That Vanished in Production
Timeline
- 09:15Deploy new feature: background task to send weekly newsletter via BackgroundTasks
- 09:20Test in staging: task runs fine with uvicorn --reload
- 09:25Deploy to production (gunicorn with 4 workers)
- 09:30Product manager reports no newsletter sent; no errors in logs
- 09:35Check application logs — nothing from the task
- 09:40Add logging to task start; redeploy; still no logs
- 09:45Inspect gunicorn config: timeout=30, workers=4
- 09:50Realize the task takes ~45 seconds to send 5000 emails
- 09:55Increase timeout to 120; task now works
We had a FastAPI endpoint that accepted newsletter submissions and used BackgroundTasks to send emails. In staging, with uvicorn --reload and a single worker, the task ran perfectly. Emails were sent, logs were clean. I was confident enough to ship to production.
Production ran four gunicorn workers behind nginx. After deployment, the product manager said no newsletter had been sent. I checked the logs—nothing. The endpoint returned 200, but the background task never produced any output. I added a log line at the very start of the task function and redeployed. Still nothing. That's when I knew the task was never even called.
I reviewed the gunicorn configuration and found timeout=30. Our email sending routine takes about 45 seconds for 5000 recipients. The worker was being killed by gunicorn before the task finished. Worse, the task was running synchronously inside the async endpoint (I used requests instead of httpx), blocking the event loop. The fix was to increase timeout to 120 and switch to async http client. After that, the task ran to completion every time.
Root cause
Gunicorn worker timeout (30 seconds) killed the worker before the synchronous background task finished sending emails. The task was also blocking the event loop because it used requests instead of an async HTTP client.
The fix
Increased gunicorn --timeout to 120 seconds and refactored email sending to use httpx.AsyncClient with await. Also added exception logging inside the task.
The lesson
Always test background tasks under production-like conditions: multiple workers, realistic timeouts, and synchronous vs async behavior. A task that works in dev can silently fail in prod.
FastAPI's BackgroundTasks is a thin wrapper around Starlette's BackgroundTasks. When you add a task via background_tasks.add_task(func, arg), it is stored in the request state and executed after the response is sent. The execution happens on the same event loop that handled the request. If your task is a coroutine function (async def), it is awaited; if it's a regular function, it is run in a thread pool (via anyio.to_thread.run_sync).
The key insight: the task is not scheduled until the response is ready to be sent. If the endpoint raises an exception before returning, the task is never executed. Also, the task runs in the same request's context, meaning request-scoped dependencies are no longer available. If you need database access, you must open a new session inside the task.
Gunicorn's default worker timeout is 30 seconds. When using Uvicorn workers (worker_class = 'uvicorn.workers.UvicornWorker'), the timeout applies to the entire worker process. If a background task takes longer than the timeout, the worker is killed and restarted. The task disappears without a trace. This is the single most common cause of 'background task not running' in production.
To fix, increase --timeout to a value larger than your longest expected task. But be careful: setting it too high can mask other issues (e.g., stuck tasks). A better approach is to offload long tasks to a separate worker process or a task queue like Celery. For tasks that must run in-process, ensure they are short (<30s) or use asyncio to yield control periodically.
If your background task is a regular function (def) that makes synchronous I/O calls (e.g., requests.get, time.sleep), it blocks the entire event loop while running. This means not only does it delay other requests, but if the task takes too long, it can cause the server to appear unresponsive. Uvicorn may even kill the worker if it detects a blocked event loop (though this is not default behavior).
The fix: either make the task async and use await on all I/O, or keep it sync but wrap blocking calls in asyncio.to_thread or run_in_executor. FastAPI's BackgroundTasks already runs sync functions in a thread pool, but if your sync function internally blocks the event loop again (e.g., by using asyncio.run()), you'll end up with nested event loops and errors like 'RuntimeError: asyncio.run() cannot be called from a running event loop'.
You cannot rely on print() alone; stdout may be buffered or redirected. Use the logging module with immediate flush: logging.basicConfig(level=logging.DEBUG, stream=sys.stderr, force=True). Then add logging.debug('Task started') as the first line of your task. If you see that log, the task was scheduled. If not, the issue is before the task runs.
For deeper inspection, enable ASGI trace logs: uvicorn main:app --log-level trace. This will show you when the lifespan starts, when the response is sent, and sometimes when the background task is executed. Also, use @app.on_event('startup') to run a simple background task that logs 'I am alive' every few seconds to verify the lifespan is active.
Sometimes you need a background task that runs continuously (e.g., a Kafka consumer) rather than per-request. In that case, BackgroundTasks is not the right tool. Use the lifespan context manager to start an asyncio task: async with lifespan(app): task = asyncio.create_task(my_consumer()); yield. The task will run for the lifetime of the application. Ensure you handle cancellation and cleanup.
A common mistake is to create the task outside the lifespan, e.g., at module level. That task will be created when the module is imported, which may happen before the event loop is running, or it may be created in every worker independently. Always use lifespan or @app.on_event('startup') for long-lived background tasks.
Frequently asked questions
Why does my FastAPI background task work with uvicorn but not with gunicorn?
Gunicorn introduces worker timeout and pre-fork model. The most common cause is the default 30-second timeout killing the worker before the task finishes. Also, with multiple workers, the task might run on a worker that gets restarted due to --max-requests. Increase timeout and test with a single worker first.
How do I pass database sessions to a background task?
You cannot reuse the request's session because it will be closed after the response. Instead, create a new session inside the task using your database factory. For SQLAlchemy, use SessionLocal() inside the task. For async, use async with async_session() as session:. Ensure you handle cleanup (close the session) in a finally block.
My background task runs but throws an exception that I never see. How do I catch it?
Wrap the entire task body in a try/except and log the exception to stderr (since stdout may be buffered). Use sys.stderr.write(f'Error: {e}\n') or logging.exception('Task failed'). In production, ensure your logging handler writes to a file or stdout immediately (e.g., with StreamHandler and flush=True).
Can I use BackgroundTasks for long-running tasks like video processing?
No. BackgroundTasks are designed for short, fire-and-forget operations that complete within a few seconds. For long-running tasks (minutes/hours), use a task queue like Celery, RQ, or Arq. Alternatively, spawn a separate process or thread, but that adds complexity. BackgroundTasks are not durable—if the server crashes, the task is lost.
What's the difference between BackgroundTasks and @app.on_event('startup')?
BackgroundTasks is per-request: it runs after the response is sent for that specific request. @app.on_event('startup') runs once when the application starts. Use startup for initialization (e.g., loading models) or to start a long-lived background task (e.g., a consumer loop). BackgroundTasks is for tasks triggered by a specific request.