What this usually means
These issues almost always indicate a logic flaw or misuse in the async/await model: missed awaits, orphaned tasks, improper use of locks or events, or exception paths that aren't surfaced. Asyncio's cooperative concurrency means a single bad actor—like a blocking call, a missed await, or an exception not handled in a task—can freeze or silently stall the whole system. Deadlocks are common if you interleave sync and async code, or if you use low-level primitives incorrectly.
The first ten minutes — establish facts before touching code.
- 1Enable asyncio debug mode by setting `PYTHONASYNCIODEBUG=1` and rerun the failing code.
- 2Wrap all entrypoint coroutines with try/except and explicit logging to stderr.
- 3Patch your event loop to log on every task creation: monkeypatch `asyncio.create_task`.
- 4List all active asyncio tasks at runtime: `for t in asyncio.all_tasks(): print(t, t._coro)`.
- 5Use `trio-asyncio`'s deadlock detector or run with `pytest-asyncio` plus `pytest-timeout` to catch hangs.
- 6Watch for blocking sync calls via `asyncio.to_thread` or offending libraries — use `strace` or `py-spy`.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchasyncio debug logs: stdout/stderr with `PYTHONASYNCIODEBUG=1`
- searchCoroutines source code in suspected modules (look for forgotten `await`)
- searchTask creation sites: search for `create_task`, `ensure_future`, or `call_later`
- searchException handlers in coroutines and event loop's `set_exception_handler`
- searchlong-running or blocking operations (e.g., file I/O, HTTP libraries)
- searchTracebacks in `.result()` or `.exception()` calls from completed tasks
- searchActive tasks at breakpoint/REPL: `asyncio.all_tasks()`
Practical causes, not theory. These are the things you will actually find.
- warningForgotten `await` on coroutine calls, causing never-scheduled coroutines
- warningUncaught exceptions inside tasks, allowing them to silently terminate
- warningCreating tasks without keeping references, causing garbage collection & warnings
- warningBlocking synchronous code running in the event loop (disk I/O, CPU work)
- warningDouble-awaiting or deadlocks via improper use of `asyncio.Lock` or `asyncio.Event`
- warningMixing threading and asyncio primitives without coordination
Concrete fix directions. Pick the one that matches your root cause.
- buildAudit and add missing `await` keywords everywhere coroutines are called
- buildWrap all task coroutines with robust try/except and explicit logging
- buildHold references to tasks until they're done; collect results or handle exceptions
- buildMove blocking logic to `asyncio.to_thread`, or use native async libraries
- buildCarefully review lock/event usage, especially for circular waits
- buildHook a custom event loop exception handler to surface and log hidden errors
A fix you cannot prove is a guess. Close the loop.
- verifiedNo more pending tasks on shutdown: `assert not asyncio.all_tasks()`
- verifiedNo 'Task was destroyed but it is pending!' warnings at exit
- verifiedAll coroutines either complete or fail visibly with tracebacks in logs
- verifiedEvent loop CPU usage remains low in idle state
- verifiedRepeat with `PYTHONASYNCIODEBUG=1` — no unexpected warnings or errors
- verifiedFunctional tests/timing checks show no more random hangs or timeouts
Things that make this bug worse or harder to find.
- warningIgnoring 'Task was destroyed but it is pending!' warnings
- warningUsing print-debugging inside coroutines only — logs may never flush if event loop dies
- warningFailing to check `.exception()` on finished tasks
- warningLetting tasks be garbage collected without handling results
- warningAssuming all libraries called are async-safe (many are not)
Silent Asyncio Deadlock Drains API Fleet
Timeline
- 13:00Alerts: API pod CPU at 98%, but zero new requests processed.
- 13:10kubectl exec + ps shows uvicorn workers alive, no error logs.
- 13:15Set PYTHONASYNCIODEBUG=1, redeploy; see 'Task was destroyed but it is pending!' warnings.
- 13:25Attaching remote Python REPL, dump asyncio.all_tasks() finds multiple long-lived tasks stuck on await.
- 13:35Review code: spot missing await on async HTTP call inside cache refresh routine.
- 13:40Hotfix: add missing await, wrap whole coroutine with try/except/logging.
- 13:50API pod throughput returns to normal, warnings gone on redeploy.
Our FastAPI-based backend began stalling under moderate load—CPU shot to 98% but the metrics dashboard showed no new requests processed and no error logs.
We redeployed with asyncio debug mode, which immediately surfaced 'Task was destroyed but it is pending!' warnings. Digging deeper with an interactive REPL inside a pod, we listed all asyncio tasks and found several stuck indefinitely.
Tracing one, we spotted that an async HTTP call in a scheduled cache updater lacked an 'await', causing an unscheduled coroutine and cascading deadlocks. After patching with the missing 'await' and adding logging and error handling, traffic returned to normal.
Root cause
A missing 'await' on an async HTTP request led to coroutines never being scheduled, resulting in orphaned pending tasks that deadlocked the event loop.
The fix
Added the missing 'await', improved logging around the task, and ensured all tasks are properly awaited and their exceptions handled.
The lesson
Asyncio failures hide in silence—always enable debug mode in staging, check for orphaned tasks, and mandate code review for missing 'await' statements.
When an asyncio coroutine throws an exception that isn't handled, and the task object isn't kept or checked, the exception disappears except for a warning at garbage collection. This is why failures can be so hard to spot.
To surface these, always add explicit try/except blocks with logging in every top-level coroutine, and use a custom event loop exception handler: loop.set_exception_handler().
Unlike thread-based deadlocks, asyncio deadlocks often occur when a coroutine never yields, misses an 'await', or all tasks are waiting on each other via locks or events.
List all tasks with asyncio.all_tasks(), then inspect each task's coroutine and stack to see where they're blocked. Tools like asyncio run with debug mode and third-party deadlock detectors (e.g., from the trio ecosystem) are invaluable for this.
Even a single blocking call (e.g., disk I/O, synchronous DB client) can freeze the event loop for all tasks. Use 'py-spy' to profile running code and look for suspiciously long on-CPU times.
Migrate all such logic to native async libraries, or explicitly delegate to a thread with asyncio.to_thread().
Search codebase for every coroutine call and verify an 'await' keyword is present. Static analysis tools like flake8-async can help, but code review still catches subtle misses.
Add tests with hooks that enumerate asyncio.all_tasks() to ensure no unexpected tasks are left pending at test teardown.
Frequently asked questions
How do I see all tasks running in my event loop?
In a running event loop, call 'asyncio.all_tasks()' and print each task with its stack. This shows live, pending, and finished tasks.
Why am I not seeing exceptions from failed coroutines?
If tasks are garbage collected without .result()/.exception() called, exceptions are only logged as warnings or may be missed entirely unless debug mode is enabled.
How can I detect blocking synchronous code in asyncio?
Profile the process with 'py-spy top' or run with 'PYTHONASYNCIODEBUG=1', which will log slow callbacks and blocking calls in the event loop.
What does 'Task was destroyed but it is pending!' mean?
It means a coroutine was never awaited to completion and was deleted mid-execution—usually a missing 'await', dropped task reference, or silent exception.
Can I set a global handler for uncaught asyncio exceptions?
Yes, use 'loop.set_exception_handler()' to catch and log all uncaught exceptions on the event loop, surfacing hidden bugs faster.