What this usually means
Virtual threads are lightweight but still execute on platform (carrier) threads. The common failure modes are: 1) Pinning—a virtual thread holds a monitor (synchronized) and blocks its carrier thread, preventing other virtual threads from using that carrier. 2) Carrier thread starvation—when all carrier threads are pinned, no virtual threads can make progress. 3) Deadlock between virtual threads or between virtual and platform threads, often masked because traditional thread dump tools show them as RUNNABLE. The root cause is usually misuse of synchronized blocks, native calls, or thread-local operations that pin the virtual thread to the carrier.
The first ten minutes — establish facts before touching code.
- 1Check pinned virtual threads count: run 'jcmd <pid> Thread.vthread_stats' and look for 'pinned' count > 0
- 2Enable JFR and record: 'jcmd <pid> JFR.start name=vtrace settings=profile duration=60s filename=vtrace.jfr'
- 3Analyze JFR in JDK Mission Control: look for 'Virtual Thread Pinned' events and 'Carrier Thread Wait' events
- 4Take a thread dump with native stack: 'jcmd <pid> Thread.print -l' and search for 'virtual' and 'pinned'
- 5Monitor carrier thread pool: use 'jcmd <pid> Thread.vthread_stats' and check 'carrier' count vs active
- 6Check for deadlocks: 'jcmd <pid> Thread.print -l' and look for 'Found one Java-level deadlock'
The specific files, logs, configs, and dashboards that usually own this bug.
- searchjcmd Thread.vthread_stats output: shows carrier threads, mounted, unmounted, pinned virtual threads
- searchJFR event 'Virtual Thread Pinned' with stack trace: identifies the exact synchronized block or native method
- searchThread dump with -l flag: shows lock information and carrier thread mapping
- searchApplication logs for thread names: virtual threads are named 'virtual-<id>' but carrier threads are 'ForkJoinPool-1-worker-<n>'
- searchJFR event 'jdk.VirtualThreadStart' and 'jdk.VirtualThreadEnd': tracks lifecycle
- searchCarrier thread pool configuration: default is ForkJoinPool with parallelism = number of cores
Practical causes, not theory. These are the things you will actually find.
- warningsynchronized blocks or methods inside virtual threads: pins the carrier thread
- warningCalling native methods (e.g., JNI, socket I/O with blocking mode) that don't yield
- warningUsing ThreadLocal with virtual threads in a way that forces pinning
- warningLimited carrier thread pool size: default is Runtime.availableProcessors(), too small for many virtual threads
- warningDeadlock between virtual threads due to improper lock ordering
- warningThird-party libraries that use synchronized or native operations internally
Concrete fix directions. Pick the one that matches your root cause.
- buildReplace synchronized blocks with ReentrantLock or other java.util.concurrent locks
- buildFor unavoidable synchronized blocks, ensure they are short and non-blocking
- buildIncrease carrier thread pool size via jdk.virtualThreadScheduler.parallelism system property
- buildUse structured concurrency (StructuredTaskScope) to manage virtual thread lifecycle and timeouts
- buildAvoid ThreadLocal in virtual threads; use ScopedValue (Java 20+) instead
- buildEnsure all I/O is non-blocking or uses virtual-thread-aware APIs
A fix you cannot prove is a guess. Close the loop.
- verifiedRun jcmd Thread.vthread_stats before and after fix: pinned count should drop to zero
- verifiedObserve throughput under load: should scale linearly with number of virtual threads
- verifiedCheck JFR 'Virtual Thread Pinned' events: should be absent after fix
- verifiedPerform stress test with many concurrent virtual threads (e.g., 10000) and verify no carrier starvation
- verifiedMonitor carrier thread usage: each virtual thread should be unmounted when blocking
- verifiedRegression test with thread dump analysis: ensure no deadlocks appear
Things that make this bug worse or harder to find.
- warningAssuming virtual threads eliminate all concurrency bugs—they don't
- warningUsing synchronized as a quick fix for thread safety without considering pinning
- warningIgnoring carrier thread pool sizing: default may be too small for high concurrency
- warningUsing ThreadLocal without understanding it pins virtual threads
- warningRelying solely on jstack for debugging virtual threads—it shows them as RUNNABLE even when blocked
- warningNot using JFR—it's the most powerful tool for virtual thread debugging
Carrier Starvation Under Load During Payment Processing
Timeline
- 14:02PagerDuty alert: payment processing latency p99 spikes from 200ms to 12s
- 14:05Checked HikariCP pool: 50 connections, utilization 80% — not the bottleneck
- 14:08jcmd Thread.vthread_stats showed 2000 virtual threads, 15 carrier threads, 12 pinned
- 14:12Captured JFR: 'Virtual Thread Pinned' events all pointing to a synchronized block in PaymentService.validate()
- 14:18Code review: validate() used synchronized(this) for caching — pinning carrier threads
- 14:25Deployed fix: replaced synchronized with ReentrantLock and scoped cache per virtual thread
- 14:30p99 latency back to 210ms, pinned count dropped to 0
We had just migrated to virtual threads in Java 21. The payment service was handling 10k req/s, and initially everything looked great—lower memory, higher throughput. Then the p99 latency graph went vertical. At first I suspected the database, but connections were fine. I checked the carrier thread pool stats and saw 12 pinned threads out of 16 carriers—that's essentially a full stall.
I enabled JFR with the profile settings and captured a 60-second recording. The 'Virtual Thread Pinned' event was a goldmine: every pinned event pointed to a synchronized block in PaymentService.validate(). The code was using synchronized(this) to guard a simple cache lookup—something that worked fine with platform threads but became a pinning bottleneck under virtual threads.
The fix was straightforward: replace synchronized with a ReentrantLock. But we also learned that the cache itself didn't need to be shared; we used ScopedValue to cache per virtual thread. After deploying, the pinned count dropped to zero, and latency returned to normal. The lesson: always profile virtual thread behavior with JFR before assuming they'll just work.
Root cause
synchronized block in PaymentService.validate() pinned carrier threads, causing starvation under load
The fix
Replaced synchronized with ReentrantLock and moved cache to ScopedValue per virtual thread
The lesson
Virtual threads require careful avoidance of pinning constructs; use JFR to detect them early.
A virtual thread is said to be 'pinned' when it is mounted on a carrier thread and cannot be unmounted during a blocking operation. This happens when the virtual thread holds a monitor (synchronized block) or is executing native code (JNI) that doesn't support unmounting.
When a virtual thread is pinned, the carrier thread is also blocked. If many virtual threads become pinned, all carrier threads can be occupied, causing starvation. The default carrier thread pool is a ForkJoinPool with parallelism equal to the number of CPU cores. With 16 cores, only 16 virtual threads can run concurrently; if they all become pinned, no other virtual threads can execute.
To detect pinning, use JFR events 'jdk.VirtualThreadPinned' and 'jdk.VirtualThreadSubmitFailed'. The JFR event includes the stack trace of the pinning operation. You can also use 'jcmd <pid> Thread.vthread_stats' to see the current pinned count.
The default carrier thread pool size is determined by the system property 'jdk.virtualThreadScheduler.parallelism'. If not set, it defaults to Runtime.availableProcessors(). This is often too small for applications that create thousands of virtual threads, especially if some get pinned.
Symptoms of carrier starvation: high response times, many virtual threads in RUNNABLE state but making no progress, and JFR events showing carrier threads waiting for work. You can increase the pool size by setting 'jdk.virtualThreadScheduler.parallelism' to a higher value, e.g., '-Djdk.virtualThreadScheduler.parallelism=64'.
However, increasing the pool size is a band-aid. The real fix is to eliminate pinning. A larger pool just delays the problem. Monitor 'jcmd Thread.vthread_stats' for pinned count; if it's non-zero, address those pinning sources.
Deadlocks can still occur with virtual threads, but they are harder to detect because traditional thread dumps (jstack) show virtual threads as RUNNABLE even when they are blocked. The lock information is present but the thread state is misleading.
To diagnose a deadlock, use 'jcmd <pid> Thread.print -l' and look for the 'Found one Java-level deadlock' section. This works for virtual threads as well. Additionally, JFR events 'jdk.JavaMonitorEnter' and 'jdk.JavaMonitorWait' can help trace lock contention.
Common deadlock patterns: two virtual threads acquire locks in different order, or a virtual thread holds a lock while waiting for a platform thread that holds another lock. Use structured concurrency (StructuredTaskScope) to enforce timeouts and cancellation, which can break deadlocks.
JFR (Java Flight Recorder) is the most powerful tool for debugging virtual threads. Enable it with: 'jcmd <pid> JFR.start name=vtrace settings=profile duration=60s filename=vtrace.jfr'. Then analyze with JDK Mission Control (JMC) or 'jfr print'.
Key JFR events: 'jdk.VirtualThreadStart', 'jdk.VirtualThreadEnd', 'jdk.VirtualThreadPinned', 'jdk.VirtualThreadSubmitFailed', 'jdk.CarrierThreadWait'. The 'VirtualThreadPinned' event includes the stack trace of the pinning operation.
The jcmd tool provides several subcommands: 'Thread.vthread_stats' for a summary, 'Thread.print -l' for a detailed thread dump with locks, and 'Thread.dump_to_file' to save the dump. Use these in combination with JFR for a complete picture.
Replace synchronized blocks with java.util.concurrent locks like ReentrantLock. These locks allow the virtual thread to be unmounted while waiting. Similarly, use Semaphore, CountDownLatch, etc.
For thread-local storage, use ScopedValue (incubator in Java 20, preview in Java 21). ScopedValue allows sharing data without pinning the virtual thread. Unlike ThreadLocal, ScopedValue is inherited by child virtual threads and is automatically cleaned up.
When using I/O, ensure it's non-blocking or uses virtual-thread-aware APIs (e.g., java.nio.channels with virtual threads). Avoid blocking on Future.get() without timeout; use CompletableFuture or StructuredTaskScope with timeouts.
Frequently asked questions
Why does jstack show virtual threads as RUNNABLE even when they are blocked?
Virtual threads are lightweight and their state is managed by the scheduler. jstack shows the state of the carrier thread, not the virtual thread. If a virtual thread is blocked but not pinned, the carrier thread is free to run other virtual threads, so jstack shows the carrier thread as RUNNABLE. To see the true state, use JFR events or jcmd with specific virtual thread details.
How do I know if my synchronized block is causing pinning?
Enable JFR with the profile setting and look for 'jdk.VirtualThreadPinned' events. The event includes the stack trace of the pinned operation. You can also use jcmd Thread.vthread_stats to see the current pinned count. If you see non-zero pinned count under load, investigate the stack traces.
Can I use ThreadLocal with virtual threads?
Yes, but with caution. ThreadLocal in a virtual thread that is pinned can cause issues because the underlying carrier thread may be reused. More importantly, ThreadLocal prevents unmounting if the virtual thread is pinned. Prefer ScopedValue for virtual threads—it's designed for this use case and does not pin.
What is the default carrier thread pool size and how do I change it?
The default is Runtime.availableProcessors(). You can change it with the system property jdk.virtualThreadScheduler.parallelism, e.g., -Djdk.virtualThreadScheduler.parallelism=64. However, increasing the pool is a workaround; the real fix is to eliminate pinning.
How do I debug a deadlock with virtual threads?
Use 'jcmd <pid> Thread.print -l' and look for 'Found one Java-level deadlock'. This works for virtual threads as well. For a more detailed analysis, use JFR events like jdk.JavaMonitorEnter and jdk.JavaMonitorWait. StructuredTaskScope with timeouts can help prevent deadlocks.