What this usually means
The multiprocessing module serializes (pickles) function and object arguments to send them to worker processes. If an object or function cannot be pickled, Python raises a PicklingError. Common culprits are top-level function definitions that are actually nested inside another function, lambda closures that capture unpicklable context, dynamically created classes or methods that aren't importable from __main__, and objects with unpicklable attributes (like open file handles). The error message tells you exactly what failed to pickle, but the root cause is often a design that relies on local state.
The first ten minutes — establish facts before touching code.
- 1Read the full traceback: the PicklingError or AttributeError will name the unpicklable object (e.g., '<locals>.inner').
- 2Check if the failing function or class is defined inside another function (nested). Move it to module level.
- 3If using lambda, replace it with a top-level function or use functools.partial with a picklable context.
- 4Verify all arguments passed to pool methods are pickleable: no open files, no sockets, no threading locks.
- 5Use multiprocessing.reduction.ForkingPickler.dumps(obj) in a test script to isolate the unpicklable attribute.
- 6If using a custom class, implement __reduce__ or __getstate__/__setstate__ methods for explicit serialization.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchThe full traceback (stderr) – the first line after PicklingError is the key.
- searchFunction definitions: are they top-level or nested inside another scope?
- searchLambda expressions inside pool.map or apply_async calls.
- searchCustom classes that inherit from unpicklable bases (e.g., a class defined inside a function).
- searchThe __main__ module – if you define a class interactively or in a script, workers can't import it.
- searchAny use of multiprocessing.Pool with methods like map, apply, starmap – inspect the callable and its arguments.
- searchThe pickle module's debug output: run 'python -c "import pickle; pickle.dumps(obj)"' to see the error.
Practical causes, not theory. These are the things you will actually find.
- warningNested function or lambda passed to pool.map – only module-level functions are picklable by default.
- warningDynamic class defined inside a function or inside __main__ without proper import path.
- warningObject attributes that are unpicklable (e.g., database connections, file handles, locks).
- warningUsing multiprocessing.Pool inside a class method that references self (self is unpicklable).
- warningCustom __reduce__ or __getstate__ not implemented or returning wrong types.
- warningThe function/class is defined in a conditional block or under __name__ == '__main__' guard, not importable by workers.
Concrete fix directions. Pick the one that matches your root cause.
- buildMove nested functions to module level: define the function outside the outer function.
- buildReplace lambdas with a top-level function or use functools.partial with a static argument.
- buildFor classes, define them at module level and ensure they are importable (not inside a conditional or function).
- buildUse pathos.multiprocessing (dill) instead of standard multiprocessing if you must pickle lambdas and closures.
- buildImplement __reduce__ or __getstate__/__setstate__ on custom classes to control serialization.
- buildRefactor code to pass simple data (tuples/dicts) instead of complex objects, and reconstruct on the other side.
A fix you cannot prove is a guess. Close the loop.
- verifiedRun the same code with multiprocessing replaced by dummy (multiprocessing.dummy) – if it works, the issue is pickling.
- verifiedUse the test script: pickle.dumps(your_callable) – if it raises, the fix hasn't worked.
- verifiedRun the pool operation on a small dataset and confirm no PicklingError.
- verifiedCheck that worker processes can import the module and access the function/class by name.
- verifiedIf using __reduce__, verify that the reconstructed object behaves correctly.
- verifiedMonitor stderr for any AttributeError or PicklingError after the fix.
Things that make this bug worse or harder to find.
- warningPutting the function definition inside the if __name__ == '__main__' block – workers can't import it.
- warningUsing multiprocessing on Windows without ensuring all module-level code is guarded by if __name__ == '__main__'.
- warningPassing a lambda that captures a large or unpicklable context (e.g., a database cursor).
- warningIgnoring the traceback and just adding try/except – masks the real problem.
- warningOverriding __reduce__ without understanding pickle protocol requirements – can make it worse.
- warningAssuming that because a function is top-level in an interactive shell, it will be importable by workers.
PicklingError in a Web Scraper Pool: The Lambda That Broke the Workers
Timeline
- 09:15Deploy new scraper to production; 64-core machine runs pool of 60 workers.
- 09:17First batch completes, then all workers crash with PicklingError: Can't pickle local object 'scrape_page.<locals>.parse'.
- 09:20Check traceback: function 'parse' is defined inside 'scrape_page'.
- 09:25Hotfix: move 'parse' to module level, but still crashes – now AttributeError: Can't pickle local object 'scrape_page.<locals>.<lambda>'.
- 09:30Identify a lambda inside pool.map that captures a config dict with a file handle.
- 09:35Replace lambda with top-level function, remove file handle from config, pass it as a separate argument.
- 09:40Test with small batch – works. Run full production load – no errors.
- 09:50Write unit test that uses pickle.dumps on all callables used in pool.
We had been running a web scraper for months without issues. The code used a nested function inside a loop to parse HTML, and it worked fine in development because we used ThreadPool. When we switched to multiprocessing.Pool for production, workers started crashing immediately after the first few tasks. The error was PicklingError with a reference to a local object. I quickly found the nested function and moved it to module level, but then a new error appeared: a lambda used in pool.map couldn't be pickled either.
The lambda was capturing a config dictionary that contained an open file handle (a logging stream). That was the real culprit. Even after moving the nested function, the lambda was still capturing an unpicklable object. I replaced the lambda with a top-level function and passed the config as a separate argument, stripping the file handle. I also added a test that calls pickle.dumps on every callable passed to the pool.
The lesson: multiprocessing pickling errors are often layered. Fixing one reveals another. The safest approach is to ensure all callables and their captured contexts are pickleable from the start. We now enforce a rule: no lambdas or nested functions in pool calls, and all arguments must be simple data structures.
Root cause
A lambda inside pool.map captured a config dict with an unpicklable file handle, and a nested function inside the main worker function was also unpicklable.
The fix
Moved the nested function to module level; replaced lambda with a top-level function; removed file handles from the config dict; added pickle serialization tests.
The lesson
Always verify pickleability of any callable or argument passed to multiprocessing.Pool. Use pickle.dumps in tests to catch errors early. Prefer module-level functions and plain data.
When you call pool.map(func, iterable), the multiprocessing module serializes both func and the iterable items into bytes using pickle. These bytes are sent via a pipe to worker processes, where they are deserialized and executed. If pickle cannot serialize an object, it raises PicklingError.
This design means that func must be importable by the worker: its fully qualified name (module + function name) must resolve when the worker imports the module. That's why nested functions and lambdas fail: they don't have a qualified name that can be imported. Similarly, objects with unpicklable attributes (file handles, sockets, locks) break serialization.
On Windows, multiprocessing imports the main module in each worker process. If your function or class is defined inside the if __name__ == '__main__': block, it won't be importable by workers because that block doesn't execute during import. The fix is to define all functions and classes at module level, outside any conditional.
Even on Linux, if you define a class interactively or inside a function, the worker may not find it. The rule: all pickled objects must be importable from the top level of a module. Use a separate module for worker code if needed.
Python's pickle module supports several protocols (0-5). By default, multiprocessing uses protocol 2 or higher. Custom classes can implement __reduce__ or __getstate__/__setstate__ to control serialization. __reduce__ should return a tuple (callable, args) that can reconstruct the object. However, the callable must itself be picklable (module-level).
A common mistake is to return a lambda or a nested function from __reduce__. That fails because the returned callable also needs to be pickled. Instead, use a module-level function or a class method. If you have a complex object, consider serializing only its data (e.g., a dict) and reconstructing it on the other side.
The 'dill' library extends pickle to serialize lambdas, nested functions, and closures. The 'pathos.multiprocessing' library provides a drop-in replacement for multiprocessing.Pool that uses dill internally. This can save you from refactoring, but it's slower and not recommended for production without thorough testing.
If you're prototyping or in a controlled environment, dill is a quick fix. But for production systems, it's better to refactor to pickleable code because it forces cleaner architecture and avoids subtle bugs with serialized closures that capture unintended state.
1. Read the full traceback: locate the object that failed to pickle. It's usually named in the error message (e.g., '<lambda>', '<locals>.inner').
2. Identify if the object is defined inside a function or a conditional. If so, move it to module level.
3. Check if the object captures any unpicklable attributes (file handles, locks, connections). Strip them or implement __reduce__.
4. Use pickle.dumps(obj) in a standalone script to reproduce the error. This isolates the problem from multiprocessing.
5. If using a class, ensure it's defined at module level and not inside __main__. Add __reduce__ if needed.
6. Test with multiprocessing.dummy.Pool (thread pool) to confirm the logic works; then switch back to multiprocessing.Pool after fixing pickling.
Frequently asked questions
Why does my code work with ThreadPool but fails with Pool?
ThreadPool uses threads within the same process, so no pickling is needed. Pool uses separate processes that require pickling to pass data. If your code relies on nested functions, lambdas, or objects with unpicklable attributes, it will fail with Pool but work with ThreadPool.
Can I pickle a lambda by using __reduce__?
No, because __reduce__ must return a callable that is itself picklable. A lambda is still a local object. The only reliable way to pickle a lambda is to use a library like dill, or to rewrite it as a module-level function.
What does 'Can't pickle <class '__main__.MyClass'>' mean?
It means the class MyClass was defined in the __main__ module. When a worker process imports __main__, it runs the entire script again, which can cause problems if the class definition is inside a conditional block. Fix: define the class in a separate module and import it.
How do I pass a database connection to a worker process?
Database connections are not pickleable. Instead, open a new connection inside each worker function (or use a connection pool that is initialized per worker). Pass connection parameters (host, port, credentials) as plain data.
Is there a way to globally enable pickling for all objects?
No, pickle requires explicit serialization support. You can override pickle behavior by registering reducers with copyreg.pickle, but this is advanced and can lead to security risks. It's better to refactor your code to be pickleable by design.