`!pickle` --- Python object serialization

Source code: Lib/pickle.py

---

The !pickle module implements binary protocols for serializing and de-serializing a Python object structure. "Pickling" is the process whereby a Python object hierarchy is converted into a byte stream, and "unpickling" is the inverse operation, whereby a byte stream (from a binary file or bytes-like object) is converted back into an object hierarchy. Pickling (and unpickling) is alternatively known as "serialization", "marshalling,"[^1] or "flattening"; however, to avoid confusion, the terms used here are "pickling" and "unpickling".

[!WARNING] The pickle module is not secure. Only unpickle data you trust.

It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.

Consider signing data with hmac if you need to ensure that it has not been tampered with.

Safer serialization formats such as json may be more appropriate if you are processing untrusted data. See comparison-with-json.

Relationship to other Python modules

Comparison with marshal

Python has a more primitive serialization module called marshal, but in general !pickle should always be the preferred way to serialize Python objects. marshal exists primarily to support Python's .pyc files.

The !pickle module differs from marshal in several significant ways:

• marshal cannot be used to serialize user-defined classes and their instances. !pickle can save and restore class instances transparently, however the class definition must be importable and live in the same module as when the object was pickled.
• The marshal serialization format is not guaranteed to be portable across Python versions. Because its primary job in life is to support .pyc files, the Python implementers reserve the right to change the serialization format in non-backwards compatible ways should the need arise. The !pickle serialization format is guaranteed to be backwards compatible across Python releases provided a compatible pickle protocol is chosen and pickling and unpickling code deals with Python 2 to Python 3 type differences if your data is crossing that unique breaking change language boundary.

Comparison with json

There are fundamental differences between the pickle protocols and JSON (JavaScript Object Notation):

• JSON is a text serialization format (it outputs unicode text, although most of the time it is then encoded to utf-8), while pickle is a binary serialization format;
• JSON is human-readable, while pickle is not;
• JSON is interoperable and widely used outside of the Python ecosystem, while pickle is Python-specific;
• JSON, by default, can only represent a subset of the Python built-in types, and no custom classes; pickle can represent an extremely large number of Python types (many of them automatically, by clever usage of Python's introspection facilities; complex cases can be tackled by implementing specific object APIs <pickle-inst>);
• Unlike pickle, deserializing untrusted JSON does not in itself create an arbitrary code execution vulnerability.

Data stream format

The data format used by !pickle is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as JSON (which can't represent pointer sharing); however it means that non-Python programs may not be able to reconstruct pickled Python objects.

By default, the !pickle data format uses a relatively compact binary representation. If you need optimal size characteristics, you can efficiently compress <archiving> pickled data.

The module pickletools contains tools for analyzing data streams generated by !pickle. pickletools source code has extensive comments about opcodes used by pickle protocols.

There are currently 6 different protocols which can be used for pickling. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.

• Protocol version 0 is the original "human-readable" protocol and is backwards compatible with earlier versions of Python.
• Protocol version 1 is an old binary format which is also compatible with earlier versions of Python.
• Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of new-style classes <new-style class>. Refer to 307 for information about improvements brought by protocol 2.
• Protocol version 3 was added in Python 3.0. It has explicit support for bytes objects and cannot be unpickled by Python 2.x. This was the default protocol in Python 3.0--3.7.
• Protocol version 4 was added in Python 3.4. It adds support for very large objects, pickling more kinds of objects, and some data format optimizations. This was the default protocol in Python 3.8--3.13. Refer to 3154 for information about improvements brought by protocol 4.
• Protocol version 5 was added in Python 3.8. It adds support for out-of-band data and speedup for in-band data. It is the default protocol starting with Python 3.14. Refer to 574 for information about improvements brought by protocol 5.

[!NOTE] Serialization is a more primitive notion than persistence; although !pickle reads and writes file objects, it does not handle the issue of naming persistent objects, nor the (even more complicated) issue of concurrent access to persistent objects. The !pickle module can transform a complex object into a byte stream and it can transform the byte stream into an object with the same internal structure. Perhaps the most obvious thing to do with these byte streams is to write them onto a file, but it is also conceivable to send them across a network or store them in a database. The shelve module provides a simple interface to pickle and unpickle objects on DBM-style database files.

Module Interface

To serialize an object hierarchy, you simply call the dumps function. Similarly, to de-serialize a data stream, you call the loads function. However, if you want more control over serialization and de-serialization, you can create a Pickler or an Unpickler object, respectively.

The !pickle module provides the following constants:

The !pickle module provides the following functions to make the pickling process more convenient:

The !pickle module defines three exceptions:

The !pickle module exports three classes, Pickler, Unpickler and PickleBuffer:

What can be pickled and unpickled?

The following types can be pickled:

• built-in constants (None, True, False, Ellipsis, and NotImplemented);
• integers, floating-point numbers, complex numbers;
• strings, bytes, bytearrays;
• tuples, lists, sets, and dictionaries containing only picklable objects;
• functions (built-in and user-defined) accessible from the top level of a module (using def, not lambda);
• classes accessible from the top level of a module;
• instances of such classes for which the result of calling ~object.__getstate__ is picklable (see section pickle-inst for details);
• weak references and the weak containers from the weakref module, provided that the weakly referenced objects are strongly referenced elsewhere in the same pickle (see weakref.ref).

Attempts to pickle unpicklable objects will raise the PicklingError exception; when this happens, an unspecified number of bytes may have already been written to the underlying file. Trying to pickle a highly recursive data structure may exceed the maximum recursion depth, a RecursionError will be raised in this case. You can carefully raise this limit with sys.setrecursionlimit.

Note that functions (built-in and user-defined) are pickled by fully qualified name, not by value.[^2] This means that only the function name is pickled, along with the name of the containing module and classes. Neither the function's code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.[^3]

Similarly, classes are pickled by fully qualified name, so the same restrictions in the unpickling environment apply. Note that none of the class's code or data is pickled, so in the following example the class attribute attr is not restored in the unpickling environment:

class Foo:
    attr = 'A class attribute'

picklestring = pickle.dumps(Foo)

These restrictions are why picklable functions and classes must be defined at the top level of a module.

Similarly, when class instances are pickled, their class's code and data are not pickled along with them. Only the instance data are pickled. This is done on purpose, so you can fix bugs in a class or add methods to the class and still load objects that were created with an earlier version of the class. If you plan to have long-lived objects that will see many versions of a class, it may be worthwhile to put a version number in the objects so that suitable conversions can be made by the class's ~object.__setstate__ method.

Pickling Class Instances

In this section, we describe the general mechanisms available to you to define, customize, and control how class instances are pickled and unpickled.

In most cases, no additional code is needed to make instances picklable. By default, pickle will retrieve the class and the attributes of an instance via introspection. When a class instance is unpickled, its ~object.__init__ method is usually not invoked. The default behaviour first creates an uninitialized instance and then restores the saved attributes. The following code shows an implementation of this behaviour:

def save(obj):
    return (obj.__class__, obj.__dict__)

def restore(cls, attributes):
    obj = cls.__new__(cls)
    obj.__dict__.update(attributes)
    return obj

Classes can alter the default behaviour by providing one or several special methods:

Refer to the section pickle-state for more information about how to use the methods ~object.__getstate__ and ~object.__setstate__.

[!NOTE] At unpickling time, some methods like ~object.__getattr__, ~object.__getattribute__, or ~object.__setattr__ may be called upon the instance. In case those methods rely on some internal invariant being true, the type should implement ~object.__new__ to establish such an invariant, as ~object.__init__ is not called when unpickling an instance.

As we shall see, pickle does not use directly the methods described above. In fact, these methods are part of the copy protocol which implements the ~object.__reduce__ special method. The copy protocol provides a unified interface for retrieving the data necessary for pickling and copying objects.[^4]

Although powerful, implementing ~object.__reduce__ directly in your classes is error prone. For this reason, class designers should use the high-level interface (i.e., ~object.__getnewargs_ex__, ~object.__getstate__ and ~object.__setstate__) whenever possible. We will show, however, cases where using !__reduce__ is the only option or leads to more efficient pickling or both.

Persistence of External Objects

For the benefit of object persistence, the !pickle module supports the notion of a reference to an object outside the pickled data stream. Such objects are referenced by a persistent ID, which should be either a string of alphanumeric characters (for protocol 0)[^5] or just an arbitrary object (for any newer protocol).

The resolution of such persistent IDs is not defined by the !pickle module; it will delegate this resolution to the user-defined methods on the pickler and unpickler, ~Pickler.persistent_id and ~Unpickler.persistent_load respectively.

To pickle objects that have an external persistent ID, the pickler must have a custom ~Pickler.persistent_id method that takes an object as an argument and returns either None or the persistent ID for that object. When None is returned, the pickler simply pickles the object as normal. When a persistent ID string is returned, the pickler will pickle that object, along with a marker so that the unpickler will recognize it as a persistent ID.

To unpickle external objects, the unpickler must have a custom ~Unpickler.persistent_load method that takes a persistent ID object and returns the referenced object.

Here is a comprehensive example presenting how persistent ID can be used to pickle external objects by reference.

Dispatch Tables

If one wants to customize pickling of some classes without disturbing any other code which depends on pickling, then one can create a pickler with a private dispatch table.

The global dispatch table managed by the copyreg module is available as !copyreg.dispatch_table. Therefore, one may choose to use a modified copy of !copyreg.dispatch_table as a private dispatch table.

For example :

f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass

creates an instance of pickle.Pickler with a private dispatch table which handles the SomeClass class specially. Alternatively, the code :

class MyPickler(pickle.Pickler):
    dispatch_table = copyreg.dispatch_table.copy()
    dispatch_table[SomeClass] = reduce_SomeClass
f = io.BytesIO()
p = MyPickler(f)

does the same but all instances of MyPickler will by default share the private dispatch table. On the other hand, the code :

copyreg.pickle(SomeClass, reduce_SomeClass)
f = io.BytesIO()
p = pickle.Pickler(f)

modifies the global dispatch table shared by all users of the copyreg module.

Handling Stateful Objects

Here's an example that shows how to modify pickling behavior for a class. The !TextReader class below opens a text file, and returns the line number and line contents each time its !readline method is called. If a !TextReader instance is pickled, all attributes except the file object member are saved. When the instance is unpickled, the file is reopened, and reading resumes from the last location. The !__setstate__ and !__getstate__ methods are used to implement this behavior. :

class TextReader:
    """Print and number lines in a text file."""

    def __init__(self, filename):
        self.filename = filename
        self.file = open(filename)
        self.lineno = 0

    def readline(self):
        self.lineno += 1
        line = self.file.readline()
        if not line:
            return None
        if line.endswith('\n'):
            line = line[:-1]
        return "%i: %s" % (self.lineno, line)

    def __getstate__(self):
        # Copy the object's state from self.__dict__ which contains
        # all our instance attributes. Always use the dict.copy()
        # method to avoid modifying the original state.
        state = self.__dict__.copy()
        # Remove the unpicklable entries.
        del state['file']
        return state

    def __setstate__(self, state):
        # Restore instance attributes (i.e., filename and lineno).
        self.__dict__.update(state)
        # Restore the previously opened file's state. To do so, we need to
        # reopen it and read from it until the line count is restored.
        file = open(self.filename)
        for _ in range(self.lineno):
            file.readline()
        # Finally, save the file.
        self.file = file

A sample usage might be something like this:

>>> reader = TextReader("hello.txt")
>>> reader.readline()
'1: Hello world!'
>>> reader.readline()
'2: I am line number two.'
>>> new_reader = pickle.loads(pickle.dumps(reader))
>>> new_reader.readline()
'3: Goodbye!'

Custom Reduction for Types, Functions, and Other Objects

Sometimes, ~Pickler.dispatch_table may not be flexible enough. In particular we may want to customize pickling based on another criterion than the object's type, or we may want to customize the pickling of functions and classes.

For those cases, it is possible to subclass from the Pickler class and implement a ~Pickler.reducer_override method. This method can return an arbitrary reduction tuple (see ~object.__reduce__). It can alternatively return NotImplemented to fallback to the traditional behavior.

If both the ~Pickler.dispatch_table and ~Pickler.reducer_override are defined, then ~Pickler.reducer_override method takes priority.

[!NOTE] For performance reasons, ~Pickler.reducer_override may not be called for the following objects: None, True, False, and exact instances of int, float, bytes, str, dict, set, frozenset, list and tuple.

Here is a simple example where we allow pickling and reconstructing a given class:

import io
import pickle

class MyClass:
    my_attribute = 1

class MyPickler(pickle.Pickler):
    def reducer_override(self, obj):
        """Custom reducer for MyClass."""
        if getattr(obj, "__name__", None) == "MyClass":
            return type, (obj.__name__, obj.__bases__,
                          {'my_attribute': obj.my_attribute})
        else:
            # For any other object, fallback to usual reduction
            return NotImplemented

f = io.BytesIO()
p = MyPickler(f)
p.dump(MyClass)

del MyClass

unpickled_class = pickle.loads(f.getvalue())

assert isinstance(unpickled_class, type)
assert unpickled_class.__name__ == "MyClass"
assert unpickled_class.my_attribute == 1

Out-of-band Buffers

In some contexts, the !pickle module is used to transfer massive amounts of data. Therefore, it can be important to minimize the number of memory copies, to preserve performance and resource consumption. However, normal operation of the !pickle module, as it transforms a graph-like structure of objects into a sequential stream of bytes, intrinsically involves copying data to and from the pickle stream.

This constraint can be eschewed if both the provider (the implementation of the object types to be transferred) and the consumer (the implementation of the communications system) support the out-of-band transfer facilities provided by pickle protocol 5 and higher.

Provider API

The large data objects to be pickled must implement a ~object.__reduce_ex__ method specialized for protocol 5 and higher, which returns a PickleBuffer instance (instead of e.g. a bytes object) for any large data.

A PickleBuffer object signals that the underlying buffer is eligible for out-of-band data transfer. Those objects remain compatible with normal usage of the !pickle module. However, consumers can also opt-in to tell !pickle that they will handle those buffers by themselves.

Consumer API

A communications system can enable custom handling of the PickleBuffer objects generated when serializing an object graph.

On the sending side, it needs to pass a buffer_callback argument to Pickler (or to the dump or dumps function), which will be called with each PickleBuffer generated while pickling the object graph. Buffers accumulated by the buffer_callback will not see their data copied into the pickle stream, only a cheap marker will be inserted.

On the receiving side, it needs to pass a buffers argument to Unpickler (or to the load or loads function), which is an iterable of the buffers which were passed to buffer_callback. That iterable should produce buffers in the same order as they were passed to buffer_callback. Those buffers will provide the data expected by the reconstructors of the objects whose pickling produced the original PickleBuffer objects.

Between the sending side and the receiving side, the communications system is free to implement its own transfer mechanism for out-of-band buffers. Potential optimizations include the use of shared memory or datatype-dependent compression.

Example

Here is a trivial example where we implement a bytearray subclass able to participate in out-of-band buffer pickling:

class ZeroCopyByteArray(bytearray):

    def __reduce_ex__(self, protocol):
        if protocol >= 5:
            return type(self)._reconstruct, (PickleBuffer(self),), None
        else:
            # PickleBuffer is forbidden with pickle protocols <= 4.
            return type(self)._reconstruct, (bytearray(self),)

    @classmethod
    def _reconstruct(cls, obj):
        with memoryview(obj) as m:
            # Get a handle over the original buffer object
            obj = m.obj
            if type(obj) is cls:
                # Original buffer object is a ZeroCopyByteArray, return it
                # as-is.
                return obj
            else:
                return cls(obj)

The reconstructor (the _reconstruct class method) returns the buffer's providing object if it has the right type. This is an easy way to simulate zero-copy behaviour on this toy example.

On the consumer side, we can pickle those objects the usual way, which when unserialized will give us a copy of the original object:

b = ZeroCopyByteArray(b"abc")
data = pickle.dumps(b, protocol=5)
new_b = pickle.loads(data)
print(b == new_b)  # True
print(b is new_b)  # False: a copy was made

But if we pass a buffer_callback and then give back the accumulated buffers when unserializing, we are able to get back the original object:

b = ZeroCopyByteArray(b"abc")
buffers = []
data = pickle.dumps(b, protocol=5, buffer_callback=buffers.append)
new_b = pickle.loads(data, buffers=buffers)
print(b == new_b)  # True
print(b is new_b)  # True: no copy was made

This example is limited by the fact that bytearray allocates its own memory: you cannot create a bytearray instance that is backed by another object's memory. However, third-party datatypes such as NumPy arrays do not have this limitation, and allow use of zero-copy pickling (or making as few copies as possible) when transferring between distinct processes or systems.

Restricting Globals

By default, unpickling will import any class or function that it finds in the pickle data. For many applications, this behaviour is unacceptable as it permits the unpickler to import and invoke arbitrary code. Just consider what this hand-crafted pickle data stream does when loaded:

>>> import pickle
>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
hello world
0

In this example, the unpickler imports the os.system function and then apply the string argument "echo hello world". Although this example is inoffensive, it is not difficult to imagine one that could damage your system.

For this reason, you may want to control what gets unpickled by customizing Unpickler.find_class. Unlike its name suggests, Unpickler.find_class is called whenever a global (i.e., a class or a function) is requested. Thus it is possible to either completely forbid globals or restrict them to a safe subset.

Here is an example of an unpickler allowing only few safe classes from the builtins module to be loaded:

import builtins
import io
import pickle

safe_builtins = {
    'range',
    'complex',
    'set',
    'frozenset',
    'slice',
}

class RestrictedUnpickler(pickle.Unpickler):

    def find_class(self, module, name):
        # Only allow safe classes from builtins.
        if module == "builtins" and name in safe_builtins:
            return getattr(builtins, name)
        # Forbid everything else.
        raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
                                     (module, name))

def restricted_loads(s):
    """Helper function analogous to pickle.loads()."""
    return RestrictedUnpickler(io.BytesIO(s)).load()

A sample usage of our unpickler working as intended:

>>> restricted_loads(pickle.dumps([1, 2, range(15)]))
[1, 2, range(0, 15)]
>>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
Traceback (most recent call last):
  ...
pickle.UnpicklingError: global 'os.system' is forbidden
>>> restricted_loads(b'cbuiltins\neval\n'
...                  b'(S\'getattr(__import__("os"), "system")'
...                  b'("echo hello world")\'\ntR.')
Traceback (most recent call last):
  ...
pickle.UnpicklingError: global 'builtins.eval' is forbidden

As our examples shows, you have to be careful with what you allow to be unpickled. Therefore if security is a concern, you may want to consider alternatives such as the marshalling API in xmlrpc.client or third-party solutions.

Performance

Recent versions of the pickle protocol (from protocol 2 and upwards) feature efficient binary encodings for several common features and built-in types. Also, the !pickle module has a transparent optimizer written in C.

Examples

For the simplest code, use the dump and load functions. :

import pickle

# An arbitrary collection of objects supported by pickle.
data = {
    'a': [1, 2.0, 3+4j],
    'b': ("character string", b"byte string"),
    'c': {None, True, False}
}

with open('data.pickle', 'wb') as f:
    # Pickle the 'data' dictionary using the highest protocol available.
    pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)

The following example reads the resulting pickled data. :

import pickle

with open('data.pickle', 'rb') as f:
    # The protocol version used is detected automatically, so we do not
    # have to specify it.
    data = pickle.load(f)

Command-line interface

The !pickle module can be invoked as a script from the command line, it will display contents of the pickle files. However, when the pickle file that you want to examine comes from an untrusted source, -m pickletools is a safer option because it does not execute pickle bytecode, see pickletools CLI usage <pickletools-cli>.

python -m pickle pickle_file [pickle_file ...]

The following option is accepted:

Footnotes

[^1]: Don't confuse this with the marshal module

[^2]: This is why lambda functions cannot be pickled: all !lambda functions share the same name: <lambda>.

[^3]: The exception raised will likely be an ImportError or an AttributeError but it could be something else.

[^4]: The copy module uses this protocol for shallow and deep copying operations.

[^5]: The limitation on alphanumeric characters is due to the fact that persistent IDs in protocol 0 are delimited by the newline character. Therefore if any kind of newline characters occurs in persistent IDs, the resulting pickled data will become unreadable.