Whatsnew

What's New in Python 2.0

Author
A.M. Kuchling and Moshe Zadka

Introduction

A new release of Python, version 2.0, was released on October 16, 2000. This article covers the exciting new features in 2.0, highlights some other useful changes, and points out a few incompatible changes that may require rewriting code.

Python's development never completely stops between releases, and a steady flow of bug fixes and improvements are always being submitted. A host of minor fixes, a few optimizations, additional docstrings, and better error messages went into 2.0; to list them all would be impossible, but they're certainly significant. Consult the publicly available CVS logs if you want to see the full list. This progress is due to the five developers working for PythonLabs are now getting paid to spend their days fixing bugs, and also due to the improved communication resulting from moving to SourceForge.

What About Python 1.6?

Python 1.6 can be thought of as the Contractual Obligations Python release. After the core development team left CNRI in May 2000, CNRI requested that a 1.6 release be created, containing all the work on Python that had been performed at CNRI. Python 1.6 therefore represents the state of the CVS tree as of May 2000, with the most significant new feature being Unicode support. Development continued after May, of course, so the 1.6 tree received a few fixes to ensure that it's forward-compatible with Python 2.0. 1.6 is therefore part of Python's evolution, and not a side branch.

So, should you take much interest in Python 1.6? Probably not. The 1.6final and 2.0beta1 releases were made on the same day (September 5, 2000), the plan being to finalize Python 2.0 within a month or so. If you have applications to maintain, there seems little point in breaking things by moving to 1.6, fixing them, and then having another round of breakage within a month by moving to 2.0; you're better off just going straight to 2.0. Most of the really interesting features described in this document are only in 2.0, because a lot of work was done between May and September.

New Development Process

The most important change in Python 2.0 may not be to the code at all, but to how Python is developed: in May 2000 the Python developers began using the tools made available by SourceForge for storing source code, tracking bug reports, and managing the queue of patch submissions. To report bugs or submit patches for Python 2.0, use the bug tracking and patch manager tools available from Python's project page, located at https://sourceforge.net/projects/python/.

The most important of the services now hosted at SourceForge is the Python CVS tree, the version-controlled repository containing the source code for Python. Previously, there were roughly 7 or so people who had write access to the CVS tree, and all patches had to be inspected and checked in by one of the people on this short list. Obviously, this wasn't very scalable. By moving the CVS tree to SourceForge, it became possible to grant write access to more people; as of September 2000 there were 27 people able to check in changes, a fourfold increase. This makes possible large-scale changes that wouldn't be attempted if they'd have to be filtered through the small group of core developers. For example, one day Peter Schneider-Kamp took it into his head to drop K&R C compatibility and convert the C source for Python to ANSI C. After getting approval on the python-dev mailing list, he launched into a flurry of checkins that lasted about a week, other developers joined in to help, and the job was done. If there were only 5 people with write access, probably that task would have been viewed as "nice, but not worth the time and effort needed" and it would never have gotten done.

The shift to using SourceForge's services has resulted in a remarkable increase in the speed of development. Patches now get submitted, commented on, revised by people other than the original submitter, and bounced back and forth between people until the patch is deemed worth checking in. Bugs are tracked in one central location and can be assigned to a specific person for fixing, and we can count the number of open bugs to measure progress. This didn't come without a cost: developers now have more e-mail to deal with, more mailing lists to follow, and special tools had to be written for the new environment. For example, SourceForge sends default patch and bug notification e-mail messages that are completely unhelpful, so Ka-Ping Yee wrote an HTML screen-scraper that sends more useful messages.

The ease of adding code caused a few initial growing pains, such as code was checked in before it was ready or without getting clear agreement from the developer group. The approval process that has emerged is somewhat similar to that used by the Apache group. Developers can vote +1, +0, -0, or -1 on a patch; +1 and -1 denote acceptance or rejection, while +0 and -0 mean the developer is mostly indifferent to the change, though with a slight positive or negative slant. The most significant change from the Apache model is that the voting is essentially advisory, letting Guido van Rossum, who has Benevolent Dictator For Life status, know what the general opinion is. He can still ignore the result of a vote, and approve or reject a change even if the community disagrees with him.

Producing an actual patch is the last step in adding a new feature, and is usually easy compared to the earlier task of coming up with a good design. Discussions of new features can often explode into lengthy mailing list threads, making the discussion hard to follow, and no one can read every posting to python-dev. Therefore, a relatively formal process has been set up to write Python Enhancement Proposals (PEPs), modelled on the internet RFC process. PEPs are draft documents that describe a proposed new feature, and are continually revised until the community reaches a consensus, either accepting or rejecting the proposal. Quoting from the introduction to 1, "PEP Purpose and Guidelines":

PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature.

We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions.

Read the rest of 1 for the details of the PEP editorial process, style, and format. PEPs are kept in the Python CVS tree on SourceForge, though they're not part of the Python 2.0 distribution, and are also available in HTML form from https://peps.python.org/. As of September 2000, there are 25 PEPs, ranging from 201, "Lockstep Iteration", to PEP 225, "Elementwise/Objectwise Operators".

Unicode

The largest new feature in Python 2.0 is a new fundamental data type: Unicode strings. Unicode uses 16-bit numbers to represent characters instead of the 8-bit number used by ASCII, meaning that 65,536 distinct characters can be supported.

The final interface for Unicode support was arrived at through countless often-stormy discussions on the python-dev mailing list, and mostly implemented by Marc-André Lemburg, based on a Unicode string type implementation by Fredrik Lundh. A detailed explanation of the interface was written up as 100, "Python Unicode Integration". This article will simply cover the most significant points about the Unicode interfaces.

In Python source code, Unicode strings are written as u"string". Arbitrary Unicode characters can be written using a new escape sequence, \\u{HHHH}, where HHHH is a 4-digit hexadecimal number from 0000 to FFFF. The existing \\x{HH} escape sequence can also be used, and octal escapes can be used for characters up to U+01FF, which is represented by \777.

Unicode strings, just like regular strings, are an immutable sequence type. They can be indexed and sliced, but not modified in place. Unicode strings have an encode( [encoding] ) method that returns an 8-bit string in the desired encoding. Encodings are named by strings, such as 'ascii', 'utf-8', 'iso-8859-1', or whatever. A codec API is defined for implementing and registering new encodings that are then available throughout a Python program. If an encoding isn't specified, the default encoding is usually 7-bit ASCII, though it can be changed for your Python installation by calling the sys.setdefaultencoding(encoding) function in a customized version of site.py.

Combining 8-bit and Unicode strings always coerces to Unicode, using the default ASCII encoding; the result of 'a' + u'bc' is u'abc'.

New built-in functions have been added, and existing built-ins modified to support Unicode:

A new module, unicodedata, provides an interface to Unicode character properties. For example, unicodedata.category(u'A') returns the 2-character string 'Lu', the 'L' denoting it's a letter, and 'u' meaning that it's uppercase. unicodedata.bidirectional(u'\u0660') returns 'AN', meaning that U+0660 is an Arabic number.

The codecs module contains functions to look up existing encodings and register new ones. Unless you want to implement a new encoding, you'll most often use the codecs.lookup(encoding) function, which returns a 4-element tuple: (encode_func, decode_func, stream_reader, stream_writer).

For example, the following code writes a Unicode string into a file, encoding it as UTF-8:

import codecs

unistr = u'\u0660\u2000ab ...'

(UTF8_encode, UTF8_decode,
 UTF8_streamreader, UTF8_streamwriter) = codecs.lookup('UTF-8')

output = UTF8_streamwriter( open( '/tmp/output', 'wb') )
output.write( unistr )
output.close()

The following code would then read UTF-8 input from the file:

input = UTF8_streamreader( open( '/tmp/output', 'rb') )
print repr(input.read())
input.close()

Unicode-aware regular expressions are available through the re module, which has a new underlying implementation called SRE written by Fredrik Lundh of Secret Labs AB.

A -U command line option was added which causes the Python compiler to interpret all string literals as Unicode string literals. This is intended to be used in testing and future-proofing your Python code, since some future version of Python may drop support for 8-bit strings and provide only Unicode strings.

List Comprehensions

Lists are a workhorse data type in Python, and many programs manipulate a list at some point. Two common operations on lists are to loop over them, and either pick out the elements that meet a certain criterion, or apply some function to each element. For example, given a list of strings, you might want to pull out all the strings containing a given substring, or strip off trailing whitespace from each line.

The existing map and filter functions can be used for this purpose, but they require a function as one of their arguments. This is fine if there's an existing built-in function that can be passed directly, but if there isn't, you have to create a little function to do the required work, and Python's scoping rules make the result ugly if the little function needs additional information. Take the first example in the previous paragraph, finding all the strings in the list containing a given substring. You could write the following to do it:

# Given the list L, make a list of all strings
# containing the substring S.
sublist = filter( lambda s, substring=S:
                     string.find(s, substring) != -1,
                  L)

Because of Python's scoping rules, a default argument is used so that the anonymous function created by the lambda expression knows what substring is being searched for. List comprehensions make this cleaner:

sublist = [ s for s in L if string.find(s, S) != -1 ]

List comprehensions have the form:

[ expression for expr in sequence1
             for expr2 in sequence2 ...
             for exprN in sequenceN
             if condition ]

The !for...!in clauses contain the sequences to be iterated over. The sequences do not have to be the same length, because they are not iterated over in parallel, but from left to right; this is explained more clearly in the following paragraphs. The elements of the generated list will be the successive values of expression. The final !if clause is optional; if present, expression is only evaluated and added to the result if condition is true.

To make the semantics very clear, a list comprehension is equivalent to the following Python code:

for expr1 in sequence1:
    for expr2 in sequence2:
    ...
        for exprN in sequenceN:
             if (condition):
                  # Append the value of
                  # the expression to the
                  # resulting list.

This means that when there are multiple !for...!in clauses, the resulting list will be equal to the product of the lengths of all the sequences. If you have two lists of length 3, the output list is 9 elements long:

seq1 = 'abc'
seq2 = (1,2,3)
>>> [ (x,y) for x in seq1 for y in seq2]
[('a', 1), ('a', 2), ('a', 3), ('b', 1), ('b', 2), ('b', 3), ('c', 1),
('c', 2), ('c', 3)]

To avoid introducing an ambiguity into Python's grammar, if expression is creating a tuple, it must be surrounded with parentheses. The first list comprehension below is a syntax error, while the second one is correct:

# Syntax error
[ x,y for x in seq1 for y in seq2]
# Correct
[ (x,y) for x in seq1 for y in seq2]

The idea of list comprehensions originally comes from the functional programming language Haskell (https://www.haskell.org). Greg Ewing argued most effectively for adding them to Python and wrote the initial list comprehension patch, which was then discussed for a seemingly endless time on the python-dev mailing list and kept up-to-date by Skip Montanaro.

Augmented Assignment

Augmented assignment operators, another long-requested feature, have been added to Python 2.0. Augmented assignment operators include +=, -=, *=, and so forth. For example, the statement a += 2 increments the value of the variable a by 2, equivalent to the slightly lengthier a = a + 2.

The full list of supported assignment operators is +=, -=, *=, /=, %=, **=, &=, |=, ^=, >>=, and <<=. Python classes can override the augmented assignment operators by defining methods named !__iadd__, !__isub__, etc. For example, the following !Number class stores a number and supports using += to create a new instance with an incremented value.

class Number:
    def __init__(self, value):
        self.value = value
    def __iadd__(self, increment):
        return Number( self.value + increment)

n = Number(5)
n += 3
print n.value

The !__iadd__ special method is called with the value of the increment, and should return a new instance with an appropriately modified value; this return value is bound as the new value of the variable on the left-hand side.

Augmented assignment operators were first introduced in the C programming language, and most C-derived languages, such as awk, C++, Java, Perl, and PHP also support them. The augmented assignment patch was implemented by Thomas Wouters.

String Methods

Until now string-manipulation functionality was in the string module, which was usually a front-end for the !strop module written in C. The addition of Unicode posed a difficulty for the !strop module, because the functions would all need to be rewritten in order to accept either 8-bit or Unicode strings. For functions such as !string.replace, which takes 3 string arguments, that means eight possible permutations, and correspondingly complicated code.

Instead, Python 2.0 pushes the problem onto the string type, making string manipulation functionality available through methods on both 8-bit strings and Unicode strings. :

>>> 'andrew'.capitalize()
'Andrew'
>>> 'hostname'.replace('os', 'linux')
'hlinuxtname'
>>> 'moshe'.find('sh')
2

One thing that hasn't changed, a noteworthy April Fools' joke notwithstanding, is that Python strings are immutable. Thus, the string methods return new strings, and do not modify the string on which they operate.

The old string module is still around for backwards compatibility, but it mostly acts as a front-end to the new string methods.

Two methods which have no parallel in pre-2.0 versions, although they did exist in JPython for quite some time, are !startswith and !endswith. s.startswith(t) is equivalent to s[:len(t)] == t, while s.endswith(t) is equivalent to s[-len(t):] == t.

One other method which deserves special mention is !join. The !join method of a string receives one parameter, a sequence of strings, and is equivalent to the !string.join function from the old string module, with the arguments reversed. In other words, s.join(seq) is equivalent to the old string.join(seq, s).

Garbage Collection of Cycles

The C implementation of Python uses reference counting to implement garbage collection. Every Python object maintains a count of the number of references pointing to itself, and adjusts the count as references are created or destroyed. Once the reference count reaches zero, the object is no longer accessible, since you need to have a reference to an object to access it, and if the count is zero, no references exist any longer.

Reference counting has some pleasant properties: it's easy to understand and implement, and the resulting implementation is portable, fairly fast, and reacts well with other libraries that implement their own memory handling schemes. The major problem with reference counting is that it sometimes doesn't realise that objects are no longer accessible, resulting in a memory leak. This happens when there are cycles of references.

Consider the simplest possible cycle, a class instance which has a reference to itself:

instance = SomeClass()
instance.myself = instance

After the above two lines of code have been executed, the reference count of instance is 2; one reference is from the variable named 'instance', and the other is from the myself attribute of the instance.

If the next line of code is del instance, what happens? The reference count of instance is decreased by 1, so it has a reference count of 1; the reference in the myself attribute still exists. Yet the instance is no longer accessible through Python code, and it could be deleted. Several objects can participate in a cycle if they have references to each other, causing all of the objects to be leaked.

Python 2.0 fixes this problem by periodically executing a cycle detection algorithm which looks for inaccessible cycles and deletes the objects involved. A new gc module provides functions to perform a garbage collection, obtain debugging statistics, and tuning the collector's parameters.

Running the cycle detection algorithm takes some time, and therefore will result in some additional overhead. It is hoped that after we've gotten experience with the cycle collection from using 2.0, Python 2.1 will be able to minimize the overhead with careful tuning. It's not yet obvious how much performance is lost, because benchmarking this is tricky and depends crucially on how often the program creates and destroys objects. The detection of cycles can be disabled when Python is compiled, if you can't afford even a tiny speed penalty or suspect that the cycle collection is buggy, by specifying the !--without-cycle-gc switch when running the configure script.

Several people tackled this problem and contributed to a solution. An early implementation of the cycle detection approach was written by Toby Kelsey. The current algorithm was suggested by Eric Tiedemann during a visit to CNRI, and Guido van Rossum and Neil Schemenauer wrote two different implementations, which were later integrated by Neil. Lots of other people offered suggestions along the way; the March 2000 archives of the python-dev mailing list contain most of the relevant discussion, especially in the threads titled "Reference cycle collection for Python" and "Finalization again".

Other Core Changes

Various minor changes have been made to Python's syntax and built-in functions. None of the changes are very far-reaching, but they're handy conveniences.

Minor Language Changes

A new syntax makes it more convenient to call a given function with a tuple of arguments and/or a dictionary of keyword arguments. In Python 1.5 and earlier, you'd use the !apply built-in function: apply(f, args, kw) calls the function !f with the argument tuple args and the keyword arguments in the dictionary kw. !apply is the same in 2.0, but thanks to a patch from Greg Ewing, f(*args, **kw) is a shorter and clearer way to achieve the same effect. This syntax is symmetrical with the syntax for defining functions:

def f(*args, **kw):
    # args is a tuple of positional args,
    # kw is a dictionary of keyword args
    ...

The print statement can now have its output directed to a file-like object by following the print with >> file, similar to the redirection operator in Unix shells. Previously you'd either have to use the !write method of the file-like object, which lacks the convenience and simplicity of print, or you could assign a new value to sys.stdout and then restore the old value. For sending output to standard error, it's much easier to write this:

print >> sys.stderr, "Warning: action field not supplied"

Modules can now be renamed on importing them, using the syntax import module as name or from module import name as othername. The patch was submitted by Thomas Wouters.

A new format style is available when using the % operator; '%r' will insert the repr of its argument. This was also added from symmetry considerations, this time for symmetry with the existing '%s' format style, which inserts the str of its argument. For example, '%r %s' % ('abc', 'abc') returns a string containing 'abc' abc.

Previously there was no way to implement a class that overrode Python's built-in in operator and implemented a custom version. obj in seq returns true if obj is present in the sequence seq; Python computes this by simply trying every index of the sequence until either obj is found or an IndexError is encountered. Moshe Zadka contributed a patch which adds a !__contains__ magic method for providing a custom implementation for !in. Additionally, new built-in objects written in C can define what !in means for them via a new slot in the sequence protocol.

Earlier versions of Python used a recursive algorithm for deleting objects. Deeply nested data structures could cause the interpreter to fill up the C stack and crash; Christian Tismer rewrote the deletion logic to fix this problem. On a related note, comparing recursive objects recursed infinitely and crashed; Jeremy Hylton rewrote the code to no longer crash, producing a useful result instead. For example, after this code:

a = []
b = []
a.append(a)
b.append(b)

The comparison a==b returns true, because the two recursive data structures are isomorphic. See the thread "trashcan and PR#7" in the April 2000 archives of the python-dev mailing list for the discussion leading up to this implementation, and some useful relevant links. Note that comparisons can now also raise exceptions. In earlier versions of Python, a comparison operation such as cmp(a,b) would always produce an answer, even if a user-defined !__cmp__ method encountered an error, since the resulting exception would simply be silently swallowed.

Work has been done on porting Python to 64-bit Windows on the Itanium processor, mostly by Trent Mick of ActiveState. (Confusingly, sys.platform is still 'win32' on Win64 because it seems that for ease of porting, MS Visual C++ treats code as 32 bit on Itanium.) PythonWin also supports Windows CE; see the Python CE page at https://pythonce.sourceforge.net/ for more information.

Another new platform is Darwin/MacOS X; initial support for it is in Python 2.0. Dynamic loading works, if you specify "configure --with-dyld --with-suffix=.x". Consult the README in the Python source distribution for more instructions.

An attempt has been made to alleviate one of Python's warts, the often-confusing NameError exception when code refers to a local variable before the variable has been assigned a value. For example, the following code raises an exception on the print statement in both 1.5.2 and 2.0; in 1.5.2 a NameError exception is raised, while 2.0 raises a new UnboundLocalError exception. UnboundLocalError is a subclass of NameError, so any existing code that expects NameError to be raised should still work. :

def f():
    print "i=",i
    i = i + 1
f()

Two new exceptions, TabError and IndentationError, have been introduced. They're both subclasses of SyntaxError, and are raised when Python code is found to be improperly indented.

Changes to Built-in Functions

A new built-in, zip(seq1, seq2, ...), has been added. zip returns a list of tuples where each tuple contains the i-th element from each of the argument sequences. The difference between zip and map(None, seq1, seq2) is that map pads the sequences with None if the sequences aren't all of the same length, while zip truncates the returned list to the length of the shortest argument sequence.

The int and !long functions now accept an optional "base" parameter when the first argument is a string. int('123', 10) returns 123, while int('123', 16) returns 291. int(123, 16) raises a TypeError exception with the message "can't convert non-string with explicit base".

A new variable holding more detailed version information has been added to the sys module. sys.version_info is a tuple (major, minor, micro, level, serial) For example, in a hypothetical 2.0.1beta1, sys.version_info would be (2, 0, 1, 'beta', 1). level is a string such as "alpha", "beta", or "final" for a final release.

Dictionaries have an odd new method, setdefault(key, default), which behaves similarly to the existing !get method. However, if the key is missing, !setdefault both returns the value of default as !get would do, and also inserts it into the dictionary as the value for key. Thus, the following lines of code:

if dict.has_key( key ): return dict[key]
else:
    dict[key] = []
    return dict[key]

can be reduced to a single return dict.setdefault(key, []) statement.

The interpreter sets a maximum recursion depth in order to catch runaway recursion before filling the C stack and causing a core dump or GPF.. Previously this limit was fixed when you compiled Python, but in 2.0 the maximum recursion depth can be read and modified using sys.getrecursionlimit and sys.setrecursionlimit. The default value is 1000, and a rough maximum value for a given platform can be found by running a new script, Misc/find_recursionlimit.py.

Porting to 2.0

New Python releases try hard to be compatible with previous releases, and the record has been pretty good. However, some changes are considered useful enough, usually because they fix initial design decisions that turned out to be actively mistaken, that breaking backward compatibility can't always be avoided. This section lists the changes in Python 2.0 that may cause old Python code to break.

The change which will probably break the most code is tightening up the arguments accepted by some methods. Some methods would take multiple arguments and treat them as a tuple, particularly various list methods such as ~list.append and ~list.insert. In earlier versions of Python, if L is a list, L.append( 1,2 ) appends the tuple (1,2) to the list. In Python 2.0 this causes a TypeError exception to be raised, with the message: 'append requires exactly 1 argument; 2 given'. The fix is to simply add an extra set of parentheses to pass both values as a tuple: L.append( (1,2) ).

The earlier versions of these methods were more forgiving because they used an old function in Python's C interface to parse their arguments; 2.0 modernizes them to use PyArg_ParseTuple, the current argument parsing function, which provides more helpful error messages and treats multi-argument calls as errors. If you absolutely must use 2.0 but can't fix your code, you can edit Objects/listobject.c and define the preprocessor symbol NO_STRICT_LIST_APPEND to preserve the old behaviour; this isn't recommended.

Some of the functions in the socket module are still forgiving in this way. For example, socket.connect( ('hostname', 25) ) is the correct form, passing a tuple representing an IP address, but socket.connect('hostname', 25) also works. socket.connect_ex <socket.socket.connect_ex> and socket.bind <socket.socket.bind> are similarly easy-going. 2.0alpha1 tightened these functions up, but because the documentation actually used the erroneous multiple argument form, many people wrote code which would break with the stricter checking. GvR backed out the changes in the face of public reaction, so for the socket module, the documentation was fixed and the multiple argument form is simply marked as deprecated; it will be tightened up again in a future Python version.

The \x escape in string literals now takes exactly 2 hex digits. Previously it would consume all the hex digits following the 'x' and take the lowest 8 bits of the result, so \x123456 was equivalent to \x56.

The AttributeError and NameError exceptions have a more friendly error message, whose text will be something like 'Spam' instance has no attribute 'eggs' or name 'eggs' is not defined. Previously the error message was just the missing attribute name eggs, and code written to take advantage of this fact will break in 2.0.

Some work has been done to make integers and long integers a bit more interchangeable. In 1.5.2, large-file support was added for Solaris, to allow reading files larger than 2 GiB; this made the !tell method of file objects return a long integer instead of a regular integer. Some code would subtract two file offsets and attempt to use the result to multiply a sequence or slice a string, but this raised a TypeError. In 2.0, long integers can be used to multiply or slice a sequence, and it'll behave as you'd intuitively expect it to; 3L * 'abc' produces 'abcabcabc', and (0,1,2,3)[2L:4L] produces (2,3). Long integers can also be used in various contexts where previously only integers were accepted, such as in the !seek method of file objects, and in the formats supported by the % operator (%d, %i, %x, etc.). For example, "%d" % 2L**64 will produce the string 18446744073709551616.

The subtlest long integer change of all is that the str of a long integer no longer has a trailing 'L' character, though repr still includes it. The 'L' annoyed many people who wanted to print long integers that looked just like regular integers, since they had to go out of their way to chop off the character. This is no longer a problem in 2.0, but code which does str(longval)[:-1] and assumes the 'L' is there, will now lose the final digit.

Taking the repr of a float now uses a different formatting precision than str. repr uses %.17g format string for C's !sprintf, while str uses %.12g as before. The effect is that repr may occasionally show more decimal places than str, for certain numbers. For example, the number 8.1 can't be represented exactly in binary, so repr(8.1) is '8.0999999999999996', while str(8.1) is '8.1'.

The -X command-line option, which turned all standard exceptions into strings instead of classes, has been removed; the standard exceptions will now always be classes. The !exceptions module containing the standard exceptions was translated from Python to a built-in C module, written by Barry Warsaw and Fredrik Lundh.

Extending/Embedding Changes

Some of the changes are under the covers, and will only be apparent to people writing C extension modules or embedding a Python interpreter in a larger application. If you aren't dealing with Python's C API, you can safely skip this section.

The version number of the Python C API was incremented, so C extensions compiled for 1.5.2 must be recompiled in order to work with 2.0. On Windows, it's not possible for Python 2.0 to import a third party extension built for Python 1.5.x due to how Windows DLLs work, so Python will raise an exception and the import will fail.

Users of Jim Fulton's ExtensionClass module will be pleased to find out that hooks have been added so that ExtensionClasses are now supported by isinstance and issubclass. This means you no longer have to remember to write code such as if type(obj) == myExtensionClass, but can use the more natural if isinstance(obj, myExtensionClass).

The Python/importdl.c file, which was a mass of #ifdefs to support dynamic loading on many different platforms, was cleaned up and reorganised by Greg Stein. importdl.c is now quite small, and platform-specific code has been moved into a bunch of Python/dynload_\*.c files. Another cleanup: there were also a number of my\*.h files in the Include/ directory that held various portability hacks; they've been merged into a single file, Include/pyport.h.

Vladimir Marangozov's long-awaited malloc restructuring was completed, to make it easy to have the Python interpreter use a custom allocator instead of C's standard malloc. For documentation, read the comments in Include/pymem.h and Include/objimpl.h. For the lengthy discussions during which the interface was hammered out, see the web archives of the 'patches' and 'python-dev' lists at python.org.

Recent versions of the GUSI development environment for MacOS support POSIX threads. Therefore, Python's POSIX threading support now works on the Macintosh. Threading support using the user-space GNU pth library was also contributed.

Threading support on Windows was enhanced, too. Windows supports thread locks that use kernel objects only in case of contention; in the common case when there's no contention, they use simpler functions which are an order of magnitude faster. A threaded version of Python 1.5.2 on NT is twice as slow as an unthreaded version; with the 2.0 changes, the difference is only 10%. These improvements were contributed by Yakov Markovitch.

Python 2.0's source now uses only ANSI C prototypes, so compiling Python now requires an ANSI C compiler, and can no longer be done using a compiler that only supports K&R C.

Previously the Python virtual machine used 16-bit numbers in its bytecode, limiting the size of source files. In particular, this affected the maximum size of literal lists and dictionaries in Python source; occasionally people who are generating Python code would run into this limit. A patch by Charles G. Waldman raises the limit from 2**16 to 2**32.

Three new convenience functions intended for adding constants to a module's dictionary at module initialization time were added: PyModule_AddObject, PyModule_AddIntConstant, and PyModule_AddStringConstant. Each of these functions takes a module object, a null-terminated C string containing the name to be added, and a third argument for the value to be assigned to the name. This third argument is, respectively, a Python object, a C long, or a C string.

A wrapper API was added for Unix-style signal handlers. PyOS_getsig gets a signal handler and PyOS_setsig will set a new handler.

Distutils: Making Modules Easy to Install

Before Python 2.0, installing modules was a tedious affair -- there was no way to figure out automatically where Python is installed, or what compiler options to use for extension modules. Software authors had to go through an arduous ritual of editing Makefiles and configuration files, which only really work on Unix and leave Windows and MacOS unsupported. Python users faced wildly differing installation instructions which varied between different extension packages, which made administering a Python installation something of a chore.

The SIG for distribution utilities, shepherded by Greg Ward, has created the Distutils, a system to make package installation much easier. They form the distutils package, a new part of Python's standard library. In the best case, installing a Python module from source will require the same steps: first you simply mean unpack the tarball or zip archive, and the run "python setup.py install". The platform will be automatically detected, the compiler will be recognized, C extension modules will be compiled, and the distribution installed into the proper directory. Optional command-line arguments provide more control over the installation process, the distutils package offers many places to override defaults -- separating the build from the install, building or installing in non-default directories, and more.

In order to use the Distutils, you need to write a setup.py script. For the simple case, when the software contains only .py files, a minimal setup.py can be just a few lines long:

from distutils.core import setup
setup (name = "foo", version = "1.0",
       py_modules = ["module1", "module2"])

The setup.py file isn't much more complicated if the software consists of a few packages:

from distutils.core import setup
setup (name = "foo", version = "1.0",
       packages = ["package", "package.subpackage"])

A C extension can be the most complicated case; here's an example taken from the PyXML package:

from distutils.core import setup, Extension

expat_extension = Extension('xml.parsers.pyexpat',
     define_macros = [('XML_NS', None)],
     include_dirs = [ 'extensions/expat/xmltok',
                      'extensions/expat/xmlparse' ],
     sources = [ 'extensions/pyexpat.c',
                 'extensions/expat/xmltok/xmltok.c',
                 'extensions/expat/xmltok/xmlrole.c', ]
       )
setup (name = "PyXML", version = "0.5.4",
       ext_modules =[ expat_extension ] )

The Distutils can also take care of creating source and binary distributions. The "sdist" command, run by "python setup.py sdist', builds a source distribution such as foo-1.0.tar.gz. Adding new commands isn't difficult, "bdist_rpm" and "bdist_wininst" commands have already been contributed to create an RPM distribution and a Windows installer for the software, respectively. Commands to create other distribution formats such as Debian packages and Solaris .pkg files are in various stages of development.

All this is documented in a new manual, Distributing Python Modules, that joins the basic set of Python documentation.

XML Modules

Python 1.5.2 included a simple XML parser in the form of the !xmllib module, contributed by Sjoerd Mullender. Since 1.5.2's release, two different interfaces for processing XML have become common: SAX2 (version 2 of the Simple API for XML) provides an event-driven interface with some similarities to !xmllib, and the DOM (Document Object Model) provides a tree-based interface, transforming an XML document into a tree of nodes that can be traversed and modified. Python 2.0 includes a SAX2 interface and a stripped-down DOM interface as part of the xml package. Here we will give a brief overview of these new interfaces; consult the Python documentation or the source code for complete details. The Python XML SIG is also working on improved documentation.

SAX2 Support

SAX defines an event-driven interface for parsing XML. To use SAX, you must write a SAX handler class. Handler classes inherit from various classes provided by SAX, and override various methods that will then be called by the XML parser. For example, the ~xml.sax.handler.ContentHandler.startElement and ~xml.sax.handler.ContentHandler.endElement methods are called for every starting and end tag encountered by the parser, the ~xml.sax.handler.ContentHandler.characters method is called for every chunk of character data, and so forth.

The advantage of the event-driven approach is that the whole document doesn't have to be resident in memory at any one time, which matters if you are processing really huge documents. However, writing the SAX handler class can get very complicated if you're trying to modify the document structure in some elaborate way.

For example, this little example program defines a handler that prints a message for every starting and ending tag, and then parses the file hamlet.xml using it:

from xml import sax

class SimpleHandler(sax.ContentHandler):
    def startElement(self, name, attrs):
        print 'Start of element:', name, attrs.keys()

    def endElement(self, name):
        print 'End of element:', name

# Create a parser object
parser = sax.make_parser()

# Tell it what handler to use
handler = SimpleHandler()
parser.setContentHandler( handler )

# Parse a file!
parser.parse( 'hamlet.xml' )

For more information, consult the Python documentation, or the XML HOWTO at https://pyxml.sourceforge.net/topics/howto/xml-howto.html.

DOM Support

The Document Object Model is a tree-based representation for an XML document. A top-level !Document instance is the root of the tree, and has a single child which is the top-level !Element instance. This !Element has children nodes representing character data and any sub-elements, which may have further children of their own, and so forth. Using the DOM you can traverse the resulting tree any way you like, access element and attribute values, insert and delete nodes, and convert the tree back into XML.

The DOM is useful for modifying XML documents, because you can create a DOM tree, modify it by adding new nodes or rearranging subtrees, and then produce a new XML document as output. You can also construct a DOM tree manually and convert it to XML, which can be a more flexible way of producing XML output than simply writing <tag1>...</tag1> to a file.

The DOM implementation included with Python lives in the xml.dom.minidom module. It's a lightweight implementation of the Level 1 DOM with support for XML namespaces. The !parse and !parseString convenience functions are provided for generating a DOM tree:

from xml.dom import minidom
doc = minidom.parse('hamlet.xml')

doc is a !Document instance. !Document, like all the other DOM classes such as !Element and Text, is a subclass of the !Node base class. All the nodes in a DOM tree therefore support certain common methods, such as !toxml which returns a string containing the XML representation of the node and its children. Each class also has special methods of its own; for example, !Element and !Document instances have a method to find all child elements with a given tag name. Continuing from the previous 2-line example:

perslist = doc.getElementsByTagName( 'PERSONA' )
print perslist[0].toxml()
print perslist[1].toxml()

For the Hamlet XML file, the above few lines output:

<PERSONA>CLAUDIUS, king of Denmark. </PERSONA>
<PERSONA>HAMLET, son to the late, and nephew to the present king.</PERSONA>

The root element of the document is available as doc.documentElement, and its children can be easily modified by deleting, adding, or removing nodes:

root = doc.documentElement

# Remove the first child
root.removeChild( root.childNodes[0] )

# Move the new first child to the end
root.appendChild( root.childNodes[0] )

# Insert the new first child (originally,
# the third child) before the 20th child.
root.insertBefore( root.childNodes[0], root.childNodes[20] )

Again, I will refer you to the Python documentation for a complete listing of the different !Node classes and their various methods.

Relationship to PyXML

The XML Special Interest Group has been working on XML-related Python code for a while. Its code distribution, called PyXML, is available from the SIG's web pages at https://www.python.org/community/sigs/current/xml-sig. The PyXML distribution also used the package name xml. If you've written programs that used PyXML, you're probably wondering about its compatibility with the 2.0 xml package.

The answer is that Python 2.0's xml package isn't compatible with PyXML, but can be made compatible by installing a recent version PyXML. Many applications can get by with the XML support that is included with Python 2.0, but more complicated applications will require that the full PyXML package will be installed. When installed, PyXML versions 0.6.0 or greater will replace the xml package shipped with Python, and will be a strict superset of the standard package, adding a bunch of additional features. Some of the additional features in PyXML include:

Module changes

Lots of improvements and bugfixes were made to Python's extensive standard library; some of the affected modules include readline, ConfigParser <configparser>, !cgi, calendar, posix, readline, !xmllib, !aifc, !chunk, wave, random, shelve, and !nntplib. Consult the CVS logs for the exact patch-by-patch details.

Brian Gallew contributed OpenSSL support for the socket module. OpenSSL is an implementation of the Secure Socket Layer, which encrypts the data being sent over a socket. When compiling Python, you can edit Modules/Setup to include SSL support, which adds an additional function to the socket module: socket.ssl(socket, keyfile, certfile), which takes a socket object and returns an SSL socket. The httplib <http> and urllib modules were also changed to support https:// URLs, though no one has implemented FTP or SMTP over SSL.

The httplib <http> module has been rewritten by Greg Stein to support HTTP/1.1.

Backward compatibility with the 1.5 version of !httplib is provided, though using HTTP/1.1 features such as pipelining will require rewriting code to use a different set of interfaces.

The !Tkinter module now supports Tcl/Tk version 8.1, 8.2, or 8.3, and support for the older 7.x versions has been dropped. The Tkinter module now supports displaying Unicode strings in Tk widgets. Also, Fredrik Lundh contributed an optimization which makes operations like create_line and create_polygon much faster, especially when using lots of coordinates.

The curses module has been greatly extended, starting from Oliver Andrich's enhanced version, to provide many additional functions from ncurses and SYSV curses, such as colour, alternative character set support, pads, and mouse support. This means the module is no longer compatible with operating systems that only have BSD curses, but there don't seem to be any currently maintained OSes that fall into this category.

As mentioned in the earlier discussion of 2.0's Unicode support, the underlying implementation of the regular expressions provided by the re module has been changed. SRE, a new regular expression engine written by Fredrik Lundh and partially funded by Hewlett Packard, supports matching against both 8-bit strings and Unicode strings.

New modules

A number of new modules were added. We'll simply list them with brief descriptions; consult the 2.0 documentation for the details of a particular module.

IDLE Improvements

IDLE is the official Python cross-platform IDE, written using Tkinter. Python 2.0 includes IDLE 0.6, which adds a number of new features and improvements. A partial list:

Deleted and Deprecated Modules

A few modules have been dropped because they're obsolete, or because there are now better ways to do the same thing. The !stdwin module is gone; it was for a platform-independent windowing toolkit that's no longer developed.

A number of modules have been moved to the lib-old subdirectory: !cmp, !cmpcache, !dircmp, !dump, !find, !grep, !packmail, !poly, !util, !whatsound, !zmod. If you have code which relies on a module that's been moved to lib-old, you can simply add that directory to sys.path to get them back, but you're encouraged to update any code that uses these modules.

Acknowledgements

The authors would like to thank the following people for offering suggestions on various drafts of this article: David Bolen, Mark Hammond, Gregg Hauser, Jeremy Hylton, Fredrik Lundh, Detlef Lannert, Aahz Maruch, Skip Montanaro, Vladimir Marangozov, Tobias Polzin, Guido van Rossum, Neil Schemenauer, and Russ Schmidt.

What's New in Python 2.1

Author
A.M. Kuchling

Introduction

This article explains the new features in Python 2.1. While there aren't as many changes in 2.1 as there were in Python 2.0, there are still some pleasant surprises in store. 2.1 is the first release to be steered through the use of Python Enhancement Proposals, or PEPs, so most of the sizable changes have accompanying PEPs that provide more complete documentation and a design rationale for the change. This article doesn't attempt to document the new features completely, but simply provides an overview of the new features for Python programmers. Refer to the Python 2.1 documentation, or to the specific PEP, for more details about any new feature that particularly interests you.

One recent goal of the Python development team has been to accelerate the pace of new releases, with a new release coming every 6 to 9 months. 2.1 is the first release to come out at this faster pace, with the first alpha appearing in January, 3 months after the final version of 2.0 was released.

The final release of Python 2.1 was made on April 17, 2001.

PEP 227: Nested Scopes

The largest change in Python 2.1 is to Python's scoping rules. In Python 2.0, at any given time there are at most three namespaces used to look up variable names: local, module-level, and the built-in namespace. This often surprised people because it didn't match their intuitive expectations. For example, a nested recursive function definition doesn't work:

def f():
    ...
    def g(value):
        ...
        return g(value-1) + 1
    ...

The function !g will always raise a NameError exception, because the binding of the name g isn't in either its local namespace or in the module-level namespace. This isn't much of a problem in practice (how often do you recursively define interior functions like this?), but this also made using the lambda expression clumsier, and this was a problem in practice. In code which uses lambda you can often find local variables being copied by passing them as the default values of arguments. :

def find(self, name):
    "Return list of any entries equal to 'name'"
    L = filter(lambda x, name=name: x == name,
               self.list_attribute)
    return L

The readability of Python code written in a strongly functional style suffers greatly as a result.

The most significant change to Python 2.1 is that static scoping has been added to the language to fix this problem. As a first effect, the name=name default argument is now unnecessary in the above example. Put simply, when a given variable name is not assigned a value within a function (by an assignment, or the def, class, or import statements), references to the variable will be looked up in the local namespace of the enclosing scope. A more detailed explanation of the rules, and a dissection of the implementation, can be found in the PEP.

This change may cause some compatibility problems for code where the same variable name is used both at the module level and as a local variable within a function that contains further function definitions. This seems rather unlikely though, since such code would have been pretty confusing to read in the first place.

One side effect of the change is that the from module import * and exec statements have been made illegal inside a function scope under certain conditions. The Python reference manual has said all along that from module import * is only legal at the top level of a module, but the CPython interpreter has never enforced this before. As part of the implementation of nested scopes, the compiler which turns Python source into bytecodes has to generate different code to access variables in a containing scope. from module import * and exec make it impossible for the compiler to figure this out, because they add names to the local namespace that are unknowable at compile time. Therefore, if a function contains function definitions or lambda expressions with free variables, the compiler will flag this by raising a SyntaxError exception.

To make the preceding explanation a bit clearer, here's an example:

x = 1
def f():
    # The next line is a syntax error
    exec 'x=2'
    def g():
        return x

Line 4 containing the exec statement is a syntax error, since exec would define a new local variable named x whose value should be accessed by !g.

This shouldn't be much of a limitation, since exec is rarely used in most Python code (and when it is used, it's often a sign of a poor design anyway).

Compatibility concerns have led to nested scopes being introduced gradually; in Python 2.1, they aren't enabled by default, but can be turned on within a module by using a future statement as described in 236. (See the following section for further discussion of 236.) In Python 2.2, nested scopes will become the default and there will be no way to turn them off, but users will have had all of 2.1's lifetime to fix any breakage resulting from their introduction.

227 - Statically Nested Scopes
Written and implemented by Jeremy Hylton.

PEP 236: __future__ Directives

The reaction to nested scopes was widespread concern about the dangers of breaking code with the 2.1 release, and it was strong enough to make the Pythoneers take a more conservative approach. This approach consists of introducing a convention for enabling optional functionality in release N that will become compulsory in release N+1.

The syntax uses a from...import statement using the reserved module name __future__. Nested scopes can be enabled by the following statement:

from __future__ import nested_scopes

While it looks like a normal import statement, it's not; there are strict rules on where such a future statement can be put. They can only be at the top of a module, and must precede any Python code or regular !import statements. This is because such statements can affect how the Python bytecode compiler parses code and generates bytecode, so they must precede any statement that will result in bytecodes being produced.

236 - Back to the __future__
Written by Tim Peters, and primarily implemented by Jeremy Hylton.

PEP 207: Rich Comparisons

In earlier versions, Python's support for implementing comparisons on user-defined classes and extension types was quite simple. Classes could implement a !__cmp__ method that was given two instances of a class, and could only return 0 if they were equal or +1 or -1 if they weren't; the method couldn't raise an exception or return anything other than a Boolean value. Users of Numeric Python often found this model too weak and restrictive, because in the number-crunching programs that numeric Python is used for, it would be more useful to be able to perform elementwise comparisons of two matrices, returning a matrix containing the results of a given comparison for each element. If the two matrices are of different sizes, then the compare has to be able to raise an exception to signal the error.

In Python 2.1, rich comparisons were added in order to support this need. Python classes can now individually overload each of the <, <=, >, >=, ==, and != operations. The new magic method names are:

Operation Method name
< ~object.__lt__
<= ~object.__le__
> ~object.__gt__
>= ~object.__ge__
== ~object.__eq__
!= ~object.__ne__

(The magic methods are named after the corresponding Fortran operators .LT.. .LE., &c. Numeric programmers are almost certainly quite familiar with these names and will find them easy to remember.)

Each of these magic methods is of the form method(self, other), where self will be the object on the left-hand side of the operator, while other will be the object on the right-hand side. For example, the expression A < B will cause A.__lt__(B) to be called.

Each of these magic methods can return anything at all: a Boolean, a matrix, a list, or any other Python object. Alternatively they can raise an exception if the comparison is impossible, inconsistent, or otherwise meaningless.

The built-in cmp(A,B) function can use the rich comparison machinery, and now accepts an optional argument specifying which comparison operation to use; this is given as one of the strings "<", "<=", ">", ">=", "==", or "!=". If called without the optional third argument, !cmp will only return -1, 0, or +1 as in previous versions of Python; otherwise it will call the appropriate method and can return any Python object.

There are also corresponding changes of interest to C programmers; there's a new slot tp_richcmp in type objects and an API for performing a given rich comparison. I won't cover the C API here, but will refer you to 207, or to 2.1's C API documentation, for the full list of related functions.

207 - Rich Comparisons
Written by Guido van Rossum, heavily based on earlier work by David Ascher, and implemented by Guido van Rossum.

PEP 230: Warning Framework

Over its 10 years of existence, Python has accumulated a certain number of obsolete modules and features along the way. It's difficult to know when a feature is safe to remove, since there's no way of knowing how much code uses it --- perhaps no programs depend on the feature, or perhaps many do. To enable removing old features in a more structured way, a warning framework was added. When the Python developers want to get rid of a feature, it will first trigger a warning in the next version of Python. The following Python version can then drop the feature, and users will have had a full release cycle to remove uses of the old feature.

Python 2.1 adds the warning framework to be used in this scheme. It adds a warnings module that provide functions to issue warnings, and to filter out warnings that you don't want to be displayed. Third-party modules can also use this framework to deprecate old features that they no longer wish to support.

For example, in Python 2.1 the !regex module is deprecated, so importing it causes a warning to be printed:

>>> import regex
__main__:1: DeprecationWarning: the regex module
         is deprecated; please use the re module
>>>

Warnings can be issued by calling the warnings.warn function:

warnings.warn("feature X no longer supported")

The first parameter is the warning message; an additional optional parameters can be used to specify a particular warning category.

Filters can be added to disable certain warnings; a regular expression pattern can be applied to the message or to the module name in order to suppress a warning. For example, you may have a program that uses the !regex module and not want to spare the time to convert it to use the re module right now. The warning can be suppressed by calling :

import warnings
warnings.filterwarnings(action = 'ignore',
                        message='.*regex module is deprecated',
                        category=DeprecationWarning,
                        module = '__main__')

This adds a filter that will apply only to warnings of the class DeprecationWarning triggered in the __main__ module, and applies a regular expression to only match the message about the !regex module being deprecated, and will cause such warnings to be ignored. Warnings can also be printed only once, printed every time the offending code is executed, or turned into exceptions that will cause the program to stop (unless the exceptions are caught in the usual way, of course).

Functions were also added to Python's C API for issuing warnings; refer to PEP 230 or to Python's API documentation for the details.

5 - Guidelines for Language Evolution
Written by Paul Prescod, to specify procedures to be followed when removing old features from Python. The policy described in this PEP hasn't been officially adopted, but the eventual policy probably won't be too different from Prescod's proposal.

230 - Warning Framework
Written and implemented by Guido van Rossum.

PEP 229: New Build System

When compiling Python, the user had to go in and edit the Modules/Setup file in order to enable various additional modules; the default set is relatively small and limited to modules that compile on most Unix platforms. This means that on Unix platforms with many more features, most notably Linux, Python installations often don't contain all useful modules they could.

Python 2.0 added the Distutils, a set of modules for distributing and installing extensions. In Python 2.1, the Distutils are used to compile much of the standard library of extension modules, autodetecting which ones are supported on the current machine. It's hoped that this will make Python installations easier and more featureful.

Instead of having to edit the Modules/Setup file in order to enable modules, a setup.py script in the top directory of the Python source distribution is run at build time, and attempts to discover which modules can be enabled by examining the modules and header files on the system. If a module is configured in Modules/Setup, the setup.py script won't attempt to compile that module and will defer to the Modules/Setup file's contents. This provides a way to specific any strange command-line flags or libraries that are required for a specific platform.

In another far-reaching change to the build mechanism, Neil Schemenauer restructured things so Python now uses a single makefile that isn't recursive, instead of makefiles in the top directory and in each of the Python/, Parser/, Objects/, and Modules/ subdirectories. This makes building Python faster and also makes hacking the Makefiles clearer and simpler.

229 - Using Distutils to Build Python
Written and implemented by A.M. Kuchling.

PEP 205: Weak References

Weak references, available through the weakref module, are a minor but useful new data type in the Python programmer's toolbox.

Storing a reference to an object (say, in a dictionary or a list) has the side effect of keeping that object alive forever. There are a few specific cases where this behaviour is undesirable, object caches being the most common one, and another being circular references in data structures such as trees.

For example, consider a memoizing function that caches the results of another function f(x) by storing the function's argument and its result in a dictionary:

_cache = {}
def memoize(x):
    if _cache.has_key(x):
        return _cache[x]

    retval = f(x)

    # Cache the returned object
    _cache[x] = retval

    return retval

This version works for simple things such as integers, but it has a side effect; the _cache dictionary holds a reference to the return values, so they'll never be deallocated until the Python process exits and cleans up. This isn't very noticeable for integers, but if !f returns an object, or a data structure that takes up a lot of memory, this can be a problem.

Weak references provide a way to implement a cache that won't keep objects alive beyond their time. If an object is only accessible through weak references, the object will be deallocated and the weak references will now indicate that the object it referred to no longer exists. A weak reference to an object obj is created by calling wr = weakref.ref(obj). The object being referred to is returned by calling the weak reference as if it were a function: wr(). It will return the referenced object, or None if the object no longer exists.

This makes it possible to write a !memoize function whose cache doesn't keep objects alive, by storing weak references in the cache. :

_cache = {}
def memoize(x):
    if _cache.has_key(x):
        obj = _cache[x]()
        # If weak reference object still exists,
        # return it
        if obj is not None: return obj

    retval = f(x)

    # Cache a weak reference
    _cache[x] = weakref.ref(retval)

    return retval

The weakref module also allows creating proxy objects which behave like weak references --- an object referenced only by proxy objects is deallocated -- but instead of requiring an explicit call to retrieve the object, the proxy transparently forwards all operations to the object as long as the object still exists. If the object is deallocated, attempting to use a proxy will cause a !weakref.ReferenceError exception to be raised. :

proxy = weakref.proxy(obj)
proxy.attr   # Equivalent to obj.attr
proxy.meth() # Equivalent to obj.meth()
del obj
proxy.attr   # raises weakref.ReferenceError

205 - Weak References
Written and implemented by Fred L. Drake, Jr.

PEP 232: Function Attributes

In Python 2.1, functions can now have arbitrary information attached to them. People were often using docstrings to hold information about functions and methods, because the ~function.__doc__ attribute was the only way of attaching any information to a function. For example, in the Zope web application server, functions are marked as safe for public access by having a docstring, and in John Aycock's SPARK parsing framework, docstrings hold parts of the BNF grammar to be parsed. This overloading is unfortunate, since docstrings are really intended to hold a function's documentation; for example, it means you can't properly document functions intended for private use in Zope.

Arbitrary attributes can now be set and retrieved on functions using the regular Python syntax:

def f(): pass

f.publish = 1
f.secure = 1
f.grammar = "A ::= B (C D)*"

The dictionary containing attributes can be accessed as the function's ~function.__dict__. Unlike the ~type.__dict__ attribute of class instances, in functions you can actually assign a new dictionary to ~function.__dict__, though the new value is restricted to a regular Python dictionary; you can't be tricky and set it to a !UserDict instance, or any other random object that behaves like a mapping.

232 - Function Attributes
Written and implemented by Barry Warsaw.

PEP 235: Importing Modules on Case-Insensitive Platforms

Some operating systems have filesystems that are case-insensitive, MacOS and Windows being the primary examples; on these systems, it's impossible to distinguish the filenames FILE.PY and file.py, even though they do store the file's name in its original case (they're case-preserving, too).

In Python 2.1, the import statement will work to simulate case-sensitivity on case-insensitive platforms. Python will now search for the first case-sensitive match by default, raising an ImportError if no such file is found, so import file will not import a module named FILE.PY. Case-insensitive matching can be requested by setting the PYTHONCASEOK environment variable before starting the Python interpreter.

PEP 217: Interactive Display Hook

When using the Python interpreter interactively, the output of commands is displayed using the built-in repr function. In Python 2.1, the variable sys.displayhook can be set to a callable object which will be called instead of repr. For example, you can set it to a special pretty-printing function:

>>> # Create a recursive data structure
... L = [1,2,3]
>>> L.append(L)
>>> L # Show Python's default output
[1, 2, 3, [...]]
>>> # Use pprint.pprint() as the display function
... import sys, pprint
>>> sys.displayhook = pprint.pprint
>>> L
[1, 2, 3,  <Recursion on list with id=135143996>]
>>>

217 - Display Hook for Interactive Use
Written and implemented by Moshe Zadka.

PEP 208: New Coercion Model

How numeric coercion is done at the C level was significantly modified. This will only affect the authors of C extensions to Python, allowing them more flexibility in writing extension types that support numeric operations.

Extension types can now set the type flag Py_TPFLAGS_CHECKTYPES in their PyTypeObject structure to indicate that they support the new coercion model. In such extension types, the numeric slot functions can no longer assume that they'll be passed two arguments of the same type; instead they may be passed two arguments of differing types, and can then perform their own internal coercion. If the slot function is passed a type it can't handle, it can indicate the failure by returning a reference to the Py_NotImplemented singleton value. The numeric functions of the other type will then be tried, and perhaps they can handle the operation; if the other type also returns Py_NotImplemented, then a TypeError will be raised. Numeric methods written in Python can also return Py_NotImplemented, causing the interpreter to act as if the method did not exist (perhaps raising a TypeError, perhaps trying another object's numeric methods).

208 - Reworking the Coercion Model
Written and implemented by Neil Schemenauer, heavily based upon earlier work by Marc-André Lemburg. Read this to understand the fine points of how numeric operations will now be processed at the C level.

PEP 241: Metadata in Python Packages

A common complaint from Python users is that there's no single catalog of all the Python modules in existence. T. Middleton's Vaults of Parnassus at www.vex.net/parnassus/ (retired in February 2009, available in the Internet Archive Wayback Machine) was the largest catalog of Python modules, but registering software at the Vaults is optional, and many people did not bother.

As a first small step toward fixing the problem, Python software packaged using the Distutils sdist command will include a file named PKG-INFO containing information about the package such as its name, version, and author (metadata, in cataloguing terminology). 241 contains the full list of fields that can be present in the PKG-INFO file. As people began to package their software using Python 2.1, more and more packages will include metadata, making it possible to build automated cataloguing systems and experiment with them. With the result experience, perhaps it'll be possible to design a really good catalog and then build support for it into Python 2.2. For example, the Distutils sdist and bdist_\* commands could support an upload option that would automatically upload your package to a catalog server.

You can start creating packages containing PKG-INFO even if you're not using Python 2.1, since a new release of the Distutils will be made for users of earlier Python versions. Version 1.0.2 of the Distutils includes the changes described in 241, as well as various bugfixes and enhancements. It will be available from the Distutils SIG at https://www.python.org/community/sigs/current/distutils-sig/.

241 - Metadata for Python Software Packages
Written and implemented by A.M. Kuchling.

243 - Module Repository Upload Mechanism
Written by Sean Reifschneider, this draft PEP describes a proposed mechanism for uploading Python packages to a central server.

New and Improved Modules

Other Changes and Fixes

There were relatively few smaller changes made in Python 2.1 due to the shorter release cycle. A search through the CVS change logs turns up 117 patches applied, and 136 bugs fixed; both figures are likely to be underestimates. Some of the more notable changes are:

And there's the usual list of minor bugfixes, minor memory leaks, docstring edits, and other tweaks, too lengthy to be worth itemizing; see the CVS logs for the full details if you want them.

Acknowledgements

The author would like to thank the following people for offering suggestions on various drafts of this article: Graeme Cross, David Goodger, Jay Graves, Michael Hudson, Marc-André Lemburg, Fredrik Lundh, Neil Schemenauer, Thomas Wouters.

What's New in Python 2.2

Author
A.M. Kuchling

Introduction

This article explains the new features in Python 2.2.2, released on October 14, 2002. Python 2.2.2 is a bugfix release of Python 2.2, originally released on December 21, 2001.

Python 2.2 can be thought of as the "cleanup release". There are some features such as generators and iterators that are completely new, but most of the changes, significant and far-reaching though they may be, are aimed at cleaning up irregularities and dark corners of the language design.

This article doesn't attempt to provide a complete specification of the new features, but instead provides a convenient overview. For full details, you should refer to the documentation for Python 2.2, such as the Python Library Reference and the Python Reference Manual. If you want to understand the complete implementation and design rationale for a change, refer to the PEP for a particular new feature.

http://www.unixreview.com/documents/s=1356/urm0109h/0109h.htm
"What's So Special About Python 2.2?" is also about the new 2.2 features, and was written by Cameron Laird and Kathryn Soraiz.

PEPs 252 and 253: Type and Class Changes

The largest and most far-reaching changes in Python 2.2 are to Python's model of objects and classes. The changes should be backward compatible, so it's likely that your code will continue to run unchanged, but the changes provide some amazing new capabilities. Before beginning this, the longest and most complicated section of this article, I'll provide an overview of the changes and offer some comments.

A long time ago I wrote a web page listing flaws in Python's design. One of the most significant flaws was that it's impossible to subclass Python types implemented in C. In particular, it's not possible to subclass built-in types, so you can't just subclass, say, lists in order to add a single useful method to them. The !UserList module provides a class that supports all of the methods of lists and that can be subclassed further, but there's lots of C code that expects a regular Python list and won't accept a ~collections.UserList instance.

Python 2.2 fixes this, and in the process adds some exciting new capabilities. A brief summary:

Some users have voiced concern about all these changes. Sure, they say, the new features are neat and lend themselves to all sorts of tricks that weren't possible in previous versions of Python, but they also make the language more complicated. Some people have said that they've always recommended Python for its simplicity, and feel that its simplicity is being lost.

Personally, I think there's no need to worry. Many of the new features are quite esoteric, and you can write a lot of Python code without ever needed to be aware of them. Writing a simple class is no more difficult than it ever was, so you don't need to bother learning or teaching them unless they're actually needed. Some very complicated tasks that were previously only possible from C will now be possible in pure Python, and to my mind that's all for the better.

I'm not going to attempt to cover every single corner case and small change that were required to make the new features work. Instead this section will paint only the broad strokes. See section sect-rellinks, "Related Links", for further sources of information about Python 2.2's new object model.

Old and New Classes

First, you should know that Python 2.2 really has two kinds of classes: classic or old-style classes, and new-style classes. The old-style class model is exactly the same as the class model in earlier versions of Python. All the new features described in this section apply only to new-style classes. This divergence isn't intended to last forever; eventually old-style classes will be dropped, possibly in Python 3.0.

So how do you define a new-style class? You do it by subclassing an existing new-style class. Most of Python's built-in types, such as integers, lists, dictionaries, and even files, are new-style classes now. A new-style class named object, the base class for all built-in types, has also been added so if no built-in type is suitable, you can just subclass object:

class C(object):
    def __init__ (self):
        ...
    ...

This means that class statements that don't have any base classes are always classic classes in Python 2.2. (Actually you can also change this by setting a module-level variable named !__metaclass__ --- see 253 for the details --- but it's easier to just subclass object.)

The type objects for the built-in types are available as built-ins, named using a clever trick. Python has always had built-in functions named int, float, and str. In 2.2, they aren't functions any more, but type objects that behave as factories when called. :

>>> int
<type 'int'>
>>> int('123')
123

To make the set of types complete, new type objects such as dict and !file have been added. Here's a more interesting example, adding a !lock method to file objects:

class LockableFile(file):
    def lock (self, operation, length=0, start=0, whence=0):
        import fcntl
        return fcntl.lockf(self.fileno(), operation,
                           length, start, whence)

The now-obsolete !posixfile module contained a class that emulated all of a file object's methods and also added a !lock method, but this class couldn't be passed to internal functions that expected a built-in file, something which is possible with our new !LockableFile.

Descriptors

In previous versions of Python, there was no consistent way to discover what attributes and methods were supported by an object. There were some informal conventions, such as defining !__members__ and !__methods__ attributes that were lists of names, but often the author of an extension type or a class wouldn't bother to define them. You could fall back on inspecting the ~object.__dict__ of an object, but when class inheritance or an arbitrary !__getattr__ hook were in use this could still be inaccurate.

The one big idea underlying the new class model is that an API for describing the attributes of an object using descriptors has been formalized. Descriptors specify the value of an attribute, stating whether it's a method or a field. With the descriptor API, static methods and class methods become possible, as well as more exotic constructs.

Attribute descriptors are objects that live inside class objects, and have a few attributes of their own:

For example, when you write obj.x, the steps that Python actually performs are:

descriptor = obj.__class__.x
descriptor.__get__(obj)

For methods, descriptor.__get__ <object.__get__> returns a temporary object that's callable, and wraps up the instance and the method to be called on it. This is also why static methods and class methods are now possible; they have descriptors that wrap up just the method, or the method and the class. As a brief explanation of these new kinds of methods, static methods aren't passed the instance, and therefore resemble regular functions. Class methods are passed the class of the object, but not the object itself. Static and class methods are defined like this:

class C(object):
    def f(arg1, arg2):
        ...
    f = staticmethod(f)

    def g(cls, arg1, arg2):
        ...
    g = classmethod(g)

The staticmethod function takes the function !f, and returns it wrapped up in a descriptor so it can be stored in the class object. You might expect there to be special syntax for creating such methods (def static f, defstatic f(), or something like that) but no such syntax has been defined yet; that's been left for future versions of Python.

More new features, such as slots and properties, are also implemented as new kinds of descriptors, and it's not difficult to write a descriptor class that does something novel. For example, it would be possible to write a descriptor class that made it possible to write Eiffel-style preconditions and postconditions for a method. A class that used this feature might be defined like this:

from eiffel import eiffelmethod

class C(object):
    def f(self, arg1, arg2):
        # The actual function
        ...
    def pre_f(self):
        # Check preconditions
        ...
    def post_f(self):
        # Check postconditions
        ...

    f = eiffelmethod(f, pre_f, post_f)

Note that a person using the new !eiffelmethod doesn't have to understand anything about descriptors. This is why I think the new features don't increase the basic complexity of the language. There will be a few wizards who need to know about it in order to write !eiffelmethod or the ZODB or whatever, but most users will just write code on top of the resulting libraries and ignore the implementation details.

Multiple Inheritance: The Diamond Rule

Multiple inheritance has also been made more useful through changing the rules under which names are resolved. Consider this set of classes (diagram taken from 253 by Guido van Rossum):

class A:
  ^ ^  def save(self): ...
 /   \
/     \
/       \
/         \
class B     class C:
^         ^  def save(self): ...
\       /
\     /
 \   /
  \ /
class D

The lookup rule for classic classes is simple but not very smart; the base classes are searched depth-first, going from left to right. A reference to !D.save will search the classes !D, !B, and then !A, where !save would be found and returned. !C.save would never be found at all. This is bad, because if !C's !save method is saving some internal state specific to !C, not calling it will result in that state never getting saved.

New-style classes follow a different algorithm that's a bit more complicated to explain, but does the right thing in this situation. (Note that Python 2.3 changes this algorithm to one that produces the same results in most cases, but produces more useful results for really complicated inheritance graphs.)

  1. List all the base classes, following the classic lookup rule and include a class multiple times if it's visited repeatedly. In the above example, the list of visited classes is [!D, !B, !A, !C, !A].
  2. Scan the list for duplicated classes. If any are found, remove all but one occurrence, leaving the last one in the list. In the above example, the list becomes [!D, !B, !C, !A] after dropping duplicates.

Following this rule, referring to !D.save will return !C.save, which is the behaviour we're after. This lookup rule is the same as the one followed by Common Lisp. A new built-in function, super, provides a way to get at a class's superclasses without having to reimplement Python's algorithm. The most commonly used form will be super(class, obj), which returns a bound superclass object (not the actual class object). This form will be used in methods to call a method in the superclass; for example, !D's !save method would look like this:

class D (B,C):
    def save (self):
        # Call superclass .save()
        super(D, self).save()
        # Save D's private information here
        ...

super can also return unbound superclass objects when called as super(class) or super(class1, class2), but this probably won't often be useful.

Attribute Access

A fair number of sophisticated Python classes define hooks for attribute access using ~object.__getattr__; most commonly this is done for convenience, to make code more readable by automatically mapping an attribute access such as obj.parent into a method call such as obj.get_parent. Python 2.2 adds some new ways of controlling attribute access.

First, __getattr__(attr_name) is still supported by new-style classes, and nothing about it has changed. As before, it will be called when an attempt is made to access obj.foo and no attribute named foo is found in the instance's dictionary.

New-style classes also support a new method, __getattribute__(attr_name). The difference between the two methods is that ~object.__getattribute__ is always called whenever any attribute is accessed, while the old ~object.__getattr__ is only called if foo isn't found in the instance's dictionary.

However, Python 2.2's support for properties will often be a simpler way to trap attribute references. Writing a !__getattr__ method is complicated because to avoid recursion you can't use regular attribute accesses inside them, and instead have to mess around with the contents of ~object.__dict__. ~object.__getattr__ methods also end up being called by Python when it checks for other methods such as ~object.__repr__ or !__coerce__, and so have to be written with this in mind. Finally, calling a function on every attribute access results in a sizable performance loss.

property is a new built-in type that packages up three functions that get, set, or delete an attribute, and a docstring. For example, if you want to define a !size attribute that's computed, but also settable, you could write:

class C(object):
    def get_size (self):
        result = ... computation ...
        return result
    def set_size (self, size):
        ... compute something based on the size
        and set internal state appropriately ...

    # Define a property.  The 'delete this attribute'
    # method is defined as None, so the attribute
    # can't be deleted.
    size = property(get_size, set_size,
                    None,
                    "Storage size of this instance")

That is certainly clearer and easier to write than a pair of !__getattr__/!__setattr__ methods that check for the !size attribute and handle it specially while retrieving all other attributes from the instance's ~object.__dict__. Accesses to !size are also the only ones which have to perform the work of calling a function, so references to other attributes run at their usual speed.

Finally, it's possible to constrain the list of attributes that can be referenced on an object using the new ~object.__slots__ class attribute. Python objects are usually very dynamic; at any time it's possible to define a new attribute on an instance by just doing obj.new_attr=1. A new-style class can define a class attribute named ~object.__slots__ to limit the legal attributes to a particular set of names. An example will make this clear:

>>> class C(object):
...     __slots__ = ('template', 'name')
...
>>> obj = C()
>>> print obj.template
None
>>> obj.template = 'Test'
>>> print obj.template
Test
>>> obj.newattr = None
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
AttributeError: 'C' object has no attribute 'newattr'

Note how you get an AttributeError on the attempt to assign to an attribute not listed in ~object.__slots__.

This section has just been a quick overview of the new features, giving enough of an explanation to start you programming, but many details have been simplified or ignored. Where should you go to get a more complete picture?

The descriptorhowto is a lengthy tutorial introduction to the descriptor features, written by Guido van Rossum. If my description has whetted your appetite, go read this tutorial next, because it goes into much more detail about the new features while still remaining quite easy to read.

Next, there are two relevant PEPs, 252 and 253. 252 is titled "Making Types Look More Like Classes", and covers the descriptor API. 253 is titled "Subtyping Built-in Types", and describes the changes to type objects that make it possible to subtype built-in objects. 253 is the more complicated PEP of the two, and at a few points the necessary explanations of types and meta-types may cause your head to explode. Both PEPs were written and implemented by Guido van Rossum, with substantial assistance from the rest of the Zope Corp. team.

Finally, there's the ultimate authority: the source code. Most of the machinery for the type handling is in Objects/typeobject.c, but you should only resort to it after all other avenues have been exhausted, including posting a question to python-list or python-dev.

PEP 234: Iterators

Another significant addition to 2.2 is an iteration interface at both the C and Python levels. Objects can define how they can be looped over by callers.

In Python versions up to 2.1, the usual way to make for item in obj work is to define a ~object.__getitem__ method that looks something like this:

def __getitem__(self, index):
    return <next item>

~object.__getitem__ is more properly used to define an indexing operation on an object so that you can write obj[5] to retrieve the sixth element. It's a bit misleading when you're using this only to support for loops. Consider some file-like object that wants to be looped over; the index parameter is essentially meaningless, as the class probably assumes that a series of ~object.__getitem__ calls will be made with index incrementing by one each time. In other words, the presence of the ~object.__getitem__ method doesn't mean that using file[5] to randomly access the sixth element will work, though it really should.

In Python 2.2, iteration can be implemented separately, and ~object.__getitem__ methods can be limited to classes that really do support random access. The basic idea of iterators is simple. A new built-in function, iter(obj) or iter(C, sentinel), is used to get an iterator. iter(obj) returns an iterator for the object obj, while iter(C, sentinel) returns an iterator that will invoke the callable object C until it returns sentinel to signal that the iterator is done.

Python classes can define an !__iter__ method, which should create and return a new iterator for the object; if the object is its own iterator, this method can just return self. In particular, iterators will usually be their own iterators. Extension types implemented in C can implement a ~PyTypeObject.tp_iter function in order to return an iterator, and extension types that want to behave as iterators can define a ~PyTypeObject.tp_iternext function.

So, after all this, what do iterators actually do? They have one required method, next, which takes no arguments and returns the next value. When there are no more values to be returned, calling next should raise the StopIteration exception. :

>>> L = [1,2,3]
>>> i = iter(L)
>>> print i
<iterator object at 0x8116870>
>>> i.next()
1
>>> i.next()
2
>>> i.next()
3
>>> i.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
StopIteration
>>>

In 2.2, Python's for statement no longer expects a sequence; it expects something for which iter will return an iterator. For backward compatibility and convenience, an iterator is automatically constructed for sequences that don't implement !__iter__ or a ~PyTypeObject.tp_iter slot, so for i in [1,2,3] will still work. Wherever the Python interpreter loops over a sequence, it's been changed to use the iterator protocol. This means you can do things like this:

>>> L = [1,2,3]
>>> i = iter(L)
>>> a,b,c = i
>>> a,b,c
(1, 2, 3)

Iterator support has been added to some of Python's basic types. Calling iter on a dictionary will return an iterator which loops over its keys:

>>> m = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6,
...      'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12}
>>> for key in m: print key, m[key]
...
Mar 3
Feb 2
Aug 8
Sep 9
May 5
Jun 6
Jul 7
Jan 1
Apr 4
Nov 11
Dec 12
Oct 10

That's just the default behaviour. If you want to iterate over keys, values, or key/value pairs, you can explicitly call the !iterkeys, !itervalues, or !iteritems methods to get an appropriate iterator. In a minor related change, the in operator now works on dictionaries, so key in dict is now equivalent to dict.has_key(key).

Files also provide an iterator, which calls the readline method until there are no more lines in the file. This means you can now read each line of a file using code like this:

for line in file:
    # do something for each line
    ...

Note that you can only go forward in an iterator; there's no way to get the previous element, reset the iterator, or make a copy of it. An iterator object could provide such additional capabilities, but the iterator protocol only requires a next method.

234 - Iterators
Written by Ka-Ping Yee and GvR; implemented by the Python Labs crew, mostly by GvR and Tim Peters.

PEP 255: Simple Generators

Generators are another new feature, one that interacts with the introduction of iterators.

You're doubtless familiar with how function calls work in Python or C. When you call a function, it gets a private namespace where its local variables are created. When the function reaches a return statement, the local variables are destroyed and the resulting value is returned to the caller. A later call to the same function will get a fresh new set of local variables. But, what if the local variables weren't thrown away on exiting a function? What if you could later resume the function where it left off? This is what generators provide; they can be thought of as resumable functions.

Here's the simplest example of a generator function:

def generate_ints(N):
    for i in range(N):
        yield i

A new keyword, yield, was introduced for generators. Any function containing a !yield statement is a generator function; this is detected by Python's bytecode compiler which compiles the function specially as a result. Because a new keyword was introduced, generators must be explicitly enabled in a module by including a from __future__ import generators statement near the top of the module's source code. In Python 2.3 this statement will become unnecessary.

When you call a generator function, it doesn't return a single value; instead it returns a generator object that supports the iterator protocol. On executing the yield statement, the generator outputs the value of i, similar to a return statement. The big difference between !yield and a !return statement is that on reaching a !yield the generator's state of execution is suspended and local variables are preserved. On the next call to the generator's next() method, the function will resume executing immediately after the !yield statement. (For complicated reasons, the !yield statement isn't allowed inside the !try block of a try...finally statement; read 255 for a full explanation of the interaction between !yield and exceptions.)

Here's a sample usage of the !generate_ints generator:

>>> gen = generate_ints(3)
>>> gen
<generator object at 0x8117f90>
>>> gen.next()
0
>>> gen.next()
1
>>> gen.next()
2
>>> gen.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 2, in generate_ints
StopIteration

You could equally write for i in generate_ints(5), or a,b,c = generate_ints(3).

Inside a generator function, the return statement can only be used without a value, and signals the end of the procession of values; afterwards the generator cannot return any further values. !return with a value, such as return 5, is a syntax error inside a generator function. The end of the generator's results can also be indicated by raising StopIteration manually, or by just letting the flow of execution fall off the bottom of the function.

You could achieve the effect of generators manually by writing your own class and storing all the local variables of the generator as instance variables. For example, returning a list of integers could be done by setting self.count to 0, and having the next method increment self.count and return it. However, for a moderately complicated generator, writing a corresponding class would be much messier. Lib/test/test_generators.py contains a number of more interesting examples. The simplest one implements an in-order traversal of a tree using generators recursively. :

# A recursive generator that generates Tree leaves in in-order.
def inorder(t):
    if t:
        for x in inorder(t.left):
            yield x
        yield t.label
        for x in inorder(t.right):
            yield x

Two other examples in Lib/test/test_generators.py produce solutions for the N-Queens problem (placing $N$ queens on an $NxN$ chess board so that no queen threatens another) and the Knight's Tour (a route that takes a knight to every square of an $NxN$ chessboard without visiting any square twice).

The idea of generators comes from other programming languages, especially Icon (https://www2.cs.arizona.edu/icon/), where the idea of generators is central. In Icon, every expression and function call behaves like a generator. One example from "An Overview of the Icon Programming Language" at https://www2.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks like:

sentence := "Store it in the neighboring harbor"
if (i := find("or", sentence)) > 5 then write(i)

In Icon the !find function returns the indexes at which the substring "or" is found: 3, 23, 33. In the if statement, i is first assigned a value of 3, but 3 is less than 5, so the comparison fails, and Icon retries it with the second value of 23. 23 is greater than 5, so the comparison now succeeds, and the code prints the value 23 to the screen.

Python doesn't go nearly as far as Icon in adopting generators as a central concept. Generators are considered a new part of the core Python language, but learning or using them isn't compulsory; if they don't solve any problems that you have, feel free to ignore them. One novel feature of Python's interface as compared to Icon's is that a generator's state is represented as a concrete object (the iterator) that can be passed around to other functions or stored in a data structure.

255 - Simple Generators
Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland. Implemented mostly by Neil Schemenauer and Tim Peters, with other fixes from the Python Labs crew.

PEP 237: Unifying Long Integers and Integers

In recent versions, the distinction between regular integers, which are 32-bit values on most machines, and long integers, which can be of arbitrary size, was becoming an annoyance. For example, on platforms that support files larger than 2**32 bytes, the !tell method of file objects has to return a long integer. However, there were various bits of Python that expected plain integers and would raise an error if a long integer was provided instead. For example, in Python 1.5, only regular integers could be used as a slice index, and 'abc'[1L:] would raise a TypeError exception with the message 'slice index must be int'.

Python 2.2 will shift values from short to long integers as required. The 'L' suffix is no longer needed to indicate a long integer literal, as now the compiler will choose the appropriate type. (Using the 'L' suffix will be discouraged in future 2.x versions of Python, triggering a warning in Python 2.4, and probably dropped in Python 3.0.) Many operations that used to raise an OverflowError will now return a long integer as their result. For example:

>>> 1234567890123
1234567890123L
>>> 2 ** 64
18446744073709551616L

In most cases, integers and long integers will now be treated identically. You can still distinguish them with the type built-in function, but that's rarely needed.

237 - Unifying Long Integers and Integers
Written by Moshe Zadka and Guido van Rossum. Implemented mostly by Guido van Rossum.

PEP 238: Changing the Division Operator

The most controversial change in Python 2.2 heralds the start of an effort to fix an old design flaw that's been in Python from the beginning. Currently Python's division operator, /, behaves like C's division operator when presented with two integer arguments: it returns an integer result that's truncated down when there would be a fractional part. For example, 3/2 is 1, not 1.5, and (-1)/2 is -1, not -0.5. This means that the results of division can vary unexpectedly depending on the type of the two operands and because Python is dynamically typed, it can be difficult to determine the possible types of the operands.

(The controversy is over whether this is really a design flaw, and whether it's worth breaking existing code to fix this. It's caused endless discussions on python-dev, and in July 2001 erupted into a storm of acidly sarcastic postings on comp.lang.python. I won't argue for either side here and will stick to describing what's implemented in 2.2. Read 238 for a summary of arguments and counter-arguments.)

Because this change might break code, it's being introduced very gradually. Python 2.2 begins the transition, but the switch won't be complete until Python 3.0.

First, I'll borrow some terminology from 238. "True division" is the division that most non-programmers are familiar with: 3/2 is 1.5, 1/4 is 0.25, and so forth. "Floor division" is what Python's / operator currently does when given integer operands; the result is the floor of the value returned by true division. "Classic division" is the current mixed behaviour of /; it returns the result of floor division when the operands are integers, and returns the result of true division when one of the operands is a floating-point number.

Here are the changes 2.2 introduces:

238 - Changing the Division Operator
Written by Moshe Zadka and Guido van Rossum. Implemented by Guido van Rossum..

Unicode Changes

Python's Unicode support has been enhanced a bit in 2.2. Unicode strings are usually stored as UCS-2, as 16-bit unsigned integers. Python 2.2 can also be compiled to use UCS-4, 32-bit unsigned integers, as its internal encoding by supplying !--enable-unicode=ucs4 to the configure script. (It's also possible to specify !--disable-unicode to completely disable Unicode support.)

When built to use UCS-4 (a "wide Python"), the interpreter can natively handle Unicode characters from U+000000 to U+110000, so the range of legal values for the !unichr function is expanded accordingly. Using an interpreter compiled to use UCS-2 (a "narrow Python"), values greater than 65535 will still cause !unichr to raise a ValueError exception. This is all described in 261, "Support for 'wide' Unicode characters"; consult it for further details.

Another change is simpler to explain. Since their introduction, Unicode strings have supported an !encode method to convert the string to a selected encoding such as UTF-8 or Latin-1. A symmetric decode([*encoding*]) method has been added to 8-bit strings (though not to Unicode strings) in 2.2. !decode assumes that the string is in the specified encoding and decodes it, returning whatever is returned by the codec.

Using this new feature, codecs have been added for tasks not directly related to Unicode. For example, codecs have been added for uu-encoding, MIME's base64 encoding, and compression with the zlib module:

>>> s = """Here is a lengthy piece of redundant, overly verbose,
... and repetitive text.
... """
>>> data = s.encode('zlib')
>>> data
'x\x9c\r\xc9\xc1\r\x80 \x10\x04\xc0?Ul...'
>>> data.decode('zlib')
'Here is a lengthy piece of redundant, overly verbose,\nand repetitive text.\n'
>>> print s.encode('uu')
begin 666 <data>
M2&5R92!I<R!A(&QE;F=T:'D@<&EE8V4@;V8@<F5D=6YD86YT+"!O=F5R;'D@
>=F5R8F]S92P*86YD(')E<&5T:71I=F4@=&5X="X*

end
>>> "sheesh".encode('rot-13')
'furrfu'

To convert a class instance to Unicode, a !__unicode__ method can be defined by a class, analogous to !__str__.

!encode, !decode, and !__unicode__ were implemented by Marc-André Lemburg. The changes to support using UCS-4 internally were implemented by Fredrik Lundh and Martin von Löwis.

261 - Support for 'wide' Unicode characters
Written by Paul Prescod.

PEP 227: Nested Scopes

In Python 2.1, statically nested scopes were added as an optional feature, to be enabled by a from __future__ import nested_scopes directive. In 2.2 nested scopes no longer need to be specially enabled, and are now always present. The rest of this section is a copy of the description of nested scopes from my "What's New in Python 2.1" document; if you read it when 2.1 came out, you can skip the rest of this section.

The largest change introduced in Python 2.1, and made complete in 2.2, is to Python's scoping rules. In Python 2.0, at any given time there are at most three namespaces used to look up variable names: local, module-level, and the built-in namespace. This often surprised people because it didn't match their intuitive expectations. For example, a nested recursive function definition doesn't work:

def f():
    ...
    def g(value):
        ...
        return g(value-1) + 1
    ...

The function !g will always raise a NameError exception, because the binding of the name g isn't in either its local namespace or in the module-level namespace. This isn't much of a problem in practice (how often do you recursively define interior functions like this?), but this also made using the lambda expression clumsier, and this was a problem in practice. In code which uses !lambda you can often find local variables being copied by passing them as the default values of arguments. :

def find(self, name):
    "Return list of any entries equal to 'name'"
    L = filter(lambda x, name=name: x == name,
               self.list_attribute)
    return L

The readability of Python code written in a strongly functional style suffers greatly as a result.

The most significant change to Python 2.2 is that static scoping has been added to the language to fix this problem. As a first effect, the name=name default argument is now unnecessary in the above example. Put simply, when a given variable name is not assigned a value within a function (by an assignment, or the def, class, or import statements), references to the variable will be looked up in the local namespace of the enclosing scope. A more detailed explanation of the rules, and a dissection of the implementation, can be found in the PEP.

This change may cause some compatibility problems for code where the same variable name is used both at the module level and as a local variable within a function that contains further function definitions. This seems rather unlikely though, since such code would have been pretty confusing to read in the first place.

One side effect of the change is that the from module import * and exec statements have been made illegal inside a function scope under certain conditions. The Python reference manual has said all along that from module import * is only legal at the top level of a module, but the CPython interpreter has never enforced this before. As part of the implementation of nested scopes, the compiler which turns Python source into bytecodes has to generate different code to access variables in a containing scope. from module import * and exec make it impossible for the compiler to figure this out, because they add names to the local namespace that are unknowable at compile time. Therefore, if a function contains function definitions or lambda expressions with free variables, the compiler will flag this by raising a SyntaxError exception.

To make the preceding explanation a bit clearer, here's an example:

x = 1
def f():
    # The next line is a syntax error
    exec 'x=2'
    def g():
        return x

Line 4 containing the exec statement is a syntax error, since exec would define a new local variable named x whose value should be accessed by !g.

This shouldn't be much of a limitation, since exec is rarely used in most Python code (and when it is used, it's often a sign of a poor design anyway).

227 - Statically Nested Scopes
Written and implemented by Jeremy Hylton.

New and Improved Modules

Interpreter Changes and Fixes

Some of the changes only affect people who deal with the Python interpreter at the C level because they're writing Python extension modules, embedding the interpreter, or just hacking on the interpreter itself. If you only write Python code, none of the changes described here will affect you very much.

Other Changes and Fixes

As usual there were a bunch of other improvements and bugfixes scattered throughout the source tree. A search through the CVS change logs finds there were 527 patches applied and 683 bugs fixed between Python 2.1 and 2.2; 2.2.1 applied 139 patches and fixed 143 bugs; 2.2.2 applied 106 patches and fixed 82 bugs. These figures are likely to be underestimates.

Some of the more notable changes are:

Acknowledgements

The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this article: Fred Bremmer, Keith Briggs, Andrew Dalke, Fred L. Drake, Jr., Carel Fellinger, David Goodger, Mark Hammond, Stephen Hansen, Michael Hudson, Jack Jansen, Marc-André Lemburg, Martin von Löwis, Fredrik Lundh, Michael McLay, Nick Mathewson, Paul Moore, Gustavo Niemeyer, Don O'Donnell, Joonas Paalasma, Tim Peters, Jens Quade, Tom Reinhardt, Neil Schemenauer, Guido van Rossum, Greg Ward, Edward Welbourne.

What's New in Python 2.3

Author
A.M. Kuchling

This article explains the new features in Python 2.3. Python 2.3 was released on July 29, 2003.

The main themes for Python 2.3 are polishing some of the features added in 2.2, adding various small but useful enhancements to the core language, and expanding the standard library. The new object model introduced in the previous version has benefited from 18 months of bugfixes and from optimization efforts that have improved the performance of new-style classes. A few new built-in functions have been added such as sum and enumerate. The in operator can now be used for substring searches (e.g. "ab" in "abc" returns True).

Some of the many new library features include Boolean, set, heap, and date/time data types, the ability to import modules from ZIP-format archives, metadata support for the long-awaited Python catalog, an updated version of IDLE, and modules for logging messages, wrapping text, parsing CSV files, processing command-line options, using BerkeleyDB databases... the list of new and enhanced modules is lengthy.

This article doesn't attempt to provide a complete specification of the new features, but instead provides a convenient overview. For full details, you should refer to the documentation for Python 2.3, such as the Python Library Reference and the Python Reference Manual. If you want to understand the complete implementation and design rationale, refer to the PEP for a particular new feature.

PEP 218: A Standard Set Datatype

The new !sets module contains an implementation of a set datatype. The Set class is for mutable sets, sets that can have members added and removed. The !ImmutableSet class is for sets that can't be modified, and instances of !ImmutableSet can therefore be used as dictionary keys. Sets are built on top of dictionaries, so the elements within a set must be hashable.

Here's a simple example:

>>> import sets
>>> S = sets.Set([1,2,3])
>>> S
Set([1, 2, 3])
>>> 1 in S
True
>>> 0 in S
False
>>> S.add(5)
>>> S.remove(3)
>>> S
Set([1, 2, 5])
>>>

The union and intersection of sets can be computed with the ~frozenset.union and ~frozenset.intersection methods; an alternative notation uses the bitwise operators & and |. Mutable sets also have in-place versions of these methods, !union_update and ~set.intersection_update. :

>>> S1 = sets.Set([1,2,3])
>>> S2 = sets.Set([4,5,6])
>>> S1.union(S2)
Set([1, 2, 3, 4, 5, 6])
>>> S1 | S2                  # Alternative notation
Set([1, 2, 3, 4, 5, 6])
>>> S1.intersection(S2)
Set([])
>>> S1 & S2                  # Alternative notation
Set([])
>>> S1.union_update(S2)
>>> S1
Set([1, 2, 3, 4, 5, 6])
>>>

It's also possible to take the symmetric difference of two sets. This is the set of all elements in the union that aren't in the intersection. Another way of putting it is that the symmetric difference contains all elements that are in exactly one set. Again, there's an alternative notation (^), and an in-place version with the ungainly name ~set.symmetric_difference_update. :

>>> S1 = sets.Set([1,2,3,4])
>>> S2 = sets.Set([3,4,5,6])
>>> S1.symmetric_difference(S2)
Set([1, 2, 5, 6])
>>> S1 ^ S2
Set([1, 2, 5, 6])
>>>

There are also !issubset and !issuperset methods for checking whether one set is a subset or superset of another:

>>> S1 = sets.Set([1,2,3])
>>> S2 = sets.Set([2,3])
>>> S2.issubset(S1)
True
>>> S1.issubset(S2)
False
>>> S1.issuperset(S2)
True
>>>

218 - Adding a Built-In Set Object Type
PEP written by Greg V. Wilson. Implemented by Greg V. Wilson, Alex Martelli, and GvR.

PEP 255: Simple Generators

In Python 2.2, generators were added as an optional feature, to be enabled by a from __future__ import generators directive. In 2.3 generators no longer need to be specially enabled, and are now always present; this means that yield is now always a keyword. The rest of this section is a copy of the description of generators from the "What's New in Python 2.2" document; if you read it back when Python 2.2 came out, you can skip the rest of this section.

You're doubtless familiar with how function calls work in Python or C. When you call a function, it gets a private namespace where its local variables are created. When the function reaches a return statement, the local variables are destroyed and the resulting value is returned to the caller. A later call to the same function will get a fresh new set of local variables. But, what if the local variables weren't thrown away on exiting a function? What if you could later resume the function where it left off? This is what generators provide; they can be thought of as resumable functions.

Here's the simplest example of a generator function:

def generate_ints(N):
    for i in range(N):
        yield i

A new keyword, yield, was introduced for generators. Any function containing a !yield statement is a generator function; this is detected by Python's bytecode compiler which compiles the function specially as a result.

When you call a generator function, it doesn't return a single value; instead it returns a generator object that supports the iterator protocol. On executing the yield statement, the generator outputs the value of i, similar to a return statement. The big difference between !yield and a !return statement is that on reaching a !yield the generator's state of execution is suspended and local variables are preserved. On the next call to the generator's .next() method, the function will resume executing immediately after the !yield statement. (For complicated reasons, the !yield statement isn't allowed inside the try block of a !try...!finally statement; read 255 for a full explanation of the interaction between !yield and exceptions.)

Here's a sample usage of the !generate_ints generator:

>>> gen = generate_ints(3)
>>> gen
<generator object at 0x8117f90>
>>> gen.next()
0
>>> gen.next()
1
>>> gen.next()
2
>>> gen.next()
Traceback (most recent call last):
  File "stdin", line 1, in ?
  File "stdin", line 2, in generate_ints
StopIteration

You could equally write for i in generate_ints(5), or a,b,c = generate_ints(3).

Inside a generator function, the return statement can only be used without a value, and signals the end of the procession of values; afterwards the generator cannot return any further values. !return with a value, such as return 5, is a syntax error inside a generator function. The end of the generator's results can also be indicated by raising StopIteration manually, or by just letting the flow of execution fall off the bottom of the function.

You could achieve the effect of generators manually by writing your own class and storing all the local variables of the generator as instance variables. For example, returning a list of integers could be done by setting self.count to 0, and having the next method increment self.count and return it. However, for a moderately complicated generator, writing a corresponding class would be much messier. Lib/test/test_generators.py contains a number of more interesting examples. The simplest one implements an in-order traversal of a tree using generators recursively. :

# A recursive generator that generates Tree leaves in in-order.
def inorder(t):
    if t:
        for x in inorder(t.left):
            yield x
        yield t.label
        for x in inorder(t.right):
            yield x

Two other examples in Lib/test/test_generators.py produce solutions for the N-Queens problem (placing $N$ queens on an $NxN$ chess board so that no queen threatens another) and the Knight's Tour (a route that takes a knight to every square of an $NxN$ chessboard without visiting any square twice).

The idea of generators comes from other programming languages, especially Icon (https://www2.cs.arizona.edu/icon/), where the idea of generators is central. In Icon, every expression and function call behaves like a generator. One example from "An Overview of the Icon Programming Language" at https://www2.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks like:

sentence := "Store it in the neighboring harbor"
if (i := find("or", sentence)) > 5 then write(i)

In Icon the !find function returns the indexes at which the substring "or" is found: 3, 23, 33. In the if statement, i is first assigned a value of 3, but 3 is less than 5, so the comparison fails, and Icon retries it with the second value of 23. 23 is greater than 5, so the comparison now succeeds, and the code prints the value 23 to the screen.

Python doesn't go nearly as far as Icon in adopting generators as a central concept. Generators are considered part of the core Python language, but learning or using them isn't compulsory; if they don't solve any problems that you have, feel free to ignore them. One novel feature of Python's interface as compared to Icon's is that a generator's state is represented as a concrete object (the iterator) that can be passed around to other functions or stored in a data structure.

255 - Simple Generators
Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland. Implemented mostly by Neil Schemenauer and Tim Peters, with other fixes from the Python Labs crew.

PEP 263: Source Code Encodings

Python source files can now be declared as being in different character set encodings. Encodings are declared by including a specially formatted comment in the first or second line of the source file. For example, a UTF-8 file can be declared with:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

Without such an encoding declaration, the default encoding used is 7-bit ASCII. Executing or importing modules that contain string literals with 8-bit characters and have no encoding declaration will result in a DeprecationWarning being signalled by Python 2.3; in 2.4 this will be a syntax error.

The encoding declaration only affects Unicode string literals, which will be converted to Unicode using the specified encoding. Note that Python identifiers are still restricted to ASCII characters, so you can't have variable names that use characters outside of the usual alphanumerics.

263 - Defining Python Source Code Encodings
Written by Marc-André Lemburg and Martin von Löwis; implemented by Suzuki Hisao and Martin von Löwis.

PEP 273: Importing Modules from ZIP Archives

The new zipimport module adds support for importing modules from a ZIP-format archive. You don't need to import the module explicitly; it will be automatically imported if a ZIP archive's filename is added to sys.path. For example:

amk@nyman:~/src/python$ unzip -l /tmp/example.zip
Archive:  /tmp/example.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
     8467  11-26-02 22:30   jwzthreading.py
 --------                   -------
     8467                   1 file
amk@nyman:~/src/python$ ./python
Python 2.3 (#1, Aug 1 2003, 19:54:32)
>>> import sys
>>> sys.path.insert(0, '/tmp/example.zip')  # Add .zip file to front of path
>>> import jwzthreading
>>> jwzthreading.__file__
'/tmp/example.zip/jwzthreading.py'
>>>

An entry in sys.path can now be the filename of a ZIP archive. The ZIP archive can contain any kind of files, but only files named \*.py, \*.pyc, or \*.pyo can be imported. If an archive only contains \*.py files, Python will not attempt to modify the archive by adding the corresponding \*.pyc file, meaning that if a ZIP archive doesn't contain \*.pyc files, importing may be rather slow.

A path within the archive can also be specified to only import from a subdirectory; for example, the path /tmp/example.zip/lib/ would only import from the lib/ subdirectory within the archive.

273 - Import Modules from Zip Archives
Written by James C. Ahlstrom, who also provided an implementation. Python 2.3 follows the specification in 273, but uses an implementation written by Just van Rossum that uses the import hooks described in 302. See section section-pep302 for a description of the new import hooks.

PEP 277: Unicode file name support for Windows NT

On Windows NT, 2000, and XP, the system stores file names as Unicode strings. Traditionally, Python has represented file names as byte strings, which is inadequate because it renders some file names inaccessible.

Python now allows using arbitrary Unicode strings (within the limitations of the file system) for all functions that expect file names, most notably the open built-in function. If a Unicode string is passed to os.listdir, Python now returns a list of Unicode strings. A new function, !os.getcwdu, returns the current directory as a Unicode string.

Byte strings still work as file names, and on Windows Python will transparently convert them to Unicode using the mbcs encoding.

Other systems also allow Unicode strings as file names but convert them to byte strings before passing them to the system, which can cause a UnicodeError to be raised. Applications can test whether arbitrary Unicode strings are supported as file names by checking os.path.supports_unicode_filenames, a Boolean value.

Under MacOS, os.listdir may now return Unicode filenames.

277 - Unicode file name support for Windows NT
Written by Neil Hodgson; implemented by Neil Hodgson, Martin von Löwis, and Mark Hammond.

single: universal newlines; What's new

PEP 278: Universal Newline Support

The three major operating systems used today are Microsoft Windows, Apple's Macintosh OS, and the various Unix derivatives. A minor irritation of cross-platform work is that these three platforms all use different characters to mark the ends of lines in text files. Unix uses the linefeed (ASCII character 10), MacOS uses the carriage return (ASCII character 13), and Windows uses a two-character sequence of a carriage return plus a newline.

Python's file objects can now support end of line conventions other than the one followed by the platform on which Python is running. Opening a file with the mode 'U' or 'rU' will open a file for reading in universal newlines mode. All three line ending conventions will be translated to a '\n' in the strings returned by the various file methods such as !read and !readline.

Universal newline support is also used when importing modules and when executing a file with the !execfile function. This means that Python modules can be shared between all three operating systems without needing to convert the line-endings.

This feature can be disabled when compiling Python by specifying the !--without-universal-newlines switch when running Python's configure script.

278 - Universal Newline Support
Written and implemented by Jack Jansen.

PEP 279: enumerate()

A new built-in function, enumerate, will make certain loops a bit clearer. enumerate(thing), where thing is either an iterator or a sequence, returns an iterator that will return (0, thing[0]), (1, thing[1]), (2, thing[2]), and so forth.

A common idiom to change every element of a list looks like this:

for i in range(len(L)):
    item = L[i]
    # ... compute some result based on item ...
    L[i] = result

This can be rewritten using enumerate as:

for i, item in enumerate(L):
    # ... compute some result based on item ...
    L[i] = result

279 - The enumerate() built-in function
Written and implemented by Raymond D. Hettinger.

PEP 282: The logging Package

A standard package for writing logs, logging, has been added to Python 2.3. It provides a powerful and flexible mechanism for generating logging output which can then be filtered and processed in various ways. A configuration file written in a standard format can be used to control the logging behavior of a program. Python includes handlers that will write log records to standard error or to a file or socket, send them to the system log, or even e-mail them to a particular address; of course, it's also possible to write your own handler classes.

The ~logging.Logger class is the primary class. Most application code will deal with one or more ~logging.Logger objects, each one used by a particular subsystem of the application. Each ~logging.Logger is identified by a name, and names are organized into a hierarchy using . as the component separator. For example, you might have ~logging.Logger instances named server, server.auth and server.network. The latter two instances are below server in the hierarchy. This means that if you turn up the verbosity for server or direct server messages to a different handler, the changes will also apply to records logged to server.auth and server.network. There's also a root ~logging.Logger that's the parent of all other loggers.

For simple uses, the logging package contains some convenience functions that always use the root log:

import logging

logging.debug('Debugging information')
logging.info('Informational message')
logging.warning('Warning:config file %s not found', 'server.conf')
logging.error('Error occurred')
logging.critical('Critical error -- shutting down')

This produces the following output:

WARNING:root:Warning:config file server.conf not found
ERROR:root:Error occurred
CRITICAL:root:Critical error -- shutting down

In the default configuration, informational and debugging messages are suppressed and the output is sent to standard error. You can enable the display of informational and debugging messages by calling the ~logging.Logger.setLevel method on the root logger.

Notice the ~logging.warning call's use of string formatting operators; all of the functions for logging messages take the arguments (msg, arg1, arg2, ...) and log the string resulting from msg % (arg1, arg2, ...).

There's also an ~logging.exception function that records the most recent traceback. Any of the other functions will also record the traceback if you specify a true value for the keyword argument exc_info. :

def f():
    try:    1/0
    except: logging.exception('Problem recorded')

f()

This produces the following output:

ERROR:root:Problem recorded
Traceback (most recent call last):
  File "t.py", line 6, in f
    1/0
ZeroDivisionError: integer division or modulo by zero

Slightly more advanced programs will use a logger other than the root logger. The getLogger(name) function is used to get a particular log, creating it if it doesn't exist yet. getLogger(None) returns the root logger. :

log = logging.getLogger('server')
 ...
log.info('Listening on port %i', port)
 ...
log.critical('Disk full')
 ...

Log records are usually propagated up the hierarchy, so a message logged to server.auth is also seen by server and root, but a ~logging.Logger can prevent this by setting its ~logging.Logger.propagate attribute to False.

There are more classes provided by the logging package that can be customized. When a ~logging.Logger instance is told to log a message, it creates a ~logging.LogRecord instance that is sent to any number of different ~logging.Handler instances. Loggers and handlers can also have an attached list of filters, and each filter can cause the ~logging.LogRecord to be ignored or can modify the record before passing it along. When they're finally output, ~logging.LogRecord instances are converted to text by a ~logging.Formatter class. All of these classes can be replaced by your own specially written classes.

With all of these features the logging package should provide enough flexibility for even the most complicated applications. This is only an incomplete overview of its features, so please see the package's reference documentation for all of the details. Reading 282 will also be helpful.

282 - A Logging System
Written by Vinay Sajip and Trent Mick; implemented by Vinay Sajip.

PEP 285: A Boolean Type

A Boolean type was added to Python 2.3. Two new constants were added to the !__builtin__ module, True and False. (True and False constants were added to the built-ins in Python 2.2.1, but the 2.2.1 versions are simply set to integer values of 1 and 0 and aren't a different type.)

The type object for this new type is named bool; the constructor for it takes any Python value and converts it to True or False. :

>>> bool(1)
True
>>> bool(0)
False
>>> bool([])
False
>>> bool( (1,) )
True

Most of the standard library modules and built-in functions have been changed to return Booleans. :

>>> obj = []
>>> hasattr(obj, 'append')
True
>>> isinstance(obj, list)
True
>>> isinstance(obj, tuple)
False

Python's Booleans were added with the primary goal of making code clearer. For example, if you're reading a function and encounter the statement return 1, you might wonder whether the 1 represents a Boolean truth value, an index, or a coefficient that multiplies some other quantity. If the statement is return True, however, the meaning of the return value is quite clear.

Python's Booleans were not added for the sake of strict type-checking. A very strict language such as Pascal would also prevent you performing arithmetic with Booleans, and would require that the expression in an if statement always evaluate to a Boolean result. Python is not this strict and never will be, as 285 explicitly says. This means you can still use any expression in an !if statement, even ones that evaluate to a list or tuple or some random object. The Boolean type is a subclass of the int class so that arithmetic using a Boolean still works. :

>>> True + 1
2
>>> False + 1
1
>>> False * 75
0
>>> True * 75
75

To sum up True and False in a sentence: they're alternative ways to spell the integer values 1 and 0, with the single difference that str and repr return the strings 'True' and 'False' instead of '1' and '0'.

285 - Adding a bool type
Written and implemented by GvR.

PEP 293: Codec Error Handling Callbacks

When encoding a Unicode string into a byte string, unencodable characters may be encountered. So far, Python has allowed specifying the error processing as either "strict" (raising UnicodeError), "ignore" (skipping the character), or "replace" (using a question mark in the output string), with "strict" being the default behavior. It may be desirable to specify alternative processing of such errors, such as inserting an XML character reference or HTML entity reference into the converted string.

Python now has a flexible framework to add different processing strategies. New error handlers can be added with codecs.register_error, and codecs then can access the error handler with codecs.lookup_error. An equivalent C API has been added for codecs written in C. The error handler gets the necessary state information such as the string being converted, the position in the string where the error was detected, and the target encoding. The handler can then either raise an exception or return a replacement string.

Two additional error handlers have been implemented using this framework: "backslashreplace" uses Python backslash quoting to represent unencodable characters and "xmlcharrefreplace" emits XML character references.

293 - Codec Error Handling Callbacks
Written and implemented by Walter Dörwald.

PEP 301: Package Index and Metadata for Distutils

Support for the long-requested Python catalog makes its first appearance in 2.3.

The heart of the catalog is the new Distutils register command. Running python setup.py register will collect the metadata describing a package, such as its name, version, maintainer, description, &c., and send it to a central catalog server. The resulting catalog is available from https://pypi.org.

To make the catalog a bit more useful, a new optional classifiers keyword argument has been added to the Distutils !setup function. A list of Trove-style strings can be supplied to help classify the software.

Here's an example setup.py with classifiers, written to be compatible with older versions of the Distutils:

from distutils import core
kw = {'name': "Quixote",
      'version': "0.5.1",
      'description': "A highly Pythonic Web application framework",
      # ...
      }

if (hasattr(core, 'setup_keywords') and
    'classifiers' in core.setup_keywords):
    kw['classifiers'] = \
        ['Topic :: Internet :: WWW/HTTP :: Dynamic Content',
         'Environment :: No Input/Output (Daemon)',
         'Intended Audience :: Developers'],

core.setup(**kw)

The full list of classifiers can be obtained by running python setup.py register --list-classifiers.

301 - Package Index and Metadata for Distutils
Written and implemented by Richard Jones.

PEP 302: New Import Hooks

While it's been possible to write custom import hooks ever since the !ihooks module was introduced in Python 1.3, no one has ever been really happy with it because writing new import hooks is difficult and messy. There have been various proposed alternatives such as the !imputil and !iu modules, but none of them has ever gained much acceptance, and none of them were easily usable from C code.

302 borrows ideas from its predecessors, especially from Gordon McMillan's !iu module. Three new items are added to the sys module:

Importer objects must have a single method, find_module(fullname, path=None). fullname will be a module or package name, e.g. string or distutils.core. !find_module must return a loader object that has a single method, load_module(fullname), that creates and returns the corresponding module object.

Pseudo-code for Python's new import logic, therefore, looks something like this (simplified a bit; see 302 for the full details):

for mp in sys.meta_path:
    loader = mp(fullname)
    if loader is not None:
        <module> = loader.load_module(fullname)

for path in sys.path:
    for hook in sys.path_hooks:
        try:
            importer = hook(path)
        except ImportError:
            # ImportError, so try the other path hooks
            pass
        else:
            loader = importer.find_module(fullname)
            <module> = loader.load_module(fullname)

# Not found!
raise ImportError

302 - New Import Hooks
Written by Just van Rossum and Paul Moore. Implemented by Just van Rossum.

PEP 305: Comma-separated Files

Comma-separated files are a format frequently used for exporting data from databases and spreadsheets. Python 2.3 adds a parser for comma-separated files.

Comma-separated format is deceptively simple at first glance:

Costs,150,200,3.95

Read a line and call line.split(','): what could be simpler? But toss in string data that can contain commas, and things get more complicated:

"Costs",150,200,3.95,"Includes taxes, shipping, and sundry items"

A big ugly regular expression can parse this, but using the new csv package is much simpler:

import csv

input = open('datafile', 'rb')
reader = csv.reader(input)
for line in reader:
    print line

The ~csv.reader function takes a number of different options. The field separator isn't limited to the comma and can be changed to any character, and so can the quoting and line-ending characters.

Different dialects of comma-separated files can be defined and registered; currently there are two dialects, both used by Microsoft Excel. A separate csv.writer class will generate comma-separated files from a succession of tuples or lists, quoting strings that contain the delimiter.

305 - CSV File API
Written and implemented by Kevin Altis, Dave Cole, Andrew McNamara, Skip Montanaro, Cliff Wells.

PEP 307: Pickle Enhancements

The pickle and !cPickle modules received some attention during the 2.3 development cycle. In 2.2, new-style classes could be pickled without difficulty, but they weren't pickled very compactly; 307 quotes a trivial example where a new-style class results in a pickled string three times longer than that for a classic class.

The solution was to invent a new pickle protocol. The pickle.dumps function has supported a text-or-binary flag for a long time. In 2.3, this flag is redefined from a Boolean to an integer: 0 is the old text-mode pickle format, 1 is the old binary format, and now 2 is a new 2.3-specific format. A new constant, pickle.HIGHEST_PROTOCOL, can be used to select the fanciest protocol available.

Unpickling is no longer considered a safe operation. 2.2's pickle provided hooks for trying to prevent unsafe classes from being unpickled (specifically, a !__safe_for_unpickling__ attribute), but none of this code was ever audited and therefore it's all been ripped out in 2.3. You should not unpickle untrusted data in any version of Python.

To reduce the pickling overhead for new-style classes, a new interface for customizing pickling was added using three special methods: ~object.__getstate__, ~object.__setstate__, and ~object.__getnewargs__. Consult 307 for the full semantics of these methods.

As a way to compress pickles yet further, it's now possible to use integer codes instead of long strings to identify pickled classes. The Python Software Foundation will maintain a list of standardized codes; there's also a range of codes for private use. Currently no codes have been specified.

307 - Extensions to the pickle protocol
Written and implemented by Guido van Rossum and Tim Peters.

Extended Slices

Ever since Python 1.4, the slicing syntax has supported an optional third "step" or "stride" argument. For example, these are all legal Python syntax: L[1:10:2], L[:-1:1], L[::-1]. This was added to Python at the request of the developers of Numerical Python, which uses the third argument extensively. However, Python's built-in list, tuple, and string sequence types have never supported this feature, raising a TypeError if you tried it. Michael Hudson contributed a patch to fix this shortcoming.

For example, you can now easily extract the elements of a list that have even indexes:

>>> L = range(10)
>>> L[::2]
[0, 2, 4, 6, 8]

Negative values also work to make a copy of the same list in reverse order:

>>> L[::-1]
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

This also works for tuples, arrays, and strings:

>>> s='abcd'
>>> s[::2]
'ac'
>>> s[::-1]
'dcba'

If you have a mutable sequence such as a list or an array you can assign to or delete an extended slice, but there are some differences between assignment to extended and regular slices. Assignment to a regular slice can be used to change the length of the sequence:

>>> a = range(3)
>>> a
[0, 1, 2]
>>> a[1:3] = [4, 5, 6]
>>> a
[0, 4, 5, 6]

Extended slices aren't this flexible. When assigning to an extended slice, the list on the right hand side of the statement must contain the same number of items as the slice it is replacing:

>>> a = range(4)
>>> a
[0, 1, 2, 3]
>>> a[::2]
[0, 2]
>>> a[::2] = [0, -1]
>>> a
[0, 1, -1, 3]
>>> a[::2] = [0,1,2]
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ValueError: attempt to assign sequence of size 3 to extended slice of size 2

Deletion is more straightforward:

>>> a = range(4)
>>> a
[0, 1, 2, 3]
>>> a[::2]
[0, 2]
>>> del a[::2]
>>> a
[1, 3]

One can also now pass slice objects to the ~object.__getitem__ methods of the built-in sequences:

>>> range(10).__getitem__(slice(0, 5, 2))
[0, 2, 4]

Or use slice objects directly in subscripts:

>>> range(10)[slice(0, 5, 2)]
[0, 2, 4]

To simplify implementing sequences that support extended slicing, slice objects now have a method indices(length) which, given the length of a sequence, returns a (start, stop, step) tuple that can be passed directly to range. !indices handles omitted and out-of-bounds indices in a manner consistent with regular slices (and this innocuous phrase hides a welter of confusing details!). The method is intended to be used like this:

class FakeSeq:
    ...
    def calc_item(self, i):
        ...
    def __getitem__(self, item):
        if isinstance(item, slice):
            indices = item.indices(len(self))
            return FakeSeq([self.calc_item(i) for i in range(*indices)])
        else:
            return self.calc_item(i)

From this example you can also see that the built-in slice object is now the type object for the slice type, and is no longer a function. This is consistent with Python 2.2, where int, str, etc., underwent the same change.

Other Language Changes

Here are all of the changes that Python 2.3 makes to the core Python language.

String Changes

Optimizations

The net result of the 2.3 optimizations is that Python 2.3 runs the pystone benchmark around 25% faster than Python 2.2.

New, Improved, and Deprecated Modules

As usual, Python's standard library received a number of enhancements and bug fixes. Here's a partial list of the most notable changes, sorted alphabetically by module name. Consult the Misc/NEWS file in the source tree for a more complete list of changes, or look through the CVS logs for all the details.

Date/Time Type

Date and time types suitable for expressing timestamps were added as the datetime module. The types don't support different calendars or many fancy features, and just stick to the basics of representing time.

The three primary types are: ~datetime.date, representing a day, month, and year; ~datetime.time, consisting of hour, minute, and second; and ~datetime.datetime, which contains all the attributes of both ~datetime.date and ~datetime.time. There's also a ~datetime.timedelta class representing differences between two points in time, and time zone logic is implemented by classes inheriting from the abstract ~datetime.tzinfo class.

You can create instances of ~datetime.date and ~datetime.time by either supplying keyword arguments to the appropriate constructor, e.g. datetime.date(year=1972, month=10, day=15), or by using one of a number of class methods. For example, the ~datetime.date.today class method returns the current local date.

Once created, instances of the date/time classes are all immutable. There are a number of methods for producing formatted strings from objects:

>>> import datetime as dt
>>> now = dt.datetime.now()
>>> now.isoformat()
'2002-12-30T21:27:03.994956'
>>> now.ctime()  # Only available on date, datetime
'Mon Dec 30 21:27:03 2002'
>>> now.strftime('%Y %d %b')
'2002 30 Dec'

The ~datetime.datetime.replace method allows modifying one or more fields of a ~datetime.date or ~datetime.datetime instance, returning a new instance:

>>> d = dt.datetime.now()
>>> d
datetime.datetime(2002, 12, 30, 22, 15, 38, 827738)
>>> d.replace(year=2001, hour=12)
datetime.datetime(2001, 12, 30, 12, 15, 38, 827738)
>>>

Instances can be compared, hashed, and converted to strings (the result is the same as that of ~datetime.datetime.isoformat). ~datetime.date and ~datetime.datetime instances can be subtracted from each other, and added to ~datetime.timedelta instances. The largest missing feature is that there's no standard library support for parsing strings and getting back a ~datetime.date or ~datetime.datetime.

For more information, refer to the module's reference documentation. (Contributed by Tim Peters.)

The optparse Module

The getopt module provides simple parsing of command-line arguments. The new optparse module (originally named Optik) provides more elaborate command-line parsing that follows the Unix conventions, automatically creates the output for !--help, and can perform different actions for different options.

You start by creating an instance of ~optparse.OptionParser and telling it what your program's options are. :

import sys
from optparse import OptionParser

op = OptionParser()
op.add_option('-i', '--input',
              action='store', type='string', dest='input',
              help='set input filename')
op.add_option('-l', '--length',
              action='store', type='int', dest='length',
              help='set maximum length of output')

Parsing a command line is then done by calling the ~optparse.OptionParser.parse_args method. :

options, args = op.parse_args(sys.argv[1:])
print options
print args

This returns an object containing all of the option values, and a list of strings containing the remaining arguments.

Invoking the script with the various arguments now works as you'd expect it to. Note that the length argument is automatically converted to an integer.

$ ./python opt.py -i data arg1
<Values at 0x400cad4c: {'input': 'data', 'length': None}>
['arg1']
$ ./python opt.py --input=data --length=4
<Values at 0x400cad2c: {'input': 'data', 'length': 4}>
[]
$

The help message is automatically generated for you:

$ ./python opt.py --help
usage: opt.py [options]

options:
  -h, --help            show this help message and exit
  -iINPUT, --input=INPUT
                        set input filename
  -lLENGTH, --length=LENGTH
                        set maximum length of output
$

See the module's documentation for more details.

Optik was written by Greg Ward, with suggestions from the readers of the Getopt SIG.

Pymalloc: A Specialized Object Allocator

Pymalloc, a specialized object allocator written by Vladimir Marangozov, was a feature added to Python 2.1. Pymalloc is intended to be faster than the system malloc and to have less memory overhead for allocation patterns typical of Python programs. The allocator uses C's malloc function to get large pools of memory and then fulfills smaller memory requests from these pools.

In 2.1 and 2.2, pymalloc was an experimental feature and wasn't enabled by default; you had to explicitly enable it when compiling Python by providing the !--with-pymalloc option to the configure script. In 2.3, pymalloc has had further enhancements and is now enabled by default; you'll have to supply !--without-pymalloc to disable it.

This change is transparent to code written in Python; however, pymalloc may expose bugs in C extensions. Authors of C extension modules should test their code with pymalloc enabled, because some incorrect code may cause core dumps at runtime.

There's one particularly common error that causes problems. There are a number of memory allocation functions in Python's C API that have previously just been aliases for the C library's malloc and free, meaning that if you accidentally called mismatched functions the error wouldn't be noticeable. When the object allocator is enabled, these functions aren't aliases of malloc and free any more, and calling the wrong function to free memory may get you a core dump. For example, if memory was allocated using PyObject_Malloc, it has to be freed using PyObject_Free, not free. A few modules included with Python fell afoul of this and had to be fixed; doubtless there are more third-party modules that will have the same problem.

As part of this change, the confusing multiple interfaces for allocating memory have been consolidated down into two API families. Memory allocated with one family must not be manipulated with functions from the other family. There is one family for allocating chunks of memory and another family of functions specifically for allocating Python objects.

Thanks to lots of work by Tim Peters, pymalloc in 2.3 also provides debugging features to catch memory overwrites and doubled frees in both extension modules and in the interpreter itself. To enable this support, compile a debugging version of the Python interpreter by running configure with !--with-pydebug.

To aid extension writers, a header file Misc/pymemcompat.h is distributed with the source to Python 2.3 that allows Python extensions to use the 2.3 interfaces to memory allocation while compiling against any version of Python since 1.5.2. You would copy the file from Python's source distribution and bundle it with the source of your extension.

https://hg.python.org/cpython/file/default/Objects/obmalloc.c
For the full details of the pymalloc implementation, see the comments at the top of the file Objects/obmalloc.c in the Python source code. The above link points to the file within the python.org SVN browser.

Build and C API Changes

Changes to Python's build process and to the C API include:

Port-Specific Changes

Support for a port to IBM's OS/2 using the EMX runtime environment was merged into the main Python source tree. EMX is a POSIX emulation layer over the OS/2 system APIs. The Python port for EMX tries to support all the POSIX-like capability exposed by the EMX runtime, and mostly succeeds; !fork and ~fcntl.fcntl are restricted by the limitations of the underlying emulation layer. The standard OS/2 port, which uses IBM's Visual Age compiler, also gained support for case-sensitive import semantics as part of the integration of the EMX port into CVS. (Contributed by Andrew MacIntyre.)

On MacOS, most toolbox modules have been weaklinked to improve backward compatibility. This means that modules will no longer fail to load if a single routine is missing on the current OS version. Instead calling the missing routine will raise an exception. (Contributed by Jack Jansen.)

The RPM spec files, found in the Misc/RPM/ directory in the Python source distribution, were updated for 2.3. (Contributed by Sean Reifschneider.)

Other new platforms now supported by Python include AtheOS (http://www.atheos.cx/), GNU/Hurd, and OpenVMS.

Other Changes and Fixes

As usual, there were a bunch of other improvements and bugfixes scattered throughout the source tree. A search through the CVS change logs finds there were 523 patches applied and 514 bugs fixed between Python 2.2 and 2.3. Both figures are likely to be underestimates.

Some of the more notable changes are:

Porting to Python 2.3

This section lists previously described changes that may require changes to your code:

Acknowledgements

The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this article: Jeff Bauer, Simon Brunning, Brett Cannon, Michael Chermside, Andrew Dalke, Scott David Daniels, Fred L. Drake, Jr., David Fraser, Kelly Gerber, Raymond Hettinger, Michael Hudson, Chris Lambert, Detlef Lannert, Martin von Löwis, Andrew MacIntyre, Lalo Martins, Chad Netzer, Gustavo Niemeyer, Neal Norwitz, Hans Nowak, Chris Reedy, Francesco Ricciardi, Vinay Sajip, Neil Schemenauer, Roman Suzi, Jason Tishler, Just van Rossum.

What's New in Python 2.4

Author
A.M. Kuchling

This article explains the new features in Python 2.4.1, released on March 30, 2005.

Python 2.4 is a medium-sized release. It doesn't introduce as many changes as the radical Python 2.2, but introduces more features than the conservative 2.3 release. The most significant new language features are function decorators and generator expressions; most other changes are to the standard library.

According to the CVS change logs, there were 481 patches applied and 502 bugs fixed between Python 2.3 and 2.4. Both figures are likely to be underestimates.

This article doesn't attempt to provide a complete specification of every single new feature, but instead provides a brief introduction to each feature. For full details, you should refer to the documentation for Python 2.4, such as the Python Library Reference and the Python Reference Manual. Often you will be referred to the PEP for a particular new feature for explanations of the implementation and design rationale.

PEP 218: Built-In Set Objects

Python 2.3 introduced the !sets module. C implementations of set data types have now been added to the Python core as two new built-in types, set(iterable) and frozenset(iterable). They provide high speed operations for membership testing, for eliminating duplicates from sequences, and for mathematical operations like unions, intersections, differences, and symmetric differences. :

>>> a = set('abracadabra')              # form a set from a string
>>> 'z' in a                            # fast membership testing
False
>>> a                                   # unique letters in a
set(['a', 'r', 'b', 'c', 'd'])
>>> ''.join(a)                          # convert back into a string
'arbcd'

>>> b = set('alacazam')                 # form a second set
>>> a - b                               # letters in a but not in b
set(['r', 'd', 'b'])
>>> a | b                               # letters in either a or b
set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])
>>> a & b                               # letters in both a and b
set(['a', 'c'])
>>> a ^ b                               # letters in a or b but not both
set(['r', 'd', 'b', 'm', 'z', 'l'])

>>> a.add('z')                          # add a new element
>>> a.update('wxy')                     # add multiple new elements
>>> a
set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z'])
>>> a.remove('x')                       # take one element out
>>> a
set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z'])

The frozenset type is an immutable version of set. Since it is immutable and hashable, it may be used as a dictionary key or as a member of another set.

The !sets module remains in the standard library, and may be useful if you wish to subclass the !Set or !ImmutableSet classes. There are currently no plans to deprecate the module.

218 - Adding a Built-In Set Object Type
Originally proposed by Greg Wilson and ultimately implemented by Raymond Hettinger.

PEP 237: Unifying Long Integers and Integers

The lengthy transition process for this PEP, begun in Python 2.2, takes another step forward in Python 2.4. In 2.3, certain integer operations that would behave differently after int/long unification triggered FutureWarning warnings and returned values limited to 32 or 64 bits (depending on your platform). In 2.4, these expressions no longer produce a warning and instead produce a different result that's usually a long integer.

The problematic expressions are primarily left shifts and lengthy hexadecimal and octal constants. For example, 2 << 32 results in a warning in 2.3, evaluating to 0 on 32-bit platforms. In Python 2.4, this expression now returns the correct answer, 8589934592.

237 - Unifying Long Integers and Integers
Original PEP written by Moshe Zadka and GvR. The changes for 2.4 were implemented by Kalle Svensson.

PEP 289: Generator Expressions

The iterator feature introduced in Python 2.2 and the itertools module make it easier to write programs that loop through large data sets without having the entire data set in memory at one time. List comprehensions don't fit into this picture very well because they produce a Python list object containing all of the items. This unavoidably pulls all of the objects into memory, which can be a problem if your data set is very large. When trying to write a functionally styled program, it would be natural to write something like:

instead of :

for link in get_all_links():
    if link.followed:
        continue
    ...

The first form is more concise and perhaps more readable, but if you're dealing with a large number of link objects you'd have to write the second form to avoid having all link objects in memory at the same time.

Generator expressions work similarly to list comprehensions but don't materialize the entire list; instead they create a generator that will return elements one by one. The above example could be written as:

Generator expressions always have to be written inside parentheses, as in the above example. The parentheses signalling a function call also count, so if you want to create an iterator that will be immediately passed to a function you could write:

print sum(obj.count for obj in list_all_objects())

Generator expressions differ from list comprehensions in various small ways. Most notably, the loop variable (obj in the above example) is not accessible outside of the generator expression. List comprehensions leave the variable assigned to its last value; future versions of Python will change this, making list comprehensions match generator expressions in this respect.

289 - Generator Expressions
Proposed by Raymond Hettinger and implemented by Jiwon Seo with early efforts steered by Hye-Shik Chang.

PEP 292: Simpler String Substitutions

Some new classes in the standard library provide an alternative mechanism for substituting variables into strings; this style of substitution may be better for applications where untrained users need to edit templates.

The usual way of substituting variables by name is the % operator:

>>> '%(page)i: %(title)s' % {'page':2, 'title': 'The Best of Times'}
'2: The Best of Times'

When writing the template string, it can be easy to forget the i or s after the closing parenthesis. This isn't a big problem if the template is in a Python module, because you run the code, get an "Unsupported format character" ValueError, and fix the problem. However, consider an application such as Mailman where template strings or translations are being edited by users who aren't aware of the Python language. The format string's syntax is complicated to explain to such users, and if they make a mistake, it's difficult to provide helpful feedback to them.

PEP 292 adds a ~string.Template class to the string module that uses $ to indicate a substitution:

>>> import string
>>> t = string.Template('$page: $title')
>>> t.substitute({'page':2, 'title': 'The Best of Times'})
'2: The Best of Times'

If a key is missing from the dictionary, the ~string.Template.substitute method will raise a KeyError. There's also a ~string.Template.safe_substitute method that ignores missing keys:

>>> t = string.Template('$page: $title')
>>> t.safe_substitute({'page':3})
'3: $title'

292 - Simpler String Substitutions
Written and implemented by Barry Warsaw.

PEP 318: Decorators for Functions and Methods

Python 2.2 extended Python's object model by adding static methods and class methods, but it didn't extend Python's syntax to provide any new way of defining static or class methods. Instead, you had to write a def statement in the usual way, and pass the resulting method to a staticmethod or classmethod function that would wrap up the function as a method of the new type. Your code would look like this:

class C:
   def meth (cls):
       ...

   meth = classmethod(meth)   # Rebind name to wrapped-up class method

If the method was very long, it would be easy to miss or forget the classmethod invocation after the function body.

The intention was always to add some syntax to make such definitions more readable, but at the time of 2.2's release a good syntax was not obvious. Today a good syntax still isn't obvious but users are asking for easier access to the feature; a new syntactic feature has been added to meet this need.

The new feature is called "function decorators". The name comes from the idea that classmethod, staticmethod, and friends are storing additional information on a function object; they're decorating functions with more details.

The notation borrows from Java and uses the '@' character as an indicator. Using the new syntax, the example above would be written:

class C:

   @classmethod
   def meth (cls):
       ...

The @classmethod is shorthand for the meth=classmethod(meth) assignment. More generally, if you have the following:

@A
@B
@C
def f ():
    ...

It's equivalent to the following pre-decorator code:

def f(): ...
f = A(B(C(f)))

Decorators must come on the line before a function definition, one decorator per line, and can't be on the same line as the def statement, meaning that @A def f(): ... is illegal. You can only decorate function definitions, either at the module level or inside a class; you can't decorate class definitions.

A decorator is just a function that takes the function to be decorated as an argument and returns either the same function or some new object. The return value of the decorator need not be callable (though it typically is), unless further decorators will be applied to the result. It's easy to write your own decorators. The following simple example just sets an attribute on the function object:

>>> def deco(func):
...    func.attr = 'decorated'
...    return func
...
>>> @deco
... def f(): pass
...
>>> f
<function f at 0x402ef0d4>
>>> f.attr
'decorated'
>>>

As a slightly more realistic example, the following decorator checks that the supplied argument is an integer:

def require_int (func):
    def wrapper (arg):
        assert isinstance(arg, int)
        return func(arg)

    return wrapper

@require_int
def p1 (arg):
    print arg

@require_int
def p2(arg):
    print arg*2

An example in 318 contains a fancier version of this idea that lets you both specify the required type and check the returned type.

Decorator functions can take arguments. If arguments are supplied, your decorator function is called with only those arguments and must return a new decorator function; this function must take a single function and return a function, as previously described. In other words, @A @B @C(args) becomes:

def f(): ...
_deco = C(args)
f = A(B(_deco(f)))

Getting this right can be slightly brain-bending, but it's not too difficult.

318 - Decorators for Functions, Methods and Classes
Written by Kevin D. Smith, Jim Jewett, and Skip Montanaro. Several people wrote patches implementing function decorators, but the one that was actually checked in was patch #979728, written by Mark Russell.

https://wiki.python.org/moin/PythonDecoratorLibrary
This Wiki page contains several examples of decorators.

PEP 322: Reverse Iteration

A new built-in function, reversed(seq), takes a sequence and returns an iterator that loops over the elements of the sequence in reverse order. :

>>> for i in reversed(xrange(1,4)):
...    print i
...
3
2
1

Compared to extended slicing, such as range(1,4)[::-1], reversed is easier to read, runs faster, and uses substantially less memory.

Note that reversed only accepts sequences, not arbitrary iterators. If you want to reverse an iterator, first convert it to a list with list. :

>>> input = open('/etc/passwd', 'r')
>>> for line in reversed(list(input)):
...   print line
...
root:*:0:0:System Administrator:/var/root:/bin/tcsh
  ...

322 - Reverse Iteration
Written and implemented by Raymond Hettinger.

PEP 324: New subprocess Module

The standard library provides a number of ways to execute a subprocess, offering different features and different levels of complexity. os.system(command) is easy to use, but slow (it runs a shell process which executes the command) and dangerous (you have to be careful about escaping the shell's metacharacters). The !popen2 module offers classes that can capture standard output and standard error from the subprocess, but the naming is confusing. The subprocess module cleans this up, providing a unified interface that offers all the features you might need.

Instead of !popen2's collection of classes, subprocess contains a single class called subprocess.Popen whose constructor supports a number of different keyword arguments. :

class Popen(args, bufsize=0, executable=None,
            stdin=None, stdout=None, stderr=None,
            preexec_fn=None, close_fds=False, shell=False,
            cwd=None, env=None, universal_newlines=False,
            startupinfo=None, creationflags=0):

args is commonly a sequence of strings that will be the arguments to the program executed as the subprocess. (If the shell argument is true, args can be a string which will then be passed on to the shell for interpretation, just as os.system does.)

stdin, stdout, and stderr specify what the subprocess's input, output, and error streams will be. You can provide a file object or a file descriptor, or you can use the constant subprocess.PIPE to create a pipe between the subprocess and the parent.

single: universal newlines; What's new

The constructor has a number of handy options:

Once you've created the ~subprocess.Popen instance, you can call its ~subprocess.Popen.wait method to pause until the subprocess has exited, ~subprocess.Popen.poll to check if it's exited without pausing, or communicate(data) to send the string data to the subprocess's standard input. communicate(data) then reads any data that the subprocess has sent to its standard output or standard error, returning a tuple (stdout_data, stderr_data).

~subprocess.call is a shortcut that passes its arguments along to the ~subprocess.Popen constructor, waits for the command to complete, and returns the status code of the subprocess. It can serve as a safer analog to os.system:

sts = subprocess.call(['dpkg', '-i', '/tmp/new-package.deb'])
if sts == 0:
    # Success
    ...
else:
    # dpkg returned an error
    ...

The command is invoked without use of the shell. If you really do want to use the shell, you can add shell=True as a keyword argument and provide a string instead of a sequence:

sts = subprocess.call('dpkg -i /tmp/new-package.deb', shell=True)

The PEP takes various examples of shell and Python code and shows how they'd be translated into Python code that uses subprocess. Reading this section of the PEP is highly recommended.

324 - subprocess - New process module
Written and implemented by Peter Åstrand, with assistance from Fredrik Lundh and others.

PEP 327: Decimal Data Type

Python has always supported floating-point (FP) numbers, based on the underlying C double type, as a data type. However, while most programming languages provide a floating-point type, many people (even programmers) are unaware that floating-point numbers don't represent certain decimal fractions accurately. The new ~decimal.Decimal type can represent these fractions accurately, up to a user-specified precision limit.

Why is Decimal needed?

The limitations arise from the representation used for floating-point numbers. FP numbers are made up of three components:

For example, the number 1.25 has positive sign, a mantissa value of 1.01 (in binary), and an exponent of 0 (the decimal point doesn't need to be shifted). The number 5 has the same sign and mantissa, but the exponent is 2 because the mantissa is multiplied by 4 (2 to the power of the exponent 2); 1.25 * 4 equals 5.

Modern systems usually provide floating-point support that conforms to a standard called IEEE 754. C's double type is usually implemented as a 64-bit IEEE 754 number, which uses 52 bits of space for the mantissa. This means that numbers can only be specified to 52 bits of precision. If you're trying to represent numbers whose expansion repeats endlessly, the expansion is cut off after 52 bits. Unfortunately, most software needs to produce output in base 10, and common fractions in base 10 are often repeating decimals in binary. For example, 1.1 decimal is binary 1.0001100110011 ...; .1 = 1/16 + 1/32 + 1/256 plus an infinite number of additional terms. IEEE 754 has to chop off that infinitely repeated decimal after 52 digits, so the representation is slightly inaccurate.

Sometimes you can see this inaccuracy when the number is printed:

>>> 1.1
1.1000000000000001

The inaccuracy isn't always visible when you print the number because the FP-to-decimal-string conversion is provided by the C library, and most C libraries try to produce sensible output. Even if it's not displayed, however, the inaccuracy is still there and subsequent operations can magnify the error.

For many applications this doesn't matter. If I'm plotting points and displaying them on my monitor, the difference between 1.1 and 1.1000000000000001 is too small to be visible. Reports often limit output to a certain number of decimal places, and if you round the number to two or three or even eight decimal places, the error is never apparent. However, for applications where it does matter, it's a lot of work to implement your own custom arithmetic routines.

Hence, the ~decimal.Decimal type was created.

The ~decimal.Decimal type

A new module, decimal, was added to Python's standard library. It contains two classes, ~decimal.Decimal and ~decimal.Context. ~decimal.Decimal instances represent numbers, and ~decimal.Context instances are used to wrap up various settings such as the precision and default rounding mode.

~decimal.Decimal instances are immutable, like regular Python integers and FP numbers; once it's been created, you can't change the value an instance represents. ~decimal.Decimal instances can be created from integers or strings:

>>> import decimal
>>> decimal.Decimal(1972)
Decimal("1972")
>>> decimal.Decimal("1.1")
Decimal("1.1")

You can also provide tuples containing the sign, the mantissa represented as a tuple of decimal digits, and the exponent:

>>> decimal.Decimal((1, (1, 4, 7, 5), -2))
Decimal("-14.75")

Cautionary note: the sign bit is a Boolean value, so 0 is positive and 1 is negative.

Converting from floating-point numbers poses a bit of a problem: should the FP number representing 1.1 turn into the decimal number for exactly 1.1, or for 1.1 plus whatever inaccuracies are introduced? The decision was to dodge the issue and leave such a conversion out of the API. Instead, you should convert the floating-point number into a string using the desired precision and pass the string to the ~decimal.Decimal constructor:

>>> f = 1.1
>>> decimal.Decimal(str(f))
Decimal("1.1")
>>> decimal.Decimal('%.12f' % f)
Decimal("1.100000000000")

Once you have ~decimal.Decimal instances, you can perform the usual mathematical operations on them. One limitation: exponentiation requires an integer exponent:

>>> a = decimal.Decimal('35.72')
>>> b = decimal.Decimal('1.73')
>>> a+b
Decimal("37.45")
>>> a-b
Decimal("33.99")
>>> a*b
Decimal("61.7956")
>>> a/b
Decimal("20.64739884393063583815028902")
>>> a ** 2
Decimal("1275.9184")
>>> a**b
Traceback (most recent call last):
  ...
decimal.InvalidOperation: x ** (non-integer)

You can combine ~decimal.Decimal instances with integers, but not with floating-point numbers:

>>> a + 4
Decimal("39.72")
>>> a + 4.5
Traceback (most recent call last):
  ...
TypeError: You can interact Decimal only with int, long or Decimal data types.
>>>

~decimal.Decimal numbers can be used with the math and cmath modules, but note that they'll be immediately converted to floating-point numbers before the operation is performed, resulting in a possible loss of precision and accuracy. You'll also get back a regular floating-point number and not a ~decimal.Decimal. :

>>> import math, cmath
>>> d = decimal.Decimal('123456789012.345')
>>> math.sqrt(d)
351364.18288201344
>>> cmath.sqrt(-d)
351364.18288201344j

~decimal.Decimal instances have a ~decimal.Decimal.sqrt method that returns a ~decimal.Decimal, but if you need other things such as trigonometric functions you'll have to implement them. :

>>> d.sqrt()
Decimal("351364.1828820134592177245001")

The ~decimal.Context type

Instances of the ~decimal.Context class encapsulate several settings for decimal operations:

There's a thread-local default context available by calling ~decimal.getcontext; you can change the properties of this context to alter the default precision, rounding, or trap handling. The following example shows the effect of changing the precision of the default context:

>>> decimal.getcontext().prec
28
>>> decimal.Decimal(1) / decimal.Decimal(7)
Decimal("0.1428571428571428571428571429")
>>> decimal.getcontext().prec = 9
>>> decimal.Decimal(1) / decimal.Decimal(7)
Decimal("0.142857143")

The default action for error conditions is selectable; the module can either return a special value such as infinity or not-a-number, or exceptions can be raised:

>>> decimal.Decimal(1) / decimal.Decimal(0)
Traceback (most recent call last):
  ...
decimal.DivisionByZero: x / 0
>>> decimal.getcontext().traps[decimal.DivisionByZero] = False
>>> decimal.Decimal(1) / decimal.Decimal(0)
Decimal("Infinity")
>>>

The ~decimal.Context instance also has various methods for formatting numbers such as ~decimal.Context.to_eng_string and ~decimal.Context.to_sci_string.

For more information, see the documentation for the decimal module, which includes a quick-start tutorial and a reference.

327 - Decimal Data Type
Written by Facundo Batista and implemented by Facundo Batista, Eric Price, Raymond Hettinger, Aahz, and Tim Peters.

http://www.lahey.com/float.htm
The article uses Fortran code to illustrate many of the problems that floating-point inaccuracy can cause.

https://speleotrove.com/decimal/
A description of a decimal-based representation. This representation is being proposed as a standard, and underlies the new Python decimal type. Much of this material was written by Mike Cowlishaw, designer of the Rexx language.

PEP 328: Multi-line Imports

One language change is a small syntactic tweak aimed at making it easier to import many names from a module. In a from module import names statement, names is a sequence of names separated by commas. If the sequence is very long, you can either write multiple imports from the same module, or you can use backslashes to escape the line endings like this:

from SimpleXMLRPCServer import SimpleXMLRPCServer,\
            SimpleXMLRPCRequestHandler,\
            CGIXMLRPCRequestHandler,\
            resolve_dotted_attribute

The syntactic change in Python 2.4 simply allows putting the names within parentheses. Python ignores newlines within a parenthesized expression, so the backslashes are no longer needed:

from SimpleXMLRPCServer import (SimpleXMLRPCServer,
                                SimpleXMLRPCRequestHandler,
                                CGIXMLRPCRequestHandler,
                                resolve_dotted_attribute)

The PEP also proposes that all import statements be absolute imports, with a leading . character to indicate a relative import. This part of the PEP was not implemented for Python 2.4, but was completed for Python 2.5.

328 - Imports: Multi-Line and Absolute/Relative
Written by Aahz. Multi-line imports were implemented by Dima Dorfman.

PEP 331: Locale-Independent Float/String Conversions

The locale modules lets Python software select various conversions and display conventions that are localized to a particular country or language. However, the module was careful to not change the numeric locale because various functions in Python's implementation required that the numeric locale remain set to the 'C' locale. Often this was because the code was using the C library's !atof function.

Not setting the numeric locale caused trouble for extensions that used third-party C libraries, however, because they wouldn't have the correct locale set. The motivating example was GTK+, whose user interface widgets weren't displaying numbers in the current locale.

The solution described in the PEP is to add three new functions to the Python API that perform ASCII-only conversions, ignoring the locale setting:

The code for these functions came from the GLib library (https://developer-old.gnome.org/glib/2.26/), whose developers kindly relicensed the relevant functions and donated them to the Python Software Foundation. The locale module can now change the numeric locale, letting extensions such as GTK+ produce the correct results.

331 - Locale-Independent Float/String Conversions
Written by Christian R. Reis, and implemented by Gustavo Carneiro.

Other Language Changes

Here are all of the changes that Python 2.4 makes to the core Python language.

Optimizations

The net result of the 2.4 optimizations is that Python 2.4 runs the pystone benchmark around 5% faster than Python 2.3 and 35% faster than Python 2.2. (pystone is not a particularly good benchmark, but it's the most commonly used measurement of Python's performance. Your own applications may show greater or smaller benefits from Python 2.4.)

New, Improved, and Deprecated Modules

As usual, Python's standard library received a number of enhancements and bug fixes. Here's a partial list of the most notable changes, sorted alphabetically by module name. Consult the Misc/NEWS file in the source tree for a more complete list of changes, or look through the CVS logs for all the details.

cookielib

The cookielib <http.cookiejar> library supports client-side handling for HTTP cookies, mirroring the Cookie <http.cookies> module's server-side cookie support. Cookies are stored in cookie jars; the library transparently stores cookies offered by the web server in the cookie jar, and fetches the cookie from the jar when connecting to the server. As in web browsers, policy objects control whether cookies are accepted or not.

In order to store cookies across sessions, two implementations of cookie jars are provided: one that stores cookies in the Netscape format so applications can use the Mozilla or Lynx cookie files, and one that stores cookies in the same format as the Perl libwww library.

urllib2 <urllib.request> has been changed to interact with cookielib <http.cookiejar>: ~urllib.request.HTTPCookieProcessor manages a cookie jar that is used when accessing URLs.

This module was contributed by John J. Lee.

doctest

The doctest module underwent considerable refactoring thanks to Edward Loper and Tim Peters. Testing can still be as simple as running doctest.testmod, but the refactorings allow customizing the module's operation in various ways

The new ~doctest.DocTestFinder class extracts the tests from a given object's docstrings:

def f (x, y):
    """>>> f(2,2)
4
>>> f(3,2)
6
    """
    return x*y

finder = doctest.DocTestFinder()

# Get list of DocTest instances
tests = finder.find(f)

The new ~doctest.DocTestRunner class then runs individual tests and can produce a summary of the results:

runner = doctest.DocTestRunner()
for t in tests:
    tried, failed = runner.run(t)

runner.summarize(verbose=1)

The above example produces the following output:

1 items passed all tests:
   2 tests in f
2 tests in 1 items.
2 passed and 0 failed.
Test passed.

~doctest.DocTestRunner uses an instance of the ~doctest.OutputChecker class to compare the expected output with the actual output. This class takes a number of different flags that customize its behaviour; ambitious users can also write a completely new subclass of ~doctest.OutputChecker.

The default output checker provides a number of handy features. For example, with the doctest.ELLIPSIS option flag, an ellipsis (...) in the expected output matches any substring, making it easier to accommodate outputs that vary in minor ways:

def o (n):
    """>>> o(1)
<__main__.C instance at 0x...>
>>>
"""

Another special string, <BLANKLINE>, matches a blank line:

def p (n):
    """>>> p(1)
<BLANKLINE>
>>>
"""

Another new capability is producing a diff-style display of the output by specifying the doctest.REPORT_UDIFF (unified diffs), doctest.REPORT_CDIFF (context diffs), or doctest.REPORT_NDIFF (delta-style) option flags. For example:

def g (n):
    """>>> g(4)
here
is
a
lengthy
>>>"""
    L = 'here is a rather lengthy list of words'.split()
    for word in L[:n]:
        print word

Running the above function's tests with doctest.REPORT_UDIFF specified, you get the following output:

**********************************************************************
File "t.py", line 15, in g
Failed example:
    g(4)
Differences (unified diff with -expected +actual):
    @@ -2,3 +2,3 @@
     is
     a
    -lengthy
    +rather
**********************************************************************

Build and C API Changes

Some of the changes to Python's build process and to the C API are:

Port-Specific Changes

Porting to Python 2.4

This section lists previously described changes that may require changes to your code:

Acknowledgements

The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this article: Koray Can, Hye-Shik Chang, Michael Dyck, Raymond Hettinger, Brian Hurt, Hamish Lawson, Fredrik Lundh, Sean Reifschneider, Sadruddin Rejeb.

What's New in Python 2.5

Author
A.M. Kuchling

This article explains the new features in Python 2.5. The final release of Python 2.5 is scheduled for August 2006; 356 describes the planned release schedule. Python 2.5 was released on September 19, 2006.

The changes in Python 2.5 are an interesting mix of language and library improvements. The library enhancements will be more important to Python's user community, I think, because several widely useful packages were added. New modules include ElementTree for XML processing (xml.etree.ElementTree), the SQLite database module (sqlite3), and the ctypes module for calling C functions.

The language changes are of middling significance. Some pleasant new features were added, but most of them aren't features that you'll use every day. Conditional expressions were finally added to the language using a novel syntax; see section pep-308. The new 'with' statement will make writing cleanup code easier (section pep-343). Values can now be passed into generators (section pep-342). Imports are now visible as either absolute or relative (section pep-328). Some corner cases of exception handling are handled better (section pep-341). All these improvements are worthwhile, but they're improvements to one specific language feature or another; none of them are broad modifications to Python's semantics.

As well as the language and library additions, other improvements and bugfixes were made throughout the source tree. A search through the SVN change logs finds there were 353 patches applied and 458 bugs fixed between Python 2.4 and 2.5. (Both figures are likely to be underestimates.)

This article doesn't try to be a complete specification of the new features; instead changes are briefly introduced using helpful examples. For full details, you should always refer to the documentation for Python 2.5 at https://docs.python.org. If you want to understand the complete implementation and design rationale, refer to the PEP for a particular new feature.

Comments, suggestions, and error reports for this document are welcome; please e-mail them to the author or open a bug in the Python bug tracker.

PEP 308: Conditional Expressions

For a long time, people have been requesting a way to write conditional expressions, which are expressions that return value A or value B depending on whether a Boolean value is true or false. A conditional expression lets you write a single assignment statement that has the same effect as the following:

if condition:
    x = true_value
else:
    x = false_value

There have been endless tedious discussions of syntax on both python-dev and comp.lang.python. A vote was even held that found the majority of voters wanted conditional expressions in some form, but there was no syntax that was preferred by a clear majority. Candidates included C's cond ? true_v : false_v, if cond then true_v else false_v, and 16 other variations.

Guido van Rossum eventually chose a surprising syntax:

x = true_value if condition else false_value

Evaluation is still lazy as in existing Boolean expressions, so the order of evaluation jumps around a bit. The condition expression in the middle is evaluated first, and the true_value expression is evaluated only if the condition was true. Similarly, the false_value expression is only evaluated when the condition is false.

This syntax may seem strange and backwards; why does the condition go in the middle of the expression, and not in the front as in C's c ? x : y? The decision was checked by applying the new syntax to the modules in the standard library and seeing how the resulting code read. In many cases where a conditional expression is used, one value seems to be the 'common case' and one value is an 'exceptional case', used only on rarer occasions when the condition isn't met. The conditional syntax makes this pattern a bit more obvious:

contents = ((doc + '\n') if doc else '')

I read the above statement as meaning "here contents is usually assigned a value of doc+'\n'; sometimes doc is empty, in which special case an empty string is returned." I doubt I will use conditional expressions very often where there isn't a clear common and uncommon case.

There was some discussion of whether the language should require surrounding conditional expressions with parentheses. The decision was made to not require parentheses in the Python language's grammar, but as a matter of style I think you should always use them. Consider these two statements:

# First version -- no parens
level = 1 if logging else 0

# Second version -- with parens
level = (1 if logging else 0)

In the first version, I think a reader's eye might group the statement into 'level = 1', 'if logging', 'else 0', and think that the condition decides whether the assignment to level is performed. The second version reads better, in my opinion, because it makes it clear that the assignment is always performed and the choice is being made between two values.

Another reason for including the brackets: a few odd combinations of list comprehensions and lambdas could look like incorrect conditional expressions. See 308 for some examples. If you put parentheses around your conditional expressions, you won't run into this case.

308 - Conditional Expressions
PEP written by Guido van Rossum and Raymond D. Hettinger; implemented by Thomas Wouters.

PEP 309: Partial Function Application

The functools module is intended to contain tools for functional-style programming.

One useful tool in this module is the ~functools.partial function. For programs written in a functional style, you'll sometimes want to construct variants of existing functions that have some of the parameters filled in. Consider a Python function f(a, b, c); you could create a new function g(b, c) that was equivalent to f(1, b, c). This is called "partial function application".

~functools.partial takes the arguments (function, arg1, arg2, ... kwarg1=value1, kwarg2=value2). The resulting object is callable, so you can just call it to invoke function with the filled-in arguments.

Here's a small but realistic example:

import functools

def log (message, subsystem):
    "Write the contents of 'message' to the specified subsystem."
    print '%s: %s' % (subsystem, message)
    ...

server_log = functools.partial(log, subsystem='server')
server_log('Unable to open socket')

Here's another example, from a program that uses PyGTK. Here a context-sensitive pop-up menu is being constructed dynamically. The callback provided for the menu option is a partially applied version of the open_item() method, where the first argument has been provided. :

...
class Application:
    def open_item(self, path):
       ...
    def init (self):
        open_func = functools.partial(self.open_item, item_path)
        popup_menu.append( ("Open", open_func, 1) )

Another function in the functools module is the update_wrapper(wrapper, wrapped) function that helps you write well-behaved decorators. ~functools.update_wrapper copies the name, module, and docstring attribute to a wrapper function so that tracebacks inside the wrapped function are easier to understand. For example, you might write:

def my_decorator(f):
    def wrapper(*args, **kwds):
        print 'Calling decorated function'
        return f(*args, **kwds)
    functools.update_wrapper(wrapper, f)
    return wrapper

~functools.wraps is a decorator that can be used inside your own decorators to copy the wrapped function's information. An alternate version of the previous example would be:

def my_decorator(f):
    @functools.wraps(f)
    def wrapper(*args, **kwds):
        print 'Calling decorated function'
        return f(*args, **kwds)
    return wrapper

309 - Partial Function Application
PEP proposed and written by Peter Harris; implemented by Hye-Shik Chang and Nick Coghlan, with adaptations by Raymond Hettinger.

PEP 314: Metadata for Python Software Packages v1.1

Some simple dependency support was added to Distutils. The !setup function now has requires, provides, and obsoletes keyword parameters. When you build a source distribution using the sdist command, the dependency information will be recorded in the PKG-INFO file.

Another new keyword parameter is download_url, which should be set to a URL for the package's source code. This means it's now possible to look up an entry in the package index, determine the dependencies for a package, and download the required packages. :

VERSION = '1.0'
setup(name='PyPackage',
      version=VERSION,
      requires=['numarray', 'zlib (>=1.1.4)'],
      obsoletes=['OldPackage']
      download_url=('http://www.example.com/pypackage/dist/pkg-%s.tar.gz'
                    % VERSION),
     )

Another new enhancement to the Python package index at https://pypi.org is storing source and binary archives for a package. The new upload Distutils command will upload a package to the repository.

Before a package can be uploaded, you must be able to build a distribution using the sdist Distutils command. Once that works, you can run python setup.py upload to add your package to the PyPI archive. Optionally you can GPG-sign the package by supplying the !--sign and !--identity options.

Package uploading was implemented by Martin von Löwis and Richard Jones.

314 - Metadata for Python Software Packages v1.1
PEP proposed and written by A.M. Kuchling, Richard Jones, and Fred Drake; implemented by Richard Jones and Fred Drake.

PEP 328: Absolute and Relative Imports

The simpler part of 328 was implemented in Python 2.4: parentheses could now be used to enclose the names imported from a module using the from ... import ... statement, making it easier to import many different names.

The more complicated part has been implemented in Python 2.5: importing a module can be specified to use absolute or package-relative imports. The plan is to move toward making absolute imports the default in future versions of Python.

Let's say you have a package directory like this:

pkg/
pkg/__init__.py
pkg/main.py
pkg/string.py

This defines a package named pkg containing the pkg.main and pkg.string submodules.

Consider the code in the main.py module. What happens if it executes the statement import string? In Python 2.4 and earlier, it will first look in the package's directory to perform a relative import, finds pkg/string.py, imports the contents of that file as the pkg.string module, and that module is bound to the name string in the pkg.main module's namespace.

That's fine if pkg.string was what you wanted. But what if you wanted Python's standard string module? There's no clean way to ignore pkg.string and look for the standard module; generally you had to look at the contents of sys.modules, which is slightly unclean. Holger Krekel's !py.std package provides a tidier way to perform imports from the standard library, import py; py.std.string.join(), but that package isn't available on all Python installations.

Reading code which relies on relative imports is also less clear, because a reader may be confused about which module, string or pkg.string, is intended to be used. Python users soon learned not to duplicate the names of standard library modules in the names of their packages' submodules, but you can't protect against having your submodule's name being used for a new module added in a future version of Python.

In Python 2.5, you can switch import's behaviour to absolute imports using a from __future__ import absolute_import directive. This absolute-import behaviour will become the default in a future version (probably Python 2.7). Once absolute imports are the default, import string will always find the standard library's version. It's suggested that users should begin using absolute imports as much as possible, so it's preferable to begin writing from pkg import string in your code.

Relative imports are still possible by adding a leading period to the module name when using the from ... import form:

# Import names from pkg.string
from .string import name1, name2
# Import pkg.string
from . import string

This imports the string module relative to the current package, so in pkg.main this will import name1 and name2 from pkg.string. Additional leading periods perform the relative import starting from the parent of the current package. For example, code in the A.B.C module can do:

from . import D                 # Imports A.B.D
from .. import E                # Imports A.E
from ..F import G               # Imports A.F.G

Leading periods cannot be used with the import modname form of the import statement, only the from ... import form.

328 - Imports: Multi-Line and Absolute/Relative
PEP written by Aahz; implemented by Thomas Wouters.

https://pylib.readthedocs.io/
The py library by Holger Krekel, which contains the !py.std package.

PEP 338: Executing Modules as Scripts

The -m switch added in Python 2.4 to execute a module as a script gained a few more abilities. Instead of being implemented in C code inside the Python interpreter, the switch now uses an implementation in a new module, runpy.

The runpy module implements a more sophisticated import mechanism so that it's now possible to run modules in a package such as !pychecker.checker. The module also supports alternative import mechanisms such as the zipimport module. This means you can add a .zip archive's path to sys.path and then use the -m switch to execute code from the archive.

338 - Executing modules as scripts
PEP written and implemented by Nick Coghlan.

PEP 341: Unified try/except/finally

Until Python 2.5, the try statement came in two flavours. You could use a finally block to ensure that code is always executed, or one or more except blocks to catch specific exceptions. You couldn't combine both !except blocks and a !finally block, because generating the right bytecode for the combined version was complicated and it wasn't clear what the semantics of the combined statement should be.

Guido van Rossum spent some time working with Java, which does support the equivalent of combining except blocks and a finally block, and this clarified what the statement should mean. In Python 2.5, you can now write:

try:
    block-1 ...
except Exception1:
    handler-1 ...
except Exception2:
    handler-2 ...
else:
    else-block
finally:
    final-block

The code in block-1 is executed. If the code raises an exception, the various except blocks are tested: if the exception is of class Exception1, handler-1 is executed; otherwise if it's of class Exception2, handler-2 is executed, and so forth. If no exception is raised, the else-block is executed.

No matter what happened previously, the final-block is executed once the code block is complete and any raised exceptions handled. Even if there's an error in an exception handler or the else-block and a new exception is raised, the code in the final-block is still run.

341 - Unifying try-except and try-finally
PEP written by Georg Brandl; implementation by Thomas Lee.

PEP 342: New Generator Features

Python 2.5 adds a simple way to pass values into a generator. As introduced in Python 2.3, generators only produce output; once a generator's code was invoked to create an iterator, there was no way to pass any new information into the function when its execution is resumed. Sometimes the ability to pass in some information would be useful. Hackish solutions to this include making the generator's code look at a global variable and then changing the global variable's value, or passing in some mutable object that callers then modify.

To refresh your memory of basic generators, here's a simple example:

def counter (maximum):
    i = 0
    while i < maximum:
        yield i
        i += 1

When you call counter(10), the result is an iterator that returns the values from 0 up to 9. On encountering the yield statement, the iterator returns the provided value and suspends the function's execution, preserving the local variables. Execution resumes on the following call to the iterator's next method, picking up after the !yield statement.

In Python 2.3, yield was a statement; it didn't return any value. In 2.5, !yield is now an expression, returning a value that can be assigned to a variable or otherwise operated on:

val = (yield i)

I recommend that you always put parentheses around a yield expression when you're doing something with the returned value, as in the above example. The parentheses aren't always necessary, but it's easier to always add them instead of having to remember when they're needed.

(342 explains the exact rules, which are that a yield-expression must always be parenthesized except when it occurs at the top-level expression on the right-hand side of an assignment. This means you can write val = yield i but have to use parentheses when there's an operation, as in val = (yield i) + 12.)

Values are sent into a generator by calling its send(value) method. The generator's code is then resumed and the yield expression returns the specified value. If the regular next method is called, the !yield returns None.

Here's the previous example, modified to allow changing the value of the internal counter. :

def counter (maximum):
    i = 0
    while i < maximum:
        val = (yield i)
        # If value provided, change counter
        if val is not None:
            i = val
        else:
            i += 1

And here's an example of changing the counter:

>>> it = counter(10)
>>> print it.next()
0
>>> print it.next()
1
>>> print it.send(8)
8
>>> print it.next()
9
>>> print it.next()
Traceback (most recent call last):
  File "t.py", line 15, in ?
    print it.next()
StopIteration

yield will usually return None, so you should always check for this case. Don't just use its value in expressions unless you're sure that the ~generator.send method will be the only method used to resume your generator function.

In addition to ~generator.send, there are two other new methods on generators:

The cumulative effect of these changes is to turn generators from one-way producers of information into both producers and consumers.

Generators also become coroutines, a more generalized form of subroutines. Subroutines are entered at one point and exited at another point (the top of the function, and a return statement), but coroutines can be entered, exited, and resumed at many different points (the yield statements). We'll have to figure out patterns for using coroutines effectively in Python.

The addition of the ~generator.close method has one side effect that isn't obvious. ~generator.close is called when a generator is garbage-collected, so this means the generator's code gets one last chance to run before the generator is destroyed. This last chance means that try...finally statements in generators can now be guaranteed to work; the finally clause will now always get a chance to run. The syntactic restriction that you couldn't mix yield statements with a try...finally suite has therefore been removed. This seems like a minor bit of language trivia, but using generators and try...finally is actually necessary in order to implement the with statement described by 343. I'll look at this new statement in the following section.

Another even more esoteric effect of this change: previously, the !gi_frame attribute of a generator was always a frame object. It's now possible for !gi_frame to be None once the generator has been exhausted.

342 - Coroutines via Enhanced Generators
PEP written by Guido van Rossum and Phillip J. Eby; implemented by Phillip J. Eby. Includes examples of some fancier uses of generators as coroutines.

Earlier versions of these features were proposed in 288 by Raymond Hettinger and 325 by Samuele Pedroni.

https://en.wikipedia.org/wiki/Coroutine
The Wikipedia entry for coroutines.

https://web.archive.org/web/20160321211320/http://www.sidhe.org/~dan/blog/archives/000178.html
An explanation of coroutines from a Perl point of view, written by Dan Sugalski.

PEP 343: The 'with' statement

The 'with' statement clarifies code that previously would use try...finally blocks to ensure that clean-up code is executed. In this section, I'll discuss the statement as it will commonly be used. In the next section, I'll examine the implementation details and show how to write objects for use with this statement.

The 'with' statement is a new control-flow structure whose basic structure is:

with expression [as variable]:
    with-block

The expression is evaluated, and it should result in an object that supports the context management protocol (that is, has ~object.__enter__ and ~object.__exit__ methods.

The object's ~object.__enter__ is called before with-block is executed and therefore can run set-up code. It also may return a value that is bound to the name variable, if given. (Note carefully that variable is not assigned the result of expression.)

After execution of the with-block is finished, the object's ~object.__exit__ method is called, even if the block raised an exception, and can therefore run clean-up code.

To enable the statement in Python 2.5, you need to add the following directive to your module:

from __future__ import with_statement

The statement will always be enabled in Python 2.6.

Some standard Python objects now support the context management protocol and can be used with the 'with' statement. File objects are one example:

with open('/etc/passwd', 'r') as f:
    for line in f:
        print line
        ... more processing code ...

After this statement has executed, the file object in f will have been automatically closed, even if the for loop raised an exception part-way through the block.

[!NOTE] In this case, f is the same object created by open, because ~object.__enter__ returns self.

The threading module's locks and condition variables also support the 'with' statement:

lock = threading.Lock()
with lock:
    # Critical section of code
    ...

The lock is acquired before the block is executed and always released once the block is complete.

The new ~decimal.localcontext function in the decimal module makes it easy to save and restore the current decimal context, which encapsulates the desired precision and rounding characteristics for computations:

from decimal import Decimal, Context, localcontext

# Displays with default precision of 28 digits
v = Decimal('578')
print v.sqrt()

with localcontext(Context(prec=16)):
    # All code in this block uses a precision of 16 digits.
    # The original context is restored on exiting the block.
    print v.sqrt()

Writing Context Managers

Under the hood, the 'with' statement is fairly complicated. Most people will only use '!with' in company with existing objects and don't need to know these details, so you can skip the rest of this section if you like. Authors of new objects will need to understand the details of the underlying implementation and should keep reading.

A high-level explanation of the context management protocol is:

Let's think through an example. I won't present detailed code but will only sketch the methods necessary for a database that supports transactions.

(For people unfamiliar with database terminology: a set of changes to the database are grouped into a transaction. Transactions can be either committed, meaning that all the changes are written into the database, or rolled back, meaning that the changes are all discarded and the database is unchanged. See any database textbook for more information.)

Let's assume there's an object representing a database connection. Our goal will be to let the user write code like this:

db_connection = DatabaseConnection()
with db_connection as cursor:
    cursor.execute('insert into ...')
    cursor.execute('delete from ...')
    # ... more operations ...

The transaction should be committed if the code in the block runs flawlessly or rolled back if there's an exception. Here's the basic interface for DatabaseConnection that I'll assume:

class DatabaseConnection:
    # Database interface
    def cursor (self):
        "Returns a cursor object and starts a new transaction"
    def commit (self):
        "Commits current transaction"
    def rollback (self):
        "Rolls back current transaction"

The ~object.__enter__ method is pretty easy, having only to start a new transaction. For this application the resulting cursor object would be a useful result, so the method will return it. The user can then add as cursor to their 'with' statement to bind the cursor to a variable name. :

class DatabaseConnection:
    ...
    def __enter__ (self):
        # Code to start a new transaction
        cursor = self.cursor()
        return cursor

The ~object.__exit__ method is the most complicated because it's where most of the work has to be done. The method has to check if an exception occurred. If there was no exception, the transaction is committed. The transaction is rolled back if there was an exception.

In the code below, execution will just fall off the end of the function, returning the default value of None. None is false, so the exception will be re-raised automatically. If you wished, you could be more explicit and add a return statement at the marked location. :

class DatabaseConnection:
    ...
    def __exit__ (self, type, value, tb):
        if tb is None:
            # No exception, so commit
            self.commit()
        else:
            # Exception occurred, so rollback.
            self.rollback()
            # return False

The contextlib module

The new contextlib module provides some functions and a decorator that are useful for writing objects for use with the 'with' statement.

The decorator is called ~contextlib.contextmanager, and lets you write a single generator function instead of defining a new class. The generator should yield exactly one value. The code up to the yield will be executed as the ~object.__enter__ method, and the value yielded will be the method's return value that will get bound to the variable in the 'with' statement's !as clause, if any. The code after the yield will be executed in the ~object.__exit__ method. Any exception raised in the block will be raised by the !yield statement.

Our database example from the previous section could be written using this decorator as:

from contextlib import contextmanager

@contextmanager
def db_transaction (connection):
    cursor = connection.cursor()
    try:
        yield cursor
    except:
        connection.rollback()
        raise
    else:
        connection.commit()

db = DatabaseConnection()
with db_transaction(db) as cursor:
    ...

The contextlib module also has a nested(mgr1, mgr2, ...) function that combines a number of context managers so you don't need to write nested 'with' statements. In this example, the single '!with' statement both starts a database transaction and acquires a thread lock:

lock = threading.Lock()
with nested (db_transaction(db), lock) as (cursor, locked):
    ...

Finally, the closing(object) function returns object so that it can be bound to a variable, and calls object.close at the end of the block. :

import urllib, sys
from contextlib import closing

with closing(urllib.urlopen('http://www.yahoo.com')) as f:
    for line in f:
        sys.stdout.write(line)

343 - The "with" statement
PEP written by Guido van Rossum and Nick Coghlan; implemented by Mike Bland, Guido van Rossum, and Neal Norwitz. The PEP shows the code generated for a 'with' statement, which can be helpful in learning how the statement works.

The documentation for the contextlib module.

PEP 352: Exceptions as New-Style Classes

Exception classes can now be new-style classes, not just classic classes, and the built-in Exception class and all the standard built-in exceptions (NameError, ValueError, etc.) are now new-style classes.

The inheritance hierarchy for exceptions has been rearranged a bit. In 2.5, the inheritance relationships are:

BaseException       # New in Python 2.5
|- KeyboardInterrupt
|- SystemExit
|- Exception
   |- (all other current built-in exceptions)

This rearrangement was done because people often want to catch all exceptions that indicate program errors. KeyboardInterrupt and SystemExit aren't errors, though, and usually represent an explicit action such as the user hitting Control-C or code calling sys.exit. A bare except: will catch all exceptions, so you commonly need to list KeyboardInterrupt and SystemExit in order to re-raise them. The usual pattern is:

try:
    ...
except (KeyboardInterrupt, SystemExit):
    raise
except:
    # Log error...
    # Continue running program...

In Python 2.5, you can now write except Exception to achieve the same result, catching all the exceptions that usually indicate errors but leaving KeyboardInterrupt and SystemExit alone. As in previous versions, a bare except: still catches all exceptions.

The goal for Python 3.0 is to require any class raised as an exception to derive from BaseException or some descendant of BaseException, and future releases in the Python 2.x series may begin to enforce this constraint. Therefore, I suggest you begin making all your exception classes derive from Exception now. It's been suggested that the bare except: form should be removed in Python 3.0, but Guido van Rossum hasn't decided whether to do this or not.

Raising of strings as exceptions, as in the statement raise "Error occurred", is deprecated in Python 2.5 and will trigger a warning. The aim is to be able to remove the string-exception feature in a few releases.

352 - Required Superclass for Exceptions
PEP written by Brett Cannon and Guido van Rossum; implemented by Brett Cannon.

PEP 353: Using ssize_t as the index type

A wide-ranging change to Python's C API, using a new Py_ssize_t type definition instead of int, will permit the interpreter to handle more data on 64-bit platforms. This change doesn't affect Python's capacity on 32-bit platforms.

Various pieces of the Python interpreter used C's int type to store sizes or counts; for example, the number of items in a list or tuple were stored in an int. The C compilers for most 64-bit platforms still define int as a 32-bit type, so that meant that lists could only hold up to 2**31 - 1 = 2147483647 items. (There are actually a few different programming models that 64-bit C compilers can use -- see https://unix.org/version2/whatsnew/lp64_wp.html for a discussion -- but the most commonly available model leaves int as 32 bits.)

A limit of 2147483647 items doesn't really matter on a 32-bit platform because you'll run out of memory before hitting the length limit. Each list item requires space for a pointer, which is 4 bytes, plus space for a PyObject representing the item. 2147483647*4 is already more bytes than a 32-bit address space can contain.

It's possible to address that much memory on a 64-bit platform, however. The pointers for a list that size would only require 16 GiB of space, so it's not unreasonable that Python programmers might construct lists that large. Therefore, the Python interpreter had to be changed to use some type other than int, and this will be a 64-bit type on 64-bit platforms. The change will cause incompatibilities on 64-bit machines, so it was deemed worth making the transition now, while the number of 64-bit users is still relatively small. (In 5 or 10 years, we may all be on 64-bit machines, and the transition would be more painful then.)

This change most strongly affects authors of C extension modules. Python strings and container types such as lists and tuples now use Py_ssize_t to store their size. Functions such as PyList_Size now return Py_ssize_t. Code in extension modules may therefore need to have some variables changed to Py_ssize_t.

The PyArg_ParseTuple and Py_BuildValue functions have a new conversion code, n, for Py_ssize_t. PyArg_ParseTuple's s# and t# still output int by default, but you can define the macro !PY_SSIZE_T_CLEAN before including Python.h to make them return Py_ssize_t.

353 has a section on conversion guidelines that extension authors should read to learn about supporting 64-bit platforms.

353 - Using ssize_t as the index type
PEP written and implemented by Martin von Löwis.

PEP 357: The '__index__' method

The NumPy developers had a problem that could only be solved by adding a new special method, ~object.__index__. When using slice notation, as in [start:stop:step], the values of the start, stop, and step indexes must all be either integers or long integers. NumPy defines a variety of specialized integer types corresponding to unsigned and signed integers of 8, 16, 32, and 64 bits, but there was no way to signal that these types could be used as slice indexes.

Slicing can't just use the existing ~object.__int__ method because that method is also used to implement coercion to integers. If slicing used ~object.__int__, floating-point numbers would also become legal slice indexes and that's clearly an undesirable behaviour.

Instead, a new special method called ~object.__index__ was added. It takes no arguments and returns an integer giving the slice index to use. For example:

class C:
    def __index__ (self):
        return self.value

The return value must be either a Python integer or long integer. The interpreter will check that the type returned is correct, and raises a TypeError if this requirement isn't met.

A corresponding ~PyNumberMethods.nb_index slot was added to the C-level PyNumberMethods structure to let C extensions implement this protocol. PyNumber_Index(obj) can be used in extension code to call the ~object.__index__ function and retrieve its result.

357 - Allowing Any Object to be Used for Slicing
PEP written and implemented by Travis Oliphant.

Other Language Changes

Here are all of the changes that Python 2.5 makes to the core Python language.

Interactive Interpreter Changes

In the interactive interpreter, quit and exit have long been strings so that new users get a somewhat helpful message when they try to quit:

>>> quit
'Use Ctrl-D (i.e. EOF) to exit.'

In Python 2.5, quit and exit are now objects that still produce string representations of themselves, but are also callable. Newbies who try quit() or exit() will now exit the interpreter as they expect. (Implemented by Georg Brandl.)

The Python executable now accepts the standard long options --help and --version; on Windows, it also accepts the /? <-?> option for displaying a help message. (Implemented by Georg Brandl.)

Optimizations

Several of the optimizations were developed at the NeedForSpeed sprint, an event held in Reykjavik, Iceland, from May 21--28 2006. The sprint focused on speed enhancements to the CPython implementation and was funded by EWT LLC with local support from CCP Games. Those optimizations added at this sprint are specially marked in the following list.

New, Improved, and Removed Modules

The standard library received many enhancements and bug fixes in Python 2.5. Here's a partial list of the most notable changes, sorted alphabetically by module name. Consult the Misc/NEWS file in the source tree for a more complete list of changes, or look through the SVN logs for all the details.

The ctypes package

The ctypes package, written by Thomas Heller, has been added to the standard library. ctypes lets you call arbitrary functions in shared libraries or DLLs. Long-time users may remember the !dl module, which provides functions for loading shared libraries and calling functions in them. The ctypes package is much fancier.

To load a shared library or DLL, you must create an instance of the ~ctypes.CDLL class and provide the name or path of the shared library or DLL. Once that's done, you can call arbitrary functions by accessing them as attributes of the ~ctypes.CDLL object. :

import ctypes

libc = ctypes.CDLL('libc.so.6')
result = libc.printf("Line of output\n")

Type constructors for the various C types are provided: ~ctypes.c_int, ~ctypes.c_float, ~ctypes.c_double, ~ctypes.c_char_p (equivalent to char \*), and so forth. Unlike Python's types, the C versions are all mutable; you can assign to their ~ctypes._SimpleCData.value attribute to change the wrapped value. Python integers and strings will be automatically converted to the corresponding C types, but for other types you must call the correct type constructor. (And I mean must; getting it wrong will often result in the interpreter crashing with a segmentation fault.)

You shouldn't use ~ctypes.c_char_p with a Python string when the C function will be modifying the memory area, because Python strings are supposed to be immutable; breaking this rule will cause puzzling bugs. When you need a modifiable memory area, use ~ctypes.create_string_buffer:

s = "this is a string"
buf = ctypes.create_string_buffer(s)
libc.strfry(buf)

C functions are assumed to return integers, but you can set the ~ctypes._CFuncPtr.restype attribute of the function object to change this:

>>> libc.atof('2.71828')
-1783957616
>>> libc.atof.restype = ctypes.c_double
>>> libc.atof('2.71828')
2.71828

ctypes also provides a wrapper for Python's C API as the ctypes.pythonapi object. This object does not release the global interpreter lock before calling a function, because the lock must be held when calling into the interpreter's code. There's a ~ctypes.py_object type constructor that will create a PyObject * pointer. A simple usage:

import ctypes

d = {}
ctypes.pythonapi.PyObject_SetItem(ctypes.py_object(d),
          ctypes.py_object("abc"),  ctypes.py_object(1))
# d is now {'abc', 1}.

Don't forget to use ~ctypes.py_object; if it's omitted you end up with a segmentation fault.

ctypes has been around for a while, but people still write and distribution hand-coded extension modules because you can't rely on ctypes being present. Perhaps developers will begin to write Python wrappers atop a library accessed through ctypes instead of extension modules, now that ctypes is included with core Python.

https://web.archive.org/web/20180410025338/http://starship.python.net/crew/theller/ctypes/
The pre-stdlib ctypes web page, with a tutorial, reference, and FAQ.

The documentation for the ctypes module.

The ElementTree package

A subset of Fredrik Lundh's ElementTree library for processing XML has been added to the standard library as !xml.etree. The available modules are !ElementTree, !ElementPath, and !ElementInclude from ElementTree 1.2.6. The !cElementTree accelerator module is also included.

The rest of this section will provide a brief overview of using ElementTree. Full documentation for ElementTree is available at https://web.archive.org/web/20201124024954/http://effbot.org/zone/element-index.htm.

ElementTree represents an XML document as a tree of element nodes. The text content of the document is stored as the ~xml.etree.ElementTree.Element.text and ~xml.etree.ElementTree.Element.tail attributes of (This is one of the major differences between ElementTree and the Document Object Model; in the DOM there are many different types of node, including !TextNode.)

The most commonly used parsing function is ~xml.etree.ElementTree.parse, that takes either a string (assumed to contain a filename) or a file-like object and returns an ~xml.etree.ElementTree.ElementTree instance:

from xml.etree import ElementTree as ET

tree = ET.parse('ex-1.xml')

feed = urllib.urlopen(
          'http://planet.python.org/rss10.xml')
tree = ET.parse(feed)

Once you have an ~xml.etree.ElementTree.ElementTree instance, you can call its ~xml.etree.ElementTree.ElementTree.getroot method to get the root ~xml.etree.ElementTree.Element node.

There's also an ~xml.etree.ElementTree.XML function that takes a string literal and returns an ~xml.etree.ElementTree.Element node (not an ~xml.etree.ElementTree.ElementTree). This function provides a tidy way to incorporate XML fragments, approaching the convenience of an XML literal:

svg = ET.XML("""<svg width="10px" version="1.0">
             </svg>""")
svg.set('height', '320px')
svg.append(elem1)

Each XML element supports some dictionary-like and some list-like access methods. Dictionary-like operations are used to access attribute values, and list-like operations are used to access child nodes.

Operation Result
elem[n] Returns n'th child element.
elem[m:n] Returns list of m'th through n'th child elements.
len(elem) Returns number of child elements.
list(elem) Returns list of child elements.
elem.append(elem2) Adds elem2 as a child.
elem.insert(index, elem2) Inserts elem2 at the specified location.
del elem[n] Deletes n'th child element.
elem.keys() Returns list of attribute names.
elem.get(name) Returns value of attribute name.
elem.set(name, value) Sets new value for attribute name.
elem.attrib Retrieves the dictionary containing attributes.
del elem.attrib[name] Deletes attribute name.

Comments and processing instructions are also represented as ~xml.etree.ElementTree.Element nodes. To check if a node is a comment or processing instructions:

if elem.tag is ET.Comment:
    ...
elif elem.tag is ET.ProcessingInstruction:
    ...

To generate XML output, you should call the xml.etree.ElementTree.ElementTree.write method. Like ~xml.etree.ElementTree.parse, it can take either a string or a file-like object:

# Encoding is US-ASCII
tree.write('output.xml')

# Encoding is UTF-8
f = open('output.xml', 'w')
tree.write(f, encoding='utf-8')

(Caution: the default encoding used for output is ASCII. For general XML work, where an element's name may contain arbitrary Unicode characters, ASCII isn't a very useful encoding because it will raise an exception if an element's name contains any characters with values greater than 127. Therefore, it's best to specify a different encoding such as UTF-8 that can handle any Unicode character.)

This section is only a partial description of the ElementTree interfaces. Please read the package's official documentation for more details.

https://web.archive.org/web/20201124024954/http://effbot.org/zone/element-index.htm
Official documentation for ElementTree.

The hashlib package

A new hashlib module, written by Gregory P. Smith, has been added to replace the !md5 and !sha modules. hashlib adds support for additional secure hashes (SHA-224, SHA-256, SHA-384, and SHA-512). When available, the module uses OpenSSL for fast platform optimized implementations of algorithms.

The old !md5 and !sha modules still exist as wrappers around hashlib to preserve backwards compatibility. The new module's interface is very close to that of the old modules, but not identical. The most significant difference is that the constructor functions for creating new hashing objects are named differently. :

# Old versions
h = md5.md5()
h = md5.new()

# New version
h = hashlib.md5()

# Old versions
h = sha.sha()
h = sha.new()

# New version
h = hashlib.sha1()

# Hash that weren't previously available
h = hashlib.sha224()
h = hashlib.sha256()
h = hashlib.sha384()
h = hashlib.sha512()

# Alternative form
h = hashlib.new('md5')          # Provide algorithm as a string

Once a hash object has been created, its methods are the same as before: update(string) hashes the specified string into the current digest state, ~hashlib.hash.digest and ~hashlib.hash.hexdigest return the digest value as a binary string or a string of hex digits, and copy returns a new hashing object with the same digest state.

The documentation for the hashlib module.

The sqlite3 package

The pysqlite module (https://www.pysqlite.org), a wrapper for the SQLite embedded database, has been added to the standard library under the package name sqlite3.

SQLite is a C library that provides a lightweight disk-based database that doesn't require a separate server process and allows accessing the database using a nonstandard variant of the SQL query language. Some applications can use SQLite for internal data storage. It's also possible to prototype an application using SQLite and then port the code to a larger database such as PostgreSQL or Oracle.

pysqlite was written by Gerhard Häring and provides a SQL interface compliant with the DB-API 2.0 specification described by 249.

If you're compiling the Python source yourself, note that the source tree doesn't include the SQLite code, only the wrapper module. You'll need to have the SQLite libraries and headers installed before compiling Python, and the build process will compile the module when the necessary headers are available.

To use the module, you must first create a ~sqlite3.Connection object that represents the database. Here the data will be stored in the /tmp/example file:

conn = sqlite3.connect('/tmp/example')

You can also supply the special name :memory: to create a database in RAM.

Once you have a ~sqlite3.Connection, you can create a ~sqlite3.Cursor object and call its ~sqlite3.Cursor.execute method to perform SQL commands:

c = conn.cursor()

# Create table
c.execute('''create table stocks
(date text, trans text, symbol text,
 qty real, price real)''')

# Insert a row of data
c.execute("""insert into stocks
          values ('2006-01-05','BUY','RHAT',100,35.14)""")

Usually your SQL operations will need to use values from Python variables. You shouldn't assemble your query using Python's string operations because doing so is insecure; it makes your program vulnerable to an SQL injection attack.

Instead, use the DB-API's parameter substitution. Put ? as a placeholder wherever you want to use a value, and then provide a tuple of values as the second argument to the cursor's ~sqlite3.Cursor.execute method. (Other database modules may use a different placeholder, such as %s or :1.) For example:

# Never do this -- insecure!
symbol = 'IBM'
c.execute("... where symbol = '%s'" % symbol)

# Do this instead
t = (symbol,)
c.execute('select * from stocks where symbol=?', t)

# Larger example
for t in (('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
          ('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00),
          ('2006-04-06', 'SELL', 'IBM', 500, 53.00),
         ):
    c.execute('insert into stocks values (?,?,?,?,?)', t)

To retrieve data after executing a SELECT statement, you can either treat the cursor as an iterator, call the cursor's ~sqlite3.Cursor.fetchone method to retrieve a single matching row, or call ~sqlite3.Cursor.fetchall to get a list of the matching rows.

This example uses the iterator form:

>>> c = conn.cursor()
>>> c.execute('select * from stocks order by price')
>>> for row in c:
...    print row
...
(u'2006-01-05', u'BUY', u'RHAT', 100, 35.140000000000001)
(u'2006-03-28', u'BUY', u'IBM', 1000, 45.0)
(u'2006-04-06', u'SELL', u'IBM', 500, 53.0)
(u'2006-04-05', u'BUY', u'MSOFT', 1000, 72.0)
>>>

For more information about the SQL dialect supported by SQLite, see https://www.sqlite.org.

https://www.pysqlite.org
The pysqlite web page.

https://www.sqlite.org
The SQLite web page; the documentation describes the syntax and the available data types for the supported SQL dialect.

The documentation for the sqlite3 module.

249 - Database API Specification 2.0
PEP written by Marc-André Lemburg.

The wsgiref package

The Web Server Gateway Interface (WSGI) v1.0 defines a standard interface between web servers and Python web applications and is described in 333. The wsgiref package is a reference implementation of the WSGI specification.

The package includes a basic HTTP server that will run a WSGI application; this server is useful for debugging but isn't intended for production use. Setting up a server takes only a few lines of code:

from wsgiref import simple_server

wsgi_app = ...

host = ''
port = 8000
httpd = simple_server.make_server(host, port, wsgi_app)
httpd.serve_forever()

https://web.archive.org/web/20160331090247/http://wsgi.readthedocs.org/en/latest/
A central web site for WSGI-related resources.

333 - Python Web Server Gateway Interface v1.0
PEP written by Phillip J. Eby.

Build and C API Changes

Changes to Python's build process and to the C API include:

Port-Specific Changes

Porting to Python 2.5

This section lists previously described changes that may require changes to your code:

Acknowledgements

The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this article: Georg Brandl, Nick Coghlan, Phillip J. Eby, Lars Gustäbel, Raymond Hettinger, Ralf W. Grosse-Kunstleve, Kent Johnson, Iain Lowe, Martin von Löwis, Fredrik Lundh, Andrew McNamara, Skip Montanaro, Gustavo Niemeyer, Paul Prescod, James Pryor, Mike Rovner, Scott Weikart, Barry Warsaw, Thomas Wouters.

What's New in Python 2.6

Author
A.M. Kuchling (amk at amk.ca)

This article explains the new features in Python 2.6, released on October 1, 2008. The release schedule is described in 361.

The major theme of Python 2.6 is preparing the migration path to Python 3.0, a major redesign of the language. Whenever possible, Python 2.6 incorporates new features and syntax from 3.0 while remaining compatible with existing code by not removing older features or syntax. When it's not possible to do that, Python 2.6 tries to do what it can, adding compatibility functions in a !future_builtins module and a !-3 switch to warn about usages that will become unsupported in 3.0.

Some significant new packages have been added to the standard library, such as the multiprocessing and json modules, but there aren't many new features that aren't related to Python 3.0 in some way.

Python 2.6 also sees a number of improvements and bugfixes throughout the source. A search through the change logs finds there were 259 patches applied and 612 bugs fixed between Python 2.5 and 2.6. Both figures are likely to be underestimates.

This article doesn't attempt to provide a complete specification of the new features, but instead provides a convenient overview. For full details, you should refer to the documentation for Python 2.6. If you want to understand the rationale for the design and implementation, refer to the PEP for a particular new feature. Whenever possible, "What's New in Python" links to the bug/patch item for each change.

Python 3.0

The development cycle for Python versions 2.6 and 3.0 was synchronized, with the alpha and beta releases for both versions being made on the same days. The development of 3.0 has influenced many features in 2.6.

Python 3.0 is a far-ranging redesign of Python that breaks compatibility with the 2.x series. This means that existing Python code will need some conversion in order to run on Python 3.0. However, not all the changes in 3.0 necessarily break compatibility. In cases where new features won't cause existing code to break, they've been backported to 2.6 and are described in this document in the appropriate place. Some of the 3.0-derived features are:

Python 3.0 adds several new built-in functions and changes the semantics of some existing builtins. Functions that are new in 3.0 such as bin have simply been added to Python 2.6, but existing builtins haven't been changed; instead, the !future_builtins module has versions with the new 3.0 semantics. Code written to be compatible with 3.0 can do from future_builtins import hex, map as necessary.

A new command-line switch, !-3, enables warnings about features that will be removed in Python 3.0. You can run code with this switch to see how much work will be necessary to port code to 3.0. The value of this switch is available to Python code as the boolean variable !sys.py3kwarning, and to C extension code as !Py_Py3kWarningFlag.

The 3xxx series of PEPs, which contains proposals for Python 3.0. 3000 describes the development process for Python 3.0. Start with 3100 that describes the general goals for Python 3.0, and then explore the higher-numbered PEPs that propose specific features.

Changes to the Development Process

While 2.6 was being developed, the Python development process underwent two significant changes: we switched from SourceForge's issue tracker to a customized Roundup installation, and the documentation was converted from LaTeX to reStructuredText.

New Issue Tracker: Roundup

For a long time, the Python developers had been growing increasingly annoyed by SourceForge's bug tracker. SourceForge's hosted solution doesn't permit much customization; for example, it wasn't possible to customize the life cycle of issues.

The infrastructure committee of the Python Software Foundation therefore posted a call for issue trackers, asking volunteers to set up different products and import some of the bugs and patches from SourceForge. Four different trackers were examined: Jira, Launchpad, Roundup, and Trac. The committee eventually settled on Jira and Roundup as the two candidates. Jira is a commercial product that offers no-cost hosted instances to free-software projects; Roundup is an open-source project that requires volunteers to administer it and a server to host it.

After posting a call for volunteers, a new Roundup installation was set up at https://bugs.python.org. One installation of Roundup can host multiple trackers, and this server now also hosts issue trackers for Jython and for the Python web site. It will surely find other uses in the future. Where possible, this edition of "What's New in Python" links to the bug/patch item for each change.

Hosting of the Python bug tracker is kindly provided by Upfront Systems of Stellenbosch, South Africa. Martin von Löwis put a lot of effort into importing existing bugs and patches from SourceForge; his scripts for this import operation are at https://svn.python.org/view/tracker/importer/ and may be useful to other projects wishing to move from SourceForge to Roundup.

https://bugs.python.org
The Python bug tracker.

https://bugs.jython.org:
The Jython bug tracker.

https://roundup.sourceforge.io/
Roundup downloads and documentation.

https://svn.python.org/view/tracker/importer/
Martin von Löwis's conversion scripts.

New Documentation Format: reStructuredText Using Sphinx

The Python documentation was written using LaTeX since the project started around 1989. In the 1980s and early 1990s, most documentation was printed out for later study, not viewed online. LaTeX was widely used because it provided attractive printed output while remaining straightforward to write once the basic rules of the markup were learned.

Today LaTeX is still used for writing publications destined for printing, but the landscape for programming tools has shifted. We no longer print out reams of documentation; instead, we browse through it online and HTML has become the most important format to support. Unfortunately, converting LaTeX to HTML is fairly complicated and Fred L. Drake Jr., the long-time Python documentation editor, spent a lot of time maintaining the conversion process. Occasionally people would suggest converting the documentation into SGML and later XML, but performing a good conversion is a major task and no one ever committed the time required to finish the job.

During the 2.6 development cycle, Georg Brandl put a lot of effort into building a new toolchain for processing the documentation. The resulting package is called Sphinx, and is available from https://www.sphinx-doc.org/.

Sphinx concentrates on HTML output, producing attractively styled and modern HTML; printed output is still supported through conversion to LaTeX. The input format is reStructuredText, a markup syntax supporting custom extensions and directives that is commonly used in the Python community.

Sphinx is a standalone package that can be used for writing, and almost two dozen other projects (listed on the Sphinx web site) have adopted Sphinx as their documentation tool.

Documenting Python
Describes how to write for Python's documentation.

Sphinx
Documentation and code for the Sphinx toolchain.

Docutils
The underlying reStructuredText parser and toolset.

PEP 343: The 'with' statement

The previous version, Python 2.5, added the 'with' statement as an optional feature, to be enabled by a from __future__ import with_statement directive. In 2.6 the statement no longer needs to be specially enabled; this means that !with is now always a keyword. The rest of this section is a copy of the corresponding section from the "What's New in Python 2.5" document; if you're familiar with the '!with' statement from Python 2.5, you can skip this section.

The 'with' statement clarifies code that previously would use try...finally blocks to ensure that clean-up code is executed. In this section, I'll discuss the statement as it will commonly be used. In the next section, I'll examine the implementation details and show how to write objects for use with this statement.

The 'with' statement is a control-flow structure whose basic structure is:

with expression [as variable]:
    with-block

The expression is evaluated, and it should result in an object that supports the context management protocol (that is, has ~object.__enter__ and ~object.__exit__ methods).

The object's ~object.__enter__ is called before with-block is executed and therefore can run set-up code. It also may return a value that is bound to the name variable, if given. (Note carefully that variable is not assigned the result of expression.)

After execution of the with-block is finished, the object's ~object.__exit__ method is called, even if the block raised an exception, and can therefore run clean-up code.

Some standard Python objects now support the context management protocol and can be used with the 'with' statement. File objects are one example:

with open('/etc/passwd', 'r') as f:
    for line in f:
        print line
        ... more processing code ...

After this statement has executed, the file object in f will have been automatically closed, even if the for loop raised an exception part-way through the block.

[!NOTE] In this case, f is the same object created by open, because ~object.__enter__ returns self.

The threading module's locks and condition variables also support the 'with' statement:

lock = threading.Lock()
with lock:
    # Critical section of code
    ...

The lock is acquired before the block is executed and always released once the block is complete.

The ~decimal.localcontext function in the decimal module makes it easy to save and restore the current decimal context, which encapsulates the desired precision and rounding characteristics for computations:

from decimal import Decimal, Context, localcontext

# Displays with default precision of 28 digits
v = Decimal('578')
print v.sqrt()

with localcontext(Context(prec=16)):
    # All code in this block uses a precision of 16 digits.
    # The original context is restored on exiting the block.
    print v.sqrt()

Writing Context Managers

Under the hood, the 'with' statement is fairly complicated. Most people will only use '!with' in company with existing objects and don't need to know these details, so you can skip the rest of this section if you like. Authors of new objects will need to understand the details of the underlying implementation and should keep reading.

A high-level explanation of the context management protocol is:

Let's think through an example. I won't present detailed code but will only sketch the methods necessary for a database that supports transactions.

(For people unfamiliar with database terminology: a set of changes to the database are grouped into a transaction. Transactions can be either committed, meaning that all the changes are written into the database, or rolled back, meaning that the changes are all discarded and the database is unchanged. See any database textbook for more information.)

Let's assume there's an object representing a database connection. Our goal will be to let the user write code like this:

db_connection = DatabaseConnection()
with db_connection as cursor:
    cursor.execute('insert into ...')
    cursor.execute('delete from ...')
    # ... more operations ...

The transaction should be committed if the code in the block runs flawlessly or rolled back if there's an exception. Here's the basic interface for !DatabaseConnection that I'll assume:

class DatabaseConnection:
    # Database interface
    def cursor(self):
        "Returns a cursor object and starts a new transaction"
    def commit(self):
        "Commits current transaction"
    def rollback(self):
        "Rolls back current transaction"

The ~object.__enter__ method is pretty easy, having only to start a new transaction. For this application the resulting cursor object would be a useful result, so the method will return it. The user can then add as cursor to their 'with' statement to bind the cursor to a variable name. :

class DatabaseConnection:
    ...
    def __enter__(self):
        # Code to start a new transaction
        cursor = self.cursor()
        return cursor

The ~object.__exit__ method is the most complicated because it's where most of the work has to be done. The method has to check if an exception occurred. If there was no exception, the transaction is committed. The transaction is rolled back if there was an exception.

In the code below, execution will just fall off the end of the function, returning the default value of None. None is false, so the exception will be re-raised automatically. If you wished, you could be more explicit and add a return statement at the marked location. :

class DatabaseConnection:
    ...
    def __exit__(self, type, value, tb):
        if tb is None:
            # No exception, so commit
            self.commit()
        else:
            # Exception occurred, so rollback.
            self.rollback()
            # return False

The contextlib module

The contextlib module provides some functions and a decorator that are useful when writing objects for use with the 'with' statement.

The decorator is called ~contextlib.contextmanager, and lets you write a single generator function instead of defining a new class. The generator should yield exactly one value. The code up to the yield will be executed as the ~object.__enter__ method, and the value yielded will be the method's return value that will get bound to the variable in the 'with' statement's !as clause, if any. The code after the !yield will be executed in the ~object.__exit__ method. Any exception raised in the block will be raised by the !yield statement.

Using this decorator, our database example from the previous section could be written as:

from contextlib import contextmanager

@contextmanager
def db_transaction(connection):
    cursor = connection.cursor()
    try:
        yield cursor
    except:
        connection.rollback()
        raise
    else:
        connection.commit()

db = DatabaseConnection()
with db_transaction(db) as cursor:
    ...

The contextlib module also has a nested(mgr1, mgr2, ...) function that combines a number of context managers so you don't need to write nested 'with' statements. In this example, the single '!with' statement both starts a database transaction and acquires a thread lock:

lock = threading.Lock()
with nested (db_transaction(db), lock) as (cursor, locked):
    ...

Finally, the ~contextlib.closing function returns its argument so that it can be bound to a variable, and calls the argument's .close() method at the end of the block. :

import urllib, sys
from contextlib import closing

with closing(urllib.urlopen('http://www.yahoo.com')) as f:
    for line in f:
        sys.stdout.write(line)

343 - The "with" statement
PEP written by Guido van Rossum and Nick Coghlan; implemented by Mike Bland, Guido van Rossum, and Neal Norwitz. The PEP shows the code generated for a 'with' statement, which can be helpful in learning how the statement works.

The documentation for the contextlib module.

PEP 366: Explicit Relative Imports From a Main Module

Python's -m switch allows running a module as a script. When you ran a module that was located inside a package, relative imports didn't work correctly.

The fix for Python 2.6 adds a module.__package__ attribute. When this attribute is present, relative imports will be relative to the value of this attribute instead of the ~module.__name__ attribute.

PEP 302-style importers can then set ~module.__package__ as necessary. The runpy module that implements the -m switch now does this, so relative imports will now work correctly in scripts running from inside a package.

PEP 370: Per-user site-packages Directory

When you run Python, the module search path sys.path usually includes a directory whose path ends in "site-packages". This directory is intended to hold locally installed packages available to all users using a machine or a particular site installation.

Python 2.6 introduces a convention for user-specific site directories. The directory varies depending on the platform:

Within this directory, there will be version-specific subdirectories, such as lib/python2.6/site-packages on Unix/Mac OS and Python26/site-packages on Windows.

If you don't like the default directory, it can be overridden by an environment variable. PYTHONUSERBASE sets the root directory used for all Python versions supporting this feature. On Windows, the directory for application-specific data can be changed by setting the !APPDATA environment variable. You can also modify the site.py file for your Python installation.

The feature can be disabled entirely by running Python with the -s option or setting the PYTHONNOUSERSITE environment variable.

370 - Per-user site-packages Directory
PEP written and implemented by Christian Heimes.

PEP 371: The multiprocessing Package

The new multiprocessing package lets Python programs create new processes that will perform a computation and return a result to the parent. The parent and child processes can communicate using queues and pipes, synchronize their operations using locks and semaphores, and can share simple arrays of data.

The multiprocessing module started out as an exact emulation of the threading module using processes instead of threads. That goal was discarded along the path to Python 2.6, but the general approach of the module is still similar. The fundamental class is the ~multiprocessing.Process, which is passed a callable object and a collection of arguments. The ~multiprocessing.Process.start method sets the callable running in a subprocess, after which you can call the ~multiprocessing.Process.is_alive method to check whether the subprocess is still running and the ~multiprocessing.Process.join method to wait for the process to exit.

Here's a simple example where the subprocess will calculate a factorial. The function doing the calculation is written strangely so that it takes significantly longer when the input argument is a multiple of 4.

import time
from multiprocessing import Process, Queue


def factorial(queue, N):
    "Compute a factorial."
    # If N is a multiple of 4, this function will take much longer.
    if (N % 4) == 0:
        time.sleep(.05 * N/4)

    # Calculate the result
    fact = 1L
    for i in range(1, N+1):
        fact = fact * i

    # Put the result on the queue
    queue.put(fact)

if __name__ == '__main__':
    queue = Queue()

    N = 5

    p = Process(target=factorial, args=(queue, N))
    p.start()
    p.join()

    result = queue.get()
    print 'Factorial', N, '=', result

A ~queue.Queue is used to communicate the result of the factorial. The ~queue.Queue object is stored in a global variable. The child process will use the value of the variable when the child was created; because it's a ~queue.Queue, parent and child can use the object to communicate. (If the parent were to change the value of the global variable, the child's value would be unaffected, and vice versa.)

Two other classes, ~multiprocessing.pool.Pool and ~multiprocessing.Manager, provide higher-level interfaces. ~multiprocessing.pool.Pool will create a fixed number of worker processes, and requests can then be distributed to the workers by calling ~multiprocessing.pool.Pool.apply or ~multiprocessing.pool.Pool.apply_async to add a single request, and ~multiprocessing.pool.Pool.map or ~multiprocessing.pool.Pool.map_async to add a number of requests. The following code uses a ~multiprocessing.pool.Pool to spread requests across 5 worker processes and retrieve a list of results:

from multiprocessing import Pool

def factorial(N, dictionary):
    "Compute a factorial."
    ...
p = Pool(5)
result = p.map(factorial, range(1, 1000, 10))
for v in result:
    print v

This produces the following output:

1
39916800
51090942171709440000
8222838654177922817725562880000000
33452526613163807108170062053440751665152000000000
...

The other high-level interface, the ~multiprocessing.Manager class, creates a separate server process that can hold master copies of Python data structures. Other processes can then access and modify these data structures using proxy objects. The following example creates a shared dictionary by calling the dict method; the worker processes then insert values into the dictionary. (Locking is not done for you automatically, which doesn't matter in this example. ~multiprocessing.Manager's methods also include ~multiprocessing.managers.SyncManager.Lock, ~multiprocessing.managers.SyncManager.RLock, and ~multiprocessing.managers.SyncManager.Semaphore to create shared locks.)

import time
from multiprocessing import Pool, Manager

def factorial(N, dictionary):
    "Compute a factorial."
    # Calculate the result
    fact = 1L
    for i in range(1, N+1):
        fact = fact * i

    # Store result in dictionary
    dictionary[N] = fact

if __name__ == '__main__':
    p = Pool(5)
    mgr = Manager()
    d = mgr.dict()         # Create shared dictionary

    # Run tasks using the pool
    for N in range(1, 1000, 10):
        p.apply_async(factorial, (N, d))

    # Mark pool as closed -- no more tasks can be added.
    p.close()

    # Wait for tasks to exit
    p.join()

    # Output results
    for k, v in sorted(d.items()):
        print k, v

This will produce the output:

1 1
11 39916800
21 51090942171709440000
31 8222838654177922817725562880000000
41 33452526613163807108170062053440751665152000000000
51 15511187532873822802242430164693032110632597200169861120000...

The documentation for the multiprocessing module.

371 - Addition of the multiprocessing package
PEP written by Jesse Noller and Richard Oudkerk; implemented by Richard Oudkerk and Jesse Noller.

PEP 3101: Advanced String Formatting

In Python 3.0, the % operator is supplemented by a more powerful string formatting method, format. Support for the str.format method has been backported to Python 2.6.

In 2.6, both 8-bit and Unicode strings have a .format() method that treats the string as a template and takes the arguments to be formatted. The formatting template uses curly brackets ({, }) as special characters:

>>> # Substitute positional argument 0 into the string.
>>> "User ID: {0}".format("root")
'User ID: root'
>>> # Use the named keyword arguments
>>> "User ID: {uid}   Last seen: {last_login}".format(
...    uid="root",
...    last_login = "5 Mar 2008 07:20")
'User ID: root   Last seen: 5 Mar 2008 07:20'

Curly brackets can be escaped by doubling them:

>>> "Empty dict: {{}}".format()
"Empty dict: {}"

Field names can be integers indicating positional arguments, such as {0}, {1}, etc. or names of keyword arguments. You can also supply compound field names that read attributes or access dictionary keys:

>>> import sys
>>> print 'Platform: {0.platform}\nPython version: {0.version}'.format(sys)
Platform: darwin
Python version: 2.6a1+ (trunk:61261M, Mar  5 2008, 20:29:41)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)]'

>>> import mimetypes
>>> 'Content-type: {0[.mp4]}'.format(mimetypes.types_map)
'Content-type: video/mp4'

Note that when using dictionary-style notation such as [.mp4], you don't need to put any quotation marks around the string; it will look up the value using .mp4 as the key. Strings beginning with a number will be converted to an integer. You can't write more complicated expressions inside a format string.

So far we've shown how to specify which field to substitute into the resulting string. The precise formatting used is also controllable by adding a colon followed by a format specifier. For example:

>>> # Field 0: left justify, pad to 15 characters
>>> # Field 1: right justify, pad to 6 characters
>>> fmt = '{0:15} ${1:>6}'
>>> fmt.format('Registration', 35)
'Registration    $    35'
>>> fmt.format('Tutorial', 50)
'Tutorial        $    50'
>>> fmt.format('Banquet', 125)
'Banquet         $   125'

Format specifiers can reference other fields through nesting:

>>> fmt = '{0:{1}}'
>>> width = 15
>>> fmt.format('Invoice #1234', width)
'Invoice #1234  '
>>> width = 35
>>> fmt.format('Invoice #1234', width)
'Invoice #1234                      '

The alignment of a field within the desired width can be specified:

Character Effect
< (default) Left-align
> Right-align
^ Center
= (For numeric types only) Pad after the sign.

Format specifiers can also include a presentation type, which controls how the value is formatted. For example, floating-point numbers can be formatted as a general number or in exponential notation:

>>> '{0:g}'.format(3.75)
'3.75'
>>> '{0:e}'.format(3.75)
'3.750000e+00'

A variety of presentation types are available. Consult the 2.6 documentation for a complete list <formatstrings>; here's a sample:
b Binary. Outputs the number in base 2.
c Character. Converts the integer to the corresponding Unicode character before printing.
d Decimal Integer. Outputs the number in base 10.
o Octal format. Outputs the number in base 8.
x Hex format. Outputs the number in base 16, using lower-case letters for the digits above 9.
e Exponent notation. Prints the number in scientific notation using the letter 'e' to indicate the exponent.
g General format. This prints the number as a fixed-point number, unless the number is too large, in which case it switches to 'e' exponent notation.
n Number. This is the same as 'g' (for floats) or 'd' (for integers), except that it uses the current locale setting to insert the appropriate number separator characters.
% Percentage. Multiplies the number by 100 and displays in fixed ('f') format, followed by a percent sign.

Classes and types can define a ~object.__format__ method to control how they're formatted. It receives a single argument, the format specifier:

def __format__(self, format_spec):
    if isinstance(format_spec, unicode):
        return unicode(str(self))
    else:
        return str(self)

There's also a format builtin that will format a single value. It calls the type's ~object.__format__ method with the provided specifier:

>>> format(75.6564, '.2f')
'75.66'

formatstrings
The reference documentation for format fields.

3101 - Advanced String Formatting
PEP written by Talin. Implemented by Eric Smith.

PEP 3105: print As a Function

The print statement becomes the print function in Python 3.0. Making print a function makes it possible to replace the function by doing def print(...) or importing a new function from somewhere else.

Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead. For example:

>>> from __future__ import print_function
>>> print('# of entries', len(dictionary), file=sys.stderr)

The signature of the new function is:

def print(*args, sep=' ', end='\n', file=None)

The parameters are:

3105 - Make print a function
PEP written by Georg Brandl.

PEP 3110: Exception-Handling Changes

One error that Python programmers occasionally make is writing the following code:

try:
    ...
except TypeError, ValueError:  # Wrong!
    ...

The author is probably trying to catch both TypeError and ValueError exceptions, but this code actually does something different: it will catch TypeError and bind the resulting exception object to the local name "ValueError". The ValueError exception will not be caught at all. The correct code specifies a tuple of exceptions:

try:
    ...
except (TypeError, ValueError):
    ...

This error happens because the use of the comma here is ambiguous: does it indicate two different nodes in the parse tree, or a single node that's a tuple?

Python 3.0 makes this unambiguous by replacing the comma with the word "as". To catch an exception and store the exception object in the variable exc, you must write:

try:
    ...
except TypeError as exc:
    ...

Python 3.0 will only support the use of "as", and therefore interprets the first example as catching two different exceptions. Python 2.6 supports both the comma and "as", so existing code will continue to work. We therefore suggest using "as" when writing new Python code that will only be executed with 2.6.

3110 - Catching Exceptions in Python 3000
PEP written and implemented by Collin Winter.

PEP 3112: Byte Literals

Python 3.0 adopts Unicode as the language's fundamental string type and denotes 8-bit literals differently, either as b'string' or using a bytes constructor. For future compatibility, Python 2.6 adds bytes as a synonym for the str type, and it also supports the b'' notation.

The 2.6 str differs from 3.0's bytes type in various ways; most notably, the constructor is completely different. In 3.0, bytes([65, 66, 67]) is 3 elements long, containing the bytes representing ABC; in 2.6, bytes([65, 66, 67]) returns the 12-byte string representing the str of the list.

The primary use of bytes in 2.6 will be to write tests of object type such as isinstance(x, bytes). This will help the 2to3 converter, which can't tell whether 2.x code intends strings to contain either characters or 8-bit bytes; you can now use either bytes or str to represent your intention exactly, and the resulting code will also be correct in Python 3.0.

There's also a __future__ import that causes all string literals to become Unicode strings. This means that \u escape sequences can be used to include Unicode characters:

from __future__ import unicode_literals

s = ('\u751f\u3080\u304e\u3000\u751f\u3054'
     '\u3081\u3000\u751f\u305f\u307e\u3054')

print len(s)               # 12 Unicode characters

At the C level, Python 3.0 will rename the existing 8-bit string type, called !PyStringObject in Python 2.x, to PyBytesObject. Python 2.6 uses #define to support using the names PyBytesObject, PyBytes_Check, PyBytes_FromStringAndSize, and all the other functions and macros used with strings.

Instances of the bytes type are immutable just as strings are. A new bytearray type stores a mutable sequence of bytes:

>>> bytearray([65, 66, 67])
bytearray(b'ABC')
>>> b = bytearray(u'\u21ef\u3244', 'utf-8')
>>> b
bytearray(b'\xe2\x87\xaf\xe3\x89\x84')
>>> b[0] = '\xe3'
>>> b
bytearray(b'\xe3\x87\xaf\xe3\x89\x84')
>>> unicode(str(b), 'utf-8')
u'\u31ef \u3244'

Byte arrays support most of the methods of string types, such as ~bytearray.startswith/~bytearray.endswith, ~bytearray.find/~bytearray.rfind, and some of the methods of lists, such as ~bytearray.append, ~bytearray.pop, and ~bytearray.reverse.

>>> b = bytearray('ABC')
>>> b.append('d')
>>> b.append(ord('e'))
>>> b
bytearray(b'ABCde')

There's also a corresponding C API, with PyByteArray_FromObject, PyByteArray_FromStringAndSize, and various other functions.

3112 - Bytes literals in Python 3000
PEP written by Jason Orendorff; backported to 2.6 by Christian Heimes.

PEP 3116: New I/O Library

Python's built-in file objects support a number of methods, but file-like objects don't necessarily support all of them. Objects that imitate files usually support !read and !write, but they may not support !readline, for example. Python 3.0 introduces a layered I/O library in the io module that separates buffering and text-handling features from the fundamental read and write operations.

There are three levels of abstract base classes provided by the io module:

In Python 2.6, the underlying implementations haven't been restructured to build on top of the io module's classes. The module is being provided to make it easier to write code that's forward-compatible with 3.0, and to save developers the effort of writing their own implementations of buffering and text I/O.

3116 - New I/O
PEP written by Daniel Stutzbach, Mike Verdone, and Guido van Rossum. Code by Guido van Rossum, Georg Brandl, Walter Doerwald, Jeremy Hylton, Martin von Löwis, Tony Lownds, and others.

PEP 3118: Revised Buffer Protocol

The buffer protocol is a C-level API that lets Python types exchange pointers into their internal representations. A memory-mapped file can be viewed as a buffer of characters, for example, and this lets another module such as re treat memory-mapped files as a string of characters to be searched.

The primary users of the buffer protocol are numeric-processing packages such as NumPy, which expose the internal representation of arrays so that callers can write data directly into an array instead of going through a slower API. This PEP updates the buffer protocol in light of experience from NumPy development, adding a number of new features such as indicating the shape of an array or locking a memory region.

The most important new C API function is PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags), which takes an object and a set of flags, and fills in the Py_buffer structure with information about the object's memory representation. Objects can use this operation to lock memory in place while an external caller could be modifying the contents, so there's a corresponding PyBuffer_Release(Py_buffer *view) to indicate that the external caller is done.

The flags argument to PyObject_GetBuffer specifies constraints upon the memory returned. Some examples are:

Two new argument codes for PyArg_ParseTuple, s* and z*, return locked buffer objects for a parameter.

3118 - Revising the buffer protocol
PEP written by Travis Oliphant and Carl Banks; implemented by Travis Oliphant.

PEP 3119: Abstract Base Classes

Some object-oriented languages such as Java support interfaces, declaring that a class has a given set of methods or supports a given access protocol. Abstract Base Classes (or ABCs) are an equivalent feature for Python. The ABC support consists of an abc module containing a metaclass called ~abc.ABCMeta, special handling of this metaclass by the isinstance and issubclass builtins, and a collection of basic ABCs that the Python developers think will be widely useful. Future versions of Python will probably add more ABCs.

Let's say you have a particular class and wish to know whether it supports dictionary-style access. The phrase "dictionary-style" is vague, however. It probably means that accessing items with obj[1] works. Does it imply that setting items with obj[2] = value works? Or that the object will have !keys, !values, and !items methods? What about the iterative variants such as !iterkeys? !copy`and :meth:!update`? Iterating over the object with !iter?

The Python 2.6 collections module includes a number of different ABCs that represent these distinctions. Iterable indicates that a class defines ~object.__iter__, and Container means the class defines a ~object.__contains__ method and therefore supports x in y expressions. The basic dictionary interface of getting items, setting items, and !keys, !values, and !items, is defined by the MutableMapping ABC.

You can derive your own classes from a particular ABC to indicate they support that ABC's interface:

import collections

class Storage(collections.MutableMapping):
    ...

Alternatively, you could write the class without deriving from the desired ABC and instead register the class by calling the ABC's ~abc.ABCMeta.register method:

import collections

class Storage:
    ...

collections.MutableMapping.register(Storage)

For classes that you write, deriving from the ABC is probably clearer. The ~abc.ABCMeta.register method is useful when you've written a new ABC that can describe an existing type or class, or if you want to declare that some third-party class implements an ABC. For example, if you defined a !PrintableType ABC, it's legal to do:

# Register Python's types
PrintableType.register(int)
PrintableType.register(float)
PrintableType.register(str)

Classes should obey the semantics specified by an ABC, but Python can't check this; it's up to the class author to understand the ABC's requirements and to implement the code accordingly.

To check whether an object supports a particular interface, you can now write:

def func(d):
    if not isinstance(d, collections.MutableMapping):
        raise ValueError("Mapping object expected, not %r" % d)

Don't feel that you must now begin writing lots of checks as in the above example. Python has a strong tradition of duck-typing, where explicit type-checking is never done and code simply calls methods on an object, trusting that those methods will be there and raising an exception if they aren't. Be judicious in checking for ABCs and only do it where it's absolutely necessary.

You can write your own ABCs by using abc.ABCMeta as the metaclass in a class definition:

from abc import ABCMeta, abstractmethod

class Drawable():
    __metaclass__ = ABCMeta

    @abstractmethod
    def draw(self, x, y, scale=1.0):
        pass

    def draw_doubled(self, x, y):
        self.draw(x, y, scale=2.0)


class Square(Drawable):
    def draw(self, x, y, scale):
        ...

In the !Drawable ABC above, the !draw_doubled method renders the object at twice its size and can be implemented in terms of other methods described in !Drawable. Classes implementing this ABC therefore don't need to provide their own implementation of !draw_doubled, though they can do so. An implementation of !draw is necessary, though; the ABC can't provide a useful generic implementation.

You can apply the ~abc.abstractmethod decorator to methods such as !draw that must be implemented; Python will then raise an exception for classes that don't define the method. Note that the exception is only raised when you actually try to create an instance of a subclass lacking the method:

>>> class Circle(Drawable):
...     pass
...
>>> c = Circle()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class Circle with abstract methods draw
>>>

Abstract data attributes can be declared using the @abstractproperty decorator:

from abc import abstractproperty
...

@abstractproperty
def readonly(self):
   return self._x

Subclasses must then define a readonly property.

3119 - Introducing Abstract Base Classes
PEP written by Guido van Rossum and Talin. Implemented by Guido van Rossum. Backported to 2.6 by Benjamin Aranguren, with Alex Martelli.

PEP 3127: Integer Literal Support and Syntax

Python 3.0 changes the syntax for octal (base-8) integer literals, prefixing them with "0o" or "0O" instead of a leading zero, and adds support for binary (base-2) integer literals, signalled by a "0b" or "0B" prefix.

Python 2.6 doesn't drop support for a leading 0 signalling an octal number, but it does add support for "0o" and "0b":

>>> 0o21, 2*8 + 1
(17, 17)
>>> 0b101111
47

The oct builtin still returns numbers prefixed with a leading zero, and a new bin builtin returns the binary representation for a number:

>>> oct(42)
'052'
>>> future_builtins.oct(42)
'0o52'
>>> bin(173)
'0b10101101'

The int and !long builtins will now accept the "0o" and "0b" prefixes when base-8 or base-2 are requested, or when the base argument is zero (signalling that the base used should be determined from the string):

>>> int ('0o52', 0)
42
>>> int('1101', 2)
13
>>> int('0b1101', 2)
13
>>> int('0b1101', 0)
13

3127 - Integer Literal Support and Syntax
PEP written by Patrick Maupin; backported to 2.6 by Eric Smith.

PEP 3129: Class Decorators

Decorators have been extended from functions to classes. It's now legal to write:

@foo
@bar
class A:
  pass

This is equivalent to:

class A:
  pass

A = foo(bar(A))

3129 - Class Decorators
PEP written by Collin Winter.

PEP 3141: A Type Hierarchy for Numbers

Python 3.0 adds several abstract base classes for numeric types inspired by Scheme's numeric tower. These classes were backported to 2.6 as the numbers module.

The most general ABC is ~numbers.Number. It defines no operations at all, and only exists to allow checking if an object is a number by doing isinstance(obj, Number).

~numbers.Complex is a subclass of ~numbers.Number. Complex numbers can undergo the basic operations of addition, subtraction, multiplication, division, and exponentiation, and you can retrieve the real and imaginary parts and obtain a number's conjugate. Python's built-in complex type is an implementation of ~numbers.Complex.

~numbers.Real further derives from ~numbers.Complex, and adds operations that only work on real numbers: ~math.floor, ~math.trunc, rounding, taking the remainder mod N, floor division, and comparisons.

~numbers.Rational numbers derive from ~numbers.Real, have ~numbers.Rational.numerator and ~numbers.Rational.denominator properties, and can be converted to floats. Python 2.6 adds a simple rational-number class, ~fractions.Fraction, in the fractions module. (It's called ~fractions.Fraction instead of Rational to avoid a name clash with numbers.Rational.)

~numbers.Integral numbers derive from ~numbers.Rational, and can be shifted left and right with << and >>, combined using bitwise operations such as & and |, and can be used as array indexes and slice boundaries.

In Python 3.0, the PEP slightly redefines the existing builtins round, math.floor, math.ceil, and adds a new one, math.trunc, that's been backported to Python 2.6. math.trunc rounds toward zero, returning the closest ~numbers.Integral that's between the function's argument and zero.

3141 - A Type Hierarchy for Numbers
PEP written by Jeffrey Yasskin.

Scheme's numerical tower, from the Guile manual.

Scheme's number datatypes from the R5RS Scheme specification.

The fractions Module

To fill out the hierarchy of numeric types, the fractions module provides a rational-number class. Rational numbers store their values as a numerator and denominator forming a fraction, and can exactly represent numbers such as 2/3 that floating-point numbers can only approximate.

The ~fractions.Fraction constructor takes two ~numbers.Integral values that will be the numerator and denominator of the resulting fraction. :

>>> from fractions import Fraction
>>> a = Fraction(2, 3)
>>> b = Fraction(2, 5)
>>> float(a), float(b)
(0.66666666666666663, 0.40000000000000002)
>>> a+b
Fraction(16, 15)
>>> a/b
Fraction(5, 3)

For converting floating-point numbers to rationals, the float type now has an ~float.as_integer_ratio method that returns the numerator and denominator for a fraction that evaluates to the same floating-point value:

>>> (2.5) .as_integer_ratio()
(5, 2)
>>> (3.1415) .as_integer_ratio()
(7074029114692207L, 2251799813685248L)
>>> (1./3) .as_integer_ratio()
(6004799503160661L, 18014398509481984L)

Note that values that can only be approximated by floating-point numbers, such as 1./3, are not simplified to the number being approximated; the fraction attempts to match the floating-point value exactly.

The fractions module is based upon an implementation by Sjoerd Mullender that was in Python's Demo/classes/ directory for a long time. This implementation was significantly updated by Jeffrey Yasskin.

Other Language Changes

Some smaller changes made to the core Python language are:

Optimizations

Interpreter Changes

Two command-line options have been reserved for use by other Python implementations. The !-J switch has been reserved for use by Jython for Jython-specific options, such as switches that are passed to the underlying JVM. -X has been reserved for options specific to a particular implementation of Python such as CPython, Jython, or IronPython. If either option is used with Python 2.6, the interpreter will report that the option isn't currently used.

Python can now be prevented from writing .pyc or .pyo files by supplying the -B switch to the Python interpreter, or by setting the PYTHONDONTWRITEBYTECODE environment variable before running the interpreter. This setting is available to Python programs as the sys.dont_write_bytecode variable, and Python code can change the value to modify the interpreter's behaviour. (Contributed by Neal Norwitz and Georg Brandl.)

The encoding used for standard input, output, and standard error can be specified by setting the PYTHONIOENCODING environment variable before running the interpreter. The value should be a string in the form <encoding> or <encoding>:<errorhandler>. The encoding part specifies the encoding's name, e.g. utf-8 or latin-1; the optional errorhandler part specifies what to do with characters that can't be handled by the encoding, and should be one of "error", "ignore", or "replace". (Contributed by Martin von Löwis.)

New and Improved Modules

As in every release, Python's standard library received a number of enhancements and bug fixes. Here's a partial list of the most notable changes, sorted alphabetically by module name. Consult the Misc/NEWS file in the source tree for a more complete list of changes, or look through the Subversion logs for all the details.

The ast module

The ast module provides an Abstract Syntax Tree representation of Python code, and Armin Ronacher contributed a set of helper functions that perform a variety of common tasks. These will be useful for HTML templating packages, code analyzers, and similar tools that process Python code.

The ~ast.parse function takes an expression and returns an AST. The ~ast.dump function outputs a representation of a tree, suitable for debugging:

import ast

t = ast.parse("""
d = {}
for i in 'abcdefghijklm':
    d[i + i] = ord(i) - ord('a') + 1
print d
""")
print ast.dump(t)

This outputs a deeply nested tree:

Module(body=[
  Assign(targets=[
    Name(id='d', ctx=Store())
   ], value=Dict(keys=[], values=[]))
  For(target=Name(id='i', ctx=Store()),
      iter=Str(s='abcdefghijklm'), body=[
    Assign(targets=[
      Subscript(value=
        Name(id='d', ctx=Load()),
          slice=
          Index(value=
            BinOp(left=Name(id='i', ctx=Load()), op=Add(),
             right=Name(id='i', ctx=Load()))), ctx=Store())
     ], value=
     BinOp(left=
      BinOp(left=
       Call(func=
        Name(id='ord', ctx=Load()), args=[
          Name(id='i', ctx=Load())
         ], keywords=[], starargs=None, kwargs=None),
       op=Sub(), right=Call(func=
        Name(id='ord', ctx=Load()), args=[
          Str(s='a')
         ], keywords=[], starargs=None, kwargs=None)),
       op=Add(), right=Num(n=1)))
    ], orelse=[])
   Print(dest=None, values=[
     Name(id='d', ctx=Load())
   ], nl=True)
 ])

The ~ast.literal_eval method takes a string or an AST representing a literal expression, parses and evaluates it, and returns the resulting value. A literal expression is a Python expression containing only strings, numbers, dictionaries, etc. but no statements or function calls. If you need to evaluate an expression but cannot accept the security risk of using an eval call, ~ast.literal_eval will handle it safely:

>>> literal = '("a", "b", {2:4, 3:8, 1:2})'
>>> print ast.literal_eval(literal)
('a', 'b', {1: 2, 2: 4, 3: 8})
>>> print ast.literal_eval('"a" + "b"')
Traceback (most recent call last):
  ...
ValueError: malformed string

The module also includes ~ast.NodeVisitor and ~ast.NodeTransformer classes for traversing and modifying an AST, and functions for common transformations such as changing line numbers.

The !future_builtins module

Python 3.0 makes many changes to the repertoire of built-in functions, and most of the changes can't be introduced in the Python 2.x series because they would break compatibility. The !future_builtins module provides versions of these built-in functions that can be imported when writing 3.0-compatible code.

The functions in this module currently include:

The json module: JavaScript Object Notation

The new json module supports the encoding and decoding of Python types in JSON (Javascript Object Notation). JSON is a lightweight interchange format often used in web applications. For more information about JSON, see http://www.json.org.

json comes with support for decoding and encoding most built-in Python types. The following example encodes and decodes a dictionary:

>>> import json
>>> data = {"spam": "foo", "parrot": 42}
>>> in_json = json.dumps(data) # Encode the data
>>> in_json
'{"parrot": 42, "spam": "foo"}'
>>> json.loads(in_json) # Decode into a Python object
{"spam": "foo", "parrot": 42}

It's also possible to write your own decoders and encoders to support more types. Pretty-printing of the JSON strings is also supported.

json (originally called simplejson) was written by Bob Ippolito.

The plistlib module: A Property-List Parser

The .plist format is commonly used on Mac OS X to store basic data types (numbers, strings, lists, and dictionaries) by serializing them into an XML-based format. It resembles the XML-RPC serialization of data types.

Despite being primarily used on Mac OS X, the format has nothing Mac-specific about it and the Python implementation works on any platform that Python supports, so the plistlib module has been promoted to the standard library.

Using the module is simple:

import sys
import plistlib
import datetime as dt

# Create data structure
data_struct = dict(lastAccessed=dt.datetime.now(),
                   version=1,
                   categories=('Personal','Shared','Private'))

# Create string containing XML.
plist_str = plistlib.writePlistToString(data_struct)
new_struct = plistlib.readPlistFromString(plist_str)
print data_struct
print new_struct

# Write data structure to a file and read it back.
plistlib.writePlist(data_struct, '/tmp/customizations.plist')
new_struct = plistlib.readPlist('/tmp/customizations.plist')

# read/writePlist accepts file-like objects as well as paths.
plistlib.writePlist(data_struct, sys.stdout)

ctypes Enhancements

Thomas Heller continued to maintain and enhance the ctypes module.

ctypes now supports a ~ctypes.c_bool datatype that represents the C99 bool type. (Contributed by David Remahl; 1649190.)

The ctypes string, buffer and array types have improved support for extended slicing syntax, where various combinations of (start, stop, step) are supplied. (Implemented by Thomas Wouters.)

All ctypes data types now support ~ctypes._CData.from_buffer and ~ctypes._CData.from_buffer_copy methods that create a ctypes instance based on a provided buffer object. ~ctypes._CData.from_buffer_copy copies the contents of the object, while ~ctypes._CData.from_buffer will share the same memory area.

A new calling convention tells ctypes to clear the errno or Win32 LastError variables at the outset of each wrapped call. (Implemented by Thomas Heller; 1798.)

You can now retrieve the Unix errno variable after a function call. When creating a wrapped function, you can supply use_errno=True as a keyword parameter to the DLL function and then call the module-level methods ~ctypes.set_errno and ~ctypes.get_errno to set and retrieve the error value.

The Win32 LastError variable is similarly supported by the DLL, ~ctypes.OleDLL, and ~ctypes.WinDLL functions. You supply use_last_error=True as a keyword parameter and then call the module-level methods ~ctypes.set_last_error and ~ctypes.get_last_error.

The ~ctypes.byref function, used to retrieve a pointer to a ctypes instance, now has an optional offset parameter that is a byte count that will be added to the returned pointer.

Improved SSL Support

Bill Janssen made extensive improvements to Python 2.6's support for the Secure Sockets Layer by adding a new module, ssl, that's built atop the OpenSSL library. This new module provides more control over the protocol negotiated, the X.509 certificates used, and has better support for writing SSL servers (as opposed to clients) in Python. The existing SSL support in the socket module hasn't been removed and continues to work, though it will be removed in Python 3.0.

To use the new module, you must first create a TCP connection in the usual way and then pass it to the !ssl.wrap_socket function. It's possible to specify whether a certificate is required, and to obtain certificate info by calling the ~ssl.SSLSocket.getpeercert method.

The documentation for the ssl module.

Deprecations and Removals

Build and C API Changes

Changes to Python's build process and to the C API include:

Port-Specific Changes: Windows

Port-Specific Changes: Mac OS X

Port-Specific Changes: IRIX

A number of old IRIX-specific modules were deprecated and will be removed in Python 3.0: !al and !AL, !cd, !cddb, !cdplayer, !CL and !cl, !DEVICE, !ERRNO, !FILE, !FL and !fl, !flp, !fm, !GET, !GLWS, !GL and !gl, !IN, !IOCTL, !jpeg, !panelparser, !readcd, !SV and !sv, !torgb, !videoreader, and !WAIT.

Porting to Python 2.6

This section lists previously described changes and other bugfixes that may require changes to your code:

For applications that embed Python:

Acknowledgements

The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this article: Georg Brandl, Steve Brown, Nick Coghlan, Ralph Corderoy, Jim Jewett, Kent Johnson, Chris Lambacher, Martin Michlmayr, Antoine Pitrou, Brian Warner.

What's New in Python 2.7

Author
A.M. Kuchling (amk at amk.ca)

This article explains the new features in Python 2.7. Python 2.7 was released on July 3, 2010.

Numeric handling has been improved in many ways, for both floating-point numbers and for the ~decimal.Decimal class. There are some useful additions to the standard library, such as a greatly enhanced unittest module, the argparse module for parsing command-line options, convenient ~collections.OrderedDict and ~collections.Counter classes in the collections module, and many other improvements.

Python 2.7 is planned to be the last of the 2.x releases, so we worked on making it a good release for the long term. To help with porting to Python 3, several new features from the Python 3.x series have been included in 2.7.

This article doesn't attempt to provide a complete specification of the new features, but instead provides a convenient overview. For full details, you should refer to the documentation for Python 2.7 at https://docs.python.org. If you want to understand the rationale for the design and implementation, refer to the PEP for a particular new feature or the issue on https://bugs.python.org in which a change was discussed. Whenever possible, "What's New in Python" links to the bug/patch item for each change.

The Future for Python 2.x

Python 2.7 is the last major release in the 2.x series, as the Python maintainers have shifted the focus of their new feature development efforts to the Python 3.x series. This means that while Python 2 continues to receive bug fixes, and to be updated to build correctly on new hardware and versions of supported operated systems, there will be no new full feature releases for the language or standard library.

However, while there is a large common subset between Python 2.7 and Python 3, and many of the changes involved in migrating to that common subset, or directly to Python 3, can be safely automated, some other changes (notably those associated with Unicode handling) may require careful consideration, and preferably robust automated regression test suites, to migrate effectively.

This means that Python 2.7 will remain in place for a long time, providing a stable and supported base platform for production systems that have not yet been ported to Python 3. The full expected lifecycle of the Python 2.7 series is detailed in 373.

Some key consequences of the long-term significance of 2.7 are:

For projects wishing to migrate from Python 2 to Python 3, or for library and framework developers wishing to support users on both Python 2 and Python 3, there are a variety of tools and guides available to help decide on a suitable approach and manage some of the technical details involved. The recommended starting point is the pyporting-howto HOWTO guide.

Changes to the Handling of Deprecation Warnings

For Python 2.7, a policy decision was made to silence warnings only of interest to developers by default. DeprecationWarning and its descendants are now ignored unless otherwise requested, preventing users from seeing warnings triggered by an application. This change was also made in the branch that became Python 3.2. (Discussed on stdlib-sig and carried out in 7319.)

In previous releases, DeprecationWarning messages were enabled by default, providing Python developers with a clear indication of where their code may break in a future major version of Python.

However, there are increasingly many users of Python-based applications who are not directly involved in the development of those applications. DeprecationWarning messages are irrelevant to such users, making them worry about an application that's actually working correctly and burdening application developers with responding to these concerns.

You can re-enable display of DeprecationWarning messages by running Python with the -Wdefault <-W> (short form: -Wd <-W>) switch, or by setting the PYTHONWARNINGS environment variable to "default" (or "d") before running Python. Python code can also re-enable them by calling warnings.simplefilter('default').

The unittest module also automatically reenables deprecation warnings when running tests.

Python 3.1 Features

Much as Python 2.6 incorporated features from Python 3.0, version 2.7 incorporates some of the new features in Python 3.1. The 2.x series continues to provide tools for migrating to the 3.x series.

A partial list of 3.1 features that were backported to 2.7:

Other new Python3-mode warnings include:

PEP 372: Adding an Ordered Dictionary to collections

Regular Python dictionaries iterate over key/value pairs in arbitrary order. Over the years, a number of authors have written alternative implementations that remember the order that the keys were originally inserted. Based on the experiences from those implementations, 2.7 introduces a new ~collections.OrderedDict class in the collections module.

The ~collections.OrderedDict API provides the same interface as regular dictionaries but iterates over keys and values in a guaranteed order depending on when a key was first inserted:

>>> from collections import OrderedDict
>>> d = OrderedDict([('first', 1),
...                  ('second', 2),
...                  ('third', 3)])
>>> d.items()
[('first', 1), ('second', 2), ('third', 3)]

If a new entry overwrites an existing entry, the original insertion position is left unchanged:

>>> d['second'] = 4
>>> d.items()
[('first', 1), ('second', 4), ('third', 3)]

Deleting an entry and reinserting it will move it to the end:

>>> del d['second']
>>> d['second'] = 5
>>> d.items()
[('first', 1), ('third', 3), ('second', 5)]

The ~collections.OrderedDict.popitem method has an optional last argument that defaults to True. If last is true, the most recently added key is returned and removed; if it's false, the oldest key is selected:

>>> od = OrderedDict([(x,0) for x in range(20)])
>>> od.popitem()
(19, 0)
>>> od.popitem()
(18, 0)
>>> od.popitem(last=False)
(0, 0)
>>> od.popitem(last=False)
(1, 0)

Comparing two ordered dictionaries checks both the keys and values, and requires that the insertion order was the same:

>>> od1 = OrderedDict([('first', 1),
...                    ('second', 2),
...                    ('third', 3)])
>>> od2 = OrderedDict([('third', 3),
...                    ('first', 1),
...                    ('second', 2)])
>>> od1 == od2
False
>>> # Move 'third' key to the end
>>> del od2['third']; od2['third'] = 3
>>> od1 == od2
True

Comparing an ~collections.OrderedDict with a regular dictionary ignores the insertion order and just compares the keys and values.

How does the ~collections.OrderedDict work? It maintains a doubly linked list of keys, appending new keys to the list as they're inserted. A secondary dictionary maps keys to their corresponding list node, so deletion doesn't have to traverse the entire linked list and therefore remains O(1).

The standard library now supports use of ordered dictionaries in several modules.

372 - Adding an ordered dictionary to collections
PEP written by Armin Ronacher and Raymond Hettinger; implemented by Raymond Hettinger.

PEP 378: Format Specifier for Thousands Separator

To make program output more readable, it can be useful to add separators to large numbers, rendering them as 18,446,744,073,709,551,616 instead of 18446744073709551616.

The fully general solution for doing this is the locale module, which can use different separators ("," in North America, "." in Europe) and different grouping sizes, but locale is complicated to use and unsuitable for multi-threaded applications where different threads are producing output for different locales.

Therefore, a simple comma-grouping mechanism has been added to the mini-language used by the str.format method. When formatting a floating-point number, simply include a comma between the width and the precision:

>>> '{:20,.2f}'.format(18446744073709551616.0)
'18,446,744,073,709,551,616.00'

When formatting an integer, include the comma after the width:

>>> '{:20,d}'.format(18446744073709551616) '18,446,744,073,709,551,616'

This mechanism is not adaptable at all; commas are always used as the separator and the grouping is always into three-digit groups. The comma-formatting mechanism isn't as general as the locale module, but it's easier to use.

378 - Format Specifier for Thousands Separator
PEP written by Raymond Hettinger; implemented by Eric Smith.

PEP 389: The argparse Module for Parsing Command Lines

The argparse module for parsing command-line arguments was added as a more powerful replacement for the optparse module.

This means Python now supports three different modules for parsing command-line arguments: getopt, optparse, and argparse. The getopt module closely resembles the C library's !getopt function, so it remains useful if you're writing a Python prototype that will eventually be rewritten in C. optparse becomes redundant, but there are no plans to remove it because there are many scripts still using it, and there's no automated way to update these scripts. (Making the argparse API consistent with optparse's interface was discussed but rejected as too messy and difficult.)

In short, if you're writing a new script and don't need to worry about compatibility with earlier versions of Python, use argparse instead of optparse.

Here's an example:

import argparse

parser = argparse.ArgumentParser(description='Command-line example.')

# Add optional switches
parser.add_argument('-v', action='store_true', dest='is_verbose',
                    help='produce verbose output')
parser.add_argument('-o', action='store', dest='output',
                    metavar='FILE',
                    help='direct output to FILE instead of stdout')
parser.add_argument('-C', action='store', type=int, dest='context',
                    metavar='NUM', default=0,
                    help='display NUM lines of added context')

# Allow any number of additional arguments.
parser.add_argument(nargs='*', action='store', dest='inputs',
                    help='input filenames (default is stdin)')

args = parser.parse_args()
print args.__dict__

Unless you override it, !-h and !--help switches are automatically added, and produce neatly formatted output:

-> ./python.exe argparse-example.py --help
usage: argparse-example.py [-h] [-v] [-o FILE] [-C NUM] [inputs [inputs ...]]

Command-line example.

positional arguments:
  inputs      input filenames (default is stdin)

optional arguments:
  -h, --help  show this help message and exit
  -v          produce verbose output
  -o FILE     direct output to FILE instead of stdout
  -C NUM      display NUM lines of added context

As with optparse, the command-line switches and arguments are returned as an object with attributes named by the dest parameters:

-> ./python.exe argparse-example.py -v
{'output': None,
 'is_verbose': True,
 'context': 0,
 'inputs': []}

-> ./python.exe argparse-example.py -v -o /tmp/output -C 4 file1 file2
{'output': '/tmp/output',
 'is_verbose': True,
 'context': 4,
 'inputs': ['file1', 'file2']}

argparse has much fancier validation than optparse; you can specify an exact number of arguments as an integer, 0 or more arguments by passing '*', 1 or more by passing '+', or an optional argument with '?'. A top-level parser can contain sub-parsers to define subcommands that have different sets of switches, as in svn commit, svn checkout, etc. You can specify an argument's type as ~argparse.FileType, which will automatically open files for you and understands that '-' means standard input or output.

argparse documentation
The documentation page of the argparse module.

upgrading-optparse-code
Part of the Python documentation, describing how to convert code that uses optparse.

389 - argparse - New Command Line Parsing Module
PEP written and implemented by Steven Bethard.

PEP 391: Dictionary-Based Configuration For Logging

The logging module is very flexible; applications can define a tree of logging subsystems, and each logger in this tree can filter out certain messages, format them differently, and direct messages to a varying number of handlers.

All this flexibility can require a lot of configuration. You can write Python statements to create objects and set their properties, but a complex set-up requires verbose but boring code. logging also supports a ~logging.config.fileConfig function that parses a file, but the file format doesn't support configuring filters, and it's messier to generate programmatically.

Python 2.7 adds a ~logging.config.dictConfig function that uses a dictionary to configure logging. There are many ways to produce a dictionary from different sources: construct one with code; parse a file containing JSON; or use a YAML parsing library if one is installed. For more information see logging-config-api.

The following example configures two loggers, the root logger and a logger named "network". Messages sent to the root logger will be sent to the system log using the syslog protocol, and messages to the "network" logger will be written to a network.log file that will be rotated once the log reaches 1MB.

import logging
import logging.config

configdict = {
 'version': 1,    # Configuration schema in use; must be 1 for now
 'formatters': {
     'standard': {
         'format': ('%(asctime)s %(name)-15s '
                    '%(levelname)-8s %(message)s')}},

 'handlers': {'netlog': {'backupCount': 10,
                     'class': 'logging.handlers.RotatingFileHandler',
                     'filename': '/logs/network.log',
                     'formatter': 'standard',
                     'level': 'INFO',
                     'maxBytes': 1000000},
              'syslog': {'class': 'logging.handlers.SysLogHandler',
                         'formatter': 'standard',
                         'level': 'ERROR'}},

 # Specify all the subordinate loggers
 'loggers': {
             'network': {
                         'handlers': ['netlog']
             }
 },
 # Specify properties of the root logger
 'root': {
          'handlers': ['syslog']
 },
}

# Set up configuration
logging.config.dictConfig(configdict)

# As an example, log two error messages
logger = logging.getLogger('/')
logger.error('Database not found')

netlogger = logging.getLogger('network')
netlogger.error('Connection failed')

Three smaller enhancements to the logging module, all implemented by Vinay Sajip, are:

391 - Dictionary-Based Configuration For Logging
PEP written and implemented by Vinay Sajip.

PEP 3106: Dictionary Views

The dictionary methods ~dict.keys, ~dict.values, and ~dict.items are different in Python 3.x. They return an object called a view instead of a fully materialized list.

It's not possible to change the return values of ~dict.keys, ~dict.values, and ~dict.items in Python 2.7 because too much code would break. Instead the 3.x versions were added under the new names !viewkeys, !viewvalues, and !viewitems.

>>> d = dict((i*10, chr(65+i)) for i in range(26))
>>> d
{0: 'A', 130: 'N', 10: 'B', 140: 'O', 20: ..., 250: 'Z'}
>>> d.viewkeys()
dict_keys([0, 130, 10, 140, 20, 150, 30, ..., 250])

Views can be iterated over, but the key and item views also behave like sets. The & operator performs intersection, and | performs a union:

>>> d1 = dict((i*10, chr(65+i)) for i in range(26))
>>> d2 = dict((i**.5, i) for i in range(1000))
>>> d1.viewkeys() & d2.viewkeys()
set([0.0, 10.0, 20.0, 30.0])
>>> d1.viewkeys() | range(0, 30)
set([0, 1, 130, 3, 4, 5, 6, ..., 120, 250])

The view keeps track of the dictionary and its contents change as the dictionary is modified:

>>> vk = d.viewkeys()
>>> vk
dict_keys([0, 130, 10, ..., 250])
>>> d[260] = '&'
>>> vk
dict_keys([0, 130, 260, 10, ..., 250])

However, note that you can't add or remove keys while you're iterating over the view:

>>> for k in vk:
...     d[k*2] = k
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: dictionary changed size during iteration

You can use the view methods in Python 2.x code, and the 2to3 converter will change them to the standard ~dict.keys, ~dict.values, and ~dict.items methods.

3106 - Revamping dict.keys(), .values() and .items()
PEP written by Guido van Rossum. Backported to 2.7 by Alexandre Vassalotti; 1967.

PEP 3137: The memoryview Object

The memoryview object provides a view of another object's memory content that matches the bytes type's interface.

>>> import string >>> m = memoryview(string.letters) >>> m <memory at 0x37f850> >>> len(m) # Returns length of underlying object 52 >>> m[0], m[25], m[26] # Indexing returns one byte ('a', 'z', 'A') >>> m2 = m[0:26] # Slicing returns another memoryview >>> m2 <memory at 0x37f080>

The content of the view can be converted to a string of bytes or a list of integers:

>>> m2.tobytes() 'abcdefghijklmnopqrstuvwxyz' >>> m2.tolist() [97, 98, 99, 100, 101, 102, 103, ... 121, 122] >>>

memoryview objects allow modifying the underlying object if it's a mutable object.

>>> m2[0] = 75 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: cannot modify read-only memory >>> b = bytearray(string.letters) # Creating a mutable object >>> b bytearray(b'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ') >>> mb = memoryview(b) >>> mb[0] = '*' # Assign to view, changing the bytearray. >>> b[0:5] # The bytearray has been changed. bytearray(b'*bcde') >>>

3137 - Immutable Bytes and Mutable Buffer
PEP written by Guido van Rossum. Implemented by Travis Oliphant, Antoine Pitrou and others. Backported to 2.7 by Antoine Pitrou; 2396.

Other Language Changes

Some smaller changes made to the core Python language are:

Interpreter Changes

A new environment variable, PYTHONWARNINGS, allows controlling warnings. It should be set to a string containing warning settings, equivalent to those used with the -W switch, separated by commas. (Contributed by Brian Curtin; 7301.)

For example, the following setting will print warnings every time they occur, but turn warnings from the Cookie <http.cookies> module into an error. (The exact syntax for setting an environment variable varies across operating systems and shells.)

export PYTHONWARNINGS=all,error:::Cookie:0

Optimizations

Several performance enhancements have been added:

New and Improved Modules

As in every release, Python's standard library received a number of enhancements and bug fixes. Here's a partial list of the most notable changes, sorted alphabetically by module name. Consult the Misc/NEWS file in the source tree for a more complete list of changes, or look through the Subversion logs for all the details.

New module: importlib

Python 3.1 includes the importlib package, a re-implementation of the logic underlying Python's import statement. importlib is useful for implementers of Python interpreters and to users who wish to write new importers that can participate in the import process. Python 2.7 doesn't contain the complete importlib package, but instead has a tiny subset that contains a single function, ~importlib.import_module.

import_module(name, package=None) imports a module. name is a string containing the module or package's name. It's possible to do relative imports by providing a string that begins with a . character, such as ..utils.errors. For relative imports, the package argument must be provided and is the name of the package that will be used as the anchor for the relative import. ~importlib.import_module both inserts the imported module into sys.modules and returns the module object.

Here are some examples:

>>> from importlib import import_module
>>> anydbm = import_module('anydbm')  # Standard absolute import
>>> anydbm
<module 'anydbm' from '/p/python/Lib/anydbm.py'>
>>> # Relative import
>>> file_util = import_module('..file_util', 'distutils.command')
>>> file_util
<module 'distutils.file_util' from '/python/Lib/distutils/file_util.pyc'>

importlib was implemented by Brett Cannon and introduced in Python 3.1.

New module: sysconfig

The sysconfig module has been pulled out of the Distutils package, becoming a new top-level module in its own right. sysconfig provides functions for getting information about Python's build process: compiler switches, installation paths, the platform name, and whether Python is running from its source directory.

Some of the functions in the module are:

Consult the sysconfig documentation for more details and for a complete list of functions.

The Distutils package and sysconfig are now maintained by Tarek Ziadé, who has also started a Distutils2 package (source repository at https://hg.python.org/distutils2/) for developing a next-generation version of Distutils.

ttk: Themed Widgets for Tk

Tcl/Tk 8.5 includes a set of themed widgets that re-implement basic Tk widgets but have a more customizable appearance and can therefore more closely resemble the native platform's widgets. This widget set was originally called Tile, but was renamed to Ttk (for "themed Tk") on being added to Tcl/Tck release 8.5.

To learn more, read the ~tkinter.ttk module documentation. You may also wish to read the Tcl/Tk manual page describing the Ttk theme engine, available at https://www.tcl.tk/man/tcl8.5/TkCmd/ttk_intro.html. Some screenshots of the Python/Ttk code in use are at https://code.google.com/archive/p/python-ttk/wikis/Screenshots.wiki.

The tkinter.ttk module was written by Guilherme Polo and added in 2983. An alternate version called Tile.py, written by Martin Franklin and maintained by Kevin Walzer, was proposed for inclusion in 2618, but the authors argued that Guilherme Polo's work was more comprehensive.

Updated module: unittest

The unittest module was greatly enhanced; many new features were added. Most of these features were implemented by Michael Foord, unless otherwise noted. The enhanced version of the module is downloadable separately for use with Python versions 2.4 to 2.6, packaged as the !unittest2 package, from unittest2.

When used from the command line, the module can automatically discover tests. It's not as fancy as py.test or nose, but provides a simple way to run tests kept within a set of package directories. For example, the following command will search the test/ subdirectory for any importable test files named test*.py:

python -m unittest discover -s test

Consult the unittest module documentation for more details. (Developed in 6001.)

The ~unittest.main function supports some other new options:

The progress messages now show 'x' for expected failures and 'u' for unexpected successes when run in verbose mode. (Contributed by Benjamin Peterson.)

Test cases can raise the ~unittest.SkipTest exception to skip a test (1034053).

The error messages for ~unittest.TestCase.assertEqual, ~unittest.TestCase.assertTrue, and ~unittest.TestCase.assertFalse failures now provide more information. If you set the ~unittest.TestCase.longMessage attribute of your ~unittest.TestCase classes to true, both the standard error message and any additional message you provide will be printed for failures. (Added by Michael Foord; 5663.)

The ~unittest.TestCase.assertRaises method now returns a context handler when called without providing a callable object to run. For example, you can write this:

with self.assertRaises(KeyError):
    {}['foo']

(Implemented by Antoine Pitrou; 4444.)

Module- and class-level setup and teardown fixtures are now supported. Modules can contain ~unittest.setUpModule and ~unittest.tearDownModule functions. Classes can have ~unittest.TestCase.setUpClass and ~unittest.TestCase.tearDownClass methods that must be defined as class methods (using @classmethod or equivalent). These functions and methods are invoked when the test runner switches to a test case in a different module or class.

The methods ~unittest.TestCase.addCleanup and ~unittest.TestCase.doCleanups were added. ~unittest.TestCase.addCleanup lets you add cleanup functions that will be called unconditionally (after ~unittest.TestCase.setUp if ~unittest.TestCase.setUp fails, otherwise after ~unittest.TestCase.tearDown). This allows for much simpler resource allocation and deallocation during tests (5679).

A number of new methods were added that provide more specialized tests. Many of these methods were written by Google engineers for use in their test suites; Gregory P. Smith, Michael Foord, and GvR worked on merging them into Python's version of unittest.

unittest.main now takes an optional exit argument. If false, ~unittest.main doesn't call sys.exit, allowing ~unittest.main to be used from the interactive interpreter. (Contributed by J. Pablo Fernández; 3379.)

~unittest.TestResult has new ~unittest.TestResult.startTestRun and ~unittest.TestResult.stopTestRun methods that are called immediately before and after a test run. (Contributed by Robert Collins; 5728.)

With all these changes, the unittest.py was becoming awkwardly large, so the module was turned into a package and the code split into several files (by Benjamin Peterson). This doesn't affect how the module is imported or used.

https://web.archive.org/web/20210619163128/http://www.voidspace.org.uk/python/articles/unittest2.shtml
Describes the new features, how to use them, and the rationale for various design decisions. (By Michael Foord.)

Updated module: ElementTree 1.3

The version of the ElementTree library included with Python was updated to version 1.3. Some of the new features are:

Fredrik Lundh develops ElementTree and produced the 1.3 version; you can read his article describing 1.3 at https://web.archive.org/web/20200703234532/http://effbot.org/zone/elementtree-13-intro.htm. Florent Xicluna updated the version included with Python, after discussions on python-dev and in 6472.)

Build and C API Changes

Changes to Python's build process and to the C API include:

Capsules

Python 3.1 adds a new C datatype, PyCapsule, for providing a C API to an extension module. A capsule is essentially the holder of a C void * pointer, and is made available as a module attribute; for example, the socket module's API is exposed as socket.CAPI, and unicodedata exposes ucnhash_CAPI. Other extensions can import the module, access its dictionary to get the capsule object, and then get the void * pointer, which will usually point to an array of pointers to the module's various API functions.

There is an existing data type already used for this, !PyCObject, but it doesn't provide type safety. Evil code written in pure Python could cause a segmentation fault by taking a !PyCObject from module A and somehow substituting it for the !PyCObject in module B. Capsules know their own name, and getting the pointer requires providing the name:

void *vtable;

if (!PyCapsule_IsValid(capsule, "mymodule.CAPI") {
        PyErr_SetString(PyExc_ValueError, "argument type invalid");
        return NULL;
}

vtable = PyCapsule_GetPointer(capsule, "mymodule.CAPI");

You are assured that vtable points to whatever you're expecting. If a different capsule was passed in, PyCapsule_IsValid would detect the mismatched name and return false. Refer to using-capsules for more information on using these objects.

Python 2.7 now uses capsules internally to provide various extension-module APIs, but the !PyCObject_AsVoidPtr was modified to handle capsules, preserving compile-time compatibility with the !PyCObject interface. Use of !PyCObject_AsVoidPtr will signal a PendingDeprecationWarning, which is silent by default.

Implemented in Python 3.1 and backported to 2.7 by Larry Hastings; discussed in 5630.

Port-Specific Changes: Windows

Port-Specific Changes: Mac OS X

Port-Specific Changes: FreeBSD

Other Changes and Fixes

Porting to Python 2.7

This section lists previously described changes and other bugfixes that may require changes to your code:

In the standard library:

For C extensions:

For applications that embed Python:

New Features Added to Python 2.7 Maintenance Releases

New features may be added to Python 2.7 maintenance releases when the situation genuinely calls for it. Any such additions must go through the Python Enhancement Proposal process, and make a compelling case for why they can't be adequately addressed by either adding the new feature solely to Python 3, or else by publishing it on the Python Package Index.

In addition to the specific proposals listed below, there is a general exemption allowing new -3 warnings to be added in any Python 2.7 maintenance release.

Two new environment variables for debug mode

In debug mode, the [xxx refs] statistic is not written by default, the !PYTHONSHOWREFCOUNT environment variable now must also be set. (Contributed by Victor Stinner; 31733.)

When Python is compiled with COUNT_ALLOC defined, allocation counts are no longer dumped by default anymore: the !PYTHONSHOWALLOCCOUNT environment variable must now also be set. Moreover, allocation counts are now dumped into stderr, rather than stdout. (Contributed by Victor Stinner; 31692.)

2.7.15

PEP 434: IDLE Enhancement Exception for All Branches

434 describes a general exemption for changes made to the IDLE development environment shipped along with Python. This exemption makes it possible for the IDLE developers to provide a more consistent user experience across all supported versions of Python 2 and 3.

For details of any IDLE changes, refer to the NEWS file for the specific release.

PEP 466: Network Security Enhancements for Python 2.7

466 describes a number of network security enhancement proposals that have been approved for inclusion in Python 2.7 maintenance releases, with the first of those changes appearing in the Python 2.7.7 release.

PEP 477: Backport ensurepip (PEP 453) to Python 2.7

477 approves the inclusion of the 453 ensurepip module and the improved documentation that was enabled by it in the Python 2.7 maintenance releases, appearing first in the Python 2.7.9 release.

Bootstrapping pip By Default

The new ensurepip module (defined in 453) provides a standard cross-platform mechanism to bootstrap the pip installer into Python installations. The version of pip included with Python 2.7.9 is pip 1.5.6, and future 2.7.x maintenance releases will update the bundled version to the latest version of pip that is available at the time of creating the release candidate.

By default, the commands pip, pipX and pipX.Y will be installed on all platforms (where X.Y stands for the version of the Python installation), along with the pip Python package and its dependencies.

For CPython source builds on POSIX systems <building-python-on-unix>, the make install and make altinstall commands do not bootstrap pip by default. This behaviour can be controlled through configure options, and overridden through Makefile options.

On Windows and Mac OS X, the CPython installers now default to installing pip along with CPython itself (users may opt out of installing it during the installation process). Window users will need to opt in to the automatic PATH modifications to have pip available from the command line by default, otherwise it can still be accessed through the Python launcher for Windows as py -m pip.

As discussed in the PEP <0477#disabling-ensurepip-by-downstream-distributors>, platform packagers may choose not to install these commands by default, as long as, when invoked, they provide clear and simple directions on how to install them on that platform (usually using the system package manager).

Documentation Changes

As part of this change, the installing-index and distributing-index sections of the documentation have been completely redesigned as short getting started and FAQ documents. Most packaging documentation has now been moved out to the Python Packaging Authority maintained Python Packaging User Guide and the documentation of the individual projects.

However, as this migration is currently still incomplete, the legacy versions of those guides remaining available as install-index and setuptools-index.

453 -- Explicit bootstrapping of pip in Python installations
PEP written by Donald Stufft and Nick Coghlan, implemented by Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily.

PEP 476: Enabling certificate verification by default for stdlib http clients

476 updated httplib <http> and modules which use it, such as urllib2 <urllib.request> and xmlrpclib <xmlrpc.client>, to now verify that the server presents a certificate which is signed by a Certificate Authority in the platform trust store and whose hostname matches the hostname being requested by default, significantly improving security for many applications. This change was made in the Python 2.7.9 release.

For applications which require the old previous behavior, they can pass an alternate context:

import urllib2
import ssl

# This disables all verification
context = ssl._create_unverified_context()

# This allows using a specific certificate for the host, which doesn't need
# to be in the trust store
context = ssl.create_default_context(cafile="/path/to/file.crt")

urllib2.urlopen("https://invalid-cert", context=context)

PEP 493: HTTPS verification migration tools for Python 2.7

493 provides additional migration tools to support a more incremental infrastructure upgrade process for environments containing applications and services relying on the historically permissive processing of server certificates when establishing client HTTPS connections. These additions were made in the Python 2.7.12 release.

These tools are intended for use in cases where affected applications and services can't be modified to explicitly pass a more permissive SSL context when establishing the connection.

For applications and services which can't be modified at all, the new PYTHONHTTPSVERIFY environment variable may be set to 0 to revert an entire Python process back to the default permissive behaviour of Python 2.7.8 and earlier.

For cases where the connection establishment code can't be modified, but the overall application can be, the new !ssl._https_verify_certificates function can be used to adjust the default behaviour at runtime.

New make regen-all build target

To simplify cross-compilation, and to ensure that CPython can reliably be compiled without requiring an existing version of Python to already be available, the autotools-based build system no longer attempts to implicitly recompile generated files based on file modification times.

Instead, a new make regen-all command has been added to force regeneration of these files when desired (e.g. after an initial version of Python has already been built based on the pregenerated versions).

More selective regeneration targets are also defined - see Makefile.pre.in for details.

(Contributed by Victor Stinner in 23404.)

2.7.14

Removal of make touch build target

The make touch build target previously used to request implicit regeneration of generated files by updating their modification times has been removed.

It has been replaced by the new make regen-all target.

(Contributed by Victor Stinner in 23404.)

2.7.14

Acknowledgements

The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this article: Nick Coghlan, Philip Jenvey, Ryan Lovett, R. David Murray, Hugh Secker-Walker.

What's New In Python 3.0

Author
Guido van Rossum

This article explains the new features in Python 3.0, compared to 2.6. Python 3.0, also known as "Python 3000" or "Py3K", is the first ever intentionally backwards incompatible Python release. Python 3.0 was released on December 3, 2008. There are more changes than in a typical release, and more that are important for all Python users. Nevertheless, after digesting the changes, you'll find that Python really hasn't changed all that much -- by and large, we're mostly fixing well-known annoyances and warts, and removing a lot of old cruft.

This article doesn't attempt to provide a complete specification of all new features, but instead tries to give a convenient overview. For full details, you should refer to the documentation for Python 3.0, and/or the many PEPs referenced in the text. If you want to understand the complete implementation and design rationale for a particular feature, PEPs usually have more details than the regular documentation; but note that PEPs usually are not kept up-to-date once a feature has been fully implemented.

Due to time constraints this document is not as complete as it should have been. As always for a new release, the Misc/NEWS file in the source distribution contains a wealth of detailed information about every small thing that was changed.

Common Stumbling Blocks

This section lists those few changes that are most likely to trip you up if you're used to Python 2.5.

Print Is A Function

The print statement has been replaced with a print function, with keyword arguments to replace most of the special syntax of the old print statement (3105). Examples:

Old: print "The answer is", 2*2
New: print("The answer is", 2*2)

Old: print x,           # Trailing comma suppresses newline
New: print(x, end=" ")  # Appends a space instead of a newline

Old: print              # Prints a newline
New: print()            # You must call the function!

Old: print >>sys.stderr, "fatal error"
New: print("fatal error", file=sys.stderr)

Old: print (x, y)       # prints repr((x, y))
New: print((x, y))      # Not the same as print(x, y)!

You can also customize the separator between items, e.g.:

print("There are <", 2**32, "> possibilities!", sep="")

which produces:

There are <4294967296> possibilities!

Note:

Views And Iterators Instead Of Lists

Some well-known APIs no longer return lists:

Ordering Comparisons

Python 3.0 has simplified the rules for ordering comparisons:

Integers

Text Vs. Data Instead Of Unicode Vs. 8-bit

Everything you thought you knew about binary data and Unicode has changed.

Overview Of Syntax Changes

This section gives a brief overview of every syntactic change in Python 3.0.

New Syntax

Changed Syntax

Removed Syntax

Changes Already Present In Python 2.6

Since many users presumably make the jump straight from Python 2.5 to Python 3.0, this section reminds the reader of new features that were originally designed for Python 3.0 but that were back-ported to Python 2.6. The corresponding sections in whats-new-in-2.6 should be consulted for longer descriptions.

Library Changes

Due to time constraints, this document does not exhaustively cover the very extensive changes to the standard library. 3108 is the reference for the major changes to the library. Here's a capsule review:

Some other changes to standard library modules, not covered by 3108:

3101: A New Approach To String Formatting

Changes To Exceptions

The APIs for raising and catching exception have been cleaned up and new powerful features added:

Miscellaneous Other Changes

Operators And Special Methods

Builtins

Build and C API Changes

Due to time constraints, here is a very incomplete list of changes to the C API.

Performance

The net result of the 3.0 generalizations is that Python 3.0 runs the pystone benchmark around 10% slower than Python 2.5. Most likely the biggest cause is the removal of special-casing for small integers. There's room for improvement, but it will happen after 3.0 is released!

Porting To Python 3.0

For porting existing Python 2.5 or 2.6 source code to Python 3.0, the best strategy is the following:

  1. (Prerequisite:) Start with excellent test coverage.
  2. Port to Python 2.6. This should be no more work than the average port from Python 2.x to Python 2.(x+1). Make sure all your tests pass.
  3. (Still using 2.6:) Turn on the !-3 command line switch. This enables warnings about features that will be removed (or change) in 3.0. Run your test suite again, and fix code that you get warnings about until there are no warnings left, and all your tests still pass.
  4. Run the 2to3 source-to-source translator over your source code tree. Run the result of the translation under Python 3.0. Manually fix up any remaining issues, fixing problems until all tests pass again.

It is not recommended to try to write source code that runs unchanged under both Python 2.6 and 3.0; you'd have to use a very contorted coding style, e.g. avoiding print statements, metaclasses, and much more. If you are maintaining a library that needs to support both Python 2.6 and Python 3.0, the best approach is to modify step 3 above by editing the 2.6 version of the source code and running the 2to3 translator again, rather than editing the 3.0 version of the source code.

For porting C extensions to Python 3.0, please see cporting-howto.

What's New In Python 3.1

Author
Raymond Hettinger

This article explains the new features in Python 3.1, compared to 3.0. Python 3.1 was released on June 27, 2009.

PEP 372: Ordered Dictionaries

Regular Python dictionaries iterate over key/value pairs in arbitrary order. Over the years, a number of authors have written alternative implementations that remember the order that the keys were originally inserted. Based on the experiences from those implementations, a new collections.OrderedDict class has been introduced.

The OrderedDict API is substantially the same as regular dictionaries but will iterate over keys and values in a guaranteed order depending on when a key was first inserted. If a new entry overwrites an existing entry, the original insertion position is left unchanged. Deleting an entry and reinserting it will move it to the end.

The standard library now supports use of ordered dictionaries in several modules. The configparser module uses them by default. This lets configuration files be read, modified, and then written back in their original order. The _asdict() method for collections.namedtuple now returns an ordered dictionary with the values appearing in the same order as the underlying tuple indices. The json module is being built-out with an object_pairs_hook to allow OrderedDicts to be built by the decoder. Support was also added for third-party tools like PyYAML.

372 - Ordered Dictionaries
PEP written by Armin Ronacher and Raymond Hettinger. Implementation written by Raymond Hettinger.

Since an ordered dictionary remembers its insertion order, it can be used in conjunction with sorting to make a sorted dictionary:

>>> # regular unsorted dictionary
>>> d = {'banana': 3, 'apple':4, 'pear': 1, 'orange': 2}

>>> # dictionary sorted by key
>>> OrderedDict(sorted(d.items(), key=lambda t: t[0]))
OrderedDict([('apple', 4), ('banana', 3), ('orange', 2), ('pear', 1)])

>>> # dictionary sorted by value
>>> OrderedDict(sorted(d.items(), key=lambda t: t[1]))
OrderedDict([('pear', 1), ('orange', 2), ('banana', 3), ('apple', 4)])

>>> # dictionary sorted by length of the key string
>>> OrderedDict(sorted(d.items(), key=lambda t: len(t[0])))
OrderedDict([('pear', 1), ('apple', 4), ('orange', 2), ('banana', 3)])

The new sorted dictionaries maintain their sort order when entries are deleted. But when new keys are added, the keys are appended to the end and the sort is not maintained.

PEP 378: Format Specifier for Thousands Separator

The built-in format function and the str.format method use a mini-language that now includes a simple, non-locale aware way to format a number with a thousands separator. That provides a way to humanize a program's output, improving its professional appearance and readability:

>>> format(1234567, ',d')
'1,234,567'
>>> format(1234567.89, ',.2f')
'1,234,567.89'
>>> format(12345.6 + 8901234.12j, ',f')
'12,345.600000+8,901,234.120000j'
>>> format(Decimal('1234567.89'), ',f')
'1,234,567.89'

The supported types are int, float, complex and decimal.Decimal.

Discussions are underway about how to specify alternative separators like dots, spaces, apostrophes, or underscores. Locale-aware applications should use the existing n format specifier which already has some support for thousands separators.

378 - Format Specifier for Thousands Separator
PEP written by Raymond Hettinger and implemented by Eric Smith and Mark Dickinson.

Other Language Changes

Some smaller changes made to the core Python language are:

New, Improved, and Deprecated Modules

(Suggested by Antoine Pitrou and Jesse Noller. Implemented by Jack Diederich; 5228.)

Optimizations

Major performance enhancements have been added:

IDLE

Build and C API Changes

Changes to Python's build process and to the C API include:

Porting to Python 3.1

This section lists previously described changes and other bugfixes that may require changes to your code:

What's New In Python 3.10

Editor
Pablo Galindo Salgado

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the git log when researching a change.

This article explains the new features in Python 3.10, compared to 3.9. Python 3.10 was released on October 4, 2021. For full details, see the changelog <changelog>.

Summary -- Release highlights

New syntax features:

New features in the standard library:

Interpreter improvements:

New typing features:

Important deprecations, removals or restrictions:

New Features

Parenthesized context managers

Using enclosing parentheses for continuation across multiple lines in context managers is now supported. This allows formatting a long collection of context managers in multiple lines in a similar way as it was previously possible with import statements. For instance, all these examples are now valid:

with (CtxManager() as example):
    ...

with (
    CtxManager1(),
    CtxManager2()
):
    ...

with (CtxManager1() as example,
      CtxManager2()):
    ...

with (CtxManager1(),
      CtxManager2() as example):
    ...

with (
    CtxManager1() as example1,
    CtxManager2() as example2
):
    ...

it is also possible to use a trailing comma at the end of the enclosed group:

with (
    CtxManager1() as example1,
    CtxManager2() as example2,
    CtxManager3() as example3,
):
    ...

This new syntax uses the non LL(1) capacities of the new parser. Check 617 for more details.

(Contributed by Guido van Rossum, Pablo Galindo and Lysandros Nikolaou in 12782 and 40334.)

Better error messages

SyntaxErrors

When parsing code that contains unclosed parentheses or brackets the interpreter now includes the location of the unclosed bracket of parentheses instead of displaying SyntaxError: unexpected EOF while parsing or pointing to some incorrect location. For instance, consider the following code (notice the unclosed '{'):

expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4,
            38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6,
some_other_code = foo()

Previous versions of the interpreter reported confusing places as the location of the syntax error:

File "example.py", line 3
    some_other_code = foo()
                    ^
SyntaxError: invalid syntax

but in Python 3.10 a more informative error is emitted:

File "example.py", line 1
    expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4,
               ^
SyntaxError: '{' was never closed

In a similar way, errors involving unclosed string literals (single and triple quoted) now point to the start of the string instead of reporting EOF/EOL.

These improvements are inspired by previous work in the PyPy interpreter.

(Contributed by Pablo Galindo in 42864 and Batuhan Taskaya in 40176.)

SyntaxError exceptions raised by the interpreter will now highlight the full error range of the expression that constitutes the syntax error itself, instead of just where the problem is detected. In this way, instead of displaying (before Python 3.10):

>>> foo(x, z for z in range(10), t, w)
  File "<stdin>", line 1
    foo(x, z for z in range(10), t, w)
           ^
SyntaxError: Generator expression must be parenthesized

now Python 3.10 will display the exception as:

>>> foo(x, z for z in range(10), t, w)
  File "<stdin>", line 1
    foo(x, z for z in range(10), t, w)
           ^^^^^^^^^^^^^^^^^^^^
SyntaxError: Generator expression must be parenthesized

This improvement was contributed by Pablo Galindo in 43914.

A considerable amount of new specialized messages for SyntaxError exceptions have been incorporated. Some of the most notable ones are as follows:

IndentationErrors

Many IndentationError exceptions now have more context regarding what kind of block was expecting an indentation, including the location of the statement:

>>> def foo():
...    if lel:
...    x = 2
  File "<stdin>", line 3
    x = 2
    ^
IndentationError: expected an indented block after 'if' statement in line 2

AttributeErrors

When printing AttributeError, !PyErr_Display will offer suggestions of similar attribute names in the object that the exception was raised from:

>>> collections.namedtoplo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'collections' has no attribute 'namedtoplo'. Did you mean: namedtuple?

(Contributed by Pablo Galindo in 38530.)

[!WARNING] Notice this won't work if !PyErr_Display is not called to display the error which can happen if some other custom error display function is used. This is a common scenario in some REPLs like IPython.

NameErrors

When printing NameError raised by the interpreter, !PyErr_Display will offer suggestions of similar variable names in the function that the exception was raised from:

>>> schwarzschild_black_hole = None
>>> schwarschild_black_hole
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'schwarschild_black_hole' is not defined. Did you mean: schwarzschild_black_hole?

(Contributed by Pablo Galindo in 38530.)

[!WARNING] Notice this won't work if !PyErr_Display is not called to display the error, which can happen if some other custom error display function is used. This is a common scenario in some REPLs like IPython.

PEP 626: Precise line numbers for debugging and other tools

PEP 626 brings more precise and reliable line numbers for debugging, profiling and coverage tools. Tracing events, with the correct line number, are generated for all lines of code executed and only for lines of code that are executed.

The ~frame.f_lineno attribute of frame objects will always contain the expected line number.

The !codeobject.co_lnotab attribute of code objects <code-objects> is deprecated and will be removed in 3.12. Code that needs to convert from offset to line number should use the new ~codeobject.co_lines method instead.

PEP 634: Structural Pattern Matching

Structural pattern matching has been added in the form of a match statement and case statements of patterns with associated actions. Patterns consist of sequences, mappings, primitive data types as well as class instances. Pattern matching enables programs to extract information from complex data types, branch on the structure of data, and apply specific actions based on different forms of data.

Syntax and operations

The generic syntax of pattern matching is:

match subject:
    case <pattern_1>:
        <action_1>
    case <pattern_2>:
        <action_2>
    case <pattern_3>:
        <action_3>
    case _:
        <action_wildcard>

A match statement takes an expression and compares its value to successive patterns given as one or more case blocks. Specifically, pattern matching operates by:

  1. using data with type and shape (the subject)
  2. evaluating the subject in the match statement
  3. comparing the subject with each pattern in a case statement from top to bottom until a match is confirmed.
  4. executing the action associated with the pattern of the confirmed match
  5. If an exact match is not confirmed, the last case, a wildcard _, if provided, will be used as the matching case. If an exact match is not confirmed and a wildcard case does not exist, the entire match block is a no-op.

Declarative approach

Readers may be aware of pattern matching through the simple example of matching a subject (data object) to a literal (pattern) with the switch statement found in C, Java or JavaScript (and many other languages). Often the switch statement is used for comparison of an object/expression with case statements containing literals.

More powerful examples of pattern matching can be found in languages such as Scala and Elixir. With structural pattern matching, the approach is "declarative" and explicitly states the conditions (the patterns) for data to match.

While an "imperative" series of instructions using nested "if" statements could be used to accomplish something similar to structural pattern matching, it is less clear than the "declarative" approach. Instead the "declarative" approach states the conditions to meet for a match and is more readable through its explicit patterns. While structural pattern matching can be used in its simplest form comparing a variable to a literal in a case statement, its true value for Python lies in its handling of the subject's type and shape.

Simple pattern: match to a literal

Let's look at this example as pattern matching in its simplest form: a value, the subject, being matched to several literals, the patterns. In the example below, status is the subject of the match statement. The patterns are each of the case statements, where literals represent request status codes. The associated action to the case is executed after a match:

def http_error(status):
    match status:
        case 400:
            return "Bad request"
        case 404:
            return "Not found"
        case 418:
            return "I'm a teapot"
        case _:
            return "Something's wrong with the internet"

If the above function is passed a status of 418, "I'm a teapot" is returned. If the above function is passed a status of 500, the case statement with _ will match as a wildcard, and "Something's wrong with the internet" is returned. Note the last block: the variable name, _, acts as a wildcard and insures the subject will always match. The use of _ is optional.

You can combine several literals in a single pattern using | ("or"):

case 401 | 403 | 404:
    return "Not allowed"
Behavior without the wildcard

If we modify the above example by removing the last case block, the example becomes:

def http_error(status):
    match status:
        case 400:
            return "Bad request"
        case 404:
            return "Not found"
        case 418:
            return "I'm a teapot"

Without the use of _ in a case statement, a match may not exist. If no match exists, the behavior is a no-op. For example, if status of 500 is passed, a no-op occurs.

Patterns with a literal and variable

Patterns can look like unpacking assignments, and a pattern may be used to bind variables. In this example, a data point can be unpacked to its x-coordinate and y-coordinate:

# point is an (x, y) tuple
match point:
    case (0, 0):
        print("Origin")
    case (0, y):
        print(f"Y={y}")
    case (x, 0):
        print(f"X={x}")
    case (x, y):
        print(f"X={x}, Y={y}")
    case _:
        raise ValueError("Not a point")

The first pattern has two literals, (0, 0), and may be thought of as an extension of the literal pattern shown above. The next two patterns combine a literal and a variable, and the variable binds a value from the subject (point). The fourth pattern captures two values, which makes it conceptually similar to the unpacking assignment (x, y) = point.

Patterns and classes

If you are using classes to structure your data, you can use as a pattern the class name followed by an argument list resembling a constructor. This pattern has the ability to capture instance attributes into variables:

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

def location(point):
    match point:
        case Point(x=0, y=0):
            print("Origin is the point's location.")
        case Point(x=0, y=y):
            print(f"Y={y} and the point is on the y-axis.")
        case Point(x=x, y=0):
            print(f"X={x} and the point is on the x-axis.")
        case Point():
            print("The point is located somewhere else on the plane.")
        case _:
            print("Not a point")
Patterns with positional parameters

You can use positional parameters with some builtin classes that provide an ordering for their attributes (e.g. dataclasses). You can also define a specific position for attributes in patterns by setting the __match_args__ special attribute in your classes. If it's set to ("x", "y"), the following patterns are all equivalent (and all bind the y attribute to the var variable):

Point(1, var)
Point(1, y=var)
Point(x=1, y=var)
Point(y=var, x=1)

Nested patterns

Patterns can be arbitrarily nested. For example, if our data is a short list of points, it could be matched like this:

match points:
    case []:
        print("No points in the list.")
    case [Point(0, 0)]:
        print("The origin is the only point in the list.")
    case [Point(x, y)]:
        print(f"A single point {x}, {y} is in the list.")
    case [Point(0, y1), Point(0, y2)]:
        print(f"Two points on the Y axis at {y1}, {y2} are in the list.")
    case _:
        print("Something else is found in the list.")

Complex patterns and the wildcard

To this point, the examples have used _ alone in the last case statement. A wildcard can be used in more complex patterns, such as ('error', code, _). For example:

match test_variable:
    case ('warning', code, 40):
        print("A warning has been received.")
    case ('error', code, _):
        print(f"An error {code} occurred.")

In the above case, test_variable will match for ('error', code, 100) and ('error', code, 800).

Guard

We can add an if clause to a pattern, known as a "guard". If the guard is false, match goes on to try the next case block. Note that value capture happens before the guard is evaluated:

match point:
    case Point(x, y) if x == y:
        print(f"The point is located on the diagonal Y=X at {x}.")
    case Point(x, y):
        print(f"Point is not on the diagonal.")

Other Key Features

Several other key features:

For the full specification see 634. Motivation and rationale are in 635, and a longer tutorial is in 636.

Optional EncodingWarning and encoding="locale" option

The default encoding of ~io.TextIOWrapper and open is platform and locale dependent. Since UTF-8 is used on most Unix platforms, omitting encoding option when opening UTF-8 files (e.g. JSON, YAML, TOML, Markdown) is a very common bug. For example:

# BUG: "rb" mode or encoding="utf-8" should be used.
with open("data.json") as f:
    data = json.load(f)

To find this type of bug, an optional EncodingWarning is added. It is emitted when sys.flags.warn_default_encoding <sys.flags> is true and locale-specific default encoding is used.

-X warn_default_encoding option and PYTHONWARNDEFAULTENCODING are added to enable the warning.

See io-text-encoding for more information.

This section covers major changes affecting 484 type hints and the typing module.

PEP 604: New Type Union Operator

A new type union operator was introduced which enables the syntax X | Y. This provides a cleaner way of expressing 'either type X or type Y' instead of using typing.Union, especially in type hints.

In previous versions of Python, to apply a type hint for functions accepting arguments of multiple types, typing.Union was used:

def square(number: Union[int, float]) -> Union[int, float]:
    return number ** 2

Type hints can now be written in a more succinct manner:

def square(number: int | float) -> int | float:
    return number ** 2

This new syntax is also accepted as the second argument to isinstance and issubclass:

>>> isinstance(1, int | str)
True

See types-union and 604 for more details.

(Contributed by Maggie Moss and Philippe Prados in 41428, with additions by Yurii Karabas and Serhiy Storchaka in 44490.)

PEP 612: Parameter Specification Variables

Two new options to improve the information provided to static type checkers for 484's Callable have been added to the typing module.

The first is the parameter specification variable. They are used to forward the parameter types of one callable to another callable -- a pattern commonly found in higher order functions and decorators. Examples of usage can be found in typing.ParamSpec. Previously, there was no easy way to type annotate dependency of parameter types in such a precise manner.

The second option is the new Concatenate operator. It's used in conjunction with parameter specification variables to type annotate a higher order callable which adds or removes parameters of another callable. Examples of usage can be found in typing.Concatenate.

See typing.Callable, typing.ParamSpec, typing.Concatenate, typing.ParamSpecArgs, typing.ParamSpecKwargs, and 612 for more details.

(Contributed by Ken Jin in 41559, with minor enhancements by Jelle Zijlstra in 43783. PEP written by Mark Mendoza.)

PEP 613: TypeAlias

484 introduced the concept of type aliases, only requiring them to be top-level unannotated assignments. This simplicity sometimes made it difficult for type checkers to distinguish between type aliases and ordinary assignments, especially when forward references or invalid types were involved. Compare:

StrCache = 'Cache[str]'  # a type alias
LOG_PREFIX = 'LOG[DEBUG]'  # a module constant

Now the typing module has a special value ~typing.TypeAlias which lets you declare type aliases more explicitly:

StrCache: TypeAlias = 'Cache[str]'  # a type alias
LOG_PREFIX = 'LOG[DEBUG]'  # a module constant

See 613 for more details.

(Contributed by Mikhail Golubev in 41923.)

PEP 647: User-Defined Type Guards

~typing.TypeGuard has been added to the typing module to annotate type guard functions and improve information provided to static type checkers during type narrowing. For more information, please see ~typing.TypeGuard's documentation, and 647.

(Contributed by Ken Jin and Guido van Rossum in 43766. PEP written by Eric Traut.)

Other Language Changes

New Modules

Improved Modules

asyncio

Add missing ~asyncio.loop.connect_accepted_socket method. (Contributed by Alex Grönholm in 41332.)

argparse

Misleading phrase "optional arguments" was replaced with "options" in argparse help. Some tests might require adaptation if they rely on exact output match. (Contributed by Raymond Hettinger in 9694.)

array

The ~array.array.index method of array.array now has optional start and stop parameters. (Contributed by Anders Lorentsen and Zackery Spytz in 31956.)

asynchat, asyncore, smtpd

These modules have been marked as deprecated in their module documentation since Python 3.6. An import-time DeprecationWarning has now been added to all three of these modules.

base64

Add base64.b32hexencode and base64.b32hexdecode to support the Base32 Encoding with Extended Hex Alphabet.

bdb

Add !clearBreakpoints to reset all set breakpoints. (Contributed by Irit Katriel in 24160.)

bisect

Added the possibility of providing a key function to the APIs in the bisect module. (Contributed by Raymond Hettinger in 4356.)

codecs

Add a codecs.unregister function to unregister a codec search function. (Contributed by Hai Shi in 41842.)

collections.abc

The __args__ of the parameterized generic <types-genericalias> for collections.abc.Callable are now consistent with typing.Callable. collections.abc.Callable generic now flattens type parameters, similar to what typing.Callable currently does. This means that collections.abc.Callable[[int, str], str] will have __args__ of (int, str, str); previously this was ([int, str], str). To allow this change, types.GenericAlias can now be subclassed, and a subclass will be returned when subscripting the collections.abc.Callable type. Note that a TypeError may be raised for invalid forms of parameterizing collections.abc.Callable which may have passed silently in Python 3.9. (Contributed by Ken Jin in 42195.)

contextlib

Add a contextlib.aclosing context manager to safely close async generators and objects representing asynchronously released resources. (Contributed by Joongi Kim and John Belmonte in 41229.)

Add asynchronous context manager support to contextlib.nullcontext. (Contributed by Tom Gringauz in 41543.)

Add ~contextlib.AsyncContextDecorator, for supporting usage of async context managers as decorators.

curses

The extended color functions added in ncurses 6.1 will be used transparently by curses.color_content, curses.init_color, curses.init_pair, and curses.pair_content. A new function, curses.has_extended_color_support, indicates whether extended color support is provided by the underlying ncurses library. (Contributed by Jeffrey Kintscher and Hans Petter Jansson in 36982.)

The BUTTON5_* constants are now exposed in the curses module if they are provided by the underlying curses library. (Contributed by Zackery Spytz in 39273.)

dataclasses

__slots__

Added slots parameter in dataclasses.dataclass decorator. (Contributed by Yurii Karabas in 42269)

Keyword-only fields

dataclasses now supports fields that are keyword-only in the generated __init__ method. There are a number of ways of specifying keyword-only fields.

You can say that every field is keyword-only:

from dataclasses import dataclass

@dataclass(kw_only=True)
class Birthday:
    name: str
    birthday: datetime.date

Both name and birthday are keyword-only parameters to the generated __init__ method.

You can specify keyword-only on a per-field basis:

from dataclasses import dataclass, field

@dataclass
class Birthday:
    name: str
    birthday: datetime.date = field(kw_only=True)

Here only birthday is keyword-only. If you set kw_only on individual fields, be aware that there are rules about re-ordering fields due to keyword-only fields needing to follow non-keyword-only fields. See the full dataclasses documentation for details.

You can also specify that all fields following a KW_ONLY marker are keyword-only. This will probably be the most common usage:

from dataclasses import dataclass, KW_ONLY

@dataclass
class Point:
    x: float
    y: float
    _: KW_ONLY
    z: float = 0.0
    t: float = 0.0

Here, z and t are keyword-only parameters, while x and y are not. (Contributed by Eric V. Smith in 43532.)

distutils

The entire distutils package is deprecated, to be removed in Python 3.12. Its functionality for specifying package builds has already been completely replaced by third-party packages setuptools and packaging, and most other commonly used APIs are available elsewhere in the standard library (such as platform, shutil, subprocess or sysconfig). There are no plans to migrate any other functionality from distutils, and applications that are using other functions should plan to make private copies of the code. Refer to 632 for discussion.

The bdist_wininst command deprecated in Python 3.8 has been removed. The bdist_wheel command is now recommended to distribute binary packages on Windows. (Contributed by Victor Stinner in 42802.)

doctest

When a module does not define __loader__, fall back to __spec__.loader. (Contributed by Brett Cannon in 42133.)

encodings

encodings.normalize_encoding now ignores non-ASCII characters. (Contributed by Hai Shi in 39337.)

enum

~enum.Enum ~object.__repr__ now returns enum_name.member_name and ~object.__str__ now returns member_name. Stdlib enums available as module constants have a repr of module_name.member_name. (Contributed by Ethan Furman in 40066.)

Add enum.StrEnum for enums where all members are strings. (Contributed by Ethan Furman in 41816.)

fileinput

Add encoding and errors parameters in fileinput.input and fileinput.FileInput. (Contributed by Inada Naoki in 43712.)

fileinput.hook_compressed now returns ~io.TextIOWrapper object when mode is "r" and file is compressed, like uncompressed files. (Contributed by Inada Naoki in 5758.)

faulthandler

The faulthandler module now detects if a fatal error occurs during a garbage collector collection. (Contributed by Victor Stinner in 44466.)

gc

Add audit hooks for gc.get_objects, gc.get_referrers and gc.get_referents. (Contributed by Pablo Galindo in 43439.)

glob

Add the root_dir and dir_fd parameters in ~glob.glob and ~glob.iglob which allow to specify the root directory for searching. (Contributed by Serhiy Storchaka in 38144.)

hashlib

The hashlib module requires OpenSSL 1.1.1 or newer. (Contributed by Christian Heimes in 644 and 43669.)

The hashlib module has preliminary support for OpenSSL 3.0.0. (Contributed by Christian Heimes in 38820 and other issues.)

The pure-Python fallback of ~hashlib.pbkdf2_hmac is deprecated. In the future PBKDF2-HMAC will only be available when Python has been built with OpenSSL support. (Contributed by Christian Heimes in 43880.)

hmac

The hmac module now uses OpenSSL's HMAC implementation internally. (Contributed by Christian Heimes in 40645.)

IDLE and idlelib

Make IDLE invoke sys.excepthook (when started without '-n'). User hooks were previously ignored. (Contributed by Ken Hilton in 43008.)

Rearrange the settings dialog. Split the General tab into Windows and Shell/Ed tabs. Move help sources, which extend the Help menu, to the Extensions tab. Make space for new options and shorten the dialog. The latter makes the dialog better fit small screens. (Contributed by Terry Jan Reedy in 40468.) Move the indent space setting from the Font tab to the new Windows tab. (Contributed by Mark Roseman and Terry Jan Reedy in 33962.)

The changes above were backported to a 3.9 maintenance release.

Add a Shell sidebar. Move the primary prompt ('>>>') to the sidebar. Add secondary prompts ('...') to the sidebar. Left click and optional drag selects one or more lines of text, as with the editor line number sidebar. Right click after selecting text lines displays a context menu with 'copy with prompts'. This zips together prompts from the sidebar with lines from the selected text. This option also appears on the context menu for the text. (Contributed by Tal Einat in 37903.)

Use spaces instead of tabs to indent interactive code. This makes interactive code entries 'look right'. Making this feasible was a major motivation for adding the shell sidebar. (Contributed by Terry Jan Reedy in 37892.)

Highlight the new soft keywords <soft-keywords> match, case <match>, and _ <wildcard-patterns> in pattern-matching statements. However, this highlighting is not perfect and will be incorrect in some rare cases, including some _-s in case patterns. (Contributed by Tal Einat in 44010.)

New in 3.10 maintenance releases.

Apply syntax highlighting to .pyi files. (Contributed by Alex Waygood and Terry Jan Reedy in 45447.)

Include prompts when saving Shell with inputs and outputs. (Contributed by Terry Jan Reedy in 95191.)

importlib.metadata

Feature parity with importlib_metadata 4.6 (history).

importlib.metadata entry points <entry-points> now provide a nicer experience for selecting entry points by group and name through a new importlib.metadata.EntryPoints <entry-points> class. See the Compatibility Note in the docs for more info on the deprecation and usage.

Added importlib.metadata.packages_distributions() <package-distributions> for resolving top-level Python modules and packages to their importlib.metadata.Distribution <distributions>.

inspect

When a module does not define __loader__, fall back to __spec__.loader. (Contributed by Brett Cannon in 42133.)

Add inspect.get_annotations, which safely computes the annotations defined on an object. It works around the quirks of accessing the annotations on various types of objects, and makes very few assumptions about the object it examines. inspect.get_annotations can also correctly un-stringize stringized annotations. inspect.get_annotations is now considered best practice for accessing the annotations dict defined on any Python object; for more information on best practices for working with annotations, please see annotations-howto. Relatedly, inspect.signature, inspect.Signature.from_callable, and !inspect.Signature.from_function now call inspect.get_annotations to retrieve annotations. This means inspect.signature and inspect.Signature.from_callable can also now un-stringize stringized annotations. (Contributed by Larry Hastings in 43817.)

itertools

Add itertools.pairwise. (Contributed by Raymond Hettinger in 38200.)

linecache

When a module does not define __loader__, fall back to __spec__.loader. (Contributed by Brett Cannon in 42133.)

os

Add os.cpu_count support for VxWorks RTOS. (Contributed by Peixing Xin in 41440.)

Add a new function os.eventfd and related helpers to wrap the eventfd2 syscall on Linux. (Contributed by Christian Heimes in 41001.)

Add os.splice that allows to move data between two file descriptors without copying between kernel address space and user address space, where one of the file descriptors must refer to a pipe. (Contributed by Pablo Galindo in 41625.)

Add ~os.O_EVTONLY, ~os.O_FSYNC, ~os.O_SYMLINK and ~os.O_NOFOLLOW_ANY for macOS. (Contributed by Donghee Na in 43106.)

os.path

os.path.realpath now accepts a strict keyword-only argument. When set to True, OSError is raised if a path doesn't exist or a symlink loop is encountered. (Contributed by Barney Gale in 43757.)

pathlib

Add slice support to PurePath.parents <pathlib.PurePath.parents>. (Contributed by Joshua Cannon in 35498.)

Add negative indexing support to PurePath.parents <pathlib.PurePath.parents>. (Contributed by Yaroslav Pankovych in 21041.)

pathlib.Path.stat and ~pathlib.Path.chmod now accept a follow_symlinks keyword-only argument for consistency with corresponding functions in the os module. (Contributed by Barney Gale in 39906.)

platform

Add platform.freedesktop_os_release to retrieve operation system identification from freedesktop.org os-release standard file. (Contributed by Christian Heimes in 28468.)

pprint

pprint.pprint now accepts a new underscore_numbers keyword argument. (Contributed by sblondon in 42914.)

pprint can now pretty-print dataclasses.dataclass instances. (Contributed by Lewis Gaul in 43080.)

py_compile

Add --quiet option to command-line interface of py_compile. (Contributed by Gregory Schevchenko in 38731.)

pyclbr

Add an end_lineno attribute to the Function and Class objects in the tree returned by pyclbr.readmodule and pyclbr.readmodule_ex. It matches the existing (start) lineno. (Contributed by Aviral Srivastava in 38307.)

shelve

The shelve module now uses pickle.DEFAULT_PROTOCOL by default instead of pickle protocol 3 when creating shelves. (Contributed by Zackery Spytz in 34204.)

statistics

Add ~statistics.covariance, Pearson's ~statistics.correlation, and simple ~statistics.linear_regression functions. (Contributed by Tymoteusz Wołodźko in 38490.)

site

When a module does not define __loader__, fall back to __spec__.loader. (Contributed by Brett Cannon in 42133.)

socket

The exception socket.timeout is now an alias of TimeoutError. (Contributed by Christian Heimes in 42413.)

Add option to create MPTCP sockets with IPPROTO_MPTCP (Contributed by Rui Cunha in 43571.)

Add IP_RECVTOS option to receive the type of service (ToS) or DSCP/ECN fields (Contributed by Georg Sauthoff in 44077.)

ssl

The ssl module requires OpenSSL 1.1.1 or newer. (Contributed by Christian Heimes in 644 and 43669.)

The ssl module has preliminary support for OpenSSL 3.0.0 and new option ~ssl.OP_IGNORE_UNEXPECTED_EOF. (Contributed by Christian Heimes in 38820, 43794, 43788, 43791, 43799, 43920, 43789, and 43811.)

Deprecated function and use of deprecated constants now result in a DeprecationWarning. ssl.SSLContext.options has ~ssl.OP_NO_SSLv2 and ~ssl.OP_NO_SSLv3 set by default and therefore cannot warn about setting the flag again. The deprecation section <whatsnew310-deprecated> has a list of deprecated features. (Contributed by Christian Heimes in 43880.)

The ssl module now has more secure default settings. Ciphers without forward secrecy or SHA-1 MAC are disabled by default. Security level 2 prohibits weak RSA, DH, and ECC keys with less than 112 bits of security. ~ssl.SSLContext defaults to minimum protocol version TLS 1.2. Settings are based on Hynek Schlawack's research. (Contributed by Christian Heimes in 43998.)

The deprecated protocols SSL 3.0, TLS 1.0, and TLS 1.1 are no longer officially supported. Python does not block them actively. However OpenSSL build options, distro configurations, vendor patches, and cipher suites may prevent a successful handshake.

Add a timeout parameter to the ssl.get_server_certificate function. (Contributed by Zackery Spytz in 31870.)

The ssl module uses heap-types and multi-phase initialization. (Contributed by Christian Heimes in 42333.)

A new verify flag ~ssl.VERIFY_X509_PARTIAL_CHAIN has been added. (Contributed by l0x in 40849.)

sqlite3

Add audit events for ~sqlite3.connect, ~sqlite3.Connection.enable_load_extension, and ~sqlite3.Connection.load_extension. (Contributed by Erlend E. Aasland in 43762.)

sys

Add sys.orig_argv attribute: the list of the original command line arguments passed to the Python executable. (Contributed by Victor Stinner in 23427.)

Add sys.stdlib_module_names, containing the list of the standard library module names. (Contributed by Victor Stinner in 42955.)

thread

_thread.interrupt_main now takes an optional signal number to simulate (the default is still signal.SIGINT). (Contributed by Antoine Pitrou in 43356.)

threading

Add threading.gettrace and threading.getprofile to retrieve the functions set by threading.settrace and threading.setprofile respectively. (Contributed by Mario Corchero in 42251.)

Add threading.__excepthook__ to allow retrieving the original value of threading.excepthook in case it is set to a broken or a different value. (Contributed by Mario Corchero in 42308.)

traceback

The ~traceback.format_exception, ~traceback.format_exception_only, and ~traceback.print_exception functions can now take an exception object as a positional-only argument. (Contributed by Zackery Spytz and Matthias Bussonnier in 26389.)

types

Reintroduce the types.EllipsisType, types.NoneType and types.NotImplementedType classes, providing a new set of types readily interpretable by type checkers. (Contributed by Bas van Beek in 41810.)

typing

For major changes, see new-feat-related-type-hints.

The behavior of typing.Literal was changed to conform with 586 and to match the behavior of static type checkers specified in the PEP.

  1. Literal now de-duplicates parameters.

  2. Equality comparisons between Literal objects are now order independent.

  3. Literal comparisons now respect types. For example, Literal[0] == Literal[False] previously evaluated to True. It is now False. To support this change, the internally used type cache now supports differentiating types.

  4. Literal objects will now raise a TypeError exception during equality comparisons if any of their parameters are not hashable. Note that declaring Literal with unhashable parameters will not throw an error:

    >>> from typing import Literal
>>> Literal[{0}]
>>> Literal[{0}] == Literal[{False}]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'set'

(Contributed by Yurii Karabas in 42345.)

Add new function typing.is_typeddict to introspect if an annotation is a typing.TypedDict. (Contributed by Patrick Reader in 41792.)

Subclasses of typing.Protocol which only have data variables declared will now raise a TypeError when checked with isinstance unless they are decorated with ~typing.runtime_checkable. Previously, these checks passed silently. Users should decorate their subclasses with the !runtime_checkable decorator if they want runtime protocols. (Contributed by Yurii Karabas in 38908.)

Importing from the typing.io and typing.re submodules will now emit DeprecationWarning. These submodules have been deprecated since Python 3.8 and will be removed in a future version of Python. Anything belonging to those submodules should be imported directly from typing instead. (Contributed by Sebastian Rittau in 38291.)

unittest

Add new method ~unittest.TestCase.assertNoLogs to complement the existing ~unittest.TestCase.assertLogs. (Contributed by Kit Yan Choi in 39385.)

urllib.parse

Python versions earlier than Python 3.10 allowed using both ; and & as query parameter separators in urllib.parse.parse_qs and urllib.parse.parse_qsl. Due to security concerns, and to conform with newer W3C recommendations, this has been changed to allow only a single separator key, with & as the default. This change also affects !cgi.parse and !cgi.parse_multipart as they use the affected functions internally. For more details, please see their respective documentation. (Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in 42967.)

The presence of newline or tab characters in parts of a URL allows for some forms of attacks. Following the WHATWG specification that updates 3986, ASCII newline \n, \r and tab \t characters are stripped from the URL by the parser in urllib.parse preventing such attacks. The removal characters are controlled by a new module level variable urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE. (See 88048)

xml

Add a ~xml.sax.handler.LexicalHandler class to the xml.sax.handler module. (Contributed by Jonathan Gossage and Zackery Spytz in 35018.)

zipimport

Add methods related to 451: ~zipimport.zipimporter.find_spec, zipimport.zipimporter.create_module, and zipimport.zipimporter.exec_module. (Contributed by Brett Cannon in 42131.)

Add ~zipimport.zipimporter.invalidate_caches method. (Contributed by Desmond Cheong in 14678.)

Optimizations

Deprecated

Removed

Porting to Python 3.10

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in the Python syntax

Changes in the Python API

Changes in the C API

CPython bytecode changes

Build Changes

C API Changes

PEP 652: Maintaining the Stable ABI

The Stable ABI (Application Binary Interface) for extension modules or embedding Python is now explicitly defined. stable describes C API and ABI stability guarantees along with best practices for using the Stable ABI.

(Contributed by Petr Viktorin in 652 and 43795.)

New Features

Porting to Python 3.10

Deprecated

Removed

Notable security feature in 3.10.7

Converting between int and str in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a ValueError if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for 2020-10735. This limit can be configured or disabled by environment variable, command line flag, or sys APIs. See the integer string conversion length limitation <int_max_str_digits> documentation. The default limit is 4300 digits in string form.

Notable security feature in 3.10.8

The deprecated !mailcap module now refuses to inject unsafe text (filenames, MIME types, parameters) into shell commands. Instead of using such text, it will warn and act as if a match was not found (or for test commands, as if the test failed). (Contributed by Petr Viktorin in 98966.)

Notable changes in 3.10.12

tarfile

What's New In Python 3.11

Editor
Pablo Galindo Salgado

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the Mercurial log when researching a change.

This article explains the new features in Python 3.11, compared to 3.10. Python 3.11 was released on October 24, 2022. For full details, see the changelog <changelog>.

Summary -- Release highlights

New syntax features:

New built-in features:

New standard library modules:

Interpreter improvements:

New typing features:

Important deprecations, removals and restrictions:

New Features

PEP 657: Fine-grained error locations in tracebacks

When printing tracebacks, the interpreter will now point to the exact expression that caused the error, instead of just the line. For example:

Traceback (most recent call last):
  File "distance.py", line 11, in <module>
    print(manhattan_distance(p1, p2))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "distance.py", line 6, in manhattan_distance
    return abs(point_1.x - point_2.x) + abs(point_1.y - point_2.y)
                           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'x'

Previous versions of the interpreter would point to just the line, making it ambiguous which object was None. These enhanced errors can also be helpful when dealing with deeply nested dict objects and multiple function calls:

Traceback (most recent call last):
  File "query.py", line 37, in <module>
    magic_arithmetic('foo')
  File "query.py", line 18, in magic_arithmetic
    return add_counts(x) / 25
           ^^^^^^^^^^^^^
  File "query.py", line 24, in add_counts
    return 25 + query_user(user1) + query_user(user2)
                ^^^^^^^^^^^^^^^^^
  File "query.py", line 32, in query_user
    return 1 + query_count(db, response['a']['b']['c']['user'], retry=True)
                               ~~~~~~~~~~~~~~~~~~^^^^^
TypeError: 'NoneType' object is not subscriptable

As well as complex arithmetic expressions:

Traceback (most recent call last):
  File "calculation.py", line 54, in <module>
    result = (x / y / z) * (a / b / c)
              ~~~~~~^~~
ZeroDivisionError: division by zero

Additionally, the information used by the enhanced traceback feature is made available via a general API, that can be used to correlate bytecode instructions <bytecodes> with source code location. This information can be retrieved using:

See 657 for more details. (Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar in 43950.)

[!NOTE] This feature requires storing column positions in codeobjects, which may result in a small increase in interpreter memory usage and disk usage for compiled Python files. To avoid storing the extra information and deactivate printing the extra traceback information, use the -X no_debug_ranges <-X> command line option or the PYTHONNODEBUGRANGES environment variable.

PEP 654: Exception Groups and except*

654 introduces language features that enable a program to raise and handle multiple unrelated exceptions simultaneously. The builtin types ExceptionGroup and BaseExceptionGroup make it possible to group exceptions and raise them together, and the new except* <except_star> syntax generalizes except to match subgroups of exception groups.

See 654 for more details.

(Contributed by Irit Katriel in 45292. PEP written by Irit Katriel, Yury Selivanov and Guido van Rossum.)

PEP 678: Exceptions can be enriched with notes

The ~BaseException.add_note method is added to BaseException. It can be used to enrich exceptions with context information that is not available at the time when the exception is raised. The added notes appear in the default traceback.

See 678 for more details.

(Contributed by Irit Katriel in 45607. PEP written by Zac Hatfield-Dodds.)

Windows py.exe launcher improvements

The copy of the launcher included with Python 3.11 has been significantly updated. It now supports company/tag syntax as defined in 514 using the -V:{<company>}/{<tag>} argument instead of the limited -{<major>}.{<minor>}. This allows launching distributions other than PythonCore, the one hosted on python.org.

When using -V: selectors, either company or tag can be omitted, but all installs will be searched. For example, -V:OtherPython/ will select the "best" tag registered for OtherPython, while -V:3.11 or -V:/3.11 will select the "best" distribution with tag 3.11.

When using the legacy -{<major>}, -{<major>}.{<minor>}, -{<major>}-{<bitness>} or -{<major>}.{<minor>}-{<bitness>} arguments, all existing behaviour should be preserved from past versions, and only releases from PythonCore will be selected. However, the -64 suffix now implies "not 32-bit" (not necessarily x86-64), as there are multiple supported 64-bit platforms. 32-bit runtimes are detected by checking the runtime's tag for a -32 suffix. All releases of Python since 3.5 have included this in their 32-bit builds.

This section covers major changes affecting 484 type hints and the typing module.

PEP 646: Variadic generics

484 previously introduced ~typing.TypeVar, enabling creation of generics parameterised with a single type. 646 adds ~typing.TypeVarTuple, enabling parameterisation with an arbitrary number of types. In other words, a ~typing.TypeVarTuple is a variadic type variable, enabling variadic generics.

This enables a wide variety of use cases. In particular, it allows the type of array-like structures in numerical computing libraries such as NumPy and TensorFlow to be parameterised with the array shape. Static type checkers will now be able to catch shape-related bugs in code that uses these libraries.

See 646 for more details.

(Contributed by Matthew Rahtz in 43224, with contributions by Serhiy Storchaka and Jelle Zijlstra. PEP written by Mark Mendoza, Matthew Rahtz, Pradeep Kumar Srinivasan, and Vincent Siles.)

PEP 655: Marking individual TypedDict items as required or not-required

~typing.Required and ~typing.NotRequired provide a straightforward way to mark whether individual items in a ~typing.TypedDict must be present. Previously, this was only possible using inheritance.

All fields are still required by default, unless the total parameter is set to False, in which case all fields are still not-required by default. For example, the following specifies a !TypedDict with one required and one not-required key:

class Movie(TypedDict):
   title: str
   year: NotRequired[int]

m1: Movie = {"title": "Black Panther", "year": 2018}  # OK
m2: Movie = {"title": "Star Wars"}  # OK (year is not required)
m3: Movie = {"year": 2022}  # ERROR (missing required field title)

The following definition is equivalent:

class Movie(TypedDict, total=False):
   title: Required[str]
   year: int

See 655 for more details.

(Contributed by David Foster and Jelle Zijlstra in 47087. PEP written by David Foster.)

PEP 673: Self type

The new ~typing.Self annotation provides a simple and intuitive way to annotate methods that return an instance of their class. This behaves the same as the ~typing.TypeVar-based approach specified in PEP 484 <484#annotating-instance-and-class-methods>, but is more concise and easier to follow.

Common use cases include alternative constructors provided as classmethod <classmethod>s, and ~object.__enter__ methods that return self:

class MyLock:
    def __enter__(self) -> Self:
        self.lock()
        return self

    ...

class MyInt:
    @classmethod
    def fromhex(cls, s: str) -> Self:
        return cls(int(s, 16))

    ...

~typing.Self can also be used to annotate method parameters or attributes of the same type as their enclosing class.

See 673 for more details.

(Contributed by James Hilton-Balfe in 46534. PEP written by Pradeep Kumar Srinivasan and James Hilton-Balfe.)

PEP 675: Arbitrary literal string type

The new ~typing.LiteralString annotation may be used to indicate that a function parameter can be of any literal string type. This allows a function to accept arbitrary literal string types, as well as strings created from other literal strings. Type checkers can then enforce that sensitive functions, such as those that execute SQL statements or shell commands, are called only with static arguments, providing protection against injection attacks.

For example, a SQL query function could be annotated as follows:

def run_query(sql: LiteralString) -> ...:
    ...

def caller(
    arbitrary_string: str,
    query_string: LiteralString,
    table_name: LiteralString,
) -> None:
    run_query("SELECT * FROM students")       # ok
    run_query(query_string)                   # ok
    run_query("SELECT * FROM " + table_name)  # ok
    run_query(arbitrary_string)               # type checker error
    run_query(                                # type checker error
        f"SELECT * FROM students WHERE name = {arbitrary_string}"
    )

See 675 for more details.

(Contributed by Jelle Zijlstra in 47088. PEP written by Pradeep Kumar Srinivasan and Graham Bleaney.)

PEP 681: Data class transforms

~typing.dataclass_transform may be used to decorate a class, metaclass, or a function that is itself a decorator. The presence of @dataclass_transform() tells a static type checker that the decorated object performs runtime "magic" that transforms a class, giving it dataclass <dataclasses.dataclass>-like behaviors.

For example:

# The create_model decorator is defined by a library.
@typing.dataclass_transform()
def create_model(cls: Type[T]) -> Type[T]:
    cls.__init__ = ...
    cls.__eq__ = ...
    cls.__ne__ = ...
    return cls

# The create_model decorator can now be used to create new model classes:
@create_model
class CustomerModel:
    id: int
    name: str

c = CustomerModel(id=327, name="Eric Idle")

See 681 for more details.

(Contributed by Jelle Zijlstra in 91860. PEP written by Erik De Bonte and Eric Traut.)

PEP 563 may not be the future

563 Postponed Evaluation of Annotations (the from __future__ import annotations future statement <future>) that was originally planned for release in Python 3.10 has been put on hold indefinitely. See this message from the Steering Council for more information.

Other Language Changes

Other CPython Implementation Changes

New Modules

Improved Modules

asyncio

contextlib

dataclasses

datetime

enum

fcntl

fractions

functools

gzip

hashlib

IDLE and idlelib

inspect

locale

logging

math

operator

os

pathlib

re

shutil

socket

sqlite3

string

sys

sysconfig

tempfile

threading

time

tkinter

traceback

typing

For major changes, see new-feat-related-type-hints-311.

unicodedata

unittest

venv

warnings

zipfile

Optimizations

This section covers specific optimizations independent of the whatsnew311-faster-cpython project, which is covered in its own section.

Faster CPython

CPython 3.11 is an average of 25% faster than CPython 3.10 as measured with the pyperformance benchmark suite, when compiled with GCC on Ubuntu Linux. Depending on your workload, the overall speedup could be 10-60%.

This project focuses on two major areas in Python: whatsnew311-faster-startup and whatsnew311-faster-runtime. Optimizations not covered by this project are listed separately under whatsnew311-optimizations.

Faster Startup

Frozen imports / Static code objects

Python caches bytecode in the __pycache__ <tut-pycache> directory to speed up module loading.

Previously in 3.10, Python module execution looked like this:

Read __pycache__ -> Unmarshal -> Heap allocated code object -> Evaluate

In Python 3.11, the core modules essential for Python startup are "frozen". This means that their codeobjects (and bytecode) are statically allocated by the interpreter. This reduces the steps in module execution process to:

Statically allocated code object -> Evaluate

Interpreter startup is now 10-15% faster in Python 3.11. This has a big impact for short-running programs using Python.

(Contributed by Eric Snow, Guido van Rossum and Kumar Aditya in many issues.)

Faster Runtime

Cheaper, lazy Python frames

Python frames, holding execution information, are created whenever Python calls a Python function. The following are new frame optimizations:

Old-style frame objects <frame-objects> are now created only when requested by debuggers or by Python introspection functions such as sys._getframe and inspect.currentframe. For most user code, no frame objects are created at all. As a result, nearly all Python functions calls have sped up significantly. We measured a 3-7% speedup in pyperformance.

(Contributed by Mark Shannon in 44590.)

Inlined Python function calls

During a Python function call, Python will call an evaluating C function to interpret that function's code. This effectively limits pure Python recursion to what's safe for the C stack.

In 3.11, when CPython detects Python code calling another Python function, it sets up a new frame, and "jumps" to the new code inside the new frame. This avoids calling the C interpreting function altogether.

Most Python function calls now consume no C stack space, speeding them up. In simple recursive functions like fibonacci or factorial, we observed a 1.7x speedup. This also means recursive functions can recurse significantly deeper (if the user increases the recursion limit with sys.setrecursionlimit). We measured a 1-3% improvement in pyperformance.

(Contributed by Pablo Galindo and Mark Shannon in 45256.)

PEP 659: Specializing Adaptive Interpreter

659 is one of the key parts of the Faster CPython project. The general idea is that while Python is a dynamic language, most code has regions where objects and types rarely change. This concept is known as type stability.

At runtime, Python will try to look for common patterns and type stability in the executing code. Python will then replace the current operation with a more specialized one. This specialized operation uses fast paths available only to those use cases/types, which generally outperform their generic counterparts. This also brings in another concept called inline caching, where Python caches the results of expensive operations directly in the bytecode.

The specializer will also combine certain common instruction pairs into one superinstruction, reducing the overhead during execution.

Python will only specialize when it sees code that is "hot" (executed multiple times). This prevents Python from wasting time on run-once code. Python can also de-specialize when code is too dynamic or when the use changes. Specialization is attempted periodically, and specialization attempts are not too expensive, allowing specialization to adapt to new circumstances.

(PEP written by Mark Shannon, with ideas inspired by Stefan Brunthaler. See 659 for more information. Implementation by Mark Shannon and Brandt Bucher, with additional help from Irit Katriel and Dennis Sweeney.)

Operation

Form

Specialization

Operation speedup (up to)

Contributor(s)

Binary operations

x + x

x - x

x * x

Binary add, multiply and subtract for common types such as int, float and str take custom fast paths for their underlying types.

10%

Mark Shannon, Donghee Na, Brandt Bucher, Dennis Sweeney

Subscript

a[i]

Subscripting container types such as list, tuple and dict directly index the underlying data structures.

Subscripting custom ~object.__getitem__ is also inlined similar to inline-calls.

10-25%

Irit Katriel, Mark Shannon

Store subscript

a[i] = z

Similar to subscripting specialization above.

10-25%

Dennis Sweeney

Calls

f(arg)

C(arg)

Calls to common builtin (C) functions and types such as len and str directly call their underlying C version. This avoids going through the internal calling convention.

20%

Mark Shannon, Ken Jin

Load global variable

print

len

The object's index in the globals/builtins namespace is cached. Loading globals and builtins require zero namespace lookups.

1

Mark Shannon

Load attribute

o.attr

Similar to loading global variables. The attribute's index inside the class/object's namespace is cached. In most cases, attribute loading will require zero namespace lookups.

2

Mark Shannon

Load methods for call

o.meth()

The actual address of the method is cached. Method loading now has no namespace lookups -- even for classes with long inheritance chains.

10-20%

Ken Jin, Mark Shannon

Store attribute

o.attr = z

Similar to load attribute optimization.

2% in pyperformance

Mark Shannon

Unpack Sequence

*seq

Specialized for common containers such as list and tuple. Avoids internal calling convention.

8%

Brandt Bucher

  1. A similar optimization already existed since Python 3.8. 3.11 specializes for more forms and reduces some overhead.↩︎

  2. A similar optimization already existed since Python 3.10. 3.11 specializes for more forms. Furthermore, all attribute loads should be sped up by 45947.↩︎

Misc

FAQ

How should I write my code to utilize these speedups?

Write Pythonic code that follows common best practices; you don't have to change your code. The Faster CPython project optimizes for common code patterns we observe.

Will CPython 3.11 use more memory?

Maybe not; we don't expect memory use to exceed 20% higher than 3.10. This is offset by memory optimizations for frame objects and object dictionaries as mentioned above.

I don't see any speedups in my workload. Why?

Certain code won't have noticeable benefits. If your code spends most of its time on I/O operations, or already does most of its computation in a C extension library like NumPy, there won't be significant speedups. This project currently benefits pure-Python workloads the most.

Furthermore, the pyperformance figures are a geometric mean. Even within the pyperformance benchmarks, certain benchmarks have slowed down slightly, while others have sped up by nearly 2x!

Is there a JIT compiler?

No. We're still exploring other optimizations.

About

Faster CPython explores optimizations for CPython. The main team is funded by Microsoft to work on this full-time. Pablo Galindo Salgado is also funded by Bloomberg LP to work on the project part-time. Finally, many contributors are volunteers from the community.

CPython bytecode changes

The bytecode now contains inline cache entries, which take the form of the newly-added CACHE instructions. Many opcodes expect to be followed by an exact number of caches, and instruct the interpreter to skip over them at runtime. Populated caches can look like arbitrary instructions, so great care should be taken when reading or modifying raw, adaptive bytecode containing quickened data.

New opcodes

Replaced opcodes

Replaced Opcode(s)

New Opcode(s)

Notes
!BINARY_*
!INPLACE_*

BINARY_OP

Replaced all numeric binary/in-place opcodes with a single opcode
!CALL_FUNCTION
!CALL_FUNCTION_KW
!CALL_METHOD
CALL
!KW_NAMES
!PRECALL
PUSH_NULL

Decouples argument shifting for methods from handling of keyword arguments; allows better specialization of calls
!DUP_TOP
!DUP_TOP_TWO
!ROT_TWO
!ROT_THREE
!ROT_FOUR
!ROT_N
COPY
SWAP

Stack manipulation instructions
!JUMP_IF_NOT_EXC_MATCH
CHECK_EXC_MATCH

Now performs check but doesn't jump
!JUMP_ABSOLUTE
!POP_JUMP_IF_FALSE
!POP_JUMP_IF_TRUE
JUMP_BACKWARD
!POP_JUMP_BACKWARD_IF_*
!POP_JUMP_FORWARD_IF_*

See1; TRUE, FALSE, NONE and NOT_NONE variants for each direction
!SETUP_WITH
!SETUP_ASYNC_WITH

!BEFORE_WITH

with block setup

  1. All jump opcodes are now relative, including the existing !JUMP_IF_TRUE_OR_POP and !JUMP_IF_FALSE_OR_POP. The argument is now an offset from the current instruction rather than an absolute location.↩︎

Changed/removed opcodes

Deprecated

This section lists Python APIs that have been deprecated in Python 3.11.

Deprecated C APIs are listed separately <whatsnew311-c-api-deprecated>.

Language/Builtins

Modules

Standard Library

Pending Removal in Python 3.12

The following Python APIs have been deprecated in earlier Python releases, and will be removed in Python 3.12.

C APIs pending removal are listed separately <whatsnew311-c-api-pending-removal>.

Removed

This section lists Python APIs that have been removed in Python 3.11.

Removed C APIs are listed separately <whatsnew311-c-api-removed>.

Porting to Python 3.11

This section lists previously described changes and other bugfixes in the Python API that may require changes to your Python code.

Porting notes for the C API are listed separately <whatsnew311-c-api-porting>.

Build Changes

C API Changes

New Features

Porting to Python 3.11

Deprecated

Pending Removal in Python 3.12

The following C APIs have been deprecated in earlier Python releases, and will be removed in Python 3.12.

Removed

Notable changes in 3.11.4

tarfile

Notable changes in 3.11.5

OpenSSL

What's New In Python 3.12

Editor
Adam Turner

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the VCS log when researching a change.

This article explains the new features in Python 3.12, compared to 3.11. Python 3.12 was released on October 2, 2023. For full details, see the changelog <changelog>.

693 -- Python 3.12 Release Schedule

Summary -- Release highlights

Python 3.12 is a stable release of the Python programming language, with a mix of changes to the language and the standard library. The library changes focus on cleaning up deprecated APIs, usability, and correctness. Of note, the !distutils package has been removed from the standard library. Filesystem support in os and pathlib has seen a number of improvements, and several modules have better performance.

The language changes focus on usability, as f-strings <f-string> have had many limitations removed and 'Did you mean ...' suggestions continue to improve. The new type parameter syntax <whatsnew312-pep695> and type statement improve ergonomics for using generic types <generic type> and type aliases <type alias> with static type checkers.

This article doesn't attempt to provide a complete specification of all new features, but instead gives a convenient overview. For full details, you should refer to the documentation, such as the Library Reference <library-index> and Language Reference <reference-index>. If you want to understand the complete implementation and design rationale for a change, refer to the PEP for a particular new feature; but note that PEPs usually are not kept up-to-date once a feature has been fully implemented.

New syntax features:

New grammar features:

Interpreter improvements:

Python data model improvements:

Significant improvements in the standard library:

Security improvements:

C API improvements:

CPython implementation improvements:

New typing features:

Important deprecations, removals or restrictions:

New Features

PEP 695: Type Parameter Syntax

Generic classes and functions under 484 were declared using a verbose syntax that left the scope of type parameters unclear and required explicit declarations of variance.

695 introduces a new, more compact and explicit way to create generic classes <generic-classes> and functions <generic-functions>:

def max[T](args: Iterable[T]) -> T:
    ...

class list[T]:
    def __getitem__(self, index: int, /) -> T:
        ...

    def append(self, element: T) -> None:
        ...

In addition, the PEP introduces a new way to declare type aliases <type-aliases> using the type statement, which creates an instance of ~typing.TypeAliasType:

type Point = tuple[float, float]

Type aliases can also be generic <generic-type-aliases>:

type Point[T] = tuple[T, T]

The new syntax allows declaring ~typing.TypeVarTuple and ~typing.ParamSpec parameters, as well as ~typing.TypeVar parameters with bounds or constraints:

type IntFunc[**P] = Callable[P, int]  # ParamSpec
type LabeledTuple[*Ts] = tuple[str, *Ts]  # TypeVarTuple
type HashableSequence[T: Hashable] = Sequence[T]  # TypeVar with bound
type IntOrStrSequence[T: (int, str)] = Sequence[T]  # TypeVar with constraints

The value of type aliases and the bound and constraints of type variables created through this syntax are evaluated only on demand (see lazy evaluation <lazy-evaluation>). This means type aliases are able to refer to other types defined later in the file.

Type parameters declared through a type parameter list are visible within the scope of the declaration and any nested scopes, but not in the outer scope. For example, they can be used in the type annotations for the methods of a generic class or in the class body. However, they cannot be used in the module scope after the class is defined. See type-params for a detailed description of the runtime semantics of type parameters.

In order to support these scoping semantics, a new kind of scope is introduced, the annotation scope <annotation-scopes>. Annotation scopes behave for the most part like function scopes, but interact differently with enclosing class scopes. In Python 3.13, annotations <annotation> will also be evaluated in annotation scopes.

See 695 for more details.

(PEP written by Eric Traut. Implementation by Jelle Zijlstra, Eric Traut, and others in 103764.)

PEP 701: Syntactic formalization of f-strings

701 lifts some restrictions on the usage of f-strings <f-string>. Expression components inside f-strings can now be any valid Python expression, including strings reusing the same quote as the containing f-string, multi-line expressions, comments, backslashes, and unicode escape sequences. Let's cover these in detail:

See 701 for more details.

As a positive side-effect of how this feature has been implemented (by parsing f-strings with the PEG parser <617>), now error messages for f-strings are more precise and include the exact location of the error. For example, in Python 3.11, the following f-string raises a SyntaxError:

>>> my_string = f"{x z y}" + f"{1 + 1}"
  File "<stdin>", line 1
    (x z y)
     ^^^
SyntaxError: f-string: invalid syntax. Perhaps you forgot a comma?

but the error message doesn't include the exact location of the error within the line and also has the expression artificially surrounded by parentheses. In Python 3.12, as f-strings are parsed with the PEG parser, error messages can be more precise and show the entire line:

>>> my_string = f"{x z y}" + f"{1 + 1}"
  File "<stdin>", line 1
    my_string = f"{x z y}" + f"{1 + 1}"
                   ^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?

(Contributed by Pablo Galindo, Batuhan Taskaya, Lysandros Nikolaou, Cristián Maureira-Fredes and Marta Gómez in 102856. PEP written by Pablo Galindo, Batuhan Taskaya, Lysandros Nikolaou and Marta Gómez).

PEP 684: A Per-Interpreter GIL

684 introduces a per-interpreter GIL <global interpreter lock>, so that sub-interpreters may now be created with a unique GIL per interpreter. This allows Python programs to take full advantage of multiple CPU cores. This is currently only available through the C-API, though a Python API is anticipated for 3.13 <554>.

Use the new Py_NewInterpreterFromConfig function to create an interpreter with its own GIL:

PyInterpreterConfig config = {
    .check_multi_interp_extensions = 1,
    .gil = PyInterpreterConfig_OWN_GIL,
};
PyThreadState *tstate = NULL;
PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);
if (PyStatus_Exception(status)) {
    return -1;
}
/* The new interpreter is now active in the current thread. */

For further examples how to use the C-API for sub-interpreters with a per-interpreter GIL, see Modules/_xxsubinterpretersmodule.c.

(Contributed by Eric Snow in 104210, etc.)

PEP 669: Low impact monitoring for CPython

669 defines a new API <sys.monitoring> for profilers, debuggers, and other tools to monitor events in CPython. It covers a wide range of events, including calls, returns, lines, exceptions, jumps, and more. This means that you only pay for what you use, providing support for near-zero overhead debuggers and coverage tools. See sys.monitoring for details.

(Contributed by Mark Shannon in 103082.)

PEP 688: Making the buffer protocol accessible in Python

688 introduces a way to use the buffer protocol <bufferobjects> from Python code. Classes that implement the ~object.__buffer__ method are now usable as buffer types.

The new collections.abc.Buffer ABC provides a standard way to represent buffer objects, for example in type annotations. The new inspect.BufferFlags enum represents the flags that can be used to customize buffer creation. (Contributed by Jelle Zijlstra in 102500.)

PEP 709: Comprehension inlining

Dictionary, list, and set comprehensions are now inlined, rather than creating a new single-use function object for each execution of the comprehension. This speeds up execution of a comprehension by up to two times. See 709 for further details.

Comprehension iteration variables remain isolated and don't overwrite a variable of the same name in the outer scope, nor are they visible after the comprehension. Inlining does result in a few visible behavior changes:

(Contributed by Carl Meyer and Vladimir Matveev in 709.)

Improved Error Messages

This section covers major changes affecting type hints <484> and the typing module.

PEP 692: Using TypedDict for more precise **kwargs typing

Typing **kwargs in a function signature as introduced by 484 allowed for valid annotations only in cases where all of the **kwargs were of the same type.

692 specifies a more precise way of typing **kwargs by relying on typed dictionaries:

from typing import TypedDict, Unpack

class Movie(TypedDict):
  name: str
  year: int

def foo(**kwargs: Unpack[Movie]): ...

See 692 for more details.

(Contributed by Franek Magiera in 103629.)

PEP 698: Override Decorator for Static Typing

A new decorator typing.override has been added to the typing module. It indicates to type checkers that the method is intended to override a method in a superclass. This allows type checkers to catch mistakes where a method that is intended to override something in a base class does not in fact do so.

Example:

from typing import override

class Base:
  def get_color(self) -> str:
    return "blue"

class GoodChild(Base):
  @override  # ok: overrides Base.get_color
  def get_color(self) -> str:
    return "yellow"

class BadChild(Base):
  @override  # type checker error: does not override Base.get_color
  def get_colour(self) -> str:
    return "red"

See 698 for more details.

(Contributed by Steven Troxler in 101561.)

Other Language Changes

New Modules

Improved Modules

array

asyncio

calendar

csv

dis

fractions

importlib.resources

inspect

itertools

math

os

os.path

pathlib

platform

pdb

random

shutil

sqlite3

statistics

sys

tempfile

threading

tkinter

tokenize

types

typing

unicodedata

unittest

Add a --durations command line option, showing the N slowest test cases:

python3 -m unittest --durations=3 lib.tests.test_threading
.....
Slowest test durations
----------------------------------------------------------------------
1.210s     test_timeout (Lib.test.test_threading.BarrierTests)
1.003s     test_default_timeout (Lib.test.test_threading.BarrierTests)
0.518s     test_timeout (Lib.test.test_threading.EventTests)

(0.000 durations hidden.  Use -v to show these durations.)
----------------------------------------------------------------------
Ran 158 tests in 9.869s

OK (skipped=3)

(Contributed by Giampaolo Rodola in 48330)

uuid

Optimizations

CPython bytecode changes

Demos and Tools

Deprecated

Pending removal in Python 3.13

Modules (see 594):

Other modules:

APIs:

Pending removal in Python 3.14

Pending removal in Python 3.15

Pending removal in Python 3.16

Pending removal in Python 3.17

Pending removal in Python 3.19

Pending removal in Python 3.20

Pending removal in Python 3.21

Pending removal in future versions

The following APIs will be removed in the future, although there is currently no date scheduled for their removal.

Removed

asynchat and asyncore

configparser

distutils

ensurepip

enum

ftplib

gzip

hashlib

importlib

imp

io

locale

smtpd

sqlite3

ssl

unittest

webbrowser

xml.etree.ElementTree

zipimport

Others

Porting to Python 3.12

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in the Python API

Build Changes

C API Changes

New Features

Porting to Python 3.12

Deprecated

Pending removal in Python 3.14

Pending removal in Python 3.15

Pending removal in Python 3.16

Pending removal in future versions

The following APIs are deprecated and will be removed, although there is currently no date scheduled for their removal.

Removed

What's New In Python 3.13

Editors
Adam Turner and Thomas Wouters

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the VCS log when researching a change.

This article explains the new features in Python 3.13, compared to 3.12. Python 3.13 was released on October 7, 2024. For full details, see the changelog <changelog>.

719 -- Python 3.13 Release Schedule

Summary -- Release Highlights

Python 3.13 is a stable release of the Python programming language, with a mix of changes to the language, the implementation and the standard library. The biggest changes include a new interactive interpreter, experimental support for running in a free-threaded mode (703), and a Just-In-Time compiler (744).

Error messages continue to improve, with tracebacks now highlighted in color by default. The locals builtin now has defined semantics <whatsnew313-locals-semantics> for changing the returned mapping, and type parameters now support default values.

The library changes contain removal of deprecated APIs and modules, as well as the usual improvements in user-friendliness and correctness. Several legacy standard library modules have now been removed following their deprecation in Python 3.11 (594).

This article doesn't attempt to provide a complete specification of all new features, but instead gives a convenient overview. For full details refer to the documentation, such as the Library Reference <library-index> and Language Reference <reference-index>. To understand the complete implementation and design rationale for a change, refer to the PEP for a particular new feature; but note that PEPs usually are not kept up-to-date once a feature has been fully implemented. See Porting to Python 3.13 for guidance on upgrading from earlier versions of Python.

Interpreter improvements:

Python data model improvements:

Significant improvements in the standard library:

Security improvements:

C API improvements:

New typing features:

Platform support:

Important removals:

Release schedule changes:

602 ("Annual Release Cycle for Python") has been updated to extend the full support ('bugfix') period for new releases to two years. This updated policy means that:

New Features

A better interactive interpreter

Python now uses a new interactive shell by default, based on code from the PyPy project. When the user starts the REPL from an interactive terminal, the following new features are now supported:

To disable the new interactive shell, set the PYTHON_BASIC_REPL environment variable. For more on interactive mode, see tut-interac.

(Contributed by Pablo Galindo Salgado, Łukasz Langa, and Lysandros Nikolaou in 111201 based on code from the PyPy project. Windows support contributed by Dino Viehland and Anthony Shaw.)

Improved error messages

Free-threaded CPython

CPython now has experimental support for running in a free-threaded mode, with the global interpreter lock (GIL) disabled. This is an experimental feature and therefore is not enabled by default. The free-threaded mode requires a different executable, usually called python3.13t or python3.13t.exe. Pre-built binaries marked as free-threaded can be installed as part of the official Windows <install-freethreaded-windows> and macOS <install-freethreaded-macos> installers, or CPython can be built from source with the --disable-gil option.

Free-threaded execution allows for full utilization of the available processing power by running threads in parallel on available CPU cores. While not all software will benefit from this automatically, programs designed with threading in mind will run faster on multi-core hardware. The free-threaded mode is experimental and work is ongoing to improve it: expect some bugs and a substantial single-threaded performance hit. Free-threaded builds of CPython support optionally running with the GIL enabled at runtime using the environment variable PYTHON_GIL or the command-line option -X gil=1.

To check if the current interpreter supports free-threading, python -VV <-V> and sys.version contain "experimental free-threading build". The new !sys._is_gil_enabled function can be used to check whether the GIL is actually disabled in the running process.

C-API extension modules need to be built specifically for the free-threaded build. Extensions that support running with the GIL disabled should use the Py_mod_gil slot. Extensions using single-phase init should use PyUnstable_Module_SetGIL to indicate whether they support running with the GIL disabled. Importing C extensions that don't use these mechanisms will cause the GIL to be enabled, unless the GIL was explicitly disabled with the PYTHON_GIL environment variable or the -X gil=0 option. pip 24.1 or newer is required to install packages with C extensions in the free-threaded build.

This work was made possible thanks to many individuals and organizations, including the large community of contributors to Python and third-party projects to test and enable free-threading support. Notable contributors include: Sam Gross, Ken Jin, Donghee Na, Itamar Oren, Matt Page, Brett Simmers, Dino Viehland, Carl Meyer, Nathan Goldbaum, Ralf Gommers, Lysandros Nikolaou, and many others. Many of these contributors are employed by Meta, which has provided significant engineering resources to support this project.

703 "Making the Global Interpreter Lock Optional in CPython" contains rationale and information surrounding this work.

Porting Extension Modules to Support Free-Threading: A community-maintained porting guide for extension authors.

An experimental just-in-time (JIT) compiler

When CPython is configured and built using the !--enable-experimental-jit option, a just-in-time (JIT) compiler is added which may speed up some Python programs. On Windows, use PCbuild/build.bat --experimental-jit to enable the JIT or --experimental-jit-interpreter to enable the Tier 2 interpreter. Build requirements and further supporting information are contained at Tools/jit/README.md.

The !--enable-experimental-jit option takes these (optional) values, defaulting to yes if !--enable-experimental-jit is present without the optional value.

The internal architecture is roughly as follows:

744

(JIT by Brandt Bucher, inspired by a paper by Haoran Xu and Fredrik Kjolstad. Tier 2 IR by Mark Shannon and Guido van Rossum. Tier 2 optimizer by Ken Jin.)

Defined mutation semantics for locals

Historically, the expected result of mutating the return value of locals has been left to individual Python implementations to define. Starting from Python 3.13, 667 standardises the historical behavior of CPython for most code execution scopes, but changes optimized scopes <optimized scope> (functions, generators, coroutines, comprehensions, and generator expressions) to explicitly return independent snapshots of the currently assigned local variables, including locally referenced nonlocal variables captured in closures.

This change to the semantics of locals in optimized scopes also affects the default behavior of code execution functions that implicitly target !locals if no explicit namespace is provided (such as exec and eval). In previous versions, whether or not changes could be accessed by calling !locals after calling the code execution function was implementation-dependent. In CPython specifically, such code would typically appear to work as desired, but could sometimes fail in optimized scopes based on other code (including debuggers and code execution tracing tools) potentially resetting the shared snapshot in that scope. Now, the code will always run against an independent snapshot of the local variables in optimized scopes, and hence the changes will never be visible in subsequent calls to !locals. To access the changes made in these cases, an explicit namespace reference must now be passed to the relevant function. Alternatively, it may make sense to update affected code to use a higher level code execution API that returns the resulting code execution namespace (e.g. runpy.run_path when executing Python files from disk).

To ensure debuggers and similar tools can reliably update local variables in scopes affected by this change, FrameType.f_locals <frame.f_locals> now returns a write-through proxy to the frame's local and locally referenced nonlocal variables in these scopes, rather than returning an inconsistently updated shared dict instance with undefined runtime semantics.

See 667 for more details, including related C API changes and deprecations. Porting notes are also provided below for the affected Python APIs <pep667-porting-notes-py> and C APIs <pep667-porting-notes-c>.

(PEP and implementation contributed by Mark Shannon and Tian Gao in 74929. Documentation updates provided by Guido van Rossum and Alyssa Coghlan.)

Support for mobile platforms

730: iOS is now a 11 supported platform, with the arm64-apple-ios and arm64-apple-ios-simulator targets at tier 3 (iPhone and iPad devices released after 2013 and the Xcode iOS simulator running on Apple silicon hardware, respectively). x86_64-apple-ios-simulator (the Xcode iOS simulator running on older x86_64 hardware) is not a tier 3 supported platform, but will have best-effort support. (PEP written and implementation contributed by Russell Keith-Magee in 114099.)

738: Android is now a 11 supported platform, with the aarch64-linux-android and x86_64-linux-android targets at tier 3. The 32-bit targets arm-linux-androideabi and i686-linux-android are not tier 3 supported platforms, but will have best-effort support. (PEP written and implementation contributed by Malcolm Smith in 116622.)

730, 738

Other Language Changes

New Modules

Improved Modules

argparse

array

ast

asyncio

base64

compileall

concurrent.futures

configparser

copy

ctypes

dbm

dis

doctest

email

enum

fractions

glob

importlib

io

ipaddress

itertools

marshal

math

mimetypes

mmap

multiprocessing

os

os.path

pathlib

pdb

queue

random

re

shutil

site

sqlite3

ssl

statistics

subprocess

sys

tempfile

time

tkinter

traceback

types

typing

unicodedata

venv

warnings

xml

zipimport

Optimizations

Removed Modules And APIs

PEP 594: Remove "dead batteries" from the standard library

594 proposed removing 19 modules from the standard library, colloquially referred to as 'dead batteries' due to their historic, obsolete, or insecure status. All of the following modules were deprecated in Python 3.11, and are now removed:

(Contributed by Victor Stinner and Zachary Ware in 104773 and 104780.)

2to3

builtins

configparser

importlib.metadata

locale

opcode

optparse

pathlib

re

tkinter.tix

turtle

typing

unittest

urllib

webbrowser

New Deprecations

Pending removal in Python 3.14

Pending removal in Python 3.15

Pending removal in Python 3.16

Pending removal in Python 3.17

Pending removal in Python 3.18

Pending removal in Python 3.19

Pending removal in Python 3.20

Pending removal in Python 3.21

Pending removal in future versions

The following APIs will be removed in the future, although there is currently no date scheduled for their removal.

CPython Bytecode Changes

C API Changes

New Features

Changed C APIs

Limited C API Changes

Removed C APIs

Deprecated C APIs

Pending removal in Python 3.14

Pending removal in Python 3.15

Pending removal in Python 3.16

Pending removal in Python 3.18

Pending removal in future versions

The following APIs are deprecated and will be removed, although there is currently no date scheduled for their removal.

Build Changes

Porting to Python 3.13

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in the Python API

Changes in the C API

Regression Test Changes

What's new in Python 3.14

Editors
Adam Turner and Hugo van Kemenade

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the VCS log when researching a change.

This article explains the new features in Python 3.14, compared to 3.13. Python 3.14 was released on 7 October 2025. For full details, see the changelog <changelog>.

745 -- Python 3.14 release schedule

Summary -- Release highlights

Python 3.14 is a stable release of the Python programming language, with a mix of changes to the language, the implementation, and the standard library. The biggest changes include template string literals <whatsnew314-template-string-literals>, deferred evaluation of annotations <whatsnew314-deferred-annotations>, and support for subinterpreters <whatsnew314-multiple-interpreters> in the standard library.

The library changes include significantly improved capabilities for introspection in asyncio <whatsnew314-asyncio-introspection>, support for Zstandard <whatsnew314-zstandard> via a new compression.zstd module, syntax highlighting in the REPL, as well as the usual deprecations and removals, and improvements in user-friendliness and correctness.

This article doesn't attempt to provide a complete specification of all new features, but instead gives a convenient overview. For full details refer to the documentation, such as the Library Reference <library-index> and Language Reference <reference-index>. To understand the complete implementation and design rationale for a change, refer to the PEP for a particular new feature; but note that PEPs usually are not kept up-to-date once a feature has been fully implemented. See Porting to Python 3.14 for guidance on upgrading from earlier versions of Python.

Interpreter improvements:

Significant improvements in the standard library:

C API improvements:

Platform support:

Release changes:

New features

649 & 749: Deferred evaluation of annotations

The annotations <annotation> on functions, classes, and modules are no longer evaluated eagerly. Instead, annotations are stored in special-purpose annotate functions <annotate function> and evaluated only when necessary (except if from __future__ import annotations is used).

This change is designed to improve performance and usability of annotations in Python in most circumstances. The runtime cost for defining annotations is minimized, but it remains possible to introspect annotations at runtime. It is no longer necessary to enclose annotations in strings if they contain forward references.

The new annotationlib module provides tools for inspecting deferred annotations. Annotations may be evaluated in the ~annotationlib.Format.VALUE format (which evaluates annotations to runtime values, similar to the behavior in earlier Python versions), the ~annotationlib.Format.FORWARDREF format (which replaces undefined names with special markers), and the ~annotationlib.Format.STRING format (which returns annotations as strings).

This example shows how these formats behave:

>>> from annotationlib import get_annotations, Format >>> def func(arg: Undefined): ... pass >>> get_annotations(func, format=Format.VALUE) Traceback (most recent call last): ... NameError: name 'Undefined' is not defined >>> get_annotations(func, format=Format.FORWARDREF) {'arg': ForwardRef('Undefined', owner=<function func at 0x...>)} >>> get_annotations(func, format=Format.STRING) {'arg': 'Undefined'}

The porting <whatsnew314-porting-annotations> section contains guidance on changes that may be needed due to these changes, though in the majority of cases, code will continue working as-is.

(Contributed by Jelle Zijlstra in 749 and 119180; 649 was written by Larry Hastings.)

649
Deferred Evaluation Of Annotations Using Descriptors

749
Implementing PEP 649

734: Multiple interpreters in the standard library

The CPython runtime supports running multiple copies of Python in the same process simultaneously and has done so for over 20 years. Each of these separate copies is called an 'interpreter'. However, the feature had been available only through the C-API <sub-interpreter-support>.

That limitation is removed in Python 3.14, with the new concurrent.interpreters module.

There are at least two notable reasons why using multiple interpreters has significant benefits:

For some use cases, concurrency in software improves efficiency and can simplify design, at a high level. At the same time, implementing and maintaining all but the simplest concurrency is often a struggle for the human brain. That especially applies to plain threads (for example, threading), where all memory is shared between all threads.

With multiple isolated interpreters, you can take advantage of a class of concurrency models, like Communicating Sequential Processes (CSP) or the actor model, that have found success in other programming languages, like Smalltalk, Erlang, Haskell, and Go. Think of multiple interpreters as threads but with opt-in sharing.

Regarding multi-core parallelism: as of Python 3.12, interpreters are now sufficiently isolated from one another to be used in parallel (see 684). This unlocks a variety of CPU-intensive use cases for Python that were limited by the GIL.

Using multiple interpreters is similar in many ways to multiprocessing, in that they both provide isolated logical "processes" that can run in parallel, with no sharing by default. However, when using multiple interpreters, an application will use fewer system resources and will operate more efficiently (since it stays within the same process). Think of multiple interpreters as having the isolation of processes with the efficiency of threads.

While the feature has been around for decades, multiple interpreters have not been used widely, due to low awareness and the lack of a standard library module. Consequently, they currently have several notable limitations, which are expected to improve significantly now that the feature is going mainstream.

Current limitations:

The impact of these limitations will depend on future CPython improvements, how interpreters are used, and what the community solves through PyPI packages. Depending on the use case, the limitations may not have much impact, so try it out!

Furthermore, future CPython releases will reduce or eliminate overhead and provide utilities that are less appropriate on PyPI. In the meantime, most of the limitations can also be addressed through extension modules, meaning PyPI packages can fill any gap for 3.14, and even back to 3.12 where interpreters were finally properly isolated and stopped sharing the GIL. Likewise, libraries on PyPI are expected to emerge for high-level abstractions on top of interpreters.

Regarding extension modules, work is in progress to update some PyPI projects, as well as tools like Cython, pybind11, nanobind, and PyO3. The steps for isolating an extension module are found at isolating-extensions-howto. Isolating a module has a lot of overlap with what is required to support free-threading <whatsnew314-free-threaded-cpython>, so the ongoing work in the community in that area will help accelerate support for multiple interpreters.

Also added in 3.14: concurrent.futures.InterpreterPoolExecutor <whatsnew314-concurrent-futures-interp-pool>.

(Contributed by Eric Snow in 134939.)

734

750: Template string literals

Template strings are a new mechanism for custom string processing. They share the familiar syntax of f-strings but, unlike f-strings, return an object representing the static and interpolated parts of the string, instead of a simple str.

To write a t-string, use a 't' prefix instead of an 'f':

>>> variety = 'Stilton' >>> template = t'Try some {variety} cheese!' >>> type(template) <class 'string.templatelib.Template'>

~string.templatelib.Template objects provide access to the static and interpolated (in curly braces) parts of a string before they are combined. Iterate over !Template instances to access their parts in order:

variety = 'Stilton' template = t'Try some {variety} cheese!'

>>> list(template) ['Try some ', Interpolation('Stilton', 'variety', None, ''), ' cheese!']

It's easy to write (or call) code to process !Template instances. For example, here's a function that renders static parts lowercase and ~string.templatelib.Interpolation instances uppercase:

from string.templatelib import Interpolation

def lower_upper(template):
    """Render static parts lowercase and interpolations uppercase."""
    parts = []
    for part in template:
        if isinstance(part, Interpolation):
            parts.append(str(part.value).upper())
        else:
            parts.append(part.lower())
    return ''.join(parts)

name = 'Wenslydale'
template = t'Mister {name}'
assert lower_upper(template) == 'mister WENSLYDALE'

Because !Template instances distinguish between static strings and interpolations at runtime, they can be useful for sanitising user input. Writing a !html function that escapes user input in HTML is an exercise left to the reader! Template processing code can provide improved flexibility. For instance, a more advanced !html function could accept a !dict of HTML attributes directly in the template:

attributes = {'src': 'limburger.jpg', 'alt': 'lovely cheese'}
template = t'<img {attributes}>'
assert html(template) == '<img src="limburger.jpg" alt="lovely cheese" />'

Of course, template processing code does not need to return a string-like result. An even more advanced !html could return a custom type representing a DOM-like structure.

With t-strings in place, developers can write systems that sanitise SQL, make safe shell operations, improve logging, tackle modern ideas in web development (HTML, CSS, and so on), and implement lightweight custom business DSLs.

(Contributed by Jim Baker, Guido van Rossum, Paul Everitt, Koudai Aono, Lysandros Nikolaou, Dave Peck, Adam Turner, Jelle Zijlstra, Bénédikt Tran, and Pablo Galindo Salgado in 132661.)

750.

768: Safe external debugger interface

Python 3.14 introduces a zero-overhead debugging interface that allows debuggers and profilers to safely attach to running Python processes without stopping or restarting them. This is a significant enhancement to Python's debugging capabilities, meaning that unsafe alternatives are no longer required.

The new interface provides safe execution points for attaching debugger code without modifying the interpreter's normal execution path or adding any overhead at runtime. Due to this, tools can now inspect and interact with Python applications in real-time, which is a crucial capability for high-availability systems and production environments.

For convenience, this interface is implemented in the sys.remote_exec function. For example:

import os, sys
from tempfile import NamedTemporaryFile

with NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
    script_path = f.name
    f.write(f'import my_debugger; my_debugger.connect({os.getpid()})')

# Execute in process with PID 1234
print('Behold! An offering:')
sys.remote_exec(1234, script_path)

This function allows sending Python code to be executed in a target process at the next safe execution point. However, tool authors can also implement the protocol directly as described in the PEP, which details the underlying mechanisms used to safely attach to running processes.

The debugging interface has been carefully designed with security in mind and includes several mechanisms to control access:

(Contributed by Pablo Galindo Salgado, Matt Wozniski, and Ivona Stojanovic in 131591.)

768.

A new type of interpreter

A new type of interpreter has been added to CPython. It uses tail calls between small C functions that implement individual Python opcodes, rather than one large C case statement. For certain newer compilers, this interpreter provides significantly better performance. Preliminary benchmarks suggest a geometric mean of 3-5% faster on the standard pyperformance benchmark suite, depending on platform and architecture. The baseline is Python 3.14 built with Clang 19, without this new interpreter.

This interpreter currently only works with Clang 19 and newer on x86-64 and AArch64 architectures. However, a future release of GCC is expected to support this as well.

This feature is opt-in for now. Enabling profile-guided optimization is highly recommended when using the new interpreter as it is the only configuration that has been tested and validated for improved performance. For further information, see --with-tail-call-interp.

[!NOTE] This is not to be confused with tail call optimization of Python functions, which is currently not implemented in CPython.

This new interpreter type is an internal implementation detail of the CPython interpreter. It doesn't change the visible behavior of Python programs at all. It can improve their performance, but doesn't change anything else.

(Contributed by Ken Jin in 128563, with ideas on how to implement this in CPython by Mark Shannon, Garrett Gu, Haoran Xu, and Josh Haberman.)

Free-threaded mode improvements

CPython's free-threaded mode (703), initially added in 3.13, has been significantly improved in Python 3.14. The implementation described in PEP 703 has been finished, including C API changes, and temporary workarounds in the interpreter were replaced with more permanent solutions. The specializing adaptive interpreter (659) is now enabled in free-threaded mode, which along with many other optimizations greatly improves its performance. The performance penalty on single-threaded code in free-threaded mode is now roughly 5-10%, depending on the platform and C compiler used.

From Python 3.14, when compiling extension modules for the free-threaded build of CPython on Windows, the preprocessor variable Py_GIL_DISABLED now needs to be specified by the build backend, as it will no longer be determined automatically by the C compiler. For a running interpreter, the setting that was used at compile time can be found using sysconfig.get_config_var.

The new -X context_aware_warnings <-X> flag controls if concurrent safe warnings control <whatsnew314-concurrent-warnings-control> is enabled. The flag defaults to true for the free-threaded build and false for the GIL-enabled build.

A new ~sys.flags.thread_inherit_context flag has been added, which if enabled means that threads created with threading.Thread start with a copy of the ~contextvars.Context() of the caller of ~threading.Thread.start. Most significantly, this makes the warning filtering context established by ~warnings.catch_warnings be "inherited" by threads (or asyncio tasks) started within that context. It also affects other modules that use context variables, such as the decimal context manager. This flag defaults to true for the free-threaded build and false for the GIL-enabled build.

(Contributed by Sam Gross, Matt Page, Neil Schemenauer, Thomas Wouters, Donghee Na, Kirill Podoprigora, Ken Jin, Itamar Oren, Brett Simmers, Dino Viehland, Nathan Goldbaum, Ralf Gommers, Lysandros Nikolaou, Kumar Aditya, Edgar Margffoy, and many others. Some of these contributors are employed by Meta, which has continued to provide significant engineering resources to support this project.)

Improved error messages

784: Zstandard support in the standard library

The new compression package contains modules !compression.lzma, !compression.bz2, !compression.gzip and !compression.zlib which re-export the lzma, bz2, gzip and zlib modules respectively. The new import names under !compression are the preferred names for importing these compression modules from Python 3.14. However, the existing modules names have not been deprecated. Any deprecation or removal of the existing compression modules will occur no sooner than five years after the release of 3.14.

The new compression.zstd module provides compression and decompression APIs for the Zstandard format via bindings to Meta's zstd library. Zstandard is a widely adopted, highly efficient, and fast compression format. In addition to the APIs introduced in !compression.zstd, support for reading and writing Zstandard compressed archives has been added to the tarfile, zipfile, and shutil modules.

Here's an example of using the new module to compress some data:

from compression import zstd
import math

data = str(math.pi).encode() * 20
compressed = zstd.compress(data)
ratio = len(compressed) / len(data)
print(f"Achieved compression ratio of {ratio}")

As can be seen, the API is similar to the APIs of the !lzma and !bz2 modules.

(Contributed by Emma Harper Smith, Adam Turner, Gregory P. Smith, Tomas Roun, Victor Stinner, and Rogdham in 132983.)

784.

Asyncio introspection capabilities

Added a new command-line interface to inspect running Python processes using asynchronous tasks, available via python -m asyncio ps PID or python -m asyncio pstree PID.

The ps subcommand inspects the given process ID (PID) and displays information about currently running asyncio tasks. It outputs a task table: a flat listing of all tasks, their names, their coroutine stacks, and which tasks are awaiting them.

The pstree subcommand fetches the same information, but instead renders a visual async call tree, showing coroutine relationships in a hierarchical format. This command is particularly useful for debugging long-running or stuck asynchronous programs. It can help developers quickly identify where a program is blocked, what tasks are pending, and how coroutines are chained together.

For example given this code:

import asyncio

async def play_track(track):
    await asyncio.sleep(5)
    print(f'🎵 Finished: {track}')

async def play_album(name, tracks):
    async with asyncio.TaskGroup() as tg:
        for track in tracks:
            tg.create_task(play_track(track), name=track)

async def main():
    async with asyncio.TaskGroup() as tg:
        tg.create_task(
          play_album('Sundowning', ['TNDNBTG', 'Levitate']),
          name='Sundowning')
        tg.create_task(
          play_album('TMBTE', ['DYWTYLM', 'Aqua Regia']),
          name='TMBTE')

if __name__ == '__main__':
    asyncio.run(main())

Executing the new tool on the running process will yield a table like this:

python -m asyncio ps 12345

tid        task id              task name            coroutine stack                                    awaiter chain                                      awaiter name    awaiter id
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1935500    0x7fc930c18050       Task-1               TaskGroup._aexit -> TaskGroup.__aexit__ -> main                                                                       0x0
1935500    0x7fc930c18230       Sundowning           TaskGroup._aexit -> TaskGroup.__aexit__ -> album   TaskGroup._aexit -> TaskGroup.__aexit__ -> main    Task-1          0x7fc930c18050
1935500    0x7fc93173fa50       TMBTE                TaskGroup._aexit -> TaskGroup.__aexit__ -> album   TaskGroup._aexit -> TaskGroup.__aexit__ -> main    Task-1          0x7fc930c18050
1935500    0x7fc93173fdf0       TNDNBTG              sleep -> play                                      TaskGroup._aexit -> TaskGroup.__aexit__ -> album   Sundowning      0x7fc930c18230
1935500    0x7fc930d32510       Levitate             sleep -> play                                      TaskGroup._aexit -> TaskGroup.__aexit__ -> album   Sundowning      0x7fc930c18230
1935500    0x7fc930d32890       DYWTYLM              sleep -> play                                      TaskGroup._aexit -> TaskGroup.__aexit__ -> album   TMBTE           0x7fc93173fa50
1935500    0x7fc93161ec30       Aqua Regia           sleep -> play                                      TaskGroup._aexit -> TaskGroup.__aexit__ -> album   TMBTE           0x7fc93173fa50

or a tree like this:

python -m asyncio pstree 12345

└── (T) Task-1
    └──  main example.py:13
        └──  TaskGroup.__aexit__ Lib/asyncio/taskgroups.py:72
            └──  TaskGroup._aexit Lib/asyncio/taskgroups.py:121
                ├── (T) Sundowning
                │   └──  album example.py:8
                │       └──  TaskGroup.__aexit__ Lib/asyncio/taskgroups.py:72
                │           └──  TaskGroup._aexit Lib/asyncio/taskgroups.py:121
                │               ├── (T) TNDNBTG
                │               │   └──  play example.py:4
                │               │       └──  sleep Lib/asyncio/tasks.py:702
                │               └── (T) Levitate
                │                   └──  play example.py:4
                │                       └──  sleep Lib/asyncio/tasks.py:702
                └── (T) TMBTE
                    └──  album example.py:8
                        └──  TaskGroup.__aexit__ Lib/asyncio/taskgroups.py:72
                            └──  TaskGroup._aexit Lib/asyncio/taskgroups.py:121
                                ├── (T) DYWTYLM
                                │   └──  play example.py:4
                                │       └──  sleep Lib/asyncio/tasks.py:702
                                └── (T) Aqua Regia
                                    └──  play example.py:4
                                        └──  sleep Lib/asyncio/tasks.py:702

If a cycle is detected in the async await graph (which could indicate a programming issue), the tool raises an error and lists the cycle paths that prevent tree construction:

python -m asyncio pstree 12345

ERROR: await-graph contains cycles - cannot print a tree!

cycle: Task-2 → Task-3 → Task-2

(Contributed by Pablo Galindo, Łukasz Langa, Yury Selivanov, and Marta Gomez Macias in 91048.)

Concurrent safe warnings control

The warnings.catch_warnings context manager will now optionally use a context variable for warning filters. This is enabled by setting the ~sys.flags.context_aware_warnings flag, either with the -X command-line option or an environment variable. This gives predictable warnings control when using ~warnings.catch_warnings combined with multiple threads or asynchronous tasks. The flag defaults to true for the free-threaded build and false for the GIL-enabled build.

(Contributed by Neil Schemenauer and Kumar Aditya in 130010.)

Other language changes

Built-ins

Command line and environment

PEP 758: Allow except and except* expressions without brackets

The except and except* <except_star> expressions now allow brackets to be omitted when there are multiple exception types and the as clause is not used. For example:

try:
    connect_to_server()
except TimeoutError, ConnectionRefusedError:
    print('The network has ceased to be!')

(Contributed by Pablo Galindo and Brett Cannon in 758 and 131831.)

PEP 765: Control flow in finally blocks

The compiler now emits a SyntaxWarning when a return, break, or continue statement have the effect of leaving a finally block. This change is specified in 765.

In situations where this change is inconvenient (such as those where the warnings are redundant due to code linting), the warning filter <warning-filter> can be used to turn off all syntax warnings by adding ignore::SyntaxWarning as a filter. This can be specified in combination with a filter that converts other warnings to errors (for example, passing -Werror -Wignore::SyntaxWarning as CLI options, or setting PYTHONWARNINGS=error,ignore::SyntaxWarning).

Note that applying such a filter at runtime using the warnings module will only suppress the warning in code that is compiled after the filter is adjusted. Code that is compiled prior to the filter adjustment (for example, when a module is imported) will still emit the syntax warning.

(Contributed by Irit Katriel in 130080.)

Garbage collection

From Python 3.14.5 onwards:

The garbage collector (GC) has changed in Python 3.14.5.

Python 3.14.0-3.14.4 shipped with a new incremental GC. However, due to a number of reports of significant memory pressure in production environments, it has been reverted back to the generational GC from 3.13. This is the GC now used in Python 3.14.5 and later.

Previously in Python 3.14.0-3.14.4:

The cycle garbage collector is now incremental. This means that maximum pause times are reduced by an order of magnitude or more for larger heaps.

There are now only two generations: young and old. When gc.collect is not called directly, the GC is invoked a little less frequently. When invoked, it collects the young generation and an increment of the old generation, instead of collecting one or more generations.

The behavior of !gc.collect changes slightly:

(Contributed by Mark Shannon in 108362.)

Default interactive shell

New modules

Improved modules

argparse

ast

asyncio

calendar

concurrent.futures

configparser

contextvars

ctypes

curses

datetime

decimal

difflib

dis

errno

faulthandler

fnmatch

fractions

functools

getopt

getpass

graphlib

heapq

hmac

http

imaplib

inspect

io

json

linecache

logging.handlers

math

mimetypes

multiprocessing

operator

os

os.path

pathlib

pdb

pickle

platform

pydoc

re

socket

ssl

struct

symtable

sys

sys.monitoring

sysconfig

tarfile

threading

tkinter

turtle

types

typing

unicodedata

unittest

urllib

uuid

webbrowser

zipfile

Optimizations

asyncio

base64

bdb

difflib

gc

io

pathlib

pdb

textwrap

uuid

zlib

Removed

argparse

ast

asyncio

email

importlib.abc

itertools

pathlib

pkgutil

pty

sqlite3

urllib

Deprecated

New deprecations

Pending removal in Python 3.15

Pending removal in Python 3.16

Pending removal in Python 3.17

Pending removal in Python 3.18

Pending removal in Python 3.19

Pending removal in Python 3.20

Pending removal in Python 3.21

Pending removal in future versions

The following APIs will be removed in the future, although there is currently no date scheduled for their removal.

CPython bytecode changes

Pseudo-instructions

C API changes

Python configuration C API

Add a PyInitConfig C API <pyinitconfig_api> to configure the Python initialization without relying on C structures and the ability to make ABI-compatible changes in the future.

Complete the 587 PyConfig C API <pyconfig_api> by adding PyInitConfig_AddModule which can be used to add a built-in extension module; a feature previously referred to as the "inittab".

Add PyConfig_Get and PyConfig_Set functions to get and set the current runtime configuration.

587 'Python Initialization Configuration' unified all the ways to configure Python's initialization. This PEP also unifies the configuration of Python's preinitialization and initialization in a single API. Moreover, this PEP only provides a single choice to embed Python, instead of having two 'Python' and 'Isolated' choices (PEP 587), to further simplify the API.

The lower level PEP 587 PyConfig API remains available for use cases with an intentionally higher level of coupling to CPython implementation details (such as emulating the full functionality of CPython's CLI, including its configuration mechanisms).

(Contributed by Victor Stinner in 107954.)

741 and 587

New features in the C API

Limited C API changes

Removed C APIs

Deprecated C APIs

Pending removal in Python 3.15

Pending removal in Python 3.16

Pending removal in Python 3.18

Pending removal in future versions

The following APIs are deprecated and will be removed, although there is currently no date scheduled for their removal.

Build changes

build-details.json

Installations of Python now contain a new file, build-details.json. This is a static JSON document containing build details for CPython, to allow for introspection without needing to run code. This is helpful for use-cases such as Python launchers, cross-compilation, and so on.

build-details.json must be installed in the platform-independent standard library directory. This corresponds to the 'stdlib' <installation_paths> sysconfig installation path, which can be found by running sysconfig.get_path('stdlib').

739 -- build-details.json 1.0 -- a static description file for Python build details

Discontinuation of PGP signatures

PGP (Pretty Good Privacy) signatures will not be provided for releases of Python 3.14 or future versions. To verify CPython artifacts, users must use Sigstore verification materials. Releases have been signed using Sigstore since Python 3.11.

This change in release process was specified in 761.

Free-threaded Python is officially supported

The free-threaded build of Python is now supported and no longer experimental. This is the start of phase II where free-threaded Python is officially supported but still optional.

The free-threading team are confident that the project is on the right path, and appreciate the continued dedication from everyone working to make free-threading ready for broader adoption across the Python community.

With these recommendations and the acceptance of this PEP, the Python developer community should broadly advertise that free-threading is a supported Python build option now and into the future, and that it will not be removed without a proper deprecation schedule.

Any decision to transition to phase III, with free-threading as the default or sole build of Python is still undecided, and dependent on many factors both within CPython itself and the community. This decision is for the future.

779

PEP 779's acceptance

Binary releases for the experimental just-in-time compiler

The official macOS and Windows release binaries now include an experimental just-in-time (JIT) compiler. Although it is not recommended for production use, it can be tested by setting PYTHON_JIT=1 <PYTHON_JIT> as an environment variable. Downstream source builds and redistributors can use the --enable-experimental-jit=yes-off configuration option for similar behavior.

The JIT is at an early stage and still in active development. As such, the typical performance impact of enabling it can range from 10% slower to 20% faster, depending on workload. To aid in testing and evaluation, a set of introspection functions has been provided in the sys._jit namespace. sys._jit.is_available can be used to determine if the current executable supports JIT compilation, while sys._jit.is_enabled can be used to tell if JIT compilation has been enabled for the current process.

Currently, the most significant missing functionality is that native debuggers and profilers like gdb and perf are unable to unwind through JIT frames (Python debuggers and profilers, like pdb or profile, continue to work without modification). Free-threaded builds do not support JIT compilation.

Please report any bugs or major performance regressions that you encounter!

744

Porting to Python 3.14

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in the Python API

Changes in annotations (649 and 749)

This section contains guidance on changes that may be needed to annotations or Python code that interacts with or introspects annotations, due to the changes related to deferred evaluation of annotations <whatsnew314-deferred-annotations>.

In the majority of cases, working code from older versions of Python will not require any changes.

Implications for annotated code

If you define annotations in your code (for example, for use with a static type checker), then this change probably does not affect you: you can keep writing annotations the same way you did with previous versions of Python.

You will likely be able to remove quoted strings in annotations, which are frequently used for forward references. Similarly, if you use from __future__ import annotations to avoid having to write strings in annotations, you may well be able to remove that import once you support only Python 3.14 and newer. However, if you rely on third-party libraries that read annotations, those libraries may need changes to support unquoted annotations before they work as expected.

Implications for readers of __annotations__

If your code reads the ~object.__annotations__ attribute on objects, you may want to make changes in order to support code that relies on deferred evaluation of annotations. For example, you may want to use annotationlib.get_annotations with the ~annotationlib.Format.FORWARDREF format, as the dataclasses module now does.

The external typing_extensions package provides partial backports of some of the functionality of the annotationlib module, such as the ~annotationlib.Format enum and the ~annotationlib.get_annotations function. These can be used to write cross-version code that takes advantage of the new behavior in Python 3.14.

The changes in Python 3.14 are designed to rework how !__annotations__ works at runtime while minimizing breakage to code that contains annotations in source code and to code that reads !__annotations__. However, if you rely on undocumented details of the annotation behavior or on private functions in the standard library, there are many ways in which your code may not work in Python 3.14. To safeguard your code against future changes, only use the documented functionality of the annotationlib module.

In particular, do not read annotations directly from the namespace dictionary attribute of type objects. Use annotationlib.get_annotate_from_class_namespace during class construction and annotationlib.get_annotations afterwards.

In previous releases, it was sometimes possible to access class annotations from an instance of an annotated class. This behavior was undocumented and accidental, and will no longer work in Python 3.14.

from __future__ import annotations

In Python 3.7, 563 introduced the from __future__ import annotations future statement <future>, which turns all annotations into strings.

However, this statement is now deprecated and it is expected to be removed in a future version of Python. This removal will not happen until after Python 3.13 reaches its end of life in 2029, being the last version of Python without support for deferred evaluation of annotations.

In Python 3.14, the behavior of code using from __future__ import annotations is unchanged.

Changes in the C API

Notable changes in 3.14.5

gc

What's new in Python 3.15

Editor
Hugo van Kemenade

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the VCS log when researching a change.

This article explains the new features in Python 3.15, compared to 3.14. Python 3.15 was released on 9 October 2026. For full details, see the changelog <changelog>.

790 -- Python 3.15 release schedule

Summary -- Release highlights

Python 3.15 is the latest stable release of the Python programming language, with a mix of changes to the language, the implementation, and the standard library. The biggest changes include lazy imports <whatsnew315-lazy-imports>, frozendict <whatsnew315-frozendict> and sentinel <whatsnew315-sentinel> builtins, UTF-8 as the default encoding <whatsnew315-utf8-default>, unpacking in comprehensions <whatsnew315-unpacking-in-comprehensions>, and a stable ABI for free-threaded builds <whatsnew315-abi3t>.

The library changes include a new profiling package <whatsnew315-profiling-package> with Tachyon <whatsnew315-sampling-profiler>, a high-frequency statistical sampling profiler, more color <whatsnew315-more-color> in command-line output, as well as the usual deprecations and removals, and improvements in user-friendliness and correctness.

This article doesn't attempt to provide a complete specification of all new features, but instead gives a convenient overview. For full details refer to the documentation, such as the Library Reference <library-index> and Language Reference <reference-index>. To understand the complete implementation and design rationale for a change, refer to the PEP for a particular new feature; but note that PEPs usually are not kept up-to-date once a feature has been fully implemented.

See Porting to Python 3.15 for guidance on upgrading from earlier versions of Python.

Interpreter improvements:

Significant improvements in the standard library:

New typing features:

C API improvements:

Build changes:

Release changes:

New features

810: Explicit lazy imports

Large Python applications often suffer from slow startup times. A significant contributor to this problem is the import system: when a module is imported, Python must locate the file, read it from disk, compile it to bytecode, and execute all top-level code. For applications with deep dependency trees, this process can take seconds, even when most of the imported code is never actually used during a particular run.

Developers have worked around this by moving imports inside functions, using importlib to load modules on demand, or restructuring code to avoid unnecessary dependencies. These approaches work but make code harder to read and maintain, scatter import statements throughout the codebase, and require discipline to apply consistently.

Python now provides a cleaner solution through explicit lazy imports using the new lazy soft keyword. When you mark an import as lazy, Python defers the actual module loading until the imported name is first used. This gives you the organizational benefits of declaring all imports at the top of the file while only paying the loading cost for modules you actually use.

The lazy keyword works with both import and from ... import statements. When you write lazy import heavy_module, Python does not immediately load the module. Instead, it creates a lightweight proxy object. The actual module loading happens transparently when you first access the name:

lazy import json
lazy from pathlib import Path

print("Starting up...")  # json and pathlib are not fully imported yet

data = json.loads('{"key": "value"}')  # json gets fully imported here
p = Path(".")  # pathlib gets fully imported here

This mechanism is particularly useful for applications that import many modules at the top level but may only use a subset of them in any given run. The deferred loading reduces startup latency without requiring code restructuring or conditional imports scattered throughout the codebase.

In the case where loading a lazily imported module fails (for example, if the module does not exist), Python raises the exception at the point of first use rather than at import time. The associated traceback includes both the location where the name was accessed and the original import statement, making it straightforward to diagnose and debug the failure.

For cases where you want to enable lazy loading globally without modifying source code, Python provides the -X lazy_imports <-X> command-line option and the PYTHON_LAZY_IMPORTS environment variable. Both accept two values: all makes all imports lazy by default, and normal (the default) respects the lazy keyword in source code. The sys.set_lazy_imports and sys.get_lazy_imports functions allow changing and querying this mode at runtime.

For more selective control, sys.set_lazy_imports_filter accepts a callable that determines whether a specific module should be loaded lazily. The filter receives three arguments: the importing module's name (or None), the imported module's name, and the fromlist (or None for regular imports). It should return True to allow the import to be lazy, or False to force eager loading. This allows patterns like making only your own application's modules lazy while keeping third-party dependencies eager:

import sys

def myapp_filter(importing, imported, fromlist):
    return imported.startswith("myapp.")
sys.set_lazy_imports_filter(myapp_filter)
sys.set_lazy_imports("all")

import myapp.slow_module  # lazy (matches filter)
import json               # eager (does not match filter)

The proxy type itself is available as types.LazyImportType for code that needs to detect lazy imports programmatically.

There are some restrictions on where the lazy keyword can be used. Lazy imports are only permitted at module scope; using lazy inside a function, class body, or try/except/finally block raises a SyntaxError. Neither star imports nor future imports can be lazy (lazy from module import * and lazy from __future__ import ... both raise SyntaxError).

For code that cannot use the lazy keyword directly (for example, when supporting Python versions older than 3.15 while still using lazy imports on 3.15+), a module can define ~module.__lazy_modules__ as a container of fully qualified module name strings. Regular import statements for those modules are then treated as lazy, with the same semantics as the lazy keyword:

__lazy_modules__ = ["json", "pathlib"]

import json     # lazy
import os       # still eager

(Contributed by Pablo Galindo Salgado and Dino Viehland in 142349.)

810 for the full specification and rationale.

814: Add frozendict built-in type

A new immutable type, frozendict, is added to the builtins module. It does not allow modification after creation. A !frozendict is not a subclass of dict; it inherits directly from object. A !frozendict is hashable as long as all of its keys and values are hashable. A !frozendict preserves insertion order, but comparison does not take order into account.

For example:

>>> a = frozendict(x=1, y=2)
>>> a
frozendict({'x': 1, 'y': 2})
>>> a['z'] = 3
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    a['z'] = 3
    ~^^^^^
TypeError: 'frozendict' object does not support item assignment
>>> b = frozendict(y=2, x=1)
>>> hash(a) == hash(b)
True
>>> a == b
True

The following standard library modules have been updated to accept !frozendict: copy, decimal, json, marshal, plistlib (only for serialization), pickle, pprint and xml.etree.ElementTree.

eval and exec accept !frozendict for globals, and type and str.maketrans accept !frozendict for dict.

Code checking for dict type using isinstance(arg, dict) can be updated to isinstance(arg, (dict, frozendict)) to accept also the !frozendict type, or to isinstance(arg, collections.abc.Mapping) to accept also other mapping types such as ~types.MappingProxyType.

(Contributed by Victor Stinner and Donghee Na in 141510.)

814 for the full specification and rationale.

661: Add sentinel built-in type

A new sentinel type is added to the builtins module for creating unique sentinel values with a concise representation. Sentinel objects preserve identity when copied, support use in type expressions with the | operator, and can be pickled when they are importable by module and name.

(Contributed by Jelle Zijlstra in 148829; PEP 661 written by Tal Einat.)

661 for further details.

799: A dedicated profiling package

A new profiling module has been added to organize Python's built-in profiling tools under a single, coherent namespace. This module contains:

The cProfile module remains as an alias for backwards compatibility. The profile module is deprecated and will be removed in Python 3.17.

(Contributed by Pablo Galindo and László Kiss Kollár in 138122.)

799 for further details.

Tachyon: High-frequency statistical sampling profiler

Tachyon profiler logo

A new statistical sampling profiler (Tachyon) has been added as profiling.sampling. This profiler enables low-overhead performance analysis of running Python processes without requiring code modification or process restart.

Unlike deterministic profilers (such as profiling.tracing) that instrument every function call, the sampling profiler periodically captures stack traces from running processes. This approach provides virtually zero overhead while achieving sampling rates of up to 1,000,000 Hz, making it the fastest sampling profiler available for Python (at the time of its contribution) and ideal for debugging performance issues in production environments. This capability is particularly valuable for debugging performance issues in production systems where traditional profiling approaches would be too intrusive.

Key features include:

See profiling.sampling for the complete documentation, including all available output formats, profiling modes, and configuration options.

(Contributed by Pablo Galindo and László Kiss Kollár in 135953 and 138122.)

831: Frame pointers enabled by default

CPython is now built with frame pointers by default on platforms that support them. This uses the compiler flags -fno-omit-frame-pointer and -mno-omit-leaf-frame-pointer, making native stack unwinding faster and more reliable for system profilers, debuggers, crash analysis tools, and eBPF-based observability tools.

The flags are exposed through sysconfig, so extension modules built by tools that consume Python's build configuration inherit frame pointers by default. This propagation is intentional: mixed Python/native profiling needs an unbroken frame-pointer chain through the interpreter, extension modules, embedding applications, and native libraries.

[!IMPORTANT] Third-party build backends and native build systems should preserve these flags when they consume Python's sysconfig values. Build systems that compile C, C++, Rust, or other native code without inheriting Python's compiler flags should enable equivalent frame-pointer flags themselves. A single native component built without frame pointers can break stack unwinding for the whole Python process.

(Contributed by Pablo Galindo Salgado and Savannah Ostrowski in 149201; PEP 831 written by Pablo Galindo Salgado, Ken Jin, Savannah Ostrowski, and Diego Russo.)

831 for further details.

798: Unpacking in comprehensions

List, set, and dictionary comprehensions, as well as generator expressions, now support unpacking with * and **. This extends the unpacking syntax from 448 to comprehensions, providing a new syntax for combining an arbitrary number of iterables or dictionaries into a single flat structure. This new syntax is a direct alternative to nested comprehensions, itertools.chain, and itertools.chain.from_iterable. For example:

>>> lists = [[1, 2], [3, 4], [5]]
>>> [*L for L in lists]  # equivalent to [x for L in lists for x in L]
[1, 2, 3, 4, 5]

>>> sets = [{1, 2}, {2, 3}, {3, 4}]
>>> {*s for s in sets}  # equivalent to {x for s in sets for x in s}
{1, 2, 3, 4}

>>> dicts = [{'a': 1}, {'b': 2}, {'a': 3}]
>>> {**d for d in dicts}  # equivalent to {k: v for d in dicts for k,v in d.items()}
{'a': 3, 'b': 2}

Generator expressions can similarly use unpacking to yield values from multiple iterables:

>>> gen = (*L for L in lists)  # equivalent to (x for L in lists for x in L)
>>> list(gen)
[1, 2, 3, 4, 5]

This change also extends to asynchronous generator expressions, such that, for example, (*a async for a in agen()) is equivalent to (x async for a in agen() for x in a).

(Contributed by Adam Hartz in 143055.)

798 for further details.

829: Package startup configuration files

Loaded by the site module when -S is not given, .pth files <site-pth-files> can contain lines that both extend sys.path and execute arbitrary code when the line starts with import (followed by a space or tab). The latter functionality can be problematic, since it is difficult to know exactly what gets executed when Python starts up.

As a step towards improving the ability to audit pre-start executable code, Python 3.15 introduces .start files <site-start-files> which contain entry point specifications of the form pkg.mod:callable where pkg.mod is the import path to the given callable. When Python starts up, the callable is located and called with no arguments.

import lines in .pth files are silently deprecated. When a matching .start file is found, import lines in .pth files are ignored. There is no change to sys.path extension lines in .pth files.

The site module also provides site.StartupState to batch startup processing for multiple site directories, ensuring all static path extensions are applied before any startup code is executed. site.main uses an instance of this class implicitly to batch process all startup configuration files during normal interpreter startup. Callers needing the same batching behavior can build a ~site.StartupState directly and drive it with ~site.StartupState.addsitedir, ~site.StartupState.addusersitepackages, and ~site.StartupState.addsitepackages, then call ~site.StartupState.process once at the end of the batch.

(Contributed by Barry Warsaw in 148641 and 150228.)

803: Stable ABI for free-threaded builds

C extensions that target the Stable ABI <stable-abi> can now be compiled for the new Stable ABI for Free-Threaded Builds (also known as abi3t), which makes them compatible with free-threaded builds <free-threaded build> of CPython. This usually requires some non-trivial changes to the source code; specifically:

Note that Stable ABI does not offer all the functionality that CPython has to offer. Extensions that cannot switch to abi3t should continue to build for the existing Stable ABI (abi3) and the version-specific ABI for free-threading (cp315t) separately.

Stable ABI for Free-Threaded Builds should typically be selected in a build tool. Tools with abi3t support include meson-python, scikit-build-core, and Maturin; see abi3t-migration-build for links to their documentation on abi3t builds. If your build tool does not support abi3t, compile for cp315t separately. If not using a build tool -- or when writing such a tool -- you can select abi3t by setting the macro !Py_TARGET_ABI3T as discussed in abi3-compiling.

A practical migration guide <abi3t-migration-howto> for switching to abi3t is available. The community-maintained Python Free-Threading Guide provides examples of building and testing abi3t wheels.

803 for further details.

788: Protecting the C API from interpreter finalization

In the C API, interpreter finalization <interpreter shutdown> can be problematic for many extensions, because attaching <attached thread state> a thread state will permanently hang the thread, resulting in deadlocks and other spurious issues. Additionally, it has historically been impossible to safely check whether an interpreter is alive before using it, leading to crashes when a thread concurrently deletes an interpreter while another thread is trying to attach to it.

There are now several new suites of APIs to circumvent these problems:

In addition, APIs in the PyGILState family (most notably PyGILState_Ensure and PyGILState_Release) have been soft deprecated. There is no plan to remove them, and existing code will continue to work, but there will be no new PyGILState APIs in future versions of Python.

(Contributed by Peter Bierma in 149101.)

788 for further details.

Improved error messages

Other language changes

Default interactive shell

New modules

math.integer

This module provides access to the mathematical functions for integer arguments (791). (Contributed by Serhiy Storchaka in 81313.)

Improved modules

argparse

array

ast

asyncio

base64

binascii

calendar

collections

concurrent.futures

contextlib

ctypes

dataclasses

dbm

difflib

email

faulthandler

functools

gc

hashlib

http.client

http.server

importlib.metadata

inspect

json

locale

math

mimetypes

mmap

os

os.path

pathlib

pdb

pickle

pickletools

pprint

re

resource

shelve

shutil

socket

sqlite3

ssl

subprocess

symtable

sys

sys.monitoring

tarfile

threading

timeit

tkinter

tokenize

tomllib

types

typing

unicodedata

unittest

urllib.parse

venv

warnings

wave

(Contributed by Lionel Koenig and Michiel W. Beijen in 60729.)

webbrowser

xml

xml.parsers.expat

zlib

Optimizations

base64 & binascii

csv

Upgraded experimental JIT compiler

Results from the pyperformance benchmark suite report a 7-8% geometric mean performance improvement for the JIT over the standard CPython interpreter built with all optimizations enabled on x86-64 Linux. On AArch64 macOS, the JIT has an 11-12% speedup over the tail calling interpreter <whatsnew314-tail-call-interpreter> with all optimizations enabled. The speedups for JIT builds versus no JIT builds range from roughly 15% slowdown to over 100% speedup (ignoring the unpack_sequence microbenchmark) on x86-64 Linux and AArch64 macOS systems.

The major upgrades to the JIT are:

LLVM 21 build-time dependency

The JIT compiler now uses LLVM 21 for build-time stencil generation. As always, LLVM is only needed when building CPython with the JIT enabled; end users running Python do not need LLVM installed. Instructions for installing LLVM can be found in the JIT compiler documentation for all supported platforms. (Contributed by Savannah Ostrowski in 140973.)

A new tracing frontend

The JIT compiler now supports significantly more bytecode operations and control flow than in Python 3.14, enabling speedups on a wider variety of code. For example, simple Python object creation is now understood by the 3.15 JIT compiler. Overloaded operations and generators are also partially supported. This was made possible by an overhauled JIT tracing frontend that records actual execution paths through code, rather than estimating them as the previous implementation did. (Contributed by Ken Jin in 139109. Support for Windows added by Mark Shannon in 141703.)

Basic register allocation in the JIT

A basic form of register allocation has been added to the JIT compiler's optimizer. This allows the JIT compiler to avoid certain stack operations altogether and instead operate on registers. This allows the JIT to produce more efficient traces by avoiding reads and writes to memory. (Contributed by Mark Shannon in 135379.)

More JIT optimizations

More constant-propagation is now performed. This means when the JIT compiler detects that certain user code results in constants, the code can be simplified by the JIT. (Contributed by Ken Jin and Savannah Ostrowski in 132732.)

Reference counts <reference count> are avoided whenever it is safe to do so. This generally reduces the cost of most operations in Python. (Contributed by Ken Jin, Donghee Na, Zheao Li, Hai Zhu, Savannah Ostrowski, Reiden Ong, Noam Cohen, Tomas Roun, PuQing, Cajetan Rodrigues, and Sacul in 134584.)

By tracking unique references to objects, the JIT optimizer can now eliminate reference count updates and perform in-place operations on ints and floats. (Contributed by Reiden Ong, and Pieter Eendebak in 143414 and 146306.)

The JIT optimizer now supports significantly more operations than in 3.14. (Contributed by Kumar Aditya, Ken Jin, Jiahao Li, and Sacul in 131798.)

GDB and GNU backtrace() unwinding support

The JIT compiler now publishes unwind information for generated machine code to the GDB interface on supported Linux ELF platforms. When libgcc frame registration is available, the same unwind information is also registered for GNU backtrace() stack walkers. This allows native debuggers, crash handlers, and diagnostic tools using these mechanisms to unwind through JIT frames instead of stopping at generated code. (Contributed by Diego Russo and Pablo Galindo Salgado in 146071 and 149104.)

Better machine code generation

The JIT compiler's machine code generator now produces better machine code for x86-64 and AArch64 macOS and Linux targets. In general, users should experience lower memory usage for generated machine code and more efficient machine code versus 3.14. (Contributed by Brandt Bucher in 136528 and 135905. Implementation for AArch64 contributed by Mark Shannon in 139855. Additional optimizations for AArch64 contributed by Mark Shannon and Diego Russo in 140683 and 142305.)

Maintainability

The JIT optimizer's operations have been simplified. This was made possible by a refactoring of JIT data structures. (Contributed by Zhongtian Zheng in 148211 and Hai Zhu in 143421.)

Removed

ast

collections.abc

ctypes

datetime

glob

http.server

importlib

importlib.resources

pathlib

platform

sre*

sysconfig

threading

types

typing

wave

zipimport

Deprecated

New deprecations

Pending removal in Python 3.16

Pending removal in Python 3.17

Pending removal in Python 3.18

Pending removal in Python 3.19

Pending removal in Python 3.20

Pending removal in Python 3.21

Pending removal in future versions

The following APIs will be removed in the future, although there is currently no date scheduled for their removal.

Soft deprecations

There are no plans to remove soft deprecated APIs.

C API changes

New features

Changed C APIs

Removed C APIs

The following functions are removed in favor of PyConfig_Get. The [|pythoncapi_compat_project_link|](https://github.com/python/pythoncapi-compat) can be used to get !PyConfig_Get on Python 3.13 and older.

Deprecated C APIs

Build changes

Porting to Python 3.15

This section lists previously described changes and other bugfixes that may require changes to your code.

What's new in Python 3.16

Editor
Savannah Ostrowski

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the VCS log when researching a change.

This article explains the new features in Python 3.16, compared to 3.15.

For full details, see the changelog <changelog>.

[!NOTE] Prerelease users should be aware that this document is currently in draft form. It will be updated substantially as Python 3.16 moves towards release, so it's worth checking back even after reading earlier versions.

Summary -- Release highlights

New features

Other language changes

Default interactive shell

New modules

Improved modules

argparse

asyncio

bdb

bz2

codecs

compression.zstd

concurrent.futures

concurrent.interpreters

csv

ctypes

curses

datetime

dbm.gnu

dbm.ndbm

decimal

difflib

encodings

gzip

imaplib

io

ipaddress

logging

lzma

math

multiprocessing

os

pydoc

pickle

quopri

re

readline

shlex

shutil

socket

sqlite3

ssl

symtable

tkinter

turtle

unicodedata

uuid

xml

zipfile

zlib

Optimizations

datetime and time

re

uuid

Removed

annotationlib

array

asyncio

functools

logging

mimetypes

shutil

symtable

sys

sysconfig

tarfile

xml.etree.ElementTree

Deprecated

New deprecations

Pending removal in Python 3.17

Pending removal in Python 3.18

Pending removal in Python 3.19

Pending removal in Python 3.20

Pending removal in Python 3.21

Pending removal in future versions

The following APIs will be removed in the future, although there is currently no date scheduled for their removal.

Porting to Python 3.16

This section lists previously described changes and other bugfixes that may require changes to your code.

Build changes

C API changes

New features

Porting to Python 3.16

Deprecated C APIs

Pending removal in Python 3.18

Pending removal in Python 3.19

Pending removal in Python 3.20

Pending removal in future versions

The following APIs are deprecated and will be removed, although there is currently no date scheduled for their removal.

Removed C APIs

What's New In Python 3.2

Author
Raymond Hettinger

This article explains the new features in Python 3.2 as compared to 3.1. Python 3.2 was released on February 20, 2011. It focuses on a few highlights and gives a few examples. For full details, see the Misc/NEWS file.

392 - Python 3.2 Release Schedule

PEP 384: Defining a Stable ABI

In the past, extension modules built for one Python version were often not usable with other Python versions. Particularly on Windows, every feature release of Python required rebuilding all extension modules that one wanted to use. This requirement was the result of the free access to Python interpreter internals that extension modules could use.

With Python 3.2, an alternative approach becomes available: extension modules which restrict themselves to a limited API (by defining Py_LIMITED_API) cannot use many of the internals, but are constrained to a set of API functions that are promised to be stable for several releases. As a consequence, extension modules built for 3.2 in that mode will also work with 3.3, 3.4, and so on. Extension modules that make use of details of memory structures can still be built, but will need to be recompiled for every feature release.

384 - Defining a Stable ABI
PEP written by Martin von Löwis.

PEP 389: Argparse Command Line Parsing Module

A new module for command line parsing, argparse, was introduced to overcome the limitations of optparse which did not provide support for positional arguments (not just options), subcommands, required options and other common patterns of specifying and validating options.

This module has already had widespread success in the community as a third-party module. Being more fully featured than its predecessor, the argparse module is now the preferred module for command-line processing. The older module is still being kept available because of the substantial amount of legacy code that depends on it.

Here's an annotated example parser showing features like limiting results to a set of choices, specifying a metavar in the help screen, validating that one or more positional arguments is present, and making a required option:

import argparse
parser = argparse.ArgumentParser(
            description = 'Manage servers',         # main description for help
            epilog = 'Tested on Solaris and Linux') # displayed after help
parser.add_argument('action',                       # argument name
            choices = ['deploy', 'start', 'stop'],  # three allowed values
            help = 'action on each target')         # help msg
parser.add_argument('targets',
            metavar = 'HOSTNAME',                   # var name used in help msg
            nargs = '+',                            # require one or more targets
            help = 'url for target machines')       # help msg explanation
parser.add_argument('-u', '--user',                 # -u or --user option
            required = True,                        # make it a required argument
            help = 'login as user')

Example of calling the parser on a command string:

>>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain'
>>> result = parser.parse_args(cmd.split())
>>> result.action
'deploy'
>>> result.targets
['sneezy.example.com', 'sleepy.example.com']
>>> result.user
'skycaptain'

Example of the parser's automatically generated help:

>>> parser.parse_args('-h'.split())

usage: manage_cloud.py [-h] -u USER
                       {deploy,start,stop} HOSTNAME [HOSTNAME ...]

Manage servers

positional arguments:
  {deploy,start,stop}   action on each target
  HOSTNAME              url for target machines

optional arguments:
  -h, --help            show this help message and exit
  -u USER, --user USER  login as user

Tested on Solaris and Linux

An especially nice argparse feature is the ability to define subparsers, each with their own argument patterns and help displays:

import argparse
parser = argparse.ArgumentParser(prog='HELM')
subparsers = parser.add_subparsers()

parser_l = subparsers.add_parser('launch', help='Launch Control')   # first subgroup
parser_l.add_argument('-m', '--missiles', action='store_true')
parser_l.add_argument('-t', '--torpedos', action='store_true')

parser_m = subparsers.add_parser('move', help='Move Vessel',        # second subgroup
                                 aliases=('steer', 'turn'))         # equivalent names
parser_m.add_argument('-c', '--course', type=int, required=True)
parser_m.add_argument('-s', '--speed', type=int, default=0)

$ ./helm.py --help                         # top level help (launch and move)
$ ./helm.py launch --help                  # help for launch options
$ ./helm.py launch --missiles              # set missiles=True and torpedos=False
$ ./helm.py steer --course 180 --speed 5   # set movement parameters

389 - New Command Line Parsing Module
PEP written by Steven Bethard.

upgrading-optparse-code for details on the differences from optparse.

PEP 391: Dictionary Based Configuration for Logging

The logging module provided two kinds of configuration, one style with function calls for each option or another style driven by an external file saved in a configparser format. Those options did not provide the flexibility to create configurations from JSON or YAML files, nor did they support incremental configuration, which is needed for specifying logger options from a command line.

To support a more flexible style, the module now offers logging.config.dictConfig for specifying logging configuration with plain Python dictionaries. The configuration options include formatters, handlers, filters, and loggers. Here's a working example of a configuration dictionary:

{"version": 1,
 "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"},
                "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"}
                },
 "handlers": {"console": {
                   "class": "logging.StreamHandler",
                   "formatter": "brief",
                   "level": "INFO",
                   "stream": "ext://sys.stdout"},
              "console_priority": {
                   "class": "logging.StreamHandler",
                   "formatter": "full",
                   "level": "ERROR",
                   "stream": "ext://sys.stderr"}
              },
 "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}}

If that dictionary is stored in a file called conf.json, it can be loaded and called with code like this:

>>> import json, logging.config
>>> with open('conf.json') as f:
...     conf = json.load(f)
...
>>> logging.config.dictConfig(conf)
>>> logging.info("Transaction completed normally")
INFO    : root           : Transaction completed normally
>>> logging.critical("Abnormal termination")
2011-02-17 11:14:36,694 root            CRITICAL Abnormal termination

391 - Dictionary Based Configuration for Logging
PEP written by Vinay Sajip.

PEP 3148: The concurrent.futures module

Code for creating and managing concurrency is being collected in a new top-level namespace, concurrent. Its first member is a futures package which provides a uniform high-level interface for managing threads and processes.

The design for concurrent.futures was inspired by the java.util.concurrent package. In that model, a running call and its result are represented by a ~concurrent.futures.Future object that abstracts features common to threads, processes, and remote procedure calls. That object supports status checks (running or done), timeouts, cancellations, adding callbacks, and access to results or exceptions.

The primary offering of the new module is a pair of executor classes for launching and managing calls. The goal of the executors is to make it easier to use existing tools for making parallel calls. They save the effort needed to setup a pool of resources, launch the calls, create a results queue, add time-out handling, and limit the total number of threads, processes, or remote procedure calls.

Ideally, each application should share a single executor across multiple components so that process and thread limits can be centrally managed. This solves the design challenge that arises when each component has its own competing strategy for resource management.

Both classes share a common interface with three methods: ~concurrent.futures.Executor.submit for scheduling a callable and returning a ~concurrent.futures.Future object; ~concurrent.futures.Executor.map for scheduling many asynchronous calls at a time, and ~concurrent.futures.Executor.shutdown for freeing resources. The class is a context manager and can be used in a with statement to assure that resources are automatically released when currently pending futures are done executing.

A simple of example of ~concurrent.futures.ThreadPoolExecutor is a launch of four parallel threads for copying files:

import concurrent.futures, shutil
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
    e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
    e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest4.txt')

3148 - Futures -- Execute Computations Asynchronously
PEP written by Brian Quinlan.

Code for Threaded Parallel URL reads<threadpoolexecutor-example>, an example using threads to fetch multiple web pages in parallel.

Code for computing prime numbers in parallel<processpoolexecutor-example>, an example demonstrating ~concurrent.futures.ProcessPoolExecutor.

PEP 3147: PYC Repository Directories

Python's scheme for caching bytecode in .pyc files did not work well in environments with multiple Python interpreters. If one interpreter encountered a cached file created by another interpreter, it would recompile the source and overwrite the cached file, thus losing the benefits of caching.

The issue of "pyc fights" has become more pronounced as it has become commonplace for Linux distributions to ship with multiple versions of Python. These conflicts also arise with CPython alternatives such as Unladen Swallow.

To solve this problem, Python's import machinery has been extended to use distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and Unladen Swallow each competing for a file called "mymodule.pyc", they will now look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and "mymodule.unladen10.pyc". And to prevent all of these new files from cluttering source directories, the pyc files are now collected in a "__pycache__" directory stored under the package directory.

Aside from the filenames and target directories, the new scheme has a few aspects that are visible to the programmer:

3147 - PYC Repository Directories
PEP written by Barry Warsaw.

PEP 3149: ABI Version Tagged .so Files

The PYC repository directory allows multiple bytecode cache files to be co-located. This PEP implements a similar mechanism for shared object files by giving them a common directory and distinct names for each version.

The common directory is "pyshared" and the file names are made distinct by identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the major and minor version numbers, and optional build flags (such as "d" for debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo", you may see these files when the distribution package is installed:

/usr/share/pyshared/foo.cpython-32m.so
/usr/share/pyshared/foo.cpython-33md.so

In Python itself, the tags are accessible from functions in the sysconfig module:

>>> import sysconfig
>>> sysconfig.get_config_var('SOABI')       # find the version tag
'cpython-32mu'
>>> sysconfig.get_config_var('EXT_SUFFIX')  # find the full filename extension
'.cpython-32mu.so'

3149 - ABI Version Tagged .so Files
PEP written by Barry Warsaw.

PEP 3333: Python Web Server Gateway Interface v1.0.1

This informational PEP clarifies how bytes/text issues are to be handled by the WSGI protocol. The challenge is that string handling in Python 3 is most conveniently handled with the str type even though the HTTP protocol is itself bytes oriented.

The PEP differentiates so-called native strings that are used for request/response headers and metadata versus byte strings which are used for the bodies of requests and responses.

The native strings are always of type str but are restricted to code points between U+0000 through U+00FF which are translatable to bytes using Latin-1 encoding. These strings are used for the keys and values in the environment dictionary and for response headers and statuses in the !start_response function. They must follow 2616 with respect to encoding. That is, they must either be ISO-8859-1 characters or use 2047 MIME encoding.

For developers porting WSGI applications from Python 2, here are the salient points:

For server implementers writing CGI-to-WSGI pathways or other CGI-style protocols, the users must to be able access the environment using native strings even though the underlying platform may have a different convention. To bridge this gap, the wsgiref module has a new function, wsgiref.handlers.read_environ for transcoding CGI variables from os.environ into native strings and returning a new dictionary.

3333 - Python Web Server Gateway Interface v1.0.1
PEP written by Phillip Eby.

Other Language Changes

Some smaller changes made to the core Python language are:

(Suggested by Raymond Hettinger and implemented by Eric Smith in 6081.)

(Required extensive work by Victor Stinner in 9425.)

New, Improved, and Deprecated Modules

Python's standard library has undergone significant maintenance efforts and quality improvements.

The biggest news for Python 3.2 is that the email package, mailbox module, and !nntplib modules now work correctly with the bytes/text model in Python 3. For the first time, there is correct handling of messages with mixed encodings.

Throughout the standard library, there has been more careful attention to encodings and text versus bytes issues. In particular, interactions with the operating system are now better able to exchange non-ASCII data using the Windows MBCS encoding, locale-aware encodings, or UTF-8.

Another significant win is the addition of substantially better support for SSL connections and security certificates.

In addition, more classes now implement a context manager to support convenient and reliable resource clean-up using a with statement.

email

The usability of the email package in Python 3 has been mostly fixed by the extensive efforts of R. David Murray. The problem was that emails are typically read and stored in the form of bytes rather than str text, and they may contain multiple encodings within a single email. So, the email package had to be extended to parse and generate email messages in bytes format.

(Proposed and implemented by R. David Murray, 4661 and 10321.)

elementtree

The xml.etree.ElementTree package and its !xml.etree.cElementTree counterpart have been updated to version 1.3.

Several new and useful functions and methods have been added:

Two methods have been deprecated:

For details of the update, see Introducing ElementTree on Fredrik Lundh's website.

(Contributed by Florent Xicluna and Fredrik Lundh, 6472.)

functools

itertools

collections

threading

The threading module has a new ~threading.Barrier synchronization class for making multiple threads wait until all of them have reached a common barrier point. Barriers are useful for making sure that a task with multiple preconditions does not run until all of the predecessor tasks are complete.

Barriers can work with an arbitrary number of threads. This is a generalization of a Rendezvous which is defined for only two threads.

Implemented as a two-phase cyclic barrier, ~threading.Barrier objects are suitable for use in loops. The separate filling and draining phases assure that all threads get released (drained) before any one of them can loop back and re-enter the barrier. The barrier fully resets after each cycle.

Example of using barriers:

from threading import Barrier, Thread

def get_votes(site):
    ballots = conduct_election(site)
    all_polls_closed.wait()        # do not count until all polls are closed
    totals = summarize(ballots)
    publish(site, totals)

all_polls_closed = Barrier(len(sites))
for site in sites:
    Thread(target=get_votes, args=(site,)).start()

In this example, the barrier enforces a rule that votes cannot be counted at any polling site until all polls are closed. Notice how a solution with a barrier is similar to one with threading.Thread.join, but the threads stay alive and continue to do work (summarizing ballots) after the barrier point is crossed.

If any of the predecessor tasks can hang or be delayed, a barrier can be created with an optional timeout parameter. Then if the timeout period elapses before all the predecessor tasks reach the barrier point, all waiting threads are released and a ~threading.BrokenBarrierError exception is raised:

def get_votes(site):
    ballots = conduct_election(site)
    try:
        all_polls_closed.wait(timeout=midnight - time.now())
    except BrokenBarrierError:
        lockbox = seal_ballots(ballots)
        queue.put(lockbox)
    else:
        totals = summarize(ballots)
        publish(site, totals)

In this example, the barrier enforces a more robust rule. If some election sites do not finish before midnight, the barrier times-out and the ballots are sealed and deposited in a queue for later handling.

See Barrier Synchronization Patterns for more examples of how barriers can be used in parallel computing. Also, there is a simple but thorough explanation of barriers in The Little Book of Semaphores, section 3.6.

(Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in 8777.)

datetime and time

(Contributed by Alexander Belopolsky and Victor Stinner in 1289118, 5094, 6641, 2706, 1777412, 8013, and 10827.)

math

The math module has been updated with six new functions inspired by the C99 standard.

The ~math.isfinite function provides a reliable and fast way to detect special values. It returns True for regular numbers and False for Nan or Infinity:

>>> from math import isfinite >>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))] [True, True, False, False]

The ~math.expm1 function computes e**x-1 for small values of x without incurring the loss of precision that usually accompanies the subtraction of nearly equal quantities:

>>> from math import expm1 >>> expm1(0.013671875) # more accurate way to compute e**x-1 for a small x 0.013765762467652909

The ~math.erf function computes a probability integral or Gaussian error function. The complementary error function, ~math.erfc, is 1 - erf(x):

>>> from math import erf, erfc, sqrt >>> erf(1.0/sqrt(2.0)) # portion of normal distribution within 1 standard deviation 0.682689492137086 >>> erfc(1.0/sqrt(2.0)) # portion of normal distribution outside 1 standard deviation 0.31731050786291404 >>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0)) 1.0

The ~math.gamma function is a continuous extension of the factorial function. See https://en.wikipedia.org/wiki/Gamma_function for details. Because the function is related to factorials, it grows large even for small values of x, so there is also a ~math.lgamma function for computing the natural logarithm of the gamma function:

>>> from math import gamma, lgamma >>> gamma(7.0) # six factorial 720.0 >>> lgamma(801.0) # log(800 factorial) 4551.950730698041

(Contributed by Mark Dickinson.)

abc

The abc module now supports ~abc.abstractclassmethod and ~abc.abstractstaticmethod.

These tools make it possible to define an abstract base class that requires a particular classmethod or staticmethod to be implemented:

class Temperature(metaclass=abc.ABCMeta):
    @abc.abstractclassmethod
    def from_fahrenheit(cls, t):
        ...
    @abc.abstractclassmethod
    def from_celsius(cls, t):
        ...

(Patch submitted by Daniel Urban; 5867.)

io

The io.BytesIO has a new method, ~io.BytesIO.getbuffer, which provides functionality similar to memoryview. It creates an editable view of the data without making a copy. The buffer's random access and support for slice notation are well-suited to in-place editing:

>>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11

>>> def change_location(buffer, record_number, location):
...     start = record_number * REC_LEN + LOC_START
...     buffer[start: start+LOC_LEN] = location

>>> import io

>>> byte_stream = io.BytesIO(
...     b'G3805  storeroom  Main chassis    '
...     b'X7899  shipping   Reserve cog     '
...     b'L6988  receiving  Primary sprocket'
... )
>>> buffer = byte_stream.getbuffer()
>>> change_location(buffer, 1, b'warehouse  ')
>>> change_location(buffer, 0, b'showroom   ')
>>> print(byte_stream.getvalue())
b'G3805  showroom   Main chassis    '
b'X7899  warehouse  Reserve cog     '
b'L6988  receiving  Primary sprocket'

(Contributed by Antoine Pitrou in 5506.)

reprlib

When writing a ~object.__repr__ method for a custom container, it is easy to forget to handle the case where a member refers back to the container itself. Python's builtin objects such as list and set handle self-reference by displaying "..." in the recursive part of the representation string.

To help write such ~object.__repr__ methods, the reprlib module has a new decorator, ~reprlib.recursive_repr, for detecting recursive calls to !__repr__ and substituting a placeholder string instead:

>>> class MyList(list):
...     @recursive_repr()
...     def __repr__(self):
...         return '<' + '|'.join(map(repr, self)) + '>'
...
>>> m = MyList('abc')
>>> m.append(m)
>>> m.append('x')
>>> print(m)
<'a'|'b'|'c'|...|'x'>

(Contributed by Raymond Hettinger in 9826 and 9840.)

logging

In addition to dictionary-based configuration described above, the logging package has many other improvements.

The logging documentation has been augmented by a basic tutorial <logging-basic-tutorial>, an advanced tutorial <logging-advanced-tutorial>, and a cookbook <logging-cookbook> of logging recipes. These documents are the fastest way to learn about logging.

The logging.basicConfig set-up function gained a style argument to support three different types of string formatting. It defaults to "%" for traditional %-formatting, can be set to "{" for the new str.format style, or can be set to "$" for the shell-style formatting provided by string.Template. The following three configurations are equivalent:

>>> from logging import basicConfig
>>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s")
>>> basicConfig(style='{', format="{name} -> {levelname} {message}")
>>> basicConfig(style='$', format="$name -> $levelname: $message")

If no configuration is set-up before a logging event occurs, there is now a default configuration using a ~logging.StreamHandler directed to sys.stderr for events of WARNING level or higher. Formerly, an event occurring before a configuration was set-up would either raise an exception or silently drop the event depending on the value of logging.raiseExceptions. The new default handler is stored in logging.lastResort.

The use of filters has been simplified. Instead of creating a ~logging.Filter object, the predicate can be any Python callable that returns True or False.

There were a number of other improvements that add flexibility and simplify configuration. See the module documentation for a full listing of changes in Python 3.2.

csv

The csv module now supports a new dialect, ~csv.unix_dialect, which applies quoting for all fields and a traditional Unix style with '\n' as the line terminator. The registered dialect name is unix.

The csv.DictWriter has a new method, ~csv.DictWriter.writeheader for writing-out an initial row to document the field names:

>>> import csv, sys
>>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix')
>>> w.writeheader()
"name","dept"
>>> w.writerows([
...     {'name': 'tom', 'dept': 'accounting'},
...     {'name': 'susan', 'dept': 'Salesl'}])
"tom","accounting"
"susan","sales"

(New dialect suggested by Jay Talbot in 5975, and the new method suggested by Ed Abraham in 1537721.)

contextlib

There is a new and slightly mind-blowing tool ~contextlib.ContextDecorator that is helpful for creating a context manager that does double duty as a function decorator.

As a convenience, this new functionality is used by ~contextlib.contextmanager so that no extra effort is needed to support both roles.

The basic idea is that both context managers and function decorators can be used for pre-action and post-action wrappers. Context managers wrap a group of statements using a with statement, and function decorators wrap a group of statements enclosed in a function. So, occasionally there is a need to write a pre-action or post-action wrapper that can be used in either role.

For example, it is sometimes useful to wrap functions or groups of statements with a logger that can track the time of entry and time of exit. Rather than writing both a function decorator and a context manager for the task, the ~contextlib.contextmanager provides both capabilities in a single definition:

from contextlib import contextmanager
import logging

logging.basicConfig(level=logging.INFO)

@contextmanager
def track_entry_and_exit(name):
    logging.info('Entering: %s', name)
    yield
    logging.info('Exiting: %s', name)

Formerly, this would have only been usable as a context manager:

with track_entry_and_exit('widget loader'):
    print('Some time consuming activity goes here')
    load_widget()

Now, it can be used as a decorator as well:

@track_entry_and_exit('widget loader')
def activity():
    print('Some time consuming activity goes here')
    load_widget()

Trying to fulfill two roles at once places some limitations on the technique. Context managers normally have the flexibility to return an argument usable by a with statement, but there is no parallel for function decorators.

In the above example, there is not a clean way for the track_entry_and_exit context manager to return a logging instance for use in the body of enclosed statements.

(Contributed by Michael Foord in 9110.)

decimal and fractions

Mark Dickinson crafted an elegant and efficient scheme for assuring that different numeric datatypes will have the same hash value whenever their actual values are equal (8188):

assert hash(Fraction(3, 2)) == hash(1.5) == \
       hash(Decimal("1.5")) == hash(complex(1.5, 0))

Some of the hashing details are exposed through a new attribute, sys.hash_info, which describes the bit width of the hash value, the prime modulus, the hash values for infinity and nan, and the multiplier used for the imaginary part of a number:

>>> sys.hash_info # doctest: +SKIP sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003)

An early decision to limit the interoperability of various numeric types has been relaxed. It is still unsupported (and ill-advised) to have implicit mixing in arithmetic expressions such as Decimal('1.1') + float('1.1') because the latter loses information in the process of constructing the binary float. However, since existing floating-point value can be converted losslessly to either a decimal or rational representation, it makes sense to add them to the constructor and to support mixed-type comparisons.

Similar changes were made to fractions.Fraction so that the ~fractions.Fraction.from_float and ~fractions.Fraction.from_decimal methods are no longer needed (8294):

>>> from decimal import Decimal >>> from fractions import Fraction >>> Decimal(1.1) Decimal('1.100000000000000088817841970012523233890533447265625') >>> Fraction(1.1) Fraction(2476979795053773, 2251799813685248)

Another useful change for the decimal module is that the Context.clamp <decimal.Context.clamp> attribute is now public. This is useful in creating contexts that correspond to the decimal interchange formats specified in IEEE 754 (see 8540).

(Contributed by Mark Dickinson and Raymond Hettinger.)

ftp

The ftplib.FTP class now supports the context management protocol to unconditionally consume socket.error exceptions and to close the FTP connection when done:

>>> from ftplib import FTP
>>> with FTP("ftp1.at.proftpd.org") as ftp:
...     ftp.login()
...     ftp.dir()
...
'230 Anonymous login ok, restrictions apply.'
dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 .
dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 ..
dr-xr-xr-x   5 ftp      ftp          4096 May  6 10:43 CentOS
dr-xr-xr-x   3 ftp      ftp            18 Jul 10  2008 Fedora

Other file-like objects such as mmap.mmap and fileinput.input also grew auto-closing context managers:

with fileinput.input(files=('log1.txt', 'log2.txt')) as f:
    for line in f:
        process(line)

(Contributed by Tarek Ziadé and Giampaolo Rodolà in 4972, and by Georg Brandl in 8046 and 1286.)

The ~ftplib.FTP_TLS class now accepts a context parameter, which is a ssl.SSLContext object allowing bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure.

(Contributed by Giampaolo Rodolà; 8806.)

popen

The os.popen and subprocess.Popen functions now support with statements for auto-closing of the file descriptors.

(Contributed by Antoine Pitrou and Brian Curtin in 7461 and 10554.)

select

The select module now exposes a new, constant attribute, ~select.PIPE_BUF, which gives the minimum number of bytes which are guaranteed not to block when select.select says a pipe is ready for writing.

>>> import select >>> select.PIPE_BUF # doctest: +SKIP 512

(Available on Unix systems. Patch by Sébastien Sablé in 9862)

gzip and zipfile

gzip.GzipFile now implements the io.BufferedIOBase abstract base class (except for truncate()). It also has a ~gzip.GzipFile.peek method and supports unseekable as well as zero-padded file objects.

The gzip module also gains the ~gzip.compress and ~gzip.decompress functions for easier in-memory compression and decompression. Keep in mind that text needs to be encoded as bytes before compressing and decompressing:

>>> import gzip >>> s = 'Three shall be the number thou shalt count, ' >>> s += 'and the number of the counting shall be three' >>> b = s.encode() # convert to utf-8 >>> len(b) 89 >>> c = gzip.compress(b) >>> len(c) 77 >>> gzip.decompress(c).decode()[:42] # decompress and convert to text 'Three shall be the number thou shalt count'

(Contributed by Anand B. Pillai in 3488; and by Antoine Pitrou, Nir Aides and Brian Curtin in 9962, 1675951, 7471 and 2846.)

Also, the zipfile.ZipExtFile <zipfile.ZipFile.open> class was reworked internally to represent files stored inside an archive. The new implementation is significantly faster and can be wrapped in an io.BufferedReader object for more speedups. It also solves an issue where interleaved calls to read and readline gave the wrong results.

(Patch submitted by Nir Aides in 7610.)

tarfile

The ~tarfile.TarFile class can now be used as a context manager. In addition, its ~tarfile.TarFile.add method has a new option, filter, that controls which files are added to the archive and allows the file metadata to be edited.

The new filter option replaces the older, less flexible exclude parameter which is now deprecated. If specified, the optional filter parameter needs to be a keyword argument. The user-supplied filter function accepts a ~tarfile.TarInfo object and returns an updated ~tarfile.TarInfo object, or if it wants the file to be excluded, the function can return None:

>>> import tarfile, glob

>>> def myfilter(tarinfo):
...     if tarinfo.isfile():             # only save real files
...         tarinfo.uname = 'monty'      # redact the user name
...         return tarinfo

>>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:
...     for filename in glob.glob('*.txt'):
...         tf.add(filename, filter=myfilter)
...     tf.list()
-rw-r--r-- monty/501        902 2011-01-26 17:59:11 annotations.txt
-rw-r--r-- monty/501        123 2011-01-26 17:59:11 general_questions.txt
-rw-r--r-- monty/501       3514 2011-01-26 17:59:11 prion.txt
-rw-r--r-- monty/501        124 2011-01-26 17:59:11 py_todo.txt
-rw-r--r-- monty/501       1399 2011-01-26 17:59:11 semaphore_notes.txt

(Proposed by Tarek Ziadé and implemented by Lars Gustäbel in 6856.)

hashlib

The hashlib module has two new constant attributes listing the hashing algorithms guaranteed to be present in all implementations and those available on the current implementation:

>>> import hashlib

>>> hashlib.algorithms_guaranteed
{'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'}

>>> hashlib.algorithms_available
{'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256',
'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2',
'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224',
'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'}

(Suggested by Carl Chenet in 7418.)

ast

The ast module has a wonderful a general-purpose tool for safely evaluating expression strings using the Python literal syntax. The ast.literal_eval function serves as a secure alternative to the builtin eval function which is easily abused. Python 3.2 adds bytes and set literals to the list of supported types: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None.

>>> from ast import literal_eval

>>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}"
>>> literal_eval(request)
{'args': (2, 0.5), 'req': 3, 'func': 'pow'}

>>> request = "os.system('do something harmful')"
>>> literal_eval(request)
Traceback (most recent call last):
  ...
ValueError: malformed node or string: <_ast.Call object at 0x101739a10>

(Implemented by Benjamin Peterson and Georg Brandl.)

os

Different operating systems use various encodings for filenames and environment variables. The os module provides two new functions, ~os.fsencode and ~os.fsdecode, for encoding and decoding filenames:

>>> import os >>> filename = 'Sehenswürdigkeiten' >>> os.fsencode(filename) b'Sehenswxc3xbcrdigkeiten'

Some operating systems allow direct access to encoded bytes in the environment. If so, the os.supports_bytes_environ constant will be true.

For direct access to encoded environment variables (if available), use the new os.getenvb function or use os.environb which is a bytes version of os.environ.

(Contributed by Victor Stinner.)

shutil

The shutil.copytree function has two new options:

(Contributed by Tarek Ziadé.)

In addition, the shutil module now supports archiving operations <archiving-operations> for zipfiles, uncompressed tarfiles, gzipped tarfiles, and bzipped tarfiles. And there are functions for registering additional archiving file formats (such as xz compressed tarfiles or custom formats).

The principal functions are ~shutil.make_archive and ~shutil.unpack_archive. By default, both operate on the current directory (which can be set by os.chdir) and on any sub-directories. The archive filename needs to be specified with a full pathname. The archiving step is non-destructive (the original files are left unchanged).

>>> import shutil, pprint

>>> os.chdir('mydata')  # change to the source directory
>>> f = shutil.make_archive('/var/backup/mydata',
...                         'zip')      # archive the current directory
>>> f                                   # show the name of archive
'/var/backup/mydata.zip'
>>> os.chdir('tmp')                     # change to an unpacking
>>> shutil.unpack_archive('/var/backup/mydata.zip')  # recover the data

>>> pprint.pprint(shutil.get_archive_formats())  # display known formats
[('bztar', "bzip2'ed tar-file"),
 ('gztar', "gzip'ed tar-file"),
 ('tar', 'uncompressed tar file'),
 ('zip', 'ZIP file')]

>>> shutil.register_archive_format(     # register a new archive format
...     name='xz',
...     function=xz.compress,           # callable archiving function
...     extra_args=[('level', 8)],      # arguments to the function
...     description='xz compression'
... )

(Contributed by Tarek Ziadé.)

sqlite3

The sqlite3 module was updated to pysqlite version 2.6.0. It has two new capabilities.

(Contributed by R. David Murray and Shashwat Anand; 8845.)

html

A new html module was introduced with only a single function, ~html.escape, which is used for escaping reserved characters from HTML markup:

>>> import html >>> html.escape('x > 2 && x < 7') 'x &gt; 2 &amp;&amp; x &lt; 7'

socket

The socket module has two new improvements.

ssl

The ssl module added a number of features to satisfy common requirements for secure (encrypted, authenticated) internet connections:

(Contributed by Antoine Pitrou in 8850, 1589, 8322, 5639, 4870, 8484, and 8321.)

nntp

The !nntplib module has a revamped implementation with better bytes and text semantics as well as more practical APIs. These improvements break compatibility with the nntplib version in Python 3.1, which was partly dysfunctional in itself.

Support for secure connections through both implicit (using !nntplib.NNTP_SSL) and explicit (using !nntplib.NNTP.starttls) TLS has also been added.

(Contributed by Antoine Pitrou in 9360 and Andrew Vant in 1926.)

certificates

http.client.HTTPSConnection, urllib.request.HTTPSHandler and urllib.request.urlopen now take optional arguments to allow for server certificate checking against a set of Certificate Authorities, as recommended in public uses of HTTPS.

(Added by Antoine Pitrou, 9003.)

imaplib

Support for explicit TLS on standard IMAP4 connections has been added through the new imaplib.IMAP4.starttls method.

(Contributed by Lorenzo M. Catucci and Antoine Pitrou, 4471.)

http.client

There were a number of small API improvements in the http.client module. The old-style HTTP 0.9 simple responses are no longer supported and the strict parameter is deprecated in all classes.

The ~http.client.HTTPConnection and ~http.client.HTTPSConnection classes now have a source_address parameter for a (host, port) tuple indicating where the HTTP connection is made from.

Support for certificate checking and HTTPS virtual hosts were added to ~http.client.HTTPSConnection.

The ~http.client.HTTPConnection.request method on connection objects allowed an optional body argument so that a file object could be used to supply the content of the request. Conveniently, the body argument now also accepts an iterable object so long as it includes an explicit Content-Length header. This extended interface is much more flexible than before.

To establish an HTTPS connection through a proxy server, there is a new ~http.client.HTTPConnection.set_tunnel method that sets the host and port for HTTP Connect tunneling.

To match the behavior of http.server, the HTTP client library now also encodes headers with ISO-8859-1 (Latin-1) encoding. It was already doing that for incoming headers, so now the behavior is consistent for both incoming and outgoing traffic. (See work by Armin Ronacher in 10980.)

unittest

The unittest module has a number of improvements supporting test discovery for packages, easier experimentation at the interactive prompt, new testcase methods, improved diagnostic messages for test failures, and better method names.

random

The integer methods in the random module now do a better job of producing uniform distributions. Previously, they computed selections with int(n*random()) which had a slight bias whenever n was not a power of two. Now, multiple selections are made from a range up to the next power of two and a selection is kept only when it falls within the range 0 <= x < n. The functions and methods affected are ~random.randrange, ~random.randint, ~random.choice, ~random.shuffle and ~random.sample.

(Contributed by Raymond Hettinger; 9025.)

poplib

~poplib.POP3_SSL class now accepts a context parameter, which is a ssl.SSLContext object allowing bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure.

(Contributed by Giampaolo Rodolà; 8807.)

asyncore

!asyncore.dispatcher now provides a !handle_accepted method returning a (sock, addr) pair which is called when a connection has actually been established with a new remote endpoint. This is supposed to be used as a replacement for old !handle_accept and avoids the user to call !accept directly.

(Contributed by Giampaolo Rodolà; 6706.)

tempfile

The tempfile module has a new context manager, ~tempfile.TemporaryDirectory which provides easy deterministic cleanup of temporary directories:

with tempfile.TemporaryDirectory() as tmpdirname:
    print('created temporary dir:', tmpdirname)

(Contributed by Neil Schemenauer and Nick Coghlan; 5178.)

inspect

(Contributed by Michael Foord.)

pydoc

The pydoc module now provides a much-improved web server interface, as well as a new command-line option -b to automatically open a browser window to display that server:

$ pydoc3.2 -b

(Contributed by Ron Adam; 2001.)

dis

The dis module gained two new functions for inspecting code, ~dis.code_info and ~dis.show_code. Both provide detailed code object information for the supplied function, method, source code string or code object. The former returns a string and the latter prints it:

>>> import dis, random
>>> dis.show_code(random.choice)
Name:              choice
Filename:          /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py
Argument count:    2
Kw-only arguments: 0
Number of locals:  3
Stack size:        11
Flags:             OPTIMIZED, NEWLOCALS, NOFREE
Constants:
   0: 'Choose a random element from a non-empty sequence.'
   1: 'Cannot choose from an empty sequence'
Names:
   0: _randbelow
   1: len
   2: ValueError
   3: IndexError
Variable names:
   0: self
   1: seq
   2: i

In addition, the ~dis.dis function now accepts string arguments so that the common idiom dis(compile(s, '', 'eval')) can be shortened to dis(s):

>>> dis('3*x+1 if x%2==1 else x//2')
  1           0 LOAD_NAME                0 (x)
              3 LOAD_CONST               0 (2)
              6 BINARY_MODULO
              7 LOAD_CONST               1 (1)
             10 COMPARE_OP               2 (==)
             13 POP_JUMP_IF_FALSE       28
             16 LOAD_CONST               2 (3)
             19 LOAD_NAME                0 (x)
             22 BINARY_MULTIPLY
             23 LOAD_CONST               1 (1)
             26 BINARY_ADD
             27 RETURN_VALUE
        >>   28 LOAD_NAME                0 (x)
             31 LOAD_CONST               0 (2)
             34 BINARY_FLOOR_DIVIDE
             35 RETURN_VALUE

Taken together, these improvements make it easier to explore how CPython is implemented and to see for yourself what the language syntax does under-the-hood.

(Contributed by Nick Coghlan in 9147.)

dbm

All database modules now support the !get and !setdefault methods.

(Suggested by Ray Allen in 9523.)

ctypes

A new type, ctypes.c_ssize_t represents the C ssize_t datatype.

site

The site module has three new functions useful for reporting on the details of a given Python installation.

>>> import site
>>> site.getsitepackages()
['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages',
 '/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python',
 '/Library/Python/3.2/site-packages']
>>> site.getuserbase()
'/Users/raymondhettinger/Library/Python/3.2'
>>> site.getusersitepackages()
'/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages'

Conveniently, some of site's functionality is accessible directly from the command-line:

$ python -m site --user-base
/Users/raymondhettinger/.local
$ python -m site --user-site
/Users/raymondhettinger/.local/lib/python3.2/site-packages

(Contributed by Tarek Ziadé in 6693.)

sysconfig

The new sysconfig module makes it straightforward to discover installation paths and configuration variables that vary across platforms and installations.

The module offers access simple access functions for platform and version information:

It also provides access to the paths and variables corresponding to one of seven named schemes used by distutils. Those include posix_prefix, posix_home, posix_user, nt, nt_user, os2, os2_home:

There is also a convenient command-line interface:

C:\Python32>python -m sysconfig
Platform: "win32"
Python version: "3.2"
Current installation scheme: "nt"

Paths:
        data = "C:\Python32"
        include = "C:\Python32\Include"
        platinclude = "C:\Python32\Include"
        platlib = "C:\Python32\Lib\site-packages"
        platstdlib = "C:\Python32\Lib"
        purelib = "C:\Python32\Lib\site-packages"
        scripts = "C:\Python32\Scripts"
        stdlib = "C:\Python32\Lib"

Variables:
        BINDIR = "C:\Python32"
        BINLIBDEST = "C:\Python32\Lib"
        EXE = ".exe"
        INCLUDEPY = "C:\Python32\Include"
        LIBDEST = "C:\Python32\Lib"
        SO = ".pyd"
        VERSION = "32"
        abiflags = ""
        base = "C:\Python32"
        exec_prefix = "C:\Python32"
        platbase = "C:\Python32"
        prefix = "C:\Python32"
        projectbase = "C:\Python32"
        py_version = "3.2"
        py_version_nodot = "32"
        py_version_short = "3.2"
        srcdir = "C:\Python32"
        userbase = "C:\Documents and Settings\Raymond\Application Data\Python"

(Moved out of Distutils by Tarek Ziadé.)

pdb

The pdb debugger module gained a number of usability improvements:

(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.)

configparser

The configparser module was modified to improve usability and predictability of the default parser and its supported INI syntax. The old !ConfigParser class was removed in favor of !SafeConfigParser which has in turn been renamed to ~configparser.ConfigParser. Support for inline comments is now turned off by default and section or option duplicates are not allowed in a single configuration source.

Config parsers gained a new API based on the mapping protocol:

>>> parser = ConfigParser()
>>> parser.read_string("""
... [DEFAULT]
... location = upper left
... visible = yes
... editable = no
... color = blue
...
... [main]
... title = Main Menu
... color = green
...
... [options]
... title = Options
... """)
>>> parser['main']['color']
'green'
>>> parser['main']['editable']
'no'
>>> section = parser['options']
>>> section['title']
'Options'
>>> section['title'] = 'Options (editable: %(editable)s)'
>>> section['title']
'Options (editable: no)'

The new API is implemented on top of the classical API, so custom parser subclasses should be able to use it without modifications.

The INI file structure accepted by config parsers can now be customized. Users can specify alternative option/value delimiters and comment prefixes, change the name of the DEFAULT section or switch the interpolation syntax.

There is support for pluggable interpolation including an additional interpolation handler ~configparser.ExtendedInterpolation:

>>> parser = ConfigParser(interpolation=ExtendedInterpolation())
>>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'},
...                   'custom': {'prefix': '/usr/local'}})
>>> parser.read_string("""
... [buildout]
... parts =
...   zope9
...   instance
... find-links =
...   ${buildout:directory}/downloads/dist
...
... [zope9]
... recipe = plone.recipe.zope9install
... location = /opt/zope
...
... [instance]
... recipe = plone.recipe.zope9instance
... zope9-location = ${zope9:location}
... zope-conf = ${custom:prefix}/etc/zope.conf
... """)
>>> parser['buildout']['find-links']
'\n/home/ambv/zope9/downloads/dist'
>>> parser['instance']['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance = parser['instance']
>>> instance['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance['zope9-location']
'/opt/zope'

A number of smaller features were also introduced, like support for specifying encoding in read operations, specifying fallback values for get-functions, or reading directly from dictionaries and strings.

(All changes contributed by Łukasz Langa.)

urllib.parse

A number of usability improvements were made for the urllib.parse module.

The ~urllib.parse.urlparse function now supports IPv6 addresses as described in 2732:

>>> import urllib.parse >>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') # doctest: +NORMALIZE_WHITESPACE ParseResult(scheme='http', netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]', path='/foo/', params='', query='', fragment='')

The ~urllib.parse.urldefrag function now returns a named tuple:

>>> r = urllib.parse.urldefrag('http://python.org/about/#target')
>>> r
DefragResult(url='http://python.org/about/', fragment='target')
>>> r[0]
'http://python.org/about/'
>>> r.fragment
'target'

And, the ~urllib.parse.urlencode function is now much more flexible, accepting either a string or bytes type for the query argument. If it is a string, then the safe, encoding, and error parameters are sent to ~urllib.parse.quote_plus for encoding:

>>> urllib.parse.urlencode([
...      ('type', 'telenovela'),
...      ('name', '¿Dónde Está Elisa?')],
...      encoding='latin-1')
'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F'

As detailed in parsing-ascii-encoded-bytes, all the urllib.parse functions now accept ASCII-encoded byte strings as input, so long as they are not mixed with regular strings. If ASCII-encoded byte strings are given as parameters, the return types will also be an ASCII-encoded byte strings:

>>> urllib.parse.urlparse(b'http://www.python.org:80/about/') # doctest: +NORMALIZE_WHITESPACE ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80', path=b'/about/', params=b'', query=b'', fragment=b'')

(Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in 2987, 5468, and 9873.)

mailbox

Thanks to a concerted effort by R. David Murray, the mailbox module has been fixed for Python 3.2. The challenge was that mailbox had been originally designed with a text interface, but email messages are best represented with bytes because various parts of a message may have different encodings.

The solution harnessed the email package's binary support for parsing arbitrary email messages. In addition, the solution required a number of API changes.

As expected, the ~mailbox.Mailbox.add method for mailbox.Mailbox objects now accepts binary input.

~io.StringIO and text file input are deprecated. Also, string input will fail early if non-ASCII characters are used. Previously it would fail when the email was processed in a later step.

There is also support for binary output. The ~mailbox.Mailbox.get_file method now returns a file in the binary mode (where it used to incorrectly set the file to text-mode). There is also a new ~mailbox.Mailbox.get_bytes method that returns a bytes representation of a message corresponding to a given key.

It is still possible to get non-binary output using the old API's ~mailbox.Mailbox.get_string method, but that approach is not very useful. Instead, it is best to extract messages from a ~mailbox.Message object or to load them from binary input.

(Contributed by R. David Murray, with efforts from Steffen Daode Nurpmeso and an initial patch by Victor Stinner in 9124.)

turtledemo

The demonstration code for the turtle module was moved from the Demo directory to main library. It includes over a dozen sample scripts with lively displays. Being on sys.path, it can now be run directly from the command-line:

$ python -m turtledemo

(Moved from the Demo directory by Alexander Belopolsky in 10199.)

Multi-threading

Optimizations

A number of small performance enhancements have been added:

There were several other minor optimizations. Set differencing now runs faster when one operand is much larger than the other (patch by Andress Bennetts in 8685). The !array.repeat method has a faster implementation (1569291 by Alexander Belopolsky). The ~http.server.BaseHTTPRequestHandler has more efficient buffering (3709 by Andrew Schaaf). The operator.attrgetter function has been sped-up (10160 by Christos Georgiou). And ~configparser.ConfigParser loads multi-line arguments a bit faster (7113 by Łukasz Langa).

Unicode

Python has been updated to Unicode 6.0.0. The update to the standard adds over 2,000 new characters including emoji symbols which are important for mobile phones.

In addition, the updated standard has altered the character properties for two Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character (U+19DA), making the former eligible for use in identifiers while disqualifying the latter. For more information, see Unicode Character Database Changes.

Codecs

Support was added for cp720 Arabic DOS encoding (1616979).

MBCS encoding no longer ignores the error handler argument. In the default strict mode, it raises an UnicodeDecodeError when it encounters an undecodable byte sequence and an UnicodeEncodeError for an unencodable character.

The MBCS codec supports 'strict' and 'ignore' error handlers for decoding, and 'strict' and 'replace' for encoding.

To emulate Python3.1 MBCS encoding, select the 'ignore' handler for decoding and the 'replace' handler for encoding.

On Mac OS X, Python decodes command line arguments with 'utf-8' rather than the locale encoding.

By default, tarfile uses 'utf-8' encoding on Windows (instead of 'mbcs') and the 'surrogateescape' error handler on all operating systems.

Documentation

The documentation continues to be improved.

IDLE

Code Repository

In addition to the existing Subversion code repository at https://svn.python.org there is now a Mercurial repository at [https://hg.python.org/\](https://hg.python.org/) .

After the 3.2 release, there are plans to switch to Mercurial as the primary repository. This distributed version control system should make it easier for members of the community to create and share external changesets. See 385 for details.

To learn to use the new version control system, see the Quick Start or the Guide to Mercurial Workflows.

Build and C API Changes

Changes to Python's build process and to the C API include:

There were a number of other small changes to the C-API. See the Misc/NEWS file for a complete list.

Also, there were a number of updates to the Mac OS X build, see Mac/BuildScript/README.txt for details. For users running a 32/64-bit build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6. Accordingly, we recommend installing an updated alternative such as ActiveState Tcl/Tk 8.5.9. See https://www.python.org/download/mac/tcltk/ for additional details.

Porting to Python 3.2

This section lists previously described changes and other bugfixes that may require changes to your code:

What's New In Python 3.3

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the Mercurial log when researching a change.

This article explains the new features in Python 3.3, compared to 3.2. Python 3.3 was released on September 29, 2012. For full details, see the changelog.

398 - Python 3.3 Release Schedule

Summary -- Release highlights

New syntax features:

New library modules:

New built-in features:

Implementation improvements:

Significantly Improved Library Modules:

Security improvements:

Please read on for a comprehensive list of user-facing changes.

PEP 405: Virtual Environments

Virtual environments help create separate Python setups while sharing a system-wide base install, for ease of maintenance. Virtual environments have their own set of private site packages (i.e. locally installed libraries), and are optionally segregated from the system-wide site packages. Their concept and implementation are inspired by the popular virtualenv third-party package, but benefit from tighter integration with the interpreter core.

This PEP adds the venv module for programmatic access, and the pyvenv script for command-line access and administration. The Python interpreter checks for a pyvenv.cfg, file whose existence signals the base of a virtual environment's directory tree.

405 - Python Virtual Environments
PEP written by Carl Meyer; implementation by Carl Meyer and Vinay Sajip

PEP 420: Implicit Namespace Packages

Native support for package directories that don't require __init__.py marker files and can automatically span multiple path segments (inspired by various third party approaches to namespace packages, as described in 420)

420 - Implicit Namespace Packages
PEP written by Eric V. Smith; implementation by Eric V. Smith and Barry Warsaw

PEP 3118: New memoryview implementation and buffer protocol documentation

The implementation of 3118 has been significantly improved.

The new memoryview implementation comprehensively fixes all ownership and lifetime issues of dynamically allocated fields in the Py_buffer struct that led to multiple crash reports. Additionally, several functions that crashed or returned incorrect results for non-contiguous or multi-dimensional input have been fixed.

The memoryview object now has a PEP-3118 compliant getbufferproc() that checks the consumer's request type. Many new features have been added, most of them work in full generality for non-contiguous arrays and arrays with suboffsets.

The documentation has been updated, clearly spelling out responsibilities for both exporters and consumers. Buffer request flags are grouped into basic and compound flags. The memory layout of non-contiguous and multi-dimensional NumPy-style arrays is explained.

Features

API changes

(Contributed by Stefan Krah in 10181.)

3118 - Revising the Buffer Protocol

PEP 393: Flexible String Representation

The Unicode string type is changed to support multiple internal representations, depending on the character with the largest Unicode ordinal (1, 2, or 4 bytes) in the represented string. This allows a space-efficient representation in common cases, but gives access to full UCS-4 on all systems. For compatibility with existing APIs, several representations may exist in parallel; over time, this compatibility should be phased out.

On the Python side, there should be no downside to this change.

On the C API side, 393 is fully backward compatible. The legacy API should remain available at least five years. Applications using the legacy API will not fully benefit of the memory reduction, or - worse - may use a bit more memory, because Python may have to maintain two versions of each string (in the legacy format and in the new efficient storage).

Functionality

Changes introduced by 393 are the following:

Performance and resource usage

The storage of Unicode strings now depends on the highest code point in the string:

The net effect is that for most applications, memory usage of string storage should decrease significantly - especially compared to former wide unicode builds - as, in many cases, strings will be pure ASCII even in international contexts (because many strings store non-human language data, such as XML fragments, HTTP headers, JSON-encoded data, etc.). We also hope that it will, for the same reasons, increase CPU cache efficiency on non-trivial applications. The memory usage of Python 3.3 is two to three times smaller than Python 3.2, and a little bit better than Python 2.7, on a Django benchmark (see the PEP for details).

393 - Flexible String Representation
PEP written by Martin von Löwis; implementation by Torsten Becker and Martin von Löwis.

PEP 397: Python Launcher for Windows

The Python 3.3 Windows installer now includes a py launcher application that can be used to launch Python applications in a version independent fashion.

This launcher is invoked implicitly when double-clicking *.py files. If only a single Python version is installed on the system, that version will be used to run the file. If multiple versions are installed, the most recent version is used by default, but this can be overridden by including a Unix-style "shebang line" in the Python script.

The launcher can also be used explicitly from the command line as the py application. Running py follows the same version selection rules as implicitly launching scripts, but a more specific version can be selected by passing appropriate arguments (such as -3 to request Python 3 when Python 2 is also installed, or -2.6 to specifically request an earlier Python version when a more recent version is installed).

In addition to the launcher, the Windows installer now includes an option to add the newly installed Python to the system PATH. (Contributed by Brian Curtin in 3561.)

397 - Python Launcher for Windows
PEP written by Mark Hammond and Martin v. Löwis; implementation by Vinay Sajip.

Launcher documentation: launcher

Installer PATH modification: windows-path-mod

PEP 3151: Reworking the OS and IO exception hierarchy

The hierarchy of exceptions raised by operating system errors is now both simplified and finer-grained.

You don't have to worry anymore about choosing the appropriate exception type between OSError, IOError, EnvironmentError, WindowsError, !mmap.error, socket.error or select.error. All these exception types are now only one: OSError. The other names are kept as aliases for compatibility reasons.

Also, it is now easier to catch a specific error condition. Instead of inspecting the errno attribute (or args[0]) for a particular constant from the errno module, you can catch the adequate OSError subclass. The available subclasses are the following:

And the ConnectionError itself has finer-grained subclasses:

Thanks to the new exceptions, common usages of the errno can now be avoided. For example, the following code written for Python 3.2:

from errno import ENOENT, EACCES, EPERM

try:
    with open("document.txt") as f:
        content = f.read()
except IOError as err:
    if err.errno == ENOENT:
        print("document.txt file is missing")
    elif err.errno in (EACCES, EPERM):
        print("You are not allowed to read document.txt")
    else:
        raise

can now be written without the errno import and without manual inspection of exception attributes:

try:
    with open("document.txt") as f:
        content = f.read()
except FileNotFoundError:
    print("document.txt file is missing")
except PermissionError:
    print("You are not allowed to read document.txt")

3151 - Reworking the OS and IO Exception Hierarchy
PEP written and implemented by Antoine Pitrou

single: yield; yield from (in What's New)

PEP 380: Syntax for Delegating to a Subgenerator

PEP 380 adds the yield from expression, allowing a generator to delegate part of its operations to another generator. This allows a section of code containing yield to be factored out and placed in another generator. Additionally, the subgenerator is allowed to return with a value, and the value is made available to the delegating generator.

While designed primarily for use in delegating to a subgenerator, the yield from expression actually allows delegation to arbitrary subiterators.

For simple iterators, yield from iterable is essentially just a shortened form of for item in iterable: yield item:

>>> def g(x):
...     yield from range(x, 0, -1)
...     yield from range(x)
...
>>> list(g(5))
[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]

However, unlike an ordinary loop, yield from allows subgenerators to receive sent and thrown values directly from the calling scope, and return a final value to the outer generator:

>>> def accumulate():
...     tally = 0
...     while 1:
...         next = yield
...         if next is None:
...             return tally
...         tally += next
...
>>> def gather_tallies(tallies):
...     while 1:
...         tally = yield from accumulate()
...         tallies.append(tally)
...
>>> tallies = []
>>> acc = gather_tallies(tallies)
>>> next(acc)  # Ensure the accumulator is ready to accept values
>>> for i in range(4):
...     acc.send(i)
...
>>> acc.send(None)  # Finish the first tally
>>> for i in range(5):
...     acc.send(i)
...
>>> acc.send(None)  # Finish the second tally
>>> tallies
[6, 10]

The main principle driving this change is to allow even generators that are designed to be used with the send and throw methods to be split into multiple subgenerators as easily as a single large function can be split into multiple subfunctions.

380 - Syntax for Delegating to a Subgenerator
PEP written by Greg Ewing; implementation by Greg Ewing, integrated into 3.3 by Renaud Blanch, Ryan Kelly and Nick Coghlan; documentation by Zbigniew Jędrzejewski-Szmek and Nick Coghlan

PEP 409: Suppressing exception context

PEP 409 introduces new syntax that allows the display of the chained exception context to be disabled. This allows cleaner error messages in applications that convert between exception types:

>>> class D:
...     def __init__(self, extra):
...         self._extra_attributes = extra
...     def __getattr__(self, attr):
...         try:
...             return self._extra_attributes[attr]
...         except KeyError:
...             raise AttributeError(attr) from None
...
>>> D({}).x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 8, in __getattr__
AttributeError: x

Without the from None suffix to suppress the cause, the original exception would be displayed by default:

>>> class C:
...     def __init__(self, extra):
...         self._extra_attributes = extra
...     def __getattr__(self, attr):
...         try:
...             return self._extra_attributes[attr]
...         except KeyError:
...             raise AttributeError(attr)
...
>>> C({}).x
Traceback (most recent call last):
  File "<stdin>", line 6, in __getattr__
KeyError: 'x'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 8, in __getattr__
AttributeError: x

No debugging capability is lost, as the original exception context remains available if needed (for example, if an intervening library has incorrectly suppressed valuable underlying details):

>>> try:
...     D({}).x
... except AttributeError as exc:
...     print(repr(exc.__context__))
...
KeyError('x',)

409 - Suppressing exception context
PEP written by Ethan Furman; implemented by Ethan Furman and Nick Coghlan.

PEP 414: Explicit Unicode literals

To ease the transition from Python 2 for Unicode aware Python applications that make heavy use of Unicode literals, Python 3.3 once again supports the "u" prefix for string literals. This prefix has no semantic significance in Python 3, it is provided solely to reduce the number of purely mechanical changes in migrating to Python 3, making it easier for developers to focus on the more significant semantic changes (such as the stricter default separation of binary and text data).

414 - Explicit Unicode literals
PEP written by Armin Ronacher.

PEP 3155: Qualified name for classes and functions

Functions and class objects have a new ~definition.__qualname__ attribute representing the "path" from the module top-level to their definition. For global functions and classes, this is the same as ~definition.__name__. For other functions and classes, it provides better information about where they were actually defined, and how they might be accessible from the global scope.

Example with (non-bound) methods:

>>> class C:
...     def meth(self):
...         pass
...
>>> C.meth.__name__
'meth'
>>> C.meth.__qualname__
'C.meth'

Example with nested classes:

>>> class C:
...     class D:
...         def meth(self):
...             pass
...
>>> C.D.__name__
'D'
>>> C.D.__qualname__
'C.D'
>>> C.D.meth.__name__
'meth'
>>> C.D.meth.__qualname__
'C.D.meth'

Example with nested functions:

>>> def outer():
...     def inner():
...         pass
...     return inner
...
>>> outer().__name__
'inner'
>>> outer().__qualname__
'outer.<locals>.inner'

The string representation of those objects is also changed to include the new, more precise information:

>>> str(C.D)
"<class '__main__.C.D'>"
>>> str(C.D.meth)
'<function C.D.meth at 0x7f46b9fe31e0>'

3155 - Qualified name for classes and functions
PEP written and implemented by Antoine Pitrou.

PEP 412: Key-Sharing Dictionary

Dictionaries used for the storage of objects' attributes are now able to share part of their internal storage between each other (namely, the part which stores the keys and their respective hashes). This reduces the memory consumption of programs creating many instances of non-builtin types.

412 - Key-Sharing Dictionary
PEP written and implemented by Mark Shannon.

PEP 362: Function Signature Object

A new function inspect.signature makes introspection of python callables easy and straightforward. A broad range of callables is supported: python functions, decorated or not, classes, and functools.partial objects. New classes inspect.Signature, inspect.Parameter and inspect.BoundArguments hold information about the call signatures, such as, annotations, default values, parameters kinds, and bound arguments, which considerably simplifies writing decorators and any code that validates or amends calling signatures or arguments.

362: - Function Signature Object
PEP written by Brett Cannon, Yury Selivanov, Larry Hastings, Jiwon Seo; implemented by Yury Selivanov.

PEP 421: Adding sys.implementation

A new attribute on the sys module exposes details specific to the implementation of the currently running interpreter. The initial set of attributes on sys.implementation are name, version, hexversion, and cache_tag.

The intention of sys.implementation is to consolidate into one namespace the implementation-specific data used by the standard library. This allows different Python implementations to share a single standard library code base much more easily. In its initial state, sys.implementation holds only a small portion of the implementation-specific data. Over time that ratio will shift in order to make the standard library more portable.

One example of improved standard library portability is cache_tag. As of Python 3.3, sys.implementation.cache_tag is used by importlib to support 3147 compliance. Any Python implementation that uses importlib for its built-in import system may use cache_tag to control the caching behavior for modules.

SimpleNamespace

The implementation of sys.implementation also introduces a new type to Python: types.SimpleNamespace. In contrast to a mapping-based namespace, like dict, SimpleNamespace is attribute-based, like object. However, unlike object, SimpleNamespace instances are writable. This means that you can add, remove, and modify the namespace through normal attribute access.

421 - Adding sys.implementation
PEP written and implemented by Eric Snow.

Using importlib as the Implementation of Import

2377 - Replace __import__ w/ importlib.__import__ 13959 - Re-implement parts of !imp in pure Python 14605 - Make import machinery explicit 14646 - Require loaders set __loader__ and __package__

The __import__ function is now powered by importlib.__import__. This work leads to the completion of "phase 2" of 302. There are multiple benefits to this change. First, it has allowed for more of the machinery powering import to be exposed instead of being implicit and hidden within the C code. It also provides a single implementation for all Python VMs supporting Python 3.3 to use, helping to end any VM-specific deviations in import semantics. And finally it eases the maintenance of import, allowing for future growth to occur.

For the common user, there should be no visible change in semantics. For those whose code currently manipulates import or calls import programmatically, the code changes that might possibly be required are covered in the Porting Python code section of this document.

New APIs

One of the large benefits of this work is the exposure of what goes into making the import statement work. That means the various importers that were once implicit are now fully exposed as part of the importlib package.

The abstract base classes defined in importlib.abc have been expanded to properly delineate between meta path finders <meta path finder> and path entry finders <path entry finder> by introducing importlib.abc.MetaPathFinder and importlib.abc.PathEntryFinder, respectively. The old ABC of !importlib.abc.Finder is now only provided for backwards-compatibility and does not enforce any method requirements.

In terms of finders, importlib.machinery.FileFinder exposes the mechanism used to search for source and bytecode files of a module. Previously this class was an implicit member of sys.path_hooks.

For loaders, the new abstract base class importlib.abc.FileLoader helps write a loader that uses the file system as the storage mechanism for a module's code. The loader for source files (importlib.machinery.SourceFileLoader), sourceless bytecode files (importlib.machinery.SourcelessFileLoader), and extension modules (importlib.machinery.ExtensionFileLoader) are now available for direct use.

ImportError now has name and path attributes which are set when there is relevant data to provide. The message for failed imports will also provide the full name of the module now instead of just the tail end of the module's name.

The importlib.invalidate_caches function will now call the method with the same name on all finders cached in sys.path_importer_cache to help clean up any stored state as necessary.

Visible Changes

For potential required changes to code, see the Porting Python code section.

Beyond the expanse of what importlib now exposes, there are other visible changes to import. The biggest is that sys.meta_path and sys.path_hooks now store all of the meta path finders and path entry hooks used by import. Previously the finders were implicit and hidden within the C code of import instead of being directly exposed. This means that one can now easily remove or change the order of the various finders to fit one's needs.

Another change is that all modules have a __loader__ attribute, storing the loader used to create the module. 302 has been updated to make this attribute mandatory for loaders to implement, so in the future once 3rd-party loaders have been updated people will be able to rely on the existence of the attribute. Until such time, though, import is setting the module post-load.

Loaders are also now expected to set the __package__ attribute from 366. Once again, import itself is already setting this on all loaders from importlib and import itself is setting the attribute post-load.

None is now inserted into sys.path_importer_cache when no finder can be found on sys.path_hooks. Since !imp.NullImporter is not directly exposed on sys.path_hooks it could no longer be relied upon to always be available to use as a value representing no finder found.

All other changes relate to semantic changes which should be taken into consideration when updating code for Python 3.3, and thus should be read about in the Porting Python code section of this document.

(Implementation by Brett Cannon)

Other Language Changes

Some smaller changes made to the core Python language are:

A Finer-Grained Import Lock

Previous versions of CPython have always relied on a global import lock. This led to unexpected annoyances, such as deadlocks when importing a module would trigger code execution in a different thread as a side-effect. Clumsy workarounds were sometimes employed, such as the !PyImport_ImportModuleNoBlock C API function.

In Python 3.3, importing a module takes a per-module lock. This correctly serializes importation of a given module from multiple threads (preventing the exposure of incompletely initialized modules), while eliminating the aforementioned annoyances.

(Contributed by Antoine Pitrou in 9260.)

Builtin functions and types

New Modules

faulthandler

This new debug module faulthandler contains functions to dump Python tracebacks explicitly, on a fault (a crash like a segmentation fault), after a timeout, or on a user signal. Call faulthandler.enable to install fault handlers for the ~signal.SIGSEGV, ~signal.SIGFPE, ~signal.SIGABRT, ~signal.SIGBUS, and ~signal.SIGILL signals. You can also enable them at startup by setting the PYTHONFAULTHANDLER environment variable or by using -X faulthandler command line option.

Example of a segmentation fault on Linux:

$ python -q -X faulthandler
>>> import ctypes
>>> ctypes.string_at(0)
Fatal Python error: Segmentation fault

Current thread 0x00007fb899f39700:
  File "/home/python/cpython/Lib/ctypes/__init__.py", line 486 in string_at
  File "<stdin>", line 1 in <module>
Segmentation fault

ipaddress

The new ipaddress module provides tools for creating and manipulating objects representing IPv4 and IPv6 addresses, networks and interfaces (i.e. an IP address associated with a specific IP subnet).

(Contributed by Google and Peter Moody in 3144.)

lzma

The newly added lzma module provides data compression and decompression using the LZMA algorithm, including support for the .xz and .lzma file formats.

(Contributed by Nadeem Vawda and Per Øyvind Karlsen in 6715.)

Improved Modules

abc

Improved support for abstract base classes containing descriptors composed with abstract methods. The recommended approach to declaring abstract descriptors is now to provide !__isabstractmethod__ as a dynamically updated property. The built-in descriptors have been updated accordingly.

(Contributed by Darren Dale in 11610.)

abc.ABCMeta.register now returns the registered subclass, which means it can now be used as a class decorator (10868).

array

The array module supports the long long type using q and Q type codes.

(Contributed by Oren Tirosh and Hirokazu Yamamoto in 1172711.)

base64

ASCII-only Unicode strings are now accepted by the decoding functions of the base64 modern interface. For example, base64.b64decode('YWJj') returns b'abc'. (Contributed by Catalin Iacob in 13641.)

binascii

In addition to the binary objects they normally accept, the a2b_ functions now all also accept ASCII-only strings as input. (Contributed by Antoine Pitrou in 13637.)

bz2

The bz2 module has been rewritten from scratch. In the process, several new features have been added:

codecs

The ~encodings.mbcs codec has been rewritten to handle correctly replace and ignore error handlers on all Windows versions. The ~encodings.mbcs codec now supports all error handlers, instead of only replace to encode and ignore to decode.

A new Windows-only codec has been added: cp65001 (13216). It is the Windows code page 65001 (Windows UTF-8, CP_UTF8). For example, it is used by sys.stdout if the console output code page is set to cp65001 (e.g., using chcp 65001 command).

Multibyte CJK decoders now resynchronize faster. They only ignore the first byte of an invalid byte sequence. For example, b'\xff\n'.decode('gb2312', 'replace') now returns a \n after the replacement character.

(12016)

Incremental CJK codec encoders are no longer reset at each call to their encode() methods. For example:

>>> import codecs
>>> encoder = codecs.getincrementalencoder('hz')('strict')
>>> b''.join(encoder.encode(x) for x in '\u52ff\u65bd\u65bc\u4eba\u3002 Bye.')
b'~{NpJ)l6HK!#~} Bye.'

This example gives b'~{Np~}~{J)~}~{l6~}~{HK~}~{!#~} Bye.' with older Python versions.

(12100)

The unicode_internal codec has been deprecated.

collections

Addition of a new ~collections.ChainMap class to allow treating a number of mappings as a single unit. (Written by Raymond Hettinger for 11089, made public in 11297.)

The abstract base classes have been moved in a new collections.abc module, to better differentiate between the abstract and the concrete collections classes. Aliases for ABCs are still present in the collections module to preserve existing imports. (11085)

The ~collections.Counter class now supports the unary + and - operators, as well as the in-place operators +=, -=, |=, and &=. (Contributed by Raymond Hettinger in 13121.)

contextlib

~contextlib.ExitStack now provides a solid foundation for programmatic manipulation of context managers and similar cleanup functionality. Unlike the previous contextlib.nested API (which was deprecated and removed), the new API is designed to work correctly regardless of whether context managers acquire their resources in their __init__ method (for example, file objects) or in their __enter__ method (for example, synchronisation objects from the threading module).

(13585)

crypt

Addition of salt and modular crypt format (hashing method) and the !mksalt function to the !crypt module.

(10924)

curses

(Contributed by Iñigo Serna in 6755.)

datetime

decimal

7652 - integrate fast native decimal arithmetic.
C-module and libmpdec written by Stefan Krah.

The new C version of the decimal module integrates the high speed libmpdec library for arbitrary precision correctly rounded decimal floating-point arithmetic. libmpdec conforms to IBM's General Decimal Arithmetic Specification.

Performance gains range from 10x for database applications to 100x for numerically intensive applications. These numbers are expected gains for standard precisions used in decimal floating-point arithmetic. Since the precision is user configurable, the exact figures may vary. For example, in integer bignum arithmetic the differences can be significantly higher.

The following table is meant as an illustration. Benchmarks are available at https://www.bytereef.org/mpdecimal/quickstart.html.

decimal.py

decimal

speedup

pi

42.02s

0.345s

120x

telco

172.19s

5.68s

30x

psycopg

3.57s

0.29s

12x

Features

API changes

email

Policy Framework

The email package now has a ~email.policy framework. A ~email.policy.Policy is an object with several methods and properties that control how the email package behaves. The primary policy for Python 3.3 is the ~email.policy.Compat32 policy, which provides backward compatibility with the email package in Python 3.2. A policy can be specified when an email message is parsed by a ~email.parser, or when a ~email.message.Message object is created, or when an email is serialized using a ~email.generator. Unless overridden, a policy passed to a parser is inherited by all the Message object and sub-objects created by the parser. By default a generator will use the policy of the Message object it is serializing. The default policy is ~email.policy.compat32.

The minimum set of controls implemented by all policy objects are:

|l|L|
max_line_length The maximum length, excluding the linesep character(s), individual lines may have when a Message is serialized. Defaults to 78.
linesep The character used to separate individual lines when a Message is serialized. Defaults to \n.
cte_type 7bit or 8bit. 8bit applies only to a Bytes generator, and means that non-ASCII may be used where allowed by the protocol (or where it exists in the original input).
raise_on_defect Causes a parser to raise error when defects are encountered instead of adding them to the Message object's defects list.

A new policy instance, with new settings, is created using the ~email.policy.Policy.clone method of policy objects. clone takes any of the above controls as keyword arguments. Any control not specified in the call retains its default value. Thus you can create a policy that uses \r\n linesep characters like this:

mypolicy = compat32.clone(linesep='\r\n')

Policies can be used to make the generation of messages in the format needed by your application simpler. Instead of having to remember to specify linesep='\r\n' in all the places you call a generator, you can specify it once, when you set the policy used by the parser or the Message, whichever your program uses to create Message objects. On the other hand, if you need to generate messages in multiple forms, you can still specify the parameters in the appropriate generator call. Or you can have custom policy instances for your different cases, and pass those in when you create the generator.

Provisional Policy with New Header API

While the policy framework is worthwhile all by itself, the main motivation for introducing it is to allow the creation of new policies that implement new features for the email package in a way that maintains backward compatibility for those who do not use the new policies. Because the new policies introduce a new API, we are releasing them in Python 3.3 as a provisional policy <provisional package>. Backwards incompatible changes (up to and including removal of the code) may occur if deemed necessary by the core developers.

The new policies are instances of ~email.policy.EmailPolicy, and add the following additional controls:

|l|L|
refold_source Controls whether or not headers parsed by a ~email.parser are refolded by the ~email.generator. It can be none, long, or all. The default is long, which means that source headers with a line longer than max_line_length get refolded. none means no line get refolded, and all means that all lines get refolded.
header_factory A callable that take a name and value and produces a custom header object.

The header_factory is the key to the new features provided by the new policies. When one of the new policies is used, any header retrieved from a Message object is an object produced by the header_factory, and any time you set a header on a Message it becomes an object produced by header_factory. All such header objects have a name attribute equal to the header name. Address and Date headers have additional attributes that give you access to the parsed data of the header. This means you can now do things like this:

>>> m = Message(policy=SMTP)
>>> m['To'] = 'Éric <foo@example.com>'
>>> m['to']
'Éric <foo@example.com>'
>>> m['to'].addresses
(Address(display_name='Éric', username='foo', domain='example.com'),)
>>> m['to'].addresses[0].username
'foo'
>>> m['to'].addresses[0].display_name
'Éric'
>>> m['Date'] = email.utils.localtime()
>>> m['Date'].datetime
datetime.datetime(2012, 5, 25, 21, 39, 24, 465484, tzinfo=datetime.timezone(datetime.timedelta(-1, 72000), 'EDT'))
>>> m['Date']
'Fri, 25 May 2012 21:44:27 -0400'
>>> print(m)
To: =?utf-8?q?=C3=89ric?= <foo@example.com>
Date: Fri, 25 May 2012 21:44:27 -0400

You will note that the unicode display name is automatically encoded as utf-8 when the message is serialized, but that when the header is accessed directly, you get the unicode version. This eliminates any need to deal with the email.header ~email.header.decode_header or ~email.header.make_header functions.

You can also create addresses from parts:

>>> m['cc'] = [Group('pals', [Address('Bob', 'bob', 'example.com'),
...                           Address('Sally', 'sally', 'example.com')]),
...            Address('Bonzo', addr_spec='bonz@laugh.com')]
>>> print(m)
To: =?utf-8?q?=C3=89ric?= <foo@example.com>
Date: Fri, 25 May 2012 21:44:27 -0400
cc: pals: Bob <bob@example.com>, Sally <sally@example.com>;, Bonzo <bonz@laugh.com>

Decoding to unicode is done automatically:

>>> m2 = message_from_string(str(m))
>>> m2['to']
'Éric <foo@example.com>'

When you parse a message, you can use the addresses and groups attributes of the header objects to access the groups and individual addresses:

>>> m2['cc'].addresses
(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com'), Address(display_name='Bonzo', username='bonz', domain='laugh.com'))
>>> m2['cc'].groups
(Group(display_name='pals', addresses=(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com')), Group(display_name=None, addresses=(Address(display_name='Bonzo', username='bonz', domain='laugh.com'),))

In summary, if you use one of the new policies, header manipulation works the way it ought to: your application works with unicode strings, and the email package transparently encodes and decodes the unicode to and from the RFC standard Content Transfer Encodings.

Other API Changes

New ~email.parser.BytesHeaderParser, added to the ~email.parser module to complement ~email.parser.HeaderParser and complete the Bytes API.

New utility functions:

ftplib

functools

The functools.lru_cache decorator now accepts a typed keyword argument (that defaults to False to ensure that it caches values of different types that compare equal in separate cache slots. (Contributed by Raymond Hettinger in 13227.)

gc

It is now possible to register callbacks invoked by the garbage collector before and after collection using the new ~gc.callbacks list.

hmac

A new ~hmac.compare_digest function has been added to prevent side channel attacks on digests through timing analysis. (Contributed by Nick Coghlan and Christian Heimes in 15061.)

http

http.server.BaseHTTPRequestHandler now buffers the headers and writes them all at once when ~http.server.BaseHTTPRequestHandler.end_headers is called. A new method ~http.server.BaseHTTPRequestHandler.flush_headers can be used to directly manage when the accumulated headers are sent. (Contributed by Andrew Schaaf in 3709.)

http.server now produces valid HTML 4.01 strict output. (Contributed by Ezio Melotti in 13295.)

http.client.HTTPResponse now has a ~http.client.HTTPResponse.readinto method, which means it can be used as an io.RawIOBase class. (Contributed by John Kuhn in 13464.)

html

html.parser.HTMLParser is now able to parse broken markup without raising errors, therefore the strict argument of the constructor and the !HTMLParseError exception are now deprecated. The ability to parse broken markup is the result of a number of bug fixes that are also available on the latest bug fix releases of Python 2.7/3.2. (Contributed by Ezio Melotti in 15114, and 14538, 13993, 13960, 13358, 1745761, 755670, 13357, 12629, 1200313, 670664, 13273, 12888, 7311.)

A new ~html.entities.html5 dictionary that maps HTML5 named character references to the equivalent Unicode character(s) (e.g. html5['gt;'] == '>') has been added to the html.entities module. The dictionary is now also used by ~html.parser.HTMLParser. (Contributed by Ezio Melotti in 11113 and 15156.)

imaplib

The ~imaplib.IMAP4_SSL constructor now accepts an SSLContext parameter to control parameters of the secure channel.

(Contributed by Sijin Joseph in 8808.)

inspect

A new ~inspect.getclosurevars function has been added. This function reports the current binding of all names referenced from the function body and where those names were resolved, making it easier to verify correct internal state when testing code that relies on stateful closures.

(Contributed by Meador Inge and Nick Coghlan in 13062.)

A new ~inspect.getgeneratorlocals function has been added. This function reports the current binding of local variables in the generator's stack frame, making it easier to verify correct internal state when testing generators.

(Contributed by Meador Inge in 15153.)

io

The ~io.open function has a new 'x' mode that can be used to exclusively create a new file, and raise a FileExistsError if the file already exists. It is based on the C11 'x' mode to fopen().

(Contributed by David Townshend in 12760.)

The constructor of the ~io.TextIOWrapper class has a new write_through optional argument. If write_through is True, calls to !write are guaranteed not to be buffered: any data written on the ~io.TextIOWrapper object is immediately handled to its underlying binary buffer.

itertools

~itertools.accumulate now takes an optional func argument for providing a user-supplied binary function.

logging

The ~logging.basicConfig function now supports an optional handlers argument taking an iterable of handlers to be added to the root logger.

A class level attribute !append_nul has been added to ~logging.handlers.SysLogHandler to allow control of the appending of the NUL (\000) byte to syslog records, since for some daemons it is required while for others it is passed through to the log.

math

The math module has a new function, ~math.log2, which returns the base-2 logarithm of x.

(Written by Mark Dickinson in 11888.)

mmap

The ~mmap.mmap.read method is now more compatible with other file-like objects: if the argument is omitted or specified as None, it returns the bytes from the current file position to the end of the mapping. (Contributed by Petri Lehtinen in 12021.)

multiprocessing

The new multiprocessing.connection.wait function allows polling multiple objects (such as connections, sockets and pipes) with a timeout. (Contributed by Richard Oudkerk in 12328.)

multiprocessing.connection.Connection objects can now be transferred over multiprocessing connections. (Contributed by Richard Oudkerk in 4892.)

multiprocessing.Process now accepts a daemon keyword argument to override the default behavior of inheriting the daemon flag from the parent process (6064).

New attribute multiprocessing.Process.sentinel allows a program to wait on multiple ~multiprocessing.Process objects at one time using the appropriate OS primitives (for example, select on posix systems).

New methods multiprocessing.pool.Pool.starmap and ~multiprocessing.pool.Pool.starmap_async provide itertools.starmap equivalents to the existing multiprocessing.pool.Pool.map and ~multiprocessing.pool.Pool.map_async functions. (Contributed by Hynek Schlawack in 12708.)

nntplib

The !nntplib.NNTP class now supports the context management protocol to unconditionally consume socket.error exceptions and to close the NNTP connection when done:

>>> from nntplib import NNTP
>>> with NNTP('news.gmane.org') as n:
...     n.group('gmane.comp.python.committers')
...
('211 1755 1 1755 gmane.comp.python.committers', 1755, 1, 1755, 'gmane.comp.python.committers')
>>>

(Contributed by Giampaolo Rodolà in 9795.)

os

pdb

Tab-completion is now available not only for command names, but also their arguments. For example, for the break command, function and file names are completed.

(Contributed by Georg Brandl in 14210)

pickle

pickle.Pickler objects now have an optional ~pickle.Pickler.dispatch_table attribute allowing per-pickler reduction functions to be set.

(Contributed by Richard Oudkerk in 14166.)

pydoc

The Tk GUI and the !serve function have been removed from the pydoc module: pydoc -g and !serve have been deprecated in Python 3.2.

re

str regular expressions now support \u and \U escapes.

(Contributed by Serhiy Storchaka in 3665.)

sched

select

Solaris and derivative platforms have a new class select.devpoll for high performance asynchronous sockets via /dev/poll. (Contributed by Jesús Cea Avión in 6397.)

shlex

The previously undocumented helper function quote from the !pipes modules has been moved to the shlex module and documented. ~shlex.quote properly escapes all characters in a string that might be otherwise given special meaning by the shell.

shutil

signal

smtpd

The !smtpd module now supports 5321 (extended SMTP) and 1870 (size extension). Per the standard, these extensions are enabled if and only if the client initiates the session with an EHLO command.

(Initial ELHO support by Alberto Trevino. Size extension by Juhana Jauhiainen. Substantial additional work on the patch contributed by Michele Orrù and Dan Boswell. 8739)

smtplib

The ~smtplib.SMTP, ~smtplib.SMTP_SSL, and ~smtplib.LMTP classes now accept a source_address keyword argument to specify the (host, port) to use as the source address in the bind call when creating the outgoing socket. (Contributed by Paulo Scardine in 11281.)

~smtplib.SMTP now supports the context management protocol, allowing an SMTP instance to be used in a with statement. (Contributed by Giampaolo Rodolà in 11289.)

The ~smtplib.SMTP_SSL constructor and the ~smtplib.SMTP.starttls method now accept an SSLContext parameter to control parameters of the secure channel. (Contributed by Kasun Herath in 8809.)

socket

socketserver

~socketserver.BaseServer now has an overridable method ~socketserver.BaseServer.service_actions that is called by the ~socketserver.BaseServer.serve_forever method in the service loop. ~socketserver.ForkingMixIn now uses this to clean up zombie child processes. (Contributed by Justin Warkentin in 11109.)

sqlite3

New sqlite3.Connection method ~sqlite3.Connection.set_trace_callback can be used to capture a trace of all sql commands processed by sqlite. (Contributed by Torsten Landschoff in 11688.)

ssl

stat

The undocumented tarfile.filemode function has been moved to stat.filemode. It can be used to convert a file's mode to a string of the form '-rwxrwxrwx'.

(Contributed by Giampaolo Rodolà in 14807.)

struct

The struct module now supports ssize_t and size_t via the new codes n and N, respectively. (Contributed by Antoine Pitrou in 3163.)

subprocess

Command strings can now be bytes objects on posix platforms. (Contributed by Victor Stinner in 8513.)

A new constant ~subprocess.DEVNULL allows suppressing output in a platform-independent fashion. (Contributed by Ross Lagerwall in 5870.)

sys

The sys module has a new ~sys.thread_info named tuple holding information about the thread implementation (11223).

tarfile

tarfile now supports lzma encoding via the lzma module. (Contributed by Lars Gustäbel in 5689.)

tempfile

tempfile.SpooledTemporaryFile's !truncate method now accepts a size parameter. (Contributed by Ryan Kelly in 9957.)

textwrap

The textwrap module has a new ~textwrap.indent that makes it straightforward to add a common prefix to selected lines in a block of text (13857).

threading

threading.Condition, threading.Semaphore, threading.BoundedSemaphore, threading.Event, and threading.Timer, all of which used to be factory functions returning a class instance, are now classes and may be subclassed. (Contributed by Éric Araujo in 10968.)

The threading.Thread constructor now accepts a daemon keyword argument to override the default behavior of inheriting the daemon flag value from the parent thread (6064).

The formerly private function _thread.get_ident is now available as the public function threading.get_ident. This eliminates several cases of direct access to the _thread module in the stdlib. Third party code that used _thread.get_ident should likewise be changed to use the new public interface.

time

The 418 added new functions to the time module:

Other new functions:

To improve cross platform consistency, ~time.sleep now raises a ValueError when passed a negative sleep value. Previously this was an error on posix, but produced an infinite sleep on Windows.

types

Add a new types.MappingProxyType class: Read-only proxy of a mapping. (14386)

The new functions types.new_class and types.prepare_class provide support for 3115 compliant dynamic type creation. (14588)

unittest

.assertRaises, .assertRaisesRegex, .assertWarns, and .assertWarnsRegex now accept a keyword argument msg when used as context managers. (Contributed by Ezio Melotti and Winston Ewert in 10775.)

unittest.TestCase.run now returns the ~unittest.TestResult object.

urllib

The ~urllib.request.Request class, now accepts a method argument used by ~urllib.request.Request.get_method to determine what HTTP method should be used. For example, this will send a 'HEAD' request:

>>> urlopen(Request('https://www.python.org', method='HEAD'))

(1673007)

webbrowser

The webbrowser module supports more "browsers": Google Chrome (named chrome, chromium, chrome-browser or chromium-browser depending on the version and operating system), and the generic launchers xdg-open, from the FreeDesktop.org project, and gvfs-open, which is the default URI handler for GNOME 3. (The former contributed by Arnaud Calmettes in 13620, the latter by Matthias Klose in 14493.)

xml.etree.ElementTree

The xml.etree.ElementTree module now imports its C accelerator by default; there is no longer a need to explicitly import !xml.etree.cElementTree (this module stays for backwards compatibility, but is now deprecated). In addition, the iter family of methods of ~xml.etree.ElementTree.Element has been optimized (rewritten in C). The module's documentation has also been greatly improved with added examples and a more detailed reference.

zlib

New attribute zlib.Decompress.eof makes it possible to distinguish between a properly formed compressed stream and an incomplete or truncated one. (Contributed by Nadeem Vawda in 12646.)

New attribute zlib.ZLIB_RUNTIME_VERSION reports the version string of the underlying zlib library that is loaded at runtime. (Contributed by Torsten Landschoff in 12306.)

Optimizations

Major performance enhancements have been added:

Build and C API Changes

Changes to Python's build process and to the C API include:

Deprecated

Unsupported Operating Systems

OS/2 and VMS are no longer supported due to the lack of a maintainer.

Windows 2000 and Windows platforms which set COMSPEC to command.com are no longer supported due to maintenance burden.

OSF support, which was deprecated in 3.2, has been completely removed.

Deprecated Python modules, functions and methods

Deprecated functions and types of the C API

The Py_UNICODE has been deprecated by 393 and will be removed in Python 4. All functions using this type are deprecated:

Unicode functions and methods using Py_UNICODE and Py_UNICODE* types:

Functions and macros manipulating Py_UNICODE* strings:

Encoders:

Deprecated features

The array module's 'u' format code is now deprecated and will be removed in Python 4 together with the rest of the (Py_UNICODE) API.

Porting to Python 3.3

This section lists previously described changes and other bugfixes that may require changes to your code.

Porting Python code

Porting C code

Building C extensions

Command Line Switch Changes

What's New In Python 3.4

Author
R. David Murray <rdmurray@bitdance.com> (Editor)

* Anyone can add text to this document, but the maintainer reserves the right to rewrite any additions. In particular, for obscure or esoteric features, the maintainer may reduce any addition to a simple reference to the new documentation rather than explaining the feature inline.

* While the maintainer will periodically go through Misc/NEWS and add changes, it's best not to rely on this. We know from experience that any changes that aren't in the What's New documentation around the time of the original release will remain largely unknown to the community for years, even if they're added later. We also know from experience that other priorities can arise, and the maintainer will run out of time to do updates -- in such cases, end users will be much better served by partial notifications that at least give a hint about new features to investigate.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. The What's New should focus on changes that are visible to Python users and that require a feature release (i.e. most bug fixes should only be recorded in Misc/NEWS)

* PEPs should not be marked Final until they have an entry in What's New. A placeholder entry that is just a section header and a link to the PEP (e.g "397 has been implemented") is acceptable. If a PEP has been implemented and noted in What's New, don't forget to mark it as Final!

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to add just a very brief note about a change. For example: "The ~socket.transmogrify() function was added to the socket module." The maintainer will research the change and write the necessary text (if appropriate). The advantage of doing this is that even if no more descriptive text is ever added, readers will at least have a notification that the new feature exists and a link to the relevant documentation.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

The ~socket.transmogrify() function was added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the Mercurial log when researching a change.

* Cross referencing tip: mod.attr will display as mod.attr, while ~mod.attr will display as attr.

This article explains the new features in Python 3.4, compared to 3.3. Python 3.4 was released on March 16, 2014. For full details, see the changelog.

429 -- Python 3.4 Release Schedule

Summary -- Release Highlights

New syntax features:

Other new features:

New library modules:

Significantly improved library modules:

Security improvements:

CPython implementation improvements:

Please read on for a comprehensive list of user-facing changes, including many other smaller improvements, CPython optimizations, deprecations, and potential porting issues.

New Features

PEP 453: Explicit Bootstrapping of PIP in Python Installations

Bootstrapping pip By Default

The new ensurepip module (defined in 453) provides a standard cross-platform mechanism to bootstrap the pip installer into Python installations and virtual environments. The version of pip included with Python 3.4.0 is pip 1.5.4, and future 3.4.x maintenance releases will update the bundled version to the latest version of pip that is available at the time of creating the release candidate.

By default, the commands pipX and pipX.Y will be installed on all platforms (where X.Y stands for the version of the Python installation), along with the pip Python package and its dependencies. On Windows and in virtual environments on all platforms, the unversioned pip command will also be installed. On other platforms, the system wide unversioned pip command typically refers to the separately installed Python 2 version.

The pyvenv command line utility and the venv module make use of the ensurepip module to make pip readily available in virtual environments. When using the command line utility, pip is installed by default, while when using the venv module venv-api installation of pip must be requested explicitly.

For CPython source builds on POSIX systems <building-python-on-unix>, the make install and make altinstall commands bootstrap pip by default. This behaviour can be controlled through configure options, and overridden through Makefile options.

On Windows and Mac OS X, the CPython installers now default to installing pip along with CPython itself (users may opt out of installing it during the installation process). Window users will need to opt in to the automatic PATH modifications to have pip available from the command line by default, otherwise it can still be accessed through the Python launcher for Windows as py -m pip.

As discussed in the PEP <0453#recommendations-for-downstream-distributors> platform packagers may choose not to install these commands by default, as long as, when invoked, they provide clear and simple directions on how to install them on that platform (usually using the system package manager).

[!NOTE] To avoid conflicts between parallel Python 2 and Python 3 installations, only the versioned pip3 and pip3.4 commands are bootstrapped by default when ensurepip is invoked directly - the --default-pip option is needed to also request the unversioned pip command. pyvenv and the Windows installer ensure that the unqualified pip command is made available in those environments, and pip can always be invoked via the -m switch rather than directly to avoid ambiguity on systems with multiple Python installations.

Documentation Changes

As part of this change, the installing-index and distributing-index sections of the documentation have been completely redesigned as short getting started and FAQ documents. Most packaging documentation has now been moved out to the Python Packaging Authority maintained Python Packaging User Guide and the documentation of the individual projects.

However, as this migration is currently still incomplete, the legacy versions of those guides remaining available as install-index and setuptools-index.

453 -- Explicit bootstrapping of pip in Python installations
PEP written by Donald Stufft and Nick Coghlan, implemented by Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily.

PEP 446: Newly Created File Descriptors Are Non-Inheritable

446 makes newly created file descriptors non-inheritable <fd_inheritance>. In general, this is the behavior an application will want: when launching a new process, having currently open files also open in the new process can lead to all sorts of hard to find bugs, and potentially to security issues.

However, there are occasions when inheritance is desired. To support these cases, the following new functions and methods are available:

446 -- Make newly created file descriptors non-inheritable
PEP written and implemented by Victor Stinner.

Improvements to Codec Handling

Since it was first introduced, the codecs module has always been intended to operate as a type-neutral dynamic encoding and decoding system. However, its close coupling with the Python text model, especially the type restricted convenience methods on the builtin str, bytes and bytearray types, has historically obscured that fact.

As a key step in clarifying the situation, the codecs.encode and codecs.decode convenience functions are now properly documented in Python 2.7, 3.3 and 3.4. These functions have existed in the codecs module (and have been covered by the regression test suite) since Python 2.4, but were previously only discoverable through runtime introspection.

Unlike the convenience methods on str, bytes and bytearray, the codecs convenience functions support arbitrary codecs in both Python 2 and Python 3, rather than being limited to Unicode text encodings (in Python 3) or basestring <-> basestring conversions (in Python 2).

In Python 3.4, the interpreter is able to identify the known non-text encodings provided in the standard library and direct users towards these general purpose convenience functions when appropriate:

>>> b"abcdef".decode("hex")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'hex' is not a text encoding; use codecs.decode() to handle arbitrary codecs

>>> "hello".encode("rot13")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'rot13' is not a text encoding; use codecs.encode() to handle arbitrary codecs

>>> open("foo.txt", encoding="hex")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'hex' is not a text encoding; use codecs.open() to handle arbitrary codecs

>>> import codecs

>>> codecs.decode(b"abcdefgh", "hex")
Traceback (most recent call last):
  File "/usr/lib/python3.4/encodings/hex_codec.py", line 20, in hex_decode
    return (binascii.a2b_hex(input), len(input))
binascii.Error: Non-hexadecimal digit found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
binascii.Error: decoding with 'hex' codec failed (Error: Non-hexadecimal digit found)

>>> codecs.encode("hello", "bz2")
Traceback (most recent call last):
  File "/usr/lib/python3.4/encodings/bz2_codec.py", line 17, in bz2_encode
    return (bz2.compress(input), len(input))
  File "/usr/lib/python3.4/bz2.py", line 498, in compress
    return comp.compress(data) + comp.flush()
TypeError: 'str' does not support the buffer interface

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: encoding with 'bz2' codec failed (TypeError: 'str' does not support the buffer interface)

Finally, as the examples above show, these improvements have permitted the restoration of the convenience aliases for the non-Unicode codecs that were themselves restored in Python 3.2. This means that encoding binary data to and from its hexadecimal representation (for example) can now be written as:

>>> from codecs import encode, decode
>>> encode(b"hello", "hex")
b'68656c6c6f'
>>> decode(b"68656c6c6f", "hex")
b'hello'

The binary and text transforms provided in the standard library are detailed in binary-transforms and text-transforms.

(Contributed by Nick Coghlan in 7475, 17827, 17828 and 19619.)

PEP 451: A ModuleSpec Type for the Import System

451 provides an encapsulation of the information about a module that the import machinery will use to load it (that is, a module specification). This helps simplify both the import implementation and several import-related APIs. The change is also a stepping stone for several future import-related improvements.

The public-facing changes from the PEP are entirely backward-compatible. Furthermore, they should be transparent to everyone but importer authors. Key finder and loader methods have been deprecated, but they will continue working. New importers should use the new methods described in the PEP. Existing importers should be updated to implement the new methods. See the deprecated-3.4 section for a list of methods that should be replaced and their replacements.

Other Language Changes

Some smaller changes made to the core Python language are:

New Modules

asyncio

The new asyncio module (defined in 3156) provides a standard pluggable event loop model for Python, providing solid asynchronous IO support in the standard library, and making it easier for other event loop implementations to interoperate with the standard library and each other.

For Python 3.4, this module is considered a provisional API.

3156 -- Asynchronous IO Support Rebooted: the "asyncio" Module
PEP written and implementation led by Guido van Rossum.

ensurepip

The new ensurepip module is the primary infrastructure for the 453 implementation. In the normal course of events end users will not need to interact with this module, but it can be used to manually bootstrap pip if the automated bootstrapping into an installation or virtual environment was declined.

ensurepip includes a bundled copy of pip, up-to-date as of the first release candidate of the release of CPython with which it ships (this applies to both maintenance releases and feature releases). ensurepip does not access the internet. If the installation has internet access, after ensurepip is run the bundled pip can be used to upgrade pip to a more recent release than the bundled one. (Note that such an upgraded version of pip is considered to be a separately installed package and will not be removed if Python is uninstalled.)

The module is named ensurepip because if called when pip is already installed, it does nothing. It also has an --upgrade option that will cause it to install the bundled copy of pip if the existing installed version of pip is older than the bundled copy.

enum

The new enum module (defined in 435) provides a standard implementation of enumeration types, allowing other modules (such as socket) to provide more informative error messages and better debugging support by replacing opaque integer constants with backwards compatible enumeration values.

435 -- Adding an Enum type to the Python standard library
PEP written by Barry Warsaw, Eli Bendersky and Ethan Furman, implemented by Ethan Furman.

pathlib

The new pathlib module offers classes representing filesystem paths with semantics appropriate for different operating systems. Path classes are divided between pure paths, which provide purely computational operations without I/O, and concrete paths, which inherit from pure paths but also provide I/O operations.

For Python 3.4, this module is considered a provisional API.

428 -- The pathlib module -- object-oriented filesystem paths
PEP written and implemented by Antoine Pitrou.

selectors

The new selectors module (created as part of implementing 3156) allows high-level and efficient I/O multiplexing, built upon the select module primitives.

statistics

The new statistics module (defined in 450) offers some core statistics functionality directly in the standard library. This module supports calculation of the mean, median, mode, variance and standard deviation of a data series.

450 -- Adding A Statistics Module To The Standard Library
PEP written and implemented by Steven D'Aprano

tracemalloc

The new tracemalloc module (defined in 454) is a debug tool to trace memory blocks allocated by Python. It provides the following information:

454 -- Add a new tracemalloc module to trace Python memory allocations
PEP written and implemented by Victor Stinner

Improved Modules

abc

New function abc.get_cache_token can be used to know when to invalidate caches that are affected by changes in the object graph. (Contributed by Łukasz Langa in 16832.)

New class ~abc.ABC has ~abc.ABCMeta as its meta class. Using ABC as a base class has essentially the same effect as specifying metaclass=abc.ABCMeta, but is simpler to type and easier to read. (Contributed by Bruno Dupuis in 16049.)

aifc

The !getparams method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in 17818.)

!aifc.open now supports the context management protocol: when used in a with block, the !close method of the returned object will be called automatically at the end of the block. (Contributed by Serhiy Storchacha in 16486.)

The !writeframesraw and !writeframes methods now accept any bytes-like object. (Contributed by Serhiy Storchaka in 8311.)

argparse

The ~argparse.FileType class now accepts encoding and errors arguments, which are passed through to open. (Contributed by Lucas Maystre in 11175.)

audioop

!audioop now supports 24-bit samples. (Contributed by Serhiy Storchaka in 12866.)

New !byteswap function converts big-endian samples to little-endian and vice versa. (Contributed by Serhiy Storchaka in 19641.)

All !audioop functions now accept any bytes-like object. Strings are not accepted: they didn't work before, now they raise an error right away. (Contributed by Serhiy Storchaka in 16685.)

base64

The encoding and decoding functions in base64 now accept any bytes-like object in cases where it previously required a bytes or bytearray instance. (Contributed by Nick Coghlan in 17839.)

New functions ~base64.a85encode, ~base64.a85decode, ~base64.b85encode, and ~base64.b85decode provide the ability to encode and decode binary data from and to Ascii85 and the git/mercurial Base85 formats, respectively. The a85 functions have options that can be used to make them compatible with the variants of the Ascii85 encoding, including the Adobe variant. (Contributed by Martin Morrison, the Mercurial project, Serhiy Storchaka, and Antoine Pitrou in 17618.)

collections

The .ChainMap.new_child method now accepts an m argument specifying the child map to add to the chain. This allows an existing mapping and/or a custom mapping type to be used for the child. (Contributed by Vinay Sajip in 16613.)

colorsys

The number of digits in the coefficients for the RGB --- YIQ conversions have been expanded so that they match the FCC NTSC versions. The change in results should be less than 1% and may better match results found elsewhere. (Contributed by Brian Landers and Serhiy Storchaka in 14323.)

contextlib

The new contextlib.suppress context manager helps to clarify the intent of code that deliberately suppresses exceptions from a single statement. (Contributed by Raymond Hettinger in 15806 and Zero Piraeus in 19266.)

The new contextlib.redirect_stdout context manager makes it easier for utility scripts to handle inflexible APIs that write their output to sys.stdout and don't provide any options to redirect it. Using the context manager, the sys.stdout output can be redirected to any other stream or, in conjunction with io.StringIO, to a string. The latter can be especially useful, for example, to capture output from a function that was written to implement a command line interface. It is recommended only for utility scripts because it affects the global state of sys.stdout. (Contributed by Raymond Hettinger in 15805.)

The contextlib documentation has also been updated to include a discussion <single-use-reusable-and-reentrant-cms> of the differences between single use, reusable and reentrant context managers.

dbm

dbm.open objects now support the context management protocol. When used in a with statement, the close method of the database object will be called automatically at the end of the block. (Contributed by Claudiu Popa and Nick Coghlan in 19282.)

dis

Functions ~dis.show_code, ~dis.dis, ~dis.distb, and ~dis.disassemble now accept a keyword-only file argument that controls where they write their output.

The dis module is now built around an ~dis.Instruction class that provides object oriented access to the details of each individual bytecode operation.

A new method, ~dis.get_instructions, provides an iterator that emits the Instruction stream for a given piece of Python code. Thus it is now possible to write a program that inspects and manipulates a bytecode object in ways different from those provided by the ~dis module itself. For example:

>>> import dis
>>> for instr in dis.get_instructions(lambda x: x + 1):
...     print(instr.opname)
LOAD_FAST
LOAD_CONST
BINARY_ADD
RETURN_VALUE

The various display tools in the dis module have been rewritten to use these new components.

In addition, a new application-friendly class ~dis.Bytecode provides an object-oriented API for inspecting bytecode in both in human-readable form and for iterating over instructions. The ~dis.Bytecode constructor takes the same arguments that ~dis.get_instructions does (plus an optional current_offset), and the resulting object can be iterated to produce ~dis.Instruction objects. But it also has a ~dis.Bytecode.dis method, equivalent to calling ~dis.dis on the constructor argument, but returned as a multi-line string:

>>> bytecode = dis.Bytecode(lambda x: x + 1, current_offset=3)
>>> for instr in bytecode:
...     print('{} ({})'.format(instr.opname, instr.opcode))
LOAD_FAST (124)
LOAD_CONST (100)
BINARY_ADD (23)
RETURN_VALUE (83)
>>> bytecode.dis().splitlines()       # doctest: +NORMALIZE_WHITESPACE
['  1           0 LOAD_FAST                0 (x)',
 '      -->     3 LOAD_CONST               1 (1)',
 '              6 BINARY_ADD',
 '              7 RETURN_VALUE']

~dis.Bytecode also has a class method, ~dis.Bytecode.from_traceback, that provides the ability to manipulate a traceback (that is, print(Bytecode.from_traceback(tb).dis()) is equivalent to distb(tb)).

(Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in 11816 and Claudiu Popa in 17916.)

New function ~dis.stack_effect computes the effect on the Python stack of a given opcode and argument, information that is not otherwise available. (Contributed by Larry Hastings in 19722.)

doctest

A new option flag <doctest-options>, ~doctest.FAIL_FAST, halts test running as soon as the first failure is detected. (Contributed by R. David Murray and Daniel Urban in 16522.)

The doctest command line interface now uses argparse, and has two new options, -o and -f. -o allows doctest options <doctest-options> to be specified on the command line, and -f is a shorthand for -o FAIL_FAST (to parallel the similar option supported by the unittest CLI). (Contributed by R. David Murray in 11390.)

doctest will now find doctests in extension module __doc__ strings. (Contributed by Zachary Ware in 3158.)

email

~email.message.Message.as_string now accepts a policy argument to override the default policy of the message when generating a string representation of it. This means that as_string can now be used in more circumstances, instead of having to create and use a ~email.generator in order to pass formatting parameters to its flatten method. (Contributed by R. David Murray in 18600.)

New method ~email.message.Message.as_bytes added to produce a bytes representation of the message in a fashion similar to how as_string produces a string representation. It does not accept the maxheaderlen argument, but does accept the unixfrom and policy arguments. The ~email.message.Message ~email.message.Message.__bytes__ method calls it, meaning that bytes(mymsg) will now produce the intuitive result: a bytes object containing the fully formatted message. (Contributed by R. David Murray in 18600.)

The .Message.set_param message now accepts a replace keyword argument. When specified, the associated header will be updated without changing its location in the list of headers. For backward compatibility, the default is False. (Contributed by R. David Murray in 18891.)

A pair of new subclasses of ~email.message.Message have been added (.EmailMessage and .MIMEPart), along with a new sub-module, ~email.contentmanager and a new ~email.policy attribute ~email.policy.EmailPolicy.content_manager. All documentation is currently in the new module, which is being added as part of email's new provisional API. These classes provide a number of new methods that make extracting content from and inserting content into email messages much easier. For details, see the ~email.contentmanager documentation and the email-examples. These API additions complete the bulk of the work that was planned as part of the email6 project. The currently provisional API is scheduled to become final in Python 3.5 (possibly with a few minor additions in the area of error handling). (Contributed by R. David Murray in 18891.)

filecmp

A new ~filecmp.clear_cache function provides the ability to clear the filecmp comparison cache, which uses os.stat information to determine if the file has changed since the last compare. This can be used, for example, if the file might have been changed and re-checked in less time than the resolution of a particular filesystem's file modification time field. (Contributed by Mark Levitt in 18149.)

New module attribute ~filecmp.DEFAULT_IGNORES provides the list of directories that are used as the default value for the ignore parameter of the ~filecmp.dircmp function. (Contributed by Eli Bendersky in 15442.)

functools

The new ~functools.partialmethod descriptor brings partial argument application to descriptors, just as ~functools.partial provides for normal callables. The new descriptor also makes it easier to get arbitrary callables (including ~functools.partial instances) to behave like normal instance methods when included in a class definition. (Contributed by Alon Horev and Nick Coghlan in 4331.)

The new ~functools.singledispatch decorator brings support for single-dispatch generic functions to the Python standard library. Where object oriented programming focuses on grouping multiple operations on a common set of data into a class, a generic function focuses on grouping multiple implementations of an operation that allows it to work with different kinds of data.

443 -- Single-dispatch generic functions
PEP written and implemented by Łukasz Langa.

~functools.total_ordering now supports a return value of NotImplemented from the underlying comparison function. (Contributed by Katie Miller in 10042.)

A pure-python version of the ~functools.partial function is now in the stdlib; in CPython it is overridden by the C accelerated version, but it is available for other implementations to use. (Contributed by Brian Thorne in 12428.)

gc

New function ~gc.get_stats returns a list of three per-generation dictionaries containing the collections statistics since interpreter startup. (Contributed by Antoine Pitrou in 16351.)

glob

A new function ~glob.escape provides a way to escape special characters in a filename so that they do not become part of the globbing expansion but are instead matched literally. (Contributed by Serhiy Storchaka in 8402.)

hashlib

A new hashlib.pbkdf2_hmac function provides the PKCS#5 password-based key derivation function 2. (Contributed by Christian Heimes in 18582.)

The ~hashlib.hash.name attribute of hashlib hash objects is now a formally supported interface. It has always existed in CPython's hashlib (although it did not return lower case names for all supported hashes), but it was not a public interface and so some other Python implementations have not previously supported it. (Contributed by Jason R. Coombs in 18532.)

hmac

hmac now accepts bytearray as well as bytes for the key argument to the ~hmac.new function, and the msg parameter to both the ~hmac.new function and the ~hmac.HMAC.update method now accepts any type supported by the hashlib module. (Contributed by Jonas Borgström in 18240.)

The digestmod argument to the hmac.new function may now be any hash digest name recognized by hashlib. In addition, the current behavior in which the value of digestmod defaults to MD5 is deprecated: in a future version of Python there will be no default value. (Contributed by Christian Heimes in 17276.)

With the addition of ~hmac.HMAC.block_size and ~hmac.HMAC.name attributes (and the formal documentation of the ~hmac.HMAC.digest_size attribute), the hmac module now conforms fully to the 247 API. (Contributed by Christian Heimes in 18775.)

html

New function ~html.unescape function converts HTML5 character references to the corresponding Unicode characters. (Contributed by Ezio Melotti in 2927.)

~html.parser.HTMLParser accepts a new keyword argument convert_charrefs that, when True, automatically converts all character references. For backward-compatibility, its value defaults to False, but it will change to True in a future version of Python, so you are invited to set it explicitly and update your code to use this new feature. (Contributed by Ezio Melotti in 13633.)

The strict argument of ~html.parser.HTMLParser is now deprecated. (Contributed by Ezio Melotti in 15114.)

http

~http.server.BaseHTTPRequestHandler.send_error now accepts an optional additional explain parameter which can be used to provide an extended error description, overriding the hardcoded default if there is one. This extended error description will be formatted using the ~http.server.BaseHTTPRequestHandler.error_message_format attribute and sent as the body of the error response. (Contributed by Karl Cow in 12921.)

The http.server command line interface <http-server-cli> now has a -b/--bind option that causes the server to listen on a specific address. (Contributed by Malte Swart in 17764.)

idlelib and IDLE

Since idlelib implements the IDLE shell and editor and is not intended for import by other programs, it gets improvements with every release. See Lib/idlelib/NEWS.txt for a cumulative list of changes since 3.3.0, as well as changes made in future 3.4.x releases. This file is also available from the IDLE Help --> About IDLE dialog.

importlib

The ~importlib.abc.InspectLoader ABC defines a new method, ~importlib.abc.InspectLoader.source_to_code that accepts source data and a path and returns a code object. The default implementation is equivalent to compile(data, path, 'exec', dont_inherit=True). (Contributed by Eric Snow and Brett Cannon in 15627.)

~importlib.abc.InspectLoader also now has a default implementation for the ~importlib.abc.InspectLoader.get_code method. However, it will normally be desirable to override the default implementation for performance reasons. (Contributed by Brett Cannon in 18072.)

The ~importlib.reload function has been moved from !imp to importlib as part of the !imp module deprecation. (Contributed by Berker Peksag in 18193.)

importlib.util now has a ~importlib.util.MAGIC_NUMBER attribute providing access to the bytecode version number. This replaces the !get_magic function in the deprecated !imp module. (Contributed by Brett Cannon in 18192.)

New importlib.util functions ~importlib.util.cache_from_source and ~importlib.util.source_from_cache replace the same-named functions in the deprecated !imp module. (Contributed by Brett Cannon in 18194.)

The importlib bootstrap .NamespaceLoader now conforms to the .InspectLoader ABC, which means that runpy and python -m can now be used with namespace packages. (Contributed by Brett Cannon in 18058.)

importlib.util has a new function ~importlib.util.decode_source that decodes source from bytes using universal newline processing. This is useful for implementing .InspectLoader.get_source methods.

importlib.machinery.ExtensionFileLoader now has a ~importlib.machinery.ExtensionFileLoader.get_filename method. This was inadvertently omitted in the original implementation. (Contributed by Eric Snow in 19152.)

inspect

The inspect module now offers a basic command line interface <inspect-module-cli> to quickly display source code and other information for modules, classes and functions. (Contributed by Claudiu Popa and Nick Coghlan in 18626.)

~inspect.unwrap makes it easy to unravel wrapper function chains created by functools.wraps (and any other API that sets the __wrapped__ attribute on a wrapper function). (Contributed by Daniel Urban, Aaron Iles and Nick Coghlan in 13266.)

As part of the implementation of the new enum module, the inspect module now has substantially better support for custom __dir__ methods and dynamic class attributes provided through metaclasses. (Contributed by Ethan Furman in 18929 and 19030.)

~inspect.getfullargspec and !getargspec now use the ~inspect.signature API. This allows them to support a much broader range of callables, including those with __signature__ attributes, those with metadata provided by argument clinic, functools.partial objects and more. Note that, unlike ~inspect.signature, these functions still ignore __wrapped__ attributes, and report the already bound first argument for bound methods, so it is still necessary to update your code to use ~inspect.signature directly if those features are desired. (Contributed by Yury Selivanov in 17481.)

~inspect.signature now supports duck types of CPython functions, which adds support for functions compiled with Cython. (Contributed by Stefan Behnel and Yury Selivanov in 17159.)

ipaddress

ipaddress was added to the standard library in Python 3.3 as a provisional API. With the release of Python 3.4, this qualification has been removed: ipaddress is now considered a stable API, covered by the normal standard library requirements to maintain backwards compatibility.

A new ~ipaddress.IPv4Address.is_global property is True if an address is globally routeable. (Contributed by Peter Moody in 17400.)

logging

The ~logging.handlers.TimedRotatingFileHandler has a new atTime parameter that can be used to specify the time of day when rollover should happen. (Contributed by Ronald Oussoren in 9556.)

~logging.handlers.SocketHandler and ~logging.handlers.DatagramHandler now support Unix domain sockets (by setting port to None). (Contributed by Vinay Sajip in commit ce46195b56a9.)

~logging.config.fileConfig now accepts a configparser.RawConfigParser subclass instance for the fname parameter. This facilitates using a configuration file when logging configuration is just a part of the overall application configuration, or where the application modifies the configuration before passing it to ~logging.config.fileConfig. (Contributed by Vinay Sajip in 16110.)

Logging configuration data received from a socket via the logging.config.listen function can now be validated before being processed by supplying a verification function as the argument to the new verify keyword argument. (Contributed by Vinay Sajip in 15452.)

marshal

The default marshal version has been bumped to 3. The code implementing the new version restores the Python2 behavior of recording only one copy of interned strings and preserving the interning on deserialization, and extends this "one copy" ability to any object type (including handling recursive references). This reduces both the size of .pyc files and the amount of memory a module occupies in memory when it is loaded from a .pyc (or .pyo) file. (Contributed by Kristján Valur Jónsson in 16475, with additional speedups by Antoine Pitrou in 19219.)

mmap

mmap objects are now weakly referenceable <mod-weakref>. (Contributed by Valerie Lambert in 4885.)

multiprocessing

On Unix two new start methods <multiprocessing-start-methods>, spawn and forkserver, have been added for starting processes using multiprocessing. These make the mixing of processes with threads more robust, and the spawn method matches the semantics that multiprocessing has always used on Windows. New function ~multiprocessing.get_all_start_methods reports all start methods available on the platform, ~multiprocessing.get_start_method reports the current start method, and ~multiprocessing.set_start_method sets the start method. (Contributed by Richard Oudkerk in 8713.)

multiprocessing also now has the concept of a context, which determines how child processes are created. New function ~multiprocessing.get_context returns a context that uses a specified start method. It has the same API as the multiprocessing module itself, so you can use it to create ~multiprocessing.pool.Pools and other objects that will operate within that context. This allows a framework and an application or different parts of the same application to use multiprocessing without interfering with each other. (Contributed by Richard Oudkerk in 18999.)

Except when using the old fork start method, child processes no longer inherit unneeded handles/file descriptors from their parents (part of 8713).

multiprocessing now relies on runpy (which implements the -m switch) to initialise __main__ appropriately in child processes when using the spawn or forkserver start methods. This resolves some edge cases where combining multiprocessing, the -m command line switch, and explicit relative imports could cause obscure failures in child processes. (Contributed by Nick Coghlan in 19946.)

operator

New function ~operator.length_hint provides an implementation of the specification for how the ~object.__length_hint__ special method should be used, as part of the 424 formal specification of this language feature. (Contributed by Armin Ronacher in 16148.)

There is now a pure-python version of the operator module available for reference and for use by alternate implementations of Python. (Contributed by Zachary Ware in 16694.)

os

There are new functions to get and set the inheritable flag <fd_inheritance> of a file descriptor (os.get_inheritable, os.set_inheritable) or a Windows handle (os.get_handle_inheritable, os.set_handle_inheritable).

New function ~os.cpu_count reports the number of CPUs available on the platform on which Python is running (or None if the count can't be determined). The multiprocessing.cpu_count function is now implemented in terms of this function). (Contributed by Trent Nelson, Yogesh Chaudhari, Victor Stinner, and Charles-François Natali in 17914.)

os.path.samestat is now available on the Windows platform (and the os.path.samefile implementation is now shared between Unix and Windows). (Contributed by Brian Curtin in 11939.)

os.path.ismount now recognizes volumes mounted below a drive root on Windows. (Contributed by Tim Golden in 9035.)

os.open supports two new flags on platforms that provide them, ~os.O_PATH (un-opened file descriptor), and ~os.O_TMPFILE (unnamed temporary file; as of 3.4.0 release available only on Linux systems with a kernel version of 3.11 or newer that have uapi headers). (Contributed by Christian Heimes in 18673 and Benjamin Peterson, respectively.)

pdb

pdb has been enhanced to handle generators, yield, and yield from in a more useful fashion. This is especially helpful when debugging asyncio based programs. (Contributed by Andrew Svetlov and Xavier de Gaye in 16596.)

The print command has been removed from pdb, restoring access to the Python print function from the pdb command line. Python2's pdb did not have a print command; instead, entering print executed the print statement. In Python3 print was mistakenly made an alias for the pdb p command. p, however, prints the repr of its argument, not the str like the Python2 print command did. Worse, the Python3 pdb print command shadowed the Python3 print function, making it inaccessible at the pdb prompt. (Contributed by Connor Osborn in 18764.)

pickle

pickle now supports (but does not use by default) a new pickle protocol, protocol 4. This new protocol addresses a number of issues that were present in previous protocols, such as the serialization of nested classes, very large strings and containers, and classes whose ~object.__new__ method takes keyword-only arguments. It also provides some efficiency improvements.

3154 -- Pickle protocol 4
PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti.

plistlib

plistlib now has an API that is similar to the standard pattern for stdlib serialization protocols, with new ~plistlib.load, ~plistlib.dump, ~plistlib.loads, and ~plistlib.dumps functions. (The older API is now deprecated.) In addition to the already supported XML plist format (~plistlib.FMT_XML), it also now supports the binary plist format (~plistlib.FMT_BINARY). (Contributed by Ronald Oussoren and others in 14455.)

poplib

Two new methods have been added to poplib: ~poplib.POP3.capa, which returns the list of capabilities advertised by the POP server, and ~poplib.POP3.stls, which switches a clear-text POP3 session into an encrypted POP3 session if the POP server supports it. (Contributed by Lorenzo Catucci in 4473.)

pprint

The pprint module's ~pprint.PrettyPrinter class and its ~pprint.pformat, and ~pprint.pprint functions have a new option, compact, that controls how the output is formatted. Currently setting compact to True means that sequences will be printed with as many sequence elements as will fit within width on each (indented) line. (Contributed by Serhiy Storchaka in 19132.)

Long strings are now wrapped using Python's normal line continuation syntax. (Contributed by Antoine Pitrou in 17150.)

pty

pty.spawn now returns the status value from os.waitpid on the child process, instead of None. (Contributed by Gregory P. Smith.)

pydoc

The pydoc module is now based directly on the inspect.signature introspection API, allowing it to provide signature information for a wider variety of callable objects. This change also means that __wrapped__ attributes are now taken into account when displaying help information. (Contributed by Larry Hastings in 19674.)

The pydoc module no longer displays the self parameter for already bound methods. Instead, it aims to always display the exact current signature of the supplied callable. (Contributed by Larry Hastings in 20710.)

In addition to the changes that have been made to pydoc directly, its handling of custom __dir__ methods and various descriptor behaviours has also been improved substantially by the underlying changes in the inspect module.

As the help builtin is based on pydoc, the above changes also affect the behaviour of help.

re

New ~re.fullmatch function and .Pattern.fullmatch method anchor the pattern at both ends of the string to match. This provides a way to be explicit about the goal of the match, which avoids a class of subtle bugs where $ characters get lost during code changes or the addition of alternatives to an existing regular expression. (Contributed by Matthew Barnett in 16203.)

The repr of regex objects <re-objects> now includes the pattern and the flags; the repr of match objects <match-objects> now includes the start, end, and the part of the string that matched. (Contributed by Hugo Lopes Tavares and Serhiy Storchaka in 13592 and 17087.)

resource

New ~resource.prlimit function, available on Linux platforms with a kernel version of 2.6.36 or later and glibc of 2.13 or later, provides the ability to query or set the resource limits for processes other than the one making the call. (Contributed by Christian Heimes in 16595.)

On Linux kernel version 2.6.36 or later, there are also some new Linux specific constants: ~resource.RLIMIT_MSGQUEUE, ~resource.RLIMIT_NICE, ~resource.RLIMIT_RTPRIO, ~resource.RLIMIT_RTTIME, and ~resource.RLIMIT_SIGPENDING. (Contributed by Christian Heimes in 19324.)

On FreeBSD version 9 and later, there some new FreeBSD specific constants: ~resource.RLIMIT_SBSIZE, ~resource.RLIMIT_SWAP, and ~resource.RLIMIT_NPTS. (Contributed by Claudiu Popa in 19343.)

select

~select.epoll objects now support the context management protocol. When used in a with statement, the ~select.epoll.close method will be called automatically at the end of the block. (Contributed by Serhiy Storchaka in 16488.)

~select.devpoll objects now have ~select.devpoll.fileno and ~select.devpoll.close methods, as well as a new attribute ~select.devpoll.closed. (Contributed by Victor Stinner in 18794.)

shelve

~shelve.Shelf instances may now be used in with statements, and will be automatically closed at the end of the !with block. (Contributed by Filip Gruszczyński in 13896.)

shutil

~shutil.copyfile now raises a specific ~shutil.Error subclass, ~shutil.SameFileError, when the source and destination are the same file, which allows an application to take appropriate action on this specific error. (Contributed by Atsuo Ishimoto and Hynek Schlawack in 1492704.)

smtpd

The !SMTPServer and !SMTPChannel classes now accept a map keyword argument which, if specified, is passed in to !asynchat.async_chat as its map argument. This allows an application to avoid affecting the global socket map. (Contributed by Vinay Sajip in 11959.)

smtplib

~smtplib.SMTPException is now a subclass of OSError, which allows both socket level errors and SMTP protocol level errors to be caught in one try/except statement by code that only cares whether or not an error occurred. (Contributed by Ned Jackson Lovely in 2118.)

socket

The socket module now supports the ~socket.CAN_BCM protocol on platforms that support it. (Contributed by Brian Thorne in 15359.)

Socket objects have new methods to get or set their inheritable flag <fd_inheritance>, ~socket.socket.get_inheritable and ~socket.socket.set_inheritable.

The socket.AF_* and socket.SOCK_* constants are now enumeration values using the new enum module. This allows meaningful names to be printed during debugging, instead of integer "magic numbers".

The ~socket.AF_LINK constant is now available on BSD and OSX.

~socket.inet_pton and ~socket.inet_ntop are now supported on Windows. (Contributed by Atsuo Ishimoto in 7171.)

sqlite3

A new boolean parameter to the ~sqlite3.connect function, uri, can be used to indicate that the database parameter is a uri (see the SQLite URI documentation). (Contributed by poq in 13773.)

ssl

~ssl.PROTOCOL_TLSv1_1 and ~ssl.PROTOCOL_TLSv1_2 (TLSv1.1 and TLSv1.2 support) have been added; support for these protocols is only available if Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orrù and Antoine Pitrou in 16692.)

New function ~ssl.create_default_context provides a standard way to obtain an ~ssl.SSLContext whose settings are intended to be a reasonable balance between compatibility and security. These settings are more stringent than the defaults provided by the ~ssl.SSLContext constructor, and may be adjusted in the future, without prior deprecation, if best-practice security requirements change. The new recommended best practice for using stdlib libraries that support SSL is to use ~ssl.create_default_context to obtain an ~ssl.SSLContext object, modify it if needed, and then pass it as the context argument of the appropriate stdlib API. (Contributed by Christian Heimes in 19689.)

~ssl.SSLContext method ~ssl.SSLContext.load_verify_locations accepts a new optional argument cadata, which can be used to provide PEM or DER encoded certificates directly via strings or bytes, respectively. (Contributed by Christian Heimes in 18138.)

New function ~ssl.get_default_verify_paths returns a named tuple of the paths and environment variables that the ~ssl.SSLContext.set_default_verify_paths method uses to set OpenSSL's default cafile and capath. This can be an aid in debugging default verification issues. (Contributed by Christian Heimes in 18143.)

~ssl.SSLContext has a new method, ~ssl.SSLContext.cert_store_stats, that reports the number of loaded X.509 certs, X.509 CA certs, and certificate revocation lists (crls), as well as a ~ssl.SSLContext.get_ca_certs method that returns a list of the loaded CA certificates. (Contributed by Christian Heimes in 18147.)

If OpenSSL 0.9.8 or later is available, ~ssl.SSLContext has a new attribute ~ssl.SSLContext.verify_flags that can be used to control the certificate verification process by setting it to some combination of the new constants ~ssl.VERIFY_DEFAULT, ~ssl.VERIFY_CRL_CHECK_LEAF, ~ssl.VERIFY_CRL_CHECK_CHAIN, or ~ssl.VERIFY_X509_STRICT. OpenSSL does not do any CRL verification by default. (Contributed by Christien Heimes in 8813.)

New ~ssl.SSLContext method ~ssl.SSLContext.load_default_certs loads a set of default "certificate authority" (CA) certificates from default locations, which vary according to the platform. It can be used to load both TLS web server authentication certificates (purpose=``~ssl.Purpose.SERVER_AUTH) for a client to use to verify a server, and certificates for a server to use in verifying client certificates (purpose=``~ssl.Purpose.CLIENT_AUTH). (Contributed by Christian Heimes in 19292.)

Two new windows-only functions, ~ssl.enum_certificates and ~ssl.enum_crls provide the ability to retrieve certificates, certificate information, and CRLs from the Windows cert store. (Contributed by Christian Heimes in 17134.)

Support for server-side SNI (Server Name Indication) using the new ssl.SSLContext.set_servername_callback method. (Contributed by Daniel Black in 8109.)

The dictionary returned by .SSLSocket.getpeercert contains additional X509v3 extension items: crlDistributionPoints, calIssuers, and OCSP URIs. (Contributed by Christian Heimes in 18379.)

stat

The stat module is now backed by a C implementation in !_stat. A C implementation is required as most of the values aren't standardized and are platform-dependent. (Contributed by Christian Heimes in 11016.)

The module supports new ~stat.ST_MODE flags, ~stat.S_IFDOOR, ~stat.S_IFPORT, and ~stat.S_IFWHT. (Contributed by Christian Hiemes in 11016.)

struct

New function ~struct.iter_unpack and a new struct.Struct.iter_unpack method on compiled formats provide streamed unpacking of a buffer containing repeated instances of a given format of data. (Contributed by Antoine Pitrou in 17804.)

subprocess

~subprocess.check_output now accepts an input argument that can be used to provide the contents of stdin for the command that is run. (Contributed by Zack Weinberg in 16624.)

~subprocess.getoutput and ~subprocess.getstatusoutput now work on Windows. This change was actually inadvertently made in 3.3.4. (Contributed by Tim Golden in 10197.)

sunau

The !getparams method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in 18901.)

!sunau.open now supports the context management protocol: when used in a with block, the close method of the returned object will be called automatically at the end of the block. (Contributed by Serhiy Storchaka in 18878.)

!AU_write.setsampwidth now supports 24 bit samples, thus adding support for writing 24 sample using the module. (Contributed by Serhiy Storchaka in 19261.)

The !writeframesraw and !writeframes methods now accept any bytes-like object. (Contributed by Serhiy Storchaka in 8311.)

sys

New function sys.getallocatedblocks returns the current number of blocks allocated by the interpreter. (In CPython with the default --with-pymalloc setting, this is allocations made through the PyObject_Malloc API.) This can be useful for tracking memory leaks, especially if automated via a test suite. (Contributed by Antoine Pitrou in 13390.)

When the Python interpreter starts in interactive mode <tut-interactive>, it checks for an ~sys.__interactivehook__ attribute on the sys module. If the attribute exists, its value is called with no arguments just before interactive mode is started. The check is made after the PYTHONSTARTUP file is read, so it can be set there. The site module sets it <rlcompleter-config> to a function that enables tab completion and history saving (in ~/.python-history) if the platform supports readline. If you do not want this (new) behavior, you can override it in PYTHONSTARTUP, sitecustomize, or usercustomize by deleting this attribute from sys (or setting it to some other callable). (Contributed by Éric Araujo and Antoine Pitrou in 5845.)

tarfile

The tarfile module now supports a simple tarfile-commandline when called as a script directly or via -m. This can be used to create and extract tarfile archives. (Contributed by Berker Peksag in 13477.)

textwrap

The ~textwrap.TextWrapper class has two new attributes/constructor arguments: ~textwrap.TextWrapper.max_lines, which limits the number of lines in the output, and ~textwrap.TextWrapper.placeholder, which is a string that will appear at the end of the output if it has been truncated because of max_lines. Building on these capabilities, a new convenience function ~textwrap.shorten collapses all of the whitespace in the input to single spaces and produces a single line of a given width that ends with the placeholder (by default, [...]). (Contributed by Antoine Pitrou and Serhiy Storchaka in 18585 and 18725.)

threading

The ~threading.Thread object representing the main thread can be obtained from the new ~threading.main_thread function. In normal conditions this will be the thread from which the Python interpreter was started. (Contributed by Andrew Svetlov in 18882.)

traceback

A new traceback.clear_frames function takes a traceback object and clears the local variables in all of the frames it references, reducing the amount of memory consumed. (Contributed by Andrew Kuchling in 1565525.)

types

A new ~types.DynamicClassAttribute descriptor provides a way to define an attribute that acts normally when looked up through an instance object, but which is routed to the class __getattr__ when looked up through the class. This allows one to have properties active on a class, and have virtual attributes on the class with the same name (see enum for an example). (Contributed by Ethan Furman in 19030.)

urllib

urllib.request now supports data: URLs via the ~urllib.request.DataHandler class. (Contributed by Mathias Panzenböck in 16423.)

The http method that will be used by a ~urllib.request.Request class can now be specified by setting a ~urllib.request.Request.method class attribute on the subclass. (Contributed by Jason R Coombs in 18978.)

~urllib.request.Request objects are now reusable: if the ~urllib.request.Request.full_url or ~urllib.request.Request.data attributes are modified, all relevant internal properties are updated. This means, for example, that it is now possible to use the same ~urllib.request.Request object in more than one .OpenerDirector.open call with different data arguments, or to modify a ~urllib.request.Request's url rather than recomputing it from scratch. There is also a new ~urllib.request.Request.remove_header method that can be used to remove headers from a ~urllib.request.Request. (Contributed by Alexey Kachayev in 16464, Daniel Wozniak in 17485, and Damien Brecht and Senthil Kumaran in 17272.)

~urllib.error.HTTPError objects now have a ~urllib.error.HTTPError.headers attribute that provides access to the HTTP response headers associated with the error. (Contributed by Berker Peksag in 15701.)

unittest

The ~unittest.TestCase class has a new method, ~unittest.TestCase.subTest, that produces a context manager whose with block becomes a "sub-test". This context manager allows a test method to dynamically generate subtests by, say, calling the subTest context manager inside a loop. A single test method can thereby produce an indefinite number of separately identified and separately counted tests, all of which will run even if one or more of them fail. For example:

class NumbersTest(unittest.TestCase):
    def test_even(self):
        for i in range(6):
            with self.subTest(i=i):
                self.assertEqual(i % 2, 0)

will result in six subtests, each identified in the unittest verbose output with a label consisting of the variable name i and a particular value for that variable (i=0, i=1, etc). See subtests for the full version of this example. (Contributed by Antoine Pitrou in 16997.)

unittest.main now accepts an iterable of test names for defaultTest, where previously it only accepted a single test name as a string. (Contributed by Jyrki Pulliainen in 15132.)

If ~unittest.SkipTest is raised during test discovery (that is, at the module level in the test file), it is now reported as a skip instead of an error. (Contributed by Zach Ware in 16935.)

~unittest.TestLoader.discover now sorts the discovered files to provide consistent test ordering. (Contributed by Martin Melin and Jeff Ramnani in 16709.)

~unittest.TestSuite now drops references to tests as soon as the test has been run, if the test is successful. On Python interpreters that do garbage collection, this allows the tests to be garbage collected if nothing else is holding a reference to the test. It is possible to override this behavior by creating a ~unittest.TestSuite subclass that defines a custom _removeTestAtIndex method. (Contributed by Tom Wardill, Matt McClure, and Andrew Svetlov in 11798.)

A new test assertion context-manager, ~unittest.TestCase.assertLogs, will ensure that a given block of code emits a log message using the logging module. By default the message can come from any logger and have a priority of INFO or higher, but both the logger name and an alternative minimum logging level may be specified. The object returned by the context manager can be queried for the ~logging.LogRecords and/or formatted messages that were logged. (Contributed by Antoine Pitrou in 18937.)

Test discovery now works with namespace packages (Contributed by Claudiu Popa in 17457.)

unittest.mock objects now inspect their specification signatures when matching calls, which means an argument can now be matched by either position or name, instead of only by position. (Contributed by Antoine Pitrou in 17015.)

~unittest.mock.mock_open objects now have readline and readlines methods. (Contributed by Toshio Kuratomi in 17467.)

venv

venv now includes activation scripts for the csh and fish shells. (Contributed by Andrew Svetlov in 15417.)

~venv.EnvBuilder and the ~venv.create convenience function take a new keyword argument with_pip, which defaults to False, that controls whether or not ~venv.EnvBuilder ensures that pip is installed in the virtual environment. (Contributed by Nick Coghlan in 19552 as part of the 453 implementation.)

wave

The ~wave.Wave_read.getparams method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in 17487.)

wave.open now supports the context management protocol. (Contributed by Claudiu Popa in 17616.)

wave can now write output to unseekable files <wave-write-objects>. (Contributed by David Jones, Guilherme Polo, and Serhiy Storchaka in 5202.)

The ~wave.Wave_write.writeframesraw and ~wave.Wave_write.writeframes methods now accept any bytes-like object. (Contributed by Serhiy Storchaka in 8311.)

weakref

New ~weakref.WeakMethod class simulates weak references to bound methods. (Contributed by Antoine Pitrou in 14631.)

New ~weakref.finalize class makes it possible to register a callback to be invoked when an object is garbage collected, without needing to carefully manage the lifecycle of the weak reference itself. (Contributed by Richard Oudkerk in 15528.)

The callback, if any, associated with a ~weakref.ref is now exposed via the ~weakref.ref.__callback__ attribute. (Contributed by Mark Dickinson in 17643.)

xml.etree

A new parser, ~xml.etree.ElementTree.XMLPullParser, allows a non-blocking applications to parse XML documents. An example can be seen at elementtree-pull-parsing. (Contributed by Antoine Pitrou in 17741.)

The xml.etree.ElementTree ~xml.etree.ElementTree.tostring and ~xml.etree.ElementTree.tostringlist functions, and the ~xml.etree.ElementTree.ElementTree ~xml.etree.ElementTree.ElementTree.write method, now have a short_empty_elements keyword-only parameter <keyword-only_parameter> providing control over whether elements with no content are written in abbreviated (<tag />) or expanded (<tag></tag>) form. (Contributed by Ariel Poliak and Serhiy Storchaka in 14377.)

zipfile

The ~zipfile.PyZipFile.writepy method of the ~zipfile.PyZipFile class has a new filterfunc option that can be used to control which directories and files are added to the archive. For example, this could be used to exclude test files from the archive. (Contributed by Christian Tismer in 19274.)

The allowZip64 parameter to ~zipfile.ZipFile and ~zipfile.PyZipFile is now True by default. (Contributed by William Mallard in 17201.)

CPython Implementation Changes

PEP 445: Customization of CPython Memory Allocators

445 adds new C level interfaces to customize memory allocation in the CPython interpreter.

445 -- Add new APIs to customize Python memory allocators
PEP written and implemented by Victor Stinner.

PEP 442: Safe Object Finalization

442 removes the current limitations and quirks of object finalization in CPython. With it, objects with ~object.__del__ methods, as well as generators with finally clauses, can be finalized when they are part of a reference cycle.

As part of this change, module globals are no longer forcibly set to None during interpreter shutdown in most cases, instead relying on the normal operation of the cyclic garbage collector. This avoids a whole class of interpreter-shutdown-time errors, usually involving __del__ methods, that have plagued Python since the cyclic GC was first introduced.

442 -- Safe object finalization
PEP written and implemented by Antoine Pitrou.

PEP 456: Secure and Interchangeable Hash Algorithm

456 follows up on earlier security fix work done on Python's hash algorithm to address certain DOS attacks to which public facing APIs backed by dictionary lookups may be subject. (See 14621 for the start of the current round of improvements.) The PEP unifies CPython's hash code to make it easier for a packager to substitute a different hash algorithm, and switches Python's default implementation to a SipHash implementation on platforms that have a 64 bit data type. Any performance differences in comparison with the older FNV algorithm are trivial.

The PEP adds additional fields to the sys.hash_info named tuple to describe the hash algorithm in use by the currently executing binary. Otherwise, the PEP does not alter any existing CPython APIs.

PEP 436: Argument Clinic

"Argument Clinic" (436) is now part of the CPython build process and can be used to simplify the process of defining and maintaining accurate signatures for builtins and standard library extension modules implemented in C.

Some standard library extension modules have been converted to use Argument Clinic in Python 3.4, and pydoc and inspect have been updated accordingly.

It is expected that signature metadata for programmatic introspection will be added to additional callables implemented in C as part of Python 3.4 maintenance releases.

[!NOTE] The Argument Clinic PEP is not fully up to date with the state of the implementation. This has been deemed acceptable by the release manager and core development team in this case, as Argument Clinic will not be made available as a public API for third party use in Python 3.4.

436 -- The Argument Clinic DSL
PEP written and implemented by Larry Hastings.

Other Build and C API Changes

Other Improvements

Significant Optimizations

Deprecated

This section covers various APIs and other features that have been deprecated in Python 3.4, and will be removed in Python 3.5 or later. In most (but not all) cases, using the deprecated APIs will produce a DeprecationWarning when the interpreter is run with deprecation warnings enabled (for example, by using -Wd).

Deprecations in the Python API

Deprecated Features

Removed

Operating Systems No Longer Supported

Support for the following operating systems has been removed from the source and build tools:

API and Feature Removals

The following obsolete and previously deprecated APIs and features have been removed:

Code Cleanups

Porting to Python 3.4

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in 'python' Command Behavior

Changes in the Python API

Changes in the C API

Changed in 3.4.3

PEP 476: Enabling certificate verification by default for stdlib http clients

http.client and modules which use it, such as urllib.request and xmlrpc.client, will now verify that the server presents a certificate which is signed by a CA in the platform trust store and whose hostname matches the hostname being requested by default, significantly improving security for many applications.

For applications which require the old previous behavior, they can pass an alternate context:

import urllib.request
import ssl

# This disables all verification
context = ssl._create_unverified_context()

# This allows using a specific certificate for the host, which doesn't need
# to be in the trust store
context = ssl.create_default_context(cafile="/path/to/file.crt")

urllib.request.urlopen("https://invalid-cert", context=context)

What's New In Python 3.5

Editors
Elvis Pranskevichus <elvis@magic.io>, Yury Selivanov <yury@magic.io>

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the Mercurial log when researching a change.

This article explains the new features in Python 3.5, compared to 3.4. Python 3.5 was released on September 13, 2015. See the changelog for a full list of changes.

478 - Python 3.5 Release Schedule

Summary -- Release highlights

New syntax features:

New library modules:

New built-in features:

CPython implementation improvements:

Significant improvements in the standard library:

Security improvements:

Windows improvements:

Please read on for a comprehensive list of user-facing changes, including many other smaller improvements, CPython optimizations, deprecations, and potential porting issues.

New Features

PEP 492 - Coroutines with async and await syntax

492 greatly improves support for asynchronous programming in Python by adding awaitable objects <awaitable>, coroutine functions <coroutine function>, asynchronous iteration <asynchronous iterable>, and asynchronous context managers <asynchronous context manager>.

Coroutine functions are declared using the new async def syntax:

>>> async def coro():
...     return 'spam'

Inside a coroutine function, the new await expression can be used to suspend coroutine execution until the result is available. Any object can be awaited, as long as it implements the awaitable protocol by defining the ~object.__await__ method.

PEP 492 also adds async for statement for convenient iteration over asynchronous iterables.

An example of a rudimentary HTTP client written using the new syntax:

import asyncio

async def http_get(domain):
    reader, writer = await asyncio.open_connection(domain, 80)

    writer.write(b'\r\n'.join([
        b'GET / HTTP/1.1',
        b'Host: %b' % domain.encode('latin-1'),
        b'Connection: close',
        b'', b''
    ]))

    async for line in reader:
        print('>>>', line)

    writer.close()

loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(http_get('example.com'))
finally:
    loop.close()

Similarly to asynchronous iteration, there is a new syntax for asynchronous context managers. The following script:

import asyncio

async def coro(name, lock):
    print('coro {}: waiting for lock'.format(name))
    async with lock:
        print('coro {}: holding the lock'.format(name))
        await asyncio.sleep(1)
        print('coro {}: releasing the lock'.format(name))

loop = asyncio.get_event_loop()
lock = asyncio.Lock()
coros = asyncio.gather(coro(1, lock), coro(2, lock))
try:
    loop.run_until_complete(coros)
finally:
    loop.close()

will output:

coro 2: waiting for lock
coro 2: holding the lock
coro 1: waiting for lock
coro 2: releasing the lock
coro 1: holding the lock
coro 1: releasing the lock

Note that both async for and async with can only be used inside a coroutine function declared with async def.

Coroutine functions are intended to be run inside a compatible event loop, such as the asyncio loop <asyncio-event-loop>.

[!NOTE]

3.5.2 Starting with CPython 3.5.2, __aiter__ can directly return asynchronous iterators <asynchronous iterator>. Returning an awaitable object will result in a PendingDeprecationWarning.

See more details in the async-iterators documentation section.

492 -- Coroutines with async and await syntax
PEP written and implemented by Yury Selivanov.

PEP 465 - A dedicated infix operator for matrix multiplication

465 adds the @ infix operator for matrix multiplication. Currently, no builtin Python types implement the new operator, however, it can be implemented by defining ~object.__matmul__, ~object.__rmatmul__, and ~object.__imatmul__ for regular, reflected, and in-place matrix multiplication. The semantics of these methods is similar to that of methods defining other infix arithmetic operators.

Matrix multiplication is a notably common operation in many fields of mathematics, science, engineering, and the addition of @ allows writing cleaner code:

S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r)

instead of:

S = dot((dot(H, beta) - r).T,
        dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r))

NumPy 1.10 has support for the new operator:

>>> import numpy

>>> x = numpy.ones(3)
>>> x
array([ 1., 1., 1.])

>>> m = numpy.eye(3)
>>> m
array([[ 1., 0., 0.],
       [ 0., 1., 0.],
       [ 0., 0., 1.]])

>>> x @ m
array([ 1., 1., 1.])

465 -- A dedicated infix operator for matrix multiplication
PEP written by Nathaniel J. Smith; implemented by Benjamin Peterson.

PEP 448 - Additional Unpacking Generalizations

448 extends the allowed uses of the * iterable unpacking operator and ** dictionary unpacking operator. It is now possible to use an arbitrary number of unpackings in function calls <calls>:

>>> print(*[1], *[2], 3, *[4, 5])
1 2 3 4 5

>>> def fn(a, b, c, d):
...     print(a, b, c, d)
...

>>> fn(**{'a': 1, 'c': 3}, **{'b': 2, 'd': 4})
1 2 3 4

Similarly, tuple, list, set, and dictionary displays allow multiple unpackings (see exprlists and dict):

>>> *range(4), 4
(0, 1, 2, 3, 4)

>>> [*range(4), 4]
[0, 1, 2, 3, 4]

>>> {*range(4), 4, *(5, 6, 7)}
{0, 1, 2, 3, 4, 5, 6, 7}

>>> {'x': 1, **{'y': 2}}
{'x': 1, 'y': 2}

448 -- Additional Unpacking Generalizations
PEP written by Joshua Landau; implemented by Neil Girdhar, Thomas Wouters, and Joshua Landau.

PEP 461 - percent formatting support for bytes and bytearray

461 adds support for the % interpolation operator <bytes-formatting> to bytes and bytearray.

While interpolation is usually thought of as a string operation, there are cases where interpolation on bytes or bytearrays makes sense, and the work needed to make up for this missing functionality detracts from the overall readability of the code. This issue is particularly important when dealing with wire format protocols, which are often a mixture of binary and ASCII compatible text.

Examples:

>>> b'Hello %b!' % b'World'
b'Hello World!'

>>> b'x=%i y=%f' % (1, 2.5)
b'x=1 y=2.500000'

Unicode is not allowed for %b, but it is accepted by %a (equivalent of repr(obj).encode('ascii', 'backslashreplace')):

>>> b'Hello %b!' % 'World'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: %b requires bytes, or an object that implements __bytes__, not 'str'

>>> b'price: %a' % '10€'
b"price: '10\\u20ac'"

Note that %s and %r conversion types, although supported, should only be used in codebases that need compatibility with Python 2.

461 -- Adding % formatting to bytes and bytearray
PEP written by Ethan Furman; implemented by Neil Schemenauer and Ethan Furman.

PEP 484 - Type Hints

Function annotation syntax has been a Python feature since version 3.0 (3107), however the semantics of annotations has been left undefined.

Experience has shown that the majority of function annotation uses were to provide type hints to function parameters and return values. It became evident that it would be beneficial for Python users, if the standard library included the base definitions and tools for type annotations.

484 introduces a provisional module <provisional API> to provide these standard definitions and tools, along with some conventions for situations where annotations are not available.

For example, here is a simple function whose argument and return type are declared in the annotations:

def greeting(name: str) -> str:
    return 'Hello ' + name

While these annotations are available at runtime through the usual ~object.__annotations__ attribute, no automatic type checking happens at runtime. Instead, it is assumed that a separate off-line type checker (e.g. mypy) will be used for on-demand source code analysis.

The type system supports unions, generic types, and a special type named ~typing.Any which is consistent with (i.e. assignable to and from) all types.

PEP 471 - os.scandir() function -- a better and faster directory iterator

471 adds a new directory iteration function, os.scandir, to the standard library. Additionally, os.walk is now implemented using scandir, which makes it 3 to 5 times faster on POSIX systems and 7 to 20 times faster on Windows systems. This is largely achieved by greatly reducing the number of calls to os.stat required to walk a directory tree.

Additionally, scandir returns an iterator, as opposed to returning a list of file names, which improves memory efficiency when iterating over very large directories.

The following example shows a simple use of os.scandir to display all the files (excluding directories) in the given path that don't start with '.'. The entry.is_file() <os.DirEntry.is_file> call will generally not make an additional system call:

for entry in os.scandir(path):
    if not entry.name.startswith('.') and entry.is_file():
        print(entry.name)

471 -- os.scandir() function -- a better and faster directory iterator
PEP written and implemented by Ben Hoyt with the help of Victor Stinner.

PEP 475: Retry system calls failing with EINTR

An errno.EINTR error code is returned whenever a system call, that is waiting for I/O, is interrupted by a signal. Previously, Python would raise InterruptedError in such cases. This meant that, when writing a Python application, the developer had two choices:

  1. Ignore the InterruptedError.
  2. Handle the InterruptedError and attempt to restart the interrupted system call at every call site.

The first option makes an application fail intermittently. The second option adds a large amount of boilerplate that makes the code nearly unreadable. Compare:

print("Hello World")

and:

while True:
    try:
        print("Hello World")
        break
    except InterruptedError:
        continue

475 implements automatic retry of system calls on EINTR. This removes the burden of dealing with EINTR or InterruptedError in user code in most situations and makes Python programs, including the standard library, more robust. Note that the system call is only retried if the signal handler does not raise an exception.

Below is a list of functions which are now retried when interrupted by a signal:

475 -- Retry system calls failing with EINTR
PEP and implementation written by Charles-François Natali and Victor Stinner, with the help of Antoine Pitrou (the French connection).

PEP 479: Change StopIteration handling inside generators

The interaction of generators and StopIteration in Python 3.4 and earlier was sometimes surprising, and could conceal obscure bugs. Previously, StopIteration raised accidentally inside a generator function was interpreted as the end of the iteration by the loop construct driving the generator.

479 changes the behavior of generators: when a StopIteration exception is raised inside a generator, it is replaced with a RuntimeError before it exits the generator frame. The main goal of this change is to ease debugging in the situation where an unguarded next call raises StopIteration and causes the iteration controlled by the generator to terminate silently. This is particularly pernicious in combination with the yield from construct.

This is a backwards incompatible change, so to enable the new behavior, a __future__ import is necessary:

>>> from __future__ import generator_stop

>>> def gen():
...     next(iter([]))
...     yield
...
>>> next(gen())
Traceback (most recent call last):
  File "<stdin>", line 2, in gen
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: generator raised StopIteration

Without a __future__ import, a PendingDeprecationWarning will be raised whenever a StopIteration exception is raised inside a generator.

479 -- Change StopIteration handling inside generators
PEP written by Chris Angelico and Guido van Rossum. Implemented by Chris Angelico, Yury Selivanov and Nick Coghlan.

PEP 485: A function for testing approximate equality

485 adds the math.isclose and cmath.isclose functions which tell whether two values are approximately equal or "close" to each other. Whether or not two values are considered close is determined according to given absolute and relative tolerances. Relative tolerance is the maximum allowed difference between isclose arguments, relative to the larger absolute value:

>>> import math
>>> a = 5.0
>>> b = 4.99998
>>> math.isclose(a, b, rel_tol=1e-5)
True
>>> math.isclose(a, b, rel_tol=1e-6)
False

It is also possible to compare two values using absolute tolerance, which must be a non-negative value:

>>> import math
>>> a = 5.0
>>> b = 4.99998
>>> math.isclose(a, b, abs_tol=0.00003)
True
>>> math.isclose(a, b, abs_tol=0.00001)
False

485 -- A function for testing approximate equality
PEP written by Christopher Barker; implemented by Chris Barker and Tal Einat.

PEP 486: Make the Python Launcher aware of virtual environments

486 makes the Windows launcher (see 397) aware of an active virtual environment. When the default interpreter would be used and the VIRTUAL_ENV environment variable is set, the interpreter in the virtual environment will be used.

486 -- Make the Python Launcher aware of virtual environments
PEP written and implemented by Paul Moore.

PEP 488: Elimination of PYO files

488 does away with the concept of .pyo files. This means that .pyc files represent both unoptimized and optimized bytecode. To prevent the need to constantly regenerate bytecode files, .pyc files now have an optional opt- tag in their name when the bytecode is optimized. This has the side-effect of no more bytecode file name clashes when running under either -O or -OO. Consequently, bytecode files generated from -O, and -OO may now exist simultaneously. importlib.util.cache_from_source has an updated API to help with this change.

488 -- Elimination of PYO files
PEP written and implemented by Brett Cannon.

PEP 489: Multi-phase extension module initialization

489 updates extension module initialization to take advantage of the two step module loading mechanism introduced by 451 in Python 3.4.

This change brings the import semantics of extension modules that opt-in to using the new mechanism much closer to those of Python source and bytecode modules, including the ability to use any valid identifier as a module name, rather than being restricted to ASCII.

489 -- Multi-phase extension module initialization
PEP written by Petr Viktorin, Stefan Behnel, and Nick Coghlan; implemented by Petr Viktorin.

Other Language Changes

Some smaller changes made to the core Python language are:

New Modules

typing

The new typing provisional <provisional API> module provides standard definitions and tools for function type annotations. See Type Hints <whatsnew-pep-484> for more information.

zipapp

The new zipapp module (specified in 441) provides an API and command line tool for creating executable Python Zip Applications, which were introduced in Python 2.6 in 1739468, but which were not well publicized, either at the time or since.

With the new module, bundling your application is as simple as putting all the files, including a __main__.py file, into a directory myapp and running:

$ python -m zipapp myapp
$ python myapp.pyz

The module implementation has been contributed by Paul Moore in 23491.

441 -- Improving Python ZIP Application Support

Improved Modules

argparse

The ~argparse.ArgumentParser class now allows disabling abbreviated usage <prefix-matching> of long options by setting allow_abbrev to False. (Contributed by Jonathan Paugh, Steven Bethard, paul j3 and Daniel Eriksson in 14910.)

asyncio

Since the asyncio module is provisional <provisional API>, all changes introduced in Python 3.5 have also been backported to Python 3.4.x.

Notable changes in the asyncio module since Python 3.4.0:

Updates in 3.5.1:

Updates in 3.5.2:

bz2

The BZ2Decompressor.decompress <bz2.BZ2Decompressor.decompress> method now accepts an optional max_length argument to limit the maximum size of decompressed data. (Contributed by Nikolaus Rath in 15955.)

cgi

The !FieldStorage class now supports the context manager protocol. (Contributed by Berker Peksag in 20289.)

cmath

A new function ~cmath.isclose provides a way to test for approximate equality. (Contributed by Chris Barker and Tal Einat in 24270.)

code

The InteractiveInterpreter.showtraceback() <code.InteractiveInterpreter.showtraceback> method now prints the full chained traceback, just like the interactive interpreter. (Contributed by Claudiu Popa in 17442.)

collections

The ~collections.OrderedDict class is now implemented in C, which makes it 4 to 100 times faster. (Contributed by Eric Snow in 16991.)

!OrderedDict.items, !OrderedDict.keys, and !OrderedDict.values views now support reversed iteration. (Contributed by Serhiy Storchaka in 19505.)

The ~collections.deque class now defines ~collections.deque.index, ~collections.deque.insert, and ~collections.deque.copy, and supports the + and * operators. This allows deques to be recognized as a ~collections.abc.MutableSequence and improves their substitutability for lists. (Contributed by Raymond Hettinger in 23704.)

Docstrings produced by ~collections.namedtuple can now be updated:

Point = namedtuple('Point', ['x', 'y'])
Point.__doc__ += ': Cartesian coordinate'
Point.x.__doc__ = 'abscissa'
Point.y.__doc__ = 'ordinate'

(Contributed by Berker Peksag in 24064.)

The ~collections.UserString class now implements the ~object.__getnewargs__, ~object.__rmod__, ~str.casefold, ~str.format_map, ~str.isprintable, and ~str.maketrans methods to match the corresponding methods of str. (Contributed by Joe Jevnik in 22189.)

collections.abc

The !Sequence.index method now accepts start and stop arguments to match the corresponding methods of tuple, list, etc. (Contributed by Devin Jeanpierre in 23086.)

A new ~collections.abc.Generator abstract base class. (Contributed by Stefan Behnel in 24018.)

New ~collections.abc.Awaitable, ~collections.abc.Coroutine, ~collections.abc.AsyncIterator, and ~collections.abc.AsyncIterable abstract base classes. (Contributed by Yury Selivanov in 24184.)

For earlier Python versions, a backport of the new ABCs is available in an external PyPI package <backports_abc>.

compileall

A new compileall option, -j {N}, allows running N workers simultaneously to perform parallel bytecode compilation. The ~compileall.compile_dir function has a corresponding workers parameter. (Contributed by Claudiu Popa in 16104.)

Another new option, -r, allows controlling the maximum recursion level for subdirectories. (Contributed by Claudiu Popa in 19628.)

The -q command line option can now be specified more than once, in which case all output, including errors, will be suppressed. The corresponding quiet parameter in ~compileall.compile_dir, ~compileall.compile_file, and ~compileall.compile_path can now accept an integer value indicating the level of output suppression. (Contributed by Thomas Kluyver in 21338.)

concurrent.futures

The Executor.map() <concurrent.futures.Executor.map> method now accepts a chunksize argument to allow batching of tasks to improve performance when ~concurrent.futures.ProcessPoolExecutor is used. (Contributed by Dan O'Reilly in 11271.)

The number of workers in the ~concurrent.futures.ThreadPoolExecutor constructor is optional now. The default value is 5 times the number of CPUs. (Contributed by Claudiu Popa in 21527.)

configparser

configparser now provides a way to customize the conversion of values by specifying a dictionary of converters in the ~configparser.ConfigParser constructor, or by defining them as methods in ConfigParser subclasses. Converters defined in a parser instance are inherited by its section proxies.

Example:

>>> import configparser
>>> conv = {}
>>> conv['list'] = lambda v: [e.strip() for e in v.split() if e.strip()]
>>> cfg = configparser.ConfigParser(converters=conv)
>>> cfg.read_string("""
... [s]
... list = a b c d e f g
... """)
>>> cfg.get('s', 'list')
'a b c d e f g'
>>> cfg.getlist('s', 'list')
['a', 'b', 'c', 'd', 'e', 'f', 'g']
>>> section = cfg['s']
>>> section.getlist('list')
['a', 'b', 'c', 'd', 'e', 'f', 'g']

(Contributed by Łukasz Langa in 18159.)

contextlib

The new ~contextlib.redirect_stderr context manager (similar to ~contextlib.redirect_stdout) makes it easier for utility scripts to handle inflexible APIs that write their output to sys.stderr and don't provide any options to redirect it:

>>> import contextlib, io, logging
>>> f = io.StringIO()
>>> with contextlib.redirect_stderr(f):
...     logging.warning('warning')
...
>>> f.getvalue()
'WARNING:root:warning\n'

(Contributed by Berker Peksag in 22389.)

csv

The ~csv.csvwriter.writerow method now supports arbitrary iterables, not just sequences. (Contributed by Serhiy Storchaka in 23171.)

curses

The new ~curses.update_lines_cols function updates the ~curses.LINES and ~curses.COLS module variables. This is useful for detecting manual screen resizing. (Contributed by Arnon Yaari in 4254.)

dbm

dumb.open <dbm.dumb.open> always creates a new database when the flag has the value "n". (Contributed by Claudiu Popa in 18039.)

difflib

The charset of HTML documents generated by HtmlDiff.make_file() <difflib.HtmlDiff.make_file> can now be customized by using a new charset keyword-only argument. The default charset of HTML document changed from "ISO-8859-1" to "utf-8". (Contributed by Berker Peksag in 2052.)

The ~difflib.diff_bytes function can now compare lists of byte strings. This fixes a regression from Python 2. (Contributed by Terry J. Reedy and Greg Ward in 17445.)

distutils

Both the build and build_ext commands now accept a -j option to enable parallel building of extension modules. (Contributed by Antoine Pitrou in 5309.)

The distutils module now supports xz compression, and can be enabled by passing xztar as an argument to bdist --format. (Contributed by Serhiy Storchaka in 16314.)

doctest

The ~doctest.DocTestSuite function returns an empty unittest.TestSuite if module contains no docstrings, instead of raising ValueError. (Contributed by Glenn Jones in 15916.)

email

A new policy option Policy.mangle_from_ <email.policy.Policy.mangle_from_> controls whether or not lines that start with "From " in email bodies are prefixed with a ">" character by generators. The default is True for ~email.policy.compat32 and False for all other policies. (Contributed by Milan Oberkirch in 20098.)

A new Message.get_content_disposition() <email.message.Message.get_content_disposition> method provides easy access to a canonical value for the Content-Disposition header. (Contributed by Abhilash Raj in 21083.)

A new policy option EmailPolicy.utf8 <email.policy.EmailPolicy.utf8> can be set to True to encode email headers using the UTF-8 charset instead of using encoded words. This allows Messages to be formatted according to 6532 and used with an SMTP server that supports the 6531 SMTPUTF8 extension. (Contributed by R. David Murray in 24211.)

The mime.text.MIMEText <email.mime.text.MIMEText> constructor now accepts a charset.Charset <email.charset.Charset> instance. (Contributed by Claude Paroz and Berker Peksag in 16324.)

enum

The ~enum.Enum callable has a new parameter start to specify the initial number of enum values if only names are provided:

>>> Animal = enum.Enum('Animal', 'cat dog', start=10)
>>> Animal.cat
<Animal.cat: 10>
>>> Animal.dog
<Animal.dog: 11>

(Contributed by Ethan Furman in 21706.)

faulthandler

The ~faulthandler.enable, ~faulthandler.register, ~faulthandler.dump_traceback and ~faulthandler.dump_traceback_later functions now accept file descriptors in addition to file-like objects. (Contributed by Wei Wu in 23566.)

functools

Most of the ~functools.lru_cache machinery is now implemented in C, making it significantly faster. (Contributed by Matt Joiner, Alexey Kachayev, and Serhiy Storchaka in 14373.)

glob

The ~glob.iglob and ~glob.glob functions now support recursive search in subdirectories, using the "**" pattern. (Contributed by Serhiy Storchaka in 13968.)

gzip

The mode argument of the ~gzip.GzipFile constructor now accepts "x" to request exclusive creation. (Contributed by Tim Heaney in 19222.)

heapq

Element comparison in ~heapq.merge can now be customized by passing a key function in a new optional key keyword argument, and a new optional reverse keyword argument can be used to reverse element comparison:

>>> import heapq
>>> a = ['9', '777', '55555']
>>> b = ['88', '6666']
>>> list(heapq.merge(a, b, key=len))
['9', '88', '777', '6666', '55555']
>>> list(heapq.merge(reversed(a), reversed(b), key=len, reverse=True))
['55555', '6666', '777', '88', '9']

(Contributed by Raymond Hettinger in 13742.)

http

A new HTTPStatus <http.HTTPStatus> enum that defines a set of HTTP status codes, reason phrases and long descriptions written in English. (Contributed by Demian Brecht in 21793.)

http.client

HTTPConnection.getresponse() <http.client.HTTPConnection.getresponse> now raises a ~http.client.RemoteDisconnected exception when a remote server connection is closed unexpectedly. Additionally, if a ConnectionError (of which RemoteDisconnected is a subclass) is raised, the client socket is now closed automatically, and will reconnect on the next request:

import http.client
conn = http.client.HTTPConnection('www.python.org')
for retries in range(3):
    try:
        conn.request('GET', '/')
        resp = conn.getresponse()
    except http.client.RemoteDisconnected:
        pass

(Contributed by Martin Panter in 3566.)

idlelib and IDLE

Since idlelib implements the IDLE shell and editor and is not intended for import by other programs, it gets improvements with every release. See Lib/idlelib/NEWS.txt for a cumulative list of changes since 3.4.0, as well as changes made in future 3.5.x releases. This file is also available from the IDLE Help --> About IDLE dialog.

imaplib

The ~imaplib.IMAP4 class now supports the context manager protocol. When used in a with statement, the IMAP4 LOGOUT command will be called automatically at the end of the block. (Contributed by Tarek Ziadé and Serhiy Storchaka in 4972.)

The imaplib module now supports 5161 (ENABLE Extension) and 6855 (UTF-8 Support) via the IMAP4.enable() <imaplib.IMAP4.enable> method. A new IMAP4.utf8_enabled <imaplib.IMAP4.utf8_enabled> attribute tracks whether or not 6855 support is enabled. (Contributed by Milan Oberkirch, R. David Murray, and Maciej Szulik in 21800.)

The imaplib module now automatically encodes non-ASCII string usernames and passwords using UTF-8, as recommended by the RFCs. (Contributed by Milan Oberkirch in 21800.)

imghdr

The !what function now recognizes the OpenEXR format (contributed by Martin Vignali and Claudiu Popa in 20295), and the WebP format (contributed by Fabrice Aneche and Claudiu Popa in 20197.)

importlib

The util.LazyLoader <importlib.util.LazyLoader> class allows for lazy loading of modules in applications where startup time is important. (Contributed by Brett Cannon in 17621.)

The abc.InspectLoader.source_to_code() <importlib.abc.InspectLoader.source_to_code> method is now a static method. This makes it easier to initialize a module object with code compiled from a string by running exec(code, module.__dict__). (Contributed by Brett Cannon in 21156.)

The new util.module_from_spec() <importlib.util.module_from_spec> function is now the preferred way to create a new module. As opposed to creating a types.ModuleType instance directly, this new function will set the various import-controlled attributes based on the passed-in spec object. (Contributed by Brett Cannon in 20383.)

inspect

Both the ~inspect.Signature and ~inspect.Parameter classes are now picklable and hashable. (Contributed by Yury Selivanov in 20726 and 20334.)

A new BoundArguments.apply_defaults() <inspect.BoundArguments.apply_defaults> method provides a way to set default values for missing arguments:

>>> def foo(a, b='ham', *args): pass
>>> ba = inspect.signature(foo).bind('spam')
>>> ba.apply_defaults()
>>> ba.arguments
OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())])

(Contributed by Yury Selivanov in 24190.)

A new class method Signature.from_callable() <inspect.Signature.from_callable> makes subclassing of ~inspect.Signature easier. (Contributed by Yury Selivanov and Eric Snow in 17373.)

The ~inspect.signature function now accepts a follow_wrapped optional keyword argument, which, when set to False, disables automatic following of __wrapped__ links. (Contributed by Yury Selivanov in 20691.)

A set of new functions to inspect coroutine functions <coroutine function> and coroutine objects <coroutine> has been added: ~inspect.iscoroutine, ~inspect.iscoroutinefunction, ~inspect.isawaitable, ~inspect.getcoroutinelocals, and ~inspect.getcoroutinestate. (Contributed by Yury Selivanov in 24017 and 24400.)

The ~inspect.stack, ~inspect.trace, ~inspect.getouterframes, and ~inspect.getinnerframes functions now return a list of named tuples. (Contributed by Daniel Shahaf in 16808.)

io

A new BufferedIOBase.readinto1() <io.BufferedIOBase.readinto1> method, that uses at most one call to the underlying raw stream's RawIOBase.read() <io.RawIOBase.read> or RawIOBase.readinto() <io.RawIOBase.readinto> methods. (Contributed by Nikolaus Rath in 20578.)

ipaddress

Both the ~ipaddress.IPv4Network and ~ipaddress.IPv6Network classes now accept an (address, netmask) tuple argument, so as to easily construct network objects from existing addresses:

>>> import ipaddress
>>> ipaddress.IPv4Network(('127.0.0.0', 8))
IPv4Network('127.0.0.0/8')
>>> ipaddress.IPv4Network(('127.0.0.0', '255.0.0.0'))
IPv4Network('127.0.0.0/8')

(Contributed by Peter Moody and Antoine Pitrou in 16531.)

A new ~ipaddress.IPv4Address.reverse_pointer attribute for the ~ipaddress.IPv4Address and ~ipaddress.IPv6Address classes returns the name of the reverse DNS PTR record:

>>> import ipaddress
>>> addr = ipaddress.IPv4Address('127.0.0.1')
>>> addr.reverse_pointer
'1.0.0.127.in-addr.arpa'
>>> addr6 = ipaddress.IPv6Address('::1')
>>> addr6.reverse_pointer
'1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'

(Contributed by Leon Weber in 20480.)

json

The json.tool command line interface now preserves the order of keys in JSON objects passed in input. The new --sort-keys option can be used to sort the keys alphabetically. (Contributed by Berker Peksag in 21650.)

JSON decoder now raises ~json.JSONDecodeError instead of ValueError to provide better context information about the error. (Contributed by Serhiy Storchaka in 19361.)

linecache

A new ~linecache.lazycache function can be used to capture information about a non-file-based module to permit getting its lines later via ~linecache.getline. This avoids doing I/O until a line is actually needed, without having to carry the module globals around indefinitely. (Contributed by Robert Collins in 17911.)

locale

A new ~locale.delocalize function can be used to convert a string into a normalized number string, taking the LC_NUMERIC settings into account:

>>> import locale
>>> locale.setlocale(locale.LC_NUMERIC, 'de_DE.UTF-8')
'de_DE.UTF-8'
>>> locale.delocalize('1.234,56')
'1234.56'
>>> locale.setlocale(locale.LC_NUMERIC, 'en_US.UTF-8')
'en_US.UTF-8'
>>> locale.delocalize('1,234.56')
'1234.56'

(Contributed by Cédric Krier in 13918.)

logging

All logging methods (~logging.Logger ~logging.Logger.log, ~logging.Logger.exception, ~logging.Logger.critical, ~logging.Logger.debug, etc.), now accept exception instances as an exc_info argument, in addition to boolean values and exception tuples:

>>> import logging
>>> try:
...     1/0
... except ZeroDivisionError as ex:
...     logging.error('exception', exc_info=ex)
ERROR:root:exception

(Contributed by Yury Selivanov in 20537.)

The handlers.HTTPHandler <logging.handlers.HTTPHandler> class now accepts an optional ssl.SSLContext instance to configure SSL settings used in an HTTP connection. (Contributed by Alex Gaynor in 22788.)

The handlers.QueueListener <logging.handlers.QueueListener> class now takes a respect_handler_level keyword argument which, if set to True, will pass messages to handlers taking handler levels into account. (Contributed by Vinay Sajip.)

lzma

The LZMADecompressor.decompress() <lzma.LZMADecompressor.decompress> method now accepts an optional max_length argument to limit the maximum size of decompressed data. (Contributed by Martin Panter in 15955.)

math

Two new constants have been added to the math module: ~math.inf and ~math.nan. (Contributed by Mark Dickinson in 23185.)

A new function ~math.isclose provides a way to test for approximate equality. (Contributed by Chris Barker and Tal Einat in 24270.)

A new ~math.gcd function has been added. The !fractions.gcd function is now deprecated. (Contributed by Mark Dickinson and Serhiy Storchaka in 22486.)

multiprocessing

sharedctypes.synchronized() <multiprocessing.sharedctypes.synchronized> objects now support the context manager protocol. (Contributed by Charles-François Natali in 21565.)

operator

~operator.attrgetter, ~operator.itemgetter, and ~operator.methodcaller objects now support pickling. (Contributed by Josh Rosenberg and Serhiy Storchaka in 22955.)

New ~operator.matmul and ~operator.imatmul functions to perform matrix multiplication. (Contributed by Benjamin Peterson in 21176.)

os

The new ~os.scandir function returning an iterator of ~os.DirEntry objects has been added. If possible, ~os.scandir extracts file attributes while scanning a directory, removing the need to perform subsequent system calls to determine file type or attributes, which may significantly improve performance. (Contributed by Ben Hoyt with the help of Victor Stinner in 22524.)

On Windows, a new stat_result.st_file_attributes <os.stat_result.st_file_attributes> attribute is now available. It corresponds to the dwFileAttributes member of the BY_HANDLE_FILE_INFORMATION structure returned by GetFileInformationByHandle(). (Contributed by Ben Hoyt in 21719.)

The ~os.urandom function now uses the getrandom() syscall on Linux 3.17 or newer, and getentropy() on OpenBSD 5.6 and newer, removing the need to use /dev/urandom and avoiding failures due to potential file descriptor exhaustion. (Contributed by Victor Stinner in 22181.)

New ~os.get_blocking and ~os.set_blocking functions allow getting and setting a file descriptor's blocking mode (~os.O_NONBLOCK.) (Contributed by Victor Stinner in 22054.)

The ~os.truncate and ~os.ftruncate functions are now supported on Windows. (Contributed by Steve Dower in 23668.)

There is a new os.path.commonpath function returning the longest common sub-path of each passed pathname. Unlike the os.path.commonprefix function, it always returns a valid path:

>>> os.path.commonprefix(['/usr/lib', '/usr/local/lib'])
'/usr/l'

>>> os.path.commonpath(['/usr/lib', '/usr/local/lib'])
'/usr'

(Contributed by Rafik Draoui and Serhiy Storchaka in 10395.)

pathlib

The new Path.samefile() <pathlib.Path.samefile> method can be used to check whether the path points to the same file as another path, which can be either another ~pathlib.Path object, or a string:

>>> import pathlib
>>> p1 = pathlib.Path('/etc/hosts')
>>> p2 = pathlib.Path('/etc/../etc/hosts')
>>> p1.samefile(p2)
True

(Contributed by Vajrasky Kok and Antoine Pitrou in 19775.)

The Path.mkdir() <pathlib.Path.mkdir> method now accepts a new optional exist_ok argument to match mkdir -p and os.makedirs functionality. (Contributed by Berker Peksag in 21539.)

There is a new Path.expanduser() <pathlib.Path.expanduser> method to expand ~ and ~user prefixes. (Contributed by Serhiy Storchaka and Claudiu Popa in 19776.)

A new Path.home() <pathlib.Path.home> class method can be used to get a ~pathlib.Path instance representing the user’s home directory. (Contributed by Victor Salgado and Mayank Tripathi in 19777.)

New Path.write_text() <pathlib.Path.write_text>, Path.read_text() <pathlib.Path.read_text>, Path.write_bytes() <pathlib.Path.write_bytes>, Path.read_bytes() <pathlib.Path.read_bytes> methods to simplify read/write operations on files.

The following code snippet will create or rewrite existing file ~/spam42:

>>> import pathlib
>>> p = pathlib.Path('~/spam42')
>>> p.expanduser().write_text('ham')
3

(Contributed by Christopher Welborn in 20218.)

pickle

Nested objects, such as unbound methods or nested classes, can now be pickled using pickle protocols <pickle-protocols> older than protocol version 4. Protocol version 4 already supports these cases. (Contributed by Serhiy Storchaka in 23611.)

poplib

A new POP3.utf8() <poplib.POP3.utf8> command enables 6856 (Internationalized Email) support, if a POP server supports it. (Contributed by Milan OberKirch in 21804.)

re

References and conditional references to groups with fixed length are now allowed in lookbehind assertions:

>>> import re
>>> pat = re.compile(r'(a|b).(?<=\1)c')
>>> pat.match('aac')
<_sre.SRE_Match object; span=(0, 3), match='aac'>
>>> pat.match('bbc')
<_sre.SRE_Match object; span=(0, 3), match='bbc'>

(Contributed by Serhiy Storchaka in 9179.)

The number of capturing groups in regular expressions is no longer limited to 100. (Contributed by Serhiy Storchaka in 22437.)

The ~re.sub and ~re.subn functions now replace unmatched groups with empty strings instead of raising an exception. (Contributed by Serhiy Storchaka in 1519638.)

The re.error <re.PatternError> exceptions have new attributes, ~re.PatternError.msg, ~re.PatternError.pattern, ~re.PatternError.pos, ~re.PatternError.lineno, and ~re.PatternError.colno, that provide better context information about the error:

>>> re.compile("""
...     (?x)
...     .++
... """)
Traceback (most recent call last):
   ...
sre_constants.error: multiple repeat at position 16 (line 3, column 7)

(Contributed by Serhiy Storchaka in 22578.)

readline

A new ~readline.append_history_file function can be used to append the specified number of trailing elements in history to the given file. (Contributed by Bruno Cauet in 22940.)

selectors

The new ~selectors.DevpollSelector supports efficient /dev/poll polling on Solaris. (Contributed by Giampaolo Rodola' in 18931.)

shutil

The ~shutil.move function now accepts a copy_function argument, allowing, for example, the ~shutil.copy function to be used instead of the default ~shutil.copy2 if there is a need to ignore file metadata when moving. (Contributed by Claudiu Popa in 19840.)

The ~shutil.make_archive function now supports the xztar format. (Contributed by Serhiy Storchaka in 5411.)

signal

On Windows, the ~signal.set_wakeup_fd function now also supports socket handles. (Contributed by Victor Stinner in 22018.)

Various SIG* constants in the signal module have been converted into Enums <enum>. This allows meaningful names to be printed during debugging, instead of integer "magic numbers". (Contributed by Giampaolo Rodola' in 21076.)

smtpd

Both the !SMTPServer and !SMTPChannel classes now accept a decode_data keyword argument to determine if the DATA portion of the SMTP transaction is decoded using the "utf-8" codec or is instead provided to the !SMTPServer.process_message method as a byte string. The default is True for backward compatibility reasons, but will change to False in Python 3.6. If decode_data is set to False, the process_message method must be prepared to accept keyword arguments. (Contributed by Maciej Szulik in 19662.)

The !SMTPServer class now advertises the 8BITMIME extension (6152) if decode_data has been set True. If the client specifies BODY=8BITMIME on the MAIL command, it is passed to !SMTPServer.process_message via the mail_options keyword. (Contributed by Milan Oberkirch and R. David Murray in 21795.)

The !SMTPServer class now also supports the SMTPUTF8 extension (6531: Internationalized Email). If the client specified SMTPUTF8 BODY=8BITMIME on the MAIL command, they are passed to !SMTPServer.process_message via the mail_options keyword. It is the responsibility of the process_message method to correctly handle the SMTPUTF8 data. (Contributed by Milan Oberkirch in 21725.)

It is now possible to provide, directly or via name resolution, IPv6 addresses in the !SMTPServer constructor, and have it successfully connect. (Contributed by Milan Oberkirch in 14758.)

smtplib

A new SMTP.auth() <smtplib.SMTP.auth> method provides a convenient way to implement custom authentication mechanisms. (Contributed by Milan Oberkirch in 15014.)

The SMTP.set_debuglevel() <smtplib.SMTP.set_debuglevel> method now accepts an additional debuglevel (2), which enables timestamps in debug messages. (Contributed by Gavin Chappell and Maciej Szulik in 16914.)

Both the SMTP.sendmail() <smtplib.SMTP.sendmail> and SMTP.send_message() <smtplib.SMTP.send_message> methods now support 6531 (SMTPUTF8). (Contributed by Milan Oberkirch and R. David Murray in 22027.)

sndhdr

The !what and !whathdr functions now return a ~collections.namedtuple. (Contributed by Claudiu Popa in 18615.)

socket

Functions with timeouts now use a monotonic clock, instead of a system clock. (Contributed by Victor Stinner in 22043.)

A new socket.sendfile() <socket.socket.sendfile> method allows sending a file over a socket by using the high-performance os.sendfile function on UNIX, resulting in uploads being from 2 to 3 times faster than when using plain socket.send() <socket.socket.send>. (Contributed by Giampaolo Rodola' in 17552.)

The socket.sendall() <socket.socket.sendall> method no longer resets the socket timeout every time bytes are received or sent. The socket timeout is now the maximum total duration to send all data. (Contributed by Victor Stinner in 23853.)

The backlog argument of the socket.listen() <socket.socket.listen> method is now optional. By default it is set to SOMAXCONN <socket.SOMAXCONN> or to 128, whichever is less. (Contributed by Charles-François Natali in 21455.)

ssl

Memory BIO Support

(Contributed by Geert Jansen in 21965.)

The new ~ssl.SSLObject class has been added to provide SSL protocol support for cases when the network I/O capabilities of ~ssl.SSLSocket are not necessary or are suboptimal. SSLObject represents an SSL protocol instance, but does not implement any network I/O methods, and instead provides a memory buffer interface. The new ~ssl.MemoryBIO class can be used to pass data between Python and an SSL protocol instance.

The memory BIO SSL support is primarily intended to be used in frameworks implementing asynchronous I/O for which ~ssl.SSLSocket's readiness model ("select/poll") is inefficient.

A new SSLContext.wrap_bio() <ssl.SSLContext.wrap_bio> method can be used to create a new SSLObject instance.

Application-Layer Protocol Negotiation Support

(Contributed by Benjamin Peterson in 20188.)

Where OpenSSL support is present, the ssl module now implements the Application-Layer Protocol Negotiation TLS extension as described in 7301.

The new SSLContext.set_alpn_protocols() <ssl.SSLContext.set_alpn_protocols> can be used to specify which protocols a socket should advertise during the TLS handshake.

The new SSLSocket.selected_alpn_protocol() <ssl.SSLSocket.selected_alpn_protocol> returns the protocol that was selected during the TLS handshake. The ~ssl.HAS_ALPN flag indicates whether ALPN support is present.

Other Changes

There is a new SSLSocket.version() <ssl.SSLSocket.version> method to query the actual protocol version in use. (Contributed by Antoine Pitrou in 20421.)

The ~ssl.SSLSocket class now implements a !SSLSocket.sendfile method. (Contributed by Giampaolo Rodola' in 17552.)

The !SSLSocket.send method now raises either the ssl.SSLWantReadError or ssl.SSLWantWriteError exception on a non-blocking socket if the operation would block. Previously, it would return 0. (Contributed by Nikolaus Rath in 20951.)

The ~ssl.cert_time_to_seconds function now interprets the input time as UTC and not as local time, per 5280. Additionally, the return value is always an int. (Contributed by Akira Li in 19940.)

New !SSLObject.shared_ciphers and SSLSocket.shared_ciphers() <ssl.SSLSocket.shared_ciphers> methods return the list of ciphers sent by the client during the handshake. (Contributed by Benjamin Peterson in 23186.)

The SSLSocket.do_handshake() <ssl.SSLSocket.do_handshake>, SSLSocket.read() <ssl.SSLSocket.read>, !SSLSocket.shutdown, and SSLSocket.write() <ssl.SSLSocket.write> methods of the ~ssl.SSLSocket class no longer reset the socket timeout every time bytes are received or sent. The socket timeout is now the maximum total duration of the method. (Contributed by Victor Stinner in 23853.)

The !match_hostname function now supports matching of IP addresses. (Contributed by Antoine Pitrou in 23239.)

sqlite3

The ~sqlite3.Row class now fully supports the sequence protocol, in particular reversed iteration and slice indexing. (Contributed by Claudiu Popa in 10203; by Lucas Sinclair, Jessica McKellar, and Serhiy Storchaka in 13583.)

subprocess

The new ~subprocess.run function has been added. It runs the specified command and returns a ~subprocess.CompletedProcess object, which describes a finished process. The new API is more consistent and is the recommended approach to invoking subprocesses in Python code that does not need to maintain compatibility with earlier Python versions. (Contributed by Thomas Kluyver in 23342.)

Examples:

>>> subprocess.run(["ls", "-l"])  # doesn't capture output
CompletedProcess(args=['ls', '-l'], returncode=0)

>>> subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
  ...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1

>>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE)
CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0,
stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n')

sys

A new !set_coroutine_wrapper function allows setting a global hook that will be called whenever a coroutine object <coroutine> is created by an async def function. A corresponding !get_coroutine_wrapper can be used to obtain a currently set wrapper. Both functions are provisional <provisional API>, and are intended for debugging purposes only. (Contributed by Yury Selivanov in 24017.)

A new ~sys.is_finalizing function can be used to check if the Python interpreter is shutting down <interpreter shutdown>. (Contributed by Antoine Pitrou in 22696.)

sysconfig

The name of the user scripts directory on Windows now includes the first two components of the Python version. (Contributed by Paul Moore in 23437.)

tarfile

The mode argument of the ~tarfile.open function now accepts "x" to request exclusive creation. (Contributed by Berker Peksag in 21717.)

The TarFile.extractall() <tarfile.TarFile.extractall> and TarFile.extract() <tarfile.TarFile.extract> methods now take a keyword argument numeric_owner. If set to True, the extracted files and directories will be owned by the numeric uid and gid from the tarfile. If set to False (the default, and the behavior in versions prior to 3.5), they will be owned by the named user and group in the tarfile. (Contributed by Michael Vogt and Eric Smith in 23193.)

The TarFile.list() <tarfile.TarFile.list> now accepts an optional members keyword argument that can be set to a subset of the list returned by TarFile.getmembers() <tarfile.TarFile.getmembers>. (Contributed by Serhiy Storchaka in 21549.)

threading

Both the Lock.acquire() <threading.Lock.acquire> and RLock.acquire() <threading.RLock.acquire> methods now use a monotonic clock for timeout management. (Contributed by Victor Stinner in 22043.)

time

The ~time.monotonic function is now always available. (Contributed by Victor Stinner in 22043.)

timeit

A new command line option -u or --unit={U} can be used to specify the time unit for the timer output. Supported options are usec, msec, or sec. (Contributed by Julian Gindi in 18983.)

The ~timeit.timeit function has a new globals parameter for specifying the namespace in which the code will be running. (Contributed by Ben Roberts in 2527.)

tkinter

The !tkinter._fix module used for setting up the Tcl/Tk environment on Windows has been replaced by a private function in the !_tkinter module which makes no permanent changes to environment variables. (Contributed by Zachary Ware in 20035.)

traceback

New ~traceback.walk_stack and ~traceback.walk_tb functions to conveniently traverse frame and traceback objects <traceback-objects>. (Contributed by Robert Collins in 17911.)

New lightweight classes: ~traceback.TracebackException, ~traceback.StackSummary, and ~traceback.FrameSummary. (Contributed by Robert Collins in 17911.)

Both the ~traceback.print_tb and ~traceback.print_stack functions now support negative values for the limit argument. (Contributed by Dmitry Kazakov in 22619.)

types

A new ~types.coroutine function to transform generator <generator iterator> and generator-like <collections.abc.Generator> objects into awaitables <awaitable>. (Contributed by Yury Selivanov in 24017.)

A new type called ~types.CoroutineType, which is used for coroutine objects created by async def functions. (Contributed by Yury Selivanov in 24400.)

unicodedata

The unicodedata module now uses data from Unicode 8.0.0.

unittest

The TestLoader.loadTestsFromModule() <unittest.TestLoader.loadTestsFromModule> method now accepts a keyword-only argument pattern which is passed to load_tests as the third argument. Found packages are now checked for load_tests regardless of whether their path matches pattern, because it is impossible for a package name to match the default pattern. (Contributed by Robert Collins and Barry A. Warsaw in 16662.)

Unittest discovery errors now are exposed in the TestLoader.errors <unittest.TestLoader.errors> attribute of the ~unittest.TestLoader instance. (Contributed by Robert Collins in 19746.)

A new command line option --locals to show local variables in tracebacks. (Contributed by Robert Collins in 22936.)

unittest.mock

The ~unittest.mock.Mock class has the following improvements:

The ~unittest.mock.MagicMock class now supports ~object.__truediv__, ~object.__divmod__ and ~object.__matmul__ operators. (Contributed by Johannes Baiter in 20968, and Håkan Lövdahl in 23581 and 23568.)

It is no longer necessary to explicitly pass create=True to the ~unittest.mock.patch function when patching builtin names. (Contributed by Kushal Das in 17660.)

urllib

A new request.HTTPPasswordMgrWithPriorAuth <urllib.request.HTTPPasswordMgrWithPriorAuth> class allows HTTP Basic Authentication credentials to be managed so as to eliminate unnecessary 401 response handling, or to unconditionally send credentials on the first request in order to communicate with servers that return a 404 response instead of a 401 if the Authorization header is not sent. (Contributed by Matej Cepl in 19494 and Akshit Khurana in 7159.)

A new quote_via argument for the parse.urlencode() <urllib.parse.urlencode> function provides a way to control the encoding of query parts if needed. (Contributed by Samwyse and Arnon Yaari in 13866.)

The request.urlopen() <urllib.request.urlopen> function accepts an ssl.SSLContext object as a context argument, which will be used for the HTTPS connection. (Contributed by Alex Gaynor in 22366.)

The parse.urljoin() <urllib.parse.urljoin> was updated to use the 3986 semantics for the resolution of relative URLs, rather than 1808 and 2396. (Contributed by Demian Brecht and Senthil Kumaran in 22118.)

wsgiref

The headers argument of the headers.Headers <wsgiref.headers.Headers> class constructor is now optional. (Contributed by Pablo Torres Navarrete and SilentGhost in 5800.)

xmlrpc

The client.ServerProxy <xmlrpc.client.ServerProxy> class now supports the context manager protocol. (Contributed by Claudiu Popa in 20627.)

The client.ServerProxy <xmlrpc.client.ServerProxy> constructor now accepts an optional ssl.SSLContext instance. (Contributed by Alex Gaynor in 22960.)

xml.sax

SAX parsers now support a character stream of the xmlreader.InputSource <xml.sax.xmlreader.InputSource> object. (Contributed by Serhiy Storchaka in 2175.)

~xml.sax.parseString now accepts a str instance. (Contributed by Serhiy Storchaka in 10590.)

zipfile

ZIP output can now be written to unseekable streams. (Contributed by Serhiy Storchaka in 23252.)

The mode argument of ZipFile.open() <zipfile.ZipFile.open> method now accepts "x" to request exclusive creation. (Contributed by Serhiy Storchaka in 21717.)

Other module-level changes

Many functions in the mmap, !ossaudiodev, socket, ssl, and codecs modules now accept writable bytes-like objects <bytes-like object>. (Contributed by Serhiy Storchaka in 23001.)

Optimizations

The os.walk function has been sped up by 3 to 5 times on POSIX systems, and by 7 to 20 times on Windows. This was done using the new os.scandir function, which exposes file information from the underlying readdir or FindFirstFile/FindNextFile system calls. (Contributed by Ben Hoyt with help from Victor Stinner in 23605.)

Construction of bytes(int) (filled by zero bytes) is faster and uses less memory for large objects. calloc() is used instead of malloc() to allocate memory for these objects. (Contributed by Victor Stinner in 21233.)

Some operations on ipaddress ~ipaddress.IPv4Network and ~ipaddress.IPv6Network have been massively sped up, such as ~ipaddress.IPv4Network.subnets, ~ipaddress.IPv4Network.supernet, ~ipaddress.summarize_address_range, ~ipaddress.collapse_addresses. The speed up can range from 3 to 15 times. (Contributed by Antoine Pitrou, Michel Albert, and Markus in 21486, 21487, 20826, 23266.)

Pickling of ipaddress objects was optimized to produce significantly smaller output. (Contributed by Serhiy Storchaka in 23133.)

Many operations on io.BytesIO are now 50% to 100% faster. (Contributed by Serhiy Storchaka in 15381 and David Wilson in 22003.)

The marshal.dumps function is now faster: 65--85% with versions 3 and 4, 20--25% with versions 0 to 2 on typical data, and up to 5 times in best cases. (Contributed by Serhiy Storchaka in 20416 and 23344.)

The UTF-32 encoder is now 3 to 7 times faster. (Contributed by Serhiy Storchaka in 15027.)

Regular expressions are now parsed up to 10% faster. (Contributed by Serhiy Storchaka in 19380.)

The json.dumps function was optimized to run with ensure_ascii=False as fast as with ensure_ascii=True. (Contributed by Naoki Inada in 23206.)

The PyObject_IsInstance and PyObject_IsSubclass functions have been sped up in the common case that the second argument has type as its metaclass. (Contributed Georg Brandl by in 22540.)

Method caching was slightly improved, yielding up to 5% performance improvement in some benchmarks. (Contributed by Antoine Pitrou in 22847.)

Objects from the random module now use 50% less memory on 64-bit builds. (Contributed by Serhiy Storchaka in 23488.)

The property getter calls are up to 25% faster. (Contributed by Joe Jevnik in 23910.)

Instantiation of fractions.Fraction is now up to 30% faster. (Contributed by Stefan Behnel in 22464.)

String methods ~str.find, ~str.rfind, ~str.split, ~str.partition and the in string operator are now significantly faster for searching 1-character substrings. (Contributed by Serhiy Storchaka in 23573.)

Build and C API Changes

New calloc functions were added:

(Contributed by Victor Stinner in 21233.)

New encoding/decoding helper functions:

(Contributed by Victor Stinner in 18395.)

A new PyCodec_NameReplaceErrors function to replace the unicode encode error with \N{...} escapes. (Contributed by Serhiy Storchaka in 19676.)

A new PyErr_FormatV function similar to PyErr_Format, but accepts a va_list argument. (Contributed by Antoine Pitrou in 18711.)

A new PyExc_RecursionError exception. (Contributed by Georg Brandl in 19235.)

New PyModule_FromDefAndSpec, PyModule_FromDefAndSpec2, and PyModule_ExecDef functions introduced by 489 -- multi-phase extension module initialization. (Contributed by Petr Viktorin in 24268.)

New PyNumber_MatrixMultiply and PyNumber_InPlaceMatrixMultiply functions to perform matrix multiplication. (Contributed by Benjamin Peterson in 21176. See also 465 for details.)

The PyTypeObject.tp_finalize slot is now part of the stable ABI.

Windows builds now require Microsoft Visual C++ 14.0, which is available as part of Visual Studio 2015.

Extension modules now include a platform information tag in their filename on some platforms (the tag is optional, and CPython will import extensions without it, although if the tag is present and mismatched, the extension won't be loaded):

Deprecated

New Keywords

async and await are not recommended to be used as variable, class, function or module names. Introduced by 492 in Python 3.5, they will become proper keywords in Python 3.7.

Deprecated Python Behavior

Raising the StopIteration exception inside a generator will now generate a silent PendingDeprecationWarning, which will become a non-silent deprecation warning in Python 3.6 and will trigger a RuntimeError in Python 3.7. See PEP 479: Change StopIteration handling inside generators <whatsnew-pep-479> for details.

Unsupported Operating Systems

Windows XP is no longer supported by Microsoft, thus, per PEP 11, CPython 3.5 is no longer officially supported on this OS.

Deprecated Python modules, functions and methods

The !formatter module has now graduated to full deprecation and is still slated for removal in Python 3.6.

The !asyncio.async function is deprecated in favor of ~asyncio.ensure_future.

The !smtpd module has in the past always decoded the DATA portion of email messages using the utf-8 codec. This can now be controlled by the new decode_data keyword to !SMTPServer. The default value is True, but this default is deprecated. Specify the decode_data keyword with an appropriate value to avoid the deprecation warning.

Directly assigning values to the ~http.cookies.Morsel.key, ~http.cookies.Morsel.value and ~http.cookies.Morsel.coded_value of http.cookies.Morsel objects is deprecated. Use the ~http.cookies.Morsel.set method instead. In addition, the undocumented LegalChars parameter of ~http.cookies.Morsel.set is deprecated, and is now ignored.

Passing a format string as keyword argument format_string to the ~string.Formatter.format method of the string.Formatter class has been deprecated. (Contributed by Serhiy Storchaka in 23671.)

The !platform.dist and !platform.linux_distribution functions are now deprecated. Linux distributions use too many different ways of describing themselves, so the functionality is left to a package. (Contributed by Vajrasky Kok and Berker Peksag in 1322.)

The previously undocumented from_function and from_builtin methods of inspect.Signature are deprecated. Use the new Signature.from_callable() <inspect.Signature.from_callable> method instead. (Contributed by Yury Selivanov in 24248.)

The !inspect.getargspec function is deprecated and scheduled to be removed in Python 3.6. (See 20438 for details.)

The inspect ~inspect.getfullargspec, ~inspect.getcallargs, and !formatargspec functions are deprecated in favor of the inspect.signature API. (Contributed by Yury Selivanov in 20438.)

~inspect.getargvalues and ~inspect.formatargvalues functions were inadvertently marked as deprecated with the release of Python 3.5.0.

Use of re.LOCALE flag with str patterns or re.ASCII is now deprecated. (Contributed by Serhiy Storchaka in 22407.)

Use of unrecognized special sequences consisting of '\' and an ASCII letter in regular expression patterns and replacement patterns now raises a deprecation warning and will be forbidden in Python 3.6. (Contributed by Serhiy Storchaka in 23622.)

The undocumented and unofficial use_load_tests default argument of the unittest.TestLoader.loadTestsFromModule method now is deprecated and ignored. (Contributed by Robert Collins and Barry A. Warsaw in 16662.)

Removed

API and Feature Removals

The following obsolete and previously deprecated APIs and features have been removed:

Porting to Python 3.5

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in Python behavior

Changes in the Python API

Changes in the C API

Notable changes in Python 3.5.4

New make regen-all build target

To simplify cross-compilation, and to ensure that CPython can reliably be compiled without requiring an existing version of Python to already be available, the autotools-based build system no longer attempts to implicitly recompile generated files based on file modification times.

Instead, a new make regen-all command has been added to force regeneration of these files when desired (e.g. after an initial version of Python has already been built based on the pregenerated versions).

More selective regeneration targets are also defined - see Makefile.pre.in for details.

(Contributed by Victor Stinner in 23404.)

3.5.4

Removal of make touch build target

The make touch build target previously used to request implicit regeneration of generated files by updating their modification times has been removed.

It has been replaced by the new make regen-all target.

(Contributed by Victor Stinner in 23404.)

3.5.4

What's New In Python 3.6

Editors
Elvis Pranskevichus <elvis@magic.io>, Yury Selivanov <yury@magic.io>

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the Mercurial log when researching a change.

This article explains the new features in Python 3.6, compared to 3.5. Python 3.6 was released on December 23, 2016. See the changelog for a full list of changes.

494 - Python 3.6 Release Schedule

Summary -- Release highlights

New syntax features:

New library modules:

CPython implementation improvements:

Significant improvements in the standard library:

Security improvements:

Windows improvements:

New Features

PEP 498: Formatted string literals

498 introduces a new kind of string literals: f-strings, or formatted string literals <f-strings>.

Formatted string literals are prefixed with 'f' and are similar to the format strings accepted by str.format. They contain replacement fields surrounded by curly braces. The replacement fields are expressions, which are evaluated at run time, and then formatted using the format protocol:

>>> name = "Fred"
>>> f"He said his name is {name}."
'He said his name is Fred.'
>>> width = 10
>>> precision = 4
>>> value = decimal.Decimal("12.34567")
>>> f"result: {value:{width}.{precision}}"  # nested fields
'result:      12.35'

498 -- Literal String Interpolation.
PEP written and implemented by Eric V. Smith.

Feature documentation <f-strings>.

PEP 526: Syntax for variable annotations

484 introduced the standard for type annotations of function parameters, a.k.a. type hints. This PEP adds syntax to Python for annotating the types of variables including class variables and instance variables:

primes: List[int] = []

captain: str  # Note: no initial value!

class Starship:
    stats: Dict[str, int] = {}

Just as for function annotations, the Python interpreter does not attach any particular meaning to variable annotations and only stores them in the __annotations__ attribute of a class or module.

In contrast to variable declarations in statically typed languages, the goal of annotation syntax is to provide an easy way to specify structured type metadata for third party tools and libraries via the abstract syntax tree and the __annotations__ attribute.

526 -- Syntax for variable annotations.
PEP written by Ryan Gonzalez, Philip House, Ivan Levkivskyi, Lisa Roach, and Guido van Rossum. Implemented by Ivan Levkivskyi.

Tools that use or will use the new syntax: mypy, pytype, PyCharm, etc.

PEP 515: Underscores in Numeric Literals

515 adds the ability to use underscores in numeric literals for improved readability. For example:

>>> 1_000_000_000_000_000
1000000000000000
>>> 0x_FF_FF_FF_FF
4294967295

Single underscores are allowed between digits and after any base specifier. Leading, trailing, or multiple underscores in a row are not allowed.

The string formatting <formatspec> language also now has support for the '_' option to signal the use of an underscore for a thousands separator for floating-point presentation types and for integer presentation type 'd'. For integer presentation types 'b', 'o', 'x', and 'X', underscores will be inserted every 4 digits:

>>> '{:_}'.format(1000000)
'1_000_000'
>>> '{:_x}'.format(0xFFFFFFFF)
'ffff_ffff'

515 -- Underscores in Numeric Literals
PEP written by Georg Brandl and Serhiy Storchaka.

PEP 525: Asynchronous Generators

492 introduced support for native coroutines and async / await syntax to Python 3.5. A notable limitation of the Python 3.5 implementation is that it was not possible to use await and yield in the same function body. In Python 3.6 this restriction has been lifted, making it possible to define asynchronous generators:

async def ticker(delay, to):
    """Yield numbers from 0 to *to* every *delay* seconds."""
    for i in range(to):
        yield i
        await asyncio.sleep(delay)

The new syntax allows for faster and more concise code.

525 -- Asynchronous Generators
PEP written and implemented by Yury Selivanov.

PEP 530: Asynchronous Comprehensions

530 adds support for using async for in list, set, dict comprehensions and generator expressions:

result = [i async for i in aiter() if i % 2]

Additionally, await expressions are supported in all kinds of comprehensions:

result = [await fun() for fun in funcs if await condition()]

530 -- Asynchronous Comprehensions
PEP written and implemented by Yury Selivanov.

PEP 487: Simpler customization of class creation

It is now possible to customize subclass creation without using a metaclass. The new __init_subclass__ classmethod will be called on the base class whenever a new subclass is created:

class PluginBase:
    subclasses = []

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.subclasses.append(cls)

class Plugin1(PluginBase):
    pass

class Plugin2(PluginBase):
    pass

In order to allow zero-argument super calls to work correctly from ~object.__init_subclass__ implementations, custom metaclasses must ensure that the new __classcell__ namespace entry is propagated to type.__new__ (as described in class-object-creation).

487 -- Simpler customization of class creation
PEP written and implemented by Martin Teichmann.

Feature documentation <class-customization>

PEP 487: Descriptor Protocol Enhancements

487 extends the descriptor protocol to include the new optional ~object.__set_name__ method. Whenever a new class is defined, the new method will be called on all descriptors included in the definition, providing them with a reference to the class being defined and the name given to the descriptor within the class namespace. In other words, instances of descriptors can now know the attribute name of the descriptor in the owner class:

class IntField:
    def __get__(self, instance, owner):
        return instance.__dict__[self.name]

    def __set__(self, instance, value):
        if not isinstance(value, int):
            raise ValueError(f'expecting integer in {self.name}')
        instance.__dict__[self.name] = value

    # this is the new initializer:
    def __set_name__(self, owner, name):
        self.name = name

class Model:
    int_field = IntField()

487 -- Simpler customization of class creation
PEP written and implemented by Martin Teichmann.

Feature documentation <descriptors>

PEP 519: Adding a file system path protocol

File system paths have historically been represented as str or bytes objects. This has led to people who write code which operate on file system paths to assume that such objects are only one of those two types (an int representing a file descriptor does not count as that is not a file path). Unfortunately that assumption prevents alternative object representations of file system paths like pathlib from working with pre-existing code, including Python's standard library.

To fix this situation, a new interface represented by os.PathLike has been defined. By implementing the ~os.PathLike.__fspath__ method, an object signals that it represents a path. An object can then provide a low-level representation of a file system path as a str or bytes object. This means an object is considered path-like <path-like object> if it implements os.PathLike or is a str or bytes object which represents a file system path. Code can use os.fspath, os.fsdecode, or os.fsencode to explicitly get a str and/or bytes representation of a path-like object.

The built-in open function has been updated to accept os.PathLike objects, as have all relevant functions in the os and os.path modules, and most other functions and classes in the standard library. The os.DirEntry class and relevant classes in pathlib have also been updated to implement os.PathLike.

The hope is that updating the fundamental functions for operating on file system paths will lead to third-party code to implicitly support all path-like objects <path-like object> without any code changes, or at least very minimal ones (e.g. calling os.fspath at the beginning of code before operating on a path-like object).

Here are some examples of how the new interface allows for pathlib.Path to be used more easily and transparently with pre-existing code:

>>> import pathlib
>>> with open(pathlib.Path("README")) as f:
...     contents = f.read()
...
>>> import os.path
>>> os.path.splitext(pathlib.Path("some_file.txt"))
('some_file', '.txt')
>>> os.path.join("/a/b", pathlib.Path("c"))
'/a/b/c'
>>> import os
>>> os.fspath(pathlib.Path("some_file.txt"))
'some_file.txt'

(Implemented by Brett Cannon, Ethan Furman, Dusty Phillips, and Jelle Zijlstra.)

519 -- Adding a file system path protocol
PEP written by Brett Cannon and Koos Zevenhoven.

PEP 495: Local Time Disambiguation

In most world locations, there have been and will be times when local clocks are moved back. In those times, intervals are introduced in which local clocks show the same time twice in the same day. In these situations, the information displayed on a local clock (or stored in a Python datetime instance) is insufficient to identify a particular moment in time.

495 adds the new fold attribute to instances of datetime.datetime and datetime.time classes to differentiate between two moments in time for which local times are the same:

>>> u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc)
>>> for i in range(4):
...     u = u0 + i*HOUR
...     t = u.astimezone(Eastern)
...     print(u.time(), 'UTC =', t.time(), t.tzname(), t.fold)
...
04:00:00 UTC = 00:00:00 EDT 0
05:00:00 UTC = 01:00:00 EDT 0
06:00:00 UTC = 01:00:00 EST 1
07:00:00 UTC = 02:00:00 EST 0

The values of the fold <datetime.datetime.fold> attribute have the value 0 for all instances except those that represent the second (chronologically) moment in time in an ambiguous case.

495 -- Local Time Disambiguation
PEP written by Alexander Belopolsky and Tim Peters, implementation by Alexander Belopolsky.

PEP 529: Change Windows filesystem encoding to UTF-8

Representing filesystem paths is best performed with str (Unicode) rather than bytes. However, there are some situations where using bytes is sufficient and correct.

Prior to Python 3.6, data loss could result when using bytes paths on Windows. With this change, using bytes to represent paths is now supported on Windows, provided those bytes are encoded with the encoding returned by sys.getfilesystemencoding, which now defaults to 'utf-8'.

Applications that do not use str to represent paths should use os.fsencode and os.fsdecode to ensure their bytes are correctly encoded. To revert to the previous behaviour, set PYTHONLEGACYWINDOWSFSENCODING or call !sys._enablelegacywindowsfsencoding.

See 529 for more information and discussion of code modifications that may be required.

PEP 528: Change Windows console encoding to UTF-8

The default console on Windows will now accept all Unicode characters and provide correctly read str objects to Python code. sys.stdin, sys.stdout and sys.stderr now default to utf-8 encoding.

This change only applies when using an interactive console, and not when redirecting files or pipes. To revert to the previous behaviour for interactive console use, set PYTHONLEGACYWINDOWSSTDIO.

528 -- Change Windows console encoding to UTF-8
PEP written and implemented by Steve Dower.

PEP 520: Preserving Class Attribute Definition Order

Attributes in a class definition body have a natural ordering: the same order in which the names appear in the source. This order is now preserved in the new class's ~type.__dict__ attribute.

Also, the effective default class execution namespace (returned from type.__prepare__() <prepare>) is now an insertion-order-preserving mapping.

520 -- Preserving Class Attribute Definition Order
PEP written and implemented by Eric Snow.

PEP 468: Preserving Keyword Argument Order

**kwargs in a function signature is now guaranteed to be an insertion-order-preserving mapping.

468 -- Preserving Keyword Argument Order
PEP written and implemented by Eric Snow.

New dict <typesmapping> implementation

The dict <typesmapping> type now uses a "compact" representation based on a proposal by Raymond Hettinger which was first implemented by PyPy. The memory usage of the new dict is between 20% and 25% smaller compared to Python 3.5.

The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon (this may change in the future, but it is desired to have this new dict implementation in the language for a few releases before changing the language spec to mandate order-preserving semantics for all current and future Python implementations; this also helps preserve backwards-compatibility with older versions of the language where random iteration order is still in effect, e.g. Python 3.5).

(Contributed by INADA Naoki in 27350. Idea originally suggested by Raymond Hettinger.)

PEP 523: Adding a frame evaluation API to CPython

While Python provides extensive support to customize how code executes, one place it has not done so is in the evaluation of frame objects. If you wanted some way to intercept frame evaluation in Python there really wasn't any way without directly manipulating function pointers for defined functions.

523 changes this by providing an API to make frame evaluation pluggable at the C level. This will allow for tools such as debuggers and JITs to intercept frame evaluation before the execution of Python code begins. This enables the use of alternative evaluation implementations for Python code, tracking frame evaluation, etc.

This API is not part of the limited C API and is marked as private to signal that usage of this API is expected to be limited and only applicable to very select, low-level use-cases. Semantics of the API will change with Python as necessary.

523 -- Adding a frame evaluation API to CPython
PEP written by Brett Cannon and Dino Viehland.

PYTHONMALLOC environment variable

The new PYTHONMALLOC environment variable allows setting the Python memory allocators and installing debug hooks.

It is now possible to install debug hooks on Python memory allocators on Python compiled in release mode using PYTHONMALLOC=debug. Effects of debug hooks:

Checking if the GIL is held is also a new feature of Python 3.6.

See the PyMem_SetupDebugHooks function for debug hooks on Python memory allocators.

It is now also possible to force the usage of the malloc allocator of the C library for all Python memory allocations using PYTHONMALLOC=malloc. This is helpful when using external memory debuggers like Valgrind on a Python compiled in release mode.

On error, the debug hooks on Python memory allocators now use the tracemalloc module to get the traceback where a memory block was allocated.

Example of fatal error on buffer overflow using python3.6 -X tracemalloc=5 (store 5 frames in traces):

Debug memory block at address p=0x7fbcd41666f8: API 'o'
    4 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x02 *** OUCH
        at tail+1: 0xfb
        at tail+2: 0xfb
        at tail+3: 0xfb
        at tail+4: 0xfb
        at tail+5: 0xfb
        at tail+6: 0xfb
        at tail+7: 0xfb
    The block was made by call #1233329 to debug malloc/realloc.
    Data at p: 1a 2b 30 00

Memory block allocated at (most recent call first):
  File "test/test_bytes.py", line 323
  File "unittest/case.py", line 600
  File "unittest/case.py", line 648
  File "unittest/suite.py", line 122
  File "unittest/suite.py", line 84

Fatal Python error: bad trailing pad byte

Current thread 0x00007fbcdbd32700 (most recent call first):
  File "test/test_bytes.py", line 323 in test_hex
  File "unittest/case.py", line 600 in run
  File "unittest/case.py", line 648 in __call__
  File "unittest/suite.py", line 122 in run
  File "unittest/suite.py", line 84 in __call__
  File "unittest/suite.py", line 122 in run
  File "unittest/suite.py", line 84 in __call__
  ...

(Contributed by Victor Stinner in 26516 and 26564.)

DTrace and SystemTap probing support

Python can now be built --with-dtrace which enables static markers for the following events in the interpreter:

This can be used to instrument running interpreters in production, without the need to recompile specific debug builds <debug-build> or providing application-specific profiling/debugging code.

More details in instrumentation.

The current implementation is tested on Linux and macOS. Additional markers may be added in the future.

(Contributed by Łukasz Langa in 21590, based on patches by Jesús Cea Avión, David Malcolm, and Nikhil Benesch.)

Other Language Changes

Some smaller changes made to the core Python language are:

New Modules

secrets

The main purpose of the new secrets module is to provide an obvious way to reliably generate cryptographically strong pseudo-random values suitable for managing secrets, such as account authentication, tokens, and similar.

[!WARNING] Note that the pseudo-random generators in the random module should NOT be used for security purposes. Use secrets on Python 3.6+ and os.urandom on Python 3.5 and earlier.

506 -- Adding A Secrets Module To The Standard Library
PEP written and implemented by Steven D'Aprano.

Improved Modules

array

Exhausted iterators of array.array will now stay exhausted even if the iterated array is extended. This is consistent with the behavior of other mutable sequences.

Contributed by Serhiy Storchaka in 26492.

ast

The new ast.Constant AST node has been added. It can be used by external AST optimizers for the purposes of constant folding.

Contributed by Victor Stinner in 26146.

asyncio

Starting with Python 3.6 the asyncio module is no longer provisional and its API is considered stable.

Notable changes in the asyncio module since Python 3.5.0 (all backported to 3.5.x due to the provisional status):

binascii

The ~binascii.b2a_base64 function now accepts an optional newline keyword argument to control whether the newline character is appended to the return value. (Contributed by Victor Stinner in 25357.)

cmath

The new cmath.tau (τ) constant has been added. (Contributed by Lisa Roach in 12345, see 628 for details.)

New constants: cmath.inf and cmath.nan to match math.inf and math.nan, and also cmath.infj and cmath.nanj to match the format used by complex repr. (Contributed by Mark Dickinson in 23229.)

collections

The new ~collections.abc.Collection abstract base class has been added to represent sized iterable container classes. (Contributed by Ivan Levkivskyi, docs by Neil Girdhar in 27598.)

The new ~collections.abc.Reversible abstract base class represents iterable classes that also provide the ~object.__reversed__ method. (Contributed by Ivan Levkivskyi in 25987.)

The new ~collections.abc.AsyncGenerator abstract base class represents asynchronous generators. (Contributed by Yury Selivanov in 28720.)

The ~collections.namedtuple function now accepts an optional keyword argument module, which, when specified, is used for the ~type.__module__ attribute of the returned named tuple class. (Contributed by Raymond Hettinger in 17941.)

The verbose and rename arguments for ~collections.namedtuple are now keyword-only. (Contributed by Raymond Hettinger in 25628.)

Recursive collections.deque instances can now be pickled. (Contributed by Serhiy Storchaka in 26482.)

concurrent.futures

The ThreadPoolExecutor <concurrent.futures.ThreadPoolExecutor> class constructor now accepts an optional thread_name_prefix argument to make it possible to customize the names of the threads created by the pool. (Contributed by Gregory P. Smith in 27664.)

contextlib

The contextlib.AbstractContextManager class has been added to provide an abstract base class for context managers. It provides a sensible default implementation for __enter__() which returns self and leaves __exit__() an abstract method. A matching class has been added to the typing module as typing.ContextManager. (Contributed by Brett Cannon in 25609.)

datetime

The ~datetime.datetime and ~datetime.time classes have the new ~datetime.time.fold attribute used to disambiguate local time when necessary. Many functions in the datetime have been updated to support local time disambiguation. See Local Time Disambiguation <whatsnew36-pep495> section for more information. (Contributed by Alexander Belopolsky in 24773.)

The datetime.strftime() <datetime.datetime.strftime> and date.strftime() <datetime.date.strftime> methods now support ISO 8601 date directives %G, %u and %V. (Contributed by Ashley Anderson in 12006.)

The datetime.isoformat() <datetime.datetime.isoformat> function now accepts an optional timespec argument that specifies the number of additional components of the time value to include. (Contributed by Alessandro Cucci and Alexander Belopolsky in 19475.)

The datetime.combine() <datetime.datetime.combine> now accepts an optional tzinfo argument. (Contributed by Alexander Belopolsky in 27661.)

decimal

New Decimal.as_integer_ratio() <decimal.Decimal.as_integer_ratio> method that returns a pair (n, d) of integers that represent the given ~decimal.Decimal instance as a fraction, in lowest terms and with a positive denominator:

>>> Decimal('-3.14').as_integer_ratio()
(-157, 50)

(Contributed by Stefan Krah amd Mark Dickinson in 25928.)

distutils

The default_format attribute has been removed from distutils.command.sdist.sdist and the formats attribute defaults to ['gztar']. Although not anticipated, any code relying on the presence of default_format may need to be adapted. See 27819 for more details.

email

The new email API, enabled via the policy keyword to various constructors, is no longer provisional. The email documentation has been reorganized and rewritten to focus on the new API, while retaining the old documentation for the legacy API. (Contributed by R. David Murray in 24277.)

The email.mime classes now all accept an optional policy keyword. (Contributed by Berker Peksag in 27331.)

The ~email.generator.DecodedGenerator now supports the policy keyword.

There is a new ~email.policy attribute, ~email.policy.Policy.message_factory, that controls what class is used by default when the parser creates new message objects. For the email.policy.compat32 policy this is ~email.message.Message, for the new policies it is ~email.message.EmailMessage. (Contributed by R. David Murray in 20476.)

encodings

On Windows, added the 'oem' encoding to use CP_OEMCP, and the 'ansi' alias for the existing 'mbcs' encoding, which uses the CP_ACP code page. (Contributed by Steve Dower in 27959.)

enum

Two new enumeration base classes have been added to the enum module: ~enum.Flag and ~enum.IntFlag. Both are used to define constants that can be combined using the bitwise operators. (Contributed by Ethan Furman in 23591.)

Many standard library modules have been updated to use the ~enum.IntFlag class for their constants.

The new enum.auto value can be used to assign values to enum members automatically:

>>> from enum import Enum, auto
>>> class Color(Enum):
...     red = auto()
...     blue = auto()
...     green = auto()
...
>>> list(Color)
[<Color.red: 1>, <Color.blue: 2>, <Color.green: 3>]

faulthandler

On Windows, the faulthandler module now installs a handler for Windows exceptions: see faulthandler.enable. (Contributed by Victor Stinner in 23848.)

fileinput

~fileinput.hook_encoded now supports the errors argument. (Contributed by Joseph Hackman in 25788.)

hashlib

hashlib supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. (Contributed by Christian Heimes in 26470.)

BLAKE2 hash functions were added to the module. ~hashlib.blake2b and ~hashlib.blake2s are always available and support the full feature set of BLAKE2. (Contributed by Christian Heimes in 26798 based on code by Dmitry Chestnykh and Samuel Neves. Documentation written by Dmitry Chestnykh.)

The SHA-3 hash functions ~hashlib.sha3_224, ~hashlib.sha3_256, ~hashlib.sha3_384, ~hashlib.sha3_512, and SHAKE hash functions ~hashlib.shake_128 and ~hashlib.shake_256 were added. (Contributed by Christian Heimes in 16113. Keccak Code Package by Guido Bertoni, Joan Daemen, Michaël Peeters, Gilles Van Assche, and Ronny Van Keer.)

The password-based key derivation function ~hashlib.scrypt is now available with OpenSSL 1.1.0 and newer. (Contributed by Christian Heimes in 27928.)

http.client

HTTPConnection.request() <http.client.HTTPConnection.request> and ~http.client.HTTPConnection.endheaders both now support chunked encoding request bodies. (Contributed by Demian Brecht and Rolf Krahl in 12319.)

idlelib and IDLE

The idlelib package is being modernized and refactored to make IDLE look and work better and to make the code easier to understand, test, and improve. Part of making IDLE look better, especially on Linux and Mac, is using ttk widgets, mostly in the dialogs. As a result, IDLE no longer runs with tcl/tk 8.4. It now requires tcl/tk 8.5 or 8.6. We recommend running the latest release of either.

'Modernizing' includes renaming and consolidation of idlelib modules. The renaming of files with partial uppercase names is similar to the renaming of, for instance, Tkinter and TkFont to tkinter and tkinter.font in 3.0. As a result, imports of idlelib files that worked in 3.5 will usually not work in 3.6. At least a module name change will be needed (see idlelib/README.txt), sometimes more. (Name changes contributed by Al Swiegart and Terry Reedy in 24225. Most idlelib patches since have been and will be part of the process.)

In compensation, the eventual result with be that some idlelib classes will be easier to use, with better APIs and docstrings explaining them. Additional useful information will be added to idlelib when available.

New in 3.6.2:

Multiple fixes for autocompletion. (Contributed by Louie Lu in 15786.)

New in 3.6.3:

Module Browser (on the File menu, formerly called Class Browser), now displays nested functions and classes in addition to top-level functions and classes. (Contributed by Guilherme Polo, Cheryl Sabella, and Terry Jan Reedy in 1612262.)

The IDLE features formerly implemented as extensions have been reimplemented as normal features. Their settings have been moved from the Extensions tab to other dialog tabs. (Contributed by Charles Wohlganger and Terry Jan Reedy in 27099.)

The Settings dialog (Options, Configure IDLE) has been partly rewritten to improve both appearance and function. (Contributed by Cheryl Sabella and Terry Jan Reedy in multiple issues.)

New in 3.6.4:

The font sample now includes a selection of non-Latin characters so that users can better see the effect of selecting a particular font. (Contributed by Terry Jan Reedy in 13802.) The sample can be edited to include other characters. (Contributed by Serhiy Storchaka in 31860.)

New in 3.6.6:

Editor code context option revised. Box displays all context lines up to maxlines. Clicking on a context line jumps the editor to that line. Context colors for custom themes is added to Highlights tab of Settings dialog. (Contributed by Cheryl Sabella and Terry Jan Reedy in 33642, 33768, and 33679.)

On Windows, a new API call tells Windows that tk scales for DPI. On Windows 8.1+ or 10, with DPI compatibility properties of the Python binary unchanged, and a monitor resolution greater than 96 DPI, this should make text and lines sharper. It should otherwise have no effect. (Contributed by Terry Jan Reedy in 33656.)

New in 3.6.7:

Output over N lines (50 by default) is squeezed down to a button. N can be changed in the PyShell section of the General page of the Settings dialog. Fewer, but possibly extra long, lines can be squeezed by right clicking on the output. Squeezed output can be expanded in place by double-clicking the button or into the clipboard or a separate window by right-clicking the button. (Contributed by Tal Einat in 1529353.)

importlib

Import now raises the new exception ModuleNotFoundError (subclass of ImportError) when it cannot find a module. Code that current checks for ImportError (in try-except) will still work. (Contributed by Eric Snow in 15767.)

importlib.util.LazyLoader now calls ~importlib.abc.Loader.create_module on the wrapped loader, removing the restriction that importlib.machinery.BuiltinImporter and importlib.machinery.ExtensionFileLoader couldn't be used with importlib.util.LazyLoader.

importlib.util.cache_from_source, importlib.util.source_from_cache, and importlib.util.spec_from_file_location now accept a path-like object.

inspect

The inspect.signature() <inspect.signature> function now reports the implicit .0 parameters generated by the compiler for comprehension and generator expression scopes as if they were positional-only parameters called implicit0. (Contributed by Jelle Zijlstra in 19611.)

To reduce code churn when upgrading from Python 2.7 and the legacy !inspect.getargspec API, the previously documented deprecation of inspect.getfullargspec has been reversed. While this function is convenient for single/source Python 2/3 code bases, the richer inspect.signature interface remains the recommended approach for new code. (Contributed by Nick Coghlan in 27172)

json

json.load and json.loads now support binary input. Encoded JSON should be represented using either UTF-8, UTF-16, or UTF-32. (Contributed by Serhiy Storchaka in 17909.)

logging

The new WatchedFileHandler.reopenIfNeeded() <logging.handlers.WatchedFileHandler.reopenIfNeeded> method has been added to add the ability to check if the log file needs to be reopened. (Contributed by Marian Horban in 24884.)

math

The tau (τ) constant has been added to the math and cmath modules. (Contributed by Lisa Roach in 12345, see 628 for details.)

multiprocessing

Proxy Objects <multiprocessing-proxy_objects> returned by multiprocessing.Manager can now be nested. (Contributed by Davin Potts in 6766.)

os

See the summary of PEP 519 <whatsnew36-pep519> for details on how the os and os.path modules now support path-like objects <path-like object>.

~os.scandir now supports bytes paths on Windows.

A new ~os.scandir.close method allows explicitly closing a ~os.scandir iterator. The ~os.scandir iterator now supports the context manager protocol. If a !scandir iterator is neither exhausted nor explicitly closed a ResourceWarning will be emitted in its destructor. (Contributed by Serhiy Storchaka in 25994.)

On Linux, os.urandom now blocks until the system urandom entropy pool is initialized to increase the security. See the 524 for the rationale.

The Linux getrandom() syscall (get random bytes) is now exposed as the new os.getrandom function. (Contributed by Victor Stinner, part of the 524)

pathlib

pathlib now supports path-like objects <path-like object>. (Contributed by Brett Cannon in 27186.)

See the summary of PEP 519 <whatsnew36-pep519> for details.

pdb

The ~pdb.Pdb class constructor has a new optional readrc argument to control whether .pdbrc files should be read.

pickle

Objects that need __new__ called with keyword arguments can now be pickled using pickle protocols <pickle-protocols> older than protocol version 4. Protocol version 4 already supports this case. (Contributed by Serhiy Storchaka in 24164.)

pickletools

pickletools.dis now outputs the implicit memo index for the MEMOIZE opcode. (Contributed by Serhiy Storchaka in 25382.)

pydoc

The pydoc module has learned to respect the MANPAGER environment variable. (Contributed by Matthias Klose in 8637.)

help and pydoc can now list named tuple fields in the order they were defined rather than alphabetically. (Contributed by Raymond Hettinger in 24879.)

random

The new ~random.choices function returns a list of elements of specified size from the given population with optional weights. (Contributed by Raymond Hettinger in 18844.)

re

Added support of modifier spans in regular expressions. Examples: '(?i:p)ython' matches 'python' and 'Python', but not 'PYTHON'; '(?i)g(?-i:v)r' matches 'GvR' and 'gvr', but not 'GVR'. (Contributed by Serhiy Storchaka in 433028.)

Match object groups can be accessed by __getitem__, which is equivalent to group(). So mo['name'] is now equivalent to mo.group('name'). (Contributed by Eric Smith in 24454.)

~re.Match objects now support index-like objects <object.__index__> as group indices. (Contributed by Jeroen Demeyer and Xiang Zhang in 27177.)

readline

Added ~readline.set_auto_history to enable or disable automatic addition of input to the history list. (Contributed by Tyler Crompton in 26870.)

rlcompleter

Private and special attribute names now are omitted unless the prefix starts with underscores. A space or a colon is added after some completed keywords. (Contributed by Serhiy Storchaka in 25011 and 25209.)

shlex

The ~shlex.shlex has much improved shell compatibility <improved-shell-compatibility> through the new punctuation_chars argument to control which characters are treated as punctuation. (Contributed by Vinay Sajip in 1521950.)

site

When specifying paths to add to sys.path in a .pth file, you may now specify file paths on top of directories (e.g. zip files). (Contributed by Wolfgang Langner in 26587).

sqlite3

sqlite3.Cursor.lastrowid now supports the REPLACE statement. (Contributed by Alex LordThorsen in 16864.)

socket

The ~socket.socket.ioctl function now supports the ~socket.SIO_LOOPBACK_FAST_PATH control code. (Contributed by Daniel Stokes in 26536.)

The ~socket.socket.getsockopt constants SO_DOMAIN, SO_PROTOCOL, SO_PEERSEC, and SO_PASSSEC are now supported. (Contributed by Christian Heimes in 26907.)

The ~socket.socket.setsockopt now supports the setsockopt(level, optname, None, optlen: int) form. (Contributed by Christian Heimes in 27744.)

The socket module now supports the address family ~socket.AF_ALG to interface with Linux Kernel crypto API. ALG_*, SOL_ALG and ~socket.socket.sendmsg_afalg were added. (Contributed by Christian Heimes in 27744 with support from Victor Stinner.)

New Linux constants TCP_USER_TIMEOUT and TCP_CONGESTION were added. (Contributed by Omar Sandoval, 26273).

socketserver

Servers based on the socketserver module, including those defined in http.server, xmlrpc.server and wsgiref.simple_server, now support the context manager protocol. (Contributed by Aviv Palivoda in 26404.)

The wfile <socketserver.DatagramRequestHandler.wfile> attribute of ~socketserver.StreamRequestHandler classes now implements the io.BufferedIOBase writable interface. In particular, calling ~io.BufferedIOBase.write is now guaranteed to send the data in full. (Contributed by Martin Panter in 26721.)

ssl

ssl supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. (Contributed by Christian Heimes in 26470.)

3DES has been removed from the default cipher suites and ChaCha20 Poly1305 cipher suites have been added. (Contributed by Christian Heimes in 27850 and 27766.)

~ssl.SSLContext has better default configuration for options and ciphers. (Contributed by Christian Heimes in 28043.)

SSL session can be copied from one client-side connection to another with the new ~ssl.SSLSession class. TLS session resumption can speed up the initial handshake, reduce latency and improve performance (Contributed by Christian Heimes in 19500 based on a draft by Alex Warhawk.)

The new ~ssl.SSLContext.get_ciphers method can be used to get a list of enabled ciphers in order of cipher priority.

All constants and flags have been converted to ~enum.IntEnum and ~enum.IntFlag. (Contributed by Christian Heimes in 28025.)

Server and client-side specific TLS protocols for ~ssl.SSLContext were added. (Contributed by Christian Heimes in 28085.)

Added ssl.SSLContext.post_handshake_auth to enable and ssl.SSLSocket.verify_client_post_handshake to initiate TLS 1.3 post-handshake authentication. (Contributed by Christian Heimes in 78851.)

statistics

A new ~statistics.harmonic_mean function has been added. (Contributed by Steven D'Aprano in 27181.)

struct

struct now supports IEEE 754 half-precision floats via the 'e' format specifier. (Contributed by Eli Stevens, Mark Dickinson in 11734.)

subprocess

subprocess.Popen destructor now emits a ResourceWarning warning if the child process is still running. Use the context manager protocol (with proc: ...) or explicitly call the ~subprocess.Popen.wait method to read the exit status of the child process. (Contributed by Victor Stinner in 26741.)

The subprocess.Popen constructor and all functions that pass arguments through to it now accept encoding and errors arguments. Specifying either of these will enable text mode for the stdin, stdout and stderr streams. (Contributed by Steve Dower in 6135.)

sys

The new ~sys.getfilesystemencodeerrors function returns the name of the error mode used to convert between Unicode filenames and bytes filenames. (Contributed by Steve Dower in 27781.)

On Windows the return value of the ~sys.getwindowsversion function now includes the platform_version field which contains the accurate major version, minor version and build number of the current operating system, rather than the version that is being emulated for the process (Contributed by Steve Dower in 27932.)

telnetlib

!telnetlib.Telnet is now a context manager (contributed by Stéphane Wirtel in 25485).

time

The ~time.struct_time attributes !tm_gmtoff and !tm_zone are now available on all platforms.

timeit

The new Timer.autorange() <timeit.Timer.autorange> convenience method has been added to call Timer.timeit() <timeit.Timer.timeit> repeatedly so that the total run time is greater or equal to 200 milliseconds. (Contributed by Steven D'Aprano in 6422.)

timeit now warns when there is substantial (4x) variance between best and worst times. (Contributed by Serhiy Storchaka in 23552.)

tkinter

Added methods !Variable.trace_add, !Variable.trace_remove and !trace_info in the !tkinter.Variable class. They replace old methods !trace_variable, !trace, !trace_vdelete and !trace_vinfo that use obsolete Tcl commands and might not work in future versions of Tcl. (Contributed by Serhiy Storchaka in 22115).

traceback

Both the traceback module and the interpreter's builtin exception display now abbreviate long sequences of repeated lines in tracebacks as shown in the following example:

>>> def f(): f()
...
>>> f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in f
  File "<stdin>", line 1, in f
  File "<stdin>", line 1, in f
  [Previous line repeated 995 more times]
RecursionError: maximum recursion depth exceeded

(Contributed by Emanuel Barry in 26823.)

tracemalloc

The tracemalloc module now supports tracing memory allocations in multiple different address spaces.

The new ~tracemalloc.DomainFilter filter class has been added to filter block traces by their address space (domain).

(Contributed by Victor Stinner in 26588.)

typing

Since the typing module is provisional <provisional API>, all changes introduced in Python 3.6 have also been backported to Python 3.5.x.

The typing module has a much improved support for generic type aliases. For example Dict[str, Tuple[S, T]] is now a valid type annotation. (Contributed by Guido van Rossum in Github #195.)

The typing.ContextManager class has been added for representing contextlib.AbstractContextManager. (Contributed by Brett Cannon in 25609.)

The typing.Collection class has been added for representing collections.abc.Collection. (Contributed by Ivan Levkivskyi in 27598.)

The typing.ClassVar type construct has been added to mark class variables. As introduced in 526, a variable annotation wrapped in ClassVar indicates that a given attribute is intended to be used as a class variable and should not be set on instances of that class. (Contributed by Ivan Levkivskyi in Github #280.)

A new ~typing.TYPE_CHECKING constant that is assumed to be True by the static type checkers, but is False at runtime. (Contributed by Guido van Rossum in Github #230.)

A new ~typing.NewType helper function has been added to create lightweight distinct types for annotations:

from typing import NewType

UserId = NewType('UserId', int)
some_id = UserId(524313)

The static type checker will treat the new type as if it were a subclass of the original type. (Contributed by Ivan Levkivskyi in Github #189.)

unicodedata

The unicodedata module now uses data from Unicode 9.0.0. (Contributed by Benjamin Peterson.)

unittest.mock

The ~unittest.mock.Mock class has the following improvements:

urllib.request

If a HTTP request has a file or iterable body (other than a bytes object) but no Content-Length header, rather than throwing an error, AbstractHTTPHandler <urllib.request.HTTPHandler> now falls back to use chunked transfer encoding. (Contributed by Demian Brecht and Rolf Krahl in 12319.)

urllib.robotparser

~urllib.robotparser.RobotFileParser now supports the Crawl-delay and Request-rate extensions. (Contributed by Nikolay Bogoychev in 16099.)

venv

venv accepts a new parameter --prompt. This parameter provides an alternative prefix for the virtual environment. (Proposed by Łukasz Balcerzak and ported to 3.6 by Stéphane Wirtel in 22829.)

warnings

A new optional source parameter has been added to the warnings.warn_explicit function: the destroyed object which emitted a ResourceWarning. A source attribute has also been added to !warnings.WarningMessage (contributed by Victor Stinner in 26568 and 26567).

When a ResourceWarning warning is logged, the tracemalloc module is now used to try to retrieve the traceback where the destroyed object was allocated.

Example with the script example.py:

import warnings

def func():
    return open(__file__)

f = func()
f = None

Output of the command python3.6 -Wd -X tracemalloc=5 example.py:

example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper name='example.py' mode='r' encoding='UTF-8'>
  f = None
Object allocated at (most recent call first):
  File "example.py", lineno 4
    return open(__file__)
  File "example.py", lineno 6
    f = func()

The "Object allocated at" traceback is new and is only displayed if tracemalloc is tracing Python memory allocations and if the warnings module was already imported.

winreg

Added the 64-bit integer type REG_QWORD <winreg.REG_QWORD>. (Contributed by Clement Rouault in 23026.)

winsound

Allowed keyword arguments to be passed to Beep <winsound.Beep>, MessageBeep <winsound.MessageBeep>, and PlaySound <winsound.PlaySound> (27982).

xmlrpc.client

The xmlrpc.client module now supports unmarshalling additional data types used by the Apache XML-RPC implementation for numerics and None. (Contributed by Serhiy Storchaka in 26885.)

zipfile

A new ZipInfo.from_file() <zipfile.ZipInfo.from_file> class method allows making a ~zipfile.ZipInfo instance from a filesystem file. A new ZipInfo.is_dir() <zipfile.ZipInfo.is_dir> method can be used to check if the ~zipfile.ZipInfo instance represents a directory. (Contributed by Thomas Kluyver in 26039.)

The ZipFile.open() <zipfile.ZipFile.open> method can now be used to write data into a ZIP file, as well as for extracting data. (Contributed by Thomas Kluyver in 26039.)

zlib

The ~zlib.compress and ~zlib.decompress functions now accept keyword arguments. (Contributed by Aviv Palivoda in 26243 and Xiang Zhang in 16764 respectively.)

Optimizations

Build and C API Changes

Other Improvements

Deprecated

New Keywords

async and await are not recommended to be used as variable, class, function or module names. Introduced by 492 in Python 3.5, they will become proper keywords in Python 3.7. Starting in Python 3.6, the use of async or await as names will generate a DeprecationWarning.

Deprecated Python behavior

Raising the StopIteration exception inside a generator will now generate a DeprecationWarning, and will trigger a RuntimeError in Python 3.7. See whatsnew-pep-479 for details.

The ~object.__aiter__ method is now expected to return an asynchronous iterator directly instead of returning an awaitable as previously. Doing the former will trigger a DeprecationWarning. Backward compatibility will be removed in Python 3.7. (Contributed by Yury Selivanov in 27243.)

A backslash-character pair that is not a valid escape sequence now generates a DeprecationWarning. Although this will eventually become a SyntaxError, that will not be for several Python releases. (Contributed by Emanuel Barry in 27364.)

When performing a relative import, falling back on __name__ and __path__ from the calling module when __spec__ or __package__ are not defined now raises an ImportWarning. (Contributed by Rose Ames in 25791.)

Deprecated Python modules, functions and methods

asynchat

The !asynchat has been deprecated in favor of asyncio. (Contributed by Mariatta in 25002.)

asyncore

The !asyncore has been deprecated in favor of asyncio. (Contributed by Mariatta in 25002.)

dbm

Unlike other dbm implementations, the dbm.dumb module creates databases with the 'rw' mode and allows modifying the database opened with the 'r' mode. This behavior is now deprecated and will be removed in 3.8. (Contributed by Serhiy Storchaka in 21708.)

distutils

The undocumented extra_path argument to the distutils.Distribution constructor is now considered deprecated and will raise a warning if set. Support for this parameter will be removed in a future Python release. See 27919 for details.

grp

The support of non-integer arguments in ~grp.getgrgid has been deprecated. (Contributed by Serhiy Storchaka in 26129.)

importlib

The importlib.machinery.SourceFileLoader.load_module and importlib.machinery.SourcelessFileLoader.load_module methods are now deprecated. They were the only remaining implementations of importlib.abc.Loader.load_module in importlib that had not been deprecated in previous versions of Python in favour of importlib.abc.Loader.exec_module.

The importlib.machinery.WindowsRegistryFinder class is now deprecated. As of 3.6.0, it is still added to sys.meta_path by default (on Windows), but this may change in future releases.

os

Undocumented support of general bytes-like objects <bytes-like object> as paths in os functions, compile and similar functions is now deprecated. (Contributed by Serhiy Storchaka in 25791 and 26754.)

re

Support for inline flags (?letters) in the middle of the regular expression has been deprecated and will be removed in a future Python version. Flags at the start of a regular expression are still allowed. (Contributed by Serhiy Storchaka in 22493.)

ssl

OpenSSL 0.9.8, 1.0.0 and 1.0.1 are deprecated and no longer supported. In the future the ssl module will require at least OpenSSL 1.0.2 or 1.1.0.

A couple of protocols and functions of the ssl module are now deprecated. Some features will no longer be available in future versions of OpenSSL. Other features are deprecated in favor of a different API. (Contributed by Christian Heimes in 28022 and 26470.)

tkinter

The !tkinter.tix module is now deprecated. tkinter users should use tkinter.ttk instead.

venv

The pyvenv script has been deprecated in favour of python3 -m venv. This prevents confusion as to what Python interpreter pyvenv is connected to and thus what Python interpreter will be used by the virtual environment. (Contributed by Brett Cannon in 25154.)

xml

Deprecated functions and types of the C API

Undocumented functions !PyUnicode_AsEncodedObject, !PyUnicode_AsDecodedObject, !PyUnicode_AsEncodedUnicode and !PyUnicode_AsDecodedUnicode are deprecated now. Use the generic codec based API <codec-registry> instead.

Deprecated Build Options

The --with-system-ffi configure flag is now on by default on non-macOS UNIX platforms. It may be disabled by using --without-system-ffi, but using the flag is deprecated and will not be accepted in Python 3.7. macOS is unaffected by this change. Note that many OS distributors already use the --with-system-ffi flag when building their system Python.

Removed

API and Feature Removals

Porting to Python 3.6

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in 'python' Command Behavior

Changes in the Python API

Changes in the C API

CPython bytecode changes

There have been several major changes to the bytecode in Python 3.6.

Notable changes in Python 3.6.2

New make regen-all build target

To simplify cross-compilation, and to ensure that CPython can reliably be compiled without requiring an existing version of Python to already be available, the autotools-based build system no longer attempts to implicitly recompile generated files based on file modification times.

Instead, a new make regen-all command has been added to force regeneration of these files when desired (e.g. after an initial version of Python has already been built based on the pregenerated versions).

More selective regeneration targets are also defined - see Makefile.pre.in for details.

(Contributed by Victor Stinner in 23404.)

3.6.2

Removal of make touch build target

The make touch build target previously used to request implicit regeneration of generated files by updating their modification times has been removed.

It has been replaced by the new make regen-all target.

(Contributed by Victor Stinner in 23404.)

3.6.2

Notable changes in Python 3.6.4

The PyExc_RecursionErrorInst singleton that was part of the public API has been removed as its members being never cleared may cause a segfault during finalization of the interpreter. (Contributed by Xavier de Gaye in 22898 and 30697.)

Notable changes in Python 3.6.5

The locale.localeconv function now sets temporarily the LC_CTYPE locale to the LC_NUMERIC locale in some cases. (Contributed by Victor Stinner in 31900.)

Notable changes in Python 3.6.7

xml.dom.minidom and xml.sax modules no longer process external entities by default. See also 61441.

In 3.6.7 the tokenize module now implicitly emits a NEWLINE token when provided with input that does not have a trailing new line. This behavior now matches what the C tokenizer does internally. (Contributed by Ammar Askar in 33899.)

Notable changes in Python 3.6.10

Due to significant security concerns, the reuse_address parameter of asyncio.loop.create_datagram_endpoint is no longer supported. This is because of the behavior of the socket option SO_REUSEADDR in UDP. For more details, see the documentation for loop.create_datagram_endpoint(). (Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in 37228.)

Notable changes in Python 3.6.13

Earlier Python versions allowed using both ; and & as query parameter separators in urllib.parse.parse_qs and urllib.parse.parse_qsl. Due to security concerns, and to conform with newer W3C recommendations, this has been changed to allow only a single separator key, with & as the default. This change also affects !cgi.parse and !cgi.parse_multipart as they use the affected functions internally. For more details, please see their respective documentation. (Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in 42967.)

Notable changes in Python 3.6.14

A security fix alters the ftplib.FTP behavior to not trust the IPv4 address sent from the remote server when setting up a passive data channel. We reuse the ftp server IP address instead. For unusual code requiring the old behavior, set a trust_server_pasv_ipv4_address attribute on your FTP instance to True. (See 87451)

The presence of newline or tab characters in parts of a URL allows for some forms of attacks. Following the WHATWG specification that updates RFC 3986, ASCII newline \n, \r and tab \t characters are stripped from the URL by the parser urllib.parse preventing such attacks. The removal characters are controlled by a new module level variable urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE. (See 88048)

What's New In Python 3.7

Editor
Elvis Pranskevichus <elvis@magic.io>

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the Git log when researching a change.

This article explains the new features in Python 3.7, compared to 3.6. Python 3.7 was released on June 27, 2018. For full details, see the changelog <changelog>.

Summary -- Release Highlights

New syntax features:

Backwards incompatible syntax changes:

New library modules:

New built-in features:

Python data model improvements:

Significant improvements in the standard library:

CPython implementation improvements:

C API improvements:

Documentation improvements:

This release features notable performance improvements in many areas. The whatsnew37-perf section lists them in detail.

For a list of changes that may affect compatibility with previous Python releases please refer to the porting-to-python-37 section.

New Features

PEP 563: Postponed Evaluation of Annotations

The advent of type hints in Python uncovered two glaring usability issues with the functionality of annotations added in 3107 and refined further in 526:

Both of these issues are fixed by postponing the evaluation of annotations. Instead of compiling code which executes expressions in annotations at their definition time, the compiler stores the annotation in a string form equivalent to the AST of the expression in question. If needed, annotations can be resolved at runtime using typing.get_type_hints. In the common case where this is not required, the annotations are cheaper to store (since short strings are interned by the interpreter) and make startup time faster.

Usability-wise, annotations now support forward references, making the following syntax valid:

class C:
    @classmethod
    def from_string(cls, source: str) -> C:
        ...

    def validate_b(self, obj: B) -> bool:
        ...

class B:
    ...

Since this change breaks compatibility, the new behavior needs to be enabled on a per-module basis in Python 3.7 using a __future__ import:

from __future__ import annotations

It will become the default in Python 3.10.

563 -- Postponed evaluation of annotations
PEP written and implemented by Łukasz Langa.

PEP 538: Legacy C Locale Coercion

An ongoing challenge within the Python 3 series has been determining a sensible default strategy for handling the "7-bit ASCII" text encoding assumption currently implied by the use of the default C or POSIX locale on non-Windows platforms.

538 updates the default interpreter command line interface to automatically coerce that locale to an available UTF-8 based locale as described in the documentation of the new PYTHONCOERCECLOCALE environment variable. Automatically setting LC_CTYPE this way means that both the core interpreter and locale-aware C extensions (such as readline) will assume the use of UTF-8 as the default text encoding, rather than ASCII.

The platform support definition in 11 has also been updated to limit full text handling support to suitably configured non-ASCII based locales.

As part of this change, the default error handler for ~sys.stdin and ~sys.stdout is now surrogateescape (rather than strict) when using any of the defined coercion target locales (currently C.UTF-8, C.utf8, and UTF-8). The default error handler for ~sys.stderr continues to be backslashreplace, regardless of locale.

Locale coercion is silent by default, but to assist in debugging potentially locale related integration problems, explicit warnings (emitted directly on ~sys.stderr) can be requested by setting PYTHONCOERCECLOCALE=warn. This setting will also cause the Python runtime to emit a warning if the legacy C locale remains active when the core interpreter is initialized.

While 538's locale coercion has the benefit of also affecting extension modules (such as GNU readline), as well as child processes (including those running non-Python applications and older versions of Python), it has the downside of requiring that a suitable target locale be present on the running system. To better handle the case where no suitable target locale is available (as occurs on RHEL/CentOS 7, for example), Python 3.7 also implements whatsnew37-pep540.

538 -- Coercing the legacy C locale to a UTF-8 based locale
PEP written and implemented by Nick Coghlan.

PEP 540: Forced UTF-8 Runtime Mode

The new -X utf8 command line option and PYTHONUTF8 environment variable can be used to enable the Python UTF-8 Mode <utf8-mode>.

When in UTF-8 mode, CPython ignores the locale settings, and uses the UTF-8 encoding by default. The error handlers for sys.stdin and sys.stdout streams are set to surrogateescape.

The forced UTF-8 mode can be used to change the text handling behavior in an embedded Python interpreter without changing the locale settings of an embedding application.

While 540's UTF-8 mode has the benefit of working regardless of which locales are available on the running system, it has the downside of having no effect on extension modules (such as GNU readline), child processes running non-Python applications, and child processes running older versions of Python. To reduce the risk of corrupting text data when communicating with such components, Python 3.7 also implements whatsnew37-pep540).

The UTF-8 mode is enabled by default when the locale is C or POSIX, and the 538 locale coercion feature fails to change it to a UTF-8 based alternative (whether that failure is due to PYTHONCOERCECLOCALE=0 being set, LC_ALL being set, or the lack of a suitable target locale).

540 -- Add a new UTF-8 mode
PEP written and implemented by Victor Stinner

PEP 553: Built-in breakpoint()

Python 3.7 includes the new built-in breakpoint function as an easy and consistent way to enter the Python debugger.

Built-in breakpoint() calls sys.breakpointhook. By default, the latter imports pdb and then calls pdb.set_trace(), but by binding sys.breakpointhook() to the function of your choosing, breakpoint() can enter any debugger. Additionally, the environment variable PYTHONBREAKPOINT can be set to the callable of your debugger of choice. Set PYTHONBREAKPOINT=0 to completely disable built-in breakpoint().

553 -- Built-in breakpoint()
PEP written and implemented by Barry Warsaw

PEP 539: New C API for Thread-Local Storage

While Python provides a C API for thread-local storage support; the existing Thread Local Storage (TLS) API <thread-local-storage-api> has used int to represent TLS keys across all platforms. This has not generally been a problem for officially support platforms, but that is neither POSIX-compliant, nor portable in any practical sense.

539 changes this by providing a new Thread Specific Storage (TSS) API <thread-specific-storage-api> to CPython which supersedes use of the existing TLS API within the CPython interpreter, while deprecating the existing API. The TSS API uses a new type Py_tss_t instead of int to represent TSS keys--an opaque type the definition of which may depend on the underlying TLS implementation. Therefore, this will allow to build CPython on platforms where the native TLS key is defined in a way that cannot be safely cast to int.

Note that on platforms where the native TLS key is defined in a way that cannot be safely cast to int, all functions of the existing TLS API will be no-op and immediately return failure. This indicates clearly that the old API is not supported on platforms where it cannot be used reliably, and that no effort will be made to add such support.

539 -- A New C-API for Thread-Local Storage in CPython
PEP written by Erik M. Bray; implementation by Masayuki Yamamoto.

PEP 562: Customization of Access to Module Attributes

Python 3.7 allows defining ~module.__getattr__ on modules and will call it whenever a module attribute is otherwise not found. Defining ~module.__dir__ on modules is now also allowed.

A typical example of where this may be useful is module attribute deprecation and lazy loading.

562 -- Module __getattr__ and __dir__
PEP written and implemented by Ivan Levkivskyi

PEP 564: New Time Functions With Nanosecond Resolution

The resolution of clocks in modern systems can exceed the limited precision of a floating-point number returned by the time.time function and its variants. To avoid loss of precision, 564 adds six new "nanosecond" variants of the existing timer functions to the time module:

The new functions return the number of nanoseconds as an integer value.

Measurements <0564#annex-clocks-resolution-in-python> show that on Linux and Windows the resolution of time.time_ns is approximately 3 times better than that of time.time.

564 -- Add new time functions with nanosecond resolution
PEP written and implemented by Victor Stinner

PEP 565: Show DeprecationWarning in __main__

The default handling of DeprecationWarning has been changed such that these warnings are once more shown by default, but only when the code triggering them is running directly in the __main__ module. As a result, developers of single file scripts and those using Python interactively should once again start seeing deprecation warnings for the APIs they use, but deprecation warnings triggered by imported application, library and framework modules will continue to be hidden by default.

As a result of this change, the standard library now allows developers to choose between three different deprecation warning behaviours:

Previously both DeprecationWarning and PendingDeprecationWarning were only visible when running tests, which meant that developers primarily writing single file scripts or using Python interactively could be surprised by breaking changes in the APIs they used.

565 -- Show DeprecationWarning in __main__
PEP written and implemented by Nick Coghlan

PEP 560: Core Support for typing module and Generic Types

Initially 484 was designed in such way that it would not introduce any changes to the core CPython interpreter. Now type hints and the typing module are extensively used by the community, so this restriction is removed. The PEP introduces two special methods ~object.__class_getitem__ and ~object.__mro_entries__, these methods are now used by most classes and special constructs in typing. As a result, the speed of various operations with types increased up to 7 times, the generic types can be used without metaclass conflicts, and several long standing bugs in typing module are fixed.

560 -- Core support for typing module and generic types
PEP written and implemented by Ivan Levkivskyi

PEP 552: Hash-based .pyc Files

Python has traditionally checked the up-to-dateness of bytecode cache files (i.e., .pyc files) by comparing the source metadata (last-modified timestamp and size) with source metadata saved in the cache file header when it was generated. While effective, this invalidation method has its drawbacks. When filesystem timestamps are too coarse, Python can miss source updates, leading to user confusion. Additionally, having a timestamp in the cache file is problematic for build reproducibility and content-based build systems.

552 extends the pyc format to allow the hash of the source file to be used for invalidation instead of the source timestamp. Such .pyc files are called "hash-based". By default, Python still uses timestamp-based invalidation and does not generate hash-based .pyc files at runtime. Hash-based .pyc files may be generated with py_compile or compileall.

Hash-based .pyc files come in two variants: checked and unchecked. Python validates checked hash-based .pyc files against the corresponding source files at runtime but doesn't do so for unchecked hash-based pycs. Unchecked hash-based .pyc files are a useful performance optimization for environments where a system external to Python (e.g., the build system) is responsible for keeping .pyc files up-to-date.

See pyc-invalidation for more information.

552 -- Deterministic pycs
PEP written and implemented by Benjamin Peterson

PEP 545: Python Documentation Translations

545 describes the process of creating and maintaining Python documentation translations.

Three new translations have been added:

545 -- Python Documentation Translations
PEP written and implemented by Julien Palard, Inada Naoki, and Victor Stinner.

Python Development Mode (-X dev)

The new -X dev command line option or the new PYTHONDEVMODE environment variable can be used to enable Python Development Mode <devmode>. When in development mode, Python performs additional runtime checks that are too expensive to be enabled by default. See Python Development Mode <devmode> documentation for the full description.

Other Language Changes

New Modules

contextvars

The new contextvars module and a set of new C APIs <contextvarsobjects> introduce support for context variables. Context variables are conceptually similar to thread-local variables. Unlike TLS, context variables support asynchronous code correctly.

The asyncio and decimal modules have been updated to use and support context variables out of the box. Particularly the active decimal context is now stored in a context variable, which allows decimal operations to work with the correct context in asynchronous code.

567 -- Context Variables
PEP written and implemented by Yury Selivanov

dataclasses

The new ~dataclasses.dataclass decorator provides a way to declare data classes. A data class describes its attributes using class variable annotations. Its constructor and other magic methods, such as ~object.__repr__, ~object.__eq__, and ~object.__hash__ are generated automatically.

Example:

@dataclass
class Point:
    x: float
    y: float
    z: float = 0.0

p = Point(1.5, 2.5)
print(p)   # produces "Point(x=1.5, y=2.5, z=0.0)"

557 -- Data Classes
PEP written and implemented by Eric V. Smith

importlib.resources

The new importlib.resources module provides several new APIs and one new ABC for access to, opening, and reading resources inside packages. Resources are roughly similar to files inside packages, but they needn't be actual files on the physical file system. Module loaders can provide a !get_resource_reader function which returns a !importlib.abc.ResourceReader instance to support this new API. Built-in file path loaders and zip file loaders both support this.

Contributed by Barry Warsaw and Brett Cannon in 32248.

importlib_resources -- a PyPI backport for earlier Python versions.

Improved Modules

argparse

The new ArgumentParser.parse_intermixed_args() <argparse.ArgumentParser.parse_intermixed_args> method allows intermixing options and positional arguments. (Contributed by paul.j3 in 14191.)

asyncio

The asyncio module has received many new features, usability and performance improvements <whatsnew37-asyncio-perf>. Notable changes include:

Several asyncio APIs have been deprecated <whatsnew37-asyncio-deprecated>.

binascii

The ~binascii.b2a_uu function now accepts an optional backtick keyword argument. When it's true, zeros are represented by '`' instead of spaces. (Contributed by Xiang Zhang in 30103.)

calendar

The ~calendar.HTMLCalendar class has new class attributes which ease the customization of CSS classes in the produced HTML calendar. (Contributed by Oz Tiram in 30095.)

collections

collections.namedtuple() now supports default values. (Contributed by Raymond Hettinger in 32320.)

compileall

compileall.compile_dir learned the new invalidation_mode parameter, which can be used to enable hash-based .pyc invalidation <whatsnew37-pep552>. The invalidation mode can also be specified on the command line using the new --invalidation-mode argument. (Contributed by Benjamin Peterson in 31650.)

concurrent.futures

ProcessPoolExecutor <concurrent.futures.ProcessPoolExecutor> and ThreadPoolExecutor <concurrent.futures.ThreadPoolExecutor> now support the new initializer and initargs constructor arguments. (Contributed by Antoine Pitrou in 21423.)

The ProcessPoolExecutor <concurrent.futures.ProcessPoolExecutor> can now take the multiprocessing context via the new mp_context argument. (Contributed by Thomas Moreau in 31540.)

contextlib

The new ~contextlib.nullcontext is a simpler and faster no-op context manager than ~contextlib.ExitStack. (Contributed by Jesse-Bakker in 10049.)

The new ~contextlib.asynccontextmanager, ~contextlib.AbstractAsyncContextManager, and ~contextlib.AsyncExitStack have been added to complement their synchronous counterparts. (Contributed by Jelle Zijlstra in 29679 and 30241, and by Alexander Mohr and Ilya Kulakov in 29302.)

cProfile

The !cProfile command line now accepts -m module_name as an alternative to script path. (Contributed by Sanyam Khurana in 21862.)

crypt

The !crypt module now supports the Blowfish hashing method. (Contributed by Serhiy Storchaka in 31664.)

The !mksalt function now allows specifying the number of rounds for hashing. (Contributed by Serhiy Storchaka in 31702.)

datetime

The new datetime.fromisoformat() <datetime.datetime.fromisoformat> method constructs a ~datetime.datetime object from a string in one of the formats output by datetime.isoformat() <datetime.datetime.isoformat>. (Contributed by Paul Ganssle in 15873.)

The tzinfo <datetime.tzinfo> class now supports sub-minute offsets. (Contributed by Alexander Belopolsky in 5288.)

dbm

dbm.dumb now supports reading read-only files and no longer writes the index file when it is not changed.

decimal

The decimal module now uses context variables <whatsnew37-pep567> to store the decimal context. (Contributed by Yury Selivanov in 32630.)

dis

The ~dis.dis function is now able to disassemble nested code objects (the code of comprehensions, generator expressions and nested functions, and the code used for building nested classes). The maximum depth of disassembly recursion is controlled by the new depth parameter. (Contributed by Serhiy Storchaka in 11822.)

distutils

README.rst is now included in the list of distutils standard READMEs and therefore included in source distributions. (Contributed by Ryan Gonzalez in 11913.)

enum

The Enum <enum.Enum> learned the new _ignore_ class property, which allows listing the names of properties which should not become enum members. (Contributed by Ethan Furman in 31801.)

In Python 3.8, attempting to check for non-Enum objects in ~enum.Enum classes will raise a TypeError (e.g. 1 in Color); similarly, attempting to check for non-Flag objects in a ~enum.Flag member will raise TypeError (e.g. 1 in Perm.RW); currently, both operations return False instead and are deprecated. (Contributed by Ethan Furman in 33217.)

functools

functools.singledispatch now supports registering implementations using type annotations. (Contributed by Łukasz Langa in 32227.)

gc

The new gc.freeze function allows freezing all objects tracked by the garbage collector and excluding them from future collections. This can be used before a POSIX fork() call to make the GC copy-on-write friendly or to speed up collection. The new gc.unfreeze functions reverses this operation. Additionally, gc.get_freeze_count can be used to obtain the number of frozen objects. (Contributed by Li Zekun in 31558.)

hmac

The hmac module now has an optimized one-shot ~hmac.digest function, which is up to three times faster than ~hmac.HMAC. (Contributed by Christian Heimes in 32433.)

http.client

~http.client.HTTPConnection and ~http.client.HTTPSConnection now support the new blocksize argument for improved upload throughput. (Contributed by Nir Soffer in 31945.)

http.server

~http.server.SimpleHTTPRequestHandler now supports the HTTP If-Modified-Since header. The server returns the 304 response status if the target file was not modified after the time specified in the header. (Contributed by Pierre Quentel in 29654.)

~http.server.SimpleHTTPRequestHandler accepts the new directory argument, in addition to the new --directory command line argument. With this parameter, the server serves the specified directory, by default it uses the current working directory. (Contributed by Stéphane Wirtel and Julien Palard in 28707.)

The new ThreadingHTTPServer <http.server.ThreadingHTTPServer> class uses threads to handle requests using ~socketserver.ThreadingMixIn. It is used when http.server is run with -m. (Contributed by Julien Palard in 31639.)

idlelib and IDLE

Multiple fixes for autocompletion. (Contributed by Louie Lu in 15786.)

Module Browser (on the File menu, formerly called Class Browser), now displays nested functions and classes in addition to top-level functions and classes. (Contributed by Guilherme Polo, Cheryl Sabella, and Terry Jan Reedy in 1612262.)

The Settings dialog (Options, Configure IDLE) has been partly rewritten to improve both appearance and function. (Contributed by Cheryl Sabella and Terry Jan Reedy in multiple issues.)

The font sample now includes a selection of non-Latin characters so that users can better see the effect of selecting a particular font. (Contributed by Terry Jan Reedy in 13802.) The sample can be edited to include other characters. (Contributed by Serhiy Storchaka in 31860.)

The IDLE features formerly implemented as extensions have been reimplemented as normal features. Their settings have been moved from the Extensions tab to other dialog tabs. (Contributed by Charles Wohlganger and Terry Jan Reedy in 27099.)

Editor code context option revised. Box displays all context lines up to maxlines. Clicking on a context line jumps the editor to that line. Context colors for custom themes is added to Highlights tab of Settings dialog. (Contributed by Cheryl Sabella and Terry Jan Reedy in 33642, 33768, and 33679.)

On Windows, a new API call tells Windows that tk scales for DPI. On Windows 8.1+ or 10, with DPI compatibility properties of the Python binary unchanged, and a monitor resolution greater than 96 DPI, this should make text and lines sharper. It should otherwise have no effect. (Contributed by Terry Jan Reedy in 33656.)

New in 3.7.1:

Output over N lines (50 by default) is squeezed down to a button. N can be changed in the PyShell section of the General page of the Settings dialog. Fewer, but possibly extra long, lines can be squeezed by right clicking on the output. Squeezed output can be expanded in place by double-clicking the button or into the clipboard or a separate window by right-clicking the button. (Contributed by Tal Einat in 1529353.)

The changes above have been backported to 3.6 maintenance releases.

NEW in 3.7.4:

Add "Run Customized" to the Run menu to run a module with customized settings. Any command line arguments entered are added to sys.argv. They re-appear in the box for the next customized run. One can also suppress the normal Shell main module restart. (Contributed by Cheryl Sabella, Terry Jan Reedy, and others in 5680 and 37627.)

New in 3.7.5:

Add optional line numbers for IDLE editor windows. Windows open without line numbers unless set otherwise in the General tab of the configuration dialog. Line numbers for an existing window are shown and hidden in the Options menu. (Contributed by Tal Einat and Saimadhav Heblikar in 17535.)

importlib

The !importlib.abc.ResourceReader ABC was introduced to support the loading of resources from packages. See also whatsnew37_importlib_resources. (Contributed by Barry Warsaw, Brett Cannon in 32248.)

importlib.reload now raises ModuleNotFoundError if the module lacks a spec. (Contributed by Garvit Khatri in 29851.)

importlib.util.find_spec now raises ModuleNotFoundError instead of AttributeError if the specified parent module is not a package (i.e. lacks a __path__ attribute). (Contributed by Milan Oberkirch in 30436.)

The new importlib.util.source_hash can be used to compute the hash of the passed source. A hash-based .pyc file <whatsnew37-pep552> embeds the value returned by this function.

io

The new TextIOWrapper.reconfigure() <io.TextIOWrapper.reconfigure> method can be used to reconfigure the text stream with the new settings. (Contributed by Antoine Pitrou in 30526 and INADA Naoki in 15216.)

ipaddress

The new subnet_of() and supernet_of() methods of ipaddress.IPv6Network and ipaddress.IPv4Network can be used for network containment tests. (Contributed by Michel Albert and Cheryl Sabella in 20825.)

itertools

itertools.islice now accepts integer-like objects <object.__index__> as start, stop, and slice arguments. (Contributed by Will Roberts in 30537.)

locale

The new monetary argument to locale.format_string can be used to make the conversion use monetary thousands separators and grouping strings. (Contributed by Garvit in 10379.)

The locale.getpreferredencoding function now always returns 'UTF-8' on Android or when in the forced UTF-8 mode <whatsnew37-pep540>.

logging

~logging.Logger instances can now be pickled. (Contributed by Vinay Sajip in 30520.)

The new StreamHandler.setStream() <logging.StreamHandler.setStream> method can be used to replace the logger stream after handler creation. (Contributed by Vinay Sajip in 30522.)

It is now possible to specify keyword arguments to handler constructors in configuration passed to logging.config.fileConfig. (Contributed by Preston Landers in 31080.)

math

The new math.remainder function implements the IEEE 754-style remainder operation. (Contributed by Mark Dickinson in 29962.)

mimetypes

The MIME type of .bmp has been changed from 'image/x-ms-bmp' to 'image/bmp'. (Contributed by Nitish Chandra in 22589.)

msilib

The new !Database.Close method can be used to close the MSI database. (Contributed by Berker Peksag in 20486.)

multiprocessing

The new Process.close() <multiprocessing.Process.close> method explicitly closes the process object and releases all resources associated with it. ValueError is raised if the underlying process is still running. (Contributed by Antoine Pitrou in 30596.)

The new Process.kill() <multiprocessing.Process.kill> method can be used to terminate the process using the ~signal.SIGKILL signal on Unix. (Contributed by Vitor Pereira in 30794.)

Non-daemonic threads created by ~multiprocessing.Process are now joined on process exit. (Contributed by Antoine Pitrou in 18966.)

os

os.fwalk now accepts the path argument as bytes. (Contributed by Serhiy Storchaka in 28682.)

os.scandir gained support for file descriptors <path_fd>. (Contributed by Serhiy Storchaka in 25996.)

The new ~os.register_at_fork function allows registering Python callbacks to be executed at process fork. (Contributed by Antoine Pitrou in 16500.)

Added os.preadv (combine the functionality of os.readv and os.pread) and os.pwritev functions (combine the functionality of os.writev and os.pwrite). (Contributed by Pablo Galindo in 31368.)

The mode argument of os.makedirs no longer affects the file permission bits of newly created intermediate-level directories. (Contributed by Serhiy Storchaka in 19930.)

os.dup2 now returns the new file descriptor. Previously, None was always returned. (Contributed by Benjamin Peterson in 32441.)

The structure returned by os.stat now contains the ~os.stat_result.st_fstype attribute on Solaris and its derivatives. (Contributed by Jesús Cea Avión in 32659.)

pathlib

The new Path.is_mount() <pathlib.Path.is_mount> method is now available on POSIX systems and can be used to determine whether a path is a mount point. (Contributed by Cooper Ry Lees in 30897.)

pdb

pdb.set_trace now takes an optional header keyword-only argument. If given, it is printed to the console just before debugging begins. (Contributed by Barry Warsaw in 31389.)

pdb command line now accepts -m module_name as an alternative to script file. (Contributed by Mario Corchero in 32206.)

py_compile

py_compile.compile -- and by extension, compileall -- now respects the SOURCE_DATE_EPOCH environment variable by unconditionally creating .pyc files for hash-based validation. This allows for guaranteeing reproducible builds of .pyc files when they are created eagerly. (Contributed by Bernhard M. Wiedemann in 29708.)

pydoc

The pydoc server can now bind to an arbitrary hostname specified by the new -n command-line argument. (Contributed by Feanil Patel in 31128.)

queue

The new ~queue.SimpleQueue class is an unbounded FIFO queue. (Contributed by Antoine Pitrou in 14976.)

re

The flags re.ASCII, re.LOCALE and re.UNICODE can be set within the scope of a group. (Contributed by Serhiy Storchaka in 31690.)

re.split now supports splitting on a pattern like r'\b', '^$' or (?=-) that matches an empty string. (Contributed by Serhiy Storchaka in 25054.)

Regular expressions compiled with the re.LOCALE flag no longer depend on the locale at compile time. Locale settings are applied only when the compiled regular expression is used. (Contributed by Serhiy Storchaka in 30215.)

FutureWarning is now emitted if a regular expression contains character set constructs that will change semantically in the future, such as nested sets and set operations. (Contributed by Serhiy Storchaka in 30349.)

Compiled regular expression and match objects can now be copied using copy.copy and copy.deepcopy. (Contributed by Serhiy Storchaka in 10076.)

signal

The new warn_on_full_buffer argument to the signal.set_wakeup_fd function makes it possible to specify whether Python prints a warning on stderr when the wakeup buffer overflows. (Contributed by Nathaniel J. Smith in 30050.)

socket

The new socket.getblocking() <socket.socket.getblocking> method returns True if the socket is in blocking mode and False otherwise. (Contributed by Yury Selivanov in 32373.)

The new socket.close function closes the passed socket file descriptor. This function should be used instead of os.close for better compatibility across platforms. (Contributed by Christian Heimes in 32454.)

The socket module now exposes the socket.TCP_CONGESTION <socket-unix-constants> (Linux 2.6.13), socket.TCP_USER_TIMEOUT <socket-unix-constants> (Linux 2.6.37), and socket.TCP_NOTSENT_LOWAT <socket-unix-constants> (Linux 3.12) constants. (Contributed by Omar Sandoval in 26273 and Nathaniel J. Smith in 29728.)

Support for socket.AF_VSOCK sockets has been added to allow communication between virtual machines and their hosts. (Contributed by Cathy Avery in 27584.)

Sockets now auto-detect family, type and protocol from file descriptor by default. (Contributed by Christian Heimes in 28134.)

socketserver

socketserver.ThreadingMixIn.server_close <socketserver.BaseServer.server_close> now waits until all non-daemon threads complete. socketserver.ForkingMixIn.server_close <socketserver.BaseServer.server_close> now waits until all child processes complete.

Add a new socketserver.ForkingMixIn.block_on_close <socketserver.ThreadingMixIn.block_on_close> class attribute to socketserver.ForkingMixIn and socketserver.ThreadingMixIn classes. Set the class attribute to False to get the pre-3.7 behaviour.

sqlite3

sqlite3.Connection now exposes the ~sqlite3.Connection.backup method when the underlying SQLite library is at version 3.6.11 or higher. (Contributed by Lele Gaifax in 27645.)

The database argument of sqlite3.connect now accepts any path-like object, instead of just a string. (Contributed by Anders Lorentsen in 31843.)

ssl

The ssl module now uses OpenSSL's builtin API instead of !match_hostname to check a host name or an IP address. Values are validated during TLS handshake. Any certificate validation error including failing the host name check now raises ~ssl.SSLCertVerificationError and aborts the handshake with a proper TLS Alert message. The new exception contains additional information. Host name validation can be customized with SSLContext.hostname_checks_common_name <ssl.SSLContext.hostname_checks_common_name>. (Contributed by Christian Heimes in 31399.)

[!NOTE] The improved host name check requires a libssl implementation compatible with OpenSSL 1.0.2 or 1.1. Consequently, OpenSSL 0.9.8 and 1.0.1 are no longer supported (see 37-platform-support-removals for more details). The ssl module is mostly compatible with LibreSSL 2.7.2 and newer.

The ssl module no longer sends IP addresses in SNI TLS extension. (Contributed by Christian Heimes in 32185.)

!match_hostname no longer supports partial wildcards like www*.example.org. (Contributed by Mandeep Singh in 23033 and Christian Heimes in 31399.)

The default cipher suite selection of the ssl module now uses a blacklist approach rather than a hard-coded whitelist. Python no longer re-enables ciphers that have been blocked by OpenSSL security updates. Default cipher suite selection can be configured at compile time. (Contributed by Christian Heimes in 31429.)

Validation of server certificates containing internationalized domain names (IDNs) is now supported. As part of this change, the SSLSocket.server_hostname <ssl.SSLSocket.server_hostname> attribute now stores the expected hostname in A-label form ("xn--pythn-mua.org"), rather than the U-label form ("pythön.org"). (Contributed by Nathaniel J. Smith and Christian Heimes in 28414.)

The ssl module has preliminary and experimental support for TLS 1.3 and OpenSSL 1.1.1. At the time of Python 3.7.0 release, OpenSSL 1.1.1 is still under development and TLS 1.3 hasn't been finalized yet. The TLS 1.3 handshake and protocol behaves slightly differently than TLS 1.2 and earlier, see ssl-tlsv1_3. (Contributed by Christian Heimes in 32947, 20995, 29136, 30622 and 33618)

~ssl.SSLSocket and ~ssl.SSLObject no longer have a public constructor. Direct instantiation was never a documented and supported feature. Instances must be created with ~ssl.SSLContext methods ~ssl.SSLContext.wrap_socket and ~ssl.SSLContext.wrap_bio. (Contributed by Christian Heimes in 32951)

OpenSSL 1.1 APIs for setting the minimum and maximum TLS protocol version are available as SSLContext.minimum_version <ssl.SSLContext.minimum_version> and SSLContext.maximum_version <ssl.SSLContext.maximum_version>. Supported protocols are indicated by several new flags, such as ~ssl.HAS_TLSv1_1. (Contributed by Christian Heimes in 32609.)

Added ssl.SSLContext.post_handshake_auth to enable and ssl.SSLSocket.verify_client_post_handshake to initiate TLS 1.3 post-handshake authentication. (Contributed by Christian Heimes in 78851.)

string

string.Template now lets you to optionally modify the regular expression pattern for braced placeholders and non-braced placeholders separately. (Contributed by Barry Warsaw in 1198569.)

subprocess

The subprocess.run function accepts the new capture_output keyword argument. When true, stdout and stderr will be captured. This is equivalent to passing subprocess.PIPE as stdout and stderr arguments. (Contributed by Bo Bayles in 32102.)

The subprocess.run function and the subprocess.Popen constructor now accept the text keyword argument as an alias to universal_newlines. (Contributed by Andrew Clegg in 31756.)

On Windows the default for close_fds was changed from False to True when redirecting the standard handles. It's now possible to set close_fds to true when redirecting the standard handles. See subprocess.Popen. This means that close_fds now defaults to True on all supported platforms. (Contributed by Segev Finer in 19764.)

The subprocess module is now more graceful when handling KeyboardInterrupt during subprocess.call, subprocess.run, or in a ~subprocess.Popen context manager. It now waits a short amount of time for the child to exit, before continuing the handling of the KeyboardInterrupt exception. (Contributed by Gregory P. Smith in 25942.)

sys

The new sys.breakpointhook hook function is called by the built-in breakpoint. (Contributed by Barry Warsaw in 31353.)

On Android, the new sys.getandroidapilevel returns the build-time Android API version. (Contributed by Victor Stinner in 28740.)

The new sys.get_coroutine_origin_tracking_depth function returns the current coroutine origin tracking depth, as set by the new sys.set_coroutine_origin_tracking_depth. asyncio has been converted to use this new API instead of the deprecated !sys.set_coroutine_wrapper. (Contributed by Nathaniel J. Smith in 32591.)

time

564 adds six new functions with nanosecond resolution to the time module:

New clock identifiers have been added:

The new time.thread_time and time.thread_time_ns functions can be used to get per-thread CPU time measurements. (Contributed by Antoine Pitrou in 32025.)

The new time.pthread_getcpuclockid function returns the clock ID of the thread-specific CPU-time clock.

tkinter

The new tkinter.ttk.Spinbox class is now available. (Contributed by Alan Moore in 32585.)

tracemalloc

tracemalloc.Traceback behaves more like regular tracebacks, sorting the frames from oldest to most recent. Traceback.format() <tracemalloc.Traceback.format> now accepts negative limit, truncating the result to the abs(limit) oldest frames. To get the old behaviour, use the new most_recent_first argument to Traceback.format(). (Contributed by Jesse Bakker in 32121.)

types

The new ~types.WrapperDescriptorType, ~types.MethodWrapperType, ~types.MethodDescriptorType, and ~types.ClassMethodDescriptorType classes are now available. (Contributed by Manuel Krebber and Guido van Rossum in 29377, and Serhiy Storchaka in 32265.)

The new types.resolve_bases function resolves MRO entries dynamically as specified by 560. (Contributed by Ivan Levkivskyi in 32717.)

unicodedata

The internal unicodedata database has been upgraded to use Unicode 11. (Contributed by Benjamin Peterson.)

unittest

The new -k command-line option allows filtering tests by a name substring or a Unix shell-like pattern. For example, python -m unittest -k foo runs foo_tests.SomeTest.test_something, bar_tests.SomeTest.test_foo, but not bar_tests.FooTest.test_something. (Contributed by Jonas Haag in 32071.)

unittest.mock

The ~unittest.mock.sentinel attributes now preserve their identity when they are copied <copy> or pickled <pickle>. (Contributed by Serhiy Storchaka in 20804.)

The new ~unittest.mock.seal function allows sealing ~unittest.mock.Mock instances, which will disallow further creation of attribute mocks. The seal is applied recursively to all attributes that are themselves mocks. (Contributed by Mario Corchero in 30541.)

urllib.parse

urllib.parse.quote has been updated from 2396 to 3986, adding ~ to the set of characters that are never quoted by default. (Contributed by Christian Theune and Ratnadeep Debnath in 16285.)

uu

The !uu.encode function now accepts an optional backtick keyword argument. When it's true, zeros are represented by '`' instead of spaces. (Contributed by Xiang Zhang in 30103.)

uuid

The new UUID.is_safe <uuid.UUID.is_safe> attribute relays information from the platform about whether generated UUIDs are generated with a multiprocessing-safe method. (Contributed by Barry Warsaw in 22807.)

uuid.getnode now prefers universally administered MAC addresses over locally administered MAC addresses. This makes a better guarantee for global uniqueness of UUIDs returned from uuid.uuid1. If only locally administered MAC addresses are available, the first such one found is returned. (Contributed by Barry Warsaw in 32107.)

warnings

The initialization of the default warnings filters has changed as follows:

(Contributed by Nick Coghlan and Victor Stinner in 20361, 32043, and 32230.)

Deprecation warnings are once again shown by default in single-file scripts and at the interactive prompt. See whatsnew37-pep565 for details. (Contributed by Nick Coghlan in 31975.)

xml

As mitigation against DTD and external entity retrieval, the xml.dom.minidom and xml.sax modules no longer process external entities by default. (Contributed by Christian Heimes in 61441.)

xml.etree

ElementPath <elementtree-xpath> predicates in the !find methods can now compare text of the current node with [. = "text"], not only text in children. Predicates also allow adding spaces for better readability. (Contributed by Stefan Behnel in 31648.)

xmlrpc.server

!SimpleXMLRPCDispatcher.register_function can now be used as a decorator. (Contributed by Xiang Zhang in 7769.)

zipapp

Function ~zipapp.create_archive now accepts an optional filter argument to allow the user to select which files should be included in the archive. (Contributed by Irmen de Jong in 31072.)

Function ~zipapp.create_archive now accepts an optional compressed argument to generate a compressed archive. A command line option --compress has also been added to support compression. (Contributed by Zhiming Wang in 31638.)

zipfile

~zipfile.ZipFile now accepts the new compresslevel parameter to control the compression level. (Contributed by Bo Bayles in 21417.)

Subdirectories in archives created by ZipFile are now stored in alphabetical order. (Contributed by Bernhard M. Wiedemann in 30693.)

C API Changes

A new API for thread-local storage has been implemented. See whatsnew37-pep539 for an overview and thread-specific-storage-api for a complete reference. (Contributed by Masayuki Yamamoto in 25658.)

The new context variables <whatsnew37-pep567> functionality exposes a number of new C APIs <contextvarsobjects>.

The new PyImport_GetModule function returns the previously imported module with the given name. (Contributed by Eric Snow in 28411.)

The new Py_RETURN_RICHCOMPARE macro eases writing rich comparison functions. (Contributed by Petr Victorin in 23699.)

The new Py_UNREACHABLE macro can be used to mark unreachable code paths. (Contributed by Barry Warsaw in 31338.)

The tracemalloc now exposes a C API through the new PyTraceMalloc_Track and PyTraceMalloc_Untrack functions. (Contributed by Victor Stinner in 30054.)

The new import__find__load__start <static-markers> and import__find__load__done <static-markers> static markers can be used to trace module imports. (Contributed by Christian Heimes in 31574.)

The fields !name and !doc of structures PyMemberDef, PyGetSetDef, PyStructSequence_Field, PyStructSequence_Desc, and !wrapperbase are now of type const char * rather of char *. (Contributed by Serhiy Storchaka in 28761.)

The result of PyUnicode_AsUTF8AndSize and PyUnicode_AsUTF8 is now of type const char * rather of char *. (Contributed by Serhiy Storchaka in 28769.)

The result of PyMapping_Keys, PyMapping_Values and PyMapping_Items is now always a list, rather than a list or a tuple. (Contributed by Oren Milman in 28280.)

Added functions PySlice_Unpack and PySlice_AdjustIndices. (Contributed by Serhiy Storchaka in 27867.)

PyOS_AfterFork is deprecated in favour of the new functions PyOS_BeforeFork, PyOS_AfterFork_Parent and PyOS_AfterFork_Child. (Contributed by Antoine Pitrou in 16500.)

The PyExc_RecursionErrorInst singleton that was part of the public API has been removed as its members being never cleared may cause a segfault during finalization of the interpreter. Contributed by Xavier de Gaye in 22898 and 30697.

Added C API support for timezones with timezone constructors PyTimeZone_FromOffset and PyTimeZone_FromOffsetAndName, and access to the UTC singleton with PyDateTime_TimeZone_UTC. Contributed by Paul Ganssle in 10381.

The type of results of !PyThread_start_new_thread and !PyThread_get_thread_ident, and the id parameter of PyThreadState_SetAsyncExc changed from long to unsigned long. (Contributed by Serhiy Storchaka in 6532.)

PyUnicode_AsWideCharString now raises a ValueError if the second argument is NULL and the wchar_t* string contains null characters. (Contributed by Serhiy Storchaka in 30708.)

Changes to the startup sequence and the management of dynamic memory allocators mean that the long documented requirement to call Py_Initialize before calling most C API functions is now relied on more heavily, and failing to abide by it may lead to segfaults in embedding applications. See the porting-to-python-37 section in this document and the pre-init-safe section in the C API documentation for more details.

The new PyInterpreterState_GetID returns the unique ID for a given interpreter. (Contributed by Eric Snow in 29102.)

Py_DecodeLocale, Py_EncodeLocale now use the UTF-8 encoding when the UTF-8 mode <whatsnew37-pep540> is enabled. (Contributed by Victor Stinner in 29240.)

PyUnicode_DecodeLocaleAndSize and PyUnicode_EncodeLocale now use the current locale encoding for surrogateescape error handler. (Contributed by Victor Stinner in 29240.)

The start and end parameters of PyUnicode_FindChar are now adjusted to behave like string slices. (Contributed by Xiang Zhang in 28822.)

Build Changes

Support for building --without-threads has been removed. The threading module is now always available. (Contributed by Antoine Pitrou in 31370.).

A full copy of libffi is no longer bundled for use when building the _ctypes <ctypes> module on non-OSX UNIX platforms. An installed copy of libffi is now required when building _ctypes on such platforms. (Contributed by Zachary Ware in 27979.)

The Windows build process no longer depends on Subversion to pull in external sources, a Python script is used to download zipfiles from GitHub instead. If Python 3.6 is not found on the system (via py -3.6), NuGet is used to download a copy of 32-bit Python for this purpose. (Contributed by Zachary Ware in 30450.)

The ssl module requires OpenSSL 1.0.2 or 1.1 compatible libssl. OpenSSL 1.0.1 has reached end of lifetime on 2016-12-31 and is no longer supported. LibreSSL is temporarily not supported as well. LibreSSL releases up to version 2.6.4 are missing required OpenSSL 1.0.2 APIs.

Optimizations

The overhead of calling many methods of various standard library classes implemented in C has been significantly reduced by porting more code to use the METH_FASTCALL convention. (Contributed by Victor Stinner in 29300, 29507, 29452, and 29286.)

Various optimizations have reduced Python startup time by 10% on Linux and up to 30% on macOS. (Contributed by Victor Stinner, INADA Naoki in 29585, and Ivan Levkivskyi in 31333.)

Method calls are now up to 20% faster due to the bytecode changes which avoid creating bound method instances. (Contributed by Yury Selivanov and INADA Naoki in 26110.)

The asyncio module received a number of notable optimizations for commonly used functions:

As a result of PEP 560 work <whatsnew37-pep560>, the import time of typing has been reduced by a factor of 7, and many typing operations are now faster. (Contributed by Ivan Levkivskyi in 32226.)

sorted and list.sort have been optimized for common cases to be up to 40-75% faster. (Contributed by Elliot Gorokhovsky in 28685.)

dict.copy is now up to 5.5 times faster. (Contributed by Yury Selivanov in 31179.)

hasattr and getattr are now about 4 times faster when name is not found and obj does not override object.__getattr__ or object.__getattribute__. (Contributed by INADA Naoki in 32544.)

Searching for certain Unicode characters (like Ukrainian capital "Є") in a string was up to 25 times slower than searching for other characters. It is now only 3 times slower in the worst case. (Contributed by Serhiy Storchaka in 24821.)

The collections.namedtuple factory has been reimplemented to make the creation of named tuples 4 to 6 times faster. (Contributed by Jelle Zijlstra with further improvements by INADA Naoki, Serhiy Storchaka, and Raymond Hettinger in 28638.)

datetime.date.fromordinal and datetime.date.fromtimestamp are now up to 30% faster in the common case. (Contributed by Paul Ganssle in 32403.)

The os.fwalk function is now up to 2 times faster thanks to the use of os.scandir. (Contributed by Serhiy Storchaka in 25996.)

The speed of the shutil.rmtree function has been improved by 20--40% thanks to the use of the os.scandir function. (Contributed by Serhiy Storchaka in 28564.)

Optimized case-insensitive matching and searching of regular expressions <re>. Searching some patterns can now be up to 20 times faster. (Contributed by Serhiy Storchaka in 30285.)

re.compile now converts flags parameter to int object if it is RegexFlag. It is now as fast as Python 3.5, and faster than Python 3.6 by about 10% depending on the pattern. (Contributed by INADA Naoki in 31671.)

The ~selectors.BaseSelector.modify methods of classes selectors.EpollSelector, selectors.PollSelector and selectors.DevpollSelector may be around 10% faster under heavy loads. (Contributed by Giampaolo Rodola' in 30014)

Constant folding has been moved from the peephole optimizer to the new AST optimizer, which is able perform optimizations more consistently. (Contributed by Eugene Toder and INADA Naoki in 29469 and 11549.)

Most functions and methods in abc have been rewritten in C. This makes creation of abstract base classes, and calling isinstance and issubclass on them 1.5x faster. This also reduces Python start-up time by up to 10%. (Contributed by Ivan Levkivskyi and INADA Naoki in 31333)

Significant speed improvements to alternate constructors for datetime.date and datetime.datetime by using fast-path constructors when not constructing subclasses. (Contributed by Paul Ganssle in 32403)

The speed of comparison of array.array instances has been improved considerably in certain cases. It is now from 10x to 70x faster when comparing arrays holding values of the same integer type. (Contributed by Adrian Wielgosik in 24700.)

The math.erf and math.erfc functions now use the (faster) C library implementation on most platforms. (Contributed by Serhiy Storchaka in 26121.)

Other CPython Implementation Changes

Deprecated Python Behavior

Yield expressions (both yield and yield from clauses) are now deprecated in comprehensions and generator expressions (aside from the iterable expression in the leftmost !for clause). This ensures that comprehensions always immediately return a container of the appropriate type (rather than potentially returning a generator iterator object), while generator expressions won't attempt to interleave their implicit output with the output from any explicit yield expressions. In Python 3.7, such expressions emit DeprecationWarning when compiled, in Python 3.8 this will be a SyntaxError. (Contributed by Serhiy Storchaka in 10544.)

Returning a subclass of complex from object.__complex__ is deprecated and will be an error in future Python versions. This makes __complex__() consistent with object.__int__ and object.__float__. (Contributed by Serhiy Storchaka in 28894.)

Deprecated Python modules, functions and methods

aifc

!aifc.openfp has been deprecated and will be removed in Python 3.9. Use !aifc.open instead. (Contributed by Brian Curtin in 31985.)

asyncio

Support for directly await-ing instances of asyncio.Lock and other asyncio synchronization primitives has been deprecated. An asynchronous context manager must be used in order to acquire and release the synchronization resource. (Contributed by Andrew Svetlov in 32253.)

The !asyncio.Task.current_task and !asyncio.Task.all_tasks methods have been deprecated. (Contributed by Andrew Svetlov in 32250.)

collections

In Python 3.8, the abstract base classes in collections.abc will no longer be exposed in the regular collections module. This will help create a clearer distinction between the concrete classes and the abstract base classes. (Contributed by Serhiy Storchaka in 25988.)

dbm

dbm.dumb now supports reading read-only files and no longer writes the index file when it is not changed. A deprecation warning is now emitted if the index file is missing and recreated in the 'r' and 'w' modes (this will be an error in future Python releases). (Contributed by Serhiy Storchaka in 28847.)

enum

In Python 3.8, attempting to check for non-Enum objects in ~enum.Enum classes will raise a TypeError (e.g. 1 in Color); similarly, attempting to check for non-Flag objects in a ~enum.Flag member will raise TypeError (e.g. 1 in Perm.RW); currently, both operations return False instead. (Contributed by Ethan Furman in 33217.)

gettext

Using non-integer value for selecting a plural form in gettext is now deprecated. It never correctly worked. (Contributed by Serhiy Storchaka in 28692.)

importlib

Methods !MetaPathFinder.find_module (replaced by MetaPathFinder.find_spec() <importlib.abc.MetaPathFinder.find_spec>) and !PathEntryFinder.find_loader (replaced by PathEntryFinder.find_spec() <importlib.abc.PathEntryFinder.find_spec>) both deprecated in Python 3.4 now emit DeprecationWarning. (Contributed by Matthias Bussonnier in 29576.)

The importlib.abc.ResourceLoader ABC has been deprecated in favour of !importlib.abc.ResourceReader.

locale

!locale.format has been deprecated, use locale.format_string instead. (Contributed by Garvit in 10379.)

macpath

The !macpath is now deprecated and will be removed in Python 3.8. (Contributed by Chi Hsuan Yen in 9850.)

threading

!dummy_threading and !_dummy_thread have been deprecated. It is no longer possible to build Python with threading disabled. Use threading instead. (Contributed by Antoine Pitrou in 31370.)

socket

The silent argument value truncation in socket.htons and socket.ntohs has been deprecated. In future versions of Python, if the passed argument is larger than 16 bits, an exception will be raised. (Contributed by Oren Milman in 28332.)

ssl

!ssl.wrap_socket is deprecated. Use ssl.SSLContext.wrap_socket instead. (Contributed by Christian Heimes in 28124.)

sunau

!sunau.openfp has been deprecated and will be removed in Python 3.9. Use !sunau.open instead. (Contributed by Brian Curtin in 31985.)

sys

Deprecated !sys.set_coroutine_wrapper and !sys.get_coroutine_wrapper.

The undocumented sys.callstats() function has been deprecated and will be removed in a future Python version. (Contributed by Victor Stinner in 28799.)

wave

!wave.openfp has been deprecated and will be removed in Python 3.9. Use wave.open instead. (Contributed by Brian Curtin in 31985.)

Deprecated functions and types of the C API

Function PySlice_GetIndicesEx is deprecated and replaced with a macro if Py_LIMITED_API is not set or set to a value in the range between 0x03050400 and 0x03060000 (not inclusive), or is 0x03060100 or higher. (Contributed by Serhiy Storchaka in 27867.)

PyOS_AfterFork has been deprecated. Use PyOS_BeforeFork, PyOS_AfterFork_Parent or PyOS_AfterFork_Child() instead. (Contributed by Antoine Pitrou in 16500.)

Platform Support Removals

API and Feature Removals

The following features and APIs have been removed from Python 3.7:

Module Removals

The fpectl module has been removed. It was never enabled by default, never worked correctly on x86-64, and it changed the Python ABI in ways that caused unexpected breakage of C extensions. (Contributed by Nathaniel J. Smith in 29137.)

Windows-only Changes

The python launcher, (py.exe), can accept 32 & 64 bit specifiers without having to specify a minor version as well. So py -3-32 and py -3-64 become valid as well as py -3.7-32, also the -m-64 and -m.n-64 forms are now accepted to force 64 bit python even if 32 bit would have otherwise been used. If the specified version is not available py.exe will error exit. (Contributed by Steve Barnes in 30291.)

The launcher can be run as py -0 to produce a list of the installed pythons, with default marked with an asterisk. Running py -0p will include the paths. If py is run with a version specifier that cannot be matched it will also print the short form list of available specifiers. (Contributed by Steve Barnes in 30362.)

Porting to Python 3.7

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in Python Behavior

Changes in the Python API

Changes in the C API

The function PySlice_GetIndicesEx is considered unsafe for resizable sequences. If the slice indices are not instances of int, but objects that implement the !__index__ method, the sequence can be resized after passing its length to !PySlice_GetIndicesEx. This can lead to returning indices out of the length of the sequence. For avoiding possible problems use new functions PySlice_Unpack and PySlice_AdjustIndices. (Contributed by Serhiy Storchaka in 27867.)

CPython bytecode changes

There are two new opcodes: !LOAD_METHOD and !CALL_METHOD. (Contributed by Yury Selivanov and INADA Naoki in 26110.)

The !STORE_ANNOTATION opcode has been removed. (Contributed by Mark Shannon in 32550.)

Windows-only Changes

The file used to override sys.path is now called <python-executable>._pth instead of 'sys.path'. See windows_finding_modules for more information. (Contributed by Steve Dower in 28137.)

Other CPython implementation changes

In preparation for potential future changes to the public CPython runtime initialization API (see 432 for an initial, but somewhat outdated, draft), CPython's internal startup and configuration management logic has been significantly refactored. While these updates are intended to be entirely transparent to both embedding applications and users of the regular CPython CLI, they're being mentioned here as the refactoring changes the internal order of various operations during interpreter startup, and hence may uncover previously latent defects, either in embedding applications, or in CPython itself. (Initially contributed by Nick Coghlan and Eric Snow as part of 22257, and further updated by Nick, Eric, and Victor Stinner in a number of other issues). Some known details affected:

Due to changes in the way the default warnings filters are configured, setting !Py_BytesWarningFlag to a value greater than one is no longer sufficient to both emit BytesWarning messages and have them converted to exceptions. Instead, the flag must be set (to cause the warnings to be emitted in the first place), and an explicit error::BytesWarning warnings filter added to convert them to exceptions.

Due to a change in the way docstrings are handled by the compiler, the implicit return None in a function body consisting solely of a docstring is now marked as occurring on the same line as the docstring, not on the function's header line.

The current exception state has been moved from the frame object to the co-routine. This simplified the interpreter and fixed a couple of obscure bugs caused by having swap exception state when entering or exiting a generator. (Contributed by Mark Shannon in 25612.)

Notable changes in Python 3.7.1

Starting in 3.7.1, Py_Initialize now consistently reads and respects all of the same environment settings as Py_Main (in earlier Python versions, it respected an ill-defined subset of those environment variables, while in Python 3.7.0 it didn't read any of them due to 34247). If this behavior is unwanted, set !Py_IgnoreEnvironmentFlag to 1 before calling Py_Initialize.

In 3.7.1 the C API for Context Variables was updated <contextvarsobjects_pointertype_change> to use PyObject pointers. See also 34762.

In 3.7.1 the tokenize module now implicitly emits a NEWLINE token when provided with input that does not have a trailing new line. This behavior now matches what the C tokenizer does internally. (Contributed by Ammar Askar in 33899.)

Notable changes in Python 3.7.2

In 3.7.2, venv on Windows no longer copies the original binaries, but creates redirector scripts named python.exe and pythonw.exe instead. This resolves a long standing issue where all virtual environments would have to be upgraded or recreated with each Python update. However, note that this release will still require recreation of virtual environments in order to get the new scripts.

Notable changes in Python 3.7.6

Due to significant security concerns, the reuse_address parameter of asyncio.loop.create_datagram_endpoint is no longer supported. This is because of the behavior of the socket option SO_REUSEADDR in UDP. For more details, see the documentation for loop.create_datagram_endpoint(). (Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in 37228.)

Notable changes in Python 3.7.10

Earlier Python versions allowed using both ; and & as query parameter separators in urllib.parse.parse_qs and urllib.parse.parse_qsl. Due to security concerns, and to conform with newer W3C recommendations, this has been changed to allow only a single separator key, with & as the default. This change also affects !cgi.parse and !cgi.parse_multipart as they use the affected functions internally. For more details, please see their respective documentation. (Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in 42967.)

Notable changes in Python 3.7.11

A security fix alters the ftplib.FTP behavior to not trust the IPv4 address sent from the remote server when setting up a passive data channel. We reuse the ftp server IP address instead. For unusual code requiring the old behavior, set a trust_server_pasv_ipv4_address attribute on your FTP instance to True. (See 87451)

The presence of newline or tab characters in parts of a URL allows for some forms of attacks. Following the WHATWG specification that updates RFC 3986, ASCII newline \n, \r and tab \t characters are stripped from the URL by the parser urllib.parse preventing such attacks. The removal characters are controlled by a new module level variable urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE. (See 88048)

Notable security feature in 3.7.14

Converting between int and str in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a ValueError if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for 2020-10735. This limit can be configured or disabled by environment variable, command line flag, or sys APIs. See the integer string conversion length limitation <int_max_str_digits> documentation. The default limit is 4300 digits in string form.

What's New In Python 3.8

* Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.)

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the Git log when researching a change.

Editor
Raymond Hettinger

This article explains the new features in Python 3.8, compared to 3.7. Python 3.8 was released on October 14, 2019. For full details, see the changelog <changelog>.

from math import cos, radians from unicodedata import normalize import re import math

Summary -- Release highlights

New Features

Assignment expressions

There is new syntax := that assigns values to variables as part of a larger expression. It is affectionately known as "the walrus operator" due to its resemblance to the eyes and tusks of a walrus.

In this example, the assignment expression helps avoid calling len twice:

if (n := len(a)) > 10:
    print(f"List is too long ({n} elements, expected <= 10)")

A similar benefit arises during regular expression matching where match objects are needed twice, once to test whether a match occurred and another to extract a subgroup:

discount = 0.0
if (mo := re.search(r'(\d+)% discount', advertisement)):
    discount = float(mo.group(1)) / 100.0

The operator is also useful with while-loops that compute a value to test loop termination and then need that same value again in the body of the loop:

# Loop over fixed length blocks
while (block := f.read(256)) != '':
    process(block)

Another motivating use case arises in list comprehensions where a value computed in a filtering condition is also needed in the expression body:

[clean_name.title() for name in names
 if (clean_name := normalize('NFC', name)) in allowed_names]

Try to limit use of the walrus operator to clean cases that reduce complexity and improve readability.

See 572 for a full description.

(Contributed by Emily Morehouse in 35224.)

Positional-only parameters

There is a new function parameter syntax / to indicate that some function parameters must be specified positionally and cannot be used as keyword arguments. This is the same notation shown by help() for C functions annotated with Larry Hastings' Argument Clinic tool.

In the following example, parameters a and b are positional-only, while c or d can be positional or keyword, and e or f are required to be keywords:

def f(a, b, /, c, d, *, e, f):
    print(a, b, c, d, e, f)

The following is a valid call:

f(10, 20, 30, d=40, e=50, f=60)

However, these are invalid calls:

f(10, b=20, c=30, d=40, e=50, f=60)   # b cannot be a keyword argument
f(10, 20, 30, 40, 50, f=60)           # e must be a keyword argument

One use case for this notation is that it allows pure Python functions to fully emulate behaviors of existing C coded functions. For example, the built-in divmod function does not accept keyword arguments:

def divmod(a, b, /):
    "Emulate the built in divmod() function"
    return (a // b, a % b)

Another use case is to preclude keyword arguments when the parameter name is not helpful. For example, the builtin len function has the signature len(obj, /). This precludes awkward calls such as:

len(obj='hello')  # The "obj" keyword argument impairs readability

A further benefit of marking a parameter as positional-only is that it allows the parameter name to be changed in the future without risk of breaking client code. For example, in the statistics module, the parameter name dist may be changed in the future. This was made possible with the following function specification:

def quantiles(dist, /, *, n=4, method='exclusive'):
    ...

Since the parameters to the left of / are not exposed as possible keywords, the parameters names remain available for use in **kwargs:

>>> def f(a, b, /, **kwargs):
...     print(a, b, kwargs)
...
>>> f(10, 20, a=1, b=2, c=3)         # a and b are used in two ways
10 20 {'a': 1, 'b': 2, 'c': 3}

This greatly simplifies the implementation of functions and methods that need to accept arbitrary keyword arguments. For example, here is an excerpt from code in the collections module:

class Counter(dict):

    def __init__(self, iterable=None, /, **kwds):
        # Note "iterable" is a possible keyword argument

See 570 for a full description.

(Contributed by Pablo Galindo in 36540.)

Parallel filesystem cache for compiled bytecode files

The new PYTHONPYCACHEPREFIX setting (also available as -X pycache_prefix) configures the implicit bytecode cache to use a separate parallel filesystem tree, rather than the default __pycache__ subdirectories within each source directory.

The location of the cache is reported in sys.pycache_prefix (None indicates the default location in __pycache__ subdirectories).

(Contributed by Carl Meyer in 33499.)

Debug build uses the same ABI as release build

The ABI of Python debug builds <debug-build> is now compatible with Python release builds. On Unix, when Python is built in debug mode, it is now possible to load C extensions built in release mode and C extensions built using the stable ABI. The inverse is not true, as debug builds expose additional symbols not available in release builds.

Defining the Py_DEBUG macro no longer implies the Py_TRACE_REFS macro, which introduces the only ABI incompatibility. The Py_TRACE_REFS macro, which adds the sys.getobjects function and the PYTHONDUMPREFS environment variable, can be set using the new ./configure --with-trace-refs <--with-trace-refs> build option. (Contributed by Victor Stinner in 36465.)

On Unix, C extensions are no longer linked to libpython except on Android and Cygwin. It is now possible for a statically linked Python to load a C extension built using a shared library Python. (Contributed by Victor Stinner in 21536.)

On Unix, when Python is built in debug mode, import now also looks for C extensions compiled in release mode and for C extensions compiled with the stable ABI. (Contributed by Victor Stinner in 36722.)

To embed Python into an application, a new --embed option must be passed to python3-config --libs --embed to get -lpython3.8 (link the application to libpython). To support both 3.8 and older, try python3-config --libs --embed first and fallback to python3-config --libs (without --embed) if the previous command fails.

Add a pkg-config python-3.8-embed module to embed Python into an application: pkg-config python-3.8-embed --libs includes -lpython3.8. To support both 3.8 and older, try pkg-config python-X.Y-embed --libs first and fallback to pkg-config python-X.Y --libs (without --embed) if the previous command fails (replace X.Y with the Python version).

On the other hand, pkg-config python3.8 --libs no longer contains -lpython3.8. C extensions must not be linked to libpython (except on Android and Cygwin, whose cases are handled by the script); this change is backward incompatible on purpose. (Contributed by Victor Stinner in 36721.)

f-strings support = for self-documenting expressions and debugging

Added an = specifier to f-strings. An f-string such as f'{expr=}' will expand to the text of the expression, an equal sign, then the representation of the evaluated expression. For example:

>>> import datetime as dt >>> user = 'eric_idle' >>> member_since = dt.date(1975, 7, 31) >>> f'{user=} {member_since=}' "user='eric_idle' member_since=datetime.date(1975, 7, 31)"

The usual f-string format specifiers <f-strings> allow more control over how the result of the expression is displayed:

>>> delta = dt.date.today() - member_since
>>> f'{user=!s}  {delta.days=:,d}'
'user=eric_idle  delta.days=16,075'

The = specifier will display the whole expression so that calculations can be shown:

>>> print(f'{theta=}  {cos(radians(theta))=:.3f}')
theta=30  cos(radians(theta))=0.866

(Contributed by Eric V. Smith and Larry Hastings in 36817.)

PEP 578: Python Runtime Audit Hooks

The PEP adds an Audit Hook and Verified Open Hook. Both are available from Python and native code, allowing applications and frameworks written in pure Python code to take advantage of extra notifications, while also allowing embedders or system administrators to deploy builds of Python where auditing is always enabled.

See 578 for full details.

PEP 587: Python Initialization Configuration

The 587 adds a new C API to configure the Python Initialization providing finer control on the whole configuration and better error reporting.

New structures:

New functions:

This PEP also adds _PyRuntimeState.preconfig (PyPreConfig type) and PyInterpreterState.config (PyConfig type) fields to these internal structures. PyInterpreterState.config becomes the new reference configuration, replacing global configuration variables and other private variables.

See Python Initialization Configuration <init-config> for the documentation.

See 587 for a full description.

(Contributed by Victor Stinner in 36763.)

PEP 590: Vectorcall: a fast calling protocol for CPython

vectorcall is added to the Python/C API. It is meant to formalize existing optimizations which were already done for various classes. Any static type <static-types> implementing a callable can use this protocol.

This is currently provisional. The aim is to make it fully public in Python 3.9.

See 590 for a full description.

(Contributed by Jeroen Demeyer, Mark Shannon and Petr Viktorin in 36974.)

Pickle protocol 5 with out-of-band data buffers

When pickle is used to transfer large data between Python processes in order to take advantage of multi-core or multi-machine processing, it is important to optimize the transfer by reducing memory copies, and possibly by applying custom techniques such as data-dependent compression.

The pickle protocol 5 introduces support for out-of-band buffers where 3118-compatible data can be transmitted separately from the main pickle stream, at the discretion of the communication layer.

See 574 for a full description.

(Contributed by Antoine Pitrou in 36785.)

Other Language Changes

New Modules

Improved Modules

ast

AST nodes now have end_lineno and end_col_offset attributes, which give the precise location of the end of the node. (This only applies to nodes that have lineno and col_offset attributes.)

New function ast.get_source_segment returns the source code for a specific AST node.

(Contributed by Ivan Levkivskyi in 33416.)

The ast.parse function has some new flags:

(Contributed by Guido van Rossum in 35766.)

asyncio

asyncio.run has graduated from the provisional to stable API. This function can be used to execute a coroutine and return the result while automatically managing the event loop. For example:

import asyncio

async def main():
    await asyncio.sleep(0)
    return 42

asyncio.run(main())

This is roughly equivalent to:

import asyncio

async def main():
    await asyncio.sleep(0)
    return 42

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
    loop.run_until_complete(main())
finally:
    asyncio.set_event_loop(None)
    loop.close()

The actual implementation is significantly more complex. Thus, asyncio.run should be the preferred way of running asyncio programs.

(Contributed by Yury Selivanov in 32314.)

Running python -m asyncio launches a natively async REPL. This allows rapid experimentation with code that has a top-level await. There is no longer a need to directly call asyncio.run() which would spawn a new event loop on every invocation:

$ python -m asyncio
asyncio REPL 3.8.0
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(10, result='hello')
hello

(Contributed by Yury Selivanov in 37028.)

The exception asyncio.CancelledError now inherits from BaseException rather than Exception and no longer inherits from concurrent.futures.CancelledError. (Contributed by Yury Selivanov in 32528.)

On Windows, the default event loop is now ~asyncio.ProactorEventLoop. (Contributed by Victor Stinner in 34687.)

~asyncio.ProactorEventLoop now also supports UDP. (Contributed by Adam Meily and Andrew Svetlov in 29883.)

~asyncio.ProactorEventLoop can now be interrupted by KeyboardInterrupt ("CTRL+C"). (Contributed by Vladimir Matveev in 23057.)

Added asyncio.Task.get_coro for getting the wrapped coroutine within an asyncio.Task. (Contributed by Alex Grönholm in 36999.)

Asyncio tasks can now be named, either by passing the name keyword argument to asyncio.create_task or the ~asyncio.loop.create_task event loop method, or by calling the ~asyncio.Task.set_name method on the task object. The task name is visible in the repr() output of asyncio.Task and can also be retrieved using the ~asyncio.Task.get_name method. (Contributed by Alex Grönholm in 34270.)

Added support for Happy Eyeballs to asyncio.loop.create_connection. To specify the behavior, two new parameters have been added: happy_eyeballs_delay and interleave. The Happy Eyeballs algorithm improves responsiveness in applications that support IPv4 and IPv6 by attempting to simultaneously connect using both. (Contributed by twisteroid ambassador in 33530.)

builtins

The compile built-in has been improved to accept the ast.PyCF_ALLOW_TOP_LEVEL_AWAIT flag. With this new flag passed, compile will allow top-level await, async for and async with constructs that are usually considered invalid syntax. Asynchronous code object marked with the CO_COROUTINE flag may then be returned. (Contributed by Matthias Bussonnier in 34616)

collections

The ~collections.somenamedtuple._asdict method for collections.namedtuple now returns a dict instead of a collections.OrderedDict. This works because regular dicts have guaranteed ordering since Python 3.7. If the extra features of OrderedDict are required, the suggested remediation is to cast the result to the desired type: OrderedDict(nt._asdict()). (Contributed by Raymond Hettinger in 35864.)

cProfile

The cProfile.Profile <profile.Profile> class can now be used as a context manager. Profile a block of code by running:

import cProfile

with cProfile.Profile() as profiler:
      # code to be profiled
      ...

(Contributed by Scott Sanderson in 29235.)

csv

The csv.DictReader now returns instances of dict instead of a collections.OrderedDict. The tool is now faster and uses less memory while still preserving the field order. (Contributed by Michael Selik in 34003.)

curses

Added a new variable holding structured version information for the underlying ncurses library: ~curses.ncurses_version. (Contributed by Serhiy Storchaka in 31680.)

ctypes

On Windows, ~ctypes.CDLL and subclasses now accept a winmode parameter to specify flags for the underlying LoadLibraryEx call. The default flags are set to only load DLL dependencies from trusted locations, including the path where the DLL is stored (if a full or partial path is used to load the initial DLL) and paths added by ~os.add_dll_directory. (Contributed by Steve Dower in 36085.)

datetime

Added new alternate constructors datetime.date.fromisocalendar and datetime.datetime.fromisocalendar, which construct ~datetime.date and ~datetime.datetime objects respectively from ISO year, week number, and weekday; these are the inverse of each class's isocalendar method. (Contributed by Paul Ganssle in 36004.)

functools

functools.lru_cache can now be used as a straight decorator rather than as a function returning a decorator. So both of these are now supported:

@lru_cache
def f(x):
    ...

@lru_cache(maxsize=256)
def f(x):
    ...

(Contributed by Raymond Hettinger in 36772.)

Added a new functools.cached_property decorator, for computed properties cached for the life of the instance. :

import functools
import statistics

class Dataset:
   def __init__(self, sequence_of_numbers):
      self.data = sequence_of_numbers

   @functools.cached_property
   def variance(self):
      return statistics.variance(self.data)

(Contributed by Carl Meyer in 21145)

Added a new functools.singledispatchmethod decorator that converts methods into generic functions <generic function> using single dispatch:

from functools import singledispatchmethod
from contextlib import suppress

class TaskManager:

    def __init__(self, tasks):
        self.tasks = list(tasks)

    @singledispatchmethod
    def discard(self, value):
        with suppress(ValueError):
            self.tasks.remove(value)

    @discard.register(list)
    def _(self, tasks):
        targets = set(tasks)
        self.tasks = [x for x in self.tasks if x not in targets]

(Contributed by Ethan Smith in 32380)

gc

~gc.get_objects can now receive an optional generation parameter indicating a generation to get objects from. (Contributed by Pablo Galindo in 36016.)

gettext

Added ~gettext.pgettext and its variants. (Contributed by Franz Glasner, Éric Araujo, and Cheryl Sabella in 2504.)

gzip

Added the mtime parameter to gzip.compress for reproducible output. (Contributed by Guo Ci Teo in 34898.)

A ~gzip.BadGzipFile exception is now raised instead of OSError for certain types of invalid or corrupt gzip files. (Contributed by Filip Gruszczyński, Michele Orrù, and Zackery Spytz in 6584.)

IDLE and idlelib

Output over N lines (50 by default) is squeezed down to a button. N can be changed in the PyShell section of the General page of the Settings dialog. Fewer, but possibly extra long, lines can be squeezed by right clicking on the output. Squeezed output can be expanded in place by double-clicking the button or into the clipboard or a separate window by right-clicking the button. (Contributed by Tal Einat in 1529353.)

Add "Run Customized" to the Run menu to run a module with customized settings. Any command line arguments entered are added to sys.argv. They also re-appear in the box for the next customized run. One can also suppress the normal Shell main module restart. (Contributed by Cheryl Sabella, Terry Jan Reedy, and others in 5680 and 37627.)

Added optional line numbers for IDLE editor windows. Windows open without line numbers unless set otherwise in the General tab of the configuration dialog. Line numbers for an existing window are shown and hidden in the Options menu. (Contributed by Tal Einat and Saimadhav Heblikar in 17535.)

OS native encoding is now used for converting between Python strings and Tcl objects. This allows IDLE to work with emoji and other non-BMP characters. These characters can be displayed or copied and pasted to or from the clipboard. Converting strings from Tcl to Python and back now never fails. (Many people worked on this for eight years but the problem was finally solved by Serhiy Storchaka in 13153.)

New in 3.8.1:

Add option to toggle cursor blink off. (Contributed by Zackery Spytz in 4603.)

Escape key now closes IDLE completion windows. (Contributed by Johnny Najera in 38944.)

The changes above have been backported to 3.7 maintenance releases.

Add keywords to module name completion list. (Contributed by Terry J. Reedy in 37765.)

inspect

The inspect.getdoc function can now find docstrings for __slots__ if that attribute is a dict where the values are docstrings. This provides documentation options similar to what we already have for property, classmethod, and staticmethod:

class AudioClip:
    __slots__ = {'bit_rate': 'expressed in kilohertz to one decimal place',
                 'duration': 'in seconds, rounded up to an integer'}
    def __init__(self, bit_rate, duration):
        self.bit_rate = round(bit_rate / 1000.0, 1)
        self.duration = ceil(duration)

(Contributed by Raymond Hettinger in 36326.)

io

In development mode (-X env) and in debug build <debug-build>, the io.IOBase finalizer now logs the exception if the close() method fails. The exception is ignored silently by default in release build. (Contributed by Victor Stinner in 18748.)

itertools

The itertools.accumulate function added an option initial keyword argument to specify an initial value:

>>> from itertools import accumulate
>>> list(accumulate([10, 5, 30, 15], initial=1000))
[1000, 1010, 1015, 1045, 1060]

(Contributed by Lisa Roach in 34659.)

json.tool

Add option --json-lines to parse every input line as a separate JSON object. (Contributed by Weipeng Hong in 31553.)

logging

Added a force keyword argument to logging.basicConfig. When set to true, any existing handlers attached to the root logger are removed and closed before carrying out the configuration specified by the other arguments.

This solves a long-standing problem. Once a logger or basicConfig() had been called, subsequent calls to basicConfig() were silently ignored. This made it difficult to update, experiment with, or teach the various logging configuration options using the interactive prompt or a Jupyter notebook.

(Suggested by Raymond Hettinger, implemented by Donghee Na, and reviewed by Vinay Sajip in 33897.)

math

Added new function math.dist for computing Euclidean distance between two points. (Contributed by Raymond Hettinger in 33089.)

Expanded the math.hypot function to handle multiple dimensions. Formerly, it only supported the 2-D case. (Contributed by Raymond Hettinger in 33089.)

Added new function, math.prod, as analogous function to sum that returns the product of a 'start' value (default: 1) times an iterable of numbers:

>>> prior = 0.8
>>> likelihoods = [0.625, 0.84, 0.30]
>>> math.prod(likelihoods, start=prior)
0.126

(Contributed by Pablo Galindo in 35606.)

Added two new combinatoric functions math.perm and math.comb:

>>> math.perm(10, 3)    # Permutations of 10 things taken 3 at a time
720
>>> math.comb(10, 3)    # Combinations of 10 things taken 3 at a time
120

(Contributed by Yash Aggarwal, Keller Fuchs, Serhiy Storchaka, and Raymond Hettinger in 37128, 37178, and 35431.)

Added a new function math.isqrt for computing accurate integer square roots without conversion to floating point. The new function supports arbitrarily large integers. It is faster than floor(sqrt(n)) but slower than math.sqrt:

>>> r = 650320427
>>> s = r ** 2
>>> isqrt(s - 1)         # correct
650320426
>>> floor(sqrt(s - 1))   # incorrect
650320427

(Contributed by Mark Dickinson in 36887.)

The function math.factorial no longer accepts arguments that are not int-like. (Contributed by Pablo Galindo in 33083.)

mmap

The mmap.mmap class now has an ~mmap.mmap.madvise method to access the madvise() system call. (Contributed by Zackery Spytz in 32941.)

multiprocessing

Added new multiprocessing.shared_memory module. (Contributed by Davin Potts in 35813.)

On macOS, the spawn start method is now used by default. (Contributed by Victor Stinner in 33725.)

os

Added new function ~os.add_dll_directory on Windows for providing additional search paths for native dependencies when importing extension modules or loading DLLs using ctypes. (Contributed by Steve Dower in 36085.)

A new os.memfd_create function was added to wrap the memfd_create() syscall. (Contributed by Zackery Spytz and Christian Heimes in 26836.)

On Windows, much of the manual logic for handling reparse points (including symlinks and directory junctions) has been delegated to the operating system. Specifically, os.stat will now traverse anything supported by the operating system, while os.lstat will only open reparse points that identify as "name surrogates" while others are opened as for os.stat. In all cases, os.stat_result.st_mode will only have S_IFLNK set for symbolic links and not other kinds of reparse points. To identify other kinds of reparse point, check the new os.stat_result.st_reparse_tag attribute.

On Windows, os.readlink is now able to read directory junctions. Note that ~os.path.islink will return False for directory junctions, and so code that checks islink first will continue to treat junctions as directories, while code that handles errors from os.readlink may now treat junctions as links.

(Contributed by Steve Dower in 37834.)

os.path

os.path functions that return a boolean result like ~os.path.exists, ~os.path.lexists, ~os.path.isdir, ~os.path.isfile, ~os.path.islink, and ~os.path.ismount now return False instead of raising ValueError or its subclasses UnicodeEncodeError and UnicodeDecodeError for paths that contain characters or bytes unrepresentable at the OS level. (Contributed by Serhiy Storchaka in 33721.)

~os.path.expanduser on Windows now prefers the USERPROFILE environment variable and does not use HOME, which is not normally set for regular user accounts. (Contributed by Anthony Sottile in 36264.)

~os.path.isdir on Windows no longer returns True for a link to a non-existent directory.

~os.path.realpath on Windows now resolves reparse points, including symlinks and directory junctions.

(Contributed by Steve Dower in 37834.)

pathlib

pathlib.Path methods that return a boolean result like ~pathlib.Path.exists, ~pathlib.Path.is_dir, ~pathlib.Path.is_file, ~pathlib.Path.is_mount, ~pathlib.Path.is_symlink, ~pathlib.Path.is_block_device, ~pathlib.Path.is_char_device, ~pathlib.Path.is_fifo, ~pathlib.Path.is_socket now return False instead of raising ValueError or its subclass UnicodeEncodeError for paths that contain characters unrepresentable at the OS level. (Contributed by Serhiy Storchaka in 33721.)

Added !pathlib.Path.link_to which creates a hard link pointing to a path. (Contributed by Joannah Nanjekye in 26978) Note that link_to was deprecated in 3.10 and removed in 3.12 in favor of a hardlink_to method added in 3.10 which matches the semantics of the existing symlink_to method.

pickle

pickle extensions subclassing the C-optimized ~pickle.Pickler can now override the pickling logic of functions and classes by defining the special ~pickle.Pickler.reducer_override method. (Contributed by Pierre Glaser and Olivier Grisel in 35900.)

plistlib

Added new plistlib.UID and enabled support for reading and writing NSKeyedArchiver-encoded binary plists. (Contributed by Jon Janzen in 26707.)

pprint

The pprint module added a sort_dicts parameter to several functions. By default, those functions continue to sort dictionaries before rendering or printing. However, if sort_dicts is set to false, the dictionaries retain the order that keys were inserted. This can be useful for comparison to JSON inputs during debugging.

In addition, there is a convenience new function, pprint.pp that is like pprint.pprint but with sort_dicts defaulting to False:

>>> from pprint import pprint, pp
>>> d = dict(source='input.txt', operation='filter', destination='output.txt')
>>> pp(d, width=40)                  # Original order
{'source': 'input.txt',
 'operation': 'filter',
 'destination': 'output.txt'}
>>> pprint(d, width=40)              # Keys sorted alphabetically
{'destination': 'output.txt',
 'operation': 'filter',
 'source': 'input.txt'}

(Contributed by Rémi Lapeyre in 30670.)

py_compile

py_compile.compile now supports silent mode. (Contributed by Joannah Nanjekye in 22640.)

shlex

The new shlex.join function acts as the inverse of shlex.split. (Contributed by Bo Bayles in 32102.)

shutil

shutil.copytree now accepts a new dirs_exist_ok keyword argument. (Contributed by Josh Bronson in 20849.)

shutil.make_archive now defaults to the modern pax (POSIX.1-2001) format for new archives to improve portability and standards conformance, inherited from the corresponding change to the tarfile module. (Contributed by C.A.M. Gerlach in 30661.)

shutil.rmtree on Windows now removes directory junctions without recursively removing their contents first. (Contributed by Steve Dower in 37834.)

socket

Added ~socket.create_server and ~socket.has_dualstack_ipv6 convenience functions to automate the necessary tasks usually involved when creating a server socket, including accepting both IPv4 and IPv6 connections on the same socket. (Contributed by Giampaolo Rodolà in 17561.)

The socket.if_nameindex, socket.if_nametoindex, and socket.if_indextoname functions have been implemented on Windows. (Contributed by Zackery Spytz in 37007.)

ssl

Added ~ssl.SSLContext.post_handshake_auth to enable and ~ssl.SSLSocket.verify_client_post_handshake to initiate TLS 1.3 post-handshake authentication. (Contributed by Christian Heimes in 34670.)

statistics

Added statistics.fmean as a faster, floating-point variant of statistics.mean. (Contributed by Raymond Hettinger and Steven D'Aprano in 35904.)

Added statistics.geometric_mean (Contributed by Raymond Hettinger in 27181.)

Added statistics.multimode that returns a list of the most common values. (Contributed by Raymond Hettinger in 35892.)

Added statistics.quantiles that divides data or a distribution in to equiprobable intervals (e.g. quartiles, deciles, or percentiles). (Contributed by Raymond Hettinger in 36546.)

Added statistics.NormalDist, a tool for creating and manipulating normal distributions of a random variable. (Contributed by Raymond Hettinger in 36018.)

>>> temperature_feb = NormalDist.from_samples([4, 12, -3, 2, 7, 14])
>>> temperature_feb.mean
6.0
>>> temperature_feb.stdev
6.356099432828281

>>> temperature_feb.cdf(3)            # Chance of being under 3 degrees
0.3184678262814532
>>> # Relative chance of being 7 degrees versus 10 degrees
>>> temperature_feb.pdf(7) / temperature_feb.pdf(10)
1.2039930378537762

>>> el_niño = NormalDist(4, 2.5)
>>> temperature_feb += el_niño        # Add in a climate effect
>>> temperature_feb
NormalDist(mu=10.0, sigma=6.830080526611674)

>>> temperature_feb * (9/5) + 32      # Convert to Fahrenheit
NormalDist(mu=50.0, sigma=12.294144947901014)
>>> temperature_feb.samples(3)        # Generate random samples
[7.672102882379219, 12.000027119750287, 4.647488369766392]

sys

Add new sys.unraisablehook function which can be overridden to control how "unraisable exceptions" are handled. It is called when an exception has occurred but there is no way for Python to handle it. For example, when a destructor raises an exception or during garbage collection (gc.collect). (Contributed by Victor Stinner in 36829.)

tarfile

The tarfile module now defaults to the modern pax (POSIX.1-2001) format for new archives, instead of the previous GNU-specific one. This improves cross-platform portability with a consistent encoding (UTF-8) in a standardized and extensible format, and offers several other benefits. (Contributed by C.A.M. Gerlach in 36268.)

threading

Add a new threading.excepthook function which handles uncaught threading.Thread.run exception. It can be overridden to control how uncaught threading.Thread.run exceptions are handled. (Contributed by Victor Stinner in 1230540.)

Add a new threading.get_native_id function and a ~threading.Thread.native_id attribute to the threading.Thread class. These return the native integral Thread ID of the current thread assigned by the kernel. This feature is only available on certain platforms, see get_native_id <threading.get_native_id> for more information. (Contributed by Jake Tesler in 36084.)

tokenize

The tokenize module now implicitly emits a NEWLINE token when provided with input that does not have a trailing new line. This behavior now matches what the C tokenizer does internally. (Contributed by Ammar Askar in 33899.)

tkinter

Added methods !selection_from, !selection_present, !selection_range and !selection_to in the !tkinter.Spinbox class. (Contributed by Juliette Monsel in 34829.)

Added method !moveto in the !tkinter.Canvas class. (Contributed by Juliette Monsel in 23831.)

The !tkinter.PhotoImage class now has !transparency_get and !transparency_set methods. (Contributed by Zackery Spytz in 25451.)

time

Added new clock ~time.CLOCK_UPTIME_RAW for macOS 10.12. (Contributed by Joannah Nanjekye in 35702.)

typing

The typing module incorporates several new features:

unicodedata

The unicodedata module has been upgraded to use the Unicode 12.1.0 release.

New function ~unicodedata.is_normalized can be used to verify a string is in a specific normal form, often much faster than by actually normalizing the string. (Contributed by Max Belanger, David Euresti, and Greg Price in 32285 and 37966).

unittest

Added ~unittest.mock.AsyncMock to support an asynchronous version of ~unittest.mock.Mock. Appropriate new assert functions for testing have been added as well. (Contributed by Lisa Roach in 26467).

Added ~unittest.addModuleCleanup and ~unittest.TestCase.addClassCleanup to unittest to support cleanups for ~unittest.setUpModule and ~unittest.TestCase.setUpClass. (Contributed by Lisa Roach in 24412.)

Several mock assert functions now also print a list of actual calls upon failure. (Contributed by Petter Strandmark in 35047.)

unittest module gained support for coroutines to be used as test cases with unittest.IsolatedAsyncioTestCase. (Contributed by Andrew Svetlov in 32972.)

Example:

import unittest


class TestRequest(unittest.IsolatedAsyncioTestCase):

    async def asyncSetUp(self):
        self.connection = await AsyncConnection()

    async def test_get(self):
        response = await self.connection.get("https://example.com")
        self.assertEqual(response.status_code, 200)

    async def asyncTearDown(self):
        await self.connection.close()


if __name__ == "__main__":
    unittest.main()

venv

venv now includes an Activate.ps1 script on all platforms for activating virtual environments under PowerShell Core 6.1. (Contributed by Brett Cannon in 32718.)

weakref

The proxy objects returned by weakref.proxy now support the matrix multiplication operators @ and @= in addition to the other numeric operators. (Contributed by Mark Dickinson in 36669.)

xml

As mitigation against DTD and external entity retrieval, the xml.dom.minidom and xml.sax modules no longer process external entities by default. (Contributed by Christian Heimes in 17239.)

The .find*() methods in the xml.etree.ElementTree module support wildcard searches like {*}tag which ignores the namespace and {namespace}* which returns all tags in the given namespace. (Contributed by Stefan Behnel in 28238.)

The xml.etree.ElementTree module provides a new function ~xml.etree.ElementTree.canonicalize that implements C14N 2.0. (Contributed by Stefan Behnel in 13611.)

The target object of xml.etree.ElementTree.XMLParser can receive namespace declaration events through the new callback methods start_ns() and end_ns(). Additionally, the xml.etree.ElementTree.TreeBuilder target can be configured to process events about comments and processing instructions to include them in the generated tree. (Contributed by Stefan Behnel in 36676 and 36673.)

xmlrpc

xmlrpc.client.ServerProxy now supports an optional headers keyword argument for a sequence of HTTP headers to be sent with each request. Among other things, this makes it possible to upgrade from default basic authentication to faster session authentication. (Contributed by Cédric Krier in 35153.)

Optimizations

Build and C API Changes

Deprecated

API and Feature Removals

The following features and APIs have been removed from Python 3.8:

Porting to Python 3.8

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in Python behavior

Changes in the Python API

Changes in the C API

CPython bytecode changes

Demos and Tools

Added a benchmark script for timing various ways to access variables: Tools/scripts/var_access_benchmark.py. (Contributed by Raymond Hettinger in 35884.)

Here's a summary of performance improvements since Python 3.3:

Python version                       3.3     3.4     3.5     3.6     3.7     3.8
--------------                       ---     ---     ---     ---     ---     ---

Variable and attribute read access:
    read_local                       4.0     7.1     7.1     5.4     5.1     3.9
    read_nonlocal                    5.3     7.1     8.1     5.8     5.4     4.4
    read_global                     13.3    15.5    19.0    14.3    13.6     7.6
    read_builtin                    20.0    21.1    21.6    18.5    19.0     7.5
    read_classvar_from_class        20.5    25.6    26.5    20.7    19.5    18.4
    read_classvar_from_instance     18.5    22.8    23.5    18.8    17.1    16.4
    read_instancevar                26.8    32.4    33.1    28.0    26.3    25.4
    read_instancevar_slots          23.7    27.8    31.3    20.8    20.8    20.2
    read_namedtuple                 68.5    73.8    57.5    45.0    46.8    18.4
    read_boundmethod                29.8    37.6    37.9    29.6    26.9    27.7

Variable and attribute write access:
    write_local                      4.6     8.7     9.3     5.5     5.3     4.3
    write_nonlocal                   7.3    10.5    11.1     5.6     5.5     4.7
    write_global                    15.9    19.7    21.2    18.0    18.0    15.8
    write_classvar                  81.9    92.9    96.0   104.6   102.1    39.2
    write_instancevar               36.4    44.6    45.8    40.0    38.9    35.5
    write_instancevar_slots         28.7    35.6    36.1    27.3    26.6    25.7

Data structure read access:
    read_list                       19.2    24.2    24.5    20.8    20.8    19.0
    read_deque                      19.9    24.7    25.5    20.2    20.6    19.8
    read_dict                       19.7    24.3    25.7    22.3    23.0    21.0
    read_strdict                    17.9    22.6    24.3    19.5    21.2    18.9

Data structure write access:
    write_list                      21.2    27.1    28.5    22.5    21.6    20.0
    write_deque                     23.8    28.7    30.1    22.7    21.8    23.5
    write_dict                      25.9    31.4    33.3    29.3    29.2    24.7
    write_strdict                   22.9    28.4    29.9    27.5    25.2    23.1

Stack (or queue) operations:
    list_append_pop                144.2    93.4   112.7    75.4    74.2    50.8
    deque_append_pop                30.4    43.5    57.0    49.4    49.2    42.5
    deque_append_popleft            30.8    43.7    57.3    49.7    49.7    42.8

Timing loop:
    loop_overhead                    0.3     0.5     0.6     0.4     0.3     0.3

The benchmarks were measured on an Intel® Core™ i7-4960HQ processor running the macOS 64-bit builds found at python.org. The benchmark script displays timings in nanoseconds.

Notable changes in Python 3.8.1

Due to significant security concerns, the reuse_address parameter of asyncio.loop.create_datagram_endpoint is no longer supported. This is because of the behavior of the socket option SO_REUSEADDR in UDP. For more details, see the documentation for loop.create_datagram_endpoint(). (Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in 37228.)

Notable changes in Python 3.8.2

Fixed a regression with the ignore callback of shutil.copytree. The argument types are now str and List[str] again. (Contributed by Manuel Barkhau and Giampaolo Rodola in 83571.)

Notable changes in Python 3.8.3

The constant values of future flags in the __future__ module are updated in order to prevent collision with compiler flags. Previously PyCF_ALLOW_TOP_LEVEL_AWAIT was clashing with CO_FUTURE_DIVISION. (Contributed by Batuhan Taskaya in 83743)

Notable changes in Python 3.8.8

Earlier Python versions allowed using both ; and & as query parameter separators in urllib.parse.parse_qs and urllib.parse.parse_qsl. Due to security concerns, and to conform with newer W3C recommendations, this has been changed to allow only a single separator key, with & as the default. This change also affects !cgi.parse and !cgi.parse_multipart as they use the affected functions internally. For more details, please see their respective documentation. (Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in 42967.)

Notable changes in Python 3.8.9

A security fix alters the ftplib.FTP behavior to not trust the IPv4 address sent from the remote server when setting up a passive data channel. We reuse the ftp server IP address instead. For unusual code requiring the old behavior, set a trust_server_pasv_ipv4_address attribute on your FTP instance to True. (See 87451)

Notable changes in Python 3.8.10

macOS 11.0 (Big Sur) and Apple Silicon Mac support

As of 3.8.10, Python now supports building and running on macOS 11 (Big Sur) and on Apple Silicon Macs (based on the ARM64 architecture). A new universal build variant, universal2, is now available to natively support both ARM64 and Intel 64 in one set of executables. Note that support for "weaklinking", building binaries targeted for newer versions of macOS that will also run correctly on older versions by testing at runtime for missing features, is not included in this backport from Python 3.9; to support a range of macOS versions, continue to target for and build on the oldest version in the range.

(Originally contributed by Ronald Oussoren and Lawrence D'Anna in 85272, with fixes by FX Coudert and Eli Rykoff, and backported to 3.8 by Maxime Bélanger and Ned Deily)

Notable changes in Python 3.8.10

urllib.parse

The presence of newline or tab characters in parts of a URL allows for some forms of attacks. Following the WHATWG specification that updates 3986, ASCII newline \n, \r and tab \t characters are stripped from the URL by the parser in urllib.parse preventing such attacks. The removal characters are controlled by a new module level variable urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE. (See 43882)

Notable changes in Python 3.8.12

Changes in the Python API

Starting with Python 3.8.12 the ipaddress module no longer accepts any leading zeros in IPv4 address strings. Leading zeros are ambiguous and interpreted as octal notation by some libraries. For example the legacy function socket.inet_aton treats leading zeros as octal notation. glibc implementation of modern ~socket.inet_pton does not accept any leading zeros.

(Originally contributed by Christian Heimes in 36384, and backported to 3.8 by Achraf Merzouki.)

Notable security feature in 3.8.14

Converting between int and str in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a ValueError if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for 2020-10735. This limit can be configured or disabled by environment variable, command line flag, or sys APIs. See the integer string conversion length limitation <int_max_str_digits> documentation. The default limit is 4300 digits in string form.

Notable changes in 3.8.17

tarfile

What's New In Python 3.9

Editor
Łukasz Langa

* Anyone can add text to this document. Your text might get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes will be too small or esoteric to include. If such a change is added to the text, it might get removed during final editing.

* If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section.

* It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text.

* You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary.

XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in 12345.)

This saves the maintainer the effort of going through the Mercurial log when researching a change.

This article explains the new features in Python 3.9, compared to 3.8. Python 3.9 was released on October 5, 2020. For full details, see the changelog <changelog>.

596 - Python 3.9 Release Schedule

Summary -- Release highlights

New syntax features:

New built-in features:

New features in the standard library:

Interpreter improvements:

New library modules:

Release process changes:

You should check for DeprecationWarning in your code

When Python 2.7 was still supported, a lot of functionality in Python 3 was kept for backward compatibility with Python 2.7. With the end of Python 2 support, these backward compatibility layers have been removed, or will be removed soon. Most of them emitted a DeprecationWarning warning for several years. For example, using collections.Mapping instead of collections.abc.Mapping emits a DeprecationWarning since Python 3.3, released in 2012.

Test your application with the -W default command-line option to see DeprecationWarning and PendingDeprecationWarning, or even with -W error to treat them as errors. Warnings Filter <warning-filter> can be used to ignore warnings from third-party code.

Python 3.9 is the last version providing those Python 2 backward compatibility layers, to give more time to Python projects maintainers to organize the removal of the Python 2 support and add support for Python 3.9.

Aliases to Abstract Base Classes <collections-abstract-base-classes> in the collections module, like collections.Mapping alias to collections.abc.Mapping, are kept for one last release for backward compatibility. They will be removed from Python 3.10.

More generally, try to run your tests in the Python Development Mode <devmode> which helps to prepare your code to make it compatible with the next Python version.

Note: a number of pre-existing deprecations were removed in this version of Python as well. Consult the removed-in-python-39 section.

New Features

Dictionary Merge & Update Operators

Merge (|) and update (|=) operators have been added to the built-in dict class. Those complement the existing dict.update and {**d1, **d2} methods of merging dictionaries.

Example:

>>> x = {"key1": "value1 from x", "key2": "value2 from x"}
>>> y = {"key2": "value2 from y", "key3": "value3 from y"}
>>> x | y
{'key1': 'value1 from x', 'key2': 'value2 from y', 'key3': 'value3 from y'}
>>> y | x
{'key2': 'value2 from x', 'key3': 'value3 from y', 'key1': 'value1 from x'}

See 584 for a full description. (Contributed by Brandt Bucher in 36144.)

New String Methods to Remove Prefixes and Suffixes

str.removeprefix(prefix)<str.removeprefix> and str.removesuffix(suffix)<str.removesuffix> have been added to easily remove an unneeded prefix or a suffix from a string. Corresponding bytes, bytearray, and collections.UserString methods have also been added. See 616 for a full description. (Contributed by Dennis Sweeney in 39939.)

Type Hinting Generics in Standard Collections

In type annotations you can now use built-in collection types such as list and dict as generic types instead of importing the corresponding capitalized types (e.g. List or Dict) from typing. Some other types in the standard library are also now generic, for example queue.Queue.

Example:

def greet_all(names: list[str]) -> None:
    for name in names:
        print("Hello", name)

See 585 for more details. (Contributed by Guido van Rossum, Ethan Smith, and Batuhan Taşkaya in 39481.)

New Parser

Python 3.9 uses a new parser, based on PEG instead of LL(1). The new parser's performance is roughly comparable to that of the old parser, but the PEG formalism is more flexible than LL(1) when it comes to designing new language features. We'll start using this flexibility in Python 3.10 and later.

The ast module uses the new parser and produces the same AST as the old parser.

In Python 3.10, the old parser will be deleted and so will all functionality that depends on it (primarily the !parser module, which has long been deprecated). In Python 3.9 only, you can switch back to the LL(1) parser using a command line switch (-X oldparser) or an environment variable (PYTHONOLDPARSER=1).

See 617 for more details. (Contributed by Guido van Rossum, Pablo Galindo and Lysandros Nikolaou in 40334.)

Other Language Changes

New Modules

zoneinfo

The zoneinfo module brings support for the IANA time zone database to the standard library. It adds zoneinfo.ZoneInfo, a concrete datetime.tzinfo implementation backed by the system's time zone data.

Example:

>>> from zoneinfo import ZoneInfo
>>> import datetime as dt

>>> # Daylight saving time
>>> when = dt.datetime(2020, 10, 31, 12, tzinfo=ZoneInfo("America/Los_Angeles"))
>>> print(when)
2020-10-31 12:00:00-07:00
>>> when.tzname()
'PDT'

>>> # Standard time
>>> when += dt.timedelta(days=7)
>>> print(when)
2020-11-07 12:00:00-08:00
>>> print(when.tzname())
PST

As a fall-back source of data for platforms that don't ship the IANA database, the tzdata module was released as a first-party package -- distributed via PyPI and maintained by the CPython core team.

615 -- Support for the IANA Time Zone Database in the Standard Library
PEP written and implemented by Paul Ganssle

graphlib

A new module, graphlib, was added that contains the graphlib.TopologicalSorter class to offer functionality to perform topological sorting of graphs. (Contributed by Pablo Galindo, Tim Peters and Larry Hastings in 17005.)

Improved Modules

ast

Added the indent option to ~ast.dump which allows it to produce a multiline indented output. (Contributed by Serhiy Storchaka in 37995.)

Added ast.unparse as a function in the ast module that can be used to unparse an ast.AST object and produce a string with code that would produce an equivalent ast.AST object when parsed. (Contributed by Pablo Galindo and Batuhan Taskaya in 38870.)

Added docstrings to AST nodes that contains the ASDL signature used to construct that node. (Contributed by Batuhan Taskaya in 39638.)

asyncio

Due to significant security concerns, the reuse_address parameter of asyncio.loop.create_datagram_endpoint is no longer supported. This is because of the behavior of the socket option SO_REUSEADDR in UDP. For more details, see the documentation for loop.create_datagram_endpoint(). (Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in 37228.)

Added a new coroutine ~asyncio.loop.shutdown_default_executor that schedules a shutdown for the default executor that waits on the ~concurrent.futures.ThreadPoolExecutor to finish closing. Also, asyncio.run has been updated to use the new coroutine. (Contributed by Kyle Stanley in 34037.)

Added !asyncio.PidfdChildWatcher, a Linux-specific child watcher implementation that polls process file descriptors. (38692)

Added a new coroutine asyncio.to_thread. It is mainly used for running IO-bound functions in a separate thread to avoid blocking the event loop, and essentially works as a high-level version of ~asyncio.loop.run_in_executor that can directly take keyword arguments. (Contributed by Kyle Stanley and Yury Selivanov in 32309.)

When cancelling the task due to a timeout, asyncio.wait_for will now wait until the cancellation is complete also in the case when timeout is <= 0, like it does with positive timeouts. (Contributed by Elvis Pranskevichus in 32751.)

asyncio now raises TypeError when calling incompatible methods with an ssl.SSLSocket socket. (Contributed by Ido Michael in 37404.)

compileall

Added new possibility to use hardlinks for duplicated .pyc files: hardlink_dupes parameter and --hardlink-dupes command line option. (Contributed by Lumír 'Frenzy' Balhar in 40495.)

Added new options for path manipulation in resulting .pyc files: stripdir, prependdir, limit_sl_dest parameters and -s, -p, -e command line options. Added the possibility to specify the option for an optimization level multiple times. (Contributed by Lumír 'Frenzy' Balhar in 38112.)

concurrent.futures

Added a new cancel_futures parameter to concurrent.futures.Executor.shutdown that cancels all pending futures which have not started running, instead of waiting for them to complete before shutting down the executor. (Contributed by Kyle Stanley in 39349.)

Removed daemon threads from ~concurrent.futures.ThreadPoolExecutor and ~concurrent.futures.ProcessPoolExecutor. This improves compatibility with subinterpreters and predictability in their shutdown processes. (Contributed by Kyle Stanley in 39812.)

Workers in ~concurrent.futures.ProcessPoolExecutor are now spawned on demand, only when there are no available idle workers to reuse. This optimizes startup overhead and reduces the amount of lost CPU time to idle workers. (Contributed by Kyle Stanley in 39207.)

curses

Added curses.get_escdelay, curses.set_escdelay, curses.get_tabsize, and curses.set_tabsize functions. (Contributed by Anthony Sottile in 38312.)

datetime

The ~datetime.date.isocalendar of datetime.date and ~datetime.datetime.isocalendar of datetime.datetime methods now returns a ~collections.namedtuple instead of a tuple. (Contributed by Donghee Na in 24416.)

distutils

The upload command now creates SHA2-256 and Blake2b-256 hash digests. It skips MD5 on platforms that block MD5 digest. (Contributed by Christian Heimes in 40698.)

fcntl

Added constants !fcntl.F_OFD_GETLK, !fcntl.F_OFD_SETLK and !fcntl.F_OFD_SETLKW. (Contributed by Donghee Na in 38602.)

ftplib

~ftplib.FTP and ~ftplib.FTP_TLS now raise a ValueError if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Donghee Na in 39259.)

gc

When the garbage collector makes a collection in which some objects resurrect (they are reachable from outside the isolated cycles after the finalizers have been executed), do not block the collection of all objects that are still unreachable. (Contributed by Pablo Galindo and Tim Peters in 38379.)

Added a new function gc.is_finalized to check if an object has been finalized by the garbage collector. (Contributed by Pablo Galindo in 39322.)

hashlib

The hashlib module can now use SHA3 hashes and SHAKE XOF from OpenSSL when available. (Contributed by Christian Heimes in 37630.)

Builtin hash modules can now be disabled with ./configure --without-builtin-hashlib-hashes or selectively enabled with e.g. ./configure --with-builtin-hashlib-hashes=sha3,blake2 to force use of OpenSSL based implementation. (Contributed by Christian Heimes in 40479)

http

HTTP status codes 103 EARLY_HINTS, 418 IM_A_TEAPOT and 425 TOO_EARLY are added to http.HTTPStatus. (Contributed by Donghee Na in 39509 and Ross Rhodes in 39507.)

IDLE and idlelib

Added option to toggle cursor blink off. (Contributed by Zackery Spytz in 4603.)

Escape key now closes IDLE completion windows. (Contributed by Johnny Najera in 38944.)

Added keywords to module name completion list. (Contributed by Terry J. Reedy in 37765.)

New in 3.9 maintenance releases

Make IDLE invoke sys.excepthook (when started without '-n'). User hooks were previously ignored. (Contributed by Ken Hilton in 43008.)

The changes above have been backported to 3.8 maintenance releases.

Rearrange the settings dialog. Split the General tab into Windows and Shell/Ed tabs. Move help sources, which extend the Help menu, to the Extensions tab. Make space for new options and shorten the dialog. The latter makes the dialog better fit small screens. (Contributed by Terry Jan Reedy in 40468.) Move the indent space setting from the Font tab to the new Windows tab. (Contributed by Mark Roseman and Terry Jan Reedy in 33962.)

Apply syntax highlighting to .pyi files. (Contributed by Alex Waygood and Terry Jan Reedy in 45447.)

imaplib

~imaplib.IMAP4 and ~imaplib.IMAP4_SSL now have an optional timeout parameter for their constructors. Also, the ~imaplib.IMAP4.open method now has an optional timeout parameter with this change. The overridden methods of ~imaplib.IMAP4_SSL and ~imaplib.IMAP4_stream were applied to this change. (Contributed by Donghee Na in 38615.)

imaplib.IMAP4.unselect is added. imaplib.IMAP4.unselect frees server's resources associated with the selected mailbox and returns the server to the authenticated state. This command performs the same actions as imaplib.IMAP4.close, except that no messages are permanently removed from the currently selected mailbox. (Contributed by Donghee Na in 40375.)

importlib

To improve consistency with import statements, importlib.util.resolve_name now raises ImportError instead of ValueError for invalid relative import attempts. (Contributed by Ngalim Siregar in 37444.)

Import loaders which publish immutable module objects can now publish immutable packages in addition to individual modules. (Contributed by Dino Viehland in 39336.)

Added importlib.resources.files function with support for subdirectories in package data, matching backport in importlib_resources version 1.5. (Contributed by Jason R. Coombs in 39791.)

Refreshed importlib.metadata from importlib_metadata version 1.6.1.

inspect

inspect.BoundArguments.arguments is changed from OrderedDict to regular dict. (Contributed by Inada Naoki in 36350 and 39775.)

ipaddress

ipaddress now supports IPv6 Scoped Addresses (IPv6 address with suffix %<scope_id>).

Scoped IPv6 addresses can be parsed using ipaddress.IPv6Address. If present, scope zone ID is available through the ~ipaddress.IPv6Address.scope_id attribute. (Contributed by Oleksandr Pavliuk in 34788.)

Starting with Python 3.9.5 the ipaddress module no longer accepts any leading zeros in IPv4 address strings. (Contributed by Christian Heimes in 36384).

math

Expanded the math.gcd function to handle multiple arguments. Formerly, it only supported two arguments. (Contributed by Serhiy Storchaka in 39648.)

Added math.lcm: return the least common multiple of specified arguments. (Contributed by Mark Dickinson, Ananthakrishnan and Serhiy Storchaka in 39479 and 39648.)

Added math.nextafter: return the next floating-point value after x towards y. (Contributed by Victor Stinner in 39288.)

Added math.ulp: return the value of the least significant bit of a float. (Contributed by Victor Stinner in 39310.)

multiprocessing

The multiprocessing.SimpleQueue class has a new ~multiprocessing.SimpleQueue.close method to explicitly close the queue. (Contributed by Victor Stinner in 30966.)

nntplib

!NNTP and !NNTP_SSL now raise a ValueError if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Donghee Na in 39259.)

os

Added ~os.CLD_KILLED and ~os.CLD_STOPPED for !si_code. (Contributed by Donghee Na in 38493.)

Exposed the Linux-specific os.pidfd_open (38692) and os.P_PIDFD (38713) for process management with file descriptors.

The os.unsetenv function is now also available on Windows. (Contributed by Victor Stinner in 39413.)

The os.putenv and os.unsetenv functions are now always available. (Contributed by Victor Stinner in 39395.)

Added os.waitstatus_to_exitcode function: convert a wait status to an exit code. (Contributed by Victor Stinner in 40094.)

pathlib

Added pathlib.Path.readlink which acts similarly to os.readlink. (Contributed by Girts Folkmanis in 30618)

pdb

On Windows now ~pdb.Pdb supports ~/.pdbrc. (Contributed by Tim Hopper and Dan Lidral-Porter in 20523.)

poplib

~poplib.POP3 and ~poplib.POP3_SSL now raise a ValueError if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Donghee Na in 39259.)

pprint

pprint can now pretty-print types.SimpleNamespace. (Contributed by Carl Bordum Hansen in 37376.)

pydoc

The documentation string is now shown not only for class, function, method etc, but for any object that has its own ~definition.__doc__ attribute. (Contributed by Serhiy Storchaka in 40257.)

random

Added a new random.Random.randbytes method: generate random bytes. (Contributed by Victor Stinner in 40286.)

signal

Exposed the Linux-specific signal.pidfd_send_signal for sending to signals to a process using a file descriptor instead of a pid. (38712)

smtplib

~smtplib.SMTP and ~smtplib.SMTP_SSL now raise a ValueError if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Donghee Na in 39259.)

~smtplib.LMTP constructor now has an optional timeout parameter. (Contributed by Donghee Na in 39329.)

socket

The socket module now exports the ~socket.CAN_RAW_JOIN_FILTERS constant on Linux 4.1 and greater. (Contributed by Stefan Tatschner and Zackery Spytz in 25780.)

The socket module now supports the ~socket.CAN_J1939 protocol on platforms that support it. (Contributed by Karl Ding in 40291.)

The socket module now has the socket.send_fds and socket.recv_fds functions. (Contributed by Joannah Nanjekye, Shinya Okano and Victor Stinner in 28724.)

time

On AIX, ~time.thread_time is now implemented with thread_cputime() which has nanosecond resolution, rather than clock_gettime(CLOCK_THREAD_CPUTIME_ID) which has a resolution of 10 milliseconds. (Contributed by Batuhan Taskaya in 40192)

sys

Added a new sys.platlibdir attribute: name of the platform-specific library directory. It is used to build the path of standard library and the paths of installed extension modules. It is equal to "lib" on most platforms. On Fedora and SuSE, it is equal to "lib64" on 64-bit platforms. (Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis and Victor Stinner in 1294959.)

Previously, sys.stderr was block-buffered when non-interactive. Now stderr defaults to always being line-buffered. (Contributed by Jendrik Seipp in 13601.)

tracemalloc

Added tracemalloc.reset_peak to set the peak size of traced memory blocks to the current size, to measure the peak of specific pieces of code. (Contributed by Huon Wilson in 40630.)

typing

593 introduced an typing.Annotated type to decorate existing types with context-specific metadata and new include_extras parameter to typing.get_type_hints to access the metadata at runtime. (Contributed by Till Varoquaux and Konstantin Kashin.)

unicodedata

The Unicode database has been updated to version 13.0.0. (39926).

venv

The activation scripts provided by venv now all specify their prompt customization consistently by always using the value specified by __VENV_PROMPT__. Previously some scripts unconditionally used __VENV_PROMPT__, others only if it happened to be set (which was the default case), and one used __VENV_NAME__ instead. (Contributed by Brett Cannon in 37663.)

xml

White space characters within attributes are now preserved when serializing xml.etree.ElementTree to XML file. EOLNs are no longer normalized to "n". This is the result of discussion about how to interpret section 2.11 of XML spec. (Contributed by Mefistotelis in 39011.)

Optimizations

Here's a summary of performance improvements from Python 3.4 through Python 3.9:

Python version                       3.4     3.5     3.6     3.7     3.8    3.9
--------------                       ---     ---     ---     ---     ---    ---

Variable and attribute read access:
    read_local                       7.1     7.1     5.4     5.1     3.9    3.9
    read_nonlocal                    7.1     8.1     5.8     5.4     4.4    4.5
    read_global                     15.5    19.0    14.3    13.6     7.6    7.8
    read_builtin                    21.1    21.6    18.5    19.0     7.5    7.8
    read_classvar_from_class        25.6    26.5    20.7    19.5    18.4   17.9
    read_classvar_from_instance     22.8    23.5    18.8    17.1    16.4   16.9
    read_instancevar                32.4    33.1    28.0    26.3    25.4   25.3
    read_instancevar_slots          27.8    31.3    20.8    20.8    20.2   20.5
    read_namedtuple                 73.8    57.5    45.0    46.8    18.4   18.7
    read_boundmethod                37.6    37.9    29.6    26.9    27.7   41.1

Variable and attribute write access:
    write_local                      8.7     9.3     5.5     5.3     4.3    4.3
    write_nonlocal                  10.5    11.1     5.6     5.5     4.7    4.8
    write_global                    19.7    21.2    18.0    18.0    15.8   16.7
    write_classvar                  92.9    96.0   104.6   102.1    39.2   39.8
    write_instancevar               44.6    45.8    40.0    38.9    35.5   37.4
    write_instancevar_slots         35.6    36.1    27.3    26.6    25.7   25.8

Data structure read access:
    read_list                       24.2    24.5    20.8    20.8    19.0   19.5
    read_deque                      24.7    25.5    20.2    20.6    19.8   20.2
    read_dict                       24.3    25.7    22.3    23.0    21.0   22.4
    read_strdict                    22.6    24.3    19.5    21.2    18.9   21.5

Data structure write access:
    write_list                      27.1    28.5    22.5    21.6    20.0   20.0
    write_deque                     28.7    30.1    22.7    21.8    23.5   21.7
    write_dict                      31.4    33.3    29.3    29.2    24.7   25.4
    write_strdict                   28.4    29.9    27.5    25.2    23.1   24.5

Stack (or queue) operations:
    list_append_pop                 93.4   112.7    75.4    74.2    50.8   50.6
    deque_append_pop                43.5    57.0    49.4    49.2    42.5   44.2
    deque_append_popleft            43.7    57.3    49.7    49.7    42.8   46.4

Timing loop:
    loop_overhead                    0.5     0.6     0.4     0.3     0.3    0.3

These results were generated from the variable access benchmark script at: Tools/scripts/var_access_benchmark.py. The benchmark script displays timings in nanoseconds. The benchmarks were measured on an Intel® Core™ i7-4960HQ processor running the macOS 64-bit builds found at python.org.

Deprecated

Removed

Porting to Python 3.9

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in the Python API

Changes in the C API

CPython bytecode changes

Build Changes

C API Changes

New Features

Porting to Python 3.9

Removed

Notable changes in Python 3.9.1

typing

The behavior of typing.Literal was changed to conform with 586 and to match the behavior of static type checkers specified in the PEP.

  1. Literal now de-duplicates parameters.

  2. Equality comparisons between Literal objects are now order independent.

  3. Literal comparisons now respect types. For example, Literal[0] == Literal[False] previously evaluated to True. It is now False. To support this change, the internally used type cache now supports differentiating types.

  4. Literal objects will now raise a TypeError exception during equality comparisons if any of their parameters are not hashable. Note that declaring Literal with mutable parameters will not throw an error:

    >>> from typing import Literal
>>> Literal[{0}]
>>> Literal[{0}] == Literal[{False}]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'set'

(Contributed by Yurii Karabas in 42345.)

macOS 11.0 (Big Sur) and Apple Silicon Mac support

As of 3.9.1, Python now fully supports building and running on macOS 11.0 (Big Sur) and on Apple Silicon Macs (based on the ARM64 architecture). A new universal build variant, universal2, is now available to natively support both ARM64 and Intel 64 in one set of executables. Binaries can also now be built on current versions of macOS to be deployed on a range of older macOS versions (tested to 10.9) while making some newer OS functions and options conditionally available based on the operating system version in use at runtime ("weaklinking").

(Contributed by Ronald Oussoren and Lawrence D'Anna in 41100.)

Notable changes in Python 3.9.2

collections.abc

collections.abc.Callable generic now flattens type parameters, similar to what typing.Callable currently does. This means that collections.abc.Callable[[int, str], str] will have __args__ of (int, str, str); previously this was ([int, str], str). To allow this change, types.GenericAlias can now be subclassed, and a subclass will be returned when subscripting the collections.abc.Callable type. Code which accesses the arguments via typing.get_args or __args__ need to account for this change. A DeprecationWarning may be emitted for invalid forms of parameterizing collections.abc.Callable which may have passed silently in Python 3.9.1. This DeprecationWarning will become a TypeError in Python 3.10. (Contributed by Ken Jin in 42195.)

urllib.parse

Earlier Python versions allowed using both ; and & as query parameter separators in urllib.parse.parse_qs and urllib.parse.parse_qsl. Due to security concerns, and to conform with newer W3C recommendations, this has been changed to allow only a single separator key, with & as the default. This change also affects !cgi.parse and !cgi.parse_multipart as they use the affected functions internally. For more details, please see their respective documentation. (Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in 42967.)

Notable changes in Python 3.9.3

A security fix alters the ftplib.FTP behavior to not trust the IPv4 address sent from the remote server when setting up a passive data channel. We reuse the ftp server IP address instead. For unusual code requiring the old behavior, set a trust_server_pasv_ipv4_address attribute on your FTP instance to True. (See 87451)

Notable changes in Python 3.9.5

urllib.parse

The presence of newline or tab characters in parts of a URL allows for some forms of attacks. Following the WHATWG specification that updates 3986, ASCII newline \n, \r and tab \t characters are stripped from the URL by the parser in urllib.parse preventing such attacks. The removal characters are controlled by a new module level variable urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE. (See 88048)

Notable security feature in 3.9.14

Converting between int and str in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a ValueError if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for 2020-10735. This limit can be configured or disabled by environment variable, command line flag, or sys APIs. See the integer string conversion length limitation <int_max_str_digits> documentation. The default limit is 4300 digits in string form.

Notable changes in 3.9.17

tarfile

changelog

Changelog

../build/NEWS

What's New in Python

The "What's New in Python" series of essays takes tours through the most important changes between major Python versions. They are a "must read" for anyone wishing to stay up-to-date after a new release.

3.16.rst 3.15.rst 3.14.rst 3.13.rst 3.12.rst 3.11.rst 3.10.rst 3.9.rst 3.8.rst 3.7.rst 3.6.rst 3.5.rst 3.4.rst 3.3.rst 3.2.rst 3.1.rst 3.0.rst 2.7.rst 2.6.rst 2.5.rst 2.4.rst 2.3.rst 2.2.rst 2.1.rst 2.0.rst

The "Changelog" is an HTML version of the file built<blurb> from the contents of the Misc/NEWS.d directory tree, which contains all nontrivial changes to Python for the current version.

changelog.rst