python on Hugo van Kemenade

Soft-deprecating re.match()

Thu, 10 Sep 2026 08:08:01 +0000

Quick, without looking it up, what does re.match() do? Which of these return a match?

import re
re.match("pi", "pi")
re.match("pi", "pie")
re.match("pi", "api")
re.match("pi", "magpie")

How does it compare to re.search() and re.fullmatch()?

Whilst you’re (quickly) thinking about it, let’s introduce soft deprecation.

Soft deprecation #

Python’s backwards compatibility policy (PEP 387) introduced soft deprecation in 2023:

A soft deprecation can be used when using an API which should no longer be used to write new code, but it remains safe to continue using it in existing code. The API remains documented and tested, but will not be developed further (no enhancement).

A soft deprecation does not imply future removal of the API, nor does it issue a warning. It’s a docs-only recommendation to not use an API, ideally with a suggested replacement.

It’s a completely separate decision whether, if ever, to turn a soft deprecation into a regular “hard” deprecation (where removal may follow); soft deprecations don’t “graduate” into regular deprecations or removals.

`re.match()` #

Now the answer:

>>> import re
>>> re.match("pi", "pi") # ✅ Matches
<re.Match object; span=(0, 2), match='pi'>
>>> re.match("pi", "pie") # ✅ Matches
<re.Match object; span=(0, 2), match='pi'>
>>> re.match("pi", "api") # ❌ No match
>>> re.match("pi", "magpie") # ❌ No match
>>>

So re.match() only matches at the beginning of a string! This can be surprising: why is the start of the string special?

`re.search()` #

If you don’t want to anchor at the start, and want to match anywhere in the string, use re.search():

>>> import re
>>> re.search("pi", "pi") # ✅ Matches
<re.Match object; span=(0, 2), match='pi'>
>>> re.search("pi", "pie") # ✅ Matches
<re.Match object; span=(0, 2), match='pi'>
>>> re.search("pi", "api") # ✅ Matches
<re.Match object; span=(1, 3), match='pi'>
>>> re.search("pi", "magpie") # ✅ Matches
<re.Match object; span=(3, 5), match='pi'>
>>>

`re.fullmatch()` #

If you want to anchor both the start and the end, and check the entire string matches, use re.fullmatch():

>>> import re
>>> re.fullmatch("pi", "pi") # ✅ Matches
<re.Match object; span=(0, 2), match='pi'>
>>> re.fullmatch("pi", "api") # ❌ No match
>>> re.fullmatch("pi", "pie") # ❌ No match
>>> re.fullmatch("pi", "magpie") # ❌ No match
>>>

Introducing `re.prefixmatch()` #

Because of the surprising half-anchored behaviour, we’ve introduced a new alias for re.match() in Python 3.15, named re.prefixmatch():

Quoting from the Zen Of Python (python3 -m this): “Explicit is better than implicit”. Anyone reading the name prefixmatch() is likely to understand the intended semantics. When reading match() there remains a seed of doubt about the intended behavior to anyone not already familiar with this old Python gotcha.

Soft-deprecating `re.match()` #

And with a more explicit replacement, we’ve soft-deprecated re.match() in Python 3.15:

We do not plan to remove the older match() name, as it has been used in code for over 30 years. It has been soft deprecated: code supporting older versions of Python should continue to use match(), while new code should prefer prefixmatch().

Use re.prefixmatch() if you only really meant to use the half-anchor; otherwise use re.search() or re.fullmatch().

Comparison #

Function	Start anchor	End anchor	Added in	With special characters
`re.search()`	❌	❌	1.5	`re.search("pi", string)`
`re.match()`	✅	❌	1.5	`re.search("^pi", string)` `re.search(r"\Api", string)`
`re.prefixmatch()`	✅	❌	3.15	`re.search("^pi", string)` `re.search(r"\Api", string)`
`re.fullmatch()`	✅	✅	3.4	`re.search("^pi$", string)` `re.search(r"\Api\z", string)`

The functions without special characters are generally a bit faster.

Lint #

You can avoid re.match() in your project with Ruff:

# pyproject.toml
[tool.ruff]
lint.extend-select = [
 "TID251", # flake8-tidy-imports: banned-api
]
lint.flake8-tidy-imports.banned-api."re.match".msg = "Use re.fullmatch() or re.search() instead"

Or run:

ruff check . --isolated --select TID251 \
 --config 'lint.flake8-tidy-imports.banned-api."re.match".msg = "use re.fullmatch() or re.search() instead"'

Help test Python 3.15!

Tue, 01 Sep 2026 14:33:38 +0000

Calling all Python library maintainers! 🐍

The second and final Python 3.15 release candidate is out! 🎉

It’s your last chance to join the cool testers’ club and say that YOU helped test 3.15 before the big day on 1st October! 😎

PEP 790 defines the release schedule for Python 3.15.0:

The first release candidate came out on 4th August 2026
The second release candidate came out on 1st September 2026
And the full release is set for 1st October 2026

In his announcement, Hugo van Kemenade (👋 hi, that’s me!), release manager for Python 3.14 and 3.15, gave a call to action:

We strongly encourage maintainers of third-party Python projects to prepare their projects for 3.15 during this phase, and publish Python 3.15 wheels on PyPI to be ready for the final release of 3.15.0, and to help other projects do their own testing. Any binary wheels built against Python 3.15.0 release candidates will work with future versions of Python 3.15. As always, report any issues to the Python bug tracker.

Test with 3.15 #

It’s now time for us library maintainers to start testing our projects with 3.15. There’s two big benefits:

There have been removals and changes in Python 3.15. Testing now helps us make our code compatible and avoid any big surprises (for us and our users) at the big launch in October.
We might find bugs in Python itself! Reporting those will help get them fixed and help everyone.

How #

To test the latest alpha, beta or release candidate on GitHub Actions with actions/setup-python, add 3.15 and allow-prereleases: true to your workflow matrix.

For example:

jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 fail-fast: false
 matrix:
 python-version: ["3.11", "3.12", "3.13", "3.14", "3.15"]

 steps:
 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

 - name: Set up Python ${{ matrix.python-version }}
 uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
 with:
 python-version: ${{ matrix.python-version }}
 allow-prereleases: true

(We can instead use 3.15-dev and omit allow-prereleases: true, but I find the above a bit neater, and when 3.15.0 final is released in October, it will continue testing with full release versions.)

When to support 3.15? #

Now is also a good time to declare support and add the Programming Language :: Python :: 3.15 Trove classifier. Many projects already have!

Especially if you have extension modules and other projects depend on yours, a release with binary wheels will help them test and prepare.

ABI breaks? #

From the announcement:

There will be no ABI changes from this point forward in the 3.15 series, and the goal is that there will be as few code changes as possible.

Let’s start testing 3.15 now! 🚀

Security: line goes up

Mon, 13 Jul 2026 20:44:50 +0000

Like many other projects, CPython is experiencing a huge increase in security reports.

CVEs per year #

Last month, PSF Security Developer-in-Residence Seth Larson posted a chart of CVEs per year, showing a large increase in 2026:

But this only represents the output of security work, and doesn’t show all the work dealing with incoming reports. Many are closed and dealt with as non-security bug reports instead; many are closed as neither security nor bug reports.

Let’s reveal some of this unseen work by the Python Security Response Team (PSRT).

GHSAs by month #

Here are the number of incoming GitHub Security Advisories (GHSA) reports created since July 2024:

GHSAs by year #

Here is the same thing by year, and remembering we’re only halfway through 2026:

Email reports by month #

We’ve only fairly recently been encouraging new reports be made via GHSA. Before this, they were usually made by email. The next chart is the number of email discussions (or threads) and participants by month:

Thanks #

Big thanks to Seth for all his work as Security Developer-in-Residence: helping shepherd all these reports, developing a security policy to improve the quality of incoming reports and help us assess them, and defining PSRT membership and responsibilities via PEP 811 to build an active team. All this would be much harder without his guidance! And thanks to Alpha-Omega for sponsoring his position at the PSF.

Fixing the dictionary with Python 3.14

Wed, 08 Jul 2026 16:40:05 +0000

Yes, but not the dict kind of dictionary.

When working on CPython, we often find obscure bugs elsewhere, in compilers, operating systems and elsewhere:

Rust/LLVM, clang-19, clang 21 and BOLT
GCC 13, GCC 15, glibc, readline and curses
musl fma

Since Python 3.8, the release notes have a section called “And now for something completely different”. These have included Monty Python sketches, astrophysics facts and poetry.

For Python 3.14, I’m doing all things π, pie and [mag]pie (more here). As part of the research for this important task, I looked up pi in the Oxford English Dictionary.

As we all recall from the Python 3.14.0b1 release notes, William Jones was the first person to use the π symbol to denote the circle’s circumference to its diameter in his Synopsis Palmariorum Matheseos (1706):

However, the OED’s first citation had a markup bug:

I duly reported this to the OED in July 2024; and by the next time I looked it up, in June 2025, it was fixed!

Hooray!

Header photo: Part of the definition for “get” in the OED’s 1901 forerunner, A New English Dictionary on Historical Principles (CC BY-NC-SA 2.0 Hugo van Kemenade).

I'm delighted to rejoin the Sovereign Tech Fellowship

Fri, 12 Jun 2026 07:35:12 +0000

I’m happy to rejoin the Sovereign Tech Fellowship!

I was one of six participants in the 2025 pilot to pay maintainers of critical open source technologies in the public interest. By all accounts this first cohort was a resounding success, and I’m glad to see the programme continue.

It was wonderful to be part of the inaugural Sovereign Tech Fellowship, and incredibly beneficial to my projects: it gave me the time to focus on releasing Python 3.14 and 3.15 smoothly, to mentor and onboard others, and to support the wider community.

2025 impact #

The 2025 evaluation report covers all six of us and the benefits of the programme at a higher level. Here’s some of the specific things I achieved.

I was happy with how the big Python 3.14.0 release went. This is in part from having a good team to work with, and building on the past, but no doubt also due to being able to focus and invest time thanks to the Fellowship.

On many occasions, having time to dedicate to the role meant I could prioritise things as they occurred. For example, when needing to make expedited releases, I could dedicate time to go through all the necessary prep, and make the release without any stress of fitting it in around a regular job. Similarly, when last-minute problems came up on release day, such as newly-committed code not passing tests, I didn’t need to rush, and could contact the contributor to arrange a fix. Some other release managers had reverted similar changes to let the contributor try again for a later release, but there was less pressure for me and I could wait longer.

I was able to mentor other project members, such as helping onboard the next release manager, and also answer questions for other triagers. I promoted two new triagers in different projects. Other community members sometimes asked me about how to contribute. I attended many community “office hours” meetings and Monthly Conference Organisers' calls to share what’s going on and answer questions, and likewise blogged and shared on social media such as Mastodon, Bluesky and LinkedIn. I was able to attend many conferences and give talks about the upcoming release, and discuss with other attendees what happens with Python releases and the project in general. This all helps improve transparency. I also chaired many monthly Docs Working Group meetings, and attended many other meetings from different projects.

I was able to make many improvements in the release process, through additional automation and testing to remove tedious manual steps. I’ve improved the accessibility of websites visited by tens of millions per month.

I created a triage dashboard that helped us close hundreds of issues, and also complete forgotten backports including security fixes.

I was also able to invest time on non-technical, social, organisational and governance improvements. I’m proud my proposal was accepted to alternate the Language Summit between PyCon US and EuroPython, rather than always being in the US, to improve the diversity of voices of who will shape the future of Python. The 2026 summit will be held at EuroPython and I’m helping organise.

Since 2009, the summit has been a one-day event that takes place at PyCon US before the main conference days. It has also been held at EuroPython twice, in 2010 and 2011. The PSF mission is “to support and facilitate the growth of a diverse and international community of Python programmers”, and not all potential attendees can travel to the US each year. This proposal took a lot of work:

In November 2024, the core team discussed on Discord the possibility of alternating the summit between PyCon US and EuroPython. People were in favour, but said it would need to be discussed at the summit at PyCon US in May 2025.
In March 2025, I asked for the topic to be added to the agenda, as I wasn’t attending.
In May, the discussion took place. The minutes simply said: “Watch out for a Discourse thread to discuss this.”
In July, during EuroPython, I asked summit attendees what the impression was, and they said people were in favour. I also spoke with the chair of EuroPython Society about the summit requirements, and they said they’d be happy to host us.
In August, because no Discourse thread had appeared, I opened a proposal to alternate.
In September, during the Steering Council Q&A at the Core Team Sprint, I asked about next steps. The consensus was for the SC to open a formal poll amongst the core team.
In October, the SC opened a poll for core team members.
In November, the poll concluded overwhelmingly in favour of alternating.
In December, the SC approved my request. I volunteered to help organise the summit and opened discussions with the EuroPython Society to make it happen in 2026.

I had more free time to spend on non-open source things, but also more free time to help the local Python community such as by co-organising two local meetups. One person I nominated became a Fellow of the Python Software Foundation and another of the EuroPython Society, which recognises the importance of community work.

Finally, I enjoyed our monthly Fellowship meetings where the six of us all gave a summary of our last month’s work. Similarly, it was great to meet most of them in person along with people from the Agency at the event to mark the inaugural Sovereign Tech Fellowship cohort and hear the results of the evaluation report.

Arbitrary statistics #

On GitHub:

Total contributions: 6,642
Issues created: 90
PRs created: 901
Issues closed: 446
PRs merged: 1,401
PRs closed: 142
Total issues involved with: 1,409
Total PRs involved with: 4,095
Repositories affected: 409

Made 55 releases:

13 of Python 3.14
3 of Python 3.15
39 of PyPI projects

Started maintaining:

Archived:

Attended eight conferences in Berlin (FOSS Backstage and Design), Bologna (PyCon Italia), Prague (EuroPython), Athens (PyCon Greece), Manchester (PyCon UK), Tallinn (PyCon Estonia), Jyväskylä (PyCon Finland) and Stockholm (PyCon Sweden)

On a discussion panel at one
Gave a lightning talk at five
Announced PyCon Finland at four
Helped new contributors at sprints at three
Hosted a barcamp session at one
Helped organise one by reviewing talks and through promotion
Volunteered at one

Attended three online conferences:

March: SustainOSS Virtual Forum
May: Maintainer Summit
December: PyLadiesCon

Other events:

September: Core Team Sprint in Cambridge
- Podcast interview
December: Sovereign Tech Agency event in Berlin to mark the inaugural Fellowship cohort

Meetups:

Co-organised 16 meetups for two groups, one which we restarted in 2025
Attended 27 meetups of 11 groups in four cities and three countries
Gave one long talk and four lightning talks

Docs Working Group:

Organised 12 monthly meetings, chaired 9, attended 10

User Success Workgroup:

Attended two meetings

Published 17 blog posts:

February: How to delay a Python release
February: I’m excited to join the Sovereign Tech Fellowship
February: Improving licence metadata
March: Free-threaded Python on GitHub Actions
April: My most used command-line commands
May: PEPs & Co.
June: Run coverage on tests
August: EuroPython 2025: A roundup of writeups
September: Ready prek go!
October: Releasing Python 3.14.0
October: Three times faster with lazy imports
November: Python Core Sprint 2025
November: Setting secrets in env vars
December: Steering Council election
December: Steering Council results
December: And now for something completely different
December: Replacing python-dateutil to remove six

Reported 67 accounts to GitHub for spam/abuse/inauthentic activity.

2026 and beyond #

This time we’re 14 Fellows, and not only maintainers but also community managers and technical writers. It’s great that Python core dev Stan Ulbrych and PSF director Georgi Ker are also joining, and I’m looking forward to meeting the other Sovereign Tech Fellows.

I’m really pleased to again be working with the Sovereign Tech Agency. They’re showing the world some of the ways we can improve open source and critical digital infrastructure, through a range of different programmes. Their success has informed the proposal for an EU Sovereign Tech Fund (EU-STF), and they have also helped shape the European Digital Infrastructure Consortium for digital commons (DC-EDIC), with an EU-STF pilot kicking off later this month which builds on their experience. And it’s good to see the focus on maintenance and long-term sustainability in the brand new EU Open Source Strategy, announced just last week.

And by the way, the Sovereign Tech Agency are currently hiring, check out their open positions.

Header photo by Jan Michalko.

A CLI to fight GitHub spam

Wed, 18 Feb 2026 13:35:25 +0000

`gh triage spam` #

We get a lot of spam in the CPython project .

A lot of it isn’t even slop, but mostly worthless “nothing” issues and PRs that barely fill in the issue template, or add a line of nonsense to some arbitrary file.

They’re often from new accounts with usernames like:

za9066559-wq
quanghuynh10111-png
riffocristobal579-cmd
sajjad5giot
satyamchoudhary1430-boop
SilaMey
standaell1234-maker
eedamhmd2005-ui
ksdmyanmar-lighter
experments-studios
madurangarathanayaka5-art

A new issue from a username following the pattern nameNNNN-short_suffix is a dead giveaway. I think they’re trying to farm “realistic” accounts: open a PR, open an issue, comment on something, make a fake review.

It’s easy but tedious to:

close the PR/issue as not planned
retitle to “spam”
apply the “invalid” label
remove other labels

I use the GitHub CLI gh a lot (for example, gh co NNN to check out a PR locally), and it’s straightforward to write your own Python-based extensions, so I wrote gh triage.

Install:

$ gh extension install hugovk/gh-triage
Cloning into '/Users/hugo/.local/share/gh/extensions/gh-triage'...
remote: Enumerating objects: 15, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 15 (delta 5), reused 12 (delta 2), pack-reused 0 (from 0)
Receiving objects: 100% (15/15), 5.09 KiB | 5.09 MiB/s, done.
Resolving deltas: 100% (5/5), done.
✓ Installed extension hugovk/gh-triage

Then run like gh triage spam <issue-or-pr-number-or-url>:

$ gh triage spam https://github.com/python/cpython/issues/144900
✅ Removed labels: type-bug
✅ Added labels: invalid
✅ Changed title: spam
✅ Closed

This can be used for any repo that you have permissions for: it applies the “invalid” or “spam” labels, but only if they exist in the repo.

Next step: perhaps it could print out the URL to make it easy to report the account to GitHub (usually for “Spam or inauthentic Activity”).

`gh triage unassign` #

Not spam, but another triage helper.

A less common occurrence is a rebase or merge from main or change of PR base branch that ends up bringing in lots of code changes. This often assigns the PR to dozens of people via CODEOWNERS, for example: python/cpython#142564.

Everyone’s already been pinged and subscribed to the PR, so it’s too late to help that, but we can automate unassigning them all so at least the PR is not in their “assigned to” list.

Run gh triage unassign <issue-or-pr-number-or-url> to:

remove all assignees (issues and PRs)
remove all requested reviewers (PRs only)

For example:

gh triage unassign 142564

Speeding up Pillow's open and save

Wed, 28 Jan 2026 10:29:04 +0000

Tachyon #

I tried out Tachyon, the new “high-frequency statistical sampling profiler” coming in Python 3.15, to see if we can speed up the Pillow imaging library. I started with a simple script to open an image:

import sys
from PIL import Image

im = Image.open(f"Tests/images/hopper.{sys.argv[1]}")

Then ran:

$ python3.15 -m profiling.sampling run --flamegraph /tmp/1.py png
Captured 35 samples in 0.04 seconds
Sample rate: 1,000.00 samples/sec
Error rate: 25.71
Flamegraph data: 1 root functions, total samples: 26, 169 unique strings
Flamegraph saved to: flamegraph_97927.html

Which generates this flame graph:

The whole thing took 40 milliseconds, with half in Image.py’s open(). If you visit the interactive HTML page we can see open() calls preinit(), which in turn imports GifImagePlugin, BmpImagePlugin, PngImagePlugin and JpegImagePlugin (hover over the <module> boxes to see them).

Do we really need to import all those plugins when we’re only interested in PNG?

Okay, let’s try another kind of image:

$ python3.15 -m profiling.sampling run --flamegraph /tmp/1.py webp
Captured 59 samples in 0.06 seconds
Sample rate: 1,000.00 samples/sec
Error rate: 22.03
Flamegraph data: 1 root functions, total samples: 46, 256 unique strings
Flamegraph saved to: flamegraph_98028.html

Hmm, 60 milliseconds with 80% in open() and most of that in init(). The HTML page shows it imports AvifImagePlugin, PdfImagePlugin, WebpImagePlugin, DcxImagePlugin, DdsImagePlugin and PalmImagePlugin. We also have preinit importing GifImagePlugin, BmpImagePlugin and PngImagePlugin.

Again, why import even more plugins when we only care about WebP?

Loading all the plugins? #

That’s enough profiling, let’s look at the code.

When open()ing or save()ing an image, if Pillow isn’t yet initialised, we call a preinit() function. This loads five drivers for five formats by importing their plugins: BMP, GIF, JPEG, PPM and PNG.

During import, each plugin registers its file extensions, MIME types and some methods used for opening and saving.

Then we check each of these plugins in turn to see if one will accept the image. Most of Pillow’s plugins detect an image by opening the file and checking if the first few bytes match a magic prefix. For example:

GIF starts with b"GIF87a" or b"GIF89a".
PNG starts with b"\211PNG\r\n\032\n" (reference).
JPEG starts with b"\xff\xd8\xff", where \xff\xd8 means “Start of Image”, and \xff is the start of the next marker (reference).

If none of these five match, we call init(), which imports the remaining 42 plugins. We then check each of these for a match.

This has been the case since at least PIL 1.1.1 released in 2000 (this is the oldest version I have to check). There were 33 builtin plugins then and 47 now.

Lazy loading #

This is all a bit wasteful if we only need one or two image formats during a program’s lifetime, especially for things like CLIs. Longer running programs may need a few more, but unlikely all 47.

A benefit of the plugin system is third parties can create their own plugins, but we can be more efficient with our builtins.

I opened a PR to add a mapping of file extensions to plugins. Before calling preinit() or init(), we can instead do a cheap lookup, which may save us importing, registering, and checking all those plugins.

Of course, we may have an image without an extension, or with the “wrong” extension, but that’s fine; I expect it’s rare and anyway we’ll fall back to the original preinit() -> init() flow.

After merging the PR, here’s a new flame graph for opening PNG (HTML page):

And for WebP (HTML page):

The flame graphs are scaled to the same width, but there’s far fewer boxes meaning there’s much less work now. We’re down from 40 and 60 milliseconds to 20 and 20 milliseconds.

The PR has a bunch of benchmarks which show opening a PNG (that previously loaded five plugins) is now 2.6 times faster. Opening a WebP (that previously loaded all 47 plugins), is now 14 times faster. Similarly, Saving PNG is improved by 2.2 times and WebP by 7.9 times. Success! This will be in Pillow 12.2.0.

Localising xkcd

Sat, 03 Jan 2026 14:06:51 +0000

I gave a lightning talk at a bunch of conferences in 2025 about some of the exciting new things coming in Python 3.14, including template strings.

One thing we can use t-strings for is to prevent SQL injection. The user gives you an untrusted t-string, and you can sanitise it, before using it in a safer way.

I illustrated this with xkcd 327, titled “Exploits of a Mom”, but commonly known as “Little Bobby Tables”.

I localised most of the slides for the PyCon I was at, including this comic. Here they are!

PyCon Italia #

May, Bologna

PyCon Greece #

August, Athens

PyCon Estonia #

October, Tallinn

PyCon Finland #

October, Jyväskylä

PyCon Sweden #

October, Stockholm

Thanks #

Thanks to Randall Munroe for licensing the comic under the Creative Commons Attribution-NonCommercial 2.5 License. These adaptations are therefore licensed the same way.

Finally, here’s links for 2026, I recommend them all:

PyCon Italia, 27-30 May: the CFP is open until 6th January
PyCon Estonia, 8-9 October
PyCon Greece, 12-13 October
PyCon Sweden, TBA
PyCon Finland, TBA

Replacing python-dateutil to remove six

Mon, 29 Dec 2025 16:53:00 +0000

The dateutil library is a popular and powerful Python library for dealing with dates and times.

However, it still supports Python 2.7 by depending on the six compatibility shim, and I’d prefer not to install for Python 3.10 and higher.

Here’s how I replaced three uses of its relativedelta in a couple of CLIs that didn’t really need to use it.

One #

norwegianblue was using it to calculate six months from now:

import datetime as dt

from dateutil.relativedelta import relativedelta

now = dt.datetime.now(dt.timezone.utc)
# datetime.datetime(2025, 12, 29, 15, 59, 44, 518240, tzinfo=datetime.timezone.utc)
six_months_from_now = now + relativedelta(months=+6)
# datetime.datetime(2026, 6, 29, 15, 59, 44, 518240, tzinfo=datetime.timezone.utc)

But we don’t need to be so precise here, and 180 days is good enough, using the standard library’s datetime.timedelta:

import datetime as dt

now = dt.datetime.now(dt.timezone.utc)
# datetime.datetime(2025, 12, 29, 15, 59, 44, 518240, tzinfo=datetime.timezone.utc)
six_months_from_now = now + dt.timedelta(days=180)
# datetime.datetime(2026, 6, 27, 15, 59, 44, 518240, tzinfo=datetime.timezone.utc)

Two #

pypistats was using it get the last day of a month:

import datetime as dt

first = dt.date(year, month, 1)
# datetime.date(2025, 12, 1)
last = first + relativedelta(months=1) - relativedelta(days=1)
# datetime.date(2025, 12, 31)

Instead, we can use the stdlib’s calendar.monthrange:

import calendar
import datetime as dt

last_day = calendar.monthrange(year, month)[1]
# 31
last = dt.date(year, month, last_day)
# datetime.date(2025, 12, 31)

Three #

Finally, to get last month as a yyyy-mm string:

import datetime as dt

from dateutil.relativedelta import relativedelta

today = dt.date.today()
# datetime.date(2025, 12, 29)
d = today - relativedelta(months=1)
# datetime.date(2025, 11, 29)
d.isoformat()[:7]
# '2025-11'

Instead:

import datetime as dt

today = dt.date.today()
# datetime.date(2025, 12, 29)
if today.month == 1:
 year, month = today.year - 1, 12
else:
 year, month = today.year, today.month - 1
 # 2025, 11
f"{year}-{month:02d}"
# '2025-11'

Goodbye six, and we also get slightly quicker install, import and run times.

Bonus #

I recommend Adam Johnson’s tip to import datetime as dt to avoid the ambiguity of which datetime is the module and which is the class.

Header photo: Ver Sacrum calendar by Alfred Roller

And now for something completely different

Tue, 23 Dec 2025 13:03:33 +0000

Starting in 2019, Python 3.8 and 3.9 release manager Łukasz Langa added a new section to the release notes called “And now for something completely different” with a sketch transcript from Monty Python.

For Python 3.10 and 3.11, the next release manager Pablo Galindo Salgado continued the section but included astrophysics facts.

For Python 3.12, the next RM Thomas Wouters shared poems (and took a break for 3.13).

And for Python 3.14, I’m doing all things π, pie and [mag]pie.

Here’s a collection of my different things for the first year (and a bit) of Python 3.14.

alpha 1 #

2024-10-15

π (or pi) is a mathematical constant, approximately 3.14, for the ratio of a circle’s circumference to its diameter. It is an irrational number, which means it cannot be written as a simple fraction of two integers. When written as a decimal, its digits go on forever without ever repeating a pattern.

Here’s 76 digits of π:

3.141592653589793238462643383279502884197169399375105820974944592307816406286

Piphilology is the creation of mnemonics to help remember digits of π.

In a pi-poem, or “piem”, the number of letters in a word equal the corresponding digit. This covers 9 digits, 3.14159265:

How I wish I could recollect pi easily today!

One of the most well-known covers 15 digits, 3.14159265358979:

How I want a drink, alcoholic of course, after the heavy chapters involving quantum mechanics!

Here’s a 35-word piem in the shape of a circle, 3.1415926535897932384626433832795728:

It’s a fact A ratio immutable Of circle round and width, Produces geometry’s deepest conundrum. For as the numerals stay random, No repeat lets out its presence, Yet it forever stretches forth. Nothing to eternity.

The Guinness World Record for memorising the most digits is held by Rajveer Meena, who recited 70,000 digits blindfold in 2015. The unofficial record is held by Akira Haraguchi who recited 100,000 digits in 2006.

alpha 2 #

2024-11-19

Ludolph van Ceulen (1540-1610) was a fencing and mathematics teacher in Leiden, Netherlands, and spent around 25 years calculating π (or pi), using essentially the same methods Archimedes employed some seventeen hundred years earlier.

Archimedes estimated π by calculating the circumferences of polygons that fit just inside and outside of a circle, reasoning the circumference of the circle lies between these two values. Archimedes went up to polygons with 96 sides, for a value between 3.1408 and 3.1428, which is accurate to two decimal places.

Van Ceulen used a polygon with half a billion sides. He published a 20-decimal value in his 1596 book Vanden Circkel (“On the Circle”), and later expanded it to 35 decimals:

3.14159265358979323846264338327950288

Van Ceulen’s 20 digits is more than enough precision for any conceivable practical purpose. For example, even if a printed circle was perfect down to the atomic scale, the thermal vibrations of the molecules of ink would make most of those digits physically meaningless. NASA Jet Propulsion Laboratory’s highest accuracy calculations, for interplanetary navigation, uses 15 decimals: 3.141592653589793.

At Van Ceulen’s request, his upper and lower bounds for π were engraved on his tombstone in Leiden. The tombstone was eventually lost but restored in 2000. In the Netherlands and Germany, π is sometimes referred to as the “Ludolphine number”, after Van Ceulen.

alpha 3 #

2024-12-17

A mince pie is a small, round covered tart filled with “mincemeat”, usually eaten during the Christmas season – the UK consumes some 800 million each Christmas. Mincemeat is a mixture of things like apple, dried fruits, candied peel and spices, and originally would have contained meat chopped small, but rarely nowadays. They are often served warm with brandy butter.

According to the Oxford English Dictionary, the earliest mention of Christmas mince pies is by Thomas Dekker, writing in the aftermath of the 1603 London plague, in Newes from Graues-end: Sent to Nobody (1604):

Ten thousand in London swore to feast their neighbors with nothing but plum-porredge, and mince-pyes all Christmas.

Here’s a meaty recipe from Rare and Excellent Receipts, Experienc’d and Taught by Mrs Mary Tillinghast and now Printed for the Use of her Scholars Only (1678):

XV. How to make Mince-pies.

To every pound of Meat, take two pound of beef Suet, a pound of Corrants, and a quarter of an Ounce of Cinnamon, one Nutmeg, a little beaten Mace, some beaten Colves, a little Sack & Rose-water, two large Pippins, some Orange and Lemon peel cut very thin, and shred very small, a few beaten Carraway-seeds, if you love them the Juyce of half a Lemon squez’d into this quantity of meat; for Sugar, sweeten it to your relish; then mix all these together and fill your Pie. The best meat for Pies is Neats-Tongues, or a leg of Veal; you may make them of a leg of Mutton if you please; the meat must be parboyl’d if you do not spend it presently; but if it be for present use, you may do it raw, and the Pies will be the better.

alpha 4 #

2025-01-14

In Python, you can use Greek letters as constants. For example:

from math import pi as π

def circumference(radius: float) -> float:
 return 2 * π * radius

print(circumference(6378.137)) # 40075.016685578485

alpha 5 #

2025-02-11

2025-01-29 marked the start of a new lunar year, the Year of the Snake :snake: (and the Year of Python?).

For centuries, π was often approximated as 3 in China. Some time between the years 1 and 5 CE, astronomer, librarian, mathematician and politician Liu Xin (劉歆) calculated π as 3.154.

Around 130 CE, mathematician, astronomer, and geographer Zhang Heng (張衡, 78–139) compared the celestial circle with the diameter of the earth as 736:232 to get 3.1724. He also came up with a formula for the ratio between a cube and inscribed sphere as 8:5, implying the ratio of a square’s area to an inscribed circle is √8:√5. From this, he calculated π as √10 (~3.162).

Third century mathematician Liu Hui (刘徽) came up with an algorithm for calculating π iteratively: calculate the area of a polygon inscribed in a circle, then as the number of sides of the polygon is increased, the area becomes closer to that of the circle, from which you can approximate π.

This algorithm is similar to the method used by Archimedes in the 3rd century BCE and Ludolph van Ceulen in the 16th century CE (see 3.14.0a2 release notes), but Archimedes only went up to a 96-sided polygon (96-gon). Liu Hui went up to a 192-gon to approximate π as 157/50 (3.14) and later a 3072-gon for 3.14159.

Liu Hu wrote a commentary on the book The Nine Chapters on the Mathematical Art which included his π approximations.

In the fifth century, astronomer, inventor, mathematician, politician, and writer Zu Chongzhi (祖沖之, 429–500) used Liu Hui’s algorithm to inscribe a 12,288-gon to compute π between 3.1415926 and 3.1415927, correct to seven decimal places. This was more accurate than Hellenistic calculations and wouldn’t be improved upon for 900 years.

Happy Year of the Snake!

alpha 6 #

2025-03-14

March 14 is celebrated as pi day, because 3.14 is an approximation of π. The day is observed by eating pies (savoury and/or sweet) and celebrating π. The first pi day was organised by physicist and tinkerer Larry Shaw of the San Francisco Exploratorium in 1988. It is also the International Day of Mathematics and Albert Einstein’s birthday. Let’s all eat some pie, recite some π, install and test some py, and wish a happy birthday to Albert, Loren and all the other pi day children!

alpha 7 #

2025-04-08

On Saturday, 5th April, 3.141592653589793 months of the year had elapsed.

beta 1 #

2025-05-07

The mathematical constant pi is represented by the Greek letter π and represents the ratio of a circle’s circumference to its diameter. The first person to use π as a symbol for this ratio was Welsh self-taught mathematician William Jones in 1706. He was a farmer’s son born in Llanfihangel Tre’r Beirdd on Angelsy (Ynys Môn) in 1675 and only received a basic education at a local charity school. However, the owner of his parents’ farm noticed his mathematical ability and arranged for him to move to London to work in a bank.

By age 20, he served at sea in the Royal Navy, teaching sailors mathematics and helping with the ship’s navigation. On return to London seven years later, he became a maths teacher in coffee houses and a private tutor. In 1706, Jones published Synopsis Palmariorum Matheseos which used the symbol π for the ratio of a circle’s circumference to diameter (hunt for it on pages 243 and 263 or here). Jones was also the first person to realise π is an irrational number, meaning it can be written as decimal number that goes on forever, but cannot be written as a fraction of two integers.

But why π? It’s thought Jones used the Greek letter π because it’s the first letter in perimetron or perimeter. Jones was the first to use π as our familiar ratio but wasn’t the first to use it in as part of the ratio. William Oughtred, in his 1631 Clavis Mathematicae (The Key of Mathematics), used π/δ to represent what we now call pi. His π was the circumference, not the ratio of circumference to diameter. James Gregory, in his 1668 Geometriae Pars Universalis (The Universal Part of Geometry) used π/ρ instead, where ρ is the radius, making the ratio 6.28… or τ. After Jones, Leonhard Euler had used π for 6.28…, and also p for 3.14…, before settling on and popularising π for the famous ratio.

beta 2 #

2025-05-26

In 1897, the State of Indiana almost passed a bill defining π as 3.2.

Of course, it’s not that simple.

Edwin J. Goodwin, M.D., claimed to have come up with a solution to an ancient geometrical problem called squaring the circle, first proposed in Greek mathematics. It involves trying to draw a circle and a square with the same area, using only a compass and a straight edge. It turns out to be impossible because π is transcendental (and this had been proved just 13 years earlier by Ferdinand von Lindemann), but Goodwin fudged things so the value of π was 3.2 (his writings have included at least nine different values of π: including 4, 3.236, 3.232, 3.2325… and even 9.2376…).

Goodwin had copyrighted his proof and offered it to the State of Indiana to use in their educational textbooks without paying royalties, provided they endorsed it. And so Indiana Bill No. 246 was introduced to the House on 18th January 1897. It was not understood and initially referred to the House Committee on Canals, also called the Committee on Swamp Lands. They then referred it to the Committee on Education, who duly recommended on 2nd February that “said bill do pass”. It passed its second reading on the 5th and the education chair moved that they suspend the constitutional rule that required bills to be read on three separate days. This passed 72-0, and the bill itself passed 67-0.

The bill was referred to the Senate on 10th February, had its first reading on the 11th, and was referred to the Committee on Temperance, whose chair on the 12th recommended “that said bill do pass”.

A mathematics professor, Clarence Abiathar Waldo, happened to be in the State Capitol on the day the House passed the bill and walked in during the debate to hear an ex-teacher argue:

The case is perfectly simple. If we pass this bill which establishes a new and correct value for pi , the author offers to our state without cost the use of his discovery and its free publication in our school text books, while everyone else must pay him a royalty.

Waldo ensured the senators were “properly coached”; and on the 12th, during the second reading, after an unsuccessful attempt to amend the bill it was postponed indefinitely. But not before the senators had some fun.

The Indiana News reported on the 13th:

…the bill was brought up and made fun of. The Senators made bad puns about it, ridiculed it and laughed over it. The fun lasted half an hour. Senator Hubbell said that it was not meet for the Senate, which was costing the State $250 a day, to waste its time in such frivolity. He said that in reading the leading newspapers of Chicago and the East, he found that the Indiana State Legislature had laid itself open to ridicule by the action already taken on the bill. He thought consideration of such a proposition was not dignified or worthy of the Senate. He moved the indefinite postponement of the bill, and the motion carried.

beta 3 #

2025-06-17

If you’re heading out to sea, remember the Maritime Approximation:

π mph = e knots

beta 4 #

2025-07-08

All this talk of π and yet some say π is wrong. Tau Day (June 28th, 6/28 in the US) celebrates τ as the “true circle constant”, as the ratio of a circle’s circumference to its radius, C/r = 6.283185… The Tau Manifesto declares π “a confusing and unnatural choice for the circle constant”, in part because “2π occurs with astonishing frequency throughout mathematics”.

If you wish to embrace τ the good news is PEP 628 added math.tau to Python 3.6 in 2016:

When working with radians, it is trivial to convert any given fraction of a circle to a value in radians in terms of tau. A quarter circle is tau/4, a half circle is tau/2, seven 25ths is 7*tau/25, etc. In contrast with the equivalent expressions in terms of pi (pi/2, pi, 14*pi/25), the unnecessary and needlessly confusing multiplication by two is gone.

release candidate 1 #

2025-07-22

Today, 22nd July, is Pi Approximation Day, because 22/7 is a common approximation of π and closer to π than 3.14.

22/7 is a Diophantine approximation, named after Diophantus of Alexandria (3rd century CE), which is a way of estimating a real number as a ratio of two integers. 22/7 has been known since antiquity; Archimedes (3rd century BCE) wrote the first known proof that 22/7 overestimates π by comparing 96-sided polygons to the circle it circumscribes.

Another approximation is 355/113. In Chinese mathematics, 22/7 and 355/113 are respectively known as Yuelü (约率; yuēlǜ; “approximate ratio”) and Milü (密率; mìlǜ; “close ratio”).

Happy Pi Approximation Day!

release candidate 2 #

2025-08-14

The magpie, Pica pica in Latin, is a black and white bird in the crow family, known for its chattering call.

The first-known use in English is from a 1589 poem, where magpie is spelled “magpy” and cuckoo is “cookow”:

Th[e]y fly to wood like breeding hauke, And leave old neighbours loue, They pearch themselves in syluane lodge, And soare in th’ aire aboue. There : magpy teacheth them to chat, And cookow soone doth hit them pat.

The name comes from Mag, short for Margery or Margaret (compare robin redbreast, jenny wren, and its corvid relative jackdaw); and pie, a magpie or other bird with black and white (or pied) plumage. The sea-pie (1552) is the oystercatcher, the grey pie (1678) and murdering pie (1688) is the great grey shrike. Others birds include the yellow and black pie, red-billed pie, wandering tree-pie, and river pie. The rain-pie, wood-pie and French pie are woodpeckers.

Pie on its own dates to before 1225, and comes from the Latin name for the bird, pica.

release candidate 3 #

2025-09-18

According to Pablo Galindo Salgado at PyCon Greece:

There are things that are supercool indeed, like for instance, this is one of the results that I’m more proud about. This equation over here, which you don’t need to understand, you don’t need to be scared about, but this equation here tells what is the maximum time that it takes for a ray of light to fall into a black hole. And as you can see the math is quite complicated but the answer is quite simple: it’s 2π times the mass of the black hole. So if you normalise by the mass of the black hole, the answer is 2π. And because there is nothing specific about your election of things in this formula, this formula is universal. It means it doesn’t depend on anything other than nature itself. Which means that you can use this as a definition of π. This is a valid alternative definition of the number π. It’s literally half the maximum time it takes to fall into a black hole, which is kind of crazy. So next time someone asks you what π means you can just drop this thing and impress them quite a lot. Maybe Hugo could use this information to put it into the release notes of πthon [yes, I can, thank you!].

3.14.0 (final) #

2025-10-07

Edgar Allen Poe died on 7th October 1849.

As we all recall from 3.14.0a1, piphilology is the creation of mnemonics to help memorise the digits of π, and the number of letters in each word in a pi-poem (or “piem”) successively correspond to the digits of π.

In 1995, Mike Keith, an American mathematician and author of constrained writing, retold Poe’s The Raven as a 740-word piem. Here’s the first two stanzas of Near A Raven:

Poe, E. Near a Raven

Midnights so dreary, tired and weary. Silently pondering volumes extolling all by-now obsolete lore. During my rather long nap - the weirdest tap! An ominous vibrating sound disturbing my chamber’s antedoor. “This”, I whispered quietly, “I ignore”.

Perfectly, the intellect remembers: the ghostly fires, a glittering ember. Inflamed by lightning’s outbursts, windows cast penumbras upon this floor. Sorrowful, as one mistreated, unhappy thoughts I heeded: That inimitable lesson in elegance - Lenore - Is delighting, exciting…nevermore.

3.14.1 #

2025-12-02

Seki Takakazu (関孝和; c. March 1642 – December 5, 1708) was a Japanese mathematician and samurai who laid the foundations of Japanese mathematics, later known as wasan (和算, from wa (“Japanese”) and san (“calculation”).

Seki was a contemporary of Isaac Newton and Gottfried Leibniz but worked independently. He created a new algebraic system, worked on infinitesimal calculus, and is credited with the discovery of Bernoulli numbers (before Bernoulli’s birth).

Seki also calculated π to 11 decimal places using a polygon with 131,072 sides inscribed within a circle, using an acceleration method now known as Aitken’s delta-squared process, which was rediscovered by Alexander Aitken in 1926.

Header photo: A scan of Seki Takakazu’s posthumous Katsuyō Sanpō (1712) showing calculations of π.

Steering Council results

Sat, 13 Dec 2025 14:40:12 +0000

The Python Steering Council 2026 election results are in and congratulations to the new Python Steering Council!

Welcome Savannah for the first time, and thank you to Greg Smith and Emily Morehouse for four and three years’ service each.

Three are starting their sixth terms, and four members have been or are release managers.

The chart above only covers the Steering Council years. Let’s also not forget Guido van Rossum’s BDFL years:

Steering Council election

Mon, 08 Dec 2025 19:50:58 +0000

🗳️ Update: See the results!

The Python Steering Council election is on!

This year six candidates are running for the five seats. See PEP 8107 for links to their nomination statements and PEP 13 for more on Python language governance.

I made a chart to show when the nominations arrived during the nomination period: six is fewer than in previous years, but they didn’t all wait until the last three days like in 2023 and 2024.

This year we’re also using a new voting system: with “Multi-winner Bloc STAR” you give between 0-6 stars per candidate.

If you’re in the core team, remember to vote by Friday!

Python Core Sprint 2025

Mon, 03 Nov 2025 08:58:00 +0000

🐍🏃In September, the annual Python Core Sprint was hosted by Arm in Cambridge, UK!

The plan: put 35 core developers and 13 special guests in a room for a week, and see what they cook up.

Monday highlights #

We kicked off the first day with a round of five-word intros (mine: “three”, “dot”, “fourteen”, “release”, “manager”), lots of talks, and lots of discussion about talks:

Ken Jin Ooi - Building a JIT Community and Demo [effect] of new C API
Antonio Cuni - Tracing JITs on real code
Brett Cannon - WASI update and Precompiled binaries from python.org
Hood Chatham - Upstreaming Pyodide FFI
Russell Keith-Magee - Managing cross-platform wheel builds
Steering Council - PEP 793 and abi3/abi3t/abi4
Matthew Parkinson - Designing Deep Immutability

I did some ~~sprint~~ spring cleaning of our PyPI projects, dropping support for then-almost-EOL Python 3.9:

And because Mariatta wasn’t with us, here’s the all-important Python T-shirt census:

Menlo Park 2016 core sprint (Greg, Guido)
EuroPython 2023 (Antonio)
EuroPython 2024 volunteer (Hugo)
PyCon Canada 2016 (Brett)
PyCon US Pittsburgh Charlas (Thomas)
Cambridge 2025 core sprint (Diego)

The sprint room

Tuesday highlights #

Release day?

I’d originally planned to release Python 3.14.0rc3 on the Tuesday, but the morning was full of presentations, and the afternoon had an early departure for the social event, so I moved it to Wednesday instead.

Tania Allard gave a presentation about the different types of mentorship and how we can improve, followed by an open discussion.

Gregory P. Smith gave a demo on how we can use tools like Claude with CPython.

Tania, Jannis Leidel, Carol Willing and I discussed the User Success Workgroup and we came up with some ideas on next steps.

We ended the day with a punting tour on the river Cam and dinner at Jesus College, thank you, Arm!

And Thomas Wouters gave a fun session of his Feuding Pythonistas game (spoiler: people are wrong on the internet).

Python/tech t-shirt census:

Python 23.7, Menlo Park 20167 core sprint (Carl)
PyCon Italia 2025 (Antonio)
Python 3.14 pie (Hugo)
WHY 2025 (Thomas)
PyCon US 2025 (Diego)
PyCascades (Guido)
EuroPython 2022 (Mark)
Pythonista (Jacob)

Punting on the Cam

Wednesday highlights #

Release day?

The Steering Council asked for an extra day to decide about a possible typing revert (python/steering-council#307), so not today.

Lightning talks:

⚡ Guido van Rossum on collecting an oral history
⚡ Ee Durbin on modernising the last bits of infra and bots
⚡ Steve Dower demoed the new Windows installer
⚡ Larry Hastings on a linked_list date type
⚡ Adam Turner asked shall we close old issues
⚡ Greg shared a draft PEP for timestamps on async tracebacks

Carol, Adam, Thomas, Petr Viktorin and I discussed a number of docs topics.

I released the Python Docs Sphinx Theme with more translations.

We had a Q&A session with the Steering Council, three in-person and two joining remotely.

Jacob Coffee and I looked into upgrading the Python Insider and PSF blogs into something a little more modern.

T-shirt census:

PyCon Greece 2025 (Hugo)
PyCon Korea 2025 (Donghee)
PyLadies (Savannah, Petr)
EuroPython 2024 (Lys)
PyCon Italia 2023 (Antonio)
Cambridge 2025 core sprint (Guido)

The Python Steering Council

Thursday highlights #

Release day? Yes!

The Steering Council decided not to revert, so full steam ahead with the release.

Savannah Ostrowski, release manager for 3.16 and 3.17, shadowed to see what the process looks like (not as bad as it looks like in PEP 101).

Time for a couple of quick PRs and an interview with Pablo Galindo Salgado and Łukasz Langa on the core.py podcast, along with 29 others!

T-census:

PyCon UA 2016 (Łukasz)
Cambridge 2025 core sprint (Donghee)
Read the Docs Budapest 2014 (Petr)
Quansight Labs “Sustaining the future of Open Source” hoodie (Lys)

The 3.14.0rc3 release in progress

Friday highlights #

I went to Manchester to attend PyCon UK. Some highlights:

The conference also included sprints, and Adam and I ran the CPython sprint. We had a big table full of contributors and a few made their very first contributions, which is always rewarding for all involved!

PyCon UK sprint

Some numbers for me during the week:

Python release candidates released: 1
Issues created: 1
PRs created: 8
Issues closed: 7
PRs merged: 28
PRs closed: 1
Total issues involved with: 18
Total PRs involved with: 70
Repositories affected: 19

Thank you #

Huge thanks to Diego Russo and Arm for arranging and hosting us. The core sprint is always a highlight of the year and an incredibly productive week.

Read writeups by Diego and Antonio, and I recommend listening to Łukasz and Pablo’s core.py podcast for interviews with 18 (part one) and 12 sprinters (part two). They’re long, but it’s fascinating to hear all the different things everyone is working on.

Header photo by Arm

Three times faster with lazy imports

Sun, 19 Oct 2025 16:05:00 +0000

PEP 810 proposes “explicit lazy imports” for Python 3.15:

Lazy imports defer the loading and execution of a module until the first time the imported name is used, in contrast to ‘normal’ imports, which eagerly load and execute a module at the point of the import statement.

By allowing developers to mark individual imports as lazy with explicit syntax, Python programs can reduce startup time, memory usage, and unnecessary work. This is particularly beneficial for command-line tools, test suites, and applications with large dependency graphs.

It’s not been accepted yet, but let’s try out the reference implementation on one of my CLI tools, pypistats.

Setup #

First fetch the reference implementation. From a CPython checkout:

git remote add LazyImportsCabal https://github.com/LazyImportsCabal/cpython
git fetch LazyImportsCabal
gco lazy # see https://hugovk.dev/blog/2025/my-most-used-command-line-commands/

Because we want to install NumPy and pandas, let’s pretend to be Python 3.14 so we can use the binary wheels instead of having to build from source:

--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
 /* Version parsed out into numeric values */
 /*--start constants--*/
 #define PY_MAJOR_VERSION 3
-#define PY_MINOR_VERSION 15
+#define PY_MINOR_VERSION 14
 #define PY_MICRO_VERSION 0
 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_ALPHA
 #define PY_RELEASE_SERIAL 0

 /* Version as a string */
-#define PY_VERSION "3.15.0a0"
+#define PY_VERSION "3.14.0a0"
 /*--end constants--*/

--- a/configure.ac
+++ b/configure.ac
-m4_define([PYTHON_VERSION], [3.15])
+m4_define([PYTHON_VERSION], [3.14])

Build non-debug CPython with optimisations:

GDBM_CFLAGS="-I$(brew --prefix gdbm)/include" \
 GDBM_LIBS="-L$(brew --prefix gdbm)/lib -lgdbm" \
 ./configure --enable-optimizations --with-lto \
 --with-system-libmpdec --config-cache \
 --with-openssl="$(brew --prefix openssl@3)" && make -s -j8

Install NumPy and pandas:

./python.exe -m pip install numpy pandas

And then an editable install of the CLI, because we’ll also test changing the imports:

./python.exe -m pip install -e ~/github/pypistats

Let’s check the dependencies with pipdeptree:

uvx "pipdeptree[graphviz]" --python ./python.exe --packages pypistats --graph-output svg > pipdeptree.svg

pypistats has seven direct dependencies, which result in a total of 41 dependencies six layers deep, not counting NumPy and pandas:

Benchmarks #

Let’s benchmark running pypistats --help, which is meant to be quick, using hyperfine:

brew install hyperfine

Inline imports #

In the pypistats CLI, I had already applied the trick of moving heavier imports into the functions that call them (the PEP calls these “inline imports”).

Instead of the lazy keyword, I’m using the PYTHON_LAZY_IMPORTS env var here to make it easy to compare two different runs.

$ hyperfine --warmup 10 --runs 20 --export-json out.json \
 "./python.exe -m pypistats --help" \
 "PYTHON_LAZY_IMPORTS=on ./python.exe -m pypistats --help"
Benchmark 1: ./python.exe -m pypistats --help
 Time (mean ± σ): 46.2 ms ± 1.1 ms [User: 38.8 ms, System: 6.4 ms]
 Range (min … max): 45.1 ms … 49.6 ms 20 runs

Benchmark 2: PYTHON_LAZY_IMPORTS=on ./python.exe -m pypistats --help
 Time (mean ± σ): 35.3 ms ± 0.5 ms [User: 29.5 ms, System: 4.8 ms]
 Range (min … max): 34.6 ms … 36.3 ms 20 runs

Summary
 PYTHON_LAZY_IMPORTS=on ./python.exe -m pypistats --help ran
 1.31 ± 0.04 times faster than ./python.exe -m pypistats --help

Plotted with plot_progression.py:

From 46 to 35 milliseconds, or, 1.31 times faster, not bad.

Fully lazy #

But we no longer need the inline imports trick with PEP 810!

I modified the CLI so all imports are at the top, and also removed if TYPE_CHECKING: guards. Here’s a diff.

$ hyperfine --warmup 10 --runs 20 --export-json out2.json \
 "./python.exe -m pypistats --help" \
 "PYTHON_LAZY_IMPORTS=on ./python.exe -m pypistats --help"
Benchmark 1: ./python.exe -m pypistats --help
 Time (mean ± σ): 104.1 ms ± 1.6 ms [User: 88.2 ms, System: 14.5 ms]
 Range (min … max): 101.9 ms … 109.5 ms 20 runs

Benchmark 2: PYTHON_LAZY_IMPORTS=on ./python.exe -m pypistats --help
 Time (mean ± σ): 35.7 ms ± 0.5 ms [User: 29.8 ms, System: 4.8 ms]
 Range (min … max): 34.7 ms … 36.5 ms 20 runs

Summary
 PYTHON_LAZY_IMPORTS=on ./python.exe -m pypistats --help ran
 2.92 ± 0.06 times faster than ./python.exe -m pypistats --help

From 104 to 36 milliseconds, or 2.92 times faster, much better!

Header photo: “Lazy Man Fishing” at Cascade Locks on the Columbia River 05/1973 in the U.S. National Archives , with no known copyright restrictions.

Releasing Python 3.14.0

Sat, 11 Oct 2025 17:04:00 +0000

Prologue #

I livetooted the release of Python 3.14.0. Here it is in blogpost form!

One week #

Only one week left until the release of Python 3.14.0 final!

What are you looking forward to?

#Python #Python314
Tue, Sep 30, 2025, 15:19 EEST

Three days #

Three days until release and a bug in the Linux kernel has turned a dozen buildbots red…

It’s already been fixed in the kernel, but will take some time to bubble up. We’ll skip that test for relevant kernel versions in the meantime.

#Python #Python314
Sat, Oct 4, 2025, 16:15 EEST

Green #

And back to green!

#Python #Python314
Sun, Oct 5, 2025, 16:58 EEST

Release day! #

First off, check blockers and buildbots.

A new release-blocker appeared yesterday (because of course) but it can wait until 3.14.1.

Three deferred-blockers are also waiting until 3.14.1.

A new tier-2 buildbot failure appeared yesterday (because of course) but it had previously been offline for a month and will need some reconfiguration. Can ignore.

OK, let’s make a Python!

#Python #Python314 #release
Tue, Oct 7, 2025, 11:40 EEST

`run_release.py` #

Next up, merge and backport the final change to What’s New in Python 3.14 to declare it latest stable.

Now start run_release.py, the main release automation script, which does a bunch of pre-checks, runs blurb to create a merged changelog, bumps some numbers, and pushes a branch and tag to my fork. It’ll go upstream at the end of a successful build.

Then kick off the CI to build source zips, docs and Android binaries.

#Python #Python314 #release
Tue, Oct 7, 2025, 12:43 EEST

Installers #

(That’s actually the second CI attempt, we had to update some script arguments following an Android test runner update.)

This build takes about half an hour.

I’ve also informed the Windows and macOS release managers about the tag and they will start up installer builds.

This takes a few hours, so I’ve got time to finish up the release notes.

PEP 101 is the full process, but much is automated and we don’t need to follow it all manually.

#Python #Python314 #release
Tue, Oct 7, 2025, 12:52 EEST

Windows #

The Windows build has been started.

The jobs with profile-guided optimisation (PGO) build once, then collect a profile by running the tests, and then build again using that profile, to see how ‘real’ code executes and optimises for that.

Meanwhile, the docs+source+Android build has finished and the artifacts have been copied to where they need to go with SBOMs created.

#Python #Python314 #release
Tue, Oct 7, 2025, 13:50 EEST

macOS #

The Windows build is ready and macOS is underway.

#Python #Python314 #release
Tue, Oct 7, 2025, 15:36 EEST

Final steps #

macOS installer done, next on to the final publishing and announcing steps.

#Python #Python314 #release
Tue, Oct 7, 2025, 17:02 EEST

🚀 It’s out! #

🥧 Please install and enjoy Python 3.14!

#Python #Python314 #release
Tue, Oct 7, 2025, 17:27 EEST

Finally #

And the last few tasks: announce also on the blog & mailing lists, update the PEP & downloads landing page, fix Discourse post links, unlock the 3.14 branch for the core team to start landing PRs that didn’t need to be in the RC, and eat the pie.

A HUGE thanks to @sovtechfund Fellowship for allowing me to dedicate my time on getting this out 🎉

#Python #Python314 #release
Tue, Oct 7, 2025, 19:28 EEST

Ready prek go!

Sat, 06 Sep 2025 15:50:00 +0000

I’ve been using prek recently as a drop-in replacement for pre-commit.

It uses uv for managing Python virtual environments and dependencies, is rewritten in Rust (because of course) and uses the same .pre-commit-config.yaml as pre-commit.

Its homepage says it’s not yet production ready, but several projects like Apache Airflow and PDM are already using it. I’ve been using it for a while and reporting issues as I find them; the maintainer is quick with fixes and releases. All the hooks I need are now supported.

Getting started #

First install using one of the many methods, then:

cd my-repo
pre-commit uninstall # remove the old hooks
prek install # install the new hooks
prek run --all-files # run all lints on all files
git commit -m "my commit" # run on the changed files in a commit

Benchmarks #

prek is noticeably quicker at installing hooks.

⚡ About 10x faster than pre-commit and uses only a third of disk space.

This 10x is a comparison of installing hooks using the excellent hyperfine benchmarking tool.

Here’s my own comparison.

In the first test, I initially ran pre-commit clean or prek clean to clear their caches. I then ran each tool with run --all-files in serial on the 126 repos I have cloned right now, 84 of which have a .pre-commit-config.yaml file. Each then downloads and installs the hooks when needed, then runs the lint tools.
Second, because running the lint tools should be independent and constant time for both tools, the next test ran install-hooks instead of run --all files.

To get an idea of the amount of work for these two tests, pre-commit reported initialising environments 217 times.
Finally, let’s run hyperfine on the Pillow config, which installs hooks from 14 repos:

❯ hyperfine \
--prepare 'rm -rf ~/.cache/prek/ && rm -rf ~/.cache/pre-commit && rm -rf ~/.cache/uv' \
--setup 'prek --version && pre-commit --version' \
'prek install-hooks' \
'pre-commit install-hooks'

Results #

	`pre-commit`	`prek`	Times faster
`run --all-files`	17m13s	7m59s	2.16
`install-hooks`	10m48s	2m24s	4.50
`hyperfine`	39.841s	5.539s	7.19

The hyperfine results:

Benchmark 1: prek install-hooks
 Time (mean ± σ): 5.539 s ± 0.176 s [User: 8.973 s, System: 5.692 s]
 Range (min … max): 5.231 s … 5.834 s 10 runs

Benchmark 2: pre-commit install-hooks
 Time (mean ± σ): 39.841 s ± 2.017 s [User: 19.760 s, System: 8.203 s]
 Range (min … max): 36.930 s … 43.976 s 10 runs

Summary
 prek install-hooks ran
 7.19 ± 0.43 times faster than pre-commit install-hooks

Give it a try and give it a ⭐!

Bonus #

These are the aliases I have set for pre-commit and prek:

alias pci="pre-commit install --allow-missing-config"
alias pcu="pre-commit uninstall"
alias pca="pre-commit autoupdate --jobs 0"
alias pcr="pre-commit run --all-files"
alias pki="prek install --allow-missing-config"
alias pku="prek uninstall"
alias pka="prek autoupdate --jobs 0"
alias pkr="prek run --all-files"

Where:

install’s --allow-missing-config prevents failing with an error code when a repo has no config file
autoupdate’s --jobs 0 uses all the available threads to make it faster

Header photo: AH-1G Aircraft Maintenance Test Flight Handbook and handwritten checklist for helicopters in the San Diego Air and Space Museum Archive, with no known copyright restrictions.

EuroPython 2025: A roundup of writeups

Mon, 25 Aug 2025 13:59:00 +0000

Some out-of-context quotes:

“We can just bump the version and move on.” – Dr. Brett Cannon
“You just show up. That’s it.” – Rodrigo Girão Serrão
“If it kwargs like a dorg, it’s a dorg.” – Sebastián Ramírez
“Our job will be to put the human in.” – Paul Everitt

20 July 2025

Is AI Leaving the Python Community Behind? by Georgi Ker

21 July 2025

23 July 2025

First EuroPython by Raffaella Suardini

24 July 2025

EuroPython 2025 Recap by Cheuk Ting Ho

11 August 2025

Personal highlights of EuroPython 2025 by Rodrigo Girão Serrão

12 August 2025

EuroPython 2025: “I came for the language, but stayed for the community” by Jürgen Gmach

17 August 2025

PyLadies, PacMan, and Public Health by Gauden Galea

21 August 2025

EuroPython 2025開幕までの道のり⁠⁠、Day 1の注目セッション by Takanori Suzuki

8 September 2025

Recap EuroPython 2025 by Victor Brinkhorst

And a bunch of LinkedIn posts:

Finally, the official photos and videos should be up soon, and here are my photos.

Update #

2025-09-06: Added Gauden Galea’s writeup
2025-09-09: Added Victor Brinkhorst’s writeup

Header photo: Savannah Bailey’s keynote (CC BY-NC-SA 2.0 Hugo van Kemenade).

Run coverage on tests

Fri, 27 Jun 2025 17:41:00 +0000

I recommend running coverage on your tests.

Here’s a couple of reasons why, from the past couple of months.

Example one #

When writing tests, it’s common to copy and paste test functions, but sometimes you forget to rename the new one (see also: the Last Line Effect).

For example:

def test_get_install_to_run_with_platform(patched_installs):
 i = installs.get_install_to_run("<none>", None, "1.0-32")
 assert i["id"] == "PythonCore-1.0-32"
 assert i["executable"].match("python.exe")
 i = installs.get_install_to_run("<none>", None, "2.0-arm64")
 assert i["id"] == "PythonCore-2.0-arm64"
 assert i["executable"].match("python.exe")


def test_get_install_to_run_with_platform(patched_installs):
 i = installs.get_install_to_run("<none>", None, "1.0-32", windowed=True)
 assert i["id"] == "PythonCore-1.0-32"
 assert i["executable"].match("pythonw.exe")
 i = installs.get_install_to_run("<none>", None, "2.0-arm64", windowed=True)
 assert i["id"] == "PythonCore-2.0-arm64"
 assert i["executable"].match("pythonw.exe")

The tests pass, but the first one is never run because its name is redefined. This clearly shows up as a non-run test in the coverage report. In this case, we only need to rename one of them, and both are covered and pass.

But sometimes there’s a bug in the test which would cause it to fail, but we just don’t know because it’s not run.

Tip 1: This can also be found by Ruff’s F811 rule.

Tip 2: pytest’s parametrize is a great way to combine similar test functions with different input data.

Example two #

This is more subtle:

im = Image.new("RGB", (1, 1))
for colors in (("#f00",), ("#f00", "#0f0")):
 append_images = (Image.new("RGB", (1, 1), color) for color in colors)
 im_reloaded = roundtrip(im, save_all=True, append_images=append_images)

 assert_image_equal(im, im_reloaded)
 assert isinstance(im_reloaded, MpoImagePlugin.MpoImageFile)
 assert im_reloaded.mpinfo is not None
 assert im_reloaded.mpinfo[45056] == b"0100"

 for im_expected in append_images:
 im_reloaded.seek(im_reloaded.tell() + 1)
 assert_image_similar(im_reloaded, im_expected, 1)

It’s not so obvious when looking at the code, but Codecov highlights a problem:

The append_images generator is being consumed inside roundtrip(), so we have nothing to iterate over in the for loop – hence no coverage. The fix is to use a list instead of a generator.

Header photo: Misplaced manhole cover (CC BY-NC-SA 2.0 Hugo van Kemenade).

PEPs & Co.

Wed, 14 May 2025 15:45:00 +0000

PEPs #

Here’s Barry Warsaw on the origin of PEPs, or Python Enhancement Proposals (edited from PyBay 2017):

I like backronyms. For those who don’t know: a backronym is where you come up with the acronym first and then you come up with the thing that the acronym stands for. And I like funny sounding words, like FLUFL was one of those. When we were working for CNRI, they also ran the IETF conferences. The IETF is the Internet Engineering Task Force, and they’re the ones who come up with the RFCs. If you look at RFC 822, it defines what an email message looks like.

We got to a point, because we were at CNRI we were more intimately involved in the IETF and how they do standards and things, we observed at the time that there were so many interesting ideas coming in being proposed for Python that Guido really just didn’t have time to dive into the details of everything.

So I thought: well, we have this RFC process, let’s try to mirror some of that so that we can capture the essence of an idea in a document that would serve as a point of discussion, and that Guido could let people discuss and then come in and read the summary of the discussion.

And I was just kind of thinking: well, PEPs, that’s kind of peppy, it’s kind of a funny sounding word. I came up with the word and then I backronymed it into Python Enhancement Proposal. And then I wrote PEP 0 and PEP 1. PEP 0 was originally handwritten, and so I was the first PEP author because I came up with the name PEP.

But the really interesting thing is that you see the E.P. part used in a lot of other places, like Debian has DEPs now. There’s a lot of other communities that have these enhancement proposals so it’s kind of interesting. And then the format of the PEP was directly from that idea of the RFC’s standard.

& Co. #

Here’s a collection of enhancement proposals from different communities.

Acronym	Name
AIP	Airflow Improvement Proposals
AIP	API Improvement Proposals
APE	Astropy Proposals for Enhancement
BIP	Bitcoin Improvement Proposals
CEP	Conda Enhancement Proposals
CFEP	conda-forge’s Enhancement Proposals
DEP	Debian Enhancement Proposals
DEP	Django Enhancement Proposals
FEP	Fediverse Enhancement Proposals
GLEP	Gentoo Linux Enhancement Proposal
IPEP	IPython Enhancement Proposals
JEP	JDK Enhancement Proposals
JEP	JMESPath Enhancement Proposals
JEP	Jupyter Enhancement Proposals
KEP	Kubernetes Enhancement Proposals
NEP	NumPy Enhancement Proposals
PEEP	Pipenv Enhancement Proposals
PEP	Python Enhancement Proposals
SKIP	scikit-image proposals
SLEP	Scikit-learn enhancement proposals
SPEC	Scientific Python Ecosystem Coordination
TIP	Tcl Improvement Proposals
WEP	Write the Docs Enhancement Proposals
XEP	XMPP Extension Protocols
YTEP	yt Enhancement Proposals
ZAP	Zensical Advancement Proposals

Are there more? Let me know!

Header photo: Grand Grocery Co., Lincoln, Nebraska, USA (1942) by The Library of Congress, with no known copyright restrictions.

Free-threaded Python on GitHub Actions

Tue, 25 Mar 2025 10:25:00 +0000

GitHub Actions now supports experimental free-threaded CPython!

There are three ways to add it to your test matrix:

actions/setup-python: t suffix
actions/setup-uv: t suffix
actions/setup-python: freethreaded variable

actions/setup-python: `t` suffix #

Using actions/setup-python, you can add the t suffix for Python versions 3.13 and higher: 3.13t and 3.14t.

This is my preferred method, we can clearly see which versions are free-threaded and it’s straightforward to test both regular and free-threaded builds.

on: [push, pull_request, workflow_dispatch]

jobs:
 build:
 runs-on: ${{ matrix.os }}
 strategy:
 fail-fast: false
 matrix:
 python-version: [
 "3.13",
 "3.13t", # add this!
 "3.14",
 "3.14t", # add this!
 ]
 os: ["windows-latest", "macos-latest", "ubuntu-latest"]

 steps:
 - uses: actions/checkout@v4

 - name: Set up Python ${{ matrix.python-version }}
 uses: actions/setup-python@v5
 with:
 python-version: ${{ matrix.python-version }}
 allow-prereleases: true # needed for 3.14

 - run: |
 python --version --version
 python -c "import sys; print('sys._is_gil_enabled:', sys._is_gil_enabled())"
 python -c "import sysconfig; print('Py_GIL_DISABLED:', sysconfig.get_config_var('Py_GIL_DISABLED'))"

Regular builds will output something like:

Python 3.14.0a6 (main, Mar 17 2025, 02:44:29) [GCC 13.3.0]
sys._is_gil_enabled: True
Py_GIL_DISABLED: 0

And free-threaded builds will output something like:

Python 3.14.0a6 experimental free-threading build (main, Mar 17 2025, 02:44:30) [GCC 13.3.0]
sys._is_gil_enabled: False
Py_GIL_DISABLED: 1

For example: hugovk/test/actions/runs/14057185035

actions/setup-uv: `t` suffix #

Similarly, you can install uv with astral/setup-uv and use that to set up free-threaded Python using the t suffix.

on: [push, pull_request, workflow_dispatch]

jobs:
 build:
 runs-on: ${{ matrix.os }}
 strategy:
 fail-fast: false
 matrix:
 python-version: [
 "3.13",
 "3.13t", # add this!
 "3.14",
 "3.14t", # add this!
 ]
 os: ["windows-latest", "macos-latest", "ubuntu-latest"]

 steps:
 - uses: actions/checkout@v4

 - name: Set up Python ${{ matrix.python-version }}
 uses: astral-sh/setup-uv@v5 # change this!
 with:
 python-version: ${{ matrix.python-version }}
 enable-cache: false # only needed for this example with no dependencies

 - run: |
 python --version --version
 python -c "import sys; print('sys._is_gil_enabled:', sys._is_gil_enabled())"
 python -c "import sysconfig; print('Py_GIL_DISABLED:', sysconfig.get_config_var('Py_GIL_DISABLED'))"

For example: hugovk/test/actions/runs/13967959519

actions/setup-python: `freethreaded` variable #

Back to actions/setup-python, you can also set the freethreaded variable for 3.13 and higher.

on: [push, pull_request, workflow_dispatch]

jobs:
 build:
 runs-on: ${{ matrix.os }}
 strategy:
 fail-fast: false
 matrix:
 python-version: ["3.13", "3.14"]
 os: ["windows-latest", "macos-latest", "ubuntu-latest"]

 steps:
 - uses: actions/checkout@v4

 - name: Set up Python ${{ matrix.python-version }}
 uses: actions/setup-python@v5
 with:
 python-version: ${{ matrix.python-version }}
 allow-prereleases: true # needed for 3.14
 freethreaded: true # add this!

 - run: |
 python --version --version
 python -c "import sys; print('sys._is_gil_enabled:', sys._is_gil_enabled())"
 python -c "import sysconfig; print('Py_GIL_DISABLED:', sysconfig.get_config_var('Py_GIL_DISABLED'))"

For example: hugovk/test/actions/runs/39359291708

`PYTHON_GIL=0` #

And you may want to set PYTHON_GIL=0 to force Python to keep the GIL disabled, even after importing a module that doesn’t support running without it.

See Running Python with the GIL Disabled for more info.

With the t suffix:

- name: Set PYTHON_GIL
 if: endsWith(matrix.python-version, 't')
 run: |
 echo "PYTHON_GIL=0" >> "$GITHUB_ENV"

With the freethreaded variable:

- name: Set PYTHON_GIL
 if: "${{ matrix.freethreaded }}"
 run: |
 echo "PYTHON_GIL=0" >> "$GITHUB_ENV"

Please test! #

For free-threaded Python to succeed and become the default, it’s essential there is ecosystem and community support. Library maintainers: please test it and where needed, adapt your code, and publish free-threaded wheels so others can test their code that depends on yours. Everyone else: please test your code too!

Improving licence metadata

Fri, 14 Feb 2025 15:11:00 +0000

What? #

PEP 639 defines a spec on how to document licences used in Python projects.

Instead of using a Trove classifier such as “License :: OSI Approved :: BSD License”, which is imprecise (for example, which BSD licence?), the SPDX licence expression syntax is used.

How? #

`pyproject.toml` #

Change pyproject.toml as follows.

I usually use Hatchling as a build backend, and support was added in 1.27:

 [build-system]
 build-backend = "hatchling.build"
 requires = [
 "hatch-vcs",
- "hatchling",
+ "hatchling>=1.27",
 ]

Replace the freeform license field with a valid SPDX license expression, and add license-files which points to the licence files in the repo. There’s often only one, but if you have more than one, list them all:

 [project]
 ...
-license = { text = "MIT" }
+license = "MIT"
+license-files = [ "LICENSE" ]

Optionally delete the deprecated licence classifier:

 classifiers = [
 "Development Status :: 5 - Production/Stable",
 "Intended Audience :: Developers",
- "License :: OSI Approved :: MIT License",
 "Operating System :: OS Independent",

For example, see humanize#236 and prettytable#350.

Upload #

Then make sure to use a PyPI uploader that supports this.

I recommend using Trusted Publishing which I use with pypa/gh-action-pypi-publish to deploy from GitHub Actions. I didn’t need to make any changes here, just make a release as usual.

Result #

PyPI #

PyPI shows the new metadata:

pip #

pip can also show you the metadata:

$ pip install prettytable==3.13.0
$ pip show prettytable
Name: prettytable
Version: 3.13.0
...
License-Expression: BSD-3-Clause
Location: /Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages
Requires: wcwidth
Required-by: norwegianblue, pypistats

Thank you! #

A lot of work went into this. Thank you to PEP authors Philippe Ombredanne for creating the first draft in 2019, to C.A.M. Gerlach for the second draft in 2021, and especially to Karolina Surma for getting the third draft over the finish line and helping with the implementation.

And many projects were updated to support this, thanks to the maintainers and contributors of at least:

Header photo: Amelia Earhart’s 1932 pilot licence in the San Diego Air and Space Museum Archive, with no known copyright restrictions.

I'm excited to join the Sovereign Tech Fellowship

Thu, 06 Feb 2025 11:40:00 +0000

For the duration of 2025, I’m thrilled to join the Sovereign Tech Fellowship for Maintainers!

This is a pilot programme from the Sovereign Tech Agency to pay maintainers of critical open source technologies in the public interest.

I’m one of six participants in this first cohort and I’ll be working on Python, with a focus on CPython, including as release manager for Python 3.14 (under development with full release due in October 2025, supported until 2030) and the next one (development starts in May 2025 for release in October 2026, supported until 2031).

I’m delighted to work with the Sovereign Tech Agency as they really get open source. Supported by Germany’s Ministry for Economic Affairs and Climate Action, they recognise the importance of open source and its place as critical digital infrastructure in the public interest. In their own words:

The Sovereign Tech Agency supports the development, improvement, and maintenance of open digital infrastructure. Our goal is to sustainably strengthen the open source ecosystem. We focus on security, resilience, technological diversity, and the people behind the code.

We often don’t notice how much our lives depend on digital infrastructure until it stops working. But making it available, accessible and secure is key for digitalization in the public interest. The Sovereign Tech Agency invests in open digital infrastructure. It is critical for innovation and economic growth, and forms the foundation for digitalization across sectors.

In addition to CPython core development and release management, I’ll continue to maintain other Python projects such as Pillow, humanize and termcolor, and CLI utilities such as norwegianblue, pypistats and pepotron, and help out more widely in the Python ecosystem.

Tasks include:

Bug fixing, issue triaging, PR review and security improvements
Mentoring new contributors to foster community growth and diversity
Communicating updates to the wider community via blog posts and social media
Workflow, CI, and infrastructure optimisations to streamline core development tasks
Maintenance and improvement of PEP editing infrastructure and related documentation tools
Automation and dashboard creation for triage scripts to assist other contributors
Process and workflow improvements for CI and core contributor environments
Ongoing documentation enhancements, including hosting monthly Documentation Working Group meetings, improving build processes, and focusing on accessibility
Updating and modernising projects for compatibility with the latest Python versions
Implementing stronger security measures and using Trusted Publishing and digital attestations
Adding and maintaining Python support for widely used PyPI packages
Improving community-wide tooling, documentation, and accessibility

Meet my fellow Sovereign Tech Fellows, and by the way, the Sovereign Tech Agency is currently hiring, check out their open positions.

How to delay a Python release

Tue, 04 Feb 2025 18:55:00 +0000

Prologue #

This was a Twitter thread from 15th January 2022 about my first CPython bug. Eight days from report to fix to merge, not bad!

Delay #

I helped delay the release of Python 3.11.0a4! But in a good way! 😇

Python 3.11 is due out in October, but they make early alpha, beta and release candidates available for people to help test Python itself and their own code before the big release.

So I tested Pillow…

Tests #

The Pillow test suite passed with 3.11 ✅

Next I tried building the documentation with 3.11 ❌

The docs program, Sphinx, emitted a couple of warnings. Warnings are often missed because they don’t error. But luckily we use the “-W” option to turn warnings into hard errors.

Sphinx #

Maybe Sphinx isn’t ready for Python 3.11?

Rather than submitting a report with the full Pillow documentation (lots of files) I made a new, “minimal” example with just enough stuff to reproduce it.

This makes it easier to investigate what’s up.

Report 1 #

I reported this to Sphinx. The problem was that a page in a subdirectory could load an image from one directory, but not from another, further away directory.

It occurs for the Python 3.11.0a3 alpha, but not 3.7-3.10.

CPython #

A few hours later the Sphinx maintainer Takeshi said it looks like a change to a part of Python itself – os.path.normpath() – since 3.11.0a3, and as it wasn’t mentioned on the “What’s New in Python 3.11” page it could be a bug in Python.

He asked me to report it to Python.

Report 2 #

I reported it to Python with Takeshi’s even more minimal example.

Half an hour later Christian pointed out a change which may have caused this.

I tested and confirmed.

The next day Steve confirmed it was a bug and set it as a “release blocker”.

Fix #

Steve also said it will needs tests, because this bug slipped out due to a gap in testing.

I didn’t know how to fix the bug, but I could write some test cases!

neonene then took the tests and fixed the bug! In doing so they found even more bugs!

Merge #

These extra bugs also existed in earlier versions.

But it turns out path handling can get pretty complicated in places, so Steve decided to only fix my bug now to get it released and the others can be sorted later.

The fix was merged and I confirmed it also worked with Sphinx ✅

Conclusion #

And that’s about it!

It’s now fixed in 3.11.0a4; much better to find these before 3.11.0 final is released to the world in October. Along the way we found more issues to address.

Short version: test your code with 3.11 now, you may find issues in your code or in Python itself 🚀

Epilogue #

Back to 2025: Please test and delay Python 3.14 alpha – but in a good way! 😇

A surprising thing about PyPI's BigQuery data

Sun, 24 Nov 2024 20:45:31 +0000

You can get download numbers for PyPI packages (or projects) from a Google BigQuery dataset. You need a Google account and credentials, and Google gives 1 TiB of free quota per month.

Each month, I have automation to fetch the download numbers for the 8,000 most popular packages over the past 30 days, and make it available as more accessible JSON and CSV files at Top PyPI Packages. This data is widely used for research in academia and industry.

However, as more packages and releases are uploaded to PyPI, and there are more and more downloads logged, the amount of billed data increases too.

This chart shows the amount of data billed per month.

At first, I was only collecting downloads data for 4,000 packages, and it was fetched for two queries: downloads over 365 days and over 30 days. But as time passed, it started using up too much quota to download data for 365 days.

So I ditched the 365-day data, and increased the 30-day data from 4,000 to 5,000 packages. Later, I checked how much quota was being used and increased from 5,000 packages to 8,000 packages.

But then I exceeded the BigQuery monthly quota of 1 TiB fetching data for July 2024.

To fetch the missing data and investigate what’s going in, I started Google Cloud’s 90-day, $300 (€277.46) free-trial 💸

Here’s what I found!

Finding: it costs more to get data for downloads from only pip than from all installers #

I use the pypinfo client to help query BigQuery. By default, it only fetches downloads for pip.

Only pip #

This command gets one day’s download data for the top 10 packages, for pip only:

$ pypinfo --limit 10 --days 1 "" project
Served from cache: False
Data processed: 58.21 GiB
Data billed: 58.21 GiB
Estimated cost: $0.29

Results:

project	download count
boto3	37,251,744
aiobotocore	16,252,824
urllib3	16,243,278
botocore	15,687,125
requests	13,271,314
s3fs	12,865,055
s3transfer	12,014,278
fsspec	11,982,305
charset-normalizer	11,684,740
certifi	11,639,584
Total	158,892,247

All installers #

Adding the --all flag gets one day’s download data for the top 10 packages, for all installers:

$ pypinfo --all --limit 10 --days 1 "" project
Served from cache: False
Data processed: 46.63 GiB
Data billed: 46.63 GiB
Estimated cost: $0.23

project	download count
boto3	39,495,624
botocore	17,281,187
urllib3	17,225,121
aiobotocore	16,430,826
requests	14,287,965
s3fs	12,958,516
charset-normalizer	12,781,405
certifi	12,647,098
setuptools	12,608,120
idna	12,510,335
Total	168,226,197

So we can see the default pip-only costs an extra 25% data processed and data billed, and costs an extra 25% in dollars.

Unsurprisingly, the actual download counts are higher for all installers. The ranking has changed a bit, but I expect we’re still getting more-or-less the same packages in the top thousands of results.

Queries #

It sends a query like this to BigQuery for only pip:

SELECT
 file.project as project,
 COUNT(*) as download_count,
FROM `bigquery-public-data.pypi.file_downloads`
WHERE timestamp BETWEEN TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -2 DAY) AND TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -1 DAY)
 AND details.installer.name = "pip"
GROUP BY
 project
ORDER BY
 download_count DESC
LIMIT 10

And for all installers:

SELECT
 file.project as project,
 COUNT(*) as download_count,
FROM `bigquery-public-data.pypi.file_downloads`
WHERE timestamp BETWEEN TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -2 DAY) AND TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -1 DAY)
GROUP BY
 project
ORDER BY
 download_count DESC
LIMIT 10

These queries are the same, except the default has an extra AND details.installer.name = "pip" condition. It seems reasonable it would cost more to do extra filtering work.

Installers #

Let’s look at the installers:

$ pypinfo --all --limit 100 --days 1 "" installer
Served from cache: False
Data processed: 29.49 GiB
Data billed: 29.49 GiB
Estimated cost: $0.15

installer name	download count
pip	1,121,198,711
uv	117,194,833
requests	29,828,272
poetry	23,009,454
None	8,916,745
bandersnatch	6,171,555
setuptools	1,362,797
Bazel	1,280,271
Browser	1,096,328
Nexus	593,230
Homebrew	510,247
Artifactory	69,063
pdm	62,904
OS	13,108
devpi	9,530
conda	2,272
pex	194
Total	1,311,319,514

pip still by far the most popular, and unsurprising uv is up there too, with about 10% of pip’s downloads.

The others are about 25% or less of uv. A lot of them are mirroring services that we wanted to exclude before.

I think given uv’s importance, and my expectation that it will continue to take a bigger share of the pie, plus especially the extra cost for filtering by just pip, means that we should switch to fetching data for all downloaders. Plus the others don’t account for that much of the pie.

Finding: the number of packages doesn’t affect the cost #

This was the biggest surprise. Earlier I’d been increasing or decreasing the number to try and remain under quota. But it turns out it makes no difference how many packages you query!

I fetched data for just one day and all installers for different package limits: 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000. Sample query:

SELECT
 file.project as project,
 COUNT(*) as download_count,
FROM `bigquery-public-data.pypi.file_downloads`
WHERE timestamp BETWEEN TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -2 DAY) AND TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -1 DAY)
GROUP BY
 project
ORDER BY
 download_count DESC
LIMIT 8000

Result: Interestingly, the cost is the same for all limits (1000-8000): $0.31.

Repeating with one day but filtering for pip only:

Result: Cost increased to $0.39 but again the same for all limits.

Let’s repeat with all installers, but for 30 days, and this time query in decreasing limits, in case we were only paying for incremental changes: 8000, 7000, 6000, 5000, 4000, 3000, 2000, 1000:

Result: Again, the cost is the same regardless of package limit: $4.89 per query.

Well then, let’s repeat with the limit increasing by powers of ten, up to 1,000,000! This last one fetches data for all 531,022 packages on PyPI:

limit	projects count	estimated cost	bytes billed	bytes processed
1	1	0.20	43,447,746,560	43,447,720,943
10	10	0.20	43,447,746,560	43,447,720,943
100	100	0.20	43,447,746,560	43,447,720,943
1000	1,000	0.20	43,447,746,560	43,447,720,943
8000	8,000	0.20	43,447,746,560	43,447,720,943
10000	10,000	0.20	43,447,746,560	43,447,720,943
100000	100,000	0.20	43,447,746,560	43,447,720,943
1000000	531,022	0.20	43,447,746,560	43,447,720,943

Result: Again, same cost, whether for 1 package or 531,022 packages!

Finding: the number of days affects the cost #

No surprise. I’d earlier noticed 365 days too took much quota, and I could continue with 30 days.

Here’s the estimated cost and bytes billed (for one package, all installers) between one and 30 days (f"pypinfo --all --json --indent 0 --days {days} --limit 1 '' project"), showing a roughly linear increase:

Conclusion #

It doesn’t matter how many packages I fetch data for, I might as well fetch all and make it available to everyone, depending on the size of the data file. It will make sense to still offer a smaller file with 8,000 or so packages: often you just need a large-ish yet manageable number.
It costs more to filter for only downloads from pip, so I’ve switched to fetching data for all installers.
The number of days affects the cost, so I will need to decrease this in the future to stay within quota. For example, at some point I may need to switch from 30 to 25 days, and later from 25 to 20 days.

More details from the investigation, the scripts and data files can be found at hugovk/top-pypi-packages#36.

And let me know if you know any tricks to reduce costs!

Header photo: “The Balancing Rock, Stonehenge, Near Glen Innes, NSW” by the Royal Australian Historical Society, with no known copyright restrictions.

Speed up CI with uv ⚡

Sat, 02 Nov 2024 13:11:22 +0000

We can use uv to make linting and testing on GitHub Actions around 1.5 times as fast.

Linting #

When using pre-commit for linting:

name: Lint

on: [push, pull_request, workflow_dispatch]

env:
 FORCE_COLOR: 1

permissions:
 contents: read

jobs:
 lint:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 with:
 persist-credentials: false
 - uses: actions/setup-python@v5
 with:
 python-version: "3.x"
 cache: pip
 - uses: pre-commit/action@v3.0.1

We can replace pre-commit/action with tox-dev/action-pre-commit-uv:

 - uses: actions/setup-python@v5
 with:
 python-version: "3.x"
- cache: pip
- - uses: pre-commit/action@v3.0.1
+ - uses: tox-dev/action-pre-commit-uv@v1

name: Lint

on: [push, pull_request, workflow_dispatch]

env:
 FORCE_COLOR: 1

permissions:
 contents: read

jobs:
 lint:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 with:
 persist-credentials: false
 - uses: actions/setup-python@v5
 with:
 python-version: "3.x"
 - uses: tox-dev/action-pre-commit-uv@v1

This means uv will create virtual environments and install packages for pre-commit, which is faster for the initial seed operation when there’s no cache.

Lint comparison #

For example: python/blurb#32

	Before	After	Times faster
No cache	60s	37s	1.62
With cache	11s	11s	1.00

Testing #

When testing with tox:

name: Test

on: [push, pull_request, workflow_dispatch]

permissions:
 contents: read

env:
 FORCE_COLOR: 1

jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 fail-fast: false
 matrix:
 python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]

 steps:
 - uses: actions/checkout@v4
 with:
 persist-credentials: false

 - name: Set up Python ${{ matrix.python-version }}
 uses: actions/setup-python@v5
 with:
 python-version: ${{ matrix.python-version }}
 allow-prereleases: true
 cache: pip

 - name: Install dependencies
 run: |
 python --version
 python -m pip install -U pip
 python -m pip install -U tox

 - name: Tox tests
 run: |
 tox -e py

We can replace tox with tox-uv:

 - name: Set up Python ${{ matrix.python-version }}
 uses: actions/setup-python@v5
 with:
 python-version: ${{ matrix.python-version }}
 allow-prereleases: true
- cache: pip

- - name: Install dependencies
- run: |
- python --version
- python -m pip install -U pip
- python -m pip install -U tox
+ - name: Install uv
+ uses: hynek/setup-cached-uv@v2

 - name: Tox tests
 run: |
- tox -e py
+ uvx --with tox-uv tox -e py

name: Test

on: [push, pull_request, workflow_dispatch]

permissions:
 contents: read

env:
 FORCE_COLOR: 1

jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 fail-fast: false
 matrix:
 python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]

 steps:
 - uses: actions/checkout@v4
 with:
 persist-credentials: false

 - name: Set up Python ${{ matrix.python-version }}
 uses: actions/setup-python@v5
 with:
 python-version: ${{ matrix.python-version }}
 allow-prereleases: true

 - name: Install uv
 uses: hynek/setup-cached-uv@v2

 - name: Tox tests
 run: |
 uvx --with tox-uv tox -e py

tox-uv is tox plugin to replace virtualenv and pip with uv in your tox environments. We only need to install uv, and use uvx to both install tox-uv and run tox, for faster installs of tox, the virtual environment, and the dependencies within it.

Test comparison #

For example: python/blurb#32

	Before	After	Times faster
No cache	2m 0s	1m 26s	1.40
With cache	1m 58s	1m 22s	1.44

No pip with tox-uv #

One difference with uv environments compared to regular venv/virtualenv ones is that they do not have pip. This means calls to pip need replacing, for example in tox.ini:

 [testenv]
-commands_pre =
- {envpython} -m pip install -U -r requirements.txt
+deps =
+ -r requirements.txt
 pass_env =
 FORCE_COLOR
 commands =
 {envpython} -m pytest {posargs}

If you still need pip (or setuptools or wheel), add uv_seed = True to your [testenv] to inject them.

Bonus tip #

Run the new tool zizmor to find security issues in GitHub Actions.

Header photo: “Road cycling at the 1952 Helsinki Olympics” by Olympia-Kuva Oy & Helsinki City Museum, Public Domain.

Python Core Sprint 2024

Sat, 05 Oct 2024 19:39:48 +0000

🐍🏃The week before last was the annual Python Core Dev Sprint, graciously hosted by Meta in Bellevue, WA!

The idea: bring a bunch of Python core team members, triagers, and special guests to the same room for a week. It’s hugely beneficial and productive, we held many in-depth discussions that just don’t happen when we’re all remote and async, and got to work on many different things together.

The sprint room

During the week, I reviewed 39 PRs, created 15, merged 10, updated 4, and closed 2 issues.

Monday highlights #

As release manager for Python 3.14, I discussed with Brett Cannon one of his project ideas which will come after lock files, and after the next big one.

Also as RM, discussed with Russell Keith-Magee, Ned Deily, Łukasz Langa and Thomas Wouters about including official binaries for iOS and Android, which wandered into ideas about security releases.

I did some maintenance of our PyPI projects, adding PEP 740 attestations, support for the new Python 3.13 and dropping support for the very-nearly-EOL 3.8.

Tuesday highlights #

Started investigating slow doctest on 3.13+ with Alex Waygood, who on Wednesday narrowed it down to a problem with the new incremental garbage collector, which would go on to be reverted by Friday and result in Python 3.13’s Monday release to be postponed and replaced with an extra release candidate. Not ideal, but much better to discover these things before the big release.

We had a Q&A session with the Steering Council: Barry Warsaw, Emily Morehouse, Gregory P. Smith, Pablo Galindo Salgado and Thomas.

The Python Steering Council

Proofread Guido van Rossum’s STAR voting proposal for electing future steering councils.

Discussed with Eric Snow his novel method for displaying many code samples in a table, using <details> disclosures to prevent the table being too wide. Looks like a good solution!

Wednesday highlights #

I applied the finishing touches to PEP 2026 (Calendar versioning for Python) and Barry gave it a final review. Ready for submission!

Seth Larson, the PSF Security Developer-in-Residence, wasn’t at the sprint but we discussed our plan to stop providing GPG signatures for CPython and rely on SigStore instead. Expect a PEP soon!

Also not at the sprint, I recommended PSF Infrastructure Engineer Jacob Coffee as a CPython triager. Welcome aboard!

The whole room discussed including static type annotations in CPython.

We had a Q&A session with two of the three Developers-in-Residence, Łukasz and Petr Viktorin.

Q&A with Łukasz and Petr

Discussed expanding the voter pool for Steering Council elections with Mariatta, Greg and Thomas.

Larry Hastings handed out, in return for oohs and aahs, some nice P.C.D.S. 2024 stickers he generously designed and printed up for us. Thanks!

PCDS 2024 stickers by Larry

Thursday highlights #

On the 26th September, at 10:26 Bellevue time (20:26 Helsinki time), I submitted PEP 2026 to the Steering Council!🤞

Brett discussed whether we should update PEP 387 to prefer 5 year deprecations instead of 2 years.

Brandt Bucher gave us all an update on the progress of the Just-in-Time (JIT) compiler (“we went from 0% slower to 0% faster!”) and we discussed plans for Python 3.14.

Because I couldn’t attend Thursday’s Helsinki Python meetup due to being at another kind of Python meetup on the other side of the world, I gave the famous HelPy quiz to the assembled core devs. Unsurprisingly they did pretty well, but the most incorrect answer was a pleasant surprise: we’ve had ~400 not ~80 new contributors to Python 3.13!

Pablo performed card tricks!

Magic from Pablo

Meta took us out for a delicious dinner at a local fish restaurant. Thank you!

Friday highlights #

Mariatta presented ideas to Jelle Zijlstra, Petr, Russell and me about to use modern tools to create a modern, interactive tutorial.

Also during the week, continued work with Adam Turner on improving the docs.python.org build. Adam wasn’t at the sprint, so tag-teamed PR reviews overnight. After much work straddling many teams, projects and repos, we’ve got the full HTML build loop for 13 languages × 3 versions down from over 40 hours to just under 9 hours, with more improvements coming.

Made a demo of the CPython docs using the PyData Sphinx Theme.

Along with around 25 others, I was on Łukasz and Pablo’s core.py podcast.

Łukasz and Pablo in their ad-hoc podcast studio in a Meta meeting room

Itamar gave us cake for the podcast’s first birthday!

cake.py. Photo by Itamar Oren.

Thank you #

It was a hugely productive week, big thanks to Itamar Oren and Meta for organising and hosting!

See also Mariatta’s excellent blog posts, and I recommend the core.py podcast with short interviews with some 25 attendees! Łukasz and Pablo were also guests on the Changelog podcast during the sprint.

Header photo by Itamar Oren

PyCon US 2024: A roundup of writeups

Thu, 13 Jun 2024 14:41:55 +0000

If you read just one, check Kati’s thorough recap!

22nd May 2024

Echos of the People API user guide by Ned Batchelder

24th May 2024

Wagtailers spread their wings at PyCon 2024 by Meagen Voss (@vossisboss)
Flet at PyCon US 2024 by Feodor Fitsner

27th May 2024

PyCon US 2024: My First PyCon in US 🫶🏻 by Tomy Hsieh

28th May 2024

PyCon 2024 Reflection by Trey Hunner
Weeknotes: PyCon US 2024 by Simon Willison
3 key takeaways from PyCon US 2024 by Luis Juncal & Yolanda Robla

30th May 2024

pyOpenSci at PyCon US 2024 - Python packaging and community by Leah Wasser
Our Experience at PyCon US 2024 in Pittsburgh by Ngazetungue Muheue (@ngazetungue)

1st June 2024

PyCon US 2024 Recap by Kati Michel (@katherinemichel)

4th June 2024

PyCon US 2024 by Yilei Yang

13th June 2024

PyCon US 2024 as Security Developer-in-Residence by Seth Michael Larson (@sethmlarson)

14th June 2024

The Python Language Summit 2024 by Seth Michael Larson (@sethmlarson)

16th June 2024

PyCon US 2024 Highlights by David Foster

19th June 2024

From Pittsburgh to New York: A PyCon US 2024 Adventure by Monica Oyugi (@monicaoyugi)

Header photo: Downtown Pittsburgh seen between the Andy Warhol Bridge and Roberto Clemente Bridge

Help test Python 3.13!

Tue, 04 Jun 2024 13:04:03 +0000

Calling all Python library maintainers! 🐍

The Python 3.13 beta is out! 🎉

PEP 719 defines the release schedule for Python 3.13.0:

The first beta candidate came out on 8th May 2024
The first release candidate is set for 30th July 2024
And the full release is set for 1st October 2024

In his announcement, Thomas Wouters, release manager for Python 3.12 and 3.13, said:

We strongly encourage maintainers of third-party Python projects to test with 3.13 during the beta phase and report issues found to the Python bug tracker as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Tuesday 2024-07-30). Our goal is to have no ABI changes after beta 4 and as few code changes as possible after 3.13.0rc1, the first release candidate. To achieve that, it will be extremely important to get as much exposure for 3.13 as possible during the beta phase.

Test with 3.13 #

It’s now time for us library maintainers to start testing our projects with 3.13. There’s two big benefits:

There have been removals and changes in Python 3.13. Testing now helps us make our code compatible and avoid any big surprises (for us and our users) at the big launch in October.
We might find bugs in Python itself! Reporting those will help get them fixed and help everyone.

How #

GitHub Actions: setup-python #

To test the latest alpha, beta or release candidate with actions/setup-python, add 3.13 and allow-prereleases: true to your workflow matrix.

For example:

jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 fail-fast: false
 matrix:
 python-version: ["3.10", "3.11", "3.12", "3.13"]

 steps:
 - uses: actions/checkout@v4

 - name: Set up Python ${{ matrix.python-version }}
 uses: actions/setup-python@v5
 with:
 python-version: ${{ matrix.python-version }}
 allow-prereleases: true

(We can instead use 3.13-dev and omit allow-prereleases: true, but I find the above a bit neater, and when 3.13.0 final is released in October, it will continue testing with full release versions.)

GitHub Actions: deadsnakes #

For the bleeding edge, we can use deadsnakes/action to test the latest nightly build:

jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 fail-fast: false
 matrix:
 python-version: ["3.10", "3.11", "3.12", "3.13-dev"]

 steps:
 - uses: actions/checkout@v4

 - name: Set up Python ${{ matrix.python-version }}
 if: "!endsWith(matrix.python-version, '-dev')"
 uses: actions/setup-python@v5
 with:
 python-version: ${{ matrix.python-version }}

 - uses: deadsnakes/action@v3.1.0
 name: Set up Python ${{ matrix.python-version }} (deadsnakes)
 if: endsWith(matrix.python-version, '-dev')
 with:
 python-version: ${{ matrix.python-version }}

When to support 3.13? #

When should you declare support and add the Programming Language :: Python :: 3.13 Trove classifier? Some projects already have!

If you have a pure Python project, you can release now.

If you have C extensions and other projects depend on yours, a preview release with wheels will help them test and prepare. I’ve already started releasing these.

ABI breaks? #

ABI breaks during the beta are infrequent and unintentional. If they happen, you can rebuild your wheels and upload them to an existing PyPI release by adding an optional build tag to the filename:

The wheel filename is {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl.

build tag: […] Acts as a tie-breaker if two wheel file names are the same in all other respects (i.e. name, version, and other tags).

For example, this updates the filename and metadata with build number 1, and removes the original file:

python -m pip install "wheel>=0.4.0"
wheel tags --build=1 --remove sampleproject-5.0.0-cp313-cp313-macosx_10_10_x86_64.whl
# this creates a file named sampleproject-5.0.0-1-cp313-cp313-macosx_10_10_x86_64.whl

Upload it, and the new file will be used instead of the old one. See also Brett Cannon’s advice on making new wheels.

In any case, let’s start testing 3.13 now! 🚀

Printable PyCon 2024 schedule

Sat, 11 May 2024 11:01:33 +0000

Want to print out the PyCon US schedule? Paper doesn’t run out of batteries, is easy to scribble on, and stuff into a pocket (technical term: the affordances of paper).

Here’s some custom CSS and JavaScript to make it nicely printable.

Install the Styler browser extension
View a PyCon schedule page such as https://us.pycon.org/2024/schedule/talks/ and click the Styler extension’s S icon
Paste this CSS into the upper box:

body.pycon-schedule div.internal-page-header,
body.pycon-schedule div.panel-heading,
body.pycon-schedule footer,
body.pycon-schedule div.badges {
 display: none;
}

body.pycon-schedule .container {
 margin: 0;
 max-width: fit-content;
}
body.pycon-schedule main.content {
 margin: 0;
 width: 100% !important;
}
body.pycon-schedule .calendar {
 left: auto !important;
 width: 100% !important;
}
body.pycon-schedule .slot {
 padding: 5px;
}
@media print {
 a:after { content:''; }
 a[href]:after { content: none !important; }
}

Paste this JavaScript into the lower box:

$(document).ready(function () {
 if (
 window.location.pathname.match("2024/schedule/") &&
 !window.location.pathname.match("2024/schedule/presentation/")
 ) {
 $("body").addClass("pycon-schedule");
 }
});

Print!

It’ll run on any of the https://us.pycon.org/2024/schedule/* pages, but not the individual presentation pages such as https://us.pycon.org/2024/schedule/presentation/61/

It’s not perfect, the right edge is slightly cut off, but it’s more printable than the original.

Once printed, you can click the Styler icon and the x button to disable Styler for the site so you can browse the original version.

Help us test free-threaded Python without the GIL

Fri, 10 May 2024 12:18:27 +0000

Python 3.13 is due out in October 2024 and work is underway to implement experimental support for PEP 703 - Making the Global Interpreter Lock Optional in CPython.

See also Free-threaded CPython in “What’s New in Python 3.13?”

As the Steering Council noted in their acceptance of the PEP, to succeed it’s important to have community support.

Projects will need to test their code with free-threaded (aka “nogil” but don’t call it that!) Python builds to help us find bugs in CPython, and to check their code is compatible.

Here’s some ways to test.

Official installers
Build it yourself
deadsnakes
GitHub Actions?
Fedora

Official installers #

The official macOS (starting in beta 2) and Windows installers has an option to install the free-threading binaries, which also installs python3.13t alongside the regular python3.13:

Build it yourself #

Check out the CPython Git repo and build yourself using the --disable-gil configuration flag.

For example, on macOS I run:

$ GDBM_CFLAGS="-I$(brew --prefix gdbm)/include" \
 GDBM_LIBS="-L$(brew --prefix gdbm)/lib -lgdbm" \
 ./configure --config-cache \
 --with-system-libmpdec \
 --with-openssl="$(brew --prefix openssl@3.0)" \
 --disable-gil && make -s -j10
$ ./python.exe -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"
1
$ ./python.exe -c "import sys; print(sys._is_gil_enabled())"
False

More details in the devguide:

https://devguide.python.org/getting-started/setup-building/

GitHub Actions: deadsnakes/action #

We can use the deadsnakes/action to test Ubuntu.

For example:

on: [push, pull_request, workflow_dispatch]

jobs:
 build:
 runs-on: ubuntu-latest
 strategy:
 matrix:
 python-version: [3.12, 3.13-dev]
 name: main
 steps:
 - uses: actions/checkout@v4
 - uses: actions/setup-python@v5
 if: "!endsWith(matrix.python-version, '-dev')"
 with:
 python-version: ${{ matrix.python-version }}
 - uses: deadsnakes/action@v3.1.0
 if: endsWith(matrix.python-version, '-dev')
 with:
 python-version: ${{ matrix.python-version }}
 nogil: true
 - run: |
 python --version --version
 which python
 python -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"
 python -c "import sys; print(sys._is_gil_enabled())"

GitHub Actions: actions/setup-python #

I’ve asked for support at actions/setup-python#771, they’re looking into it 🤞

In the meantime, give it an upvote, and use deadsnakes/action ⤴️

deadsnakes: PPA #

The deadsnakes project provides Personal Package Archives of Python packaged for Ubuntu, included free-threaded builds.

For example, python3.13-nogil under python3.13 - 3.13.0~a2-1+focal1 and python3.13 - 3.13.0~a2-1+jammy1 at https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa/+packages

Fedora #

Fedora uses the python3.13t executable name as decided by the Steering Council:

$ # Install
$ sudo dnf install python3.13-freethreading
$ # Run
$ /usr/bin/python3.13t -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"
1
$ /usr/bin/python3.13t -c "import sys; print(sys._is_gil_enabled())"
False

How to check your build #

To confirm if you’re using a free-threaded build, check the double --version option (starting in beta 2):

$ # Regular build
$ python3.13 --version --version
Python 3.13.0b2 (v3.13.0b2:3a83b172af, Jun 5 2024, 12:50:24) [Clang 15.0.0 (clang-1500.3.9.4)]
$ # Free-threaded build
$ python3.13t --version --version
Python 3.13.0b2 experimental free-threading build (v3.13.0b2:3a83b172af, Jun 5 2024, 12:57:31) [Clang 15.0.0 (clang-1500.3.9.4)]

Or the Py_GIL_DISABLED macro:

$ # Regular build
$ python3.13 -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"
0
$ # Free-threaded build
$ python3.13t -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"
1

By default, the GIL is disabled for free-threaded builds, and can be re-enabled by setting the PYTHON_GIL environment variable to 1 or running Python with -X gil 1. You can check with sys._is_gil_enabled():

$ # Regular build: GIL is always enabled
$ python3.13 -c "import sys; print(sys._is_gil_enabled())"
True
$ # Free-threaded build: GIL is disabled by default
$ python3.13t -c "import sys; print(sys._is_gil_enabled())"
False
$ # Free-threaded build: re-enable with -X
$ python3.13t -X gil=1 -c "import sys; print(sys._is_gil_enabled())"
True
$ # Free-threaded build: re-enable with env var
$ PYTHON_GIL=1 python3.13t -c "import sys; print(sys._is_gil_enabled())"
True

References:

Sphinx docs: How to activate tabs for your OS

Tue, 02 Apr 2024 18:41:30 +0000

On the Python Developer’s Guide and Pillow documentation we have some pages with tabs for different operating systems:

It’s possible to add some JavaScript so that the matching tab is activated based on the visitor’s operating system.

Here’s how!

Sphinx Inline Tabs #

First add the Sphinx Inline Tabs extension to your docs' requirements.txt:

# requirements.txt
sphinx-inline-tabs>=2023.4.21

JavaScript #

Next, add activate_tab.js to your _static/ directory:

// activate_tab.js
// Based on https://stackoverflow.com/a/38241481/724176
function getOS() {
 const userAgent = window.navigator.userAgent,
 platform = window.navigator?.userAgentData?.platform || window.navigator.platform,
 macosPlatforms = ["macOS", "Macintosh", "MacIntel", "MacPPC", "Mac68K"],
 windowsPlatforms = ["Win32", "Win64", "Windows", "WinCE"],
 iosPlatforms = ["iPhone", "iPad", "iPod"];

 if (macosPlatforms.includes(platform)) {
 return "macOS";
 } else if (iosPlatforms.includes(platform)) {
 return "iOS";
 } else if (windowsPlatforms.includes(platform)) {
 return "Windows";
 } else if (/Android/.test(userAgent)) {
 return "Android";
 } else if (/Linux/.test(platform)) {
 return "Unix";
 }

 return "unknown";
}

function activateTab(tabName) {
 // Find all label elements containing the specified tab name
 const labels = document.querySelectorAll(".tab-label");

 labels.forEach((label) => {
 if (label.textContent.includes(tabName)) {
 // Find the associated input element using the 'for' attribute
 const tabInputId = label.getAttribute("for");
 const tabInput = document.getElementById(tabInputId);

 // Check if the input element exists before attempting to set the "checked" attribute
 if (tabInput) {
 // Activate the tab by setting its "checked" attribute to true
 tabInput.checked = true;
 }
 }
 });
}

`conf.py` #

Add the extension and JavaScript to your conf.py:

# conf.py
extensions = [
 "sphinx_inline_tabs",
]

html_js_files = [
 "activate_tab.js",
]

reStructuredText #

Almost there!

Add tabs to the reStructuredText files.

For example, here we have three different commands; one for Unix, one for macOS, and one for Windows:

.. tab:: Unix

 .. code-block:: shell

 ./python -m test -h

.. tab:: macOS

 .. code-block:: shell

 ./python.exe -m test -h

.. tab:: Windows

 .. code-block:: dosbatch

 .\python.bat -m test -h

Finally, add the JavaScript to the same reST page to activate the correct tab:

.. raw:: html

 <script>
 document.addEventListener('DOMContentLoaded', function() {
 activateTab(getOS());
 });
 </script>

You can see the results here. When the page loads, the browser finds the browser name, and activates the tabs with a matching name.

If you have many sets of tabs on a page, the corresponding OS tab will be activated for all. And if you click on another OS tab, all the others with the same name are activated.

Happy tabbing!

Header photo: “The Great Pyramid and the Sphinx” by National Galleries of Scotland, with no known copyright restrictions.

TIL: exclude_also with coverage.py

Mon, 06 Nov 2023 20:05:55 +0000

Sometimes you have code you want to exclude from the test coverage report, because it doesn’t really make sense to test it.

For example, maybe you want to exclude:

if __name__ == "__main__":
 main()

The old advice was to add something like this to .coveragerc:

[report]
# Regexes for lines to exclude from consideration
exclude_lines =
 # Have to re-enable the standard pragma:
 pragma: no cover

 # Don't complain if non-runnable code isn't run:
 if __name__ == .__main__.:

But since coverage.py 7.2.0 (2023-02-22) you can use exclude_also instead and skip that pragma:

[report]
# Regexes for lines to exclude from consideration
exclude_also =
 # Don't complain if non-runnable code isn't run:
 if __name__ == .__main__.:

Which is:

 [report]
 # Regexes for lines to exclude from consideration
-exclude_lines =
- # Have to re-enable the standard pragma:
- pragma: no cover
-
+exclude_also =
 # Don't complain if non-runnable code isn't run:
 if __name__ == .__main__.:

Thanks #

To Brian Okken for the tip.

To Ned Batchelder for maintaining Coverage.py.

To the Library of Congress and Flickr Commons for the photo of a covered wagon.

Why are there still so many downloads for EOL Python 3.7?

Tue, 22 Aug 2023 19:25:29 +0000

Python 3.7 was first released on 2018-06-27 and recently reached end-of-life on 2023-06-27 (PEP 537).

This means it is no longer receiving security updates and you should upgrade to a newer version (at least 3.8, but preferably 3.11):

Source: Python Developer's Guide

However, if you look at download numbers from PyPI, 3.7 still accounts for a large share. 3.7 accounted for 25% of all downloads from PyPI in July 2023, compared with 27% for 3.8:

Source: pypi-tools

But why does such an old Python version have so many downloads?

All downloads #

Let’s dig into the numbers using a handy tool called pypinfo, which helps us analyse the PyPI data from Google BigQuery.

This command fetches all of yesterday’s downloads per Python version:

pypinfo --days 1 --percent --markdown "" pyversion

Python version	percent	download count
3.8	25.00%	189,678,872
3.7	23.35%	177,150,010
3.9	20.25%	153,663,903
3.10	15.52%	117,751,108
3.11	6.90%	52,381,884
3.6	6.29%	47,749,879
2.7	2.32%	17,602,650
3.5	0.23%	1,778,388
3.4	0.10%	770,135
3.12	0.03%	224,223
3.13	0.00%	3,920
3.3	0.00%	1,165
2.8	0.00%	57
3.2	0.00%	35
None	0.00%	3
Total		758,756,232

All downloads by OS #

But what happens if we check which distros are responsible for those downloads?

This command gives us the top 20:

pypinfo --days 1 --limit 20 --percent --markdown "" system distro pyversion

system name	distro name	Python version	percent	download count
Linux	Ubuntu	3.8	18.87%	128,991,062
Linux	Amazon Linux	3.7	14.44%	98,738,952
Linux	Ubuntu	3.9	12.14%	83,019,828
Linux	Ubuntu	3.10	9.66%	66,019,309
Linux	Ubuntu	3.7	5.89%	40,257,060
Linux	Debian GNU/Linux	3.8	5.70%	38,958,367
Linux	Debian GNU/Linux	3.9	5.63%	38,482,436
Linux	Debian GNU/Linux	3.7	4.25%	29,035,532
Linux	Debian GNU/Linux	3.10	4.15%	28,348,346
Linux	Debian GNU/Linux	3.6	3.14%	21,441,883
Linux	Ubuntu	3.11	3.01%	20,570,619
Linux	Debian GNU/Linux	3.11	2.72%	18,588,584
Linux	Amazon Linux	3.9	2.43%	16,593,595
Linux	CentOS Linux	3.6	1.70%	11,605,142
Linux	Amazon Linux	3.8	1.47%	10,035,087
Linux	Amazon Linux	3.10	1.46%	9,969,514
Linux	Ubuntu	2.7	1.02%	6,960,697
Linux	Amazon Linux AMI	3.6	0.79%	5,390,823
Linux	Ubuntu	3.6	0.79%	5,388,370
Windows	None	3.10	0.76%	5,227,519
Total				683,622,725

We can see Ubuntu with 3.8 is responsible for the largest share of 17%. (That’s fine, 3.8 is supported until 2024-10-14.)

The next is Amazon Linux with 3.7, responsible for a whopping 15% of all downloads!

The others responsible for 3.7 have a much lower share: Ubuntu (6%) and Debian (4%).

Tip: replace "" in the commands above with a package name to get data for just that package, for example:

pypinfo --days 1 --limit 20 --percent --markdown requests system distro pyversion

Header photo: Space shuttle Discovery landing at Edwards Air Force Base, California, 9th December, 1992 (source: NASA on The Commons)

Help test the Python 3.12 release candidate!

Fri, 26 May 2023 12:30:43 +0000

Calling all Python library maintainers! 🐍

The third and final Python 3.12 release candidate is out! 🎉

PEP 693 defines the release schedule for Python 3.12.0:

The first release candidate came out on 6th August 2023
The second ~~and final~~ release candidate came out on 6th September 2023
The third and final release candidate came out on 19th September 2023
And the full release is set for 2nd October 2023

In his announcement, Thomas Wouters, release manager for Python 3.12 and 3.13, said:

We strongly encourage maintainers of third-party Python projects to prepare their projects for 3.12 compatibilities during this phase, and where necessary publish Python 3.12 wheels on PyPI to be ready for the final release of 3.12.0. Any binary wheels built against Python 3.12.0rc3 will work with future versions of Python 3.12. As always, report any issues to the Python bug tracker.

Test with 3.12 #

It’s now time for us library maintainers to start testing our projects with 3.12. There’s two big benefits:

There have been removals and changes in Python 3.12. Testing now helps us make our code compatible and avoid any big surprises (for us and our users) at the big launch in October.
We might find bugs in Python itself! Reporting those will help get them fixed and help everyone.

How #

GitHub Actions: setup-python #

To test the latest alpha, beta or release candidate with actions/setup-python, add 3.12 and allow-prereleases: true to your workflow matrix.

For example:

jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 fail-fast: false
 matrix:
 python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]

 steps:
 - uses: actions/checkout@v4

 - name: Set up Python ${{ matrix.python-version }}
 uses: actions/setup-python@v4
 with:
 python-version: ${{ matrix.python-version }}
 allow-prereleases: true

(We can instead use 3.12-dev and omit allow-prereleases: true, but I find the above a bit neater, and when 3.12.0 final is released in October, it will continue testing with full release versions.)

GitHub Actions: deadsnakes #

For the bleeding edge, we can use deadsnakes/action to test the latest nightly build:

jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 fail-fast: false
 matrix:
 python-version: ["3.8", "3.9", "3.10", "3.11", "3.12-dev"]

 steps:
 - uses: actions/checkout@v3

 - name: Set up Python ${{ matrix.python-version }}
 if: "!endsWith(matrix.python-version, '-dev')"
 uses: actions/setup-python@v4
 with:
 python-version: ${{ matrix.python-version }}

 - uses: deadsnakes/action@v3.0.0
 name: Set up Python ${{ matrix.python-version }} (deadsnakes)
 if: endsWith(matrix.python-version, '-dev')
 with:
 python-version: ${{ matrix.python-version }}

Travis CI #

I recommend moving to another CI.

In the meantime, you can also add 3.12-dev to .travis.yml, although at the time of writing it’s pointing to 3.12.0a3+ from 2022-12-07, which is better than nothing.

language: python
python:
 - "3.8"
 - "3.9"
 - "3.10"
 - "3.11"
 - "3.12-dev"

Again, I recommend moving to another CI.

Other CIs #

Do you use other CIs? Please leave a comment if you know how to test 3.12!

When to support 3.12? #

Now is also a good time to declare support and add the Programming Language :: Python :: 3.12 Trove classifier.

Especially if you have C extensions and other projects depend on yours, a release with wheels will help them test and prepare.

ABI breaks? #

From the announcement:

There will be no ABI changes from this point forward in the 3.12 series. The intent is for the final release of 3.12.0, scheduled for Monday, 2023-10-02, to be identical to this release candidate. This really is the last chance to find critical problems in Python 3.12.

Let’s start testing and releasing for 3.12 now! 🚀

Header photo: The carpet of the Salt Palace Convention Center grand ballroom, host of PyCon 2023, with a couple of googly eyes added to make them Pythony (source)

2023-08-14: Updated for RC1
2023-09-06: Updated for RC2
2023-09-19: Updated for RC3

Printable PyCon 2023 schedule

Sat, 15 Apr 2023 10:10:02 +0000

Want to print out the PyCon US 2023 schedule? Paper doesn’t run out of batteries, is easy to scribble on, and stuff into a pocket.

Here’s some custom CSS and JavaScript to make it nicely printable.

Install the Styler browser extension
View a PyCon schedule page such as https://us.pycon.org/2023/schedule/talks/ and click the Styler extension’s S icon
Paste this CSS into the upper box:

body.pycon-schedule aside.sidebar,
body.pycon-schedule div.badges,
body.pycon-schedule button.menu-button.open-menu,
body.pycon-schedule button#theme-switch,
body.pycon-schedule footer {
 display: none;
}
body.pycon-schedule {
 font-size: 14px;
}
body.pycon-schedule main {
 padding: 0 !important;
}
body.pycon-schedule .calendar {
	left: auto !important;
 width: 100% !important;
}
@media print {
 a:after { content:''; }
 a[href]:after { content: none !important; }
}

Paste this JavaScript into the lower box:

$(document).ready(function () {
 if (
 window.location.pathname.match("2023/schedule/") &&
 !window.location.pathname.match("2023/schedule/presentation/")
 ) {
 $("body").addClass("pycon-schedule");
 }
});

Print!

It’ll run on any of the https://us.pycon.org/2023/schedule/* pages, but not the individual presentation pages such as https://us.pycon.org/2023/schedule/presentation/88/

Once printed, you can click the Styler icon and the x button to disable Styler for the site so you can browse the original version.

How to search 5,000 Python projects

Fri, 09 Dec 2022 12:44:21 +0000

Why? #

Often Python core developers think about deprecating and removing old bits of the language. But first it’s a good idea to get an idea of how much the old bits are used. Searching the 5,000 most-popular projects on PyPI is a helpful proxy to gauge community use.

How? #

Core developer Victor Stinner has written a couple of useful scripts that live in his misc repo.

Setup #

First, clone the repo somewhere, doesn’t matter where:

mkdir -p ~/github
cd ~/github/
git clone https://github.com/vstinner/misc

# Optionally for some colour in the logs:
python3 -m pip install termcolor

Then download 5,000 sdists! Again, doesn’t matter where:

$ mkdir -p ~/top-pypi/output
$ cd ~/top-pypi/output/
$ python3 ~/github/misc/cpython/download_pypi_top.py .
Download JSON from: https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.min.json
Project#: 5000
[1/5000] Saving to ./boto3-1.26.25.tar.gz (101.7 kB)
[2/5000] Saving to ./botocore-1.29.25.tar.gz (10426.8 kB)
[3/5000] Saving to ./urllib3-1.26.13.tar.gz (293.4 kB)
...
[4990/5000] Saving to ./meld3-2.0.1.tar.gz (35.3 kB)
[4991/5000] Saving to ./browserstack-local-1.2.4.tar.gz (6.6 kB)
Cannot find URL for project: allensdk
Cannot find URL for project: dataframe-image
[4994/5000] Saving to ./SQLAlchemy-Continuum-1.3.13.tar.gz (79.2 kB)
[4995/5000] Saving to ./sarge-0.1.7.post1.tar.gz (25.1 kB)
[4996/5000] Saving to ./confusable_homoglyphs-3.2.0.tar.gz (158.1 kB)
[4997/5000] Saving to ./YTThumb-1.4.5.tar.gz (3.0 kB)
[4998/5000] Saving to ./azure-ai-ml-1.2.0.zip (4486.3 kB)
[4999/5000] Saving to ./pythena-1.6.0.tar.gz (5.2 kB)
[5000/5000] Saving to ./interchange-2021.0.4.tar.gz (25.2 kB)
Downloaded 5000 projects in 1602.4 seconds

With colour:

⏳ This will take a bit of time. Some projects don’t have sdists, nothing to worry about, we will still end up with a good number. At the time of writing (2022-12-09), it took me just under 27 minutes to download 4,748 files, taking up 5.37 GB of space.

If you want to download fewer, specify how many:

python3 ~/github/misc/cpython/download_pypi_top.py . 100

Search #

Next, we can search all the sdists using another script. And we don’t need to extract them!

For example, configparser’s LegacyInterpolation was deprecated in Python 3.2 (released February 2011), but only in docs and without raising a DeprecationWarning.

How much is it used in the top 5k?

$ cd ~/top-pypi/output/
$ python3 ~/github/misc/cpython/search_pypi_top.py -q . "LegacyInterpolation"
./hexbytes-0.3.0.tar.gz: hexbytes-0.3.0/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/configparser.pyi: "LegacyInterpolation",
./hexbytes-0.3.0.tar.gz: hexbytes-0.3.0/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/configparser.pyi: class LegacyInterpolation(Interpolation):
./hexbytes-0.3.0.tar.gz: hexbytes-0.3.0/.tox/py39-lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/configparser.pyi: "LegacyInterpolation",
./hexbytes-0.3.0.tar.gz: hexbytes-0.3.0/.tox/py39-lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/configparser.pyi: class LegacyInterpolation(Interpolation):
./jedi-0.18.2.tar.gz: jedi-0.18.2/jedi/third_party/typeshed/stdlib/3/configparser.pyi: class LegacyInterpolation(Interpolation): ...
./mypy-0.991.tar.gz: mypy-0.991/mypy/typeshed/stdlib/configparser.pyi: "LegacyInterpolation",
./mypy-0.991.tar.gz: mypy-0.991/mypy/typeshed/stdlib/configparser.pyi: class LegacyInterpolation(Interpolation):
./eth-hash-0.5.1.tar.gz: eth-hash-0.5.1/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/configparser.pyi: "LegacyInterpolation",
./eth-hash-0.5.1.tar.gz: eth-hash-0.5.1/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/configparser.pyi: class LegacyInterpolation(Interpolation):
./eth-account-0.7.0.tar.gz: eth-account-0.7.0/.tox/lint/lib/python3.10/site-packages/mypy/typeshed/stdlib/configparser.pyi: class LegacyInterpolation(Interpolation):
./eth-account-0.7.0.tar.gz: eth-account-0.7.0/.tox/py310-lint/lib/python3.10/site-packages/mypy/typeshed/stdlib/configparser.pyi: class LegacyInterpolation(Interpolation):
./eth-utils-2.1.0.tar.gz: eth-utils-2.1.0/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/configparser.pyi: class LegacyInterpolation(Interpolation):
./pytype-2022.11.29.tar.gz: pytype-2022.11.29/pytype/typeshed/stdlib/configparser.pyi: "LegacyInterpolation",
./pytype-2022.11.29.tar.gz: pytype-2022.11.29/pytype/typeshed/stdlib/configparser.pyi: class LegacyInterpolation(Interpolation):
./pylint-2.15.8.tar.gz: pylint-2.15.8/pylint/checkers/stdlib.py: "LegacyInterpolation",
./pyre-check-0.9.17.tar.gz: pyre-check-0.9.17/typeshed/stdlib/configparser.pyi: "LegacyInterpolation",
./pyre-check-0.9.17.tar.gz: pyre-check-0.9.17/typeshed/stdlib/configparser.pyi: class LegacyInterpolation(Interpolation):
./configparser-5.3.0.tar.gz: configparser-5.3.0/src/backports/configparser/__init__.py: "LegacyInterpolation",
./configparser-5.3.0.tar.gz: configparser-5.3.0/src/backports/configparser/__init__.py: class LegacyInterpolation(Interpolation):
./configparser-5.3.0.tar.gz: configparser-5.3.0/src/backports/configparser/__init__.py: "LegacyInterpolation has been deprecated since Python 3.2 "
./configparser-5.3.0.tar.gz: configparser-5.3.0/src/configparser.py: LegacyInterpolation,
./configparser-5.3.0.tar.gz: configparser-5.3.0/src/configparser.py: "LegacyInterpolation",
./configparser-5.3.0.tar.gz: configparser-5.3.0/src/test_configparser.py: elif isinstance(self.interpolation, configparser.LegacyInterpolation):
./configparser-5.3.0.tar.gz: configparser-5.3.0/src/test_configparser.py: elif isinstance(self.interpolation, configparser.LegacyInterpolation):
./configparser-5.3.0.tar.gz: configparser-5.3.0/src/test_configparser.py: elif isinstance(self.interpolation, configparser.LegacyInterpolation):
./configparser-5.3.0.tar.gz: configparser-5.3.0/src/test_configparser.py: class ConfigParserTestCaseLegacyInterpolation(ConfigParserTestCase):
./configparser-5.3.0.tar.gz: configparser-5.3.0/src/test_configparser.py: interpolation = configparser.LegacyInterpolation()
./configparser-5.3.0.tar.gz: configparser-5.3.0/src/test_configparser.py: configparser.LegacyInterpolation()
./eth-rlp-0.3.0.tar.gz: eth-rlp-0.3.0/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/3/configparser.pyi: class LegacyInterpolation(Interpolation): ...
./eth-rlp-0.3.0.tar.gz: eth-rlp-0.3.0/venv-erlp/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/3/configparser.pyi: class LegacyInterpolation(Interpolation): ...
./eth-rlp-0.3.0.tar.gz: eth-rlp-0.3.0/venv-erlp/lib/python3.9/site-packages/mypy/typeshed/stdlib/3/configparser.pyi: class LegacyInterpolation(Interpolation): ...
./eth_abi-3.0.1.tar.gz: eth_abi-3.0.1/.tox/lint/lib/python3.10/site-packages/mypy/typeshed/stdlib/configparser.pyi: class LegacyInterpolation(Interpolation):

Time: 0:00:17.957695
Found 32 matching lines in 12 projects

With colour:

Answer: very little, mostly backports and type stubs. This told us it’s a good candidate for removal, so a proper DeprecationWarning was added in Python 3.11 (released October 2022) and it will be removed in Python 3.13 (October 2024).

Happy searching! 🔎

Header photo: “The card index department” by Boston Public Library is licensed under CC BY 2.0.

Help test Python 3.11 beta!

Tue, 31 May 2022 17:10:00 +0000

Calling all Python library maintainers! 🐍

Python 3.11 is in beta! 🎉

PEP 664 defines the release schedule for Python 3.11.0:

The second beta came out on 31st May 2022
The first release candidate is set for 1st August 2022
And the full release is set for 3rd October 2022

In his announcement, Pablo Galindo Salgado, release manager for Python 3.10 and 3.11, said:

We strongly encourage maintainers of third-party Python projects to test with 3.11 during the beta phase and report issues found to the Python bug tracker as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Monday, 2021-08-02). Our goal is have no ABI changes after beta 4 and as few code changes as possible after 3.11.0rc1, the first release candidate. To achieve that, it will be extremely important to get as much exposure for 3.11 as possible during the beta phase.

Test with 3.11 #

It’s now time for library maintainers to start testing 3.11 with your project. You don’t need to declare support and release for 3.11 yet, but there’s two big benefits to testing with 3.11 on CI:

There have been removals and changes in Python 3.11. Testing now will help you make your code compatible and avoid any big surprises (for you and your users) at the big launch in October.
You might find bugs in Python itself! Reporting those will help get them fixed and help everyone.

How #

GitHub Actions: setup-python #

To test the latest alpha, beta or release candidate with actions/setup-python, add 3.11-dev to your workflow matrix.

For example:

jobs:
 build:
 runs-on: ubuntu-latest
 strategy:
 fail-fast: false
 matrix:
 python-version: ["3.7", "3.8", "3.9", "3.10", "3.11-dev"]

 steps:
 - uses: actions/checkout@v3

 - name: Set up Python ${{ matrix.python-version }}
 uses: actions/setup-python@v4
 with:
 python-version: ${{ matrix.python-version }}

GitHub Actions: deadsnakes #

For the bleeding edge, we can use deadsnakes/action to test the latest nightly build:

jobs:
 build:
 runs-on: ubuntu-latest
 strategy:
 fail-fast: false
 matrix:
 python-version: ["3.7", "3.8", "3.9", "3.10", "3.11-dev"]

 steps:
 - uses: actions/checkout@v3

 - name: Set up Python ${{ matrix.python-version }}
 if: "!endsWith(matrix.python-version, '-dev')"
 uses: actions/setup-python@v3
 with:
 python-version: ${{ matrix.python-version }}

 - uses: deadsnakes/action@v2.1.1
 name: Set up Python ${{ matrix.python-version }} (deadsnakes)
 if: endsWith(matrix.python-version, '-dev')
 with:
 python-version: ${{ matrix.python-version }}

Travis CI #

I recommend moving to another CI.

In the meantime, you can also add 3.11-dev to .travis.yml, although at the time of writing it’s pointing to 3.11.0a3 from 2021-12-08, which is better than nothing.

language: python
python:
 - "3.7"
 - "3.8"
 - "3.9"
 - "3.10"
 - "3.11-dev"

Again, I recommend moving to another CI.

Other CIs #

Do you use other CIs? Please leave a comment if you know how to test 3.11!

When to support 3.11? #

When should you declare support and add the Programming Language :: Python :: 3.11 Trove classifier?

First of all, make sure your tests pass on 3.11 beta. One option is waiting until 3.11.0 final is released.

Or, as mentioned above:

Our goal is have no ABI changes after beta 4 and as few code changes as possible after 3.11.0rc1, the first release candidate.

If you have a pure Python project, you could release now.

If you have C extensions, you might want to wait until the release candidate phase, although if other projects depend on yours, a preview release would help them test and prepare.

In any case, start testing 3.11 now!

Header photo: Uppland Runic Inscription 53, a 1,000 year old runestone in the old town of Stockholm (source)

Top PyPI Packages

Sun, 29 May 2022 15:53:36 +0000

Top PyPI Packages is a website that creates a monthly dump of the 5,000 most-downloaded packages from the Python Package Index (PyPI). It provides a human-readable list and a machine-readable JSON file for programmatic use.

How it’s used #

The generated data is important for the Python community: it has been cited by many academic papers, covering research on topics such as software supply-chain attacks, genetic algorithms, neural type hints and technical debt.

Websites also use the data for analysing things like community adoption of Python 3 and of wheels package files, and for automated dependency updates.

Some are listed here.

Last but not least, CPython core developers often use the data for community analysis, for example to check how widespread certain language features are when considering deprecations and removals, or API changes of Python itself.

Some examples:

https://mail.python.org/archives/search?q=top-pypi-packages

How it works #

It runs on the cheapest DigitalOcean Droplet to query a Google BigQuery dataset from the Python Software Foundation, and builds the JSON file from that. The data is automatically committed from the droplet back to the GitHub repository, and GitHub Actions is used to automatically tag and create a release, which then creates a Digital Object Identifier (DOI) at Zenodo to help make it citable for researchers.

In more detail #

A “Droplet” is the name DigitalOcean uses for virtual machines, which is basically a Linux server. I’m using the cheapest $5/month running Ubuntu 20.04 (LTS) plus $1/month for automated backups. (In July 2022, I’ll switch to the new $4/month Droplet.)

On the first of the month, it runs a cron job:

13 11 1 * * ( eval "$(ssh-agent -s)"; ssh-add ~/.ssh/id_rsa-top-pypi-packages; /home/botuser/github/top-pypi-packages/top-pypi-packages.sh ) > /tmp/top-pypi-packages.log 2>&1

This script runs some others to call pypinfo:

/home/botuser/.local/bin/pypinfo --json --indent 0 --limit 5000 --days 30 "" project > top-pypi-packages-30-days.json

PyPI streams data about downloads to Google BigQuery which are accessible as a public dataset. Google provides a free amount of queries per month, and I’ve been adjusting the amount of data fetched to stay within the free quota (e.g. changing from the top 5k packages to 4k; it used to get data for top packages over the past 30 days and 365 days, but now only for 30 days; and bumping back up to 5k packages).

pypinfo is a handy command-line interface (CLI) to access this BigQuery data and dump it to a JSON file.

Another handy CLI tool called jq minifies the JSON data:

jq -c . < top-pypi-packages-30-days.json > top-pypi-packages-30-days.min.json

These are then committed back to the repo, tagged using CalVer (e.g. 2022-05) and pushed. GitHub Actions creates a release from the tag.

This then creates a Digital Object Identifier (DOI) at Zenodo to help make it citable for researchers.

The website, on GitHub Pages, reads in the generated JSON file and shows the top 100 (or 1,000 or 5,000) packages in human-readable form. It’s based on Python Wheels, which in nice circular fashion, uses the JSON data from this project.

Thanks #

Thanks to PyPI and Google BigQuery for the data; pypinfo and jq for the tools; Python Wheels for making their code open source; and DigitalOcean for sponsoring this project’s hosting. Visit https://do.co/oss-sponsorship to see if your project is eligible.

Header photo: “PACKAGES” by marc falardeau is licensed under CC BY 2.0.

The Python 3.1 problem

Wed, 19 May 2021 14:08:31 +0000

Or, a variation on the Norway problem #

Short version: put quotes around version numbers in YAML.

The Norway problem #

The Norway problem is when you put this in YAML:

countries:
 - GB
 - IE
 - FR
 - DE
 - NO

But get this out:

>>> import yaml
>>> with open("countries.yml") as f:
... yaml.safe_load(f)
...
{'countries': ['GB', 'IE', 'FR', 'DE', False]}

:scream:

The Norway fix #

Use quotes:

countries:
 - "GB"
 - "IE"
 - "FR"
 - "DE"
 - "NO"

>>> with open("countries.yml") as f:
... yaml.safe_load(f)
...
{'countries': ['GB', 'IE', 'FR', 'DE', 'NO']}

🇳🇴 ✅

The Python 3.1 problem #

A similar problem will affect the Python community in October 2021, when Python 3.10 comes out.

Why?

When 3.10 is added to YAML, for example in CI test matrix config, it’s interpreted as a float. This:

python-version: [3.6, 3.7, 3.8, 3.9, 3.10, pypy3]

Turns into this:

>>> import yaml
>>> with open("versions.yml") as f:
... yaml.safe_load(f)
...
{'python-version': [3.6, 3.7, 3.8, 3.9, 3.1, 'pypy3']}

CI failed! It’s not 2009! Python 3.1 not found!

:scream:

Relatedly, 3.10-dev without quotes works because it’s interpreted as a string. But when deleting -dev, 3.10 is interpreted as a float.

The Python 3.10 fix #

Version numbers are strings, not floats. Use quotes:

python-version: ["3.6", "3.7", "3.8", "3.9", "3.10", "pypy3"]

>>> import yaml
>>> with open("versions.yml") as f:
... yaml.safe_load(f)
...
{'python-version': ['3.6', '3.7', '3.8', '3.9', '3.10', 'pypy3']}

🐍 ✅

Python version share over time, 6

Wed, 01 Jan 2020 16:36:25 +0000

January 2016 — December 2019 #

To celebrate the end of life of Python 2 on 1st January 2020, here’s some statistics showing how much different Python versions have been used over four years.

Here’s the pip installs for all packages from the Python Package Index (PyPI), between January 2016 and December 2019:

pip #

The package installer

six #

Python 2 and 3 compatibility library

NumPy #

Scientific computing library

pytest #

Testing framework

pandas #

Data analysis toolkit

Coverage.py #

Code coverage testing

Pillow #

Imaging library

Django #

Web framework

Matplotlib #

2D plotting library

Flake8 #

Linter

Pylint #

Linter

TensorFlow #

Machine learning library

pylast #

Interface to Last.fm

How #

Statistics were collected using pypi-trends.py, a wrapper around pypinfo and pypistats to fetch all monthly downloads from the PyPI database on Google BigQuery and save them as JSON files. Data was downloaded over several days as getting all months uses up a lot of free BigQuery quota. Then jsons2csv.py plots a chart using matplotlib. Raw JSON data is in the repo.

Python version share over time, 5

Mon, 04 Nov 2019 14:40:48 +0000

January 2016 — October 2019 #

To celebrate the release of Python 3.8.0 on 14th October 2019, and with less than two months left for Python 2, here’s some statistics showing how much different Python versions have been used over nearly four years.

Here’s the pip installs for all packages from the Python Package Index (PyPI), between January 2016 and October 2019:

pip #

The package installer

six #

Python 2 and 3 compatibility library

NumPy #

Scientific computing library

pytest #

Testing framework

pandas #

Data analysis toolkit

Coverage.py #

Code coverage testing

Pillow #

Imaging library

Django #

Web framework

Matplotlib #

2D plotting library

Flake8 #

Linter

Pylint #

Linter

pylast #

Interface to Last.fm

How #

Python version share over time, 4

Thu, 04 Apr 2019 14:58:18 +0000

January 2016 — March 2019 #

To celebrate the release of Python 3.7.3 on 25th March 2019, and with under nine months left for Python 2, here’s some statistics showing how much different Python versions have been used over the past three years.

Here’s the pip installs for all packages from the Python Package Index (PyPI), between January 2016 and March 2019:

pip #

The package installer

six #

Python 2 and 3 compatibility library

NumPy #

Scientific computing library

pytest #

Testing framework

Coverage.py #

Code coverage testing

Pillow #

Imaging library

Django #

Web framework

Matplotlib #

2D plotting library

Flake8 #

Linter

Pylint #

Linter

pylast #

Interface to Last.fm

How #

Statistics were collected using pypi-trends.py, a wrapper around pypinfo to fetch all monthly downloads from the PyPI database on Google BigQuery and save them as JSON files. Data was downloaded over several days as getting all months uses up a lot of free BigQuery quota. Then jsons2csv.py plots a chart using matplotlib. Raw JSON data is in the repo.

Python version share over time, 3

Tue, 01 Jan 2019 14:40:48 +0000

January 2016 — December 2018 #

To celebrate the release of Python 3.7.2 on Christmas Eve 2018, and with under a year left for Python 2, here’s some statistics showing how much different Python versions have been used over the past three years.

Here’s the pip installs for all packages from the Python Package Index (PyPI), between January 2016 and December 2018:

For the NumPy scientific computing library:

For the pytest testing framework:

For the Pillow imaging library:

For the Django web framework:

For the matplotlib 2D plotting library:

For the Pylint linter:

And for the pylast interface to Last.fm:

How #

Statistics were collected using pypi-trends.py a wrapper around pypinfo to fetch all monthly downloads from the PyPI database on Google BigQuery and save them as JSON files. Data was downloaded over several days as getting all months uses up a lot of free BigQuery quota. Then jsons2csv.py plots a chart using matplotlib. Raw JSON data is in the repo.

Python version share over time, 2

Sat, 03 Nov 2018 14:40:48 +0000

January 2016 — October 2018 #

To celebrate the release of Python 3.7.1 on 20th October 2018, here’s some statistics showing how much different Python versions have been used over the past two and five-sixths years.

Here’s the pip installs for all packages from the Python Package Index (PyPI), between January 2016 and October 2018:

pip installs from PyPI over time, by Python version

For the NumPy scientific computing library:

Numpy's pip installs from PyPI over time, by Python version

For the Pillow imaging library:

Pillow's pip installs from PyPI over time, by Python version

For the Django web framework:

pylast's pip installs from PyPI over time, by Python version

And for the pylast interface to Last.fm:

Pillow's pip installs from PyPI over time, by Python version

How #

Python version share over time, 1

Thu, 05 Jul 2018 14:40:48 +0000

January 2016 — June 2018 #

To celebrate the release of Python 3.7 on 27th June 2018, here’s some statistics showing how much different Python versions have been used over the past two and a half years.

Here’s the pip installs for all packages from the Python Package Index (PyPI), between January 2016 and June 2018:

pip installs from PyPI over time, by Python version

And for the Pillow imaging library:

Pillow's pip installs from PyPI over time, by Python version

How #

Statistics were collected using pypi-trends.py, a wrapper around pypinfo to fetch all monthly downloads from the PyPI database on Google BigQuery and save them as JSON files. Data was downloaded over three or four days as getting all months uses up a lot of free BigQuery quota. Then jsons2csv.py converts them into a single CSV file for chart-wrangling in Excel. Raw CSV and JSON data is in the repo.

python on Hugo van Kemenade

Soft-deprecating re.match()

Soft deprecation #

re.match() #

re.search() #

re.fullmatch() #

Introducing re.prefixmatch() #

Soft-deprecating re.match() #

Comparison #

Lint #

See also #

Help test Python 3.15!

Test with 3.15 #

How #

When to support 3.15? #

ABI breaks? #

See also #

Security: line goes up

CVEs per year #

GHSAs by month #

GHSAs by year #

Email reports by month #

Thanks #

Fixing the dictionary with Python 3.14

I'm delighted to rejoin the Sovereign Tech Fellowship

2025 impact #

Arbitrary statistics #

2026 and beyond #

A CLI to fight GitHub spam

gh triage spam #

gh triage unassign #

See also #

Speeding up Pillow's open and save

Tachyon #

Loading all the plugins? #

Lazy loading #

See also #

Localising xkcd

PyCon Italia #

PyCon Greece #

PyCon Estonia #

PyCon Finland #

PyCon Sweden #

Thanks #

Replacing python-dateutil to remove six

One #

Two #

Three #

Bonus #

And now for something completely different

alpha 1 #

alpha 2 #

alpha 3 #

alpha 4 #

alpha 5 #

alpha 6 #

alpha 7 #

beta 1 #

beta 2 #

beta 3 #

beta 4 #

release candidate 1 #

release candidate 2 #

release candidate 3 #

3.14.0 (final) #

3.14.1 #

Steering Council results

Steering Council election

Python Core Sprint 2025

Monday highlights #

Tuesday highlights #

Wednesday highlights #

Thursday highlights #

Friday highlights #

Thank you #

Three times faster with lazy imports

Setup #

Benchmarks #

Inline imports #

Fully lazy #

`re.match()` #

`re.search()` #

`re.fullmatch()` #

Introducing `re.prefixmatch()` #

Soft-deprecating `re.match()` #

`gh triage spam` #

`gh triage unassign` #

`run_release.py` #

actions/setup-python: `t` suffix #

actions/setup-uv: `t` suffix #

actions/setup-python: `freethreaded` variable #

`PYTHON_GIL=0` #

`pyproject.toml` #