pypi-tools

Command-line Python scripts to do things with the Python Package Index (PyPI).

pypi-trends.py

pypi-trends.py is a wrapper around pypinfo and pypistats to fetch all monthly downloads from the PyPI database on Google BigQuery and save them as JSON files.

With --clickpy, it instead queries ClickPy’s public ClickHouse mirror of the PyPI downloads dataset: free, no auth needed, and a single month’s all-PyPI query no longer exceeds the entire free BigQuery quota.

jsons2img.py

jsons2img.py converts the JSON files into a single CSV file for chart-wrangling in a spreadsheet, and can generate a chart using Matplotlib. Once an image exists, it can be re-generated with make -j4.

Examples

Here the pip installs for all packages from the Python Package Index (PyPI), from January 2016 onwards:

urllib3

HTTP client

Requests

HTTP library

See more charts here.

See also

• Python version share over time

source-finder.py

Given a PyPI package, source_finder.py looks for the source repository in its metadata.

$ python source_finder.py six
https://github.com/benjaminp/six
$ python source_finder.py urllib3
None

It caches the JSON metadata downloaded from PyPI in a temporary directory, use the --verbose option to see where. The cache files will be deleted the next month.

$ python source_finder.py s3transfer --verbose
API URL: https://pypi.org/pypi/s3transfer/json
Cache file: /Users/hugo/Library/Caches/source-finder/2019-10-https-pypi-org-pypi-s3transfer-json.json
Cache file exists
project_urls    Homepage        https://github.com/boto/s3transfer
Success!
project_urls    Homepage        https://github.com/boto/s3transfer
Success!
https://github.com/boto/s3transfer

top_repos.py

This will look for the source repo for the most-downloaded packages, using a JSON file from Top PyPI Packages, and save them to data/top-repos.json.

First, fetch fresh copy of the top packages:

$ wget https://hugovk.dev/top-pypi-packages/top-pypi-packages.min.json -O  data/top-pypi-packages.json

--2019-10-14 18:12:45--  https://hugovk.dev/top-pypi-packages/top-pypi-packages.min.json
Resolving hugovk.dev (hugovk.dev)... 185.199.110.153, 185.199.108.153, 185.199.111.153, ...
Connecting to hugovk.dev (hugovk.dev)|185.199.110.153|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 250885 (245K) [application/json]
Saving to: ‘data/top-pypi-packages.json’

data/top-pypi-packages.json      100%[========================================================>] 245.00K  --.-KB/s    in 0.02s

2019-10-14 18:12:45 (14.7 MB/s) - ‘data/top-pypi-packages.json’ saved [250885/250885]

Check the first 10 packages:

$ python top_repos.py -n 10
Load data/top-repos.json...
Load top-pypi-packages.json...
Already done: 0
Find new repos...
1 urllib3
2 six       https://github.com/benjaminp/six
3 requests
4 botocore  https://github.com/boto/botocore
5 python-dateutil
6 certifi
7 s3transfer        https://github.com/boto/s3transfer
8 pip
9 idna      https://github.com/kjd/idna
10 docutils
Old repos: 0
New repos: 4
Not found: 6
Save data/top-repos.json...

When running again:

• if a package already has a repo, it’s not checked again in case it’s changed
• if a package doesn’t have a repo, it will be checked

Currently, it finds 3,951 repos for the top 5,000 packages.

I’m not planning on automating this, but can run it from time to time to update it.