Random presentations (both in Spanish and English)
This project is maintained by jgbarah
These are some notes that I use in the workshops I facilitate, about packaging Python projects as distributable files (that is, producing distributions from projects).
There are two major systems for producing distributions in Python: Python Packaging Authority (pip packages) and conda. These notes are about the former.
Usually, you want to package a Python project, including a hierarchy of packages (collections of modules), with one or more scripts with act as drivers for it. This will be the scenario described in this document.
Naming used for Python packaging may be a bit confusing. Have at Nomenclature for a reference to the names used in these notes.
Before starting to package, you need to structure your code, so that there is a Python package, one or more scripts for driving it, and maybe other stuff (tests, data needed by the module, etc.).
You also need to think about naming. The name of the Python distribution should be (as much as posible) unique, and matching the name of the package, to avoid confusion. Knowing that the name of the package is unique is difficult, because as far as I know, there is no registry of Python package names. So, you can only due your due diligence, some guessing, or both. Knowing that the name of the distribution is unique is easier: check in the Pypi website, or better, in the Pypi Warehouse (the work in progress system that will deprecate the current Pypi).
If a package name is already “taken” by someone in Pypi, you can try to contact the person who registered that name (use the Pypi Warehowse for that). There is a conflict resolution PEP in the works, as far as I know it is not yet in force.
If you want to use names with more than one word,
use _ (underscore) to separate the words in the package name,
and - (hyphen) to separate the words in the distribution name.
Other combinations may work, but this one seems to be customary,
and minimizes surprises for users.
In the rest of this document, I will use pkg_name for
the package (collection of modules) name and dist-name for the
distribution name (distributable package).
I assume that the project source code is in a single directory (in most cases a single git repository), where there is nothing else than the code for the package. Usually, it is structured as:
pkg_name directory: the source code for the packagebin directory: scripts (usually driving scripts)tests directory: tests (usually unit tests)setup.py: configuration for creating the distributionsetup.cfg: some extra configuration that may be neededManifest.in: to list additional files not automatically includedIn addition, it is convenient to have:
module_name/_version.py: file with the version of the package,
with a content such as:# Versions compliant with PEP 440 https://www.python.org/dev/peps/pep-0440
__version__ = "0.30.23"
README.md: file with the description of the package,
which will be converted to README.rst (rst is the traditional format in Pypi,
although now Markdown is also supported).Templates for most the files listed above can be found in the PyPA sample Python project.
As a result of building the distributions, usually the following directories are created:
build directory: used to build the packagedist directory: used to store the resulting packagesFirst of all, ensure you have Python installed (of course),
and that pip is available for that installation.
If not, follow instructions in
Requirements for installing packages.
Now, assuming dir is the directory with your project,
with the above layout, and configuration files, you can type:
$ python3 -m venv /tmp/pip
$ source /tmp/pip/bin/activate
(pip) $ pip install --upgrade pip setuptools wheel pandoc
(pip) $ cd dir
(pip) $ rm -rf build dist
(pip) $ python setup.py sdist
(pip) $ python setup.py bdist_wheel
This will produce your distributions (a source distribution, sdist,
and a binary distribution in wheel format, bdist_wheel) in the
dist directory.
Once you have produced your distributions, you can test
them by installing them in a fresh virtual environment. For that,
you can use the --no-index --find-links options.
For example (asuming newly created distributions are in
dir/dist):
$ python3 -m venv /tmp/test
$ source /tmp/test/bin/activate
(test) pip install --no-index --find-links dir/dist
If you have a project configured for producing distributions, you can install it in the current virtual environment in development (editable) mode. In this mode, the project is installed in a way that any modification to its files is immediately used in the next excution, even if no new reinstallation is done. In other words, you can just edit the files, run anything using the project (for instance running a script that imports any of its modules), and the edited version will be used.
For installing in developer mode, move to the project directory and type (notice the dot at the end of the line):
$ pip install -e .
This will install the project in development mode, and all its dependencies as specified in the project configuration, downloading them from Pypi.
The usual workflow using this development mode is:
The recommended program for uploading distributions to Pypi is
twine. In a virtual environment
(for example, the one used for building the distributions),
assuming current directory is dir, just type:
(pip) $ pip install --upgrade twine
(pip) $ twine upload dist/*
The latest one needs a working twine configuration for credentials to Pypi, including the user and password to upload to Pypi. If it is not previously configured, it will just ask for those credentials.