Preparing a distribution at some point is as easy as running make release ( on windows it would be something like make-release.bat ). Until that point is reached though, one will have to go through an iterative setup and testing procedure which will be described here.
The workflow guide assumes that you have a default setup of your info module and setup.py script, and if you started from the template project this will already be in working order. Once your project grows and you decide to distribute only parts of it, the somewhat advanced techniques described here later on will help getting good results faster.
In the following sections it will also be assumed that you made yourself familiar with the setup.py script and its general use.
No matter what kind of distribution you are going to create, it is wise to setup good means of tracking them. The MRV distribution system supports this by integrating git, using tags, branches, and commit messages to track where each distribution came from and from which source tree it was created.
This is vital in case patches need to be back-ported to existing releases, to mention just one example.
To enable git when creating a distribution, set the --use-git flag to 1 on the setup script.
Details about the git-handling can be found in the setup script’s documentation.
One thing you want to do before most releases is to run the unit-tests in the most possible variations and on all supported platforms. In practice this won’t always be possible, but MRV facilitates this through the integration of regression testing provided by the tmrv tool.
It will run all tests on all installed maya versions and report failure if one test fails.
Depending on your distribution target, you would set the --regression-tests flag to 1.
The final decision would be to determine which kind of distributions you need.
Currently there is native support for source and binary distributions, possibly distinguishing between beta or final release distributions. The first ones mentioned are less strict with their version and git tags, by allowing the program version to remain the same, by simply moving the same tag around. This can be accomplished by the --force-git-tag distribution flag.
The documentation usually needs to be build from an intermediate markup language, which is accomplished by makedoc, hence it an own distribution target.
Each distribution target usually ends up in one compressed archive and possibly in a dedicated git branch to connect it to previous versions of the respective distribution target.
All of these targets can be combined freely according to your needs.
This kind of distribution is the most complex as it turns out, and starts by creating a MANIFEST.in file.
With it you define the exact set of files to distribute.
Once you are sure about the set of files to include in your source distribution, adjust your Manifest Template accordingly.
The following command will prepare a source distribution and create a zip archive from it:
$ python setup.py sdist --format=zip
You will find the zip archive in the dist folder. But is this source distribution complete ? Are all files included that the project needs to run ?
To verify that it is operational, it is required to re-run the unit-tests, using the given maya version:
$ python setup.py sdist --format=zip --post-testing=2011
If that works as well, you have a good indication that you are ready to proceed to the next step, The Binary Distribution, which is required if you would like the user to be able to automatically install your project into his python installation. It is somewhat unfortunate that you can’t just go with the working source package.
Please note that the commandline presented here represented the bare minimum, one which might be more suitable for final releases would look more like this:
$ python setup.py --use-git=1 --regression-tests=1 sdist --format=zip --post-testing=8.5,2011 --root-remotes=origin --dist-remotes=distro,hubdistro
The preceding command tells the distribution system to run all regression tests, setup a git tag in your root repository with the proper version number, create a source distribution, run all unit tests in that distribution for maya 8.5 and 2011, to finally push the tagged changes in the root repository ( containing your sources ) to the remote named origin, and the tagged distribution data itself to the remotes named distro and hubdistro.
This step is required if you want to make your distribution installable, or if you don’t want to distribute the sources of your project. In the latter case please keep in mind that you will have to byte-compile for each minor version of the python interpreter, each interpreter represents a separate distribution target.
If you are coming from The Source Distribution, you were used to simple and straightforward definition of the files and folders to distribute. This nice time might be over now, depending on the complexity of your project of course.
The cause of this is mainly the lack of something like the MANIFEST template, hence the files to build need to be specified using keyword arguments given to the setup() routine of your setup.py script.
This information is bundled in the setup_kwargs member variable in your info module. Here it is best to have a look at MRVs well documented version of this file.
Generally, you have to watch out for the following:
- To exclude specific modules from byte-compilation, list patterns in the exclude_from_compile keyword of the build_py command.
- To exclude packages, modules and data files from being included in the build, list the modules in the exclude_items keyword of the build_py command.
- To prevent scripts you previously supplied in the scripts keyword from being installed, mention their paths in the exclude_scripts keyword of the build_scripts command.
Now you would execute a simple build and go back and forth between your info module configuration and the execution of this line until the unit-tests succeed and the distribution as contained in the build directory passes your visual examination:
$ python setup.py --maya-version=2011 build build_py--compile
The specified --maya-version indicates the target platform and is required whenever you want to compile your sources, which is optional by the way. Specifying build first makes sure we get a complete build of everything, while telling build_py to compile the sources. Its hard to know which subcommands are executed implicitly without studying the sources of the distutils or without trying to get smart from studying the scattered distutils documentation.
Once everything appears to be okay, you are free to produce optimized byte code to have smaller files ( as docstrings will be stripped ), which will automatically exclude the unit test library:
$ python setup.py --maya-version=2011 build build_py--optimize=2
A fully upgraded binary distribution could look like this:
$ python setup.py --use-git=1 --regression-tests=1 build build_py --compile --post-testing=8.5,2011 --root-remotes=origin --dist-remotes=distro,hubdistro bdist bdist_dumb --format=zip
The given commandline will create a full build into the build subdirectory, build_py byte-compiles the sources and deletes the source files afterwards. The directory contents will be picked up by bdist to be put into an archive which will extract into the right spot of your platform. bdist allows to specify other target platforms as well, as to be read in the official bdist documentation.
The documentation needs to be built as well, and there is a specialized command which does exactly that by calling ./doc/makedoc ( or whatever you have specified in your info module ).
To be sure the documentation reflects the modules actually used in your source or binary distribution, use the --from-build-version flag. If your distribution does not omit any files, it is usually alright to create ( and possibly reuse ) documentation in your root project which happens if the said flag is not specified.
A basic example would be:
$ python setup.py sdist docdist --from-build-version
The doc-distribution can also be combined with the source or the binary distribution, just append the docdist part presented here to the commandlines shown in the preceding sections.
If you were following the previous steps, you have two or three zip files ( and a bunch of new commits in your root and distribution repositories ). These zip files correspond to the source-, the binary- and the documentation distribution.
Now one has to verify that these packages actually install correctly. For the sake of brevity, we skip ‘dumb’ binary- and documentation installations as these just extract into place, but focus on the source distribution which has to be installed using your very own setup.py script that previously created the distribution package.
To test the source distribution with the distutils, we need to extract it into place, change directory into it to execute the following line:
$ python setup.py install --prefix=install
The purpose of this test is to see whether the script succeeds - if it does the install subdirectory will contain our scripts and modules, by default these have been precompiled to byte-code with the interpreter that ran the setup script. Internally it uses build_py which we already verified to produce a consistent distribution in a prior step, hence it is not strictly required to rerun the unit tests.
Another way to install an archive is by using easy_install. For the end-user, this is a much more convenient way of installing a package, provided that you have uploaded it to the Python Package Index before, allowing lines like these:
$ easy_install your_project
Internally, the ez_setup implementation will call setup.py install. One could assume that it will work if your distutil installation worked, but this is not necessarily true.
Easy_install runs your installation routine within a sandboxed environment that restricts certain operation that try to alter files outside of the installation directory. Unfortunately the implementation does not provide the exact signatures of common os methods, which in turn could cause your script to fail even though you didn’t try anything ‘forbidden’.
The following commandline tells easy_install not to use pypi, but instead to use your package directly from the dist directory with your distribution archives. Please note that it will install the package in your system, hence you need root permissions and to possibly manually remove the installed egg afterwards.:
$ easy_install<-py-version> --allow-hosts None --find-links dist your_project
Where <-py-version> is the python version you want to use for the installation. If the installation succeeds, you should be able to use your package and scripts right away in an interactive python session:
$ python<py-version>
$ >>> import your_project
If you are handling paths a lot and manipulate them, or in short, use plenty of operating system facilities, it is vital to make sure your installation works on other platforms as well.
If feasible, run your unit-tests on all supported platforms and try the installation yourself. Usually, the first time you try it on another system, it won’t work, but is fixable with trivial patches. It good though that you find out about it before the end-user does.
If software doesn’t work right after the installation, the first impression of “your project is not working” is hard to get rid of later.
Once the actual commandlines have been figured out and are definitely working, its just about aliasing them to make their execution easy.
On Linux, the simplest way is to write a makefile which contains a few phony targets, this way you can easily choose the distribution targets like:
$ make preview
$ make beta-release
$ make final-release
$ make binary-release
Writing shell scripts would be possible as well, its up to your personal preference.
On Windows, one might want to resort to writing batch files which serve as an alias, allowing to write something like:
$ make-preview.bat
$ make-beta-release.bat
$ make-final-release.bat
$ make-binary-release.bat
Note
It is very recommended to maintain targets which are for release-testing only, hence the do not interact with git. This is especially important in the case of the --post-testing flag as it will always do the testing after the changes were pushed to the git repositories, and if the test fails, the changes possibly already reached the public repository. A good insurance against post-testing failures are pre-distribution regression tests, but as these apply to a possibly different file set, post-testing might still fail.
The Python Package Index helps you to store your release archives in a central place, which makes it available to easy_install automatically. Additionally it provides a spot for your static html documentation, the zip archive docdist creates can be uploaded with just one click.