germinate/0000755000000000000000000000000012326425437007727 5ustar germinate/man/0000755000000000000000000000000012326425437010502 5ustar germinate/man/germinate-update-metapackage.10000644000000000000000000001320112326425437016254 0ustar .Dd Jul 18, 2007 .Os Ubuntu .ds volume-operating-system Ubuntu .Dt GERMINATE\-UPDATE\-METAPACKAGE 1 .Sh NAME .Nm germinate\-update\-metapackage .Nd update a set of metapackages generated from seeds .Sh SYNOPSIS .Nm .Op Fl Fl bzr .Op Fl Fl output-directory Ar dir .Op Ar dist .Sh DESCRIPTION .Nm assists with the construction and update of .Dq metapackages (packages consisting solely of a list of dependencies) from a list of seed packages. It updates binary package stanzas in .Pa debian/control to reflect the current contents of the seeds, and updates .Pa debian/changelog with a description of the changes it made. .Pp .Nm requires a configuration file, called .Pa update.cfg , in the current directory. The format is described below. .Pp If a non-option argument is given, it specifies the distribution for which metapackages should be generated (e.g.\& .Dq unstable ) . .Sh OPTIONS .Bl -tag -width 4n .It Fl Fl nodch Don't modify .Pa debian/changelog . .It Fl Fl bzr Check out seeds from the .Ic bzr branch defined in the configuration file rather than fetching them directly from the URL defined there. Requires .Ic bzr to be installed. .It Xo Fl o , .Fl Fl output-directory Ar dir .Xc Output the package lists in the specified directory. .El .Sh CONFIGURATION FILE .Pa update.cfg uses Python's ConfigParser configuration file syntax, supporting interpolation as defined by SafeConfigParser. It should have a .Li DEFAULT section with a .Li dist key indicating the default distribution, and a .Ar dist section (corresponding to that distribution). It may also have a .Ar dist Ns Li /bzr section which can override the .Li seed_base and .Li seed_dist values from the .Ar dist section if the .Fl Fl bzr option is given. The following keys are recognised in distribution sections: .Bl -tag -width 4n .It Li seeds No (mandatory) Pass these seeds to the germinator for processing. .It Li output_seeds No (optional, deprecated) Generate metapackages for these seeds. If not specified, the value of .Li seeds is used. This option is usually no longer necessary now that the value of .Li seeds is automatically expanded for seed inheritance. .It Li architectures No (mandatory) Generate metapackages for these architectures. .It Li archive_base/default No (optional) Use this URL as the default base for fetching package indices from the archive; for examples of valid URLs, see .Li deb lines in .Pa /etc/apt/sources.list , or the .Ar MIRROR argument to .Xr debootstrap 8 . .It Li archive_base/ Ns Ar arch No (optional) Use this URL as the base for fetching package indices from the archive for the specified architecture. For each architecture being processed, at least one of .Li archive_base/default and .Li archive_base/ Ns Ar arch must be present. To try multiple URLs, separate them with commas or spaces; the newest version of each package across all archives will win. Note that .Xr debootstrap 8 will only use the first archive. .It Li seed_base No (mandatory) The base URL for fetching seeds. To try multiple URLs (for example if a seed branch includes another branch stored at a different location), separate them with commas or spaces. .It Li seed_dist No (optional) The tail of the URL for fetching seeds. This will be appended to .Li seed_base . You will often want to interpolate the value of .Li dist into this value using ConfigParser's .Li %(dist)s syntax. If not specified, the value of .Li dist is used. .It Li dists No (optional) The distributions from which to fetch package indices. Listing multiple distributions may be useful, for example, when examining both a released distribution and its security updates. If not specified, the value of .Li dist is used. .It Li components No (mandatory) The archive components from which to fetch package indices. .It Li seed_map/ Ns Ar seed No (optional, deprecated) The seeds to be used as input for the metapackage corresponding to .Ar seed . If specified, this will typically be the list of seeds from which .Ar seed inherits, plus .Ar seed itself. This option is usually no longer necessary; use a .Li Task\-Seeds header in the seed file instead. .It Li metapackage_map/ Ns Ar seed No (optional, deprecated) The metapackage name to output for .Ar seed . If not specified, .Nm will look for the name of the source package in which it is being run, remove "meta" from the end, and append the seed name. This option is usually no longer necessary; use a .Li Task\-Metapackage header in the seed file instead. .El .Sh EXAMPLE At the time of writing, the following configuration file is used to generate the .Li kubuntu-meta source package in the Ubuntu archive: .Bd -literal -offset indent [DEFAULT] dist: trusty [trusty] seeds: desktop full active architectures: i386 amd64 powerpc armhf arm64 ppc64el seed_base: http://people.canonical.com/~ubuntu-archive/seeds/ seed_dist: kubuntu.%(dist)s archive_base/default: http://archive.ubuntu.com/ubuntu/ archive_base/ports: http://ports.ubuntu.com/ubuntu-ports/ archive_base/powerpc: %(archive_base/ports)s archive_base/armhf: %(archive_base/ports)s archive_base/arm64: %(archive_base/ports)s archive_base/ppc64el: %(archive_base/ports)s components: main restricted universe [trusty/bzr] seed_base: bzr+ssh://bazaar.launchpad.net/~kubuntu-dev/ubuntu-seeds/ bzr+ssh://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/ seed_dist: kubuntu.%(dist)s .Ed .Sh AUTHORS .An Gustavo Franco Aq stratus@debian.org .An Colin Watson Aq cjwatson@canonical.com .Pp .An -nosplit .Nm is copyright \(co 2004, 2005, 2006, 2007, 2008 .An Canonical Ltd. and copyright \(co 2006 .An Gustavo Franco . See the GNU General Public License version 2 or later for copying conditions. A copy of the GNU General Public License is available in .Pa /usr/share/common\-licenses/GPL . germinate/man/germinate-pkg-diff.10000644000000000000000000000512112326425437014223 0ustar .Dd Jul 18, 2007 .Os Ubuntu .ds volume-operating-system Ubuntu .Dt GERMINATE\-PKG\-DIFF 1 .Sh NAME .Nm germinate\-pkg\-diff .Nd compare seeds against currently installed packages .Sh SYNOPSIS .Nm .Op Fl l Ar file .Op Fl m Brq Li i Ns | Ns Li r Ns | Ns Li d .Op Fl a Ar arch .Op Ar seeds .Sh DESCRIPTION .Nm compares the expansion of a list of seed packages against the set of packages installed on the current system. When constructing seeds for a software distribution, it can be used to iteratively find packages installed on developers' systems that should be included in the seeds. .Pp A list of seeds against which to compare may be supplied as non-option arguments. Seeds from which they inherit will be added automatically. The default is .Sq desktop . .Sh OPTIONS .Bl -tag -width 4n .It Xo Fl l , .Fl Fl list Ar file .Xc Read the list of currently installed packages from .Ar file . The default is to read the output of .Ic dpkg Fl Fl get\-selections , and any supplied file should be in the same format. .It Xo Fl m , .Fl Fl mode Brq Li i Ns | Ns Li r Ns | Ns Li d .Xc Set the output mode as follows: .Bl -tag -width 4n .It Li i Show the .Ic dpkg selections needed to install just these seeds. List unseeded but installed files as .Dq deinstall , and seeded but uninstalled files as .Dq install . .It Li r List unseeded but installed files as .Dq install , and seeded but uninstalled files as .Dq deinstall . .It Li d Show the differences between the packages specified by the seeds and the list of installed packages, in a somewhat .Ic diff Ns -like format. .El .It Xo Fl S , .Fl Fl seed\-source Ar source Ns \&,... .Xc Fetch seeds from the specified sources. The default is .Pa http://people.canonical.com/~ubuntu-archive/seeds/ . .It Xo Fl s , .Fl Fl seed\-dist Ar dist .Xc Fetch seeds for distribution .Ar dist . The default is .Li ubuntu.trusty . .It Xo Fl d , .Fl Fl dist Ar dist Ns \&,... .Xc Operate on the specified distributions. The default is .Li trusty . Listing multiple distributions may be useful, for example, when examining both a released distribution and its security updates. .It Xo Fl a , .Fl Fl arch Ar arch .Xc Operate on architecture .Ar arch . The default is .Li i386 . .El .Sh BUGS .Fl Fl mode Li r is useless as .Ic dpkg Fl Fl set\-selections input. .Sh AUTHORS .An Lamont Jones Aq lamont@ubuntu.com .An Colin Watson Aq cjwatson@canonical.com .Pp .An -nosplit .Nm is copyright \(co 2004, 2005, 2006, 2007, 2008 .An Canonical Ltd . See the GNU General Public License version 2 or later for copying conditions. A copy of the GNU General Public License is available in .Pa /usr/share/common\-licenses/GPL . germinate/man/germinate.10000644000000000000000000002320212326425437012536 0ustar .Dd May 27, 2005 .Os Ubuntu .ds volume-operating-system Ubuntu .Dt GERMINATE 1 .Sh NAME .Nm germinate .Nd expand dependencies in a list of seed packages .Sh SYNOPSIS .Nm .Op Fl v .Op Fl S Ar source .Op Fl s Ar dist .Op Fl m Ar mirror .Op Fl d Ar dist Ns \&,... .Op Fl a Ar arch .Op Fl c Ar component Ns \&,... .Op Fl Fl bzr .Op Fl Fl no\-rdepends .Op Fl Fl no\-installer .Sh DESCRIPTION .Nm is a program to help with the maintenance of large software distributions. It takes a list of seed packages and a mirror of the distribution, and produces outputs with the seed packages and their dependencies and build-dependencies expanded out in full. .Ss Seeds The contents of the Ubuntu distribution, and others, are managed by means of .Em seeds . At their simplest, these are lists of packages which are considered important to have in the main component of the distribution, without explicitly listing all their dependencies and build-dependencies. .Pp Seed lists are typically divided up by category: a .Li base or .Li minimal seed might list the core set of packages required to make the system run at all, while a .Li desktop seed might list the set of packages installed as part of a default desktop installation. .Nm takes these seeds, adds their dependency trees, and produces an .Em output for each seed which contains a dependency-expanded list of package names. These outputs may be handed on to archive maintenance or CD-building tools. .Pp Some seeds may .Em inherit from other seeds: they rely on those seeds to be installed. For example, a .Li desktop seed will typically inherit from a .Li minimal seed. .Nm understands these inheritance relationships. If a package in the .Li desktop seed depends on .Sq foo , but .Sq foo is already part of the .Li minimal seed or dependency list, then .Sq foo will not be added to the .Li desktop output. .Pp Seeds are stored in text files downloaded from a given URL. Lines not beginning with .Sq "\ *\ " (wiki-style list markup) are ignored. .Pp Seed entries may simply consist of a package name, or may include any of the following special syntax: .Bl -tag -width 6n .It % Seed entries beginning with .Sq % expand to all binaries from the given source package. .It [...] Seed entries may be followed with .Sq " [" Ns Ar arch1 Ar arch2 ... Ns \&] to indicate that they should only be used on the given architectures, or with .Sq " [!" Ns Ar arch1 No ! Ns Ar arch2 ... Ns \&] to indicate that they should not be used on the given architectures. .It (...) Seed entries in parentheses indicate that the seed should be treated as a recommendation of metapackages generated from this seed, rather than as a dependency. .It ! Seed entries beginning with .Sq \&! cause the given package to be blacklisted from the given seed and any seeds from which it inherits; this may be followed by .Sq % as above to blacklist all binaries from the given source package. Note that this may result in uninstallable packages whose dependencies have been blacklisted, so use this feature sparingly. The purpose of a blacklist is to make it obvious when a package that is not supposed to be installed ends up in .Nm Ns 's output, so that package relationships can be fixed to stop that happening. It is not intended for the purpose of working around buggy package relationships, and attempts to do so will not work because .Ic apt has no way to know about blacklist entries in seeds. .It key: value Some seeds also contain headers at the top of the file, in .Dq key: value format. For the most part, these are not parsed by .Nm itself. The Ubuntu .Ic tasksel package uses keys beginning with .Sq Task\- to define fields of similar names in its .Pa .desc files. .Xr germinate\-update\-metapackage 1 uses some of these headers to reduce the need for fragile configuration; see its documentation for further details. .El .Pp A .Pa STRUCTURE file alongside the seeds lists their inheritance relationships. It may also include lines beginning with .Sq include , causing other collections of seeds to be included as if they were part of the collection currently being germinated, or lines beginning with .Sq feature , which set flags for the processing of seeds. The only flag currently defined is .Sq follow\-recommends , which causes .Nm to treat Recommends fields as if they were Depends. (Features may also be set on a per-seed basis using lines beginning with .Sq "\ *\ Feature:" in the seed file; here, .Sq no\-follow\-recommends is also supported to allow Recommends-following to be turned off for individual seeds.) .Ss Build-dependencies and Sq supported There is typically no need for a default desktop installation to contain all the compilers and development libraries needed to build itself from source; if nothing else, it would consume much more space. Nevertheless, it is normally a requirement for the maintainers of a distribution to support all the packages necessary to build that distribution. .Pp .Nm therefore does not add all the packages that result from following build-dependencies of seed packages and of their dependencies (the .Dq build-dependency tree ) to every output, unless they are also in the seed or in the dependency list. Instead, it adds them to the output for the last seed in the .Pa STRUCTURE file, conventionally called .Li supported . .Pp Like any other seed, the supported seed may contain its own list of packages. It is common to provide support for many software packages which are not in the default installation, such as debugging libraries, optimised kernels, alternative language support, and the like. .Ss Outputs The output files are named after the seed to which they correspond. An additional output file is needed for supported, namely .Sq supported+build\-depends , which contains the supported list and the build-depends lists of the other seeds all joined together. An .Sq all output is produced to represent the entire archive. .Pp Some other files are produced for occasional use by experts. See the .Pa README file for full details on these. .Sh OPTIONS .Bl -tag -width 4n .It Xo Fl v , .Fl Fl verbose .Xc Be more verbose when processing seeds. .It Xo Fl S , .Fl Fl seed\-source Ar source Ns \&,... .Xc Fetch seeds from the specified sources. The default is .Pa http://people.canonical.com/~ubuntu-archive/seeds/ , or .Pa http://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/ if the .Fl Fl bzr option is used. You may use .Pa file:// URLs here to fetch seeds from the local file system; for example, if your seeds are stored in .Pa /home/username/seeds/debian.unstable , then you would use the options .Fl S Ar file:///home/username/seeds/ .Fl s Ar debian.unstable . .It Xo Fl s , .Fl Fl seed\-dist Ar dist .Xc Fetch seeds for distribution .Ar dist . The default is .Li ubuntu.trusty . .It Xo Fl m , .Fl Fl mirror Ar mirror .Xc Get package lists from .Ar mirror . The default is .Pa http://archive.ubuntu.com/ubuntu/ . May be supplied multiple times; the newest version of each package across all archives will win. .It Fl Fl source\-mirror Ar mirror Get source package lists from .Ar mirror . The default is to use package lists mirrors. May be supplied multiple times; the newest version of each source package across all archives will win. .It Xo Fl d , .Fl Fl dist Ar dist Ns \&,... .Xc Operate on the specified distributions. The default is .Li trusty . Listing multiple distributions may be useful, for example, when examining both a released distribution and its security updates. .It Xo Fl a , .Fl Fl arch Ar arch .Xc Operate on architecture .Ar arch . The default is .Li i386 . .It Xo Fl c , .Fl Fl components Ar component Ns \&,... .Xc Operate on the specified components. The default is .Li main . .It Fl Fl bzr Check out seeds from the .Ic bzr branch found at .Ar seed\-source Ns / Ns Ar seed\-dist rather than fetching them directly from a URL. Requires .Ic bzr to be installed. .It Fl Fl no\-rdepends Disable reverse-dependency calculations. These calculations cause a large number of small files to be written out in the .Pa rdepends/ directory, and may take some time. .It Fl Fl no\-installer Do not consider debian-installer udeb packages. While generally not the desired outcome, sometimes you might wish to omit consideration of installer packages when processing your seeds, perhaps if sending the output directly to the package manager on an already-installed system. .It Fl Fl seed\-packages Ar parent Ns / Ns Ar pkg Ns \&,... Treat each .Ar pkg as a seed by itself, inheriting from .Ar parent (i.e. assuming that all packages in the .Ar parent seed are already installed while calculating the additional dependencies of .Ar pkg ) . This allows the use of .Nm to calculate the dependencies of individual extra packages. For example, .Fl Fl seed\-packages Ar desktop Ns / Ns Ar epiphany\-browser will create an .Pa epiphany\-browser output file listing the additional packages that need to be installed over and above the .Ar desktop seed in order to install .Ar epiphany\-browser . .El .Sh BUGS The wiki-style markup in seeds was inherited from an early implementation, and is a wart. .Pp .Nm can sometimes be confused by complicated situations involving the order in which it encounters dependencies on virtual packages. Explicit entries in seeds may be required to work around this. .Pp Handling of installer packages (udebs) is complicated, poorly documented, and doesn't always work quite right: in particular, packages aren't demoted to the supported seed when they should be. .Sh AUTHORS .An Scott James Remnant Aq scott@canonical.com .An Colin Watson Aq cjwatson@canonical.com .Pp .An -nosplit .Nm is copyright \(co 2004, 2005, 2006, 2007, 2008 .An Canonical Ltd . See the GNU General Public License version 2 or later for copying conditions. A copy of the GNU General Public License is available in .Pa /usr/share/common\-licenses/GPL . germinate/run-pychecker0000755000000000000000000000066412326425437012442 0ustar #! /bin/sh set -e if ! which pychecker >/dev/null 2>&1; then echo "pychecker not installed, so not running it" >&2 exit 0 fi cleanup () { find germinate -name tests -prune -o -type f \( -name \*.pyc -o -name \*.pyo \) -print0 | xargs -0r rm -f } trap cleanup EXIT HUP INT QUIT TERM ret=0 for x in $(find germinate -name tests -prune -o -name \*.py -printf '%p\n' | sort); do pychecker "$x" | grep "^$x:" && ret=1 done exit $ret germinate/debian/0000755000000000000000000000000012326425461011146 5ustar germinate/debian/rules0000755000000000000000000000244512326425437012236 0ustar #! /usr/bin/make -f %: dh $@ --with python2,python3 PY2REQUESTED := $(shell pyversions -r) PY2DEFAULT := $(shell pyversions -d) PY3REQUESTED := $(shell py3versions -r) PY3DEFAULT := $(shell py3versions -d) # Run setup.py with the default python/python3 last so that the scripts use # #!/usr/bin/python(3) and not #!/usr/bin/pythonX.Y. PY2 := $(filter-out $(PY2DEFAULT),$(PY2REQUESTED)) python PY3 := $(filter-out $(PY3DEFAULT),$(PY3REQUESTED)) python3 override_dh_auto_build: dh_auto_build set -e; for python in $(PY3); do \ $$python setup.py build; \ done ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS))) override_dh_auto_test: set -e; for python in $(PY2) $(PY3); do \ $$python setup.py test; \ done endif override_dh_auto_clean: dh_auto_clean rm -rf build override_dh_auto_install: # setuptools likes to leave some debris around, which confuses things. find build -name \*.pyc -print0 | xargs -0r rm -f find build -name __pycache__ -print0 | xargs -0r rm -rf find build -name \*.egg-info -print0 | xargs -0r rm -rf dh_auto_install # Allow the Python 3 top-level scripts to take priority. set -e; for python in $(PY3); do \ $$python setup.py install --force --root=$(CURDIR)/debian/tmp \ --no-compile -O0 --install-layout=deb; \ done override_dh_python2: dh_python2 -ppython-germinate germinate/debian/python-germinate.install0000644000000000000000000000002212326425437016025 0ustar usr/lib/python2.* germinate/debian/copyright0000644000000000000000000000254412326425437013111 0ustar Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Upstream-Name: germinate Upstream-Contact: Source: https://launchpad.net/germinate Comment: Written by Scott James Remnant and Colin Watson . Files: * Copyright: 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, Canonical Ltd. License: GPL-2+ Files: germinate/scripts/germinate_update_metapackage.py Copyright: 2004, 2005, 2006, 2007, 2008, 2009, 2011, 2012, Canonical Ltd. 2006, Gustavo Franco License: GPL-2+ License: GPL-2+ Germinate is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. . Germinate is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. . You should have received a copy of the GNU General Public License along with Germinate; see the file COPYING. If not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA. . A copy of the GNU General Public License version 2 is available in /usr/share/common-licenses/GPL-2. germinate/debian/docs0000644000000000000000000000000712326425437012021 0ustar README germinate/debian/source/0000755000000000000000000000000012326425437012451 5ustar germinate/debian/source/format0000644000000000000000000000001512326425437013660 0ustar 3.0 (native) germinate/debian/germinate.install0000644000000000000000000000004612326425437014514 0ustar usr/bin usr/share/man usr/share/perl5 germinate/debian/compat0000644000000000000000000000000212326425437012347 0ustar 7 germinate/debian/python3-germinate.install0000644000000000000000000000002012326425437016106 0ustar usr/lib/python3 germinate/debian/control0000644000000000000000000000376712326425437012571 0ustar Source: germinate Section: utils Priority: optional Maintainer: Colin Watson Standards-Version: 3.9.5 Build-Depends: debhelper (>= 7.0.50~) Build-Depends-Indep: python (>= 2.6.6-3~), python-all, python3 (>= 3.1.2-8~), python3-all, python-setuptools, python3-setuptools, python-apt (>= 0.7.93.2~), python3-apt (>= 0.7.93.2~), python-unittest2 Vcs-Bzr: http://bazaar.launchpad.net/+branch/germinate Vcs-Browser: http://bazaar.launchpad.net/+branch/germinate X-Python-Version: >= 2.6 X-Python3-Version: >= 3.0 Package: germinate Architecture: all Depends: ${misc:Depends}, ${python3:Depends}, ${perl:Depends}, python3-germinate (= ${binary:Version}) Description: expand dependencies in a list of seed packages Germinate takes lists of seed packages and expands their dependencies to produce a full list of packages. This can be used for purposes such as managing the list of packages present in a derived distribution's archive or CD builds. Package: python-germinate Section: python Architecture: all Depends: ${misc:Depends}, ${python:Depends}, python-apt (>= 0.7.93.2~) Description: expand dependencies in seed packages (Python 2 interface) Germinate takes lists of seed packages and expands their dependencies to produce a full list of packages. This can be used for purposes such as managing the list of packages present in a derived distribution's archive or CD builds. . This package provides Python 2 modules used by Germinate, which may also be used directly. Package: python3-germinate Section: python Architecture: all Depends: ${misc:Depends}, ${python3:Depends}, python3-apt (>= 0.7.93.2~) Description: expand dependencies in seed packages (Python 3 interface) Germinate takes lists of seed packages and expands their dependencies to produce a full list of packages. This can be used for purposes such as managing the list of packages present in a derived distribution's archive or CD builds. . This package provides Python 3 modules used by Germinate, which may also be used directly. germinate/debian/changelog0000644000000000000000000012070212326425457013027 0ustar germinate (2.16.1) trusty; urgency=medium * Bump recursion limit to 3000; apparently utopic trips over the existing limit (LP: #1312478). -- Colin Watson Fri, 25 Apr 2014 10:30:22 +0100 germinate (2.16) unstable; urgency=medium * Quote slashes in suite names when constructing local tag file names (thanks, Eugene Paskevich). * Change default distribution to trusty. * Update kubuntu-meta example in germinate-update-metapackage(1). * Policy version 3.9.5: no changes required. -- Colin Watson Thu, 03 Apr 2014 15:40:02 +0100 germinate (2.15) unstable; urgency=medium * Use type and selector attributes of urllib.request.Request in Python >= 3.1 rather than the deprecated get_type and get_selector accessor methods, fixing a build failure with Python 3.4 which removed these methods entirely. -- Colin Watson Wed, 15 Jan 2014 00:46:59 +0000 germinate (2.14) unstable; urgency=low * Micro-optimise lesser seeds calculation in Germinator._promote_dependency. * Amend dh_germinate_metapackage(1) to note that this program must be run after dh_prep. * Tolerate missing reasons when writing output. This is possible when germinate is being run over multiple flavours for the same architecture, and packages are promoted differently between seeds depending on the flavour. We can't generate a correct reason without a heavy performance cost in this case, but we can at least not crash. * Remove a few unnecessary "if condition: return True; else: return False" patterns. * Support multiarch (build-)dependency qualifiers (:any, :native). -- Colin Watson Fri, 11 Oct 2013 20:00:19 +0100 germinate (2.13) unstable; urgency=low * Fix dh_germinate_clean(1) and dh_germinate_metapackage(1) to recommend syntax compatible with debhelper 8. * Policy version 3.9.4: - Add a Vcs-Browser field. * Change default distribution to saucy. -- Colin Watson Thu, 13 Jun 2013 12:50:22 +0100 germinate (2.12) unstable; urgency=low * Add dists option to distribution sections in germinate-update-metapackage configuration, making it more useful for released distributions. -- Colin Watson Wed, 28 Nov 2012 16:13:58 +0000 germinate (2.11) unstable; urgency=low * Use collections.defaultdict instead of manually initialising elements. * Simplify some sorted() calls, relying on the default iterator for mappings being .keys(). * Simplify code to clean up bzr cache, removing an unnecessary helper function. * Add support for reading seeds from relative filesystem paths (thanks, Sjoerd Simons; LP: #1010186). * Only allow dh_python2 to operate on python-germinate, to avoid chaos caused by shebang rewriting. -- Colin Watson Tue, 18 Sep 2012 10:12:41 +0100 germinate (2.10) unstable; urgency=low * Always open Packages files as UTF-8, regardless of the current locale. LP: #1025818. -- Steve Langasek Mon, 23 Jul 2012 11:33:43 +0100 germinate (2.9) unstable; urgency=low * Support both Python 2 and 3 directly rather than using 2to3. * Make various cosmetic changes to conform to PEP-8. * Switch the top-level scripts to Python 3 by default. (They can still be run manually with /usr/bin/python if need be.) * Update copyright dates. * Policy version 3.9.3: - Convert debian/copyright to copyright-format 1.0. -- Colin Watson Thu, 14 Jun 2012 15:30:05 +0100 germinate (2.8) unstable; urgency=low * Build-depend on python-unittest2 so that the test suite works with Python 2.6 (closes: #661608). -- Colin Watson Wed, 29 Feb 2012 12:20:40 +0000 germinate (2.7) unstable; urgency=low * Explicitly specify the encoding to io.open so that tests pass even in the C locale. -- Colin Watson Tue, 31 Jan 2012 19:40:47 +0000 germinate (2.6) unstable; urgency=low * Fix processing of multiple suites where there are different versions of a package with different Provides fields. * Fix SeedStructure.write_seed_text to handle UTF-8 text in seeds correctly. -- Colin Watson Tue, 31 Jan 2012 01:53:34 +0000 germinate (2.5) unstable; urgency=low * Stop fetching Suggests from Packages; we don't use it. * Build a cache of seed entries when analysing reverse-dependencies, so that we don't have to expensively call get_seed_entries once per package per seed (LP: #915569). * Convert to setuptools. * Add the beginnings of a test suite. * Make the main bodies of scripts into modules so that they can be tested more easily. * Remove private AtomicUTF8File class; just write all files as UTF-8. * Port to Python 3: - Use "raise Exception(value)" syntax rather than the old-style "raise Exception, value". - Use Python 3-style print functions. - Use a list comprehension rather than filter (which behaves differently in Python 3). - Make GraphCycleError a subclass of Exception rather than of StandardError. - Simplify stringiness tests in TagFile.__init__ and Seed.__init__ using basestring. - Implement rich comparison methods instead of __cmp__ for Seed and GerminatedSeed. (Seed needs all of them for interface-compatibility; GerminatedSeed only needs __eq__ and __ne__.) - Use Python 3 replacements for urllib, urllib2, and ConfigParser if available. - When decompressing tag files from the archive, explicitly treat everything as binary data. - In Python 3, decode seed data read from URLs as UTF-8. - Pass universal_newlines=True to subprocess.Popen to get Unicode output. - Make sure to close stdout of subprocess.Popen objects. - Use 2to3 to handle the few remaining 2/3 differences at build time. - Add a python3-germinate package. -- Colin Watson Tue, 17 Jan 2012 00:51:28 +0000 germinate (2.4) unstable; urgency=low * Make Kernel-Version lexically scoped from the point in the seed where it's encountered to either the end of the seed or the next Kernel-Version entry, whichever comes first. Previously, the set of Kernel-Version values allowed for a seed was the union of all Kernel-Version entries in the seed, which is not what we want in practice. -- Colin Watson Thu, 05 Jan 2012 17:48:31 +0000 germinate (2.3) unstable; urgency=low * Adjust AtomicFile to not rename the .new file into place if the context exited with an exception (thanks, Jeroen T. Vermeulen). * Cope with an Archive implementation returning Maintainer values that are already unicode. -- Colin Watson Mon, 12 Dec 2011 14:24:36 +0000 germinate (2.2) unstable; urgency=low * Fix Germinator._follow_recommends to prevent a crash while calculating reverse-dependencies (LP: #900404). -- Colin Watson Wed, 07 Dec 2011 01:48:38 +0000 germinate (2.1) unstable; urgency=low * Call superclass constructor in GerminateFormatter's constructor. * Build-depend on python-all. * Store instances of Seed as GerminatedSeed._raw_seed, rather than instances of BytesIO which are transient and won't compare the way we need them to. * Remove build-sources entries that also appear in sources outputs at output time rather than immediately, which has the effect of removing sources that appear in any seed in the structure rather than just those processed earlier. This fixes behaviour when processing multiple seed structures, but also seems to fit more closely with the original intent of build-sources as documented in README, which I think was broken all the way back in r70 when seed hierarchies were first allowed to fork. * Fix inclusion/exclusion of packages from seeds other than "extra". -- Colin Watson Wed, 07 Dec 2011 00:51:12 +0000 germinate (2.0) unstable; urgency=low * Make sure to always close files after finishing with them. Mostly this is done using the 'with' statement in Python 2.6, but pychecker gets unhappy with contextlib.closing so I carried on using traditional try/finally blocks in cases that would require that. * Remove all uses of os.system and os.popen, replacing them with uses of the better-designed subprocess module. * Remove all code supporting the germinate -i/--ipv6 option; this has been off by default since November 2004, the service behind this was discontinued in March 2007 (http://lists.debian.org/debian-ipv6/2007/02/msg00015.html), and germinate was never a great place to track this kind of thing anyway. * Convert all option parsing to optparse. Consolidate defaults into a new Germinate.defaults module. * Update copyright dates. * Move canonical source location from people.canonical.com to a hosted branch on Launchpad. * Slightly modernise use of dh_python2. * Forbid seed names containing slashes. * Eliminate most uses of list.sort() in favour of sorted(iterable). * When promoting dependencies from lesser seeds, remove them from the lesser seed lists at output time rather than immediately. This is mostly to make it easier to process multiple seed structures, but also fixes a long-standing bug where promoted dependencies were only removed from a single arbitrary lesser seed rather than from all possible ones. * Memoise the results of Germinator's _inner_seeds, _strictly_outer_seeds, and _outer_seeds methods. This saves nearly a third of germinate's runtime in common cases. * Write all output files atomically. * Change default distribution to precise. * Update kubuntu-meta example in germinate-update-metapackage(1). * Refer to versioned GPL file in debian/copyright. * Policy version 3.9.2: no changes required. * Massive API cleanup: - Move output-writing functions from the top-level germinate program into Germinator. - Redesign how Germinator gets Packages/Sources sections from the archive. This now works via an abstract interface, which should make it easier to plug in alternative archive sources (e.g. a database). - Move all apt_pkg interaction into library code. Germinator.__init__ now takes an architecture argument so that it can set APT::Architecture. - Turn open_seed into a Seed class, allowing it to be a context manager. - Move code pertaining to the structure of seeds into a SeedStructure class, simplifying the interface. - Make all module names lower-case, per PEP-8. Remove the separate Germinate.Archive.tagfile module; this is now in germinate.archive directly. Adjust build system and pychecker handling to support this. - Remove unnecessary logging helper functions. - Don't modify level names on the root logger simply as a result of importing germinate.germinator; move this into a function. - Prefix all private methods with an underscore. - Remove germinate.tsort from germinate's public API. - Convert all method names to the PEP-8 preferred style (method_name rather than methodName). - Introduce wrapper functions for the various uses of write_list and write_source_list, and make the underlying methods private. - Make most instance variables private by prefixing an underscore, adding a few accessor methods. - Convert build system to distutils, make the germinate Python package public, and create a new python-germinate binary package. - Improve the Seed class so that seeds can be read multiple times without having to redownload them, and so that they remember which branch they came from. - Don't modify inner seeds when processing outer ones; filter build-dependencies on output instead. - Don't plant or grow seeds that have already had functionally-identical versions planted or grown respectively. - Automatically convert string dists/components/mirrors/source_mirrors arguments to lists in TagFile constructor. - Make it possible for a single Germinator to process multiple seed structures, reusing the work done on common seeds. - Canonicalise mirrors (by appending '/' if necessary) in TagFile rather than in the main germinate program. - Handle the extra seed entirely within Germinator rather than modifying SeedStructure (which doesn't fit well with processing the same seed structure on multiple architectures). - Use module-level loggers. - Get rid of the custom PROGRESS log level. - Change germinate.archive to use logging rather than print. - Add docstrings for all public classes and methods, and tidy up a few existing ones per PEP-257. -- Colin Watson Sun, 04 Dec 2011 14:16:54 +0000 germinate (1.27) unstable; urgency=low [ Alexandros Frantzis ] * Change local tag file name format to permit multiple repositories on the same host (LP: #634831). -- Colin Watson Thu, 25 Aug 2011 16:32:12 +0100 germinate (1.26) unstable; urgency=low * Use 'bzr branch' rather than 'bzr get'; the latter is apparently deprecated in bzr 2.4. * Change default distribution to oneiric. -- Colin Watson Fri, 12 Aug 2011 13:25:22 +0100 germinate (1.25.1) unstable; urgency=low * Add a new option, --no-installer, allowing one to process seeds without considering debian-installer udebs in the output. -- Adam Conrad Thu, 21 Jul 2011 22:19:29 -0600 germinate (1.25) unstable; urgency=low * Only raise an exception from open_tag_files if an appropriate file is not found on any mirror (thanks, Tom Gall, Steve Langasek, and Barry Warsaw; LP: #717879). -- Colin Watson Tue, 05 Apr 2011 15:36:38 +0100 germinate (1.24) unstable; urgency=low * Always refresh copies of local Packages and Sources files (i.e. when using the file: scheme). -- Colin Watson Tue, 21 Dec 2010 14:32:07 +0000 germinate (1.23) unstable; urgency=low * Write out entire decompressed Packages/Sources files in one go, rather than line-by-line. This is a bit faster. * Convert to dh_python2. * Change default distribution to natty. * Update kubuntu-meta example in germinate-update-metapackage(1) (remove ia64 and sparc). -- Colin Watson Fri, 15 Oct 2010 22:09:07 +0100 germinate (1.22) unstable; urgency=low * Change default distribution to maverick. * Update kubuntu-meta example in germinate-update-metapackage(1) (remove lpia). -- Colin Watson Wed, 18 Aug 2010 13:18:27 +0100 germinate (1.21) unstable; urgency=low [ Julian Andres Klode ] * Port the code to the new python-apt API and require at least version 0.7.93.2 of python-apt (closes: #571744). [ Colin Watson ] * Convert to source format 3.0 (native). * Policy version 3.8.4: no changes required. -- Colin Watson Tue, 06 Apr 2010 11:24:24 +0100 germinate (1.20) unstable; urgency=low * Add 'make check' which runs pychecker if available, and make it pass. * Change default distribution to lucid. * Update kubuntu-meta example in germinate-update-metapackage(1) (add netbook; remove hppa; add armel). -- Colin Watson Thu, 26 Nov 2009 12:50:56 +0000 germinate (1.19) unstable; urgency=low [ Colin Watson ] * Fix interpretation of architecture-specific regex seed entries where the regex contains a character class (which looks a bit like an architecture specification in the wrong light). [ Loïc Minier ] * Document --source-mirror option in germinate(1). -- Colin Watson Wed, 21 Oct 2009 19:55:16 +0100 germinate (1.18) unstable; urgency=low * Make germinate-update-metapackage write out a metapackage-map file, which is useful for later determining the set of seeds and metapackages to build. * Add a debhelper addon, useful for building metapackages. -- Colin Watson Mon, 10 Aug 2009 23:36:23 +0100 germinate (1.17) unstable; urgency=low * Use people.canonical.com everywhere rather than people.ubuntu.com. * Change default distribution to karmic. * Convert to debhelper 7. I've taken some care to avoid use of the new override targets since Ubuntu hardy-jaunty don't have a debhelper that supports those. * Policy version 3.8.2: no changes required. -- Colin Watson Sun, 26 Jul 2009 15:38:01 +0100 germinate (1.16) unstable; urgency=low * Pass germinate-update-metapackage's list of components to debootstrap. * Replace internal wiki link in README with something world-readable. -- Colin Watson Wed, 17 Jun 2009 21:16:40 +0100 germinate (1.15) unstable; urgency=low * Fix handling of archives that only contain uncompressed index files (thanks to Chris Cheney for the report). -- Colin Watson Wed, 03 Jun 2009 17:34:09 +0100 germinate (1.14) unstable; urgency=low * If germinate-update-metapackage fails, just print a message on stderr and exit 1, rather than raising RuntimeError (LP: #323714). -- Colin Watson Fri, 06 Mar 2009 09:59:32 +0000 germinate (1.13) unstable; urgency=low * Document 'germinate -v' in its manual page. * Report stderr as well as stdout from debootstrap failures (thanks, Loïc Minier; LP: #331488). * Use 'key in dict' rather than 'dict.has_key(key)'; dict.has_key is deprecated in Python 2.6. -- Colin Watson Wed, 25 Feb 2009 16:33:13 +0000 germinate (1.12) unstable; urgency=low * Change default distribution to jaunty. -- Colin Watson Mon, 02 Feb 2009 10:52:21 +0000 germinate (1.11) unstable; urgency=low * Keep going if the blacklist file can't be downloaded. (This was always the intent, but foiled by an implementation error.) * Clarify the purpose of individual blacklist entries in seeds in germinate(1). * Render the description of special seed syntax in germinate(1) as a tag list, to make it easier to read. -- Colin Watson Tue, 30 Sep 2008 23:07:21 +0100 germinate (1.10) unstable; urgency=low * Increase the recursion limit to 2000 for now to avoid running into problems with very deep dependency chains following code changes in 1.9. Other solutions are possible; this is a quick temporary hack. -- Colin Watson Tue, 23 Sep 2008 22:24:46 +0100 germinate (1.9) unstable; urgency=low * Fix fallback from .bz2 files when OSError is raised. * Try promoting alternative dependencies before adding new packages to the output. While the first dependency in an |-ed list may be promoted from any lesser seed, doing this for all alternatives produces pathological output, so instead we only promote from "close-by" lesser seeds, i.e. those that list the current seed in a Task-Seeds header (LP: #271309). -- Colin Watson Fri, 19 Sep 2008 01:05:19 +0100 germinate (1.8) unstable; urgency=low [ Evan Dandrea ] * Fix -S example to use file:// where the rest of the documentation says you should (LP: #264471). [ Colin Watson ] * Prefer Packages.bz2 and Sources.bz2 to .gz if possible; if neither works, fall back to the uncompressed files. -- Colin Watson Thu, 11 Sep 2008 11:37:09 +0100 germinate (1.7) unstable; urgency=low * Add support for setting per-seed features, so that following Recommends can be disabled for some seeds but not others (LP: #254042). -- Colin Watson Mon, 01 Sep 2008 01:40:19 +0100 germinate (1.6) unstable; urgency=low * Adjust printing of helpful ssh error messages when bzr fails (and re-raising of SeedError) to avoid causing problems when update.cfg contains multiple entries in seed_base. -- Colin Watson Thu, 28 Aug 2008 22:47:40 +0100 germinate (1.5) unstable; urgency=low * Consider whether there are any moves as well as additions or removals when deciding whether to run dch. -- Colin Watson Fri, 18 Jul 2008 10:36:56 +0100 germinate (1.4) unstable; urgency=low [ François-Denis Gonthier ] * Display stdout from debootstrap on error. * --nodch shouldn't make germinate-update-metapackage display the 'No changes found' message. * Add a general description to germinate-update-metapackage's help output. * Add --output-directory to germinate-update-metapackage. -- Colin Watson Fri, 11 Jul 2008 15:36:10 +0100 germinate (1.3) unstable; urgency=low [ François-Denis Gonthier ] * Add --nodch option to germinate-update-metapackage (LP: #242374). [ Colin Watson ] * Follow Recommends if 'feature follow-recommends' is set in the seed STRUCTURE file. * Document seed headers in germinate(1) (suggestion from François-Denis Gonthier). -- Colin Watson Fri, 04 Jul 2008 19:47:13 +0100 germinate (1.2) unstable; urgency=low [ Nicolas Barcet ] * Generate a seedstructure.dot diagram of seed inheritance. [ Colin Watson ] * Ensure that /usr/sbin and /sbin are on PATH when running debootstrap (closes: #487706). Requires Python 2.4. -- Colin Watson Tue, 24 Jun 2008 15:40:08 +0100 germinate (1.1) unstable; urgency=low * Only raise SeedError from open_seed if all attempts to open the seed failed; otherwise using multiple seed sources breaks. -- Colin Watson Thu, 29 May 2008 16:27:37 +0100 germinate (1.0) unstable; urgency=low * Check whether versions in dependencies are satisfied (LP: #74514). * Exclude Essential packages from metapackage dependencies (LP: #42261). * Make metapackage changelog entries more concise by merging items for multiple architectures (LP: #217963). * Print a helpful error message if an ssh connection fails, on the assumption that it's often due to an incorrect username (LP: #99123). * Document fetching seeds from the local file system (closes: #363536). * Add an example of using --seed-packages. * Change default distribution to intrepid. Update kubuntu-meta example in germinate-update-metapackage(1). * With all reported Ubuntu bugs fixed and only one organisationally-tricky Debian bug left, I think it's about time to bump the version to 1.0. -- Colin Watson Fri, 09 May 2008 11:37:11 +0100 germinate (0.45) unstable; urgency=low * Add missing apt_pkg.InitSystem() calls (LP: #215625). -- Colin Watson Fri, 11 Apr 2008 12:19:47 +0100 germinate (0.44) unstable; urgency=low * Support multiple archives (LP: #182915). Patch mostly from Francois-Denis Gonthier, adjusted and extended by me. -- Colin Watson Thu, 10 Apr 2008 08:40:59 +0100 germinate (0.43) unstable; urgency=low * Fix crash when using --seed-packages option. -- Colin Watson Wed, 02 Apr 2008 06:52:24 +0100 germinate (0.42) unstable; urgency=low * Change Maintainer address back to cjwatson@ubuntu.com; I generally work on this package on work time. * germinate-update-metapackage honours new Task-Seeds and Task-Metapackage seed headers, replacing previous seed_map/* and metapackage_map/* entries in update.cfg. * Fix error message on failing to download non-bzr seeds. * Update kubuntu-meta example in germinate-update-metapackage(1). * Simplify germinate-update-metapackage changelog output when a package moves from depends to recommends or vice versa. * Remove obsolete dh_python call from debian/rules. -- Colin Watson Fri, 22 Feb 2008 17:54:28 +0000 germinate (0.41) unstable; urgency=low * Plant seeds in topologically-sorted order. The intent of this change is the same as that in 0.40, only this time it should actually work in the face of seeds that inherit from other seeds that are overridden in a different branch. * On the other hand, continue to use the last entry in the source STRUCTURE file as the supported seed. It may not be topologically last if there are seeds in included branches that are not inherited by supported. * Remove overridden entries from 'structure' output file. -- Colin Watson Mon, 18 Feb 2008 15:06:24 +0000 germinate (0.40) unstable; urgency=low * Fix ordering of seed planting in the event that a seed in an included branch is overridden by one with different inheritance. -- Colin Watson Thu, 14 Feb 2008 18:33:25 +0000 germinate (0.39) unstable; urgency=low [ Jonathan Riddell ] * Use 'bzr checkout --lightweight' to speed up seed checkouts. [ Colin Watson ] * Allow "metapackage_map/SEED" configuration file entries in update-metapackage, which can be used to generate output metapackages whose names do not match the generating seed names. -- Colin Watson Thu, 14 Feb 2008 14:38:56 +0000 germinate (0.38) unstable; urgency=low * Fix germination from multiple seed sources in bzr mode. * germinate-update-metapackage now automatically expands the seeds option for seed inheritance, so you no longer usually need to have separate seeds and output_seeds options in update.cfg. -- Colin Watson Tue, 12 Feb 2008 11:30:53 +0000 germinate (0.37) unstable; urgency=low * Output a .seedtext file for each seed, containing the verbatim text of the seed. -- Colin Watson Thu, 07 Feb 2008 11:30:39 +0000 germinate (0.36) unstable; urgency=low * Add special-case handling of supported+build-depends to germinate-pkg-diff. * Add -S/--seed-source, -s/--seed-dist, and -d/--dist options to germinate-pkg-diff. * Add support for germinating from multiple seed sources as well as multiple branches. This is needed to support "third-party" seed branches that are stored in a different location from branches they include. * Fix rescuing of packages from extra, broken in 0.32. -- Colin Watson Tue, 29 Jan 2008 21:53:30 +0000 germinate (0.35) unstable; urgency=low * Make germinate-update-metapackage only blacklist the metapackage being generated, not all metapackages. See LP #148075. * Change default distribution to hardy. * Vcs-Bzr is now an official field. * Fix 'structure' output file; records were being separated by the empty string rather than by newlines, which made it rather useless. * Add an 'include' facility to the STRUCTURE file to allow seeds from one branch to inherit from seeds in another branch. * Automatically compute the transitive closure of inheritance relationships in the STRUCTURE file, rather than requiring them to be written out by hand. In other words, if you have 'minimal: required' then you can simply write 'standard: minimal' rather than 'standard: required minimal'. This is a requirement for sane use of 'include', since it may not be straightforward to hardcode the inheritance relationships among seeds in another branch. * Unhardcode the 'supported' seed. The last seed in the STRUCTURE file is now used for build-depends calculations. * Policy version 3.7.3: no changes required. -- Colin Watson Thu, 24 Jan 2008 14:25:16 +0000 germinate (0.34) unstable; urgency=low * Follow Recommends in Section: metapackages as if they were Depends. * Ignore leading and trailing whitespace and blank lines in STRUCTURE files. * Allow comments (starting with #) in STRUCTURE files. -- Colin Watson Thu, 02 Aug 2007 12:05:40 +0100 germinate (0.33) unstable; urgency=low * Honour recommends in germinate-pkg-diff. -- Colin Watson Thu, 19 Jul 2007 09:07:16 +0100 germinate (0.32) unstable; urgency=low * Use dch -iU rather than dch -U. (I use DEBCHANGE_RELEASE_HEURISTIC=changelog so never noticed the problem.) * Fix germinate-pkg-diff --mode option (was incorrectly --mirror, contrary to its meaning and the documentation). * Make germinate-pkg-diff work again even if supported is not in its list of seeds. * germinate-pkg-diff now automatically adds inherited seeds as necessary. Thus, the default list of seeds is reduced to simply 'desktop'. * Download package index files to a temporary directory if they aren't going to be saved. * Clean up package index files downloaded for germinate-pkg-diff. * Add germinate-pkg-diff -a/--arch option. * Update copyright dates. * Add manual pages for germinate-pkg-diff and germinate-update-metapackage. -- Colin Watson Wed, 18 Jul 2007 14:00:50 +0100 germinate (0.31) unstable; urgency=low * Fix excluding packages from seeds other than extra. -- Colin Watson Fri, 13 Jul 2007 10:08:24 +0100 germinate (0.30) unstable; urgency=low * Generalise Extra-Include and Extra-Exclude to allow rescuing packages from any seed, not just extra. For example, Mobile-Include: *-dev will rescue all packages from sources that generate binaries named *-dev in the mobile output. -- Colin Watson Tue, 10 Jul 2007 17:43:30 +0100 germinate (0.29) unstable; urgency=low * Add Vcs-Bzr control field. * Use dch -U if available. * Document meaning of seed entries in parentheses. * Remove old arch-tag lines. * PEP-8 import ordering. * Add --version option to all programs. -- Colin Watson Fri, 06 Jul 2007 12:26:52 +0100 germinate (0.28) unstable; urgency=low * Try to fall back to alternative dependencies if one is blacklisted. * Honour blacklist entries even if they're explicitly seeded. * Don't apply blacklists (apart from the "supported" blacklist) to build-dependencies. * Be a little more verbose when blacklisting packages. * Allow "seed_map/SEED" configuration file entries in update-metapackage to have an output metapackage read its package list from multiple seeds (needed for splitting required and minimal in the Ubuntu seeds). -- Colin Watson Sat, 23 Jun 2007 20:56:11 +0100 germinate (0.27) unstable; urgency=low * Add support for negative architecture specifications, e.g. [!powerpc]. * Document % and [arch] seed syntaxes in germinate(1). * Add support for prefixing seed entries with "!" to enforce blacklisting packages from the containing seed and any of its inner seeds. For example, this allows a package to be in Ubuntu main but blacklisted from being shipped on CDs. * Change default distribution to gutsy. -- Colin Watson Fri, 01 Jun 2007 12:54:27 +0100 germinate (0.26) unstable; urgency=low * Change default non-bzr seed location to http://people.ubuntu.com/~ubuntu-archive/seeds/. -- Colin Watson Mon, 12 Mar 2007 12:01:14 +0000 germinate (0.25) unstable; urgency=low * Change default distribution to feisty. * Fix file descriptor leak in Germinator.parseStructure(). * Put the contents of the STRUCTURE seed file in a new "structure" output file. * Change default seed distribution to "ubuntu.feisty"; note new naming scheme. * Fetch seeds from http://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/ by default if the --bzr option is used. -- Colin Watson Tue, 12 Dec 2006 09:29:46 +0000 germinate (0.24) unstable; urgency=low * When processing whole-source seed entries, only include real packages. This avoids slews of warnings for binaries that only exist on other architectures, and avoids mistakenly picking up virtual packages. -- Colin Watson Tue, 3 Oct 2006 11:59:45 +0100 germinate (0.23) unstable; urgency=low * Fix recommends-related changelog entries from update-metapackage. -- Colin Watson Sat, 9 Sep 2006 18:44:37 +0100 germinate (0.22) unstable; urgency=low * Fix crash when promoting from lesser seeds. * Include recommends in main outputs (they should continue to appear in Task headers and such, just not in metapackage dependencies), and create a .seed-recommends output for just the seeded recommendations. -- Colin Watson Sat, 9 Sep 2006 15:28:57 +0100 germinate (0.21ubuntu1) edgy; urgency=low * added support for recommends in seeds via enclosing the packagename in '('..')'. E.g. * (pkgname) # this will produce a recommends -- Michael Vogt Thu, 7 Sep 2006 16:44:36 +0200 germinate (0.21) unstable; urgency=low * Add optional output_seeds setting (defaults to value of seeds) in update.cfg, to allow seeds to be germinated for inheritance purposes without being output. * Sort added and removed packages in metapackage changelogs. -- Colin Watson Mon, 7 Aug 2006 16:19:15 +0100 germinate (0.20) unstable; urgency=low * Consider sources from inner seeds while rescuing packages from extra (so e.g. in Ubuntu Extra-Include in supported should look at source packages from minimal etc. as well as from supported). * Rescue from sources in the build tree in a separate pass, and add rescued packages to the depends/build-depends lists rather than pretending that they're explicitly seeded. This means that rescued packages end up correctly in supported or supported+build-depends depending on whether their source packages were found while processing dependencies or build-dependencies. -- Colin Watson Wed, 5 Jul 2006 13:08:34 +0100 germinate (0.19) unstable; urgency=low * Allow shell-style globs in seeds wherever regular expressions are allowed. (Regular expressions are surrounded by /.../, making this unambiguous.) * Add support for Extra-Include and Extra-Exclude variables, which allow automatically rescuing packages that match certain patterns from extra (https://launchpad.net/distros/ubuntu/+spec/seed-cleanup). * Tweak logging priorities for messages about virtual seed entries. * Don't display debugging messages by default; add germinate -v/--verbose option to display them. * Use sets rather than lists where possible. This roughly halves the runtime for the Ubuntu seeds. * Add extras recursively: binaries generated by sources pulled in by dependencies or build-dependencies of extra binaries should themselves be extra. -- Colin Watson Tue, 4 Jul 2006 11:57:47 +0100 germinate (0.18) unstable; urgency=low * Convert to python-support. * Move python and python-support build-dependencies to Build-Depends-Indep. * Policy version 3.7.2. -- Colin Watson Sat, 1 Jul 2006 12:33:24 +0100 germinate (0.17) unstable; urgency=low * Fix d-i kernel version pruning to happen early enough to exclude seeded packages for the wrong kernel version (which can easily happen e.g. if regex seeds are used). * If no Kernel-Version variable is given, allow all udebs with Kernel-Version headers rather than none of them. * Change default distribution to edgy. -- Colin Watson Wed, 21 Jun 2006 18:10:40 +0100 germinate (0.16) unstable; urgency=low * Add --bzr option to automatically check out seeds from bzr. * Switch from urllib to urllib2. * Refactor duplicated seed-fetching code into new Germinate.seeds module. * Add optional seed_dist setting in update.cfg files, to allow setting a default seed distribution. * Add --bzr option to update-metapackage as well (mostly implementing https://launchpad.net/distros/ubuntu/+spec/ubuntu-meta-from-bzr). -- Colin Watson Fri, 9 Jun 2006 12:15:01 +0100 germinate (0.15) unstable; urgency=low * Source moved to bzr; update copyright file. * Add germinate-update-metapackage script to help manage updating ubuntu-meta et al (thanks, Gustavo Franco; closes: https://launchpad.net/bugs/37917). * Improve man page documentation of --seed-packages slightly (see #363536). -- Colin Watson Fri, 9 Jun 2006 01:04:21 +0100 germinate (0.14) unstable; urgency=low * Move Python modules to /usr/lib/germinate; .pyc/.pyo files are architecture-dependent. -- Colin Watson Sat, 13 May 2006 10:05:37 +0100 germinate (0.13) unstable; urgency=low * Build-depend on python (closes: #363040). -- Colin Watson Mon, 17 Apr 2006 11:44:13 +0100 germinate (0.12) unstable; urgency=low * Upload to Debian (closes: #360631). * Switch to my debian.org maintainer address. -- Colin Watson Mon, 3 Apr 2006 23:25:23 +0100 germinate (0.11) dapper; urgency=low * Fix plantSeed() backward incompatibility for *-meta/update scripts by defaulting seedrelease to None. -- Colin Watson Wed, 22 Feb 2006 15:56:35 +0000 germinate (0.10) dapper; urgency=low * Add seed distribution name to seed descriptions in the "Why" column. -- Colin Watson Tue, 21 Feb 2006 18:07:16 +0000 germinate (0.9) dapper; urgency=low * When processing the build tree, germinate is (correctly) careful not to promote packages from lesser seeds, since we still want to consider them (e.g.) part of ship even if they're build-dependencies of desktop. However, this sometimes caused us to select the wrong alternative from an or-ed build-dependency (e.g. redland Build-Depends: libmysqlclient15-dev | libmysqlclient12-dev | libmysqlclient10-dev and libmysqlclient15-dev was in the Ubuntu supported seed). To fix this, process the packages at that point anyway and just avoid promoting them. -- Colin Watson Mon, 20 Feb 2006 13:51:30 +0000 germinate (0.8) dapper; urgency=low * When identifying extras, check for binaries built from multiple source packages and skip any superseded binaries. * Change default for --components to main and restricted, since the Ubuntu seeds rely on packages from restricted in order to germinate correctly. -- Colin Watson Tue, 14 Feb 2006 12:11:24 +0000 germinate (0.7) dapper; urgency=low * Depend on python-apt (thanks, Jani Monoses). -- Colin Watson Tue, 29 Nov 2005 19:23:45 +0000 germinate (0.6) dapper; urgency=low * Add regular expression support in seeds. Weaken architecture specification error checking in order to be able to use [...] in regular expressions more easily. * Add support for including all the binaries from a given source package (using "%source") in seeds. * Add germinate --cleanup option, to prevent caching of Packages and Sources files. * Add germinate --source-mirror option, to allow Packages and Sources files to be on different mirrors (e.g. Ubuntu ports architectures). -- Colin Watson Wed, 16 Nov 2005 14:37:37 +0000 germinate (0.5) dapper; urgency=high * Stop accidentally including extra in the all and supported+build-depends outputs. -- Colin Watson Sun, 30 Oct 2005 12:35:47 -0500 germinate (0.4) dapper; urgency=low * Don't attempt to satisfy versioned dependencies of .debs using virtual packages, as the packaging toolchain doesn't support that. (udpkg/anna do support this for .udebs, and the installer relies on that.) * Change default distribution to dapper. -- Colin Watson Tue, 25 Oct 2005 12:14:07 +0100 germinate (0.3) breezy; urgency=low * Add a man page for germinate. * Explain in pkg-diff --help that you can supply a list of seeds to compare against as non-option arguments. * Update pkg-diff's default seeds to 'minimal standard desktop', to match the breezy seeds. * Add germinate --seed-packages option, to allow calculating dependencies of individual extra packages during a germinate run. * Policy version 3.6.2. No changes required. * Update GPL notices with the FSF's new address. -- Colin Watson Wed, 31 Aug 2005 18:47:20 +0100 germinate (0.2) breezy; urgency=low * Fix formatting of output lines with UTF-8 maintainer strings. * Really pick the first reason from the build-dependency tree to display in output files. * Record allowed d-i kernel versions on a per-seed basis, so that we can have some kernel versions supported for netboot installations only. * Force a trailing slash onto the end of any --seed-source argument. -- Colin Watson Thu, 26 May 2005 16:32:38 +0100 germinate (0.1) breezy; urgency=low * Initial release. -- Colin Watson Wed, 27 Apr 2005 20:24:53 +1000 germinate/setup.cfg0000644000000000000000000000002512326425437011545 0ustar [build] pod2man=True germinate/germinate/0000755000000000000000000000000012326425443011677 5ustar germinate/germinate/archive.py0000644000000000000000000002264212326425437013703 0ustar # -*- coding: UTF-8 -*- """Representations of archives for use by Germinate.""" # Copyright (c) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 # Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. from __future__ import print_function import sys import os try: from urllib.parse import quote from urllib.request import Request, urlopen except ImportError: from urllib import quote from urllib2 import Request, urlopen import tempfile import shutil import logging import codecs import io import apt_pkg __pychecker__ = 'no-reuseattr' _logger = logging.getLogger(__name__) def _progress(msg, *args, **kwargs): _logger.info(msg, *args, extra={'progress': True}, **kwargs) if sys.version >= '3': _string_types = str else: _string_types = basestring if sys.version >= '3.1': def get_request_type(req): return req.type def get_request_selector(req): return req.selector else: def get_request_type(req): return req.get_type() def get_request_selector(req): return req.get_selector() class IndexType: """Types of archive index files.""" PACKAGES = 1 SOURCES = 2 INSTALLER_PACKAGES = 3 class Archive: """An abstract representation of an archive for use by Germinate.""" def sections(self): """Yield a sequence of the index sections found in this archive. A section is an entry in an index file corresponding to a single binary or source package. Each yielded value should be an (IndexType, section) pair, where section is a dictionary mapping control file keys to their values. """ raise NotImplementedError class TagFile(Archive): """Fetch package lists from a Debian-format archive as apt tag files.""" def __init__(self, dists, components, arch, mirrors, source_mirrors=None, installer_packages=True, cleanup=False): """Create a representation of a Debian-format apt archive.""" if isinstance(dists, _string_types): dists = [dists] if isinstance(components, _string_types): components = [components] if isinstance(mirrors, _string_types): mirrors = [mirrors] if isinstance(source_mirrors, _string_types): source_mirrors = [source_mirrors] self._dists = dists self._components = components self._arch = arch self._mirrors = mirrors self._installer_packages = installer_packages if source_mirrors: self._source_mirrors = source_mirrors else: self._source_mirrors = mirrors self._cleanup = cleanup def _open_tag_files(self, mirrors, dirname, tagfile_type, dist, component, ftppath): def _open_tag_file(mirror, suffix): """Download an apt tag file if needed, then open it.""" if not mirror.endswith('/'): mirror += '/' url = (mirror + "dists/" + dist + "/" + component + "/" + ftppath + suffix) req = Request(url) filename = None if get_request_type(req) != "file": filename = "%s_%s_%s_%s" % (quote(mirror, safe=""), quote(dist, safe=""), component, tagfile_type) else: # Make a more or less dummy filename for local URLs. filename = os.path.split(get_request_selector(req))[0].replace( os.sep, "_") fullname = os.path.join(dirname, filename) if get_request_type(req) == "file": # Always refresh. TODO: we should use If-Modified-Since for # remote HTTP tag files. try: os.unlink(fullname) except OSError: pass if not os.path.exists(fullname): _progress("Downloading %s file ...", req.get_full_url()) compressed = os.path.join(dirname, filename + suffix) try: url_f = urlopen(req) try: with open(compressed, "wb") as compressed_f: compressed_f.write(url_f.read()) finally: url_f.close() # apt_pkg is weird and won't accept GzipFile if suffix: _progress("Decompressing %s file ...", req.get_full_url()) if suffix == ".gz": import gzip compressed_f = gzip.GzipFile(compressed) elif suffix == ".bz2": import bz2 compressed_f = bz2.BZ2File(compressed) else: raise RuntimeError("Unknown suffix '%s'" % suffix) # This can be simplified once we can require Python # 2.7, where gzip.GzipFile and bz2.BZ2File are # context managers. try: with open(fullname, "wb") as f: f.write(compressed_f.read()) f.flush() finally: compressed_f.close() finally: if suffix: try: os.unlink(compressed) except OSError: pass if sys.version_info[0] < 3: return codecs.open(fullname, 'r', 'UTF-8', 'replace') else: return io.open(fullname, mode='r', encoding='UTF-8', errors='replace') tag_files = [] for mirror in mirrors: tag_file = None for suffix in (".bz2", ".gz", ""): try: tag_file = _open_tag_file(mirror, suffix) tag_files.append(tag_file) break except (IOError, OSError): pass if len(tag_files) == 0: raise IOError("no %s files found" % tagfile_type) return tag_files def sections(self): """Yield a sequence of the index sections found in this archive. A section is an entry in an index file corresponding to a single binary or source package. Each yielded value is an (IndexType, section) pair, where section is a dictionary mapping control file keys to their values. """ if self._cleanup: dirname = tempfile.mkdtemp(prefix="germinate-") else: dirname = '.' for dist in self._dists: for component in self._components: packages = self._open_tag_files( self._mirrors, dirname, "Packages", dist, component, "binary-" + self._arch + "/Packages") for tag_file in packages: try: for section in apt_pkg.TagFile(tag_file): yield (IndexType.PACKAGES, section) finally: tag_file.close() sources = self._open_tag_files( self._source_mirrors, dirname, "Sources", dist, component, "source/Sources") for tag_file in sources: try: for section in apt_pkg.TagFile(tag_file): yield (IndexType.SOURCES, section) finally: tag_file.close() instpackages = "" if self._installer_packages: try: instpackages = self._open_tag_files( self._mirrors, dirname, "InstallerPackages", dist, component, "debian-installer/binary-" + self._arch + "/Packages") except IOError: # can live without these _progress("Missing installer Packages file for %s " "(ignoring)", component) else: for tag_file in instpackages: try: for section in apt_pkg.TagFile(tag_file): yield (IndexType.INSTALLER_PACKAGES, section) finally: tag_file.close() if self._cleanup: shutil.rmtree(dirname) germinate/germinate/germinator.py0000644000000000000000000022360612326425443014431 0ustar # -*- coding: UTF-8 -*- """Expand seeds into dependency-closed lists of packages.""" # Copyright (c) 2004, 2005, 2006, 2007, 2008, 2009, 2011 Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. from __future__ import print_function import sys import re import fnmatch import logging from collections import defaultdict, MutableMapping import apt_pkg from germinate.archive import IndexType from germinate.seeds import AtomicFile, SeedStructure, _ensure_unicode # TODO: would be much more elegant to reduce our recursion depth! sys.setrecursionlimit(3000) __all__ = [ 'Germinator', ] _logger = logging.getLogger(__name__) try: apt_pkg.parse_src_depends("dummy:any", False) _apt_pkg_multiarch = True except TypeError: _apt_pkg_multiarch = False def _progress(msg, *args, **kwargs): _logger.info(msg, *args, extra={'progress': True}, **kwargs) class SeedReason(object): def __init__(self, branch, name): self._branch = branch self._name = name def __str__(self): if self._branch is None: return '%s seed' % self._name.title() else: return '%s %s seed' % (self._branch.title(), self._name) class BuildDependsReason(object): def __init__(self, src): self._src = src def __str__(self): return '%s (Build-Depend)' % self._src class RecommendsReason(object): def __init__(self, pkg): self._pkg = pkg def __str__(self): return '%s (Recommends)' % self._pkg class DependsReason(object): def __init__(self, pkg): self._pkg = pkg def __str__(self): return self._pkg class ExtraReason(object): def __init__(self, src): self._src = src def __str__(self): return 'Generated by %s' % self._src class RescueReason(object): def __init__(self, src): self._src = src def __str__(self): return 'Rescued from %s' % self._src class SeedKernelVersions(object): """A virtual seed entry representing lexically scoped Kernel-Version.""" def __init__(self, kernel_versions): self.kernel_versions = set(kernel_versions) class GerminatedSeed(object): def __init__(self, germinator, name, structure, raw_seed): self._germinator = germinator self._name = name self._structure = structure self._raw_seed = raw_seed self._copy = None self._entries = [] self._features = set() self._recommends_entries = [] self._close_seeds = set() self._depends = set() self._build_depends = set() self._sourcepkgs = set() self._build_sourcepkgs = set() self._pkgprovides = defaultdict(set) self._build = set() self._not_build = set() self._build_srcs = set() self._not_build_srcs = set() self._reasons = {} self._blacklist = set() self._blacklist_seen = False # Note that this relates to the vestigial global blacklist file, not # to the per-seed blacklist entries in _blacklist. self._blacklisted = set() self._includes = defaultdict(list) self._excludes = defaultdict(list) self._seed_reason = SeedReason(structure.branch, name) self._grown = False self._cache_inner_seeds = None self._cache_strictly_outer_seeds = None self._cache_outer_seeds = None def copy_plant(self, structure): """Return a copy of this seed attached to a different structure. At this point, we only copy the parts of this seed that were filled in by planting; the copy may be aborted if it transpires that it needs to be grown independently after all. The copy may be completed after all seeds have been planted by calling copy_grow.""" new = GerminatedSeed(self._germinator, self._name, structure, self._raw_seed) new._copy = self # We deliberately don't take copies of anything; deep copies would # take up substantial amounts of memory. new._entries = self._entries new._features = self._features new._recommends_entries = self._recommends_entries new._close_seeds = self._close_seeds new._blacklist = self._blacklist new._includes = self._includes new._excludes = self._excludes new._seed_reason = SeedReason(structure.branch, self._name) return new def copy_growth(self): """Complete copying of this seed.""" if self._copy is None: return # Does this seed still look the same as the one it was copied from # after we've finished planting it? if self != self._copy: self._copy = None return copy = self._copy assert copy._grown # We deliberately don't take copies of anything; this seed has been # grown and thus should not be modified further, and deep copies # would take up substantial amounts of memory. self._entries = copy._entries self._recommends_entries = copy._recommends_entries self._depends = copy._depends self._build_depends = copy._build_depends self._sourcepkgs = copy._sourcepkgs self._build_sourcepkgs = copy._build_sourcepkgs self._build = copy._build self._not_build = copy._not_build self._build_srcs = copy._build_srcs self._not_build_srcs = copy._not_build_srcs self._reasons = copy._reasons self._blacklist_seen = False self._blacklisted = copy._blacklisted self._grown = True @property def name(self): return self._name @property def structure(self): return self._structure def __str__(self): return self._name @property def entries(self): return [ e for e in self._entries if not isinstance(e, SeedKernelVersions)] @property def recommends_entries(self): return [ e for e in self._recommends_entries if not isinstance(e, SeedKernelVersions)] @property def depends(self): return set(self._depends) @property def build_depends(self): return set(self._build_depends) def __eq__(self, other): def eq_blacklist_seen(left_name, right_name): # Ignore KeyError in the following; if seeds haven't been # planted yet, they can't have seen blacklist entries from outer # seeds. try: left_seed = self._germinator._seeds[left_name] if left_seed._blacklist_seen: return False except KeyError: pass try: right_seed = other._germinator._seeds[right_name] if right_seed._blacklist_seen: return False except KeyError: pass return True def eq_inheritance(left_name, right_name): if left_name == "extra": left_inherit = self.structure.names + ["extra"] else: left_inherit = self.structure.inner_seeds(left_name) if right_name == "extra": right_inherit = other.structure.names + ["extra"] else: right_inherit = other.structure.inner_seeds(right_name) if len(left_inherit) != len(right_inherit): return False left_branch = self.structure.branch right_branch = other.structure.branch for left, right in zip(left_inherit, right_inherit): if left != right: return False left_seedname = self._germinator._make_seed_name( left_branch, left) right_seedname = other._germinator._make_seed_name( right_branch, right) if not eq_blacklist_seen(left_seedname, right_seedname): return False return True if isinstance(other, GerminatedSeed): if self._raw_seed != other._raw_seed: return False if not eq_inheritance(self.name, other.name): return False try: left_lesser = self._germinator._strictly_outer_seeds(self) right_lesser = other._germinator._strictly_outer_seeds(other) left_close = [l for l in left_lesser if self.name in l._close_seeds] right_close = [l for l in right_lesser if other.name in l._close_seeds] if left_close != right_close: return False left_branch = self.structure.branch right_branch = other.structure.branch for close_seed in left_close: left_seedname = self._germinator._make_seed_name( left_branch, close_seed) right_seedname = self._germinator._make_seed_name( right_branch, close_seed) if not eq_inheritance(left_seedname, right_seedname): return False except KeyError: pass return True else: return NotImplemented def __ne__(self, other): return not self == other __hash__ = None class GerminatedSeedStructure(object): def __init__(self, structure): self._structure = structure # TODO: move to collections.OrderedDict with 2.7 self._seednames = [] self._all = set() self._all_srcs = set() self._all_reasons = {} self._blacklist = {} self._rdepends_cache_entries = None class GerminatorOutput(MutableMapping, object): def __init__(self): self._dict = {} def __iter__(self): return iter(self._dict) def __len__(self): return len(self._dict) def __getitem__(self, key): if isinstance(key, SeedStructure): return self._dict[key.branch] else: return self._dict[key] def __setitem__(self, key, value): if isinstance(key, SeedStructure): self._dict[key.branch] = value else: self._dict[key] = value def __delitem__(self, key): if isinstance(key, SeedStructure): del self._dict[key.branch] else: del self._dict[key] class Germinator(object): """A dependency expander.""" # Initialisation. # --------------- def __init__(self, arch): """Create a dependency expander. Each instance of this class can only process a single architecture, but can process multiple seed structures against it. """ self._arch = arch apt_pkg.config.set("APT::Architecture", self._arch) # Global hints file. self._hints = {} # Parsed representation of the archive. self._packages = {} self._packagetype = {} self._provides = {} self._sources = {} # All the seeds we know about, regardless of seed structure. self._seeds = {} # The current Kernel-Version value for the seed currently being # processed. This just saves us passing a lot of extra method # arguments around. self._di_kernel_versions = None # Results of germination for each seed structure. self._output = GerminatorOutput() # Parsing. # -------- def parse_hints(self, f): """Parse a hints file.""" for line in f: if line.startswith("#") or not len(line.rstrip()): continue words = line.rstrip().split(None) if len(words) != 2: continue self._hints[words[1]] = words[0] f.close() def _parse_depends(self, value): """Parse Depends from value, without stripping qualifiers.""" if _apt_pkg_multiarch: return apt_pkg.parse_depends(value, False) else: return apt_pkg.parse_depends(value) def _parse_package(self, section, pkgtype): """Parse a section from a Packages file.""" pkg = section["Package"] ver = section["Version"] # If we have already seen an equal or newer version of this package, # then skip this section. if pkg in self._packages: last_ver = self._packages[pkg]["Version"] if apt_pkg.version_compare(last_ver, ver) >= 0: return self._packages[pkg] = {} self._packagetype[pkg] = pkgtype self._packages[pkg]["Section"] = \ section.get("Section", "").split('/')[-1] self._packages[pkg]["Version"] = ver self._packages[pkg]["Maintainer"] = \ _ensure_unicode(section.get("Maintainer", "")) self._packages[pkg]["Essential"] = section.get("Essential", "") for field in "Pre-Depends", "Depends", "Recommends": value = section.get(field, "") self._packages[pkg][field] = self._parse_depends(value) for field in "Size", "Installed-Size": value = section.get(field, "0") self._packages[pkg][field] = int(value) src = section.get("Source", pkg) idx = src.find("(") if idx != -1: src = src[:idx].strip() self._packages[pkg]["Source"] = src self._packages[pkg]["Provides"] = apt_pkg.parse_depends( section.get("Provides", "")) if pkg in self._provides: self._provides[pkg].append(pkg) self._packages[pkg]["Multi-Arch"] = section.get("Multi-Arch", "none") self._packages[pkg]["Kernel-Version"] = section.get( "Kernel-Version", "") def _parse_src_depends(self, value): """Parse Build-Depends from value, without stripping qualifiers.""" if _apt_pkg_multiarch: return apt_pkg.parse_src_depends(value, False) else: return apt_pkg.parse_src_depends(value) def _parse_source(self, section): """Parse a section from a Sources file.""" src = section["Package"] ver = section["Version"] # If we have already seen an equal or newer version of this source, # then skip this section. if src in self._sources: last_ver = self._sources[src]["Version"] if apt_pkg.version_compare(last_ver, ver) >= 0: return self._sources[src] = {} self._sources[src]["Maintainer"] = \ _ensure_unicode(section.get("Maintainer", "")) self._sources[src]["Version"] = ver for field in "Build-Depends", "Build-Depends-Indep": value = section.get(field, "") self._sources[src][field] = self._parse_src_depends(value) binaries = apt_pkg.parse_depends(section.get("Binary", src)) self._sources[src]["Binaries"] = [b[0][0] for b in binaries] def parse_archive(self, archive): """Parse an archive. This must be called before planting any seeds. """ for indextype, section in archive.sections(): if indextype == IndexType.PACKAGES: self._parse_package(section, "deb") elif indextype == IndexType.SOURCES: self._parse_source(section) elif indextype == IndexType.INSTALLER_PACKAGES: self._parse_package(section, "udeb") else: raise ValueError("Unknown index type %d" % indextype) # Construct a more convenient representation of Provides fields. for pkg in sorted(self._packages): for prov in self._packages[pkg]["Provides"]: if prov[0][0] not in self._provides: self._provides[prov[0][0]] = [] if prov[0][0] in self._packages: self._provides[prov[0][0]].append(prov[0][0]) self._provides[prov[0][0]].append(pkg) def parse_blacklist(self, structure, f): """Parse a blacklist file, used to indicate unwanted packages.""" output = self._output[structure] name = '' for line in f: line = line.strip() if line.startswith('# blacklist: '): name = line[13:] elif not line or line.startswith('#'): continue else: output._blacklist[line] = name f.close() # Seed structure handling. We need to wrap a few methods. # -------------------------------------------------------- def _inner_seeds(self, seed): if seed._cache_inner_seeds is None: branch = seed.structure.branch if seed.name == "extra": seed._cache_inner_seeds = [ self._seeds[self._make_seed_name(branch, seedname)] for seedname in seed.structure.names + ["extra"]] else: seed._cache_inner_seeds = [ self._seeds[self._make_seed_name(branch, seedname)] for seedname in seed.structure.inner_seeds(seed.name)] return seed._cache_inner_seeds def _strictly_outer_seeds(self, seed): if seed._cache_strictly_outer_seeds is None: branch = seed.structure.branch ret = [] for seedname in seed.structure.strictly_outer_seeds(seed.name): ret.append(self._seeds[self._make_seed_name(branch, seedname)]) try: ret.append(self._seeds[self._make_seed_name(branch, "extra")]) except KeyError: pass seed._cache_strictly_outer_seeds = ret return seed._cache_strictly_outer_seeds def _outer_seeds(self, seed): if seed._cache_outer_seeds is None: branch = seed.structure.branch ret = [] for seedname in seed.structure.outer_seeds(seed.name): ret.append(self._seeds[self._make_seed_name(branch, seedname)]) try: ret.append(self._seeds[self._make_seed_name(branch, "extra")]) except KeyError: pass seed._cache_outer_seeds = ret return seed._cache_outer_seeds def _supported(self, seed): try: return self._get_seed(seed.structure, seed.structure.supported) except KeyError: return None # The main germination algorithm. # ------------------------------- def _filter_packages(self, packages, pattern): """Filter a list of packages, returning those that match the given pattern. The pattern may either be a shell-style glob, or (if surrounded by slashes) an extended regular expression.""" if pattern.startswith('/') and pattern.endswith('/'): patternre = re.compile(pattern[1:-1]) filtered = [p for p in packages if patternre.search(p) is not None] elif '*' in pattern or '?' in pattern or '[' in pattern: filtered = fnmatch.filter(packages, pattern) else: # optimisation for common case if pattern in packages: return [pattern] else: return [] return sorted(filtered) def _substitute_seed_vars(self, substvars, pkg): """Process substitution variables. These look like ${name} (e.g. "kernel-image-${Kernel-Version}"). The name is case-insensitive. Substitution variables are set with a line that looks like " * name: value [value ...]", values being whitespace-separated. A package containing substitution variables will be expanded into one package for each possible combination of values of those variables.""" pieces = re.split(r'(\${.*?})', pkg) substituted = [[]] for piece in pieces: if piece.startswith("${") and piece.endswith("}"): name = piece[2:-1].lower() if name in substvars: # Duplicate substituted once for each available substvar # expansion. newsubst = [] for value in substvars[name]: for substpieces in substituted: newsubstpieces = list(substpieces) newsubstpieces.append(value) newsubst.append(newsubstpieces) substituted = newsubst else: _logger.error("Undefined seed substvar: %s", name) else: for substpieces in substituted: substpieces.append(piece) substpkgs = [] for substpieces in substituted: substpkgs.append("".join(substpieces)) return substpkgs def _already_seeded(self, seed, pkg): """Has pkg already been seeded in this seed or in one from which we inherit?""" for innerseed in self._inner_seeds(seed): if (pkg in innerseed._entries or pkg in innerseed._recommends_entries): return True return False def _make_seed_name(self, branch, seedname): return '%s/%s' % (branch, seedname) def _plant_seed(self, structure, seedname, raw_seed): """Add a seed.""" seed = GerminatedSeed(self, seedname, structure, structure[seedname]) full_seedname = self._make_seed_name(structure.branch, seedname) for existing in self._seeds.values(): if seed == existing: _logger.info("Already planted seed %s" % seed) self._seeds[full_seedname] = existing.copy_plant(structure) self._output[structure]._seednames.append(seedname) return self._seeds[full_seedname] = seed self._output[structure]._seednames.append(seedname) seedpkgs = [] seedrecommends = [] substvars = {} for line in raw_seed: if line.lower().startswith('task-seeds:'): seed._close_seeds.update(line[11:].strip().split()) seed._close_seeds.discard(seedname) continue if not line.startswith(" * "): continue pkg = line[3:].strip() if pkg.find("#") != -1: pkg = pkg[:pkg.find("#")] colon = pkg.find(":") if colon != -1: # Special header name = pkg[:colon] name = name.lower() value = pkg[colon + 1:] values = value.strip().split() if name == "kernel-version": # Allows us to pick the right modules later _logger.warning("Allowing d-i kernel versions: %s", values) # Remember the point in the seed at which we saw this, # so that we can handle it correctly while expanding # dependencies. seedpkgs.append(SeedKernelVersions(values)) elif name == "feature": _logger.warning("Setting features {%s} for seed %s", ', '.join(values), seed) seed._features.update(values) elif name.endswith("-include"): included_seed = name[:-8] if (included_seed not in structure.names and included_seed != "extra"): _logger.error("Cannot include packages from unknown " "seed: %s", included_seed) else: _logger.warning("Including packages from %s: %s", included_seed, values) seed._includes[included_seed].extend(values) elif name.endswith("-exclude"): excluded_seed = name[:-8] if (excluded_seed not in structure.names and excluded_seed != "extra"): _logger.error("Cannot exclude packages from unknown " "seed: %s", excluded_seed) else: _logger.warning("Excluding packages from %s: %s", excluded_seed, values) seed._excludes[excluded_seed].extend(values) substvars[name] = values continue pkg = pkg.strip() if pkg.endswith("]"): archspec = [] startarchspec = pkg.rfind("[") if startarchspec != -1: archspec = pkg[startarchspec + 1:-1].split() pkg = pkg[:startarchspec - 1] posarch = [x for x in archspec if not x.startswith('!')] negarch = [x[1:] for x in archspec if x.startswith('!')] if self._arch in negarch: continue if posarch and self._arch not in posarch: continue pkg = pkg.split()[0] # a leading ! indicates a per-seed blacklist; never include this # package in the given seed or any of its inner seeds, no matter # what if pkg.startswith('!'): pkg = pkg[1:] is_blacklist = True else: is_blacklist = False # a (pkgname) indicates that this is a recommend # and not a depends if pkg.startswith('(') and pkg.endswith(')'): pkg = pkg[1:-1] pkgs = self._filter_packages(self._packages, pkg) if not pkgs: pkgs = [pkg] # virtual or expanded; check again later for pkg in pkgs: seedrecommends.extend(self._substitute_seed_vars( substvars, pkg)) if pkg.startswith('%'): pkg = pkg[1:] if pkg in self._sources: pkgs = [p for p in self._sources[pkg]["Binaries"] if p in self._packages] else: _logger.warning("Unknown source package: %s", pkg) pkgs = [] else: pkgs = self._filter_packages(self._packages, pkg) if not pkgs: pkgs = [pkg] # virtual or expanded; check again later if is_blacklist: for pkg in pkgs: _logger.info("Blacklisting %s from %s", pkg, seed) seed._blacklist.update(self._substitute_seed_vars( substvars, pkg)) else: for pkg in pkgs: seedpkgs.extend(self._substitute_seed_vars(substvars, pkg)) di_kernel_versions = None for pkg in seedpkgs: if isinstance(pkg, SeedKernelVersions): di_kernel_versions = pkg continue if pkg in self._hints and self._hints[pkg] != seed.name: _logger.warning("Taking the hint: %s", pkg) continue if pkg in self._packages: # Ordinary package if self._already_seeded(seed, pkg): _logger.warning("Duplicated seed: %s", pkg) elif self._is_pruned(di_kernel_versions, pkg): _logger.warning("Pruned %s from %s", pkg, seed) else: if pkg in seedrecommends: seed._recommends_entries.append(pkg) else: seed._entries.append(pkg) elif pkg in self._provides: # Virtual package, include everything msg = "Virtual %s package: %s" % (seed, pkg) for vpkg in self._provides[pkg]: if self._already_seeded(seed, vpkg): pass elif self._is_pruned(di_kernel_versions, vpkg): pass else: msg += "\n - %s" % vpkg if pkg in seedrecommends: seed._recommends_entries.append(vpkg) else: seed._entries.append(vpkg) _logger.info("%s", msg) else: # No idea _logger.error("Unknown %s package: %s", seed, pkg) for pkg in self._hints: if (self._hints[pkg] == seed.name and not self._already_seeded(seed, pkg)): if pkg in self._packages: if pkg in seedrecommends: seed._recommends_entries.append(pkg) else: seed._entries.append(pkg) else: _logger.error("Unknown hinted package: %s", pkg) def plant_seeds(self, structure, seeds=None): """Add all seeds found in a seed structure.""" if structure not in self._output: self._output[structure] = GerminatedSeedStructure(structure) if seeds is not None: structure.limit(seeds) for name in structure.names: with structure[name] as seed: self._plant_seed(structure, name, seed) def _is_pruned(self, di_kernel_versions, pkg): """Test whether pkg is for a forbidden d-i kernel version.""" if not di_kernel_versions or not di_kernel_versions.kernel_versions: return False kernvers = di_kernel_versions.kernel_versions kernver = self._packages[pkg]["Kernel-Version"] return kernver != "" and kernver not in kernvers def _weed_blacklist(self, pkgs, seed, build_tree, why): """Weed out blacklisted seed entries from a list.""" white = [] if build_tree: outerseeds = [self._supported(seed)] else: outerseeds = self._outer_seeds(seed) for pkg in pkgs: for outerseed in outerseeds: if outerseed is not None and pkg in outerseed._blacklist: _logger.error("Package %s blacklisted in %s but seeded in " "%s (%s)", pkg, outerseed, seed, why) seed._blacklist_seen = True break else: white.append(pkg) return white def grow(self, structure): """Grow the seeds.""" output = self._output[structure] for seedname in output._seednames: self._di_kernel_versions = None seed = self._get_seed(structure, seedname) if seed._copy: seed.copy_growth() if seed._grown: _logger.info("Already grown seed %s" % seed) # We still need to update a few structure-wide outputs. output._all.update(seed._build) output._all_srcs.update(seed._build_srcs) for pkg, (why, build_tree, recommends) in seed._reasons.items(): if isinstance(why, SeedReason): # Adjust this reason to refer to the correct branch. why = seed._seed_reason seed._reasons[pkg] = (why, build_tree, recommends) self._remember_why(output._all_reasons, pkg, why, build_tree, recommends) continue _progress("Resolving %s dependencies ...", seed) # Check for blacklisted seed entries. seed._entries = self._weed_blacklist( seed._entries, seed, False, seed._seed_reason) seed._recommends_entries = self._weed_blacklist( seed._recommends_entries, seed, False, seed._seed_reason) # Note that seedrecommends are not processed with # recommends=True; that is reserved for Recommends of packages, # not packages recommended by the seed. Changing this results in # less helpful output when a package is recommended by an inner # seed and required by an outer seed. # We go through _get_seed_entries/_get_seed_recommends_entries # here so that promoted dependencies are filtered out. for pkg in self._get_seed_entries(structure, seedname): if isinstance(pkg, SeedKernelVersions): self._di_kernel_versions = pkg else: self._add_package(seed, pkg, seed._seed_reason) for pkg in self._get_seed_recommends_entries(structure, seedname): if isinstance(pkg, SeedKernelVersions): self._di_kernel_versions = pkg else: self._add_package(seed, pkg, seed._seed_reason) self._di_kernel_versions = None for rescue_seedname in output._seednames: self._rescue_includes(structure, seed.name, rescue_seedname, build_tree=False) if rescue_seedname == seed.name: # only rescue from seeds up to and including the current # seed; later ones have not been grown break self._rescue_includes(structure, seed.name, "extra", build_tree=False) seed._grown = True try: supported = self._get_seed(structure, structure.supported) except KeyError: supported = None if supported is not None: self._rescue_includes(structure, supported.name, "extra", build_tree=True) def add_extras(self, structure): """Add packages generated by the sources but not in any seed.""" output = self._output[structure] seed = GerminatedSeed(self, "extra", structure, None) self._seeds[self._make_seed_name(structure.branch, "extra")] = seed output._seednames.append("extra") self._di_kernel_versions = None _progress("Identifying extras ...") found = True while found: found = False sorted_srcs = sorted(output._all_srcs) for srcname in sorted_srcs: for pkg in self._sources[srcname]["Binaries"]: if pkg not in self._packages: continue if self._packages[pkg]["Source"] != srcname: continue if pkg in output._all: continue if pkg in self._hints and self._hints[pkg] != "extra": _logger.warning("Taking the hint: %s", pkg) continue seed._entries.append(pkg) self._add_package(seed, pkg, ExtraReason(srcname), second_class=True) found = True def _allowed_dependency(self, pkg, depend, seed, build_depend): """Test whether a dependency arc is allowed. Return True if pkg is allowed to satisfy a (build-)dependency using depend within seed. Note that depend must be a real package. If seed is None, check whether the (build-)dependency is allowed within any seed. """ if ":" in depend: depname, depqual = depend.split(":", 1) else: depname = depend depqual = None if depname not in self._packages: _logger.warning("_allowed_dependency called with virtual package " "%s", depend) return False if (seed is not None and self._is_pruned(self._di_kernel_versions, depname)): return False depmultiarch = self._packages[depname]["Multi-Arch"] if depqual == "any" and depmultiarch != "allowed": return False if build_depend: if depqual not in (None, "any", "native"): return False if (depqual == "native" and depmultiarch not in ("none", "same", "allowed")): return False return self._packagetype[depname] == "deb" else: if depqual not in (None, "any"): return False if self._packagetype[pkg] == self._packagetype[depname]: # If both packages have a Kernel-Version field, they must # match. pkgkernver = self._packages[pkg]["Kernel-Version"] depkernver = self._packages[depname]["Kernel-Version"] if (pkgkernver != "" and depkernver != "" and pkgkernver != depkernver): return False return True else: return False def _allowed_virtual_dependency(self, pkg, deptype): """Test whether a virtual dependency type is allowed. Return True if pkg's dependency relationship type deptype may be satisfied by a virtual package. (Versioned dependencies may not be satisfied by virtual packages, unless pkg is a udeb.) """ if pkg in self._packagetype and self._packagetype[pkg] == "udeb": return True else: return deptype == "" def _check_versioned_dependency(self, depname, depver, deptype): """Test whether a versioned dependency can be satisfied.""" depname = depname.split(":", 1)[0] if depname not in self._packages: return False if deptype == "": return True ver = self._packages[depname]["Version"] compare = apt_pkg.version_compare(ver, depver) if deptype == "<=": return compare <= 0 elif deptype == ">=": return compare >= 0 elif deptype == "<": return compare < 0 elif deptype == ">": return compare > 0 elif deptype == "=": return compare == 0 elif deptype == "!=": return compare != 0 else: _logger.error("Unknown dependency comparator: %s" % deptype) return False def _unparse_dependency(self, depname, depver, deptype): """Return a string representation of a dependency.""" if deptype == "": return depname else: return "%s (%s %s)" % (depname, deptype, depver) def _follow_recommends(self, structure, seed=None): """Test whether we should follow Recommends for this seed.""" if seed is not None: if "follow-recommends" in seed._features: return True if "no-follow-recommends" in seed._features: return False return "follow-recommends" in structure.features def _add_reverse(self, pkg, field, rdep): """Add a reverse dependency entry.""" if "Reverse-Depends" not in self._packages[pkg]: self._packages[pkg]["Reverse-Depends"] = defaultdict(list) self._packages[pkg]["Reverse-Depends"][field].append(rdep) def reverse_depends(self, structure): """Calculate the reverse dependency relationships.""" output = self._output[structure] for pkg in output._all: fields = ["Pre-Depends", "Depends"] if (self._follow_recommends(structure) or self._packages[pkg]["Section"] == "metapackages"): fields.append("Recommends") for field in fields: for deplist in self._packages[pkg][field]: for dep in deplist: depname = dep[0].split(":", 1)[0] if depname in output._all and \ self._allowed_dependency(pkg, dep[0], None, False): self._add_reverse(depname, field, pkg) for src in output._all_srcs: for field in "Build-Depends", "Build-Depends-Indep": for deplist in self._sources[src][field]: for dep in deplist: depname = dep[0].split(":", 1)[0] if depname in output._all and \ self._allowed_dependency(src, dep[0], None, True): self._add_reverse(depname, field, src) for pkg in output._all: if "Reverse-Depends" not in self._packages[pkg]: continue fields = ["Pre-Depends", "Depends"] if (self._follow_recommends(structure) or self._packages[pkg]["Section"] == "metapackages"): fields.append("Recommends") fields.extend(["Build-Depends", "Build-Depends-Indep"]) for field in fields: if field not in self._packages[pkg]["Reverse-Depends"]: continue self._packages[pkg]["Reverse-Depends"][field].sort() def _already_satisfied(self, seed, pkg, depend, build_depend=False, with_build=False): """Test whether a dependency has already been satisfied.""" (depname, depver, deptype) = depend if (self._allowed_virtual_dependency(pkg, deptype) and depname in self._provides): trylist = [d for d in self._provides[depname] if d in self._packages and self._allowed_dependency(pkg, d, seed, build_depend)] elif (self._check_versioned_dependency(depname, depver, deptype) and self._allowed_dependency(pkg, depname, seed, build_depend)): trylist = [depname.split(":", 1)[0]] else: return False for trydep in trylist: if with_build: for innerseed in self._inner_seeds(seed): if trydep in innerseed._build: return True else: for innerseed in self._inner_seeds(seed): if trydep in innerseed._not_build: return True if (trydep in seed._entries or trydep in seed._recommends_entries): return True else: return False def _add_dependency(self, seed, pkg, dependlist, build_depend, second_class, build_tree, recommends): """Add a single dependency. Return True if a dependency was added, otherwise False. """ if build_tree and build_depend: why = BuildDependsReason(self._packages[pkg]["Source"]) elif recommends: why = RecommendsReason(pkg) else: why = DependsReason(pkg) dependlist = self._weed_blacklist(dependlist, seed, build_tree, why) if not dependlist: return False if build_tree: for dep in dependlist: seed._build_depends.add(dep) else: for dep in dependlist: seed._depends.add(dep) for dep in dependlist: self._add_package(seed, dep, why, build_tree, second_class, recommends) return True def _promote_dependency(self, seed, pkg, depend, close, build_depend, second_class, build_tree, recommends): """Try to satisfy a dependency by promoting from a lesser seed. If close is True, only "close-by" seeds (ones that generate the same task, as defined by Task-Seeds headers) are considered. Return True if a dependency was added, otherwise False. """ (depname, depver, deptype) = depend if (self._check_versioned_dependency(depname, depver, deptype) and self._allowed_dependency(pkg, depname, seed, build_depend)): trylist = [depname.split(":", 1)[0]] elif (self._allowed_virtual_dependency(pkg, deptype) and depname in self._provides): trylist = [d for d in self._provides[depname] if d in self._packages and self._allowed_dependency(pkg, d, seed, build_depend)] else: return False lesserseeds = self._strictly_outer_seeds(seed) if close: lesserseeds = [l for l in lesserseeds if seed.name in l._close_seeds] for trydep in trylist: for lesserseed in lesserseeds: if (trydep in lesserseed._entries or trydep in lesserseed._recommends_entries): # Has it already been promoted from this seed? already_promoted = False for innerseed in self._inner_seeds(lesserseed): if innerseed.name == lesserseed.name: continue if trydep in innerseed._depends: already_promoted = True break if already_promoted: continue if second_class: # "I'll get you next time, Gadget!" # When processing the build tree, we don't promote # packages from lesser seeds, since we still want to # consider them (e.g.) part of ship even if they're # build-dependencies of desktop. However, we do need # to process them now anyway, since otherwise we # might end up selecting the wrong alternative from # an or-ed build-dependency. pass else: # We want to remove trydep from lesserseed._entries # and lesserseed._recommends_entries, but we can't # because those might need to be copied for another # seed structure; the removal is handled on output # instead. Even so, it's still useful to log it # here. _logger.warning("Promoted %s from %s to %s to satisfy " "%s", trydep, lesserseed, seed, pkg) return self._add_dependency(seed, pkg, [trydep], build_depend, second_class, build_tree, recommends) return False def _new_dependency(self, seed, pkg, depend, build_depend, second_class, build_tree, recommends): """Try to satisfy a dependency by adding a new package. Return True if a dependency was added, otherwise False. """ (depname, depver, deptype) = depend if (self._check_versioned_dependency(depname, depver, deptype) and self._allowed_dependency(pkg, depname, seed, build_depend)): virtual = None elif (self._allowed_virtual_dependency(pkg, deptype) and depname in self._provides): virtual = depname else: if build_depend: desc = "build-dependency" elif recommends: desc = "recommendation" else: desc = "dependency" _logger.error("Unknown %s %s by %s", desc, self._unparse_dependency(depname, depver, deptype), pkg) return False dependlist = [depname.split(":", 1)[0]] if virtual is not None: reallist = [d for d in self._provides[virtual] if d in self._packages and self._allowed_dependency( pkg, d, seed, build_depend)] if len(reallist): depname = reallist[0] # If the depending package isn't a d-i kernel module but the # dependency is, then pick all the modules for other allowed # kernel versions too. if (self._packages[pkg]["Kernel-Version"] == "" and self._packages[depname]["Kernel-Version"] != ""): dependlist = [d for d in reallist if not self._di_kernel_versions or (self._packages[d]["Kernel-Version"] in self._di_kernel_versions)] else: dependlist = [depname] _logger.info("Chose %s out of %s to satisfy %s", ", ".join(dependlist), virtual, pkg) else: _logger.error("Nothing to choose out of %s to satisfy %s", virtual, pkg) return False return self._add_dependency(seed, pkg, dependlist, build_depend, second_class, build_tree, recommends) def _add_dependency_tree(self, seed, pkg, depends, build_depend=False, second_class=False, build_tree=False, recommends=False): """Add a package's dependency tree.""" if build_depend: build_tree = True if build_tree: second_class = True for deplist in depends: for dep in deplist: # TODO cjwatson 2008-07-02: At the moment this check will # catch an existing Recommends and we'll never get as far as # calling _remember_why with a dependency, so seed._reasons # will be a bit inaccurate. We may need another pass for # Recommends to fix this. if self._already_satisfied( seed, pkg, dep, build_depend, second_class): break else: firstdep = True for dep in deplist: if firstdep: # For the first (preferred) alternative, we may # consider promoting it from any lesser seed. close = False firstdep = False else: # Other alternatives are less favoured, and will # only be promoted from closely-allied seeds. close = True if self._promote_dependency(seed, pkg, dep, close, build_depend, second_class, build_tree, recommends): if len(deplist) > 1: _logger.info("Chose %s to satisfy %s", dep[0], pkg) break else: for dep in deplist: if self._new_dependency(seed, pkg, dep, build_depend, second_class, build_tree, recommends): if len(deplist) > 1: _logger.info("Chose %s to satisfy %s", dep[0], pkg) break else: if len(deplist) > 1: _logger.error("Nothing to choose to satisfy %s", pkg) def _remember_why(self, reasons, pkg, why, build_tree=False, recommends=False): """Remember why this package was added to the output for this seed.""" if pkg in reasons: old_why, old_build_tree, old_recommends = reasons[pkg] # Reasons from the dependency tree beat reasons from the # build-dependency tree; but pick the first of either type that # we see. Within either tree, dependencies beat recommendations. if not old_build_tree and build_tree: return if old_build_tree == build_tree: if not old_recommends or recommends: return reasons[pkg] = (why, build_tree, recommends) def _add_package(self, seed, pkg, why, second_class=False, build_tree=False, recommends=False): """Add a package and its dependency trees.""" if self._is_pruned(self._di_kernel_versions, pkg): _logger.warning("Pruned %s from %s", pkg, seed) return if build_tree: outerseeds = [self._supported(seed)] else: outerseeds = self._outer_seeds(seed) for outerseed in outerseeds: if outerseed is not None and pkg in outerseed._blacklist: _logger.error("Package %s blacklisted in %s but seeded in %s " "(%s)", pkg, outerseed, seed, why) seed._blacklist_seen = True return if build_tree: second_class = True output = self._output[seed.structure] if pkg not in output._all: output._all.add(pkg) for innerseed in self._inner_seeds(seed): if pkg in innerseed._build: break else: seed._build.add(pkg) if not build_tree: for innerseed in self._inner_seeds(seed): if pkg in innerseed._not_build: break else: seed._not_build.add(pkg) # Remember why the package was added to the output for this seed. # Also remember a reason for "all" too, so that an aggregated list # of all selected packages can be constructed easily. self._remember_why(seed._reasons, pkg, why, build_tree, recommends) self._remember_why(output._all_reasons, pkg, why, build_tree, recommends) for prov in self._packages[pkg]["Provides"]: seed._pkgprovides[prov[0][0]].add(pkg) self._add_dependency_tree(seed, pkg, self._packages[pkg]["Pre-Depends"], second_class=second_class, build_tree=build_tree) self._add_dependency_tree(seed, pkg, self._packages[pkg]["Depends"], second_class=second_class, build_tree=build_tree) if (self._follow_recommends(seed.structure, seed) or self._packages[pkg]["Section"] == "metapackages"): self._add_dependency_tree(seed, pkg, self._packages[pkg]["Recommends"], second_class=second_class, build_tree=build_tree, recommends=True) src = self._packages[pkg]["Source"] if src not in self._sources: _logger.error("Missing source package: %s (for %s)", src, pkg) return if second_class: for innerseed in self._inner_seeds(seed): if src in innerseed._build_srcs: return else: for innerseed in self._inner_seeds(seed): if src in innerseed._not_build_srcs: return if build_tree: seed._build_sourcepkgs.add(src) if src in output._blacklist: seed._blacklisted.add(src) else: seed._not_build_srcs.add(src) seed._sourcepkgs.add(src) output._all_srcs.add(src) seed._build_srcs.add(src) self._add_dependency_tree(seed, pkg, self._sources[src]["Build-Depends"], build_depend=True) self._add_dependency_tree(seed, pkg, self._sources[src]["Build-Depends-Indep"], build_depend=True) def _rescue_includes(self, structure, seedname, rescue_seedname, build_tree): """Rescue packages matching certain patterns from other seeds.""" output = self._output[structure] try: seed = self._get_seed(structure, seedname) except KeyError: return if (rescue_seedname not in structure.names and rescue_seedname != "extra"): return # Find all the source packages. rescue_srcs = set() if rescue_seedname == "extra": rescue_seeds = self._inner_seeds(seed) else: rescue_seeds = [self._get_seed(structure, rescue_seedname)] for one_rescue_seed in rescue_seeds: if build_tree: rescue_srcs |= one_rescue_seed._build_srcs else: rescue_srcs |= one_rescue_seed._not_build_srcs # For each source, add any binaries that match the include/exclude # patterns. for src in rescue_srcs: rescue = [p for p in self._sources[src]["Binaries"] if p in self._packages] included = set() if rescue_seedname in seed._includes: for include in seed._includes[rescue_seedname]: included |= set(self._filter_packages(rescue, include)) if rescue_seedname in seed._excludes: for exclude in seed._excludes[rescue_seedname]: included -= set(self._filter_packages(rescue, exclude)) for pkg in included: if pkg in output._all: continue for lesserseed in self._strictly_outer_seeds(seed): if pkg in lesserseed._entries: seed._entries.remove(pkg) _logger.warning("Promoted %s from %s to %s due to " "%s-Includes", pkg, lesserseed, seed, rescue_seedname.title()) break _logger.debug("Rescued %s from %s to %s", pkg, rescue_seedname, seed) if build_tree: seed._build_depends.add(pkg) else: seed._depends.add(pkg) self._add_package(seed, pkg, RescueReason(src), build_tree=build_tree) # Accessors. # ---------- def get_source(self, pkg): """Return the name of the source package that builds pkg.""" return self._packages[pkg]["Source"] def is_essential(self, pkg): """Test whether pkg is Essential.""" return self._packages[pkg].get("Essential", "no") == "yes" def _get_seed(self, structure, seedname): """Return the GerminatedSeed for this structure and seed name.""" full_seedname = self._make_seed_name(structure.branch, seedname) return self._seeds[full_seedname] def _get_seed_entries(self, structure, seedname): """Return the explicitly seeded entries for this seed. This includes virtual SeedKernelVersions entries. """ seed = self._get_seed(structure, seedname) output = set(seed._entries) for innerseed in self._inner_seeds(seed): if innerseed.name == seed.name: continue output -= innerseed._depends # Take care to preserve the original ordering. # Use a temporary variable to work around a pychecker bug. ret = [e for e in seed._entries if e in output] return ret def get_seed_entries(self, structure, seedname): """Return the explicitly seeded entries for this seed.""" # Use a temporary variable to work around a pychecker bug. ret = [ e for e in self._get_seed_entries(structure, seedname) if not isinstance(e, SeedKernelVersions)] return ret def _get_seed_recommends_entries(self, structure, seedname): """Return the explicitly seeded Recommends entries for this seed. This includes virtual SeedKernelVersions entries. """ seed = self._get_seed(structure, seedname) output = set(seed._recommends_entries) for innerseed in self._inner_seeds(seed): if innerseed.name == seed.name: continue output -= innerseed._depends # Take care to preserve the original ordering. # Use a temporary variable to work around a pychecker bug. ret = [e for e in seed._recommends_entries if e in output] return ret def get_seed_recommends_entries(self, structure, seedname): """Return the explicitly seeded Recommends entries for this seed.""" # Use a temporary variable to work around a pychecker bug. ret = [ e for e in self._get_seed_recommends_entries(structure, seedname) if not isinstance(e, SeedKernelVersions)] return ret def get_depends(self, structure, seedname): """Return the dependencies of this seed.""" return self._get_seed(structure, seedname).depends def get_full(self, structure, seedname): """Return the full (run-time) dependency expansion of this seed.""" seed = self._get_seed(structure, seedname) return (set(self.get_seed_entries(structure, seedname)) | set(self.get_seed_recommends_entries(structure, seedname)) | seed._depends) def get_build_depends(self, structure, seedname): """Return the build-dependencies of this seed.""" output = set(self._get_seed(structure, seedname)._build_depends) for outerseedname in structure.outer_seeds(seedname): output -= self.get_full(structure, outerseedname) return output def get_all(self, structure): """Return all the packages in this structure.""" return list(self._output[structure]._all) # Methods for writing output to files. # ------------------------------------ def _write_list(self, reasons, filename, pkgset): pkglist = sorted(pkgset) pkg_len = len("Package") src_len = len("Source") why_len = len("Why") mnt_len = len("Maintainer") for pkg in pkglist: _pkg_len = len(pkg) if _pkg_len > pkg_len: pkg_len = _pkg_len _src_len = len(self._packages[pkg]["Source"]) if _src_len > src_len: src_len = _src_len why = reasons[pkg][0] if pkg in reasons else "" _why_len = len(str(why)) if _why_len > why_len: why_len = _why_len _mnt_len = len(self._packages[pkg]["Maintainer"]) if _mnt_len > mnt_len: mnt_len = _mnt_len size = 0 installed_size = 0 with AtomicFile(filename) as f: print("%-*s | %-*s | %-*s | %-*s | %-15s | %-15s" % (pkg_len, "Package", src_len, "Source", why_len, "Why", mnt_len, "Maintainer", "Deb Size (B)", "Inst Size (KB)"), file=f) print(("-" * pkg_len) + "-+-" + ("-" * src_len) + "-+-" + ("-" * why_len) + "-+-" + ("-" * mnt_len) + "-+-" + ("-" * 15) + "-+-" + ("-" * 15) + "-", file=f) for pkg in pkglist: why = reasons[pkg][0] if pkg in reasons else "" size += self._packages[pkg]["Size"] installed_size += self._packages[pkg]["Installed-Size"] print("%-*s | %-*s | %-*s | %-*s | %15d | %15d" % (pkg_len, pkg, src_len, self._packages[pkg]["Source"], why_len, why, mnt_len, self._packages[pkg]["Maintainer"], self._packages[pkg]["Size"], self._packages[pkg]["Installed-Size"]), file=f) print(("-" * (pkg_len + src_len + why_len + mnt_len + 9)) + "-+-" + ("-" * 15) + "-+-" + ("-" * 15) + "-", file=f) print("%*s | %15d | %15d" % ((pkg_len + src_len + why_len + mnt_len + 9), "", size, installed_size), file=f) def _write_source_list(self, filename, srcset): srclist = sorted(srcset) src_len = len("Source") mnt_len = len("Maintainer") for src in srclist: _src_len = len(src) if _src_len > src_len: src_len = _src_len _mnt_len = len(self._sources[src]["Maintainer"]) if _mnt_len > mnt_len: mnt_len = _mnt_len with AtomicFile(filename) as f: fmt = "%-*s | %-*s" print(fmt % (src_len, "Source", mnt_len, "Maintainer"), file=f) print(("-" * src_len) + "-+-" + ("-" * mnt_len) + "-", file=f) for src in srclist: print(fmt % (src_len, src, mnt_len, self._sources[src]["Maintainer"]), file=f) def write_full_list(self, structure, filename, seedname): """Write the full (run-time) dependency expansion of this seed.""" seed = self._get_seed(structure, seedname) self._write_list(seed._reasons, filename, self.get_full(structure, seedname)) def write_seed_list(self, structure, filename, seedname): """Write the explicitly seeded entries for this seed.""" seed = self._get_seed(structure, seedname) self._write_list(seed._reasons, filename, self.get_seed_entries(structure, seedname)) def write_seed_recommends_list(self, structure, filename, seedname): """Write the explicitly seeded Recommends entries for this seed.""" seed = self._get_seed(structure, seedname) self._write_list(seed._reasons, filename, self.get_seed_recommends_entries(structure, seedname)) def write_depends_list(self, structure, filename, seedname): """Write the dependencies of this seed.""" seed = self._get_seed(structure, seedname) self._write_list(seed._reasons, filename, seed._depends) def write_build_depends_list(self, structure, filename, seedname): """Write the build-dependencies of this seed.""" seed = self._get_seed(structure, seedname) self._write_list(seed._reasons, filename, self.get_build_depends(structure, seedname)) def write_sources_list(self, structure, filename, seedname): """Write the source packages for this seed and its dependencies.""" seed = self._get_seed(structure, seedname) self._write_source_list(filename, seed._sourcepkgs) def write_build_sources_list(self, structure, filename, seedname): """Write the source packages for this seed's build-dependencies.""" seed = self._get_seed(structure, seedname) build_sourcepkgs = seed._build_sourcepkgs for buildseedname in structure.names: buildseed = self._get_seed(structure, buildseedname) build_sourcepkgs -= buildseed._sourcepkgs self._write_source_list(filename, build_sourcepkgs) def write_all_list(self, structure, filename): """Write all the packages in this structure.""" all_bins = set() for seedname in structure.names: all_bins |= self.get_full(structure, seedname) all_bins |= self.get_build_depends(structure, seedname) self._write_list(self._output[structure]._all_reasons, filename, all_bins) def write_all_source_list(self, structure, filename): """Write all the source packages for this structure.""" all_srcs = set() for seedname in structure.names: seed = self._get_seed(structure, seedname) all_srcs |= seed._sourcepkgs all_srcs |= seed._build_sourcepkgs self._write_source_list(filename, all_srcs) def write_supported_list(self, structure, filename): """Write the "supported+build-depends" list.""" sup_bins = set() for seedname in structure.names: if seedname == structure.supported: sup_bins |= self.get_full(structure, seedname) # Only include those build-dependencies that aren't already in # the dependency outputs for inner seeds of supported. This # allows supported+build-depends to be usable as an "everything # else" output. build_depends = set(self.get_build_depends(structure, seedname)) for innerseedname in structure.inner_seeds(structure.supported): build_depends -= self.get_full(structure, innerseedname) sup_bins |= build_depends self._write_list(self._output[structure]._all_reasons, filename, sup_bins) def write_supported_source_list(self, structure, filename): """Write the "supported+build-depends" sources list.""" sup_srcs = set() for seedname in structure.names: seed = self._get_seed(structure, seedname) if seedname == structure.supported: sup_srcs |= seed._sourcepkgs # Only include those build-dependencies that aren't already in # the dependency outputs for inner seeds of supported. This # allows supported+build-depends to be usable as an "everything # else" output. build_sourcepkgs = set(seed._build_sourcepkgs) for innerseed in self._inner_seeds(self._supported(seed)): build_sourcepkgs -= innerseed._sourcepkgs sup_srcs |= build_sourcepkgs self._write_source_list(filename, sup_srcs) def write_all_extra_list(self, structure, filename): """Write the "all+extra" list.""" output = self._output[structure] self._write_list(output._all_reasons, filename, output._all) def write_all_extra_source_list(self, structure, filename): """Write the "all+extra" sources list.""" output = self._output[structure] self._write_source_list(filename, output._all_srcs) def write_rdepend_list(self, structure, filename, pkg): """Write a detailed analysis of reverse-dependencies.""" # First, build a cache of entries for each seed. output = self._output[structure] if output._rdepends_cache_entries is None: cache_entries = {} for seedname in output._seednames: cache_entries[seedname] = self.get_seed_entries( structure, seedname) output._rdepends_cache_entries = cache_entries # Then write out the list itself. with AtomicFile(filename) as f: print(pkg, file=f) self._write_rdepend_list(structure, f, pkg, "", done=set()) def _write_rdepend_list(self, structure, f, pkg, prefix, stack=None, done=None): if stack is None: stack = [] else: stack = list(stack) if pkg in stack: print(prefix + "! loop", file=f) return stack.append(pkg) if done is None: done = set() elif pkg in done: print(prefix + "! skipped", file=f) return done.add(pkg) output = self._output[structure] cache_entries = output._rdepends_cache_entries for seedname in output._seednames: if pkg in cache_entries[seedname]: print(prefix + "*", seedname.title(), "seed", file=f) if "Reverse-Depends" not in self._packages[pkg]: return for field in ("Pre-Depends", "Depends", "Recommends", "Build-Depends", "Build-Depends-Indep"): if field not in self._packages[pkg]["Reverse-Depends"]: continue i = 0 print(prefix + "*", "Reverse", field + ":", file=f) for dep in self._packages[pkg]["Reverse-Depends"][field]: i += 1 print(prefix + " +- " + dep, file=f) if field.startswith("Build-"): continue if i == len(self._packages[pkg]["Reverse-Depends"][field]): extra = " " else: extra = " | " self._write_rdepend_list(structure, f, dep, prefix + extra, stack, done) def write_provides_list(self, structure, filename): """Write a summary of which packages satisfied Provides.""" output = self._output[structure] with AtomicFile(filename) as f: all_pkgprovides = defaultdict(set) for seedname in output._seednames: seed = self._get_seed(structure, seedname) for prov, provset in seed._pkgprovides.items(): all_pkgprovides[prov].update(provset) for prov in sorted(all_pkgprovides): print(prov, file=f) for pkg in sorted(all_pkgprovides[prov]): print("\t%s" % (pkg,), file=f) print(file=f) def write_blacklisted(self, structure, filename): """Write the list of blacklisted packages we encountered.""" output = self._output[structure] with AtomicFile(filename) as fh: all_blacklisted = set() for seedname in output._seednames: seed = self._get_seed(structure, seedname) all_blacklisted.update(seed._blacklisted) for pkg in sorted(all_blacklisted): blacklist = output._blacklist[pkg] fh.write('%s\t%s\n' % (pkg, blacklist)) germinate/germinate/seeds.py0000644000000000000000000004524612326425437013372 0ustar # -*- coding: UTF-8 -*- """Fetch seeds from a URL collection or from bzr.""" # Copyright (c) 2004, 2005, 2006, 2008, 2009, 2011, 2012 Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. from __future__ import print_function import sys import os import tempfile import atexit import logging try: from urllib.parse import urljoin, urlparse as _urlparse from urllib.request import Request, URLError, urlopen except ImportError: from urlparse import urljoin, urlparse as _urlparse from urllib2 import Request, URLError, urlopen import shutil import re import subprocess import codecs import io import collections import germinate.defaults from germinate.tsort import topo_sort # pychecker gets confused by __next__ for Python 3 support. __pychecker__ = 'no-special' __all__ = [ 'SeedError', 'Seed', 'SeedStructure', ] _logger = logging.getLogger(__name__) _bzr_cache_dir = None if sys.version >= '3': _string_types = str _text_type = str else: _string_types = basestring _text_type = unicode class AtomicFile(object): """Facilitate atomic writing of files. Forces UTF-8 encoding.""" def __init__(self, filename): self.filename = filename if sys.version_info[0] < 3: self.fd = codecs.open( '%s.new' % self.filename, 'w', 'UTF-8', 'replace') else: # io.open is available from Python 2.6, but we only use it with # Python 3 because it raises exceptions when passed bytes. self.fd = io.open( '%s.new' % self.filename, mode='w', encoding='UTF-8', errors='replace') def __enter__(self): return self.fd def __exit__(self, exc_type, unused_exc_value, unused_exc_tb): self.fd.close() if exc_type is None: os.rename('%s.new' % self.filename, self.filename) # Not really necessary, but reduces pychecker confusion. def write(self, s): self.fd.write(s) class SeedError(RuntimeError): """An error opening or parsing a seed.""" pass def _ensure_unicode(s): if isinstance(s, _text_type): return s else: return _text_type(s, "utf8", "replace") class Seed(object): """A single seed from a collection.""" def _open_seed(self, base, branch, name, bzr=False): path = os.path.join(base, branch) if not path.endswith('/'): path += '/' if bzr: global _bzr_cache_dir if _bzr_cache_dir is None: _bzr_cache_dir = tempfile.mkdtemp(prefix='germinate-') atexit.register( shutil.rmtree, _bzr_cache_dir, ignore_errors=True) checkout = os.path.join(_bzr_cache_dir, branch) if not os.path.isdir(checkout): command = ['bzr'] # https://bugs.launchpad.net/bzr/+bug/39542 if path.startswith('http:'): command.append('branch') _logger.info("Fetching branch of %s", path) else: command.extend(['checkout', '--lightweight']) _logger.info("Checking out %s", path) command.extend([path, checkout]) status = subprocess.call(command) if status != 0: raise SeedError("Command failed with exit status %d:\n" " '%s'" % (status, ' '.join(command))) return open(os.path.join(checkout, name)) else: url = urljoin(path, name) if not _urlparse(url).scheme: fullpath = os.path.join(path, name) _logger.info("Using %s", fullpath) return open(fullpath) _logger.info("Downloading %s", url) req = Request(url) req.add_header('Cache-Control', 'no-cache') req.add_header('Pragma', 'no-cache') return urlopen(req) def __init__(self, bases, branches, name, bzr=False): """Read a seed from a collection.""" if isinstance(branches, _string_types): branches = [branches] self._name = name self._base = None self._branch = None self._file = None fd = None ssh_host = None for base in bases: for branch in branches: try: fd = self._open_seed(base, branch, name, bzr) self._base = base self._branch = branch break except SeedError: ssh_match = re.match( r'bzr\+ssh://(?:[^/]*?@)?(.*?)(?:/|$)', base) if ssh_match: ssh_host = ssh_match.group(1) except (OSError, IOError, URLError): pass if fd is not None: break if fd is None: if bzr: _logger.warning("Could not open %s from checkout of (any of):", name) for base in bases: for branch in branches: _logger.warning(' %s' % os.path.join(base, branch)) if ssh_host is not None: _logger.error("Do you need to set your user name on %s?", ssh_host) _logger.error("Try a section such as this in " "~/.ssh/config:") _logger.error("") _logger.error("Host %s", ssh_host) _logger.error(" User YOUR_USER_NAME") else: _logger.warning("Could not open (any of):") for base in bases: for branch in branches: path = os.path.join(base, branch) if not path.endswith('/'): path += '/' _logger.warning(' %s' % urljoin(path, name)) raise SeedError("Could not open %s" % name) try: self._text = fd.read() # In Python 3, we need to decode seed text read from URLs. if sys.version_info[0] >= 3 and isinstance(self._text, bytes): self._text = self._text.decode(errors="replace") finally: fd.close() def open(self): """Open a file object with the text of this seed.""" if sys.version_info[0] < 3: self._file = io.BytesIO(self._text) else: self._file = io.StringIO(self._text) return self._file def read(self, *args, **kwargs): """Read text from this seed.""" return self._file.read(*args, **kwargs) def readline(self, *args, **kwargs): """Read a line from this seed.""" return self._file.readline(*args, **kwargs) def readlines(self, *args, **kwargs): """Read a list of lines from this seed.""" return self._file.readlines(*args, **kwargs) def __next__(self): """Read the next line from this seed.""" return next(self._file) if sys.version < '3': next = __next__ def close(self): """Close the file object for this seed.""" self._file.close() def __enter__(self): """Open a seed context, returning a file object.""" return self.open() def __exit__(self, unused_exc_type, unused_exc_value, unused_exc_tb): """Close a seed context.""" self.close() @property def name(self): """The seed's name.""" return self._name @property def base(self): """The base URL where this seed was found.""" return self._base @property def branch(self): """The name of the branch containing this seed.""" return self._branch @property def text(self): """The text of this seed.""" return self._text def __lt__(self, other): if not isinstance(other, Seed): return NotImplemented return self.text < other.text def __le__(self, other): if not isinstance(other, Seed): return NotImplemented return self.text <= other.text def __eq__(self, other): if not isinstance(other, Seed): return NotImplemented return self.text == other.text def __ne__(self, other): if not isinstance(other, Seed): return NotImplemented return self.text != other.text def __ge__(self, other): if not isinstance(other, Seed): return NotImplemented return self.text >= other.text def __gt__(self, other): if not isinstance(other, Seed): return NotImplemented return self.text > other.text __hash__ = None class CustomSeed(Seed): """A seed created from custom input data.""" def __init__(self, name, entries): self._name = name self._base = None self._branch = None self._text = '\n'.join(entries) + '\n' class SingleSeedStructure(object): """A single seed collection structure file. The input data is an ordered sequence of lines as follows: SEED:[ INHERITED] INHERITED is a space-separated list of seeds from which SEED inherits. For example, "ship: base desktop" indicates that packages in the "ship" seed may depend on packages in the "base" or "desktop" seeds without requiring those packages to appear in the "ship" output. INHERITED may be empty. The lines should be topologically sorted with respect to inheritance, with inherited-from seeds at the start. Any line as follows: include BRANCH causes another seed branch to be included. Seed names will be resolved in included branches if they cannot be found in the current branch. This is for internal use; applications should use the SeedStructure class instead. """ def __init__(self, branch, f): """Parse a single seed structure file.""" self.seed_order = [] self.inherit = {} self.branches = [branch] self.lines = [] self.features = set() for line in f: line = line.strip() if not line: continue if line.startswith('#'): continue words = line.split() if words[0].endswith(':'): seed = words[0][:-1] if '/' in seed: raise SeedError( "seed name '%s' may not contain '/'" % seed) self.seed_order.append(seed) self.inherit[seed] = list(words[1:]) self.lines.append(line) elif words[0] == 'include': self.branches.extend(words[1:]) elif words[0] == 'feature': self.features.update(words[1:]) else: _logger.error("Unparseable seed structure entry: %s", line) class SeedStructure(collections.Mapping, object): """The full structure of a seed collection. This deals with acquiring the seed structure files and recursively acquiring any seed structure files it includes. """ def __init__(self, branch, seed_bases=None, bzr=False): """Open a seed collection and read all the seeds it contains.""" if seed_bases is None: if bzr: seed_bases = germinate.defaults.seeds_bzr else: seed_bases = germinate.defaults.seeds seed_bases = seed_bases.split(',') self._seed_bases = seed_bases self._branch = branch self._bzr = bzr self._features = set() self._seed_order, self._inherit, branches, self._lines = \ self._parse(self._branch, set()) self._seeds = {} for seed in self._seed_order: self._seeds[seed] = Seed(seed_bases, branches, seed, bzr=bzr) self._expand_inheritance() def _parse(self, branch, got_branches): all_seed_order = [] all_inherit = {} all_branches = [] all_structure = [] # Fetch this one with Seed(self._seed_bases, branch, "STRUCTURE", self._bzr) as seed: structure = SingleSeedStructure(branch, seed) got_branches.add(branch) # Recursively expand included branches for child_branch in structure.branches: if child_branch in got_branches: continue (child_seed_order, child_inherit, child_branches, child_structure) = self._parse(child_branch, got_branches) all_seed_order.extend(child_seed_order) all_inherit.update(child_inherit) for grandchild_branch in child_branches: if grandchild_branch not in all_branches: all_branches.append(grandchild_branch) for child_structure_line in child_structure: child_structure_name = child_structure_line.split()[0][:-1] for i in range(len(all_structure)): if (all_structure[i].split()[0][:-1] == child_structure_name): del all_structure[i] break all_structure.append(child_structure_line) # Attach the main branch's data to the end all_seed_order.extend(structure.seed_order) all_inherit.update(structure.inherit) for child_branch in structure.branches: if child_branch not in all_branches: all_branches.append(child_branch) for structure_line in structure.lines: structure_name = structure_line.split()[0][:-1] for i in range(len(all_structure)): if all_structure[i].split()[0][:-1] == structure_name: del all_structure[i] break all_structure.append(structure_line) self._features.update(structure.features) # We generally want to process branches in reverse order, so that # later branches can override seeds from earlier branches all_branches.reverse() return all_seed_order, all_inherit, all_branches, all_structure def _expand_inheritance(self): """Expand out incomplete inheritance lists.""" self._original_inherit = dict(self._inherit) self._names = topo_sort(self._inherit) for name in self._names: seen = set() new_inherit = [] for inheritee in self._inherit[name]: for expanded in self._inherit[inheritee]: if expanded not in seen: new_inherit.append(expanded) seen.add(expanded) if inheritee not in seen: new_inherit.append(inheritee) seen.add(inheritee) self._inherit[name] = new_inherit def limit(self, seeds): """Restrict the seeds we care about to this list.""" self._names = [] for name in seeds: for inherit in self._inherit[name]: if inherit not in self._names: self._names.append(inherit) if name not in self._names: self._names.append(name) def add(self, name, entries, parent): """Add a custom seed.""" self._names.append(name) self._inherit[name] = self._inherit[parent] + [parent] self._seeds[name] = CustomSeed(name, entries) def inner_seeds(self, seedname): """Return this seed and the seeds from which it inherits.""" innerseeds = list(self._inherit[seedname]) innerseeds.append(seedname) return innerseeds def strictly_outer_seeds(self, seedname): """Return the seeds that inherit from this seed.""" outerseeds = [] for seed in self._names: if seedname in self._inherit[seed]: outerseeds.append(seed) return outerseeds def outer_seeds(self, seedname): """Return this seed and the seeds that inherit from it.""" outerseeds = [seedname] outerseeds.extend(self.strictly_outer_seeds(seedname)) return outerseeds def __iter__(self): """Return an iterator over the seeds in this collection.""" return iter(self._seeds) def __len__(self): """Return the number of seeds in this collection.""" return len(self._seeds) def __getitem__(self, seedname): """Get a particular seed from this collection.""" return self._seeds[seedname] @property def branch(self): """The name of this seed collection branch.""" return self._branch @property def features(self): """The feature flags set for this seed collection.""" return set(self._features) @property def supported(self): """The name of the "supported" seed (the last one in the structure).""" return self._seed_order[-1] @property def names(self): """All the seed names in this collection.""" return list(self._names) def write(self, filename): """Write the text of the seed STRUCTURE file.""" with AtomicFile(filename) as f: for line in self._lines: print(_ensure_unicode(line), file=f) def write_dot(self, filename): """Write a dot file representing this structure.""" with AtomicFile(filename) as dotfile: print("digraph structure {", file=dotfile) print(" node [color=lightblue2, style=filled];", file=dotfile) for seed in self._seed_order: if seed not in self._original_inherit: continue for inherit in self._original_inherit[seed]: print(" \"%s\" -> \"%s\";" % (inherit, seed), file=dotfile) print("}", file=dotfile) def write_seed_text(self, filename, seedname): """Write the text of a seed in this collection.""" with AtomicFile(filename) as f: with self._seeds[seedname] as seed: for line in seed: print(_ensure_unicode(line.rstrip('\n')), file=f) germinate/germinate/version.py0000644000000000000000000000002612326425437013737 0ustar VERSION = '@VERSION@' germinate/germinate/defaults.py0000644000000000000000000000220612326425437014063 0ustar # -*- coding: UTF-8 -*- """Defaults for Germinate.""" # Copyright (c) 2011 Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. # Where do we get up-to-date seeds from? seeds = 'http://people.canonical.com/~ubuntu-archive/seeds/' seeds_bzr = 'http://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/' release = 'ubuntu.trusty' # If we need to download Packages.gz and/or Sources.gz, where do we get # them from? mirror = 'http://archive.ubuntu.com/ubuntu/' dist = 'trusty' arch = 'i386' germinate/germinate/tests/0000755000000000000000000000000012326425437013044 5ustar germinate/germinate/tests/helpers.py0000644000000000000000000001103212326425437015055 0ustar #! /usr/bin/env python """Testing helpers.""" # Copyright (C) 2012 Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. from __future__ import print_function import errno import io import os import shutil import sys import tempfile try: import unittest2 as unittest except ImportError: import unittest from germinate.seeds import SeedStructure if sys.version >= "3": def u(s): return s else: def u(s): return unicode(s, "unicode_escape") class TestCase(unittest.TestCase): def __init__(self, *args, **kwargs): unittest.TestCase.__init__(self, *args, **kwargs) self.temp_dir = None self.archive_dir = None self.seeds_dir = None def useTempDir(self): if self.temp_dir is not None: return self.temp_dir = tempfile.mkdtemp(prefix="germinate") self.addCleanup(shutil.rmtree, self.temp_dir) cwd = os.open(".", os.O_RDONLY | os.O_DIRECTORY) self.addCleanup(os.close, cwd) os.chdir(self.temp_dir) self.addCleanup(os.fchdir, cwd) def setUpDirs(self): if self.archive_dir is not None: return self.useTempDir() self.archive_dir = os.path.join(self.temp_dir, "archive") os.makedirs(self.archive_dir) self.seeds_dir = os.path.join(self.temp_dir, "seeds") os.makedirs(self.seeds_dir) def ensureDir(self, path): try: os.makedirs(path) except OSError as e: if e.errno != errno.EEXIST: raise def ensureParentDir(self, path): self.ensureDir(os.path.dirname(path)) def addSource(self, dist, component, src, ver, bins, fields={}): self.setUpDirs() compdir = os.path.join(self.archive_dir, "dists", dist, component) sources_path = os.path.join(compdir, "source", "Sources") self.ensureParentDir(sources_path) with open(sources_path, "a") as sources: print("Package: %s" % src, file=sources) print("Version: %s" % ver, file=sources) print("Binary: %s" % ", ".join(bins), file=sources) for key, value in fields.items(): print("%s: %s" % (key, value), file=sources) print(file=sources) def addPackage(self, dist, component, arch, pkg, ver, udeb=False, fields={}): self.setUpDirs() compdir = os.path.join(self.archive_dir, "dists", dist, component) if udeb: packages_path = os.path.join(compdir, "debian-installer", "binary-%s" % arch, "Packages") else: packages_path = os.path.join(compdir, "binary-%s" % arch, "Packages") self.ensureParentDir(packages_path) with open(packages_path, "a") as packages: print("Package: %s" % pkg, file=packages) print("Version: %s" % ver, file=packages) for key, value in fields.items(): print("%s: %s" % (key, value), file=packages) print(file=packages) def addStructureLine(self, seed_dist, line): self.setUpDirs() structure_path = os.path.join(self.seeds_dir, seed_dist, "STRUCTURE") self.ensureParentDir(structure_path) with open(structure_path, "a") as structure: print(line, file=structure) def addSeed(self, seed_dist, name, parents=[]): self.addStructureLine(seed_dist, "%s: %s" % (name, " ".join(parents))) def addSeedPackage(self, seed_dist, seed_name, pkg): self.setUpDirs() seed_path = os.path.join(self.seeds_dir, seed_dist, seed_name) self.ensureParentDir(seed_path) with io.open(seed_path, "a", encoding="UTF-8") as seed: print(u(" * %s") % pkg, file=seed) def openSeedStructure(self, branch): return SeedStructure(branch, seed_bases=["file://%s" % self.seeds_dir]) germinate/germinate/tests/test_germinator.py0000644000000000000000000003153112326425437016627 0ustar #! /usr/bin/env python """Unit tests for germinate.germinator.""" # Copyright (C) 2012 Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. import shutil from germinate.archive import TagFile from germinate.germinator import ( BuildDependsReason, DependsReason, ExtraReason, GerminatedSeed, Germinator, RecommendsReason, RescueReason, SeedReason, ) from germinate.tests.helpers import TestCase, u class TestSeedReason(TestCase): def test_str(self): """SeedReason stringifies to a description of the seed.""" reason = SeedReason("branch", "base") self.assertEqual("Branch base seed", str(reason)) class TestBuildDependsReason(TestCase): def test_str(self): """BuildDependsReason stringifies to a description of the reason.""" reason = BuildDependsReason("foo") self.assertEqual("foo (Build-Depend)", str(reason)) class TestRecommendsReason(TestCase): def test_str(self): """RecommendsReason stringifies to a description of the reason.""" reason = RecommendsReason("foo") self.assertEqual("foo (Recommends)", str(reason)) class TestDependsReason(TestCase): def test_str(self): """DependsReason stringifies to the package name.""" reason = DependsReason("foo") self.assertEqual("foo", str(reason)) class TestExtraReason(TestCase): def test_str(self): """ExtraReason stringifies to a description of the reason.""" reason = ExtraReason("foo") self.assertEqual("Generated by foo", str(reason)) class TestRescueReason(TestCase): def test_str(self): """RescueReason stringifies to a description of the reason.""" reason = RescueReason("foo") self.assertEqual("Rescued from foo", str(reason)) class TestGerminatedSeed(TestCase): def test_basic(self): """GerminatedSeed has reasonable basic properties.""" branch = "collection.dist" self.addSeed(branch, "one") self.addSeedPackage(branch, "one", "one-package") structure = self.openSeedStructure(branch) seed = GerminatedSeed(None, "one", structure, structure["one"]) self.assertEqual("one", seed.name) self.assertEqual(structure, seed.structure) self.assertEqual("one", str(seed)) self.assertEqual(structure["one"], seed._raw_seed) def test_equal_if_same_contents(self): """GerminatedSeeds with the same seeds and inheritance are equal.""" one = "one.dist" two = "two.dist" self.addSeed(one, "base") self.addSeedPackage(one, "base", "base") self.addSeed(one, "desktop", parents=["base"]) self.addSeedPackage(one, "desktop", "desktop") self.addSeed(two, "base") self.addSeedPackage(two, "base", "base") self.addSeed(two, "desktop", parents=["base"]) self.addSeedPackage(two, "desktop", "desktop") structure_one = self.openSeedStructure(one) structure_two = self.openSeedStructure(two) germinator = Germinator("i386") desktop_one = GerminatedSeed( germinator, "desktop", structure_one, structure_one["desktop"]) desktop_two = GerminatedSeed( germinator, "desktop", structure_two, structure_two["desktop"]) self.assertEqual(desktop_one, desktop_two) def test_not_equal_if_different_contents(self): """GerminatedSeeds with different seeds/inheritance are not equal.""" one = "one.dist" two = "two.dist" self.addSeed(one, "base") self.addSeedPackage(one, "base", "base") self.addSeed(one, "desktop", parents=["base"]) self.addSeedPackage(one, "desktop", "desktop") self.addSeed(two, "base") self.addSeedPackage(two, "base", "base") self.addSeed(two, "desktop") self.addSeedPackage(two, "desktop", "desktop") structure_one = self.openSeedStructure(one) structure_two = self.openSeedStructure(two) germinator = Germinator("i386") desktop_one = GerminatedSeed( germinator, "desktop", structure_one, structure_one["desktop"]) desktop_two = GerminatedSeed( germinator, "desktop", structure_two, structure_two["desktop"]) self.assertNotEqual(desktop_one, desktop_two) class TestGerminator(TestCase): def test_parse_archive(self): """Germinator.parse_archive successfully parses a simple archive.""" self.addSource("warty", "main", "hello", "1.0-1", ["hello", "hello-dependency"], fields={"Maintainer": "Test Person "}) self.addPackage("warty", "main", "i386", "hello", "1.0-1", fields={ "Maintainer": "Test Person ", "Depends": "hello-dependency", }) self.addPackage("warty", "main", "i386", "hello-dependency", "1.0-1", fields={"Source": "hello", "Multi-Arch": "foreign"}) self.addSeed("ubuntu.warty", "supported") self.addSeedPackage("ubuntu.warty", "supported", "hello") germinator = Germinator("i386") archive = TagFile( "warty", "main", "i386", "file://%s" % self.archive_dir) germinator.parse_archive(archive) self.assertIn("hello", germinator._sources) self.assertEqual({ "Maintainer": u("Test Person "), "Version": "1.0-1", "Build-Depends": [], "Build-Depends-Indep": [], "Binaries": ["hello", "hello-dependency"], }, germinator._sources["hello"]) self.assertIn("hello", germinator._packages) self.assertEqual({ "Section": "", "Version": "1.0-1", "Maintainer": u("Test Person "), "Essential": "", "Pre-Depends": [], "Depends": [[("hello-dependency", "", "")]], "Recommends": [], "Size": 0, "Installed-Size": 0, "Source": "hello", "Provides": [], "Kernel-Version": "", "Multi-Arch": "none", }, germinator._packages["hello"]) self.assertEqual("deb", germinator._packagetype["hello"]) self.assertIn("hello-dependency", germinator._packages) self.assertEqual({ "Section": "", "Version": "1.0-1", "Maintainer": u(""), "Essential": "", "Pre-Depends": [], "Depends": [], "Recommends": [], "Size": 0, "Installed-Size": 0, "Source": "hello", "Provides": [], "Kernel-Version": "", "Multi-Arch": "foreign", }, germinator._packages["hello-dependency"]) self.assertEqual("deb", germinator._packagetype["hello-dependency"]) self.assertEqual({}, germinator._provides) def test_different_providers_between_suites(self): """Provides from later versions override those from earlier ones.""" self.addSource("warty", "main", "hello", "1.0-1", ["hello"]) self.addPackage("warty", "main", "i386", "hello", "1.0-1", fields={"Provides": "goodbye"}) self.addSource("warty-updates", "main", "hello", "1.0-1.1", ["hello"]) self.addPackage("warty-updates", "main", "i386", "hello", "1.0-1.1", fields={"Provides": "hello-goodbye"}) germinator = Germinator("i386") archive = TagFile( ["warty", "warty-updates"], "main", "i386", "file://%s" % self.archive_dir) germinator.parse_archive(archive) self.assertNotIn("goodbye", germinator._provides) self.assertIn("hello-goodbye", germinator._provides) self.assertEqual(["hello"], germinator._provides["hello-goodbye"]) def test_depends_multiarch(self): """Compare Depends behaviour against the multiarch specification. https://wiki.ubuntu.com/MultiarchSpec """ for ma, qual, allowed in ( (None, "", True), (None, ":any", False), (None, ":native", False), ("same", "", True), ("same", ":any", False), ("same", ":native", False), ("foreign", "", True), ("foreign", ":any", False), ("foreign", ":native", False), ("allowed", "", True), ("allowed", ":any", True), ("allowed", ":native", False), ): self.addSource("precise", "main", "hello", "1.0-1", ["hello"]) self.addPackage("precise", "main", "i386", "hello", "1.0-1", fields={"Depends": "gettext%s" % qual}) self.addSource("precise", "main", "gettext", "0.18.1.1-5ubuntu3", ["gettext"]) package_fields = {} if ma is not None: package_fields["Multi-Arch"] = ma self.addPackage("precise", "main", "i386", "gettext", "0.18.1.1-5ubuntu3", fields=package_fields) branch = "collection.precise" self.addSeed(branch, "base") self.addSeedPackage(branch, "base", "hello") germinator = Germinator("i386") archive = TagFile( "precise", "main", "i386", "file://%s" % self.archive_dir) germinator.parse_archive(archive) structure = self.openSeedStructure(branch) germinator.plant_seeds(structure) germinator.grow(structure) expected = set() if allowed: expected.add("gettext") self.assertEqual( expected, germinator.get_depends(structure, "base"), "Depends: gettext%s on Multi-Arch: %s incorrectly %s" % ( qual, ma if ma else "none", "disallowed" if allowed else "allowed")) shutil.rmtree(self.archive_dir) shutil.rmtree(self.seeds_dir) def test_build_depends_multiarch(self): """Compare Build-Depends behaviour against the multiarch specification. https://wiki.ubuntu.com/MultiarchCross#Build_Dependencies """ for ma, qual, allowed in ( (None, "", True), (None, ":any", False), (None, ":native", True), ("same", "", True), ("same", ":any", False), ("same", ":native", True), ("foreign", "", True), ("foreign", ":any", False), ("foreign", ":native", False), ("allowed", "", True), ("allowed", ":any", True), ("allowed", ":native", True), ): self.addSource("precise", "main", "hello", "1.0-1", ["hello"], fields={"Build-Depends": "gettext%s" % qual}) self.addPackage("precise", "main", "i386", "hello", "1.0-1") self.addSource("precise", "main", "gettext", "0.18.1.1-5ubuntu3", ["gettext"]) package_fields = {} if ma is not None: package_fields["Multi-Arch"] = ma self.addPackage("precise", "main", "i386", "gettext", "0.18.1.1-5ubuntu3", fields=package_fields) branch = "collection.precise" self.addSeed(branch, "base") self.addSeedPackage(branch, "base", "hello") germinator = Germinator("i386") archive = TagFile( "precise", "main", "i386", "file://%s" % self.archive_dir) germinator.parse_archive(archive) structure = self.openSeedStructure(branch) germinator.plant_seeds(structure) germinator.grow(structure) expected = set() if allowed: expected.add("gettext") self.assertEqual( expected, germinator.get_build_depends(structure, "base"), "Build-Depends: gettext%s on Multi-Arch: %s incorrectly %s" % ( qual, ma if ma else "none", "disallowed" if allowed else "allowed")) shutil.rmtree(self.archive_dir) shutil.rmtree(self.seeds_dir) # TODO: Germinator needs many more unit tests. germinate/germinate/tests/test_integration.py0000644000000000000000000000573012326425437017005 0ustar #! /usr/bin/env python """Integration tests for germinate.""" # Copyright (C) 2011, 2012 Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. import logging from germinate.scripts import germinate_main from germinate.tests.helpers import TestCase try: NullHandler = logging.NullHandler except AttributeError: # < 2.7 class NullHandler(logging.Handler): def handle(self, record): pass def emit(self, record): pass def createLock(self): self.lock = None class TestGerminate(TestCase): def addNullHandler(self): handler = NullHandler() logger = logging.getLogger("germinate") logger.addHandler(handler) logger.propagate = False def runGerminate(self, *args): self.useTempDir() self.addNullHandler() argv = ["germinate"] argv.extend(["-S", "file://%s" % self.seeds_dir]) argv.extend(["-m", "file://%s" % self.archive_dir]) argv.extend(args) self.assertEqual(0, germinate_main.main(argv)) def parseOutput(self, output_name): output_dict = {} with open(output_name) as output: output.readline() output.readline() for line in output: if line.startswith("-"): break fields = [field.strip() for field in line.split("|")] output_dict[fields[0]] = fields[1:] return output_dict def test_trivial(self): self.addSource("warty", "main", "hello", "1.0-1", ["hello", "hello-dependency"]) self.addPackage("warty", "main", "i386", "hello", "1.0-1", fields={"Depends": "hello-dependency"}) self.addPackage("warty", "main", "i386", "hello-dependency", "1.0-1", fields={"Source": "hello"}) self.addSeed("ubuntu.warty", "supported") self.addSeedPackage("ubuntu.warty", "supported", "hello") self.runGerminate("-s", "ubuntu.warty", "-d", "warty", "-c", "main") supported = self.parseOutput("supported") self.assertTrue("hello" in supported) self.assertTrue("hello-dependency" in supported) all_ = self.parseOutput("supported") self.assertTrue("hello" in all_) self.assertTrue("hello-dependency" in all_) germinate/germinate/tests/test_archive.py0000644000000000000000000000704012326425437016077 0ustar #! /usr/bin/env python """Unit tests for germinate.archive.""" # Copyright (C) 2012 Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. import gzip import os import textwrap from germinate.archive import IndexType, TagFile from germinate.tests.helpers import TestCase class TestTagFile(TestCase): def test_init_lists(self): """TagFile may be constructed with list parameters.""" tagfile = TagFile( ["dist"], ["component"], "arch", ["mirror"], source_mirrors=["source_mirror"]) self.assertEqual(["dist"], tagfile._dists) self.assertEqual(["component"], tagfile._components) self.assertEqual(["mirror"], tagfile._mirrors) self.assertEqual(["source_mirror"], tagfile._source_mirrors) def test_init_strings(self): """TagFile may be constructed with string parameters.""" tagfile = TagFile( "dist", "component", "arch", "mirror", source_mirrors="source_mirror") self.assertEqual(["dist"], tagfile._dists) self.assertEqual(["component"], tagfile._components) self.assertEqual(["mirror"], tagfile._mirrors) self.assertEqual(["source_mirror"], tagfile._source_mirrors) def test_sections(self): """Test fetching sections from a basic TagFile archive.""" self.useTempDir() main_dir = os.path.join("mirror", "dists", "unstable", "main") binary_dir = os.path.join(main_dir, "binary-i386") source_dir = os.path.join(main_dir, "source") os.makedirs(binary_dir) os.makedirs(source_dir) packages = gzip.GzipFile(os.path.join(binary_dir, "Packages.gz"), "w") try: packages.write(textwrap.dedent(b"""\ Package: test Version: 1.0 Architecture: i386 Maintainer: \xc3\xba\xe1\xb8\x83\xc3\xba\xc3\xb1\xc5\xa7\xc5\xaf\x20\xc4\x91\xc9\x99\x76\xe1\xba\xbd\xc5\x82\xc3\xb5\xe1\xb9\x97\xc3\xa8\xc5\x97\xe1\xb9\xa1 """.decode("UTF-8")).encode("UTF-8")) finally: packages.close() sources = gzip.GzipFile(os.path.join(source_dir, "Sources.gz"), "w") try: sources.write(textwrap.dedent("""\ Source: test Version: 1.0 """).encode("UTF-8")) finally: sources.close() tagfile = TagFile( "unstable", "main", "i386", "file://%s/mirror" % self.temp_dir) sections = list(tagfile.sections()) self.assertEqual(IndexType.PACKAGES, sections[0][0]) self.assertEqual("test", sections[0][1]["Package"]) self.assertEqual("1.0", sections[0][1]["Version"]) self.assertEqual("i386", sections[0][1]["Architecture"]) self.assertEqual(IndexType.SOURCES, sections[1][0]) self.assertEqual("test", sections[1][1]["Source"]) self.assertEqual("1.0", sections[1][1]["Version"]) germinate/germinate/tests/run0000755000000000000000000000014712326425437013600 0ustar #! /bin/sh # Alternatively, run 'python setup.py test' from the top level. python -m unittest discover germinate/germinate/tests/test_seeds.py0000644000000000000000000002744412326425437015573 0ustar #! /usr/bin/env python """Unit tests for germinate.seeds.""" # Copyright (C) 2012 Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. import io import os import textwrap from germinate.seeds import ( AtomicFile, Seed, SingleSeedStructure, ) from germinate.tests.helpers import TestCase, u class TestAtomicFile(TestCase): def test_creates_file(self): """AtomicFile creates the named file with the requested contents.""" self.useTempDir() with AtomicFile("foo") as test: test.write("string") with open("foo") as handle: self.assertEqual("string", handle.read()) def test_removes_dot_new(self): """AtomicFile does not leave .new files lying around.""" self.useTempDir() with AtomicFile("foo"): pass self.assertFalse(os.path.exists("foo.new")) class TestSeed(TestCase): def setUp(self): self.addSeed("collection.dist", "test") self.addSeedPackage("collection.dist", "test", "foo") self.addSeed("collection.dist", "test2") self.addSeedPackage("collection.dist", "test2", "foo") self.addSeed("collection.dist", "test3") self.addSeedPackage("collection.dist", "test3", "bar") def test_init_no_bzr(self): """__init__ can open a seed from a collection without bzr.""" seed = Seed( ["file://%s" % self.seeds_dir], ["collection.dist"], "test") self.assertEqual("test", seed.name) self.assertEqual("file://%s" % self.seeds_dir, seed.base) self.assertEqual("collection.dist", seed.branch) self.assertEqual(" * foo\n", seed.text) def test_behaves_as_file(self): """A Seed context can be read from as a file object.""" seed = Seed( ["file://%s" % self.seeds_dir], ["collection.dist"], "test") with seed as seed_file: lines = list(seed_file) self.assertTrue(1, len(lines)) self.assertTrue(" * foo\n", lines[0]) def test_equal_if_same_contents(self): """Two Seed objects with the same text contents are equal.""" one = Seed( ["file://%s" % self.seeds_dir], ["collection.dist"], "test") two = Seed( ["file://%s" % self.seeds_dir], ["collection.dist"], "test2") self.assertEqual(one, two) def test_not_equal_if_different_contents(self): """Two Seed objects with different text contents are not equal.""" one = Seed( ["file://%s" % self.seeds_dir], ["collection.dist"], "test") three = Seed( ["file://%s" % self.seeds_dir], ["collection.dist"], "test3") self.assertNotEqual(one, three) def test_open_without_scheme(self): """A Seed can be opened from a relative path on the filesystem.""" seed = Seed([self.seeds_dir], ["collection.dist"], "test") with seed as seed_file: lines = list(seed_file) self.assertTrue(1, len(lines)) self.assertTrue(" * foo\n", lines[0]) class TestSingleSeedStructure(TestCase): def test_basic(self): """A SingleSeedStructure object has the correct basic properties.""" branch = "collection.dist" self.addSeed(branch, "base") self.addSeed(branch, "desktop", parents=["base"]) seed = Seed(["file://%s" % self.seeds_dir], branch, "STRUCTURE") with seed as seed_file: structure = SingleSeedStructure(branch, seed_file) self.assertEqual(["base", "desktop"], structure.seed_order) self.assertEqual({"base": [], "desktop": ["base"]}, structure.inherit) self.assertEqual([branch], structure.branches) self.assertEqual(["base:", "desktop: base"], structure.lines) self.assertEqual(set(), structure.features) def test_include(self): """SingleSeedStructure parses the "include" directive correctly.""" branch = "collection.dist" self.addStructureLine(branch, "include other.dist") seed = Seed(["file://%s" % self.seeds_dir], branch, "STRUCTURE") with seed as seed_file: structure = SingleSeedStructure(branch, seed_file) self.assertEqual([branch, "other.dist"], structure.branches) def test_feature(self): """SingleSeedStructure parses the "feature" directive correctly.""" branch = "collection.dist" self.addStructureLine(branch, "feature follow-recommends") seed = Seed(["file://%s" % self.seeds_dir], branch, "STRUCTURE") with seed as seed_file: structure = SingleSeedStructure(branch, seed_file) self.assertEqual(set(["follow-recommends"]), structure.features) class TestSeedStructure(TestCase): def test_basic(self): """A SeedStructure object has the correct basic properties.""" branch = "collection.dist" self.addSeed(branch, "base") self.addSeedPackage(branch, "base", "base-package") self.addSeed(branch, "desktop", parents=["base"]) self.addSeedPackage(branch, "desktop", "desktop-package") structure = self.openSeedStructure(branch) self.assertEqual(branch, structure.branch) self.assertEqual(set(), structure.features) self.assertEqual("desktop", structure.supported) self.assertEqual(["base", "desktop"], structure.names) self.assertEqual({"base": [], "desktop": ["base"]}, structure._inherit) def test_dict(self): """A SeedStructure can be treated as a dictionary of seeds.""" branch = "collection.dist" self.addSeed(branch, "base") self.addSeedPackage(branch, "base", "base-package") structure = self.openSeedStructure(branch) self.assertEqual(1, len(structure)) self.assertEqual(["base"], list(structure)) self.assertEqual("base", structure["base"].name) self.assertEqual(" * base-package\n", structure["base"].text) def test_multiple(self): """SeedStructure follows "include" links to other seed collections.""" one = "one.dist" two = "two.dist" self.addSeed(one, "base") self.addSeedPackage(one, "base", "base-package") self.addStructureLine(two, "include one.dist") self.addSeed(two, "desktop") self.addSeedPackage(two, "desktop", "desktop-package") structure = self.openSeedStructure(two) self.assertEqual(two, structure.branch) self.assertEqual(one, structure["base"].branch) self.assertEqual(" * base-package\n", structure["base"].text) self.assertEqual(two, structure["desktop"].branch) self.assertEqual(" * desktop-package\n", structure["desktop"].text) def test_later_branches_override_earlier_branches(self): """Seeds from later branches override seeds from earlier branches.""" one = "one.dist" two = "two.dist" self.addSeed(one, "base") self.addSeedPackage(one, "base", "base-package") self.addSeed(one, "desktop") self.addSeedPackage(one, "desktop", "desktop-package-one") self.addStructureLine(two, "include one.dist") self.addSeed(two, "desktop") self.addSeedPackage(two, "desktop", "desktop-package-two") structure = self.openSeedStructure(two) self.assertEqual(["base", "desktop"], sorted(structure)) self.assertEqual(" * desktop-package-two\n", structure["desktop"].text) def test_limit(self): """SeedStructure.limit restricts the set of seed names.""" branch = "collection.dist" self.addSeed(branch, "one") self.addSeedPackage(branch, "one", "one") self.addSeed(branch, "two", parents=["one"]) self.addSeedPackage(branch, "two", "two") self.addSeed(branch, "three") self.addSeedPackage(branch, "three", "three") self.addSeed(branch, "four") self.addSeedPackage(branch, "four", "four") structure = self.openSeedStructure(branch) self.assertEqual( sorted(["one", "two", "three", "four"]), sorted(structure.names)) structure.limit(["two", "three"]) self.assertEqual( sorted(["one", "two", "three"]), sorted(structure.names)) def test_add(self): """SeedStructure.add adds a custom seed.""" branch = "collection.dist" self.addSeed(branch, "base") self.addSeedPackage(branch, "base", "base") structure = self.openSeedStructure(branch) structure.add("custom", [" * custom-one", " * custom-two"], "base") self.assertIn("custom", structure) self.assertIn("custom", structure.names) self.assertIn("custom", structure._inherit) self.assertEqual(["base"], structure._inherit["custom"]) self.assertEqual("custom", structure["custom"].name) self.assertIsNone(structure["custom"].base) self.assertIsNone(structure["custom"].branch) self.assertEqual( " * custom-one\n * custom-two\n", structure["custom"].text) def test_write(self): """SeedStructure.write writes the text of STRUCTURE.""" branch = "collection.dist" self.addSeed(branch, "one") self.addSeedPackage(branch, "one", "one") self.addSeed(branch, "two", parents=["one"]) self.addSeedPackage(branch, "two", "two") structure = self.openSeedStructure(branch) structure.write("structure") with open("structure") as structure_file: self.assertEqual("one:\ntwo: one\n", structure_file.read()) def test_write_dot(self): """SeedStructure.write_dot writes an appropriate dot file.""" branch = "collection.dist" self.addSeed(branch, "one") self.addSeedPackage(branch, "one", "one") self.addSeed(branch, "two", parents=["one"]) self.addSeedPackage(branch, "two", "two") structure = self.openSeedStructure(branch) structure.write_dot("structure.dot") with open("structure.dot") as structure_dot_file: self.assertEqual(textwrap.dedent("""\ digraph structure { node [color=lightblue2, style=filled]; "one" -> "two"; } """), structure_dot_file.read()) def test_write_seed_text(self): """SeedStructure.write_seed_text writes the text of a seed.""" branch = "collection.dist" self.addSeed(branch, "one") self.addSeedPackage(branch, "one", "one-package") self.addSeed(branch, "two") self.addSeedPackage(branch, "two", "two-package") structure = self.openSeedStructure(branch) structure.write_seed_text("one.seedtext", "one") with open("one.seedtext") as seed_file: self.assertEqual(" * one-package\n", seed_file.read()) def test_write_seed_text_utf8(self): """SeedStructure.write_seed_text handles UTF-8 text in seeds.""" branch = "collection.dist" self.addSeed(branch, "base") self.addSeedPackage(branch, "base", u("base # \u00e4\u00f6\u00fc")) structure = self.openSeedStructure(branch) structure.write_seed_text("base.seedtext", "base") with io.open("base.seedtext", encoding="UTF-8") as seed_file: self.assertEqual( u(" * base # \u00e4\u00f6\u00fc\n"), seed_file.read()) germinate/germinate/tests/__init__.py0000644000000000000000000000000012326425437015143 0ustar germinate/germinate/log.py0000644000000000000000000000356612326425437013047 0ustar # -*- coding: UTF-8 -*- """Custom logging for Germinate.""" # Copyright (c) 2011, 2012 Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. import sys import logging class GerminateFormatter(logging.Formatter): """Format messages in Germinate's preferred concise style.""" def __init__(self): logging.Formatter.__init__(self) self.levels = { logging.DEBUG: ' ', logging.INFO: '* ', logging.WARNING: '! ', logging.ERROR: '? ', } def format(self, record): try: if record.progress: return record.getMessage() except AttributeError: pass try: return '%s%s' % (self.levels[record.levelno], record.getMessage()) except KeyError: return record.getMessage() def germinate_logging(level): """Configure logging as preferred by Germinate command-line tools.""" logging.basicConfig() logger = logging.getLogger('germinate') if not logger.handlers: logger.setLevel(level) handler = logging.StreamHandler(sys.stdout) handler.setFormatter(GerminateFormatter()) logger.addHandler(handler) logger.propagate = False germinate/germinate/scripts/0000755000000000000000000000000012326425437013371 5ustar germinate/germinate/scripts/germinate_update_metapackage.py0000644000000000000000000004252012326425437021605 0ustar # -*- coding: UTF-8 -*- # Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2011, 2012 Canonical Ltd. # Copyright (C) 2006 Gustavo Franco # # This file is part of Germinate. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. # TODO: # - Exclude essential packages from dependencies from __future__ import print_function import sys import re import os import optparse import logging try: # >= 3.0 from configparser import NoOptionError, NoSectionError if (sys.version_info[0] < 3 or (sys.version_info[0] == 3 and sys.version_info[1] < 2)): # < 3.2 from configparser import SafeConfigParser else: # >= 3.2 from configparser import ConfigParser as SafeConfigParser except ImportError: # < 3.0 from ConfigParser import NoOptionError, NoSectionError, SafeConfigParser import subprocess from collections import defaultdict from germinate.germinator import Germinator import germinate.archive from germinate.log import germinate_logging from germinate.seeds import SeedError, SeedStructure import germinate.version __pychecker__ = 'maxlocals=80' def error_exit(message): print("%s: %s" % (sys.argv[0], message), file=sys.stderr) sys.exit(1) def parse_options(argv): description = '''\ Update metapackage lists for distribution 'dist' as defined in update.cfg.''' parser = optparse.OptionParser( prog='germinate-update-metapackage', usage='%prog [options] [dist]', version='%prog ' + germinate.version.VERSION, description=description) parser.add_option('-o', '--output-directory', dest='outdir', default='.', metavar='DIR', help='output in specific directory') parser.add_option('--nodch', dest='nodch', action='store_true', default=False, help="don't modify debian/changelog") parser.add_option('--bzr', dest='bzr', action='store_true', default=False, help='fetch seeds using bzr (requires bzr to be ' 'installed)') return parser.parse_args(argv[1:]) def main(argv): options, args = parse_options(argv) if not os.path.exists('debian/control'): error_exit('must be run from the top level of a source package') this_source = None with open('debian/control') as control: for line in control: if line.startswith('Source:'): this_source = line[7:].strip() break elif line == '': break if this_source is None: error_exit('cannot find Source: in debian/control') if not this_source.endswith('-meta'): error_exit('source package name must be *-meta') metapackage = this_source[:-5] print("[info] Initialising %s-* package lists update..." % metapackage) config = SafeConfigParser() with open('update.cfg') as config_file: try: # >= 3.2 config.read_file(config_file) except AttributeError: # < 3.2 config.readfp(config_file) if len(args) > 0: dist = args[0] else: dist = config.get('DEFAULT', 'dist') seeds = config.get(dist, 'seeds').split() try: output_seeds = config.get(dist, 'output_seeds').split() except NoOptionError: output_seeds = list(seeds) architectures = config.get(dist, 'architectures').split() try: archive_base_default = config.get(dist, 'archive_base/default') archive_base_default = re.split(r'[, ]+', archive_base_default) except (NoSectionError, NoOptionError): archive_base_default = None archive_base = {} for arch in architectures: try: archive_base[arch] = config.get(dist, 'archive_base/%s' % arch) archive_base[arch] = re.split(r'[, ]+', archive_base[arch]) except (NoSectionError, NoOptionError): if archive_base_default is not None: archive_base[arch] = archive_base_default else: error_exit('no archive_base configured for %s' % arch) if options.bzr and config.has_option("%s/bzr" % dist, 'seed_base'): seed_base = config.get("%s/bzr" % dist, 'seed_base') else: seed_base = config.get(dist, 'seed_base') seed_base = re.split(r'[, ]+', seed_base) if options.bzr and config.has_option("%s/bzr" % dist, 'seed_dist'): seed_dist = config.get("%s/bzr" % dist, 'seed_dist') elif config.has_option(dist, 'seed_dist'): seed_dist = config.get(dist, 'seed_dist') else: seed_dist = dist if config.has_option(dist, 'dists'): dists = config.get(dist, 'dists').split() else: dists = [dist] components = config.get(dist, 'components').split() def seed_packages(germinator_method, structure, seed_name): if config.has_option(dist, "seed_map/%s" % seed_name): mapped_seeds = config.get(dist, "seed_map/%s" % seed_name).split() else: mapped_seeds = [] task_seeds_re = re.compile('^Task-Seeds:\s*(.*)', re.I) with structure[seed_name] as seed: for line in seed: task_seeds_match = task_seeds_re.match(line) if task_seeds_match is not None: mapped_seeds = task_seeds_match.group(1).split() break if seed_name not in mapped_seeds: mapped_seeds.append(seed_name) packages = [] for mapped_seed in mapped_seeds: packages.extend(germinator_method(structure, mapped_seed)) return packages def metapackage_name(structure, seed_name): if config.has_option(dist, "metapackage_map/%s" % seed_name): return config.get(dist, "metapackage_map/%s" % seed_name) else: task_meta_re = re.compile('^Task-Metapackage:\s*(.*)', re.I) with structure[seed_name] as seed: for line in seed: task_meta_match = task_meta_re.match(line) if task_meta_match is not None: return task_meta_match.group(1) return "%s-%s" % (metapackage, seed_name) debootstrap_version_file = 'debootstrap-version' def get_debootstrap_version(): version_cmd = subprocess.Popen( ['dpkg-query', '-W', '--showformat', '${Version}', 'debootstrap'], stdout=subprocess.PIPE, universal_newlines=True) version, _ = version_cmd.communicate() if not version: error_exit('debootstrap does not appear to be installed') return version def debootstrap_packages(arch): env = dict(os.environ) if 'PATH' in env: env['PATH'] = '/usr/sbin:/sbin:%s' % env['PATH'] else: env['PATH'] = '/usr/sbin:/sbin:/usr/bin:/bin' debootstrap = subprocess.Popen( ['debootstrap', '--arch', arch, '--components', ','.join(components), '--print-debs', dist, 'debootstrap-dir', archive_base[arch][0]], stdout=subprocess.PIPE, env=env, stderr=subprocess.PIPE, universal_newlines=True) (debootstrap_stdout, debootstrap_stderr) = debootstrap.communicate() if debootstrap.returncode != 0: error_exit('Unable to retrieve package list from debootstrap; ' 'stdout: %s\nstderr: %s' % (debootstrap_stdout, debootstrap_stderr)) # sometimes debootstrap gives empty packages / multiple separators packages = [pkg for pkg in debootstrap_stdout.split() if pkg] return sorted(packages) def check_debootstrap_version(): if os.path.exists(debootstrap_version_file): with open(debootstrap_version_file) as debootstrap: old_debootstrap_version = debootstrap.read().strip() debootstrap_version = get_debootstrap_version() failed = subprocess.call( ['dpkg', '--compare-versions', debootstrap_version, 'ge', old_debootstrap_version]) if failed: error_exit('Installed debootstrap is older than in the ' 'previous version! (%s < %s)' % (debootstrap_version, old_debootstrap_version)) def update_debootstrap_version(): with open(debootstrap_version_file, 'w') as debootstrap: debootstrap.write(get_debootstrap_version() + '\n') def format_changes(items): by_arch = defaultdict(set) for pkg, arch in items: by_arch[pkg].add(arch) all_pkgs = sorted(by_arch) chunks = [] for pkg in all_pkgs: arches = by_arch[pkg] if set(architectures) - arches: # only some architectures chunks.append('%s [%s]' % (pkg, ' '.join(sorted(arches)))) else: # all architectures chunks.append(pkg) return ', '.join(chunks) germinate_logging(logging.DEBUG) check_debootstrap_version() additions = defaultdict(list) removals = defaultdict(list) moves = defaultdict(list) metapackage_map = {} for architecture in architectures: print("[%s] Downloading available package lists..." % architecture) germinator = Germinator(architecture) archive = germinate.archive.TagFile( dists, components, architecture, archive_base[architecture], source_mirrors=archive_base_default, cleanup=True) germinator.parse_archive(archive) debootstrap_base = set(debootstrap_packages(architecture)) print("[%s] Loading seed lists..." % architecture) try: structure = SeedStructure(seed_dist, seed_base, options.bzr) germinator.plant_seeds(structure, seeds=seeds) except SeedError: sys.exit(1) print("[%s] Merging seeds with available package lists..." % architecture) for seed_name in output_seeds: meta_name = metapackage_name(structure, seed_name) metapackage_map[seed_name] = meta_name output_filename = os.path.join( options.outdir, '%s-%s' % (seed_name, architecture)) old_list = None if os.path.exists(output_filename): with open(output_filename) as output: old_list = set(map(str.strip, output.readlines())) os.rename(output_filename, output_filename + '.old') # work on the depends new_list = [] packages = seed_packages(germinator.get_seed_entries, structure, seed_name) for package in packages: if package == meta_name: print("%s/%s: Skipping package %s (metapackage)" % (seed_name, architecture, package)) elif (seed_name == 'minimal' and package not in debootstrap_base): print("%s/%s: Skipping package %s (package not in " "debootstrap)" % (seed_name, architecture, package)) elif germinator.is_essential(package): print("%s/%s: Skipping package %s (essential)" % (seed_name, architecture, package)) else: new_list.append(package) new_list.sort() with open(output_filename, 'w') as output: for package in new_list: output.write(package) output.write('\n') # work on the recommends old_recommends_list = None new_recommends_list = [] packages = seed_packages(germinator.get_seed_recommends_entries, structure, seed_name) for package in packages: if package == meta_name: print("%s/%s: Skipping package %s (metapackage)" % (seed_name, architecture, package)) continue if seed_name == 'minimal' and package not in debootstrap_base: print("%s/%s: Skipping package %s (package not in " "debootstrap)" % (seed_name, architecture, package)) else: new_recommends_list.append(package) new_recommends_list.sort() seed_name_recommends = '%s-recommends' % seed_name output_recommends_filename = os.path.join( options.outdir, '%s-%s' % (seed_name_recommends, architecture)) if os.path.exists(output_recommends_filename): with open(output_recommends_filename) as output: old_recommends_list = set( map(str.strip, output.readlines())) os.rename( output_recommends_filename, output_recommends_filename + '.old') with open(output_recommends_filename, 'w') as output: for package in new_recommends_list: output.write(package) output.write('\n') # Calculate deltas merged = defaultdict(int) recommends_merged = defaultdict(int) if old_list is not None: for package in new_list: merged[package] += 1 for package in old_list: merged[package] -= 1 if old_recommends_list is not None: for package in new_recommends_list: recommends_merged[package] += 1 for package in old_recommends_list: recommends_merged[package] -= 1 mergeditems = sorted(merged.items()) for package, value in mergeditems: #print(package, value) if value == 1: if recommends_merged.get(package, 0) == -1: moves[package].append([seed_name, architecture]) recommends_merged[package] += 1 else: additions[package].append([seed_name, architecture]) elif value == -1: if recommends_merged.get(package, 0) == 1: moves[package].append([seed_name_recommends, architecture]) recommends_merged[package] -= 1 else: removals[package].append([seed_name, architecture]) mergedrecitems = sorted(recommends_merged.items()) for package, value in mergedrecitems: #print(package, value) if value == 1: additions[package].append([seed_name_recommends, architecture]) elif value == -1: removals[package].append([seed_name_recommends, architecture]) with open('metapackage-map', 'w') as metapackage_map_file: for seed_name in output_seeds: print(seed_name, metapackage_map[seed_name], file=metapackage_map_file) if not options.nodch and (additions or removals or moves): dch_help = subprocess.Popen(['dch', '--help'], stdout=subprocess.PIPE, universal_newlines=True) try: have_U = '-U' in dch_help.stdout.read() finally: if dch_help.stdout: dch_help.stdout.close() dch_help.wait() if have_U: subprocess.check_call(['dch', '-iU', 'Refreshed dependencies']) else: subprocess.check_call(['dch', '-i', 'Refreshed dependencies']) changes = [] for package in sorted(additions): changes.append('Added %s to %s' % (package, format_changes(additions[package]))) for package in sorted(removals): changes.append('Removed %s from %s' % (package, format_changes(removals[package]))) for package in sorted(moves): # TODO: We should really list where it moved from as well, but # that gets wordy very quickly, and at the moment this is only # implemented for depends->recommends or vice versa. In future, # using this for moves between seeds might also be useful. changes.append('Moved %s to %s' % (package, format_changes(moves[package]))) for change in changes: print(change) subprocess.check_call(['dch', '-a', change]) update_debootstrap_version() else: if not options.nodch: print("No changes found") return 0 germinate/germinate/scripts/germinate_pkg_diff.py0000644000000000000000000001734712326425437017563 0ustar # -*- coding: UTF-8 -*- # Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 # Canonical Ltd. # # This file is part of Germinate. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. import sys import optparse import logging import subprocess from germinate.germinator import Germinator import germinate.archive import germinate.defaults from germinate.log import germinate_logging from germinate.seeds import SeedError, SeedStructure import germinate.version MIRRORS = [germinate.defaults.mirror] COMPONENTS = ["main"] class Package: def __init__(self, name): self.name = name self.seed = {} self.installed = 0 def set_seed(self, seed): self.seed[seed] = 1 def set_installed(self): self.installed = 1 def output(self, outmode): ret = self.name.ljust(30) + "\t" if outmode == "i": if self.installed and not len(self.seed): ret += "deinstall" elif not self.installed and len(self.seed): ret += "install" else: return "" elif outmode == "r": if self.installed and not len(self.seed): ret += "install" elif not self.installed and len(self.seed): ret += "deinstall" else: return "" else: # default case if self.installed and not len(self.seed): ret = "- " + ret elif not self.installed and len(self.seed): ret = "+ " + ret else: ret = " " + ret ret += ",".join(sorted(self.seed)) return ret class Globals: def __init__(self): self.package = {} self.seeds = [] self.outputs = {} self.outmode = "" def set_seeds(self, options, seeds): self.seeds = seeds # Suppress most log information germinate_logging(logging.CRITICAL) logging.getLogger('germinate.archive').setLevel(logging.INFO) global MIRRORS, COMPONENTS print("Germinating") g = Germinator(options.arch) archive = germinate.archive.TagFile( options.dist, COMPONENTS, options.arch, MIRRORS, cleanup=True) g.parse_archive(archive) needed_seeds = [] build_tree = False try: structure = SeedStructure(options.release, options.seeds) for seedname in self.seeds: if seedname == ('%s+build-depends' % structure.supported): seedname = structure.supported build_tree = True needed_seeds.append(seedname) g.plant_seeds(structure, seeds=needed_seeds) except SeedError: sys.exit(1) g.grow(structure) for seedname in structure.names: for pkg in g.get_seed_entries(structure, seedname): self.package.setdefault(pkg, Package(pkg)) self.package[pkg].set_seed(seedname + ".seed") for pkg in g.get_seed_recommends_entries(structure, seedname): self.package.setdefault(pkg, Package(pkg)) self.package[pkg].set_seed(seedname + ".seed-recommends") for pkg in g.get_depends(structure, seedname): self.package.setdefault(pkg, Package(pkg)) self.package[pkg].set_seed(seedname + ".depends") if build_tree: build_depends = set(g.get_build_depends(structure, seedname)) for inner in structure.inner_seeds(structure.supported): build_depends -= set(g.get_seed_entries(structure, inner)) build_depends -= set(g.get_seed_recommends_entries( structure, inner)) build_depends -= g.get_depends(structure, inner) for pkg in build_depends: self.package.setdefault(pkg, Package(pkg)) self.package[pkg].set_seed(structure.supported + ".build-depends") def parse_dpkg(self, fname): if fname is None: dpkg_cmd = subprocess.Popen(['dpkg', '--get-selections'], stdout=subprocess.PIPE, universal_newlines=True) try: lines = dpkg_cmd.stdout.readlines() finally: if dpkg_cmd.stdout: dpkg_cmd.stdout.close() dpkg_cmd.wait() else: with open(fname) as f: lines = f.readlines() for l in lines: pkg, st = l.split(None) self.package.setdefault(pkg, Package(pkg)) if st == "install" or st == "hold": self.package[pkg].set_installed() def set_output(self, mode): self.outmode = mode def output(self): for k in sorted(self.package): l = self.package[k].output(self.outmode) if len(l): print(l) def parse_options(argv): epilog = '''\ A list of seeds against which to compare may be supplied as non-option arguments. Seeds from which they inherit will be added automatically. The default is 'desktop'.''' parser = optparse.OptionParser( prog='germinate-pkg-diff', usage='%prog [options] [seeds]', version='%prog ' + germinate.version.VERSION, epilog=epilog) parser.add_option('-l', '--list', dest='dpkg_file', metavar='FILE', help='read list of packages from this file ' '(default: read from dpkg --get-selections)') parser.add_option('-m', '--mode', dest='mode', type='choice', choices=('i', 'r', 'd'), default='d', metavar='[i|r|d]', help='show packages to install/remove/diff (default: d)') parser.add_option('-S', '--seed-source', dest='seeds', metavar='SOURCE', default=germinate.defaults.seeds, help='fetch seeds from SOURCE (default: %s)' % germinate.defaults.seeds) parser.add_option('-s', '--seed-dist', dest='release', metavar='DIST', default=germinate.defaults.release, help='fetch seeds for distribution DIST ' '(default: %default)') parser.add_option('-d', '--dist', dest='dist', default=germinate.defaults.dist, help='operate on distribution DIST (default: %default)') parser.add_option('-a', '--arch', dest='arch', default=germinate.defaults.arch, help='operate on architecture ARCH (default: %default)') options, args = parser.parse_args(argv[1:]) options.seeds = options.seeds.split(',') options.dist = options.dist.split(',') return options, args def main(argv): g = Globals() options, args = parse_options(argv) g.set_output(options.mode) g.parse_dpkg(options.dpkg_file) if not len(args): args = ["desktop"] g.set_seeds(options, args) g.output() return 0 germinate/germinate/scripts/germinate_main.py0000644000000000000000000001720612326425437016730 0ustar # -*- coding: UTF-8 -*- """Expand dependencies in a list of seed packages.""" # Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 # Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. import os import shutil import sys import optparse import logging from germinate.germinator import Germinator import germinate.archive import germinate.defaults from germinate.log import germinate_logging from germinate.seeds import Seed, SeedError, SeedStructure import germinate.version def parse_options(argv): parser = optparse.OptionParser( prog='germinate', version='%prog ' + germinate.version.VERSION) parser.add_option('-v', '--verbose', dest='verbose', action='store_true', default=False, help='be more verbose when processing seeds') parser.add_option('-S', '--seed-source', dest='seeds', metavar='SOURCE', help='fetch seeds from SOURCE (default: %s)' % germinate.defaults.seeds) parser.add_option('-s', '--seed-dist', dest='release', metavar='DIST', default=germinate.defaults.release, help='fetch seeds for distribution DIST ' '(default: %default)') parser.add_option('-m', '--mirror', dest='mirrors', action='append', metavar='MIRROR', help='get package lists from MIRROR (default: %s)' % germinate.defaults.mirror) parser.add_option('--source-mirror', dest='source_mirrors', action='append', metavar='MIRROR', help='get source package lists from mirror ' '(default: value of --mirror)') parser.add_option('-d', '--dist', dest='dist', default=germinate.defaults.dist, help='operate on distribution DIST (default: %default)') parser.add_option('-a', '--arch', dest='arch', default=germinate.defaults.arch, help='operate on architecture ARCH (default: %default)') parser.add_option('-c', '--components', dest='components', default='main,restricted', metavar='COMPS', help='operate on components COMPS (default: %default)') parser.add_option('--bzr', dest='bzr', action='store_true', default=False, help='fetch seeds using bzr (requires bzr to be ' 'installed)') parser.add_option('--cleanup', dest='cleanup', action='store_true', default=False, help="don't cache Packages or Sources files") parser.add_option('--no-rdepends', dest='want_rdepends', action='store_false', default=True, help='disable reverse-dependency calculations') parser.add_option('--no-installer', dest='installer', action='store_false', default=True, help='do not consider debian-installer udeb packages') parser.add_option('--seed-packages', dest='seed_packages', metavar='PARENT/PKG,PARENT/PKG,...', help='treat each PKG as a seed by itself, inheriting ' 'from PARENT') options, _ = parser.parse_args(argv[1:]) if options.seeds is None: if options.bzr: options.seeds = germinate.defaults.seeds_bzr else: options.seeds = germinate.defaults.seeds options.seeds = options.seeds.split(',') if options.mirrors is None: options.mirrors = [germinate.defaults.mirror] options.dist = options.dist.split(',') options.components = options.components.split(',') if options.seed_packages is None: options.seed_packages = [] else: options.seed_packages = options.seed_packages.split(',') return options def main(argv): options = parse_options(argv) if options.verbose: germinate_logging(logging.DEBUG) else: germinate_logging(logging.INFO) g = Germinator(options.arch) archive = germinate.archive.TagFile( options.dist, options.components, options.arch, options.mirrors, source_mirrors=options.source_mirrors, installer_packages=options.installer, cleanup=options.cleanup) g.parse_archive(archive) if os.path.isfile("hints"): with open("hints") as hints: g.parse_hints(hints) try: structure = SeedStructure(options.release, options.seeds, options.bzr) for seed_package in options.seed_packages: parent, pkg = seed_package.split('/') structure.add(pkg, [" * " + pkg], parent) g.plant_seeds(structure) except SeedError: sys.exit(1) try: with Seed(options.seeds, options.release, "blacklist", options.bzr) as blacklist: g.parse_blacklist(structure, blacklist) except SeedError: pass g.grow(structure) g.add_extras(structure) if options.want_rdepends: g.reverse_depends(structure) for seedname in structure.names + ["extra"]: g.write_full_list(structure, seedname, seedname) g.write_seed_list(structure, seedname + ".seed", seedname) g.write_seed_recommends_list(structure, seedname + ".seed-recommends", seedname) g.write_depends_list(structure, seedname + ".depends", seedname) g.write_build_depends_list(structure, seedname + ".build-depends", seedname) if seedname != "extra" and seedname in structure: structure.write_seed_text(seedname + ".seedtext", seedname) g.write_sources_list(structure, seedname + ".sources", seedname) g.write_build_sources_list(structure, seedname + ".build-sources", seedname) g.write_all_list(structure, "all") g.write_all_source_list(structure, "all.sources") g.write_supported_list(structure, "%s+build-depends" % structure.supported) g.write_supported_source_list( structure, "%s+build-depends.sources" % structure.supported) g.write_all_extra_list(structure, "all+extra") g.write_all_extra_source_list(structure, "all+extra.sources") g.write_provides_list(structure, "provides") structure.write("structure") structure.write_dot("structure.dot") if os.path.exists("rdepends"): shutil.rmtree("rdepends") if options.want_rdepends: os.mkdir("rdepends") os.mkdir(os.path.join("rdepends", "ALL")) for pkg in g.get_all(structure): dirname = os.path.join("rdepends", g.get_source(pkg)) if not os.path.exists(dirname): os.mkdir(dirname) g.write_rdepend_list(structure, os.path.join(dirname, pkg), pkg) os.symlink(os.path.join("..", g.get_source(pkg), pkg), os.path.join("rdepends", "ALL", pkg)) g.write_blacklisted(structure, "blacklisted") return 0 germinate/germinate/scripts/__init__.py0000644000000000000000000000011212326425437015474 0ustar # None of the modules in this package have a guaranteed stable interface. germinate/germinate/__init__.py0000644000000000000000000000015212326425437014011 0ustar import apt_pkg apt_pkg.init() __all__ = ['archive', 'defaults', 'germinator', 'log', 'seeds', 'version'] germinate/germinate/tsort.py0000644000000000000000000001655312326425437013441 0ustar # Copyright (C) 2005, 2006, 2008, 2009, 2012 Canonical Ltd # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # This code originated in bzrlib. In order to make it suitable for use in # Germinate, Colin Watson removed its bzrlib.errors dependency and stripped # out the implementation of specialised merge-graph sorting. """Topological sorting routines.""" import sys __all__ = ["topo_sort", "TopoSorter"] class GraphCycleError(Exception): _fmt = "Cycle in graph %(graph)r" def __init__(self, graph): Exception.__init__(self) self.graph = graph def __str__(self): d = dict(self.__dict__) # special case: python2.5 puts the 'message' attribute in a # slot, so it isn't seen in __dict__ d['message'] = getattr(self, 'message', 'no message') s = self._fmt % d if sys.version < '3': # __str__() should always return a 'str' object # never a 'unicode' object. if isinstance(s, unicode): return s.encode('utf8') return s def topo_sort(graph): """Topological sort a graph. graph -- sequence of pairs of node->parents_list. The result is a list of node names, such that all parents come before their children. node identifiers can be any hashable object, and are typically strings. """ return TopoSorter(graph).sorted() class TopoSorter(object): def __init__(self, graph): """Topological sorting of a graph. :param graph: sequence of pairs of node_name->parent_names_list. i.e. [('C', ['B']), ('B', ['A']), ('A', [])] For this input the output from the sort or iter_topo_order routines will be: 'A', 'B', 'C' node identifiers can be any hashable object, and are typically strings. If you have a graph like [('a', ['b']), ('a', ['c'])] this will only use one of the two values for 'a'. The graph is sorted lazily: until you iterate or sort the input is not processed other than to create an internal representation. iteration or sorting may raise GraphCycleError if a cycle is present in the graph. """ # a dict of the graph. self._graph = dict(graph) self._visitable = set(self._graph) ### if debugging: # self._original_graph = dict(graph) # this is a stack storing the depth first search into the graph. self._node_name_stack = [] # at each level of 'recursion' we have to check each parent. This # stack stores the parents we have not yet checked for the node at the # matching depth in _node_name_stack self._pending_parents_stack = [] # this is a set of the completed nodes for fast checking whether a # parent in a node we are processing on the stack has already been # emitted and thus can be skipped. self._completed_node_names = set() def sorted(self): """Sort the graph and return as a list. After calling this the sorter is empty and you must create a new one. """ return list(self.iter_topo_order()) ### Useful if fiddling with this code. ### # cross check ### sorted_names = list(self.iter_topo_order()) ### for index in range(len(sorted_names)): ### rev = sorted_names[index] ### for left_index in range(index): ### if rev in self.original_graph[sorted_names[left_index]]: ### print("revision in parent list of earlier revision") ### import pdb;pdb.set_trace() def iter_topo_order(self): """Yield the nodes of the graph in a topological order. After finishing iteration the sorter is empty and you cannot continue iteration. """ while self._graph: # now pick a random node in the source graph, and transfer it to # the top of the depth first search stack. node_name, parents = self._graph.popitem() self._push_node(node_name, parents) while self._node_name_stack: # loop until this call completes. parents_to_visit = self._pending_parents_stack[-1] # if all parents are done, the revision is done if not parents_to_visit: # append the revision to the topo sorted list # all the nodes parents have been added to the output, now # we can add it to the output. yield self._pop_node() else: while self._pending_parents_stack[-1]: # recurse depth first into a single parent next_node_name = self._pending_parents_stack[-1].pop() if next_node_name in self._completed_node_names: # this parent was completed by a child on the # call stack. skip it. continue if next_node_name not in self._visitable: continue # otherwise transfer it from the source graph into the # top of the current depth first search stack. try: parents = self._graph.pop(next_node_name) except KeyError: # if the next node is not in the source graph it # has already been popped from it and placed # into the current search stack (but not # completed or we would have hit the continue 4 # lines up. this indicates a cycle. raise GraphCycleError(self._node_name_stack) self._push_node(next_node_name, parents) # and do not continue processing parents until this # 'call' has recursed. break def _push_node(self, node_name, parents): """Add node_name to the pending node stack. Names in this stack will get emitted into the output as they are popped off the stack. """ self._node_name_stack.append(node_name) self._pending_parents_stack.append(list(parents)) def _pop_node(self): """Pop the top node off the stack The node is appended to the sorted output. """ # we are returning from the flattened call frame: # pop off the local variables node_name = self._node_name_stack.pop() self._pending_parents_stack.pop() self._completed_node_names.add(node_name) return node_name germinate/setup.py0000755000000000000000000001100612326425437011442 0ustar #! /usr/bin/env python import os import re import sys import subprocess from setuptools import setup, Command, find_packages from distutils.command.build import build from setuptools.command.test import test from setuptools.command.install import install from distutils.command.clean import clean # We probably ought to use debian.changelog, but let's avoid that dependency # unless and until Germinate needs it somewhere else. changelog_heading = re.compile(r'\w[-+0-9a-z.]* \(([^\(\) \t]+)\)') with open('debian/changelog') as changelog: line = changelog.readline() match = changelog_heading.match(line) if match is None: raise ValueError( "Failed to parse first line of debian/changelog: '%s'" % line) germinate_version = match.group(1) class build_extra(build): def __init__(self, dist): build.__init__(self, dist) self.user_options.extend([('pod2man', None, 'use pod2man')]) def initialize_options(self): build.initialize_options(self) self.pod2man = False def finalize_options(self): def has_pod2man(command): return self.pod2man == 'True' build.finalize_options(self) self.sub_commands.append(('build_pod2man', has_pod2man)) class build_pod2man(Command): description = "build POD manual pages" user_options = [('pod-files=', None, 'POD files to build')] def initialize_options(self): self.pod_files = [] def finalize_options(self): pass def run(self): for pod_file in self.distribution.scripts: if not pod_file.startswith('debhelper/'): continue if os.path.exists('%s.1' % pod_file): continue self.spawn(['pod2man', '-c', 'Debhelper', '-r', germinate_version, pod_file, '%s.1' % pod_file]) class test_extra(test): def run(self): # Only useful for Python 2 right now. if sys.version_info[0] < 3: self.spawn(['./run-pychecker']) test.run(self) class install_extra(install): def run(self): install.run(self) self.spawn(['sed', '-i', 's/@VERSION@/%s/' % germinate_version, os.path.join(self.install_lib, 'germinate', 'version.py')]) class clean_extra(clean): def run(self): clean.run(self) for path, dirs, files in os.walk('.'): for i in reversed(range(len(dirs))): if dirs[i].startswith('.') or dirs[i] == 'debian': del dirs[i] elif dirs[i] == '__pycache__' or dirs[i].endswith('.egg-info'): self.spawn(['rm', '-r', os.path.join(path, dirs[i])]) del dirs[i] for f in files: f = os.path.join(path, f) if f.endswith('.pyc'): self.spawn(['rm', f]) elif f.startswith('./debhelper') and f.endswith('.1'): self.spawn(['rm', f]) perl_vendorlib = subprocess.Popen( ['perl', '-MConfig', '-e', 'print $Config{vendorlib}'], stdout=subprocess.PIPE, universal_newlines=True).communicate()[0] if not perl_vendorlib: raise ValueError("Failed to get $Config{vendorlib} from perl") perllibdir = '%s/Debian/Debhelper/Sequence' % perl_vendorlib setup( name='germinate', version=germinate_version, description='Expand dependencies in a list of seed packages', author='Scott James Remnant', author_email='scott@ubuntu.com', maintainer='Colin Watson', maintainer_email='cjwatson@ubuntu.com', url='https://wiki.ubuntu.com/Germinate', license='GNU GPL', packages=find_packages(), scripts=[ 'bin/germinate', 'bin/germinate-pkg-diff', 'bin/germinate-update-metapackage', 'debhelper/dh_germinate_clean', 'debhelper/dh_germinate_metapackage', ], data_files=[ (perllibdir, ['debhelper/germinate.pm']), ('/usr/share/man/man1', [ 'man/germinate.1', 'man/germinate-pkg-diff.1', 'man/germinate-update-metapackage.1', 'debhelper/dh_germinate_clean.1', 'debhelper/dh_germinate_metapackage.1', ])], cmdclass={ 'build': build_extra, 'build_pod2man': build_pod2man, 'test': test_extra, 'install': install_extra, 'clean': clean_extra, }, test_suite='germinate.tests', # python-apt doesn't build an egg, so we can't use this. #install_requires=['apt>=0.7.93'], #tests_require=['apt>=0.7.93'], ) germinate/README0000644000000000000000000001664112326425437010617 0ustar LaMont thinks this is fairly complicated ... We have "seed"s, these are lists of packages which we consider important to be in the distribution. We divide these into three sets for the archive. base A core set of packages to build a server or desktop on. desktop The functionality for our CD. supported Everything else we want to support, including the build-dependencies of base and desktop. To form each set, the seed and dependency tree need to be added to it. In order to form the supported set, the build-dependency tree for each set needs to be calculated as well, with any dependencies resulting added to the build-dependency list to end up in the supported set. Example. foo; Depends: bar, baz; Build-Depends: quux. bar; Depends: wibble; Build-Depends: splat. quux; Depends: wobble; Build-Depends: snarf. If foo is placed in the base seed, then the base dependency list will contain its dependencies (bar, baz) and their dependencies (wibble). The base build-dependency list will contain the build-dependencies of the base seed and dependency list (quux, splat) their dependencies (wobble) and their build-dependencies (snarf). So we end up with: base seed foo base depends bar (D foo), baz (D foo), wibble (D bar) base build-deps quux (BD foo), wobble (D quux), snarf (BD quux) If wobble had dependencies or build-dependencies, they would also be placed in the base build-dependency list. You therefore get the following output lists; base.seed (the seed list itself), base.depends (the dependency list) and base.build-depends (the build-dependency list). That was the easy bit... here's the tricky bit. Across all three lists for each of the all three sets, every package only occurs in only one list. So going back to our previous example, if a package in the desktop seed depends on 'foo' it won't be added to the desktop dependency list because it is already in the base seed. Likewise if a package in the desktop dependency list depends on 'bar', it won't be added to the desktop dependency list because it's already in the base dependency list. *However* build-dependencies are considered lesser-favoured than any seed or dependency list, so if a package needed in the desktop seed or dependency list is in the base build-dependency list it will be moved out of it and into the list it's needed in. The actual order of priority is: base.seed, base.depends, desktop.seed, desktop.depends,... supported.seed, supported.depends, base.build-depends,... desktop.build-depends, supported.build-depends With things being moved out of build-depends and into the right set (this doesn't affect the others because they're created in the same order as the priority.) For each set, the seed and dependency list then get joined together to form a list named after the set itself. This isn't quite all that's needed for supported, so another list is output named supported+build-depends containing the supported list and the build-depends lists of the other sets all joined together. We then join these together to output an "all" list that represents our entire archive. So that's the principal set of lists out of the way, but Germinate also keeps track of when source packages are first used (bearing in mind that their binaries could be in different sets). So for each set, there's a sources list containing the source packages used in that set and not used in any of higher priority. There's also a build-sources list for the source packages used by the build-dependency list but none of the sets themselves nor any build-sources list of a higher priority. For maximum greppability, we also get supported+build-depends.sources and all.sources files. What happens to the binary packages created by our sources that we don't place in any set? I'm glad you asked that, they become the "extra" seed (the sources list of which is obviously empty so omitted) have the dependency tree calculated and placed in the extra dependency list and the build-dependencies in the extra build-dependency list (although the seeded packages' build-dependencies will already have been processed, the build-dependencies of the seeded packages' dependencies may not have been). All three extra lists are at a lower priority even than the build dependency lists of the other sets, so a package won't be added to the extra dependency or build-dependency lists if they are already in any other. So we join this up to get extra, extra.sources and extra.build-sources lists. We add these to the all lists we made earlier and get all+extra and all+extra.sources. These won't all be in the archive, we will almost certainly want to cherry-pick from extra but there's a lot of crack in it too. Germinate tries very hard to satisfy package dependencies using the existing lists, and in any given sequence of alternative packages or virtual dependency it will generally pick a package already added. An interesting list of virtual packages and those chosen that provide that is output in the provides file. Lastly (oh gods, there's more?!) Germinate waffles quite a bit as it runs and there's a _germinate_output file containing this. Lines beginning '?' are missing or unresolvable packages, '*' are choices made by Germinate that you might wish to influence and '!' are problems with the sets. ! Duplicated seed: FOO FOO has been placed in multiple seeds. The better seed always takes precedence (because it gets processed first). ! Promoted BAR from SET to BETTER-SET to satisfy FOO BAR was in the SET seed, but is a dependency of FOO which is in BETTER-SET. Germinate promotes the package to the better set, you can safely remove it from SET if you want as FOO depends on it. * Virtual SET package: WIBBLE - WOBBLE - SNARFLE You placed WIBBLE in SET seed, and it is a virtual package (one provided by multiple packages). Rather than picking one, Germinate assumes you want all of the packages that provide this and lists the ones it adds. * Chose WOBBLE out of WIBBLE to satisfy FOO FOO depends on the virtual package WIBBLE, and nothing providing that has been seeded or added yet. Germinate randomly chose WOBBLE, the first package in the list, to satisfy this dependency. You can influence this choice by seeding the one you want. * Chose BAR to satisfy FOO FOO depends on a list a possible packages, and none of them has been seeded or added yet. Germinate randomly chose BAR, the first in the list. You can influence this choice by seeding the one you want. ? Unknown SET package: FOO FOO in the SET seed doesn't exist in the Debian archive, at least not in unstable/main currently. Try removing the cached Packages and Sources files. ? Unknown dependency BAR by FOO FOO declares a dependency on BAR which can't be satisfied within Debian unstable/main currently. Try removing the cached Packages and Sources files. ? Nothing to choose out of WIBBLE to satisfy FOO FOO declares a dependency on virtual package WIBBLE, but nothing in Debian unstable/main currently provides that. Try removing the cached Packages and Sources files. ? Nothing to choose to satisfy FOO FOO declares a dependency on a list of packages, none of which are currently in Debian unstable/main. Try removing the cached Packages and Sources files. ? Missing source package: QUUX (for FOO) FOO's source package is QUUX, but that is missing from the Sources file. Try removing the Packages and Sources files. germinate/debhelper/0000755000000000000000000000000012326425437011661 5ustar germinate/debhelper/dh_germinate_clean0000755000000000000000000000277012326425437015405 0ustar #!/usr/bin/perl -w =head1 NAME dh_germinate_clean - clean up files left by germinate-update-metapackage =cut use strict; use Debian::Debhelper::Dh_Lib; =head1 SYNOPSIS B [S>] =head1 DESCRIPTION dh_germinate_clean is a debhelper program that cleans up some files created by C that are useful for debugging but that should not appear in source packages. It removes any C<*.old> files that match defined seeds (as listed in the C file by a previous run of C), and the C file. =head1 EXAMPLES dh_germinate_clean is usually called indirectly in a rules file via the dh command. %: dh $@ --with germinate You must build-depend on at least debhelper (>= 7.0.8) to use this form, and in any case you must build-depend on at least germinate (>= 1.18) to use this program at all. It can also be called directly in the clean target at any time before dh_clean. clean: dh_testdir dh_germinate_clean dh_clean =cut init(); open MAP, 'metapackage-map' or die "Can't open metapackage-map: $!"; while () { chomp; my ($seed, $metapackage) = split; complex_doit("rm -f $seed-*.old"); } close MAP; doit("rm", "-f", "debootstrap-dir"); =head1 SEE ALSO L, L This program is a part of germinate. =head1 AUTHOR Colin Watson Copyright (C) 2009 Canonical Ltd., licensed under the GNU GPL v2 or later. =cut germinate/debhelper/germinate.pm0000644000000000000000000000034412326425437014173 0ustar #!/usr/bin/perl # debhelper sequence file for germinate use warnings; use strict; use Debian::Debhelper::Dh_Lib; insert_before("dh_installdeb", "dh_germinate_metapackage"); insert_before("dh_clean", "dh_germinate_clean"); 1; germinate/debhelper/dh_germinate_metapackage0000755000000000000000000000511512326425437016561 0ustar #!/usr/bin/perl -w =head1 NAME dh_germinate_metapackage - create metapackages based on seeds =cut use strict; use Debian::Debhelper::Dh_Lib; =head1 SYNOPSIS B [S>] =head1 DESCRIPTION dh_germinate_metapackage is a debhelper program that generates ${germinate:Depends} substitutions and adds them to substvars files. The program will read the C file that should have been created by a previous run of C for your package, and for each seed listed there it will generate dependencies and recommendations based on the previous results of C. Dependencies will be substituted into your package's control file wherever you place the token "${germinate:Depends}", and similarly recommendations will be substituted wherever you place the token "${germinate:Recommends}". You must run C before building the source package. Normally, source packages that use this program will include an C script that will call it with appropriate arguments. =head1 EXAMPLES dh_germinate_metapackage is usually called indirectly in a rules file via the dh command. %: dh $@ --with germinate You must build-depend on at least debhelper (>= 7.0.8) to use this form, and in any case you must build-depend on at least germinate (>= 1.18) to use this program at all. It can also be called directly at any time between C and C. install: build dh_testdir dh_testroot dh_prep dh_installdirs dh_germinate_metapackage binary-arch: build install ... =cut init(); my %seeds = (); open MAP, 'metapackage-map' or die "Can't open metapackage-map: $!"; while () { chomp; my ($seed, $metapackage) = split; $seeds{$metapackage} = $seed; } close MAP; foreach my $package (@{$dh{DOPACKAGES}}) { next unless exists $seeds{$package}; my $seed = $seeds{$package}; my $arch = package_arch($package); open DEPENDS, "$seed-$arch" or die "Can't open $seed-$arch: $!"; while () { chomp; s/^\s*//; s/\s*$//; addsubstvar($package, "germinate:Depends", $_); } close DEPENDS; open RECOMMENDS, "$seed-recommends-$arch" or die "Can't open $seed-recommends-$arch: $!"; while () { chomp; s/^\s*//; s/\s*$//; addsubstvar($package, "germinate:Recommends", $_); } close RECOMMENDS; } =head1 SEE ALSO L, L This program is a part of germinate. =head1 AUTHOR Colin Watson Copyright (C) 2009 Canonical Ltd., licensed under the GNU GPL v2 or later. =cut germinate/bin/0000755000000000000000000000000012326425437010477 5ustar germinate/bin/germinate-update-metapackage0000755000000000000000000000216312326425437016122 0ustar #! /usr/bin/env python # -*- coding: UTF-8 -*- # Copyright (C) 2012 Canonical Ltd. # # This file is part of Germinate. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. import sys try: import imp imp.find_module('germinate.germinator') except ImportError: # Running from build tree? import os sys.path.insert(0, os.path.join(sys.path[0], os.pardir)) from germinate.scripts.germinate_update_metapackage import main if __name__ == "__main__": main(sys.argv) germinate/bin/germinate-pkg-diff0000755000000000000000000000215112326425437014064 0ustar #! /usr/bin/env python # -*- coding: UTF-8 -*- # Copyright (C) 2012 Canonical Ltd. # # This file is part of Germinate. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. import sys try: import imp imp.find_module('germinate.germinator') except ImportError: # Running from build tree? import os sys.path.insert(0, os.path.join(sys.path[0], os.pardir)) from germinate.scripts.germinate_pkg_diff import main if __name__ == "__main__": main(sys.argv) germinate/bin/germinate0000755000000000000000000000216712326425437012406 0ustar #! /usr/bin/env python # -*- coding: UTF-8 -*- """Expand dependencies in a list of seed packages.""" # Copyright (C) 2012 Canonical Ltd. # # Germinate is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2, or (at your option) any # later version. # # Germinate is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Germinate; see the file COPYING. If not, write to the Free # Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA # 02110-1301, USA. import sys try: import imp imp.find_module('germinate.germinator') except ImportError: # Running from build tree? import os sys.path.insert(0, os.path.join(sys.path[0], os.pardir)) from germinate.scripts.germinate_main import main if __name__ == "__main__": main(sys.argv) germinate/COPYING0000644000000000000000000004310512326425437010765 0ustar GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Library General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Library General Public License instead of this License.