debian/0000775000000000000000000000000012601265670007175 5ustar debian/changelog.trunk0000664000000000000000000000023412524713763012215 0ustar simplestreams (0.1.0~bzrREVNO-1~trunk1) UNRELEASED; urgency=low * Initial release -- Scott Moser Tue, 26 Mar 2013 01:10:01 +0000 debian/simplestreams.install0000664000000000000000000000011012524713763013452 0ustar usr/bin/* usr/lib/simplestreams/hook-debug usr/share/doc/simplestreams/ debian/control0000664000000000000000000000406712524713763010614 0ustar Source: simplestreams Section: python Priority: extra Standards-Version: 3.9.4 Maintainer: Ubuntu Developers Build-Depends: debhelper (>= 7), python-all, python-nose, python-setuptools, python-yaml, python3, python3-nose, python3-setuptools, python3-yaml Homepage: http://launchpad.net/simplestreams X-Python-Version: >= 2.7 X-Python3-Version: >= 3.2 Package: simplestreams Architecture: all Priority: extra Depends: python3-simplestreams, python3-yaml, ${misc:Depends}, ${python3:Depends} Replaces: python-simplestreams (<= 0.1.0~bzr230) Conflicts: python-simplestreams (<= 0.1.0~bzr230) Description: Library and tools for using Simple Streams data This package provides a client for interacting with simple streams data as is produced to describe Ubuntu's cloud images. Package: python3-simplestreams Architecture: all Priority: extra Depends: gnupg, ${misc:Depends}, ${python3:Depends} Suggests: python3-requests (>= 1.1) Description: Library and tools for using Simple Streams data This package provides a client for interacting with simple streams data as is produced to describe Ubuntu's cloud images. Package: python-simplestreams Architecture: all Priority: extra Depends: gnupg, python-boto, ${misc:Depends}, ${python:Depends} Suggests: python-requests (>= 1.1) Description: Library and tools for using Simple Streams data This package provides a client for interacting with simple streams data as is produced to describe Ubuntu's cloud images. Package: python-simplestreams-openstack Architecture: all Priority: extra Depends: python-glanceclient, python-keystoneclient, python-simplestreams, python-swiftclient, ${misc:Depends} Description: Library and tools for using Simple Streams data This package depends on libraries necessary to use the openstack dependent functionality in simplestreams. That includes interacting with glance, swift and keystone. debian/changelog0000664000000000000000000001464112601265670011055 0ustar simplestreams (0.1.0~bzr341-0ubuntu2.3) trusty-security; urgency=high * export checksummer in simplestreams.util (LP: #1499749) Users of simplestreams.util checksummer would get an AttributeError because this was moved. -- Scott Moser Fri, 25 Sep 2015 11:15:24 -0400 simplestreams (0.1.0~bzr341-0ubuntu2.2) trusty-security; urgency=medium * SECURITY UPDATE: insufficient verification of GPG signatures allowing malicious injection into images - debian/patches/lp1487004-use-checksumming-reader.patch: Ensure that users of the BasicMirrorWriter get exceptions when importing data that has invalid checksum or sizes. (LP: #1487004) - CVE-2015-1337 - debian/patches/lp1487004-sru-safetynet.patch: provide a backwards compatible behavior via setting SS_MISSING_ITEM_CHECKSUM_BEHAVIOR=silent. See bug for more info. -- Scott Moser Tue, 22 Sep 2015 17:12:43 -0400 simplestreams (0.1.0~bzr341-0ubuntu2.1) trusty-proposed; urgency=medium * GlanceMirror: identify images as i686 not i386 (LP: #1454775) When uploading i386 images to glance, mark their architecture as i686 rather than i386 so that the openstack scheduler will schedule them. -- Scott Moser Wed, 13 May 2015 14:33:29 -0400 simplestreams (0.1.0~bzr341-0ubuntu2) trusty-proposed; urgency=medium * debian/patches/1-add-item-filter-to-glancemirror.patch: add filter support, expose in sstream-mirror-glance (LP: #1339842) -- Michael McCracken Fri, 11 Jul 2014 13:01:54 -0700 simplestreams (0.1.0~bzr341-0ubuntu1) trusty; urgency=medium * New upstream snapshot. * sstream-mirror: dry-run support * sstream-mirror: support for mirroring streams that re-use a path across items (such as maas ephemerals) * sstream-mirror: show gpg error message on gpg verification * debian/control: add build-depends on python-nose and python-yaml for running python 2 tests. -- Scott Moser Wed, 26 Mar 2014 12:25:02 -0400 simplestreams (0.1.0~bzr331-0ubuntu1) trusty; urgency=medium * New upstream snapshot. * support writing sparse files in FileStore * stringitems: make ints and floats available as strings * resolvwork: enforce that max and keep must be integers * load_keystone_creds: documentation * fix bug with contentsource (LP: #1241711) -- Scott Moser Tue, 18 Mar 2014 17:01:45 -0400 simplestreams (0.1.0~bzr323-0ubuntu2) trusty; urgency=medium * Rebuild to drop files installed into /usr/share/pyshared. -- Matthias Klose Sun, 23 Feb 2014 13:54:02 +0000 simplestreams (0.1.0~bzr323-0ubuntu1) trusty; urgency=low * New upstream snapshot. * fix bug with sstream-sync (LP: #1241711) * fix bug in glance mirror (LP: #1243433) * fix odd behavior bug when filtering. The result was that if you had ever run with a filter, subsequent runs with modified filter would not see different results (LP: #1238227). -- Scott Moser Tue, 22 Oct 2013 19:35:38 -0400 simplestreams (0.1.0~bzr318-0ubuntu1) saucy-proposed; urgency=low * New upstream snapshot. * fix odd behavior bug when filtering. The result was that if you had ever run with a filter, subsequent runs with modified filter would not see different results (LP: #1238227). -- Scott Moser Tue, 15 Oct 2013 17:25:40 -0400 simplestreams (0.1.0~bzr316-0ubuntu1) saucy; urgency=low * New upstream snapshot. * fix bug in resuming a partial download in sstream-mirror or anything using the FileObjectStore (LP: #1237990) -- Scott Moser Thu, 10 Oct 2013 12:07:34 -0400 simplestreams (0.1.0~bzr315-0ubuntu1) saucy; urgency=low * New upstream snapshot. * fix interpreter written on sstream-sync, sstream-query and sstream-mirror. It should be python3, not python2.7. (LP: #1237637) * simplestreams/mirrors/__init__.py: fix bug causing resumed downloads to stack trace (LP: #1237658) -- Scott Moser Wed, 09 Oct 2013 17:30:23 -0400 simplestreams (0.1.0~bzr313-0ubuntu1) saucy; urgency=low * New upstream snapshot. * handle sigpipe gracefully (LP: #1207779) * as user-friendly cleanup add a trailing '/' to a url if necessary * add --delete flag to sstream-sync * add python-simplestreams-openstack metapackage so other packages could depend on this, but the python-simplestreams package will not incur those dependencies. (LP: #1233269) * debian/control: Suggest rather than Depend on python-requests. Also, up the version to the actual required version (1.1). -- Scott Moser Mon, 30 Sep 2013 15:00:54 -0400 simplestreams (0.1.0~bzr307-0ubuntu1) saucy; urgency=low * New upstream snapshot. * support for progress callbacks in file object store. * support for partial download resume support to file object store. * inclusion of sstream-mirror * support for reading from gpg keyring rather than user's default keyring. * sstream-query --pretty output format. * fix for python2 if no python-requests is available -- Scott Moser Wed, 11 Sep 2013 21:55:18 -0400 simplestreams (0.1.0~bzr272-0ubuntu1) saucy; urgency=low * New upstream snapshot. * fix bug where .sjson files did not work with sstream-query and python3. * fix python2 issue in library that resulted in items not having all tags. -- Scott Moser Wed, 10 Jul 2013 11:20:09 -0400 simplestreams (0.1.0~bzr266-0ubuntu1) saucy; urgency=low * New upstream snapshot. * include sstream-query, improved sstream-sync * supports python3 -- Scott Moser Fri, 28 Jun 2013 16:23:11 -0400 simplestreams (0.1.0~bzr229-0ubuntu1) raring; urgency=low * New upstream snapshot. * fix a bug in 'products_condense' * support http simple and digest auth -- Scott Moser Thu, 18 Apr 2013 18:19:11 -0400 simplestreams (0.1.0~bzr223-0ubuntu1) raring; urgency=low * New upstream snapshot. * some fixes to resolvework * include upstream work on glance mirror with support for writing simplestreams output to swift -- Scott Moser Thu, 11 Apr 2013 13:05:52 -0400 simplestreams (0.1.0~bzr191-0ubuntu1) raring; urgency=low * Initial release -- Scott Moser Tue, 26 Mar 2013 01:10:01 +0000 debian/source/0000775000000000000000000000000012524714405010474 5ustar debian/source/format0000664000000000000000000000001412524713763011710 0ustar 3.0 (quilt) debian/compat0000664000000000000000000000000212524713763010400 0ustar 7 debian/python-simplestreams.install0000664000000000000000000000005412524713763015000 0ustar usr/lib/python2*/*-packages/simplestreams/* debian/patches/0000775000000000000000000000000012601265657010631 5ustar debian/patches/lp1487004-use-checksumming-reader.patch0000664000000000000000000007266412600634724017553 0ustar Origin: upstream, revno 400 Bug: https://bugs.launchpad.net/bugs/1487004 Description: provide insert_item with a contentsource that does checksumming . Ensure that users of the BasicMirrorWriter get exceptions when importing data that has invalid checksum or sizes. ------------------------------------------------------------ revno: 400 [merge] fixes bug: https://launchpad.net/bugs/1487004 committer: Scott Moser branch nick: trunk timestamp: Tue 2015-09-22 16:28:53 -0400 message: provide insert_item with a contentsource that does checksumming Previously, insert_item would receive a 'contentsource' (essentially a file like object to be read as a stream). The user of that object needed to calculate checksums and verify the data they read. Now, instead the contentsource will do checksumming as read() operations are done, and will raise a checksum error in any failure case. Thus to use this, the user now simply has to loop over reads and catch the exception. stream data is now expected to have valid checksums and size on all items with a path. If the user is using a stream source that does not have either size or checksum information, they have a few options: a.) [legacy/SRU only] set environment variable SS_MISSING_ITEM_CHECKSUM_BEHAVIOR can be set to silent: behave exactly as before. No checksumming is done, no warnings are emitted. The consumer of the contentsource must check checksums. warn: log messages at WARN level (same as default/unset) fail: the new behavior. raise an InvalidChecksum exception. b.) instantiate the BasicMirrorWriter with config checksumming_reader=False the default for that config setting is True, meaning that you will get a reader that checksums content as it goes and raises exception on bad data. c.) fix their source to have a sha256sum and a size The 'sstream-mirror' program now has a '--no-checksumming-reader' flag that does 'b' for this mirror. ------------------------------------------------------------ Use --include-merged or -n0 to see merged revisions. === modified file 'bin/sstream-mirror' --- a/bin/sstream-mirror +++ b/bin/sstream-mirror @@ -50,6 +50,10 @@ def main(): parser.add_argument('--keyring', action='store', default=None, help='keyring to be specified to gpg via --keyring') + parser.add_argument('--no-checksumming-reader', action='store_false', + dest='checksumming_reader', default=True, + help=("do not call 'insert_item' with a reader" + " that does checksumming.")) parser.add_argument('source_mirror') parser.add_argument('output_d') @@ -69,7 +73,8 @@ def main(): filter_list = filters.get_filters(args.filters) mirror_config = {'max_items': args.max, 'keep_items': args.keep, 'filters': filter_list, - 'item_download': not args.no_item_download} + 'item_download': not args.no_item_download, + 'checksumming_reader': args.checksumming_reader} level = (log.ERROR, log.INFO, log.DEBUG)[min(args.verbose, 2)] log.basicConfig(stream=args.log_file, level=level) --- a/examples/foocloud/streams/v1/com.example.foovendor:released:download.json +++ b/examples/foocloud/streams/v1/com.example.foovendor:released:download.json @@ -40,18 +40,21 @@ "name": "foovendor-pinky-6.1-beta2-amd64-server-20120328.tar.gz", "path": "files/beta-2/foovendor-6.1-beta2-server-cloudimg-amd64.tar.gz", "md5": "c245123c1a7c16dd43962b71c604c5ee", + "size": 63, "ftype": "tar.gz" }, "disk1.img": { "name": "foovendor-pinky-6.1-beta2-amd64-server-20120328-disk1.img", "path": "files/beta-2/foovendor-6.1-beta2-server-cloudimg-amd64-disk1.img", "md5": "34cec541a18352783e736ba280a12201", + "size": 66, "ftype": "disk1.img" }, "root.tar.gz": { "name": "foovendor-pinky-6.1-beta2-amd64-server-20120328-root.tar.gz", "path": "files/beta-2/foovendor-6.1-beta2-server-cloudimg-amd64-root.tar.gz", "md5": "55686ef088f7baf0ebea9349055daa85", + "size": 68, "ftype": "root.tar.gz" } }, @@ -124,18 +127,21 @@ "name": "foovendor-pinky-6.1-beta2-i386-server-20120328.tar.gz", "path": "files/beta-2/foovendor-6.1-beta2-server-cloudimg-i386.tar.gz", "md5": "2cd18b60f892af68c9d49c64ce1638e4", + "size": 62, "ftype": "tar.gz" }, "disk1.img": { "name": "foovendor-pinky-6.1-beta2-i386-server-20120328-disk1.img", "path": "files/beta-2/foovendor-6.1-beta2-server-cloudimg-i386-disk1.img", "md5": "e80df7995beb31571e104947e4d7b001", + "size": 65, "ftype": "disk1.img" }, "root.tar.gz": { "name": "foovendor-pinky-6.1-beta2-i386-server-20120328-root.tar.gz", "path": "files/beta-2/foovendor-6.1-beta2-server-cloudimg-i386-root.tar.gz", "md5": "5d86b3e75e56e10e1019fe1153fe488f", + "size": 67, "ftype": "root.tar.gz" } }, --- /dev/null +++ b/examples/minimal/product1/20150915/root.img @@ -0,0 +1 @@ +content of product1/20150915/root.img --- /dev/null +++ b/examples/minimal/product1/20150915/text.txt @@ -0,0 +1 @@ +content of product1/20150915/text.txt --- /dev/null +++ b/examples/minimal/streams/v1/download.json @@ -0,0 +1,28 @@ +{ + "datatype": "image-downloads", + "format": "products:1.0", + "updated": "Tue, 15 Sep 2015 16:15:24 -0400", + "content_id": "com.example:download", + "products": { + "com.example:product1": { + "versions": { + "20150915": { + "items": { + "item1": { + "ftype": "text", + "path": "product1/20150915/text.txt", + "sha256": "f4797060c79363d37129656e5069e817cfd144ba9a1db0a392eeba0c534316dd", + "size": 38 + }, + "item2": { + "ftype": "root.img", + "path": "product1/20150915/root.img", + "sha256": "0e2f431f4e4063483708ce24448af8586e115c984e483d60bdb532194536ece4", + "size": 38 + } + } + } + } + } + } +} --- /dev/null +++ b/examples/minimal/streams/v1/index.json @@ -0,0 +1,15 @@ +{ + "index": { + "com.example:download": { + "datatype": "image-downloads", + "path": "streams/v1/download.json", + "updated": "Tue, 15 Sep 2015 16:15:24 -0400", + "products": [ + "com.example:product1" + ], + "format": "products:1.0" + } + }, + "updated": "Tue, 15 Sep 2015 16:15:24 -0400", + "format": "index:1.0" +} --- /dev/null +++ b/simplestreams/checksum_util.py @@ -0,0 +1,110 @@ +# Copyright (C) 2015 Canonical Ltd. +# +# Author: Scott Moser +# +# Simplestreams is free software: you can redistribute it and/or modify it +# under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or (at your +# option) any later version. +# +# Simplestreams is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY +# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public +# License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Simplestreams. If not, see . +import hashlib + +# these are in order of increasing preference +CHECKSUMS = ("md5", "sha256", "sha512") + +try: + ALGORITHMS = list(getattr(hashlib, 'algorithms')) +except AttributeError: + ALGORITHMS = list(hashlib.algorithms_available) + + +class checksummer(object): + _hasher = None + algorithm = None + expected = None + + def __init__(self, checksums): + if not checksums: + self._hasher = None + return + + for meth in CHECKSUMS: + if meth in checksums and meth in ALGORITHMS: + self._hasher = hashlib.new(meth) + self.algorithm = meth + + self.expected = checksums.get(self.algorithm, None) + + if not self._hasher: + raise TypeError("Unable to find suitable hash algorithm") + + def update(self, data): + if self._hasher is None: + return + self._hasher.update(data) + + def hexdigest(self): + if self._hasher is None: + return None + return self._hasher.hexdigest() + + def check(self): + return (self.expected is None or self.expected == self.hexdigest()) + + def __str__(self): + return ("checksummer (algorithm=%s expected=%s)" % + (self.algorithm, self.expected)) + + +def item_checksums(item): + return {k: item[k] for k in CHECKSUMS if k in item} + + +class SafeCheckSummer(checksummer): + """SafeCheckSummer raises ValueError if checksums are not provided.""" + def __init__(self, checksums, allowed=None): + if allowed is None: + allowed = CHECKSUMS + super(SafeCheckSummer, self).__init__(checksums) + if self.algorithm not in allowed: + raise ValueError( + "provided checksums (%s) did not include any allowed (%s)" % + (checksums, allowed)) + + +class InvalidChecksum(ValueError): + def __init__(self, path, cksum, size=None, expected_size=None, msg=None): + self.path = path + self.cksum = cksum + self.size = size + self.expected_size = expected_size + self.msg = msg + + def __str__(self): + if self.msg is not None: + return self.msg + if not isinstance(self.expected_size, int): + msg = "Invalid size '%s' at %s." % (self.expected_size, self.path) + else: + msg = ("Invalid %s Checksum at %s. Found %s. Expected %s. " + "read %s bytes expected %s bytes." % + (self.cksum.algorithm, self.path, + self.cksum.hexdigest(), self.cksum.expected, + self.size, self.expected_size)) + if self.size: + msg += (" (size %s expected %s)" % + (self.size, self.expected_size)) + return msg + + +def invalid_checksum_for_reader(reader, msg=None): + return InvalidChecksum(path=reader.url, cksum=reader.checksummer, + size=reader.bytes_read, expected_size=reader.size, + msg=msg) --- a/simplestreams/contentsource.py +++ b/simplestreams/contentsource.py @@ -20,6 +20,8 @@ import io import os import sys +from . import checksum_util + if sys.version_info > (3, 0): import urllib.parse as urlparse # pylint: disable=F0401,E0611 else: @@ -242,6 +244,70 @@ class MemoryContentSource(FdContentSourc super(MemoryContentSource, self).__init__(fd=fd, url=url) +class ChecksummingContentSource(ContentSource): + def __init__(self, csrc, checksums, size=None): + self.cs = csrc + self.bytes_read = 0 + self.checksummer = None + self.size = size + + try: + csummer = checksum_util.SafeCheckSummer(checksums) + except ValueError as e: + raise checksum_util.invalid_checksum_for_reader(self, msg=str(e)) + + self._set_checksummer(csummer) + + try: + self.size = int(size) + except TypeError: + self.size = size + raise checksum_util.invalid_checksum_for_reader(self) + + def resume(self, offset, checksummer): + self.cs.set_start_pos(offset) + self._set_checksummer(checksummer) + self.bytes_read = offset + + @property + def algorithm(self): + return self.checksummer.algorithm + + def _set_checksummer(self, checksummer): + if checksummer.algorithm not in checksum_util.CHECKSUMS: + raise ValueError("algorithm %s is not valid (%s)" % + (checksummer.algorithm, checksum_util.CHECKSUMS)) + self.checksummer = checksummer + + def check(self): + return self.bytes_read == self.size and self.checksummer.check() + + def read(self, size=-1): + buf = self.cs.read(size) + buflen = len(buf) + self.checksummer.update(buf) + self.bytes_read += buflen + + # read size was different size than expected. + # if its not the end, something wrong + if buflen != size and self.size != self.bytes_read: + raise checksum_util.invalid_checksum_for_reader(self) + + if self.bytes_read == self.size and not self.check(): + raise checksum_util.invalid_checksum_for_reader(self) + return buf + + def open(self): + return self.cs.open() + + def close(self): + return self.cs.close() + + @property + def url(self): + return self.cs.url + + class UrlReader(object): def read(self, size=-1): raise NotImplementedError() --- a/simplestreams/mirrors/__init__.py +++ b/simplestreams/mirrors/__init__.py @@ -14,13 +14,13 @@ # # You should have received a copy of the GNU Affero General Public License # along with Simplestreams. If not, see . - import errno import io import json import simplestreams.filters as filters import simplestreams.util as util +from simplestreams import checksum_util import simplestreams.contentsource as cs from simplestreams.log import LOG @@ -218,6 +218,7 @@ class BasicMirrorWriter(MirrorWriter): if config is None: config = {} self.config = config + self.checksumming_reader = self.config.get('checksumming_reader', True) def load_products(self, path=None, content_id=None): super(BasicMirrorWriter, self).load_products(path, content_id) @@ -313,8 +314,16 @@ class BasicMirrorWriter(MirrorWriter): ipath = item.get('path', None) ipath_cs = None - if ipath: - ipath_cs = reader.source(ipath) if reader else None + if ipath and reader: + if self.checksumming_reader: + flat = util.products_exdata(src, pgree) + ipath_cs = cs.ChecksummingContentSource( + csrc=reader.source(ipath), + size=flat.get('size'), + checksums=checksum_util.item_checksums(flat)) + else: + ipath_cs = reader.source(ipath) + self.insert_item(item, src, target, pgree, ipath_cs) if len(added_items): @@ -449,8 +458,8 @@ class ObjectStoreMirrorWriter(BasicMirro return LOG.debug("inserting %s to %s", contentsource.url, data['path']) self.store.insert(data['path'], contentsource, - checksums=util.item_checksums(data), mutable=False, - size=data.get('size')) + checksums=checksum_util.item_checksums(data), + mutable=False, size=data.get('size')) self._inc_rc(data['path'], src, pedigree) def insert_index_entry(self, data, src, pedigree, contentsource): @@ -458,7 +467,7 @@ class ObjectStoreMirrorWriter(BasicMirro if not epath: return self.store.insert(epath, contentsource, - checksums=util.item_checksums(data)) + checksums=checksum_util.item_checksums(data)) def insert_products(self, path, target, content): dpath = self.products_data_path(target['content_id']) @@ -557,5 +566,4 @@ def check_tree_paths(tree, fmt=None): for content_id in index: util.assert_safe_path(index[content_id].get('path')) - # vi: ts=4 expandtab --- a/simplestreams/mirrors/glance.py +++ b/simplestreams/mirrors/glance.py @@ -18,6 +18,7 @@ import simplestreams.filters as filters import simplestreams.mirrors as mirrors import simplestreams.util as util +from simplestreams import checksum_util import simplestreams.openstack as openstack from simplestreams.log import LOG @@ -271,7 +272,7 @@ class GlanceMirror(mirrors.BasicMirrorWr def _checksum_file(fobj, read_size=util.READ_SIZE, checksums=None): if checksums is None: checksums = {'md5': None} - cksum = util.checksummer(checksums=checksums) + cksum = checksum_util.checksummer(checksums=checksums) while True: buf = fobj.read(read_size) cksum.update(buf) --- a/simplestreams/objectstores/__init__.py +++ b/simplestreams/objectstores/__init__.py @@ -21,6 +21,7 @@ import os import simplestreams.contentsource as cs import simplestreams.util as util +from simplestreams import checksum_util from simplestreams.log import LOG READ_BUFFER_SIZE = 1024 * 10 @@ -88,10 +89,6 @@ class FileStore(ObjectStore): def insert(self, path, reader, checksums=None, mutable=True, size=None, sparse=False): - zeros = None - if sparse is True: - zeros = '\0' * self.read_size - wpath = self._fullpath(path) if os.path.isfile(wpath): if not mutable: @@ -102,17 +99,27 @@ class FileStore(ObjectStore): read_size=self.read_size): return - cksum = util.checksummer(checksums) + zeros = None + if sparse is True: + zeros = '\0' * self.read_size + + cksum = checksum_util.checksummer(checksums) out_d = os.path.dirname(wpath) partfile = os.path.join(out_d, "%s.part" % os.path.basename(wpath)) util.mkdir_p(out_d) orig_part_size = 0 + reader_does_checksum = ( + isinstance(reader, cs.ChecksummingContentSource) and + cksum.algorithm == reader.algorithm) if os.path.exists(partfile): try: orig_part_size = os.path.getsize(partfile) - reader.set_start_pos(orig_part_size) + if reader_does_checksum: + reader.resume(orig_part_size, cksum) + else: + reader.set_start_pos(orig_part_size) LOG.debug("resuming partial (%s) download of '%s' from '%s'", orig_part_size, path, partfile) @@ -131,7 +138,10 @@ class FileStore(ObjectStore): with open(partfile, "ab") as wfp: while True: - buf = reader.read(self.read_size) + try: + buf = reader.read(self.read_size) + except checksum_util.InvalidChecksum: + break buflen = len(buf) if (buflen != self.read_size and zeros is not None and zeros[0:buflen] == buf): @@ -140,7 +150,9 @@ class FileStore(ObjectStore): wfp.seek(wfp.tell() + buflen) else: wfp.write(buf) - cksum.update(buf) + + if not reader_does_checksum: + cksum.update(buf) if size is not None: if self.complete_callback: @@ -156,14 +168,19 @@ class FileStore(ObjectStore): if zeros is not None: wfp.truncate(wfp.tell()) - if not cksum.check(): - os.unlink(partfile) - if orig_part_size: - LOG.warn("resumed download of '%s' had bad checksum.", path) - - msg = "unexpected checksum '%s' on %s (found: %s expected: %s)" - raise Exception(msg % (cksum.algorithm, path, - cksum.hexdigest(), cksum.expected)) + resume_msg = "resumed download of '%s' had bad checksum." % path + if reader_does_checksum: + if not reader.check(): + os.unlink(partfile) + if orig_part_size: + LOG.warn(resume_msg) + raise checksum_util.invalid_checksum_for_reader(reader) + else: + if not cksum.check(): + os.unlink(partfile) + if orig_part_size: + LOG.warn(resume_msg) + raise checksum_util.InvalidChecksum(path=path, cksum=cksum) os.rename(partfile, wpath) def remove(self, path): @@ -194,8 +211,8 @@ def has_valid_checksum(path, reader, che read_size=READ_BUFFER_SIZE): if checksums is None: return False - cksum = util.checksummer(checksums) try: + cksum = checksum_util.SafeCheckSummer(checksums) with reader(path) as rfp: while True: buf = rfp.read(read_size) --- a/simplestreams/util.py +++ b/simplestreams/util.py @@ -16,7 +16,6 @@ # along with Simplestreams. If not, see . import errno -import hashlib import os import re import subprocess @@ -25,13 +24,9 @@ import time import json import simplestreams.contentsource as cs +import simplestreams.checksum_util as checksum_util from simplestreams.log import LOG -try: - ALGORITHMS = list(getattr(hashlib, 'algorithms')) -except AttributeError: - ALGORITHMS = list(hashlib.algorithms_available) # pylint: disable=E1101 - ALIASNAME = "_aliases" PGP_SIGNED_MESSAGE_HEADER = "-----BEGIN PGP SIGNED MESSAGE-----" @@ -39,7 +34,6 @@ PGP_SIGNATURE_HEADER = "-----BEGIN PGP S PGP_SIGNATURE_FOOTER = "-----END PGP SIGNATURE-----" _UNSET = object() -CHECKSUMS = ("md5", "sha256", "sha512") READ_SIZE = (1024 * 10) @@ -313,44 +307,6 @@ def timestamp(ts=None): return time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime(ts)) -def item_checksums(item): - return {k: item[k] for k in CHECKSUMS if k in item} - - -class checksummer(object): - _hasher = None - algorithm = None - expected = None - - def __init__(self, checksums): - # expects a dict of hashname/value - if not checksums: - self._hasher = None - return - for meth in CHECKSUMS: - if meth in checksums and meth in ALGORITHMS: - self._hasher = hashlib.new(meth) - self.algorithm = meth - - self.expected = checksums.get(self.algorithm, None) - - if not self._hasher: - raise TypeError("Unable to find suitable hash algorithm") - - def update(self, data): - if self._hasher is None: - return - self._hasher.update(data) - - def hexdigest(self): - if self._hasher is None: - return None - return self._hasher.hexdigest() - - def check(self): - return (self.expected is None or self.expected == self.hexdigest()) - - def move_dups(src, target, sticky=None): # given src = {e1: {a:a, b:c}, e2: {a:a, b:d, e:f}} # update target with {a:a}, and delete 'a' from entries in dict1 @@ -556,4 +512,11 @@ def path_from_mirror_url(mirror, path): return (mirror, path) + +# these are legacy +CHECKSUMS = checksum_util.CHECKSUMS +item_checksums = checksum_util.item_checksums +ALGORITHMS = checksum_util.ALGORITHMS + + # vi: ts=4 expandtab --- /dev/null +++ b/tests/unittests/test_badmirrors.py @@ -0,0 +1,138 @@ +from unittest import TestCase +from tests.testutil import get_mirror_reader +from simplestreams.mirrors import ( + ObjectStoreMirrorWriter, ObjectStoreMirrorReader) +from simplestreams.objectstores import MemoryObjectStore +from simplestreams import util +from simplestreams import checksum_util +from simplestreams import mirrors + + +class TestBadDataSources(TestCase): + """Test of Bad Data in a datasource.""" + + dlpath = "streams/v1/download.json" + pedigree = ("com.example:product1", "20150915", "item1") + item_path = "product1/20150915/text.txt" + example = "minimal" + + def setUp(self): + self.src = self.get_clean_src(self.example, path=self.dlpath) + self.target = ObjectStoreMirrorWriter( + config={}, objectstore=MemoryObjectStore()) + + def get_clean_src(self, exname, path): + good_src = get_mirror_reader(exname) + objectstore = MemoryObjectStore(None) + target = ObjectStoreMirrorWriter(config={}, objectstore=objectstore) + target.sync(good_src, path) + + # clean the .data out of the mirror so it doesn't get read + keys = list(objectstore.data.keys()) + for k in keys: + if k.startswith(".data"): + del objectstore.data[k] + + return ObjectStoreMirrorReader( + objectstore=objectstore, policy=lambda content, path: content) + + def test_sanity_valid(self): + # verify that the tests are fine on expected pass + _moditem(self.src, self.dlpath, self.pedigree, lambda c: c) + self.target.sync(self.src, self.dlpath) + + def test_larger_size_causes_bad_checksum(self): + def size_plus_1(item): + item['size'] = int(item['size']) + 1 + return item + + _moditem(self.src, self.dlpath, self.pedigree, size_plus_1) + self.assertRaises(checksum_util.InvalidChecksum, + self.target.sync, self.src, self.dlpath) + + def test_smaller_size_causes_bad_checksum(self): + def size_minus_1(item): + item['size'] = int(item['size']) - 1 + return item + _moditem(self.src, self.dlpath, self.pedigree, size_minus_1) + self.assertRaises(checksum_util.InvalidChecksum, + self.target.sync, self.src, self.dlpath) + + def test_too_much_content_causes_bad_checksum(self): + self.src.objectstore.data[self.item_path] += b"extra" + self.assertRaises(checksum_util.InvalidChecksum, + self.target.sync, self.src, self.dlpath) + + def test_too_little_content_causes_bad_checksum(self): + orig = self.src.objectstore.data[self.item_path] + self.src.objectstore.data[self.item_path] = orig[0:-1] + self.assertRaises(checksum_util.InvalidChecksum, + self.target.sync, self.src, self.dlpath) + + def test_busted_checksum_causes_bad_checksum(self): + def break_checksum(item): + chars = "0123456789abcdef" + orig = item['sha256'] + item['sha256'] = ''.join( + [chars[(chars.find(c) + 1) % len(chars)] for c in orig]) + return item + + _moditem(self.src, self.dlpath, self.pedigree, break_checksum) + self.assertRaises(checksum_util.InvalidChecksum, + self.target.sync, self.src, self.dlpath) + + def test_changed_content_causes_bad_checksum(self): + # correct size but different content should raise bad checksum + self.src.objectstore.data[self.item_path] = ''.join( + ["x" for c in self.src.objectstore.data[self.item_path]]) + self.assertRaises(checksum_util.InvalidChecksum, + self.target.sync, self.src, self.dlpath) + + def test_no_checksums_cause_bad_checksum(self): + def del_checksums(item): + for c in checksum_util.item_checksums(item).keys(): + del item[c] + return item + + _moditem(self.src, self.dlpath, self.pedigree, del_checksums) + with _patched_missing_sum("fail"): + self.assertRaises(checksum_util.InvalidChecksum, + self.target.sync, self.src, self.dlpath) + + def test_missing_size_causes_bad_checksum(self): + def del_size(item): + del item['size'] + return item + + _moditem(self.src, self.dlpath, self.pedigree, del_size) + with _patched_missing_sum("fail"): + self.assertRaises(checksum_util.InvalidChecksum, + self.target.sync, self.src, self.dlpath) + + +class _patched_missing_sum(object): + """This patches the legacy mode for missing checksum info so + that it behaves like the new code path. Thus we can make + the test run correctly""" + def __init__(self, mode="fail"): + self.mode = mode + + def __enter__(self): + self.modmcb = getattr(mirrors, '_missing_cksum_behavior', {}) + self.orig = self.modmcb.copy() + if self.modmcb: + self.modmcb['mode'] = self.mode + return self + + def __exit__(self, type, value, traceback): + self.patch = self.orig + + +def _moditem(src, path, pedigree, modfunc): + # load the products data at 'path' in 'src' mirror, then call modfunc + # on the data found at pedigree. and store the updated data. + sobj = src.objectstore + tree = util.load_content(sobj.source(path).read()) + item = util.products_exdata(tree, pedigree, insert_fieldnames=False) + util.products_set(tree, modfunc(item), pedigree) + sobj.insert_content(path, util.dump_data(tree)) --- a/tests/unittests/test_mirrorwriters.py +++ b/tests/unittests/test_mirrorwriters.py @@ -2,10 +2,14 @@ from tests.testutil import get_mirror_re from simplestreams.mirrors import DryRunMirrorWriter from simplestreams.objectstores import MemoryObjectStore -def test_DryRunMirrorWriter_foocloud_no_filters(): - src = get_mirror_reader("foocloud") - config = {} - objectstore = MemoryObjectStore(None) - target = DryRunMirrorWriter(config, objectstore) - target.sync(src, "streams/v1/index.json") - assert target.size == 886 +from unittest import TestCase + + +class TestMirrorWriters(TestCase): + def test_DryRunMirrorWriter_foocloud_no_filters(self): + src = get_mirror_reader("foocloud") + config = {} + objectstore = MemoryObjectStore(None) + target = DryRunMirrorWriter(config, objectstore) + target.sync(src, "streams/v1/index.json") + self.assertEqual(1277, target.size) debian/patches/series0000664000000000000000000000027212601265657012047 0ustar 1-add-item-filter-to-glancemirror glance-upload-i386-image-as-i686.patch lp1487004-use-checksumming-reader.patch lp1487004-sru-safetynet.patch lp1499749-export-checksummer-in-util.patch debian/patches/lp1499749-export-checksummer-in-util.patch0000664000000000000000000000177312601265657020272 0ustar Author: Scott Moser Bug: https://bugs.launchpad.net/bugs/1499749 Description: export checksummer in simplestreams.util Applied-Upstream: rev 401 ------------------------------------------------------------ revno: 401 fixes bug: https://launchpad.net/bugs/1499749 committer: Scott Moser branch nick: trunk timestamp: Fri 2015-09-25 10:49:02 -0400 message: export checksummer in simplestreams.util When moving the checksumming things out of checksum_util, I added names for CHECKSUMS, item_checksums and ALGORITHMS, but failed to get checksummer. Simply export checksummer in util, pointing it to checksum_util.util === modified file 'simplestreams/util.py' --- a/simplestreams/util.py +++ b/simplestreams/util.py @@ -517,6 +517,7 @@ def path_from_mirror_url(mirror, path): CHECKSUMS = checksum_util.CHECKSUMS item_checksums = checksum_util.item_checksums ALGORITHMS = checksum_util.ALGORITHMS +checksummer = checksum_util.checksummer # vi: ts=4 expandtab debian/patches/lp1487004-sru-safetynet.patch0000664000000000000000000000755212600634724015647 0ustar Author: Scott Moser Bug: https://bugs.launchpad.net/bugs/1487004 Description: provide a path to old behavior via environment variable . In order to provide a safe SRU backport that has minimal breakage we offer the ability to set an environment variable to make enable the old behavior. . SS_MISSING_ITEM_CHECKSUM_BEHAVIOR= . values are: silent: behave exactly as before. No checksumming is done, no warnings are emitted. The consumer of the contentsource must check checksums. warn: log messages at WARN level (same as default/unset) fail: the new behavior. raise an InvalidChecksum exception. . Note: only legacy versions of this library respect the SS_MISSING_ITEM_CHECKSUM_BEHAVIOR environment variable. All other versions require checksums and size on items.""" --- a/simplestreams/mirrors/__init__.py +++ b/simplestreams/mirrors/__init__.py @@ -14,6 +14,9 @@ # # You should have received a copy of the GNU Affero General Public License # along with Simplestreams. If not, see . +import os +import sys + import errno import io import json @@ -317,7 +320,7 @@ class BasicMirrorWriter(MirrorWriter): if ipath and reader: if self.checksumming_reader: flat = util.products_exdata(src, pgree) - ipath_cs = cs.ChecksummingContentSource( + ipath_cs = _maybe_checksumming_cs( csrc=reader.source(ipath), size=flat.get('size'), checksums=checksum_util.item_checksums(flat)) @@ -566,4 +569,58 @@ def check_tree_paths(tree, fmt=None): for content_id in index: util.assert_safe_path(index[content_id].get('path')) + +def _maybe_checksumming_cs(csrc, size, checksums): + """wraps calls to ChecksummingContentSource consulting environment + + SS_MISSING_ITEM_CHECKSUM_BEHAVIOR= + + values are: + silent: behave exactly as before. No checksumming is done, + no warnings are emitted. The consumer of the + contentsource must check checksums. + warn: log messages at WARN level (same as default/unset) + fail: the new behavior. raise an InvalidChecksum exception. + + Note: only legacy versions of this library respect + the SS_MISSING_ITEM_CHECKSUM_BEHAVIOR environment variable. + All other versions require checksums and size on items.""" + + def handle_exception(e, cs): + mode = _missing_cksum_behavior['mode'] + if (not _missing_cksum_behavior['messaged'] and + mode not in ('silent', 'fail')): + sys.stderr.write( + "WARNING: consider setting environment variable " + "'SS_MISSING_ITEM_CHECKSUM_BEHAVIOR' to " + "'silent', 'warn', or 'fail'. See " + "https://bugs.launchpad.net/bugs/1487004 for more info.") + _missing_cksum_behavior['messaged'] = True + + if mode == 'silent': + return cs + elif mode in ("warn", "unset"): + LOG.warn(e) + return cs + else: + raise e + + try: + return cs.ChecksummingContentSource( + csrc=csrc, size=size, checksums=checksums) + except ValueError as e: + return handle_exception(e, csrc) + + +_missing_cksum_behavior = { + 'mode': os.environ.get("SS_MISSING_ITEM_CHECKSUM_BEHAVIOR", "unset"), + 'messaged': False, +} +if _missing_cksum_behavior['mode'] not in ("warn", "fail", "silent", "unset"): + raise ValueError( + "SS_MISSING_ITEM_CHECKSUM_BEHAVIOR (%s) must be one of:" + "'warn', 'fail', 'silent', 'unset'." % + _missing_cksum_behavior['mode']) + + # vi: ts=4 expandtab debian/patches/1-add-item-filter-to-glancemirror0000664000000000000000000000776012524713763016775 0ustar Description: add filter support, expose in sstream-mirror-glance Forwarded: not-needed Origin: upstream, http://bazaar.launchpad.net/~smoser/simplestreams/trunk/revision/347 Bug-Ubuntu: https://bugs.launchpad.net/ubuntu/+source/simplestreams/+bug/1339842 Author: Michael McCracken === modified file 'simplestreams/filters.py' --- a/simplestreams/filters.py +++ b/simplestreams/filters.py @@ -64,7 +64,12 @@ def filter_item(filters, data, src, pedigree): - data = util.products_exdata(src, pedigree) + "Apply filter list to a products entity. Flatten before doing so." + return filter_dict(filters, util.products_exdata(src, pedigree)) + + +def filter_dict(filters, data): + "Apply filter list to dict. Does not flatten." for f in filters: if not f.matches(data): return False --- a/simplestreams/mirrors/glance.py +++ b/simplestreams/mirrors/glance.py @@ -15,6 +15,7 @@ # You should have received a copy of the GNU Affero General Public License # along with Simplestreams. If not, see . +import simplestreams.filters as filters import simplestreams.mirrors as mirrors import simplestreams.util as util import simplestreams.openstack as openstack @@ -44,6 +45,17 @@ name_prefix=None): super(GlanceMirror, self).__init__(config=config) + self.item_filters = self.config.get('item_filters', []) + if len(self.item_filters) == 0: + self.item_filters = ['ftype~(disk1.img|disk.img)', + 'arch~(x86_64|amd64|i386)'] + self.item_filters = filters.get_filters(self.item_filters) + + self.index_filters = self.config.get('index_filters', []) + if len(self.index_filters) == 0: + self.index_filters = ['datatype=image-downloads'] + self.index_filters = filters.get_filters(self.index_filters) + self.loaded_content = {} self.store = objectstore @@ -134,9 +146,7 @@ return glance_t def filter_item(self, data, src, target, pedigree): - flat = util.products_exdata(src, pedigree, include_top=False) - return (flat.get('ftype') in ('disk1.img', 'disk.img') and - flat.get('arch') in ('x86_64', 'amd64', 'i386')) + return filters.filter_item(self.item_filters, data, src, pedigree) def insert_item(self, data, src, target, pedigree, contentsource): flat = util.products_exdata(src, pedigree, include_top=False) @@ -214,7 +224,7 @@ self.gclient.images.delete(data['id']) def filter_index_entry(self, data, src, pedigree): - return data.get('datatype') in ("image-downloads", None) + return filters.filter_dict(self.index_filters, data) def insert_products(self, path, target, content): if not self.store: --- a/tools/sstream-mirror-glance +++ b/tools/sstream-mirror-glance @@ -20,6 +20,7 @@ # glanceclient) are not python3. # import argparse +import logging import os.path import sys @@ -79,6 +80,14 @@ parser.add_argument('source_mirror') parser.add_argument('path', nargs='?', default="streams/v1/index.sjson") + parser.add_argument('--item-filter', action='append', default=[], + dest="item_filters", + help="Filter expression for mirrored items. " + "Multiple filter arguments can be specified" + "and will be combined with logical AND. " + "Expressions are key[!]=literal_string " + "or key[!]~regexp.") + args = parser.parse_args() modify_hook = None @@ -87,7 +96,8 @@ mirror_config = {'max_items': args.max, 'keep_items': args.keep, 'cloud_name': args.cloud_name, - 'modify_hook': modify_hook} + 'modify_hook': modify_hook, + 'item_filters': args.item_filters} def policy(content, path): # pylint: disable=W0613 if args.path.endswith('sjson'): debian/patches/glance-upload-i386-image-as-i686.patch0000664000000000000000000000125012524714147017222 0ustar Origin: upstream, revno 366 Bug: https://bugs.launchpad.net/ubuntu/+source/simplestreams/+bug/1454775 Description: GlanceMirror: on arch of i386, set it to i686 in glance. . basically with i386, nova by default won't schedule an instance as the hypervisors all report i686. === modified file 'o/simplestreams/mirrors/glance.py' --- a/simplestreams/mirrors/glance.py +++ b/simplestreams/mirrors/glance.py @@ -171,6 +171,8 @@ class GlanceMirror(mirrors.BasicMirrorWr arch = flat.get('arch') if arch == "amd64": arch = "x86_64" + if arch == "i386": + arch = "i686" if arch: props['architecture'] = arch debian/copyright0000664000000000000000000000115612524713763011140 0ustar Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Upstream-Name: simplestreams Upstream-Contact: Scott Moser Source: https://launchpad.net/simplestreams Files: * Copyright: 2013, Canonical Ltd. License: AGPLv3 GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007 . Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. . The complete text of the AGPL version 3 can be seen in http://www.gnu.org/licenses/agpl-3.0.html debian/python3-simplestreams.install0000664000000000000000000000005412524713763015063 0ustar usr/lib/python3*/*-packages/simplestreams/* debian/rules0000775000000000000000000000122512524713763010262 0ustar #!/usr/bin/make -f PYVERS := $(shell pyversions -r) PY3VERS := $(shell py3versions -r) %: dh $@ --with=python2,python3 override_dh_auto_install: dh_auto_install set -ex; for python in $(PY3VERS) $(PYVERS); do \ $$python setup.py build --executable=/usr/bin/python3 && \ $$python setup.py install --root=$(CURDIR)/debian/tmp --install-layout=deb; \ done # there:are no packages for python3-{boto, swiftclient, # glanceclient, keystonclient}, so do not package the bits # that would depend on them. for bad in openstack mirrors/glance objectstores/swift objectstores/s3; do \ rm $(CURDIR)/debian/tmp/usr/lib/python3/*/simplestreams/$$bad.py; done