zope.catalog-3.8.2/0000775000177100020040000000000011665177633015257 5ustar menesismenesis00000000000000zope.catalog-3.8.2/setup.py0000664000177100020040000000552511665177362016777 0ustar menesismenesis00000000000000############################################################################## # # Copyright (c) 2006 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## # This package is developed by the Zope Toolkit project, documented here: # http://docs.zope.org/zopetoolkit # When developing and releasing this package, please follow the documented # Zope Toolkit policies as described by this documentation. ############################################################################## """Setup for zope.catalog package """ import os from setuptools import setup, find_packages def read(*rnames): return open(os.path.join(os.path.dirname(__file__), *rnames)).read() setup(name = 'zope.catalog', version='3.8.2', author='Zope Corporation and Contributors', author_email='zope-dev@zope.org', description='Cataloging and Indexing Framework for the Zope Toolkit', long_description=( read('README.txt') + '\n\n' + 'Detailed Documentation\n' '**********************\n' + '\n\n' + read('src', 'zope', 'catalog', 'README.txt') + '\n\n' + read('src', 'zope', 'catalog', 'event.txt') + '\n\n' + read('CHANGES.txt') ), keywords = "zope3 catalog index", classifiers = [ 'Development Status :: 5 - Production/Stable', 'Environment :: Web Environment', 'Intended Audience :: Developers', 'License :: OSI Approved :: Zope Public License', 'Programming Language :: Python', 'Natural Language :: English', 'Operating System :: OS Independent', 'Topic :: Internet :: WWW/HTTP', 'Framework :: Zope3'], url='http://pypi.python.org/pypi/zope.catalog', license='ZPL 2.1', packages=find_packages('src'), package_dir = {'': 'src'}, namespace_packages=['zope'], extras_require = dict( test=['zope.testing', 'zope.site', ]), install_requires = [ 'setuptools', 'ZODB3', 'zope.annotation', 'zope.intid', 'zope.component', 'zope.container', 'zope.index>=3.5.0', 'zope.interface', 'zope.lifecycleevent', 'zope.location', 'zope.schema', ], include_package_data = True, zip_safe = False, ) zope.catalog-3.8.2/CHANGES.txt0000664000177100020040000000062511665177362017072 0ustar menesismenesis00000000000000======= CHANGES ======= 3.8.2 (2011-11-29) ------------------ - Conform to repository policy. 3.8.1 (2009-12-27) ------------------ - Removed ``zope.app.testing`` dependency. 3.8.0 (2009-02-01) ------------------ - Move core functionality from ``zope.app.catalog`` to this package. The ``zope.app.catalog`` package now only contains ZMI-related browser views and backward-compatibility imports. zope.catalog-3.8.2/PKG-INFO0000664000177100020040000004710411665177633016362 0ustar menesismenesis00000000000000Metadata-Version: 1.0 Name: zope.catalog Version: 3.8.2 Summary: Cataloging and Indexing Framework for the Zope Toolkit Home-page: http://pypi.python.org/pypi/zope.catalog Author: Zope Corporation and Contributors Author-email: zope-dev@zope.org License: ZPL 2.1 Description: Catalogs provide management of collections of related indexes with a basic search algorithm. Detailed Documentation ********************** Catalogs ======== Catalogs provide management of collections of related indexes with a basic search algorithm. Let's look at an example: >>> from zope.catalog.catalog import Catalog >>> cat = Catalog() We can add catalog indexes to catalogs. A catalog index is, among other things, an attribute index. It indexes attributes of objects. To see how this works, we'll create a demonstration attribute index. Our attribute index will simply keep track of objects that have a given attribute value. The `catalog` package provides an attribute-index mix-in class that is meant to work with a base indexing class. First, we'll write the base index class: >>> import persistent, BTrees.OOBTree, BTrees.IFBTree, BTrees.IOBTree >>> import zope.interface, zope.index.interfaces >>> class BaseIndex(persistent.Persistent): ... zope.interface.implements( ... zope.index.interfaces.IInjection, ... zope.index.interfaces.IIndexSearch, ... zope.index.interfaces.IIndexSort, ... ) ... ... def clear(self): ... self.forward = BTrees.OOBTree.OOBTree() ... self.backward = BTrees.IOBTree.IOBTree() ... ... __init__ = clear ... ... def index_doc(self, docid, value): ... if docid in self.backward: ... self.unindex_doc(docid) ... self.backward[docid] = value ... ... set = self.forward.get(value) ... if set is None: ... set = BTrees.IFBTree.IFTreeSet() ... self.forward[value] = set ... set.insert(docid) ... ... def unindex_doc(self, docid): ... value = self.backward.get(docid) ... if value is None: ... return ... self.forward[value].remove(docid) ... del self.backward[docid] ... ... def apply(self, value): ... set = self.forward.get(value) ... if set is None: ... set = BTrees.IFBTree.IFTreeSet() ... return set ... ... def sort(self, docids, limit=None, reverse=False): ... for i, docid in enumerate(sorted(docids, key=self.backward.get, reverse=reverse)): ... yield docid ... if limit and i >= (limit - 1): ... break The class implements `IInjection` to allow values to be indexed and unindexed and `IIndexSearch` to support searching via the `apply` method. Now, we can use the AttributeIndex mixin to make this an attribute index: >>> import zope.catalog.attribute >>> import zope.catalog.interfaces >>> import zope.container.contained >>> class Index(zope.catalog.attribute.AttributeIndex, ... BaseIndex, ... zope.container.contained.Contained, ... ): ... zope.interface.implements(zope.catalog.interfaces.ICatalogIndex) Unfortunately, because of the way we currently handle containment constraints, we have to provide `ICatalogIndex`, which extends `IContained`. We subclass `Contained` to get an implementation for `IContained`. Now let's add some of these indexes to our catalog. Let's create some indexes. First we'll define some interfaces providing data to index: >>> class IFavoriteColor(zope.interface.Interface): ... color = zope.interface.Attribute("Favorite color") >>> class IPerson(zope.interface.Interface): ... def age(): ... """Return the person's age, in years""" We'll create color and age indexes: >>> cat['color'] = Index('color', IFavoriteColor) >>> cat['age'] = Index('age', IPerson, True) >>> cat['size'] = Index('sz') The indexes are created with: - the name of the of the attribute to index - the interface defining the attribute, and - a flag indicating whether the attribute should be called, which defaults to false. If an interface is provided, then we'll only be able to index an object if it can be adapted to the interface, otherwise, we'll simply try to get the attribute from the object. If the attribute isn't present, then we'll ignore the object. Now, let's create some objects and index them: >>> class Person: ... zope.interface.implements(IPerson) ... def __init__(self, age): ... self._age = age ... def age(self): ... return self._age >>> class Discriminating: ... zope.interface.implements(IFavoriteColor) ... def __init__(self, color): ... self.color = color >>> class DiscriminatingPerson(Discriminating, Person): ... def __init__(self, age, color): ... Discriminating.__init__(self, color) ... Person.__init__(self, age) >>> class Whatever: ... def __init__(self, **kw): ... self.__dict__.update(kw) >>> o1 = Person(10) >>> o2 = DiscriminatingPerson(20, 'red') >>> o3 = Discriminating('blue') >>> o4 = Whatever(a=10, c='blue', sz=5) >>> o5 = Whatever(a=20, c='red', sz=6) >>> o6 = DiscriminatingPerson(10, 'blue') >>> cat.index_doc(1, o1) >>> cat.index_doc(2, o2) >>> cat.index_doc(3, o3) >>> cat.index_doc(4, o4) >>> cat.index_doc(5, o5) >>> cat.index_doc(6, o6) We search by providing query mapping objects that have a key for every index we want to use: >>> list(cat.apply({'age': 10})) [1, 6] >>> list(cat.apply({'age': 10, 'color': 'blue'})) [6] >>> list(cat.apply({'age': 10, 'color': 'blue', 'size': 5})) [] >>> list(cat.apply({'size': 5})) [4] We can unindex objects: >>> cat.unindex_doc(4) >>> list(cat.apply({'size': 5})) [] and reindex objects: >>> o5.sz = 5 >>> cat.index_doc(5, o5) >>> list(cat.apply({'size': 5})) [5] If we clear the catalog, we'll clear all of the indexes: >>> cat.clear() >>> [len(index.forward) for index in cat.values()] [0, 0, 0] Note that you don't have to use the catalog's search methods. You can access its indexes directly, since the catalog is a mapping: >>> [(name, cat[name].field_name) for name in cat] [(u'age', 'age'), (u'color', 'color'), (u'size', 'sz')] Catalogs work with int-id utilities, which are responsible for maintaining id <-> object mappings. To see how this works, we'll create a utility to work with our catalog: >>> import zope.intid.interfaces >>> class Ids: ... zope.interface.implements(zope.intid.interfaces.IIntIds) ... def __init__(self, data): ... self.data = data ... def getObject(self, id): ... return self.data[id] ... def __iter__(self): ... return self.data.iterkeys() >>> ids = Ids({1: o1, 2: o2, 3: o3, 4: o4, 5: o5, 6: o6}) >>> from zope.component import provideUtility >>> provideUtility(ids, zope.intid.interfaces.IIntIds) With this utility in place, catalogs can recompute indexes: >>> cat.updateIndex(cat['size']) >>> list(cat.apply({'size': 5})) [4, 5] Of course, that only updates *that* index: >>> list(cat.apply({'age': 10})) [] We can update all of the indexes: >>> cat.updateIndexes() >>> list(cat.apply({'age': 10})) [1, 6] >>> list(cat.apply({'color': 'red'})) [2] There's an alternate search interface that returns "result sets". Result sets provide access to objects, rather than object ids: >>> result = cat.searchResults(size=5) >>> len(result) 2 >>> list(result) == [o4, o5] True The searchResults method also provides a way to sort, limit and reverse results. When not using sorting, limiting and reversing are done by simple slicing and list reversing. >>> list(cat.searchResults(size=5, _reverse=True)) == [o5, o4] True >>> list(cat.searchResults(size=5, _limit=1)) == [o4] True >>> list(cat.searchResults(size=5, _limit=1, _reverse=True)) == [o5] True However, when using sorting by index, the limit and reverse parameters are passed to the index ``sort`` method so it can do it efficiently. Let's index more objects to work with: >>> o7 = DiscriminatingPerson(7, 'blue') >>> o8 = DiscriminatingPerson(3, 'blue') >>> o9 = DiscriminatingPerson(14, 'blue') >>> o10 = DiscriminatingPerson(1, 'blue') >>> ids.data.update({7: o7, 8: o8, 9: o9, 10: o10}) >>> cat.index_doc(7, o7) >>> cat.index_doc(8, o8) >>> cat.index_doc(9, o9) >>> cat.index_doc(10, o10) Now we can search all people who like blue, ordered by age: >>> results = list(cat.searchResults(color='blue', _sort_index='age')) >>> results == [o3, o10, o8, o7, o6, o9] True >>> results = list(cat.searchResults(color='blue', _sort_index='age', _limit=3)) >>> results == [o3, o10, o8] True >>> results = list(cat.searchResults(color='blue', _sort_index='age', _reverse=True)) >>> results == [o9, o6, o7, o8, o10, o3] True >>> results = list(cat.searchResults(color='blue', _sort_index='age', _reverse=True, _limit=4)) >>> results == [o9, o6, o7, o8] True The index example we looked at didn't provide document scores. Simple indexes normally don't, but more complex indexes might give results scores, according to how closely a document matches a query. Let's create a new index, a "keyword index" that indexes sequences of values: >>> class BaseKeywordIndex(persistent.Persistent): ... zope.interface.implements( ... zope.index.interfaces.IInjection, ... zope.index.interfaces.IIndexSearch, ... ) ... ... def clear(self): ... self.forward = BTrees.OOBTree.OOBTree() ... self.backward = BTrees.IOBTree.IOBTree() ... ... __init__ = clear ... ... def index_doc(self, docid, values): ... if docid in self.backward: ... self.unindex_doc(docid) ... self.backward[docid] = values ... ... for value in values: ... set = self.forward.get(value) ... if set is None: ... set = BTrees.IFBTree.IFTreeSet() ... self.forward[value] = set ... set.insert(docid) ... ... def unindex_doc(self, docid): ... values = self.backward.get(docid) ... if values is None: ... return ... for value in values: ... self.forward[value].remove(docid) ... del self.backward[docid] ... ... def apply(self, values): ... result = BTrees.IFBTree.IFBucket() ... for value in values: ... set = self.forward.get(value) ... if set is not None: ... _, result = BTrees.IFBTree.weightedUnion(result, set) ... return result >>> class KeywordIndex(zope.catalog.attribute.AttributeIndex, ... BaseKeywordIndex, ... zope.container.contained.Contained, ... ): ... zope.interface.implements(zope.catalog.interfaces.ICatalogIndex) Now, we'll add a hobbies index: >>> cat['hobbies'] = KeywordIndex('hobbies') >>> o1.hobbies = 'camping', 'music' >>> o2.hobbies = 'hacking', 'sailing' >>> o3.hobbies = 'music', 'camping', 'sailing' >>> o6.hobbies = 'cooking', 'dancing' >>> cat.updateIndexes() When we apply the catalog: >>> cat.apply({'hobbies': ['music', 'camping', 'sailing']}) BTrees.IFBTree.IFBucket([(1, 2.0), (2, 1.0), (3, 3.0)]) We found objects 1-3, because they each contained at least some of the words in the query. The scores represent the number of words that matched. If we also include age: >>> cat.apply({'hobbies': ['music', 'camping', 'sailing'], 'age': 10}) BTrees.IFBTree.IFBucket([(1, 3.0)]) The score increased because we used an additional index. If an index doesn't provide scores, scores of 1.0 are assumed. ============================== Automatic indexing with events ============================== In order to automatically keep the catalog up-to-date any objects that are added to a intid utility are indexed automatically. Also when an object gets modified it is reindexed by listening to IObjectModified events. Let us create a fake catalog to demonstrate this behaviour. We only need to implement the index_doc method for this test. >>> from zope.catalog.interfaces import ICatalog >>> from zope import interface, component >>> class FakeCatalog(object): ... indexed = [] ... interface.implements(ICatalog) ... def index_doc(self, docid, obj): ... self.indexed.append((docid, obj)) >>> cat = FakeCatalog() >>> component.provideUtility(cat) We also need an intid util and a keyreference adapter. >>> from zope.intid import IntIds >>> from zope.intid.interfaces import IIntIds >>> intids = IntIds() >>> component.provideUtility(intids, IIntIds) >>> from zope.keyreference.testing import SimpleKeyReference >>> component.provideAdapter(SimpleKeyReference) >>> from zope.container.contained import Contained >>> class Dummy(Contained): ... def __init__(self, name): ... self.__name__ = name ... def __repr__(self): ... return '' % self.__name__ We have a subscriber to IIntidAddedEvent. >>> from zope.catalog import catalog >>> from zope.intid.interfaces import IntIdAddedEvent >>> d1 = Dummy(u'one') >>> id1 = intids.register(d1) >>> catalog.indexDocSubscriber(IntIdAddedEvent(d1, None)) Now we have indexed the object. >>> cat.indexed.pop() (..., ) When an object is modified an objectmodified event should be fired by the application. Here is the handler for such an event. >>> from zope.lifecycleevent import ObjectModifiedEvent >>> catalog.reindexDocSubscriber(ObjectModifiedEvent(d1)) >>> len(cat.indexed) 1 >>> cat.indexed.pop() (..., ) Preventing automatic indexing ============================= Sometimes it is not accurate to automatically index an object. For example when a lot of indexes are in the catalog and only specific indexes needs to be updated. There are marker interfaces to achieve this. >>> from zope.catalog.interfaces import INoAutoIndex If an object provides this interface it is not automatically indexed. >>> interface.alsoProvides(d1, INoAutoIndex) >>> catalog.indexDocSubscriber(IntIdAddedEvent(d1, None)) >>> len(cat.indexed) 0 >>> from zope.catalog.interfaces import INoAutoReindex If an object provides this interface it is not automatically reindexed. >>> interface.alsoProvides(d1, INoAutoReindex) >>> catalog.reindexDocSubscriber(ObjectModifiedEvent(d1)) >>> len(cat.indexed) 0 ======= CHANGES ======= 3.8.2 (2011-11-29) ------------------ - Conform to repository policy. 3.8.1 (2009-12-27) ------------------ - Removed ``zope.app.testing`` dependency. 3.8.0 (2009-02-01) ------------------ - Move core functionality from ``zope.app.catalog`` to this package. The ``zope.app.catalog`` package now only contains ZMI-related browser views and backward-compatibility imports. Keywords: zope3 catalog index Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Web Environment Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Zope Public License Classifier: Programming Language :: Python Classifier: Natural Language :: English Classifier: Operating System :: OS Independent Classifier: Topic :: Internet :: WWW/HTTP Classifier: Framework :: Zope3 zope.catalog-3.8.2/bootstrap.py0000664000177100020040000000330211665177362017643 0ustar menesismenesis00000000000000############################################################################## # # Copyright (c) 2006 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Bootstrap a buildout-based project Simply run this script in a directory containing a buildout.cfg. The script accepts buildout command-line options, so you can use the -c option to specify an alternate configuration file. """ import os, shutil, sys, tempfile, urllib2 tmpeggs = tempfile.mkdtemp() ez = {} exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py' ).read() in ez ez['use_setuptools'](to_dir=tmpeggs, download_delay=0) import pkg_resources cmd = 'from setuptools.command.easy_install import main; main()' if sys.platform == 'win32': cmd = '"%s"' % cmd # work around spawn lamosity on windows ws = pkg_resources.working_set assert os.spawnle( os.P_WAIT, sys.executable, sys.executable, '-c', cmd, '-mqNxd', tmpeggs, 'zc.buildout', dict(os.environ, PYTHONPATH= ws.find(pkg_resources.Requirement.parse('setuptools')).location ), ) == 0 ws.add_entry(tmpeggs) ws.require('zc.buildout') import zc.buildout.buildout zc.buildout.buildout.main(sys.argv[1:] + ['bootstrap']) shutil.rmtree(tmpeggs) zope.catalog-3.8.2/COPYRIGHT.txt0000664000177100020040000000004011665177362017361 0ustar menesismenesis00000000000000Zope Foundation and Contributorszope.catalog-3.8.2/src/0000775000177100020040000000000011665177633016046 5ustar menesismenesis00000000000000zope.catalog-3.8.2/src/zope.catalog.egg-info/0000775000177100020040000000000011665177633022126 5ustar menesismenesis00000000000000zope.catalog-3.8.2/src/zope.catalog.egg-info/PKG-INFO0000664000177100020040000004710411665177625023232 0ustar menesismenesis00000000000000Metadata-Version: 1.0 Name: zope.catalog Version: 3.8.2 Summary: Cataloging and Indexing Framework for the Zope Toolkit Home-page: http://pypi.python.org/pypi/zope.catalog Author: Zope Corporation and Contributors Author-email: zope-dev@zope.org License: ZPL 2.1 Description: Catalogs provide management of collections of related indexes with a basic search algorithm. Detailed Documentation ********************** Catalogs ======== Catalogs provide management of collections of related indexes with a basic search algorithm. Let's look at an example: >>> from zope.catalog.catalog import Catalog >>> cat = Catalog() We can add catalog indexes to catalogs. A catalog index is, among other things, an attribute index. It indexes attributes of objects. To see how this works, we'll create a demonstration attribute index. Our attribute index will simply keep track of objects that have a given attribute value. The `catalog` package provides an attribute-index mix-in class that is meant to work with a base indexing class. First, we'll write the base index class: >>> import persistent, BTrees.OOBTree, BTrees.IFBTree, BTrees.IOBTree >>> import zope.interface, zope.index.interfaces >>> class BaseIndex(persistent.Persistent): ... zope.interface.implements( ... zope.index.interfaces.IInjection, ... zope.index.interfaces.IIndexSearch, ... zope.index.interfaces.IIndexSort, ... ) ... ... def clear(self): ... self.forward = BTrees.OOBTree.OOBTree() ... self.backward = BTrees.IOBTree.IOBTree() ... ... __init__ = clear ... ... def index_doc(self, docid, value): ... if docid in self.backward: ... self.unindex_doc(docid) ... self.backward[docid] = value ... ... set = self.forward.get(value) ... if set is None: ... set = BTrees.IFBTree.IFTreeSet() ... self.forward[value] = set ... set.insert(docid) ... ... def unindex_doc(self, docid): ... value = self.backward.get(docid) ... if value is None: ... return ... self.forward[value].remove(docid) ... del self.backward[docid] ... ... def apply(self, value): ... set = self.forward.get(value) ... if set is None: ... set = BTrees.IFBTree.IFTreeSet() ... return set ... ... def sort(self, docids, limit=None, reverse=False): ... for i, docid in enumerate(sorted(docids, key=self.backward.get, reverse=reverse)): ... yield docid ... if limit and i >= (limit - 1): ... break The class implements `IInjection` to allow values to be indexed and unindexed and `IIndexSearch` to support searching via the `apply` method. Now, we can use the AttributeIndex mixin to make this an attribute index: >>> import zope.catalog.attribute >>> import zope.catalog.interfaces >>> import zope.container.contained >>> class Index(zope.catalog.attribute.AttributeIndex, ... BaseIndex, ... zope.container.contained.Contained, ... ): ... zope.interface.implements(zope.catalog.interfaces.ICatalogIndex) Unfortunately, because of the way we currently handle containment constraints, we have to provide `ICatalogIndex`, which extends `IContained`. We subclass `Contained` to get an implementation for `IContained`. Now let's add some of these indexes to our catalog. Let's create some indexes. First we'll define some interfaces providing data to index: >>> class IFavoriteColor(zope.interface.Interface): ... color = zope.interface.Attribute("Favorite color") >>> class IPerson(zope.interface.Interface): ... def age(): ... """Return the person's age, in years""" We'll create color and age indexes: >>> cat['color'] = Index('color', IFavoriteColor) >>> cat['age'] = Index('age', IPerson, True) >>> cat['size'] = Index('sz') The indexes are created with: - the name of the of the attribute to index - the interface defining the attribute, and - a flag indicating whether the attribute should be called, which defaults to false. If an interface is provided, then we'll only be able to index an object if it can be adapted to the interface, otherwise, we'll simply try to get the attribute from the object. If the attribute isn't present, then we'll ignore the object. Now, let's create some objects and index them: >>> class Person: ... zope.interface.implements(IPerson) ... def __init__(self, age): ... self._age = age ... def age(self): ... return self._age >>> class Discriminating: ... zope.interface.implements(IFavoriteColor) ... def __init__(self, color): ... self.color = color >>> class DiscriminatingPerson(Discriminating, Person): ... def __init__(self, age, color): ... Discriminating.__init__(self, color) ... Person.__init__(self, age) >>> class Whatever: ... def __init__(self, **kw): ... self.__dict__.update(kw) >>> o1 = Person(10) >>> o2 = DiscriminatingPerson(20, 'red') >>> o3 = Discriminating('blue') >>> o4 = Whatever(a=10, c='blue', sz=5) >>> o5 = Whatever(a=20, c='red', sz=6) >>> o6 = DiscriminatingPerson(10, 'blue') >>> cat.index_doc(1, o1) >>> cat.index_doc(2, o2) >>> cat.index_doc(3, o3) >>> cat.index_doc(4, o4) >>> cat.index_doc(5, o5) >>> cat.index_doc(6, o6) We search by providing query mapping objects that have a key for every index we want to use: >>> list(cat.apply({'age': 10})) [1, 6] >>> list(cat.apply({'age': 10, 'color': 'blue'})) [6] >>> list(cat.apply({'age': 10, 'color': 'blue', 'size': 5})) [] >>> list(cat.apply({'size': 5})) [4] We can unindex objects: >>> cat.unindex_doc(4) >>> list(cat.apply({'size': 5})) [] and reindex objects: >>> o5.sz = 5 >>> cat.index_doc(5, o5) >>> list(cat.apply({'size': 5})) [5] If we clear the catalog, we'll clear all of the indexes: >>> cat.clear() >>> [len(index.forward) for index in cat.values()] [0, 0, 0] Note that you don't have to use the catalog's search methods. You can access its indexes directly, since the catalog is a mapping: >>> [(name, cat[name].field_name) for name in cat] [(u'age', 'age'), (u'color', 'color'), (u'size', 'sz')] Catalogs work with int-id utilities, which are responsible for maintaining id <-> object mappings. To see how this works, we'll create a utility to work with our catalog: >>> import zope.intid.interfaces >>> class Ids: ... zope.interface.implements(zope.intid.interfaces.IIntIds) ... def __init__(self, data): ... self.data = data ... def getObject(self, id): ... return self.data[id] ... def __iter__(self): ... return self.data.iterkeys() >>> ids = Ids({1: o1, 2: o2, 3: o3, 4: o4, 5: o5, 6: o6}) >>> from zope.component import provideUtility >>> provideUtility(ids, zope.intid.interfaces.IIntIds) With this utility in place, catalogs can recompute indexes: >>> cat.updateIndex(cat['size']) >>> list(cat.apply({'size': 5})) [4, 5] Of course, that only updates *that* index: >>> list(cat.apply({'age': 10})) [] We can update all of the indexes: >>> cat.updateIndexes() >>> list(cat.apply({'age': 10})) [1, 6] >>> list(cat.apply({'color': 'red'})) [2] There's an alternate search interface that returns "result sets". Result sets provide access to objects, rather than object ids: >>> result = cat.searchResults(size=5) >>> len(result) 2 >>> list(result) == [o4, o5] True The searchResults method also provides a way to sort, limit and reverse results. When not using sorting, limiting and reversing are done by simple slicing and list reversing. >>> list(cat.searchResults(size=5, _reverse=True)) == [o5, o4] True >>> list(cat.searchResults(size=5, _limit=1)) == [o4] True >>> list(cat.searchResults(size=5, _limit=1, _reverse=True)) == [o5] True However, when using sorting by index, the limit and reverse parameters are passed to the index ``sort`` method so it can do it efficiently. Let's index more objects to work with: >>> o7 = DiscriminatingPerson(7, 'blue') >>> o8 = DiscriminatingPerson(3, 'blue') >>> o9 = DiscriminatingPerson(14, 'blue') >>> o10 = DiscriminatingPerson(1, 'blue') >>> ids.data.update({7: o7, 8: o8, 9: o9, 10: o10}) >>> cat.index_doc(7, o7) >>> cat.index_doc(8, o8) >>> cat.index_doc(9, o9) >>> cat.index_doc(10, o10) Now we can search all people who like blue, ordered by age: >>> results = list(cat.searchResults(color='blue', _sort_index='age')) >>> results == [o3, o10, o8, o7, o6, o9] True >>> results = list(cat.searchResults(color='blue', _sort_index='age', _limit=3)) >>> results == [o3, o10, o8] True >>> results = list(cat.searchResults(color='blue', _sort_index='age', _reverse=True)) >>> results == [o9, o6, o7, o8, o10, o3] True >>> results = list(cat.searchResults(color='blue', _sort_index='age', _reverse=True, _limit=4)) >>> results == [o9, o6, o7, o8] True The index example we looked at didn't provide document scores. Simple indexes normally don't, but more complex indexes might give results scores, according to how closely a document matches a query. Let's create a new index, a "keyword index" that indexes sequences of values: >>> class BaseKeywordIndex(persistent.Persistent): ... zope.interface.implements( ... zope.index.interfaces.IInjection, ... zope.index.interfaces.IIndexSearch, ... ) ... ... def clear(self): ... self.forward = BTrees.OOBTree.OOBTree() ... self.backward = BTrees.IOBTree.IOBTree() ... ... __init__ = clear ... ... def index_doc(self, docid, values): ... if docid in self.backward: ... self.unindex_doc(docid) ... self.backward[docid] = values ... ... for value in values: ... set = self.forward.get(value) ... if set is None: ... set = BTrees.IFBTree.IFTreeSet() ... self.forward[value] = set ... set.insert(docid) ... ... def unindex_doc(self, docid): ... values = self.backward.get(docid) ... if values is None: ... return ... for value in values: ... self.forward[value].remove(docid) ... del self.backward[docid] ... ... def apply(self, values): ... result = BTrees.IFBTree.IFBucket() ... for value in values: ... set = self.forward.get(value) ... if set is not None: ... _, result = BTrees.IFBTree.weightedUnion(result, set) ... return result >>> class KeywordIndex(zope.catalog.attribute.AttributeIndex, ... BaseKeywordIndex, ... zope.container.contained.Contained, ... ): ... zope.interface.implements(zope.catalog.interfaces.ICatalogIndex) Now, we'll add a hobbies index: >>> cat['hobbies'] = KeywordIndex('hobbies') >>> o1.hobbies = 'camping', 'music' >>> o2.hobbies = 'hacking', 'sailing' >>> o3.hobbies = 'music', 'camping', 'sailing' >>> o6.hobbies = 'cooking', 'dancing' >>> cat.updateIndexes() When we apply the catalog: >>> cat.apply({'hobbies': ['music', 'camping', 'sailing']}) BTrees.IFBTree.IFBucket([(1, 2.0), (2, 1.0), (3, 3.0)]) We found objects 1-3, because they each contained at least some of the words in the query. The scores represent the number of words that matched. If we also include age: >>> cat.apply({'hobbies': ['music', 'camping', 'sailing'], 'age': 10}) BTrees.IFBTree.IFBucket([(1, 3.0)]) The score increased because we used an additional index. If an index doesn't provide scores, scores of 1.0 are assumed. ============================== Automatic indexing with events ============================== In order to automatically keep the catalog up-to-date any objects that are added to a intid utility are indexed automatically. Also when an object gets modified it is reindexed by listening to IObjectModified events. Let us create a fake catalog to demonstrate this behaviour. We only need to implement the index_doc method for this test. >>> from zope.catalog.interfaces import ICatalog >>> from zope import interface, component >>> class FakeCatalog(object): ... indexed = [] ... interface.implements(ICatalog) ... def index_doc(self, docid, obj): ... self.indexed.append((docid, obj)) >>> cat = FakeCatalog() >>> component.provideUtility(cat) We also need an intid util and a keyreference adapter. >>> from zope.intid import IntIds >>> from zope.intid.interfaces import IIntIds >>> intids = IntIds() >>> component.provideUtility(intids, IIntIds) >>> from zope.keyreference.testing import SimpleKeyReference >>> component.provideAdapter(SimpleKeyReference) >>> from zope.container.contained import Contained >>> class Dummy(Contained): ... def __init__(self, name): ... self.__name__ = name ... def __repr__(self): ... return '' % self.__name__ We have a subscriber to IIntidAddedEvent. >>> from zope.catalog import catalog >>> from zope.intid.interfaces import IntIdAddedEvent >>> d1 = Dummy(u'one') >>> id1 = intids.register(d1) >>> catalog.indexDocSubscriber(IntIdAddedEvent(d1, None)) Now we have indexed the object. >>> cat.indexed.pop() (..., ) When an object is modified an objectmodified event should be fired by the application. Here is the handler for such an event. >>> from zope.lifecycleevent import ObjectModifiedEvent >>> catalog.reindexDocSubscriber(ObjectModifiedEvent(d1)) >>> len(cat.indexed) 1 >>> cat.indexed.pop() (..., ) Preventing automatic indexing ============================= Sometimes it is not accurate to automatically index an object. For example when a lot of indexes are in the catalog and only specific indexes needs to be updated. There are marker interfaces to achieve this. >>> from zope.catalog.interfaces import INoAutoIndex If an object provides this interface it is not automatically indexed. >>> interface.alsoProvides(d1, INoAutoIndex) >>> catalog.indexDocSubscriber(IntIdAddedEvent(d1, None)) >>> len(cat.indexed) 0 >>> from zope.catalog.interfaces import INoAutoReindex If an object provides this interface it is not automatically reindexed. >>> interface.alsoProvides(d1, INoAutoReindex) >>> catalog.reindexDocSubscriber(ObjectModifiedEvent(d1)) >>> len(cat.indexed) 0 ======= CHANGES ======= 3.8.2 (2011-11-29) ------------------ - Conform to repository policy. 3.8.1 (2009-12-27) ------------------ - Removed ``zope.app.testing`` dependency. 3.8.0 (2009-02-01) ------------------ - Move core functionality from ``zope.app.catalog`` to this package. The ``zope.app.catalog`` package now only contains ZMI-related browser views and backward-compatibility imports. Keywords: zope3 catalog index Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Web Environment Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Zope Public License Classifier: Programming Language :: Python Classifier: Natural Language :: English Classifier: Operating System :: OS Independent Classifier: Topic :: Internet :: WWW/HTTP Classifier: Framework :: Zope3 zope.catalog-3.8.2/src/zope.catalog.egg-info/dependency_links.txt0000664000177100020040000000000111665177625026175 0ustar menesismenesis00000000000000 zope.catalog-3.8.2/src/zope.catalog.egg-info/namespace_packages.txt0000664000177100020040000000000511665177625026455 0ustar menesismenesis00000000000000zope zope.catalog-3.8.2/src/zope.catalog.egg-info/not-zip-safe0000664000177100020040000000000111665177363024354 0ustar menesismenesis00000000000000 zope.catalog-3.8.2/src/zope.catalog.egg-info/top_level.txt0000664000177100020040000000000511665177625024654 0ustar menesismenesis00000000000000zope zope.catalog-3.8.2/src/zope.catalog.egg-info/requires.txt0000664000177100020040000000026711665177625024534 0ustar menesismenesis00000000000000setuptools ZODB3 zope.annotation zope.intid zope.component zope.container zope.index>=3.5.0 zope.interface zope.lifecycleevent zope.location zope.schema [test] zope.testing zope.sitezope.catalog-3.8.2/src/zope.catalog.egg-info/SOURCES.txt0000664000177100020040000000143211665177625024013 0ustar menesismenesis00000000000000CHANGES.txt COPYRIGHT.txt LICENSE.txt README.txt bootstrap.py buildout.cfg setup.py src/zope/__init__.py src/zope.catalog.egg-info/PKG-INFO src/zope.catalog.egg-info/SOURCES.txt src/zope.catalog.egg-info/dependency_links.txt src/zope.catalog.egg-info/namespace_packages.txt src/zope.catalog.egg-info/not-zip-safe src/zope.catalog.egg-info/requires.txt src/zope.catalog.egg-info/top_level.txt src/zope/catalog/README.txt src/zope/catalog/__init__.py src/zope/catalog/apidoc.zcml src/zope/catalog/attribute.py src/zope/catalog/catalog.py src/zope/catalog/classes.zcml src/zope/catalog/configure.zcml src/zope/catalog/event.txt src/zope/catalog/field.py src/zope/catalog/interfaces.py src/zope/catalog/keyword.py src/zope/catalog/subscribers.zcml src/zope/catalog/tests.py src/zope/catalog/text.pyzope.catalog-3.8.2/src/zope/0000775000177100020040000000000011665177633017023 5ustar menesismenesis00000000000000zope.catalog-3.8.2/src/zope/catalog/0000775000177100020040000000000011665177633020435 5ustar menesismenesis00000000000000zope.catalog-3.8.2/src/zope/catalog/catalog.py0000664000177100020040000001554011665177362022425 0ustar menesismenesis00000000000000############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Catalog """ __docformat__ = 'restructuredtext' import BTrees import zope.index.interfaces from zope import component from zope.annotation.interfaces import IAttributeAnnotatable from zope.container.btree import BTreeContainer from zope.container.interfaces import IObjectAddedEvent from zope.interface import implements from zope.intid.interfaces import IIntIds, IIntIdAddedEvent, IIntIdRemovedEvent from zope.lifecycleevent import IObjectModifiedEvent from zope.location import location from zope.location.interfaces import ILocationInfo from zope.catalog.interfaces import ICatalog, INoAutoIndex, INoAutoReindex, ICatalogIndex class ResultSet: """Lazily accessed set of objects.""" def __init__(self, uids, uidutil): self.uids = uids self.uidutil = uidutil def __len__(self): return len(self.uids) def __iter__(self): for uid in self.uids: obj = self.uidutil.getObject(uid) yield obj class Catalog(BTreeContainer): implements(ICatalog, IAttributeAnnotatable, zope.index.interfaces.IIndexSearch, ) family = BTrees.family32 def __init__(self, family=None): super(Catalog, self).__init__() if family is not None: self.family = family def clear(self): for index in self.values(): index.clear() def index_doc(self, docid, texts): """Register the data in indexes of this catalog.""" for index in self.values(): index.index_doc(docid, texts) def unindex_doc(self, docid): """Unregister the data from indexes of this catalog.""" for index in self.values(): index.unindex_doc(docid) def _visitSublocations(self) : """Restricts the access to the objects that live within the nearest site if the catalog itself is locatable. """ uidutil = None locatable = ILocationInfo(self, None) if locatable is not None : site = locatable.getNearestSite() sm = site.getSiteManager() uidutil = sm.queryUtility(IIntIds) if uidutil not in [c.component for c in sm.registeredUtilities()]: # we do not have a local inits utility uidutil = component.getUtility(IIntIds, context=self) for uid in uidutil: obj = uidutil.getObject(uid) if location.inside(obj, site) : yield uid, obj return if uidutil is None: uidutil = component.getUtility(IIntIds) for uid in uidutil: yield uid, uidutil.getObject(uid) def updateIndex(self, index): for uid, obj in self._visitSublocations() : index.index_doc(uid, obj) def updateIndexes(self): for uid, obj in self._visitSublocations() : for index in self.values(): index.index_doc(uid, obj) def apply(self, query): results = [] for index_name, index_query in query.items(): index = self[index_name] r = index.apply(index_query) if r is None: continue if not r: # empty results return r results.append((len(r), r)) if not results: # no applicable indexes, so catalog was not applicable return None results.sort() # order from smallest to largest _, result = results.pop(0) for _, r in results: _, result = self.family.IF.weightedIntersection(result, r) return result def searchResults(self, **searchterms): sort_index = searchterms.pop('_sort_index', None) limit = searchterms.pop('_limit', None) reverse = searchterms.pop('_reverse', False) results = self.apply(searchterms) if results is not None: if sort_index is not None: index = self[sort_index] if not zope.index.interfaces.IIndexSort.providedBy(index): raise ValueError('Index %s does not support sorting.' % sort_index) results = list(index.sort(results, limit=limit, reverse=reverse)) else: if reverse or limit: results = list(results) if reverse: results.reverse() if limit: del results[limit:] uidutil = component.getUtility(IIntIds) results = ResultSet(results, uidutil) return results @component.adapter(ICatalogIndex, IObjectAddedEvent) def indexAdded(index, event): """When an index is added to a catalog, we have to index existing objects When an index is added, we tell it's parent to index it: >>> class FauxCatalog: ... def updateIndex(self, index): ... self.updated = index >>> class FauxIndex: ... pass >>> index = FauxIndex() >>> index.__parent__ = FauxCatalog() >>> indexAdded(index, None) >>> index.__parent__.updated is index True """ index.__parent__.updateIndex(index) @component.adapter(IIntIdAddedEvent) def indexDocSubscriber(event): """A subscriber to IntIdAddedEvent""" ob = event.object if INoAutoIndex.providedBy(ob): return for cat in component.getAllUtilitiesRegisteredFor(ICatalog, context=ob): id = component.getUtility(IIntIds, context=cat).getId(ob) cat.index_doc(id, ob) @component.adapter(IObjectModifiedEvent) def reindexDocSubscriber(event): """A subscriber to ObjectModifiedEvent""" ob = event.object if INoAutoReindex.providedBy(ob): return for cat in component.getAllUtilitiesRegisteredFor(ICatalog, context=ob): id = component.getUtility(IIntIds, context=cat).queryId(ob) if id is not None: cat.index_doc(id, ob) @component.adapter(IIntIdRemovedEvent) def unindexDocSubscriber(event): """A subscriber to IntIdRemovedEvent""" ob = event.object for cat in component.getAllUtilitiesRegisteredFor(ICatalog, context=ob): id = component.getUtility(IIntIds, context=cat).queryId(ob) if id is not None: cat.unindex_doc(id) zope.catalog-3.8.2/src/zope/catalog/apidoc.zcml0000664000177100020040000000122011665177362022555 0ustar menesismenesis00000000000000 zope.catalog-3.8.2/src/zope/catalog/text.py0000664000177100020040000000360611665177362021777 0ustar menesismenesis00000000000000############################################################################## # # Copyright (c) 2002 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Text catalog indexes """ import zope.index.text import zope.index.text.interfaces import zope.interface import zope.catalog.attribute import zope.catalog.interfaces import zope.container.contained from zope.i18nmessageid import ZopeMessageFactory as _ class ITextIndex(zope.catalog.interfaces.IAttributeIndex, zope.catalog.interfaces.ICatalogIndex): """Interface-based catalog text index """ interface = zope.schema.Choice( title=_(u"Interface"), description=_(u"Objects will be adapted to this interface"), vocabulary=_("Interfaces"), required=False, default=zope.index.text.interfaces.ISearchableText, ) field_name = zope.schema.BytesLine( title=_(u"Field Name"), description=_(u"Name of the field to index"), default="getSearchableText" ) field_callable = zope.schema.Bool( title=_(u"Field Callable"), description=_(u"If true, then the field should be called to get the " u"value to be indexed"), default=True, ) class TextIndex(zope.catalog.attribute.AttributeIndex, zope.index.text.TextIndex, zope.container.contained.Contained): zope.interface.implements(ITextIndex) zope.catalog-3.8.2/src/zope/catalog/configure.zcml0000664000177100020040000000025111665177362023302 0ustar menesismenesis00000000000000 zope.catalog-3.8.2/src/zope/catalog/tests.py0000664000177100020040000005733111665177362022161 0ustar menesismenesis00000000000000############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests for catalog Note that indexes &c already have test suites, we only have to check that a catalog passes on events that it receives. """ import unittest from zope.testing import doctest from zope.interface import implements, Interface from zope.interface.verify import verifyObject from BTrees.IFBTree import IFSet from zope.intid.interfaces import IIntIds from zope.location.location import Location from zope.component import provideUtility from zope.component import provideAdapter from zope.component import provideHandler from zope.component import testing, eventtesting from zope.component.interfaces import ISite, IComponentLookup from zope.site.hooks import setSite, setHooks, resetHooks from zope.site.folder import Folder, rootFolder from zope.site.site import SiteManagerAdapter, LocalSiteManager from zope.traversing import api from zope.traversing.testing import setUp as traversingSetUp from zope.traversing.interfaces import ITraversable from zope.container.traversal import ContainerTraversable from zope.container.interfaces import ISimpleReadContainer from zope.index.interfaces import IInjection, IIndexSearch from zope.catalog.interfaces import ICatalog from zope.catalog.catalog import Catalog from zope.catalog.field import FieldIndex class ReferenceStub: def __init__(self, obj): self.obj = obj def __call__(self): return self.obj class IntIdsStub: """A stub for IntIds.""" implements(IIntIds) def __init__(self): self.ids = {} self.objs = {} self.lastid = 0 def _generateId(self): self.lastid += 1 return self.lastid def register(self, ob): if ob not in self.ids: uid = self._generateId() self.ids[ob] = uid self.objs[uid] = ob return uid else: return self.ids[ob] def unregister(self, ob): uid = self.ids[ob] del self.ids[ob] del self.objs[uid] def getObject(self, uid): return self.objs[uid] def getId(self, ob): return self.ids[ob] def queryId(self, ob, default=None): return self.ids.get(ob, default) def __iter__(self): return self.objs.iterkeys() class StubIndex: """A stub for Index.""" implements(IIndexSearch, IInjection) def __init__(self, field_name, interface=None): self._field_name = field_name self.interface = interface self.doc = {} def index_doc(self, docid, obj): self.doc[docid] = obj def unindex_doc(self, docid): del self.doc[docid] def apply(self, term): results = [] for docid in self.doc: obj = self.doc[docid] fieldname = getattr(obj, self._field_name, '') if fieldname == term: results.append(docid) return IFSet(results) class stoopid: def __init__(self, **kw): self.__dict__ = kw class PlacelessSetup(testing.PlacelessSetup, eventtesting.PlacelessSetup): def setUp(self, doctesttest=None): testing.PlacelessSetup.setUp(self) eventtesting.PlacelessSetup.setUp(self) class Test(PlacelessSetup, unittest.TestCase): def test_catalog_add_del_indexes(self): catalog = Catalog() verifyObject(ICatalog, catalog) index = StubIndex('author', None) catalog['author'] = index self.assertEqual(list(catalog.keys()), ['author']) index = StubIndex('title', None) catalog['title'] = index indexes = list(catalog.keys()) indexes.sort() self.assertEqual(indexes, ['author', 'title']) del catalog['author'] self.assertEqual(list(catalog.keys()), ['title']) def _frob_intidutil(self, ints=True, apes=True): uidutil = IntIdsStub() provideUtility(uidutil, IIntIds) # whack some objects in our little objecthub if ints: for i in range(10): uidutil.register(""%i) if apes: uidutil.register(stoopid(simiantype='monkey', name='bobo')) uidutil.register(stoopid(simiantype='monkey', name='bubbles')) uidutil.register(stoopid(simiantype='monkey', name='ginger')) uidutil.register(stoopid(simiantype='bonobo', name='ziczac')) uidutil.register(stoopid(simiantype='bonobo', name='bobo')) uidutil.register(stoopid(simiantype='punyhuman', name='anthony')) uidutil.register(stoopid(simiantype='punyhuman', name='andy')) uidutil.register(stoopid(simiantype='punyhuman', name='kev')) def test_updateindexes(self): """Test a full refresh.""" self._frob_intidutil() catalog = Catalog() catalog['author'] = StubIndex('author', None) catalog['title'] = StubIndex('author', None) catalog.updateIndexes() for index in catalog.values(): checkNotifies = index.doc self.assertEqual(len(checkNotifies), 18) def test_updateindex(self): """Test a full refresh.""" self._frob_intidutil() catalog = Catalog() catalog['author'] = StubIndex('author', None) catalog['title'] = StubIndex('author', None) catalog.updateIndex(catalog['author']) checkNotifies = catalog['author'].doc self.assertEqual(len(checkNotifies), 18) checkNotifies = catalog['title'].doc self.assertEqual(len(checkNotifies), 0) def test_basicsearch(self): """Test the simple search results interface.""" self._frob_intidutil(ints=0) catalog = Catalog() catalog['simiantype'] = StubIndex('simiantype', None) catalog['name'] = StubIndex('name', None) catalog.updateIndexes() res = catalog.searchResults(simiantype='monkey') names = [x.name for x in res] names.sort() self.assertEqual(len(names), 3) self.assertEqual(names, ['bobo', 'bubbles', 'ginger']) res = catalog.searchResults(name='bobo') names = [x.simiantype for x in res] names.sort() self.assertEqual(len(names), 2) self.assertEqual(names, ['bonobo', 'monkey']) res = catalog.searchResults(simiantype='punyhuman', name='anthony') self.assertEqual(len(res), 1) ob = iter(res).next() self.assertEqual((ob.name, ob.simiantype), ('anthony', 'punyhuman')) res = catalog.searchResults(simiantype='ape', name='bobo') self.assertEqual(len(res), 0) res = catalog.searchResults(simiantype='ape', name='mwumi') self.assertEqual(len(res), 0) self.assertRaises(KeyError, catalog.searchResults, simiantype='monkey', hat='beret') class CatalogStub: implements(ICatalog) def __init__(self): self.regs = [] self.unregs = [] def index_doc(self, docid, doc): self.regs.append((docid, doc)) def unindex_doc(self, docid): self.unregs.append(docid) class Stub(Location): pass class TestEventSubscribers(unittest.TestCase): def setUp(self): self.root = placefulSetUp(True) sm = self.root.getSiteManager() self.utility = addUtility(sm, '', IIntIds, IntIdsStub()) self.cat = addUtility(sm, '', ICatalog, CatalogStub()) setSite(self.root) def tearDown(self): placefulTearDown() def test_indexDocSubscriber(self): from zope.catalog.catalog import indexDocSubscriber from zope.container.contained import ObjectAddedEvent from zope.intid.interfaces import IntIdAddedEvent ob = Stub() ob2 = Stub() self.root['ob'] = ob self.root['ob2'] = ob2 id = self.utility.register(ob) indexDocSubscriber(IntIdAddedEvent(ob, ObjectAddedEvent(ob2))) self.assertEqual(self.cat.regs, [(id, ob)]) self.assertEqual(self.cat.unregs, []) def test_reindexDocSubscriber(self): from zope.catalog.catalog import reindexDocSubscriber from zope.lifecycleevent import ObjectModifiedEvent ob = Stub() self.root['ob'] = ob id = self.utility.register(ob) reindexDocSubscriber(ObjectModifiedEvent(ob)) self.assertEqual(self.cat.regs, [(1, ob)]) self.assertEqual(self.cat.unregs, []) ob2 = Stub() self.root['ob2'] = ob2 reindexDocSubscriber(ObjectModifiedEvent(ob2)) self.assertEqual(self.cat.regs, [(1, ob)]) self.assertEqual(self.cat.unregs, []) def test_unindexDocSubscriber(self): from zope.catalog.catalog import unindexDocSubscriber from zope.container.contained import ObjectRemovedEvent from zope.intid.interfaces import IntIdRemovedEvent ob = Stub() ob2 = Stub() ob3 = Stub() self.root['ob'] = ob self.root['ob2'] = ob2 self.root['ob3'] = ob3 id = self.utility.register(ob) unindexDocSubscriber( IntIdRemovedEvent(ob2, ObjectRemovedEvent(ob3))) self.assertEqual(self.cat.unregs, []) self.assertEqual(self.cat.regs, []) unindexDocSubscriber( IntIdRemovedEvent(ob, ObjectRemovedEvent(ob3))) self.assertEqual(self.cat.unregs, [id]) self.assertEqual(self.cat.regs, []) class TestIndexUpdating(unittest.TestCase) : """Issue #466: When reindexing a catalog it takes all objects from the nearest IntId utility. This is a problem when IntId utility lives in another site than the one. To solve this issue we simply check whether the objects are living within the nearest site. """ def setUp(self): placefulSetUp(True) from zope.catalog.catalog import Catalog self.root = buildSampleFolderTree() subfolder = self.root[u'folder1'][u'folder1_1'] root_sm = self.root_sm = createSiteManager(self.root) local_sm = self.local_sm = createSiteManager(subfolder) self.utility = addUtility(root_sm, '', IIntIds, IntIdsStub()) self.cat = addUtility(local_sm, '', ICatalog, Catalog()) self.cat['name'] = StubIndex('__name__', None) for obj in self.iterAll(self.root) : self.utility.register(obj) def tearDown(self): placefulTearDown() def iterAll(self, container) : from zope.container.interfaces import IContainer for value in container.values() : yield value if IContainer.providedBy(value) : for obj in self.iterAll(value) : yield obj def test_visitSublocations(self) : """ Test the iterContained method which should return only the sublocations which are registered by the IntIds. """ names = sorted([ob.__name__ for i, ob in self.cat._visitSublocations()]) self.assertEqual(names, [u'folder1_1', u'folder1_1_1', u'folder1_1_2']) def test_updateIndex(self): """ Setup a catalog deeper within the containment hierarchy and call the updateIndexes method. The indexed objects should should be restricted to the sublocations. """ self.cat.updateIndexes() index = self.cat['name'] names = sorted([ob.__name__ for i, ob in index.doc.items()]) self.assertEqual(names, [u'folder1_1', u'folder1_1_1', u'folder1_1_2']) def test_optimizedUpdateIndex(self): """ Setup a catalog deeper within the containment hierarchy together with its intid utility. The catalog will not visit sublocations because the intid utility can not contain objects outside the site where it is registered. """ utility = addUtility(self.local_sm, '', IIntIds, IntIdsStub()) subfolder = self.root[u'folder1'][u'folder1_1'] for obj in self.iterAll(subfolder) : utility.register(obj) self.cat.updateIndexes() index = self.cat['name'] names = sorted([ob.__name__ for i, ob in index.doc.items()]) self.assertEqual(names, [u'folder1_1_1', u'folder1_1_2']) class TestSubSiteCatalog(unittest.TestCase) : """If a catalog is defined in a sub site and the hooks.setSite was not set the catalog will not be found unless the context in getAllUtilitiesRegisteredFor is set. """ def setUp(self): placefulSetUp(True) from zope.catalog.catalog import Catalog self.root = buildSampleFolderTree() self.subfolder = self.root[u'folder1'][u'folder1_1'] root_sm = self.root_sm = createSiteManager(self.root) local_sm = self.local_sm = createSiteManager(self.subfolder) self.utility = addUtility(root_sm, '', IIntIds, IntIdsStub()) self.cat = addUtility(local_sm, '', ICatalog, Catalog()) self.cat['name'] = StubIndex('__name__', None) for obj in self.iterAll(self.root) : self.utility.register(obj) def tearDown(self): placefulTearDown() def iterAll(self, container) : from zope.container.interfaces import IContainer for value in container.values() : yield value if IContainer.providedBy(value) : for obj in self.iterAll(value) : yield obj def test_Index(self): """ Setup a catalog deeper within the containment hierarchy and call the updateIndexes method. The indexed objects should should be restricted to the sublocations. """ from zope.catalog.catalog import indexDocSubscriber from zope.container.contained import ObjectAddedEvent ob = Stub() self.subfolder['ob'] = ob id = self.utility.register(ob) setSite(self.subfolder) res = self.cat.searchResults(name='ob') self.assertEqual(len(res), 0) setSite(None) indexDocSubscriber(ObjectAddedEvent(ob)) setSite(self.subfolder) res = self.cat.searchResults(name='ob') self.assertEqual(len(res), 1) def test_updateIndex(self): """ Setup a catalog deeper within the containment hierarchy and call the updateIndexes method. The indexed objects should should be restricted to the sublocations. """ from zope.catalog.catalog import reindexDocSubscriber from zope.lifecycleevent import ObjectModifiedEvent ob = Stub() self.subfolder['ob'] = ob id = self.utility.register(ob) setSite(self.subfolder) res = self.cat.searchResults(name='ob') self.assertEqual(len(res), 0) setSite(None) reindexDocSubscriber(ObjectModifiedEvent(ob)) setSite(self.subfolder) res = self.cat.searchResults(name='ob') self.assertEqual(len(res), 1) def test_UnIndex(self): """ Setup a catalog deeper within the containment hierarchy and call the updateIndexes method. The indexed objects should should be restricted to the sublocations. """ from zope.catalog.catalog import indexDocSubscriber from zope.catalog.catalog import unindexDocSubscriber from zope.container.contained import ObjectAddedEvent from zope.container.contained import ObjectRemovedEvent ob = Stub() self.subfolder['ob'] = ob id = self.utility.register(ob) setSite(self.subfolder) res = self.cat.searchResults(name='ob') self.assertEqual(len(res), 0) setSite(None) indexDocSubscriber(ObjectAddedEvent(ob)) setSite(self.subfolder) res = self.cat.searchResults(name='ob') self.assertEqual(len(res), 1) setSite(None) unindexDocSubscriber(ObjectRemovedEvent(ob)) setSite(self.subfolder) res = self.cat.searchResults(name='ob') self.assertEqual(len(res), 0) class TestCatalogBugs(PlacelessSetup, unittest.TestCase): """I found that z.a.catalog, AttributeIndex failed to remove the previous value/object from the index IF the NEW value is None. """ def test_updateIndexWithNone(self): uidutil = IntIdsStub() provideUtility(uidutil, IIntIds) catalog = Catalog() index = FieldIndex('author', None) catalog['author'] = index ob1 = stoopid(author = "joe") ob1id = uidutil.register(ob1) catalog.index_doc(ob1id, ob1) res = catalog.searchResults(author=('joe','joe')) names = [x.author for x in res] names.sort() self.assertEqual(len(names), 1) self.assertEqual(names, ['joe']) ob1.author = None catalog.index_doc(ob1id, ob1) #the index must be empty now because None values are never indexed res = catalog.searchResults(author=(None, None)) self.assertEqual(len(res), 0) def test_updateIndexFromCallableWithNone(self): uidutil = IntIdsStub() provideUtility(uidutil, IIntIds) catalog = Catalog() index = FieldIndex('getAuthor', None, field_callable=True) catalog['author'] = index ob1 = stoopidCallable(author = "joe") ob1id = uidutil.register(ob1) catalog.index_doc(ob1id, ob1) res = catalog.searchResults(author=('joe','joe')) names = [x.author for x in res] names.sort() self.assertEqual(len(names), 1) self.assertEqual(names, ['joe']) ob1.author = None catalog.index_doc(ob1id, ob1) #the index must be empty now because None values are never indexed res = catalog.searchResults(author=(None, None)) self.assertEqual(len(res), 0) class stoopidCallable(object): def __init__(self, **kw): #leave the door open to not to set self.author self.__dict__.update(kw) def getAuthor(self): return self.author class TestIndexRaisingValueGetter(PlacelessSetup, unittest.TestCase): """ """ def test_IndexRaisingValueGetter(self): """We can have indexes whose values are determined by callable methods. Raising an exception in the method should not be silently ignored That would cause index corruption -- the index would be out of sync""" uidutil = IntIdsStub() provideUtility(uidutil, IIntIds) catalog = Catalog() index = FieldIndex('getAuthor', None, field_callable=True) catalog['author'] = index ob1 = stoopidCallable(author = "joe") ob1id = uidutil.register(ob1) catalog.index_doc(ob1id, ob1) res = catalog.searchResults(author=('joe','joe')) names = [x.author for x in res] names.sort() self.assertEqual(len(names), 1) self.assertEqual(names, ['joe']) ob2 = stoopidCallable() # no author here, will raise AttributeError ob2id = uidutil.register(ob2) try: catalog.index_doc(ob2id, ob2) self.fail("AttributeError exception should be raised") except AttributeError: #this is OK, we WANT to have the exception pass #------------------------------------------------------------------------ # placeful setUp/tearDown def placefulSetUp(site=False): testing.setUp() eventtesting.setUp() traversingSetUp() setHooks() provideAdapter(ContainerTraversable, (ISimpleReadContainer,), ITraversable) provideAdapter(SiteManagerAdapter, (Interface,), IComponentLookup) if site: root = rootFolder() createSiteManager(root, setsite=True) return root def placefulTearDown(): resetHooks() setSite() testing.tearDown() #------------------------------------------------------------------------ # placeless setUp/tearDown ps = PlacelessSetup() placelessSetUp = ps.setUp def placelessTearDown(): tearDown_ = ps.tearDown def tearDown(doctesttest=None): tearDown_() return tearDown placelessTearDown = placelessTearDown() del ps #------------------------------------------------------------------------ # setup site manager def createSiteManager(folder, setsite=False): if not ISite.providedBy(folder): folder.setSiteManager(LocalSiteManager(folder)) if setsite: setSite(folder) return api.traverse(folder, "++etc++site") #------------------------------------------------------------------------ # Local Utility Addition def addUtility(sitemanager, name, iface, utility, suffix=''): """Add a utility to a site manager This helper function is useful for tests that need to set up utilities. """ folder_name = (name or (iface.__name__ + 'Utility')) + suffix default = sitemanager['default'] default[folder_name] = utility utility = default[folder_name] sitemanager.registerUtility(utility, iface, name) return utility #------------------------------------------------------------------------ # Sample Folder Creation def buildSampleFolderTree(): # set up a reasonably complex folder structure # # ____________ rootFolder ______________________________ # / \ \ # folder1 __________________ folder2 folder3 # | \ | | # folder1_1 ____ folder1_2 folder2_1 folder3_1 # | \ | | # folder1_1_1 folder1_1_2 folder1_2_1 folder2_1_1 root = rootFolder() root[u'folder1'] = Folder() root[u'folder1'][u'folder1_1'] = Folder() root[u'folder1'][u'folder1_1'][u'folder1_1_1'] = Folder() root[u'folder1'][u'folder1_1'][u'folder1_1_2'] = Folder() root[u'folder1'][u'folder1_2'] = Folder() root[u'folder1'][u'folder1_2'][u'folder1_2_1'] = Folder() root[u'folder2'] = Folder() root[u'folder2'][u'folder2_1'] = Folder() root[u'folder2'][u'folder2_1'][u'folder2_1_1'] = Folder() root[u"\N{CYRILLIC SMALL LETTER PE}" u"\N{CYRILLIC SMALL LETTER A}" u"\N{CYRILLIC SMALL LETTER PE}" u"\N{CYRILLIC SMALL LETTER KA}" u"\N{CYRILLIC SMALL LETTER A}3"] = Folder() root[u"\N{CYRILLIC SMALL LETTER PE}" u"\N{CYRILLIC SMALL LETTER A}" u"\N{CYRILLIC SMALL LETTER PE}" u"\N{CYRILLIC SMALL LETTER KA}" u"\N{CYRILLIC SMALL LETTER A}3"][ u"\N{CYRILLIC SMALL LETTER PE}" u"\N{CYRILLIC SMALL LETTER A}" u"\N{CYRILLIC SMALL LETTER PE}" u"\N{CYRILLIC SMALL LETTER KA}" u"\N{CYRILLIC SMALL LETTER A}3_1"] = Folder() return root def setUp(test): root = placefulSetUp(True) test.globs['root'] = root def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(Test)) suite.addTest(unittest.makeSuite(TestEventSubscribers)) suite.addTest(unittest.makeSuite(TestIndexUpdating)) suite.addTest(unittest.makeSuite(TestSubSiteCatalog)) suite.addTest(unittest.makeSuite(TestCatalogBugs)) suite.addTest(unittest.makeSuite(TestIndexRaisingValueGetter)) suite.addTest(doctest.DocTestSuite('zope.catalog.attribute')) suite.addTest(doctest.DocFileSuite( 'README.txt', setUp=placelessSetUp, tearDown=placelessTearDown, )) suite.addTest(doctest.DocFileSuite( 'event.txt', setUp=setUp, tearDown=lambda x: placefulTearDown(), optionflags=doctest.NORMALIZE_WHITESPACE|doctest.ELLIPSIS, )) return suite if __name__ == "__main__": unittest.main() zope.catalog-3.8.2/src/zope/catalog/field.py0000664000177100020040000000222711665177362022074 0ustar menesismenesis00000000000000############################################################################## # # Copyright (c) 2002 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Field catalog indexes """ import zope.index.field import zope.interface import zope.catalog.attribute import zope.catalog.interfaces import zope.container.contained class IFieldIndex(zope.catalog.interfaces.IAttributeIndex, zope.catalog.interfaces.ICatalogIndex): """Interface-based catalog field index """ class FieldIndex(zope.catalog.attribute.AttributeIndex, zope.index.field.FieldIndex, zope.container.contained.Contained): zope.interface.implements(IFieldIndex) zope.catalog-3.8.2/src/zope/catalog/README.txt0000664000177100020040000002741011665177362022136 0ustar menesismenesis00000000000000Catalogs ======== Catalogs provide management of collections of related indexes with a basic search algorithm. Let's look at an example: >>> from zope.catalog.catalog import Catalog >>> cat = Catalog() We can add catalog indexes to catalogs. A catalog index is, among other things, an attribute index. It indexes attributes of objects. To see how this works, we'll create a demonstration attribute index. Our attribute index will simply keep track of objects that have a given attribute value. The `catalog` package provides an attribute-index mix-in class that is meant to work with a base indexing class. First, we'll write the base index class: >>> import persistent, BTrees.OOBTree, BTrees.IFBTree, BTrees.IOBTree >>> import zope.interface, zope.index.interfaces >>> class BaseIndex(persistent.Persistent): ... zope.interface.implements( ... zope.index.interfaces.IInjection, ... zope.index.interfaces.IIndexSearch, ... zope.index.interfaces.IIndexSort, ... ) ... ... def clear(self): ... self.forward = BTrees.OOBTree.OOBTree() ... self.backward = BTrees.IOBTree.IOBTree() ... ... __init__ = clear ... ... def index_doc(self, docid, value): ... if docid in self.backward: ... self.unindex_doc(docid) ... self.backward[docid] = value ... ... set = self.forward.get(value) ... if set is None: ... set = BTrees.IFBTree.IFTreeSet() ... self.forward[value] = set ... set.insert(docid) ... ... def unindex_doc(self, docid): ... value = self.backward.get(docid) ... if value is None: ... return ... self.forward[value].remove(docid) ... del self.backward[docid] ... ... def apply(self, value): ... set = self.forward.get(value) ... if set is None: ... set = BTrees.IFBTree.IFTreeSet() ... return set ... ... def sort(self, docids, limit=None, reverse=False): ... for i, docid in enumerate(sorted(docids, key=self.backward.get, reverse=reverse)): ... yield docid ... if limit and i >= (limit - 1): ... break The class implements `IInjection` to allow values to be indexed and unindexed and `IIndexSearch` to support searching via the `apply` method. Now, we can use the AttributeIndex mixin to make this an attribute index: >>> import zope.catalog.attribute >>> import zope.catalog.interfaces >>> import zope.container.contained >>> class Index(zope.catalog.attribute.AttributeIndex, ... BaseIndex, ... zope.container.contained.Contained, ... ): ... zope.interface.implements(zope.catalog.interfaces.ICatalogIndex) Unfortunately, because of the way we currently handle containment constraints, we have to provide `ICatalogIndex`, which extends `IContained`. We subclass `Contained` to get an implementation for `IContained`. Now let's add some of these indexes to our catalog. Let's create some indexes. First we'll define some interfaces providing data to index: >>> class IFavoriteColor(zope.interface.Interface): ... color = zope.interface.Attribute("Favorite color") >>> class IPerson(zope.interface.Interface): ... def age(): ... """Return the person's age, in years""" We'll create color and age indexes: >>> cat['color'] = Index('color', IFavoriteColor) >>> cat['age'] = Index('age', IPerson, True) >>> cat['size'] = Index('sz') The indexes are created with: - the name of the of the attribute to index - the interface defining the attribute, and - a flag indicating whether the attribute should be called, which defaults to false. If an interface is provided, then we'll only be able to index an object if it can be adapted to the interface, otherwise, we'll simply try to get the attribute from the object. If the attribute isn't present, then we'll ignore the object. Now, let's create some objects and index them: >>> class Person: ... zope.interface.implements(IPerson) ... def __init__(self, age): ... self._age = age ... def age(self): ... return self._age >>> class Discriminating: ... zope.interface.implements(IFavoriteColor) ... def __init__(self, color): ... self.color = color >>> class DiscriminatingPerson(Discriminating, Person): ... def __init__(self, age, color): ... Discriminating.__init__(self, color) ... Person.__init__(self, age) >>> class Whatever: ... def __init__(self, **kw): ... self.__dict__.update(kw) >>> o1 = Person(10) >>> o2 = DiscriminatingPerson(20, 'red') >>> o3 = Discriminating('blue') >>> o4 = Whatever(a=10, c='blue', sz=5) >>> o5 = Whatever(a=20, c='red', sz=6) >>> o6 = DiscriminatingPerson(10, 'blue') >>> cat.index_doc(1, o1) >>> cat.index_doc(2, o2) >>> cat.index_doc(3, o3) >>> cat.index_doc(4, o4) >>> cat.index_doc(5, o5) >>> cat.index_doc(6, o6) We search by providing query mapping objects that have a key for every index we want to use: >>> list(cat.apply({'age': 10})) [1, 6] >>> list(cat.apply({'age': 10, 'color': 'blue'})) [6] >>> list(cat.apply({'age': 10, 'color': 'blue', 'size': 5})) [] >>> list(cat.apply({'size': 5})) [4] We can unindex objects: >>> cat.unindex_doc(4) >>> list(cat.apply({'size': 5})) [] and reindex objects: >>> o5.sz = 5 >>> cat.index_doc(5, o5) >>> list(cat.apply({'size': 5})) [5] If we clear the catalog, we'll clear all of the indexes: >>> cat.clear() >>> [len(index.forward) for index in cat.values()] [0, 0, 0] Note that you don't have to use the catalog's search methods. You can access its indexes directly, since the catalog is a mapping: >>> [(name, cat[name].field_name) for name in cat] [(u'age', 'age'), (u'color', 'color'), (u'size', 'sz')] Catalogs work with int-id utilities, which are responsible for maintaining id <-> object mappings. To see how this works, we'll create a utility to work with our catalog: >>> import zope.intid.interfaces >>> class Ids: ... zope.interface.implements(zope.intid.interfaces.IIntIds) ... def __init__(self, data): ... self.data = data ... def getObject(self, id): ... return self.data[id] ... def __iter__(self): ... return self.data.iterkeys() >>> ids = Ids({1: o1, 2: o2, 3: o3, 4: o4, 5: o5, 6: o6}) >>> from zope.component import provideUtility >>> provideUtility(ids, zope.intid.interfaces.IIntIds) With this utility in place, catalogs can recompute indexes: >>> cat.updateIndex(cat['size']) >>> list(cat.apply({'size': 5})) [4, 5] Of course, that only updates *that* index: >>> list(cat.apply({'age': 10})) [] We can update all of the indexes: >>> cat.updateIndexes() >>> list(cat.apply({'age': 10})) [1, 6] >>> list(cat.apply({'color': 'red'})) [2] There's an alternate search interface that returns "result sets". Result sets provide access to objects, rather than object ids: >>> result = cat.searchResults(size=5) >>> len(result) 2 >>> list(result) == [o4, o5] True The searchResults method also provides a way to sort, limit and reverse results. When not using sorting, limiting and reversing are done by simple slicing and list reversing. >>> list(cat.searchResults(size=5, _reverse=True)) == [o5, o4] True >>> list(cat.searchResults(size=5, _limit=1)) == [o4] True >>> list(cat.searchResults(size=5, _limit=1, _reverse=True)) == [o5] True However, when using sorting by index, the limit and reverse parameters are passed to the index ``sort`` method so it can do it efficiently. Let's index more objects to work with: >>> o7 = DiscriminatingPerson(7, 'blue') >>> o8 = DiscriminatingPerson(3, 'blue') >>> o9 = DiscriminatingPerson(14, 'blue') >>> o10 = DiscriminatingPerson(1, 'blue') >>> ids.data.update({7: o7, 8: o8, 9: o9, 10: o10}) >>> cat.index_doc(7, o7) >>> cat.index_doc(8, o8) >>> cat.index_doc(9, o9) >>> cat.index_doc(10, o10) Now we can search all people who like blue, ordered by age: >>> results = list(cat.searchResults(color='blue', _sort_index='age')) >>> results == [o3, o10, o8, o7, o6, o9] True >>> results = list(cat.searchResults(color='blue', _sort_index='age', _limit=3)) >>> results == [o3, o10, o8] True >>> results = list(cat.searchResults(color='blue', _sort_index='age', _reverse=True)) >>> results == [o9, o6, o7, o8, o10, o3] True >>> results = list(cat.searchResults(color='blue', _sort_index='age', _reverse=True, _limit=4)) >>> results == [o9, o6, o7, o8] True The index example we looked at didn't provide document scores. Simple indexes normally don't, but more complex indexes might give results scores, according to how closely a document matches a query. Let's create a new index, a "keyword index" that indexes sequences of values: >>> class BaseKeywordIndex(persistent.Persistent): ... zope.interface.implements( ... zope.index.interfaces.IInjection, ... zope.index.interfaces.IIndexSearch, ... ) ... ... def clear(self): ... self.forward = BTrees.OOBTree.OOBTree() ... self.backward = BTrees.IOBTree.IOBTree() ... ... __init__ = clear ... ... def index_doc(self, docid, values): ... if docid in self.backward: ... self.unindex_doc(docid) ... self.backward[docid] = values ... ... for value in values: ... set = self.forward.get(value) ... if set is None: ... set = BTrees.IFBTree.IFTreeSet() ... self.forward[value] = set ... set.insert(docid) ... ... def unindex_doc(self, docid): ... values = self.backward.get(docid) ... if values is None: ... return ... for value in values: ... self.forward[value].remove(docid) ... del self.backward[docid] ... ... def apply(self, values): ... result = BTrees.IFBTree.IFBucket() ... for value in values: ... set = self.forward.get(value) ... if set is not None: ... _, result = BTrees.IFBTree.weightedUnion(result, set) ... return result >>> class KeywordIndex(zope.catalog.attribute.AttributeIndex, ... BaseKeywordIndex, ... zope.container.contained.Contained, ... ): ... zope.interface.implements(zope.catalog.interfaces.ICatalogIndex) Now, we'll add a hobbies index: >>> cat['hobbies'] = KeywordIndex('hobbies') >>> o1.hobbies = 'camping', 'music' >>> o2.hobbies = 'hacking', 'sailing' >>> o3.hobbies = 'music', 'camping', 'sailing' >>> o6.hobbies = 'cooking', 'dancing' >>> cat.updateIndexes() When we apply the catalog: >>> cat.apply({'hobbies': ['music', 'camping', 'sailing']}) BTrees.IFBTree.IFBucket([(1, 2.0), (2, 1.0), (3, 3.0)]) We found objects 1-3, because they each contained at least some of the words in the query. The scores represent the number of words that matched. If we also include age: >>> cat.apply({'hobbies': ['music', 'camping', 'sailing'], 'age': 10}) BTrees.IFBTree.IFBucket([(1, 3.0)]) The score increased because we used an additional index. If an index doesn't provide scores, scores of 1.0 are assumed. zope.catalog-3.8.2/src/zope/catalog/keyword.py0000664000177100020040000000266711665177362022505 0ustar menesismenesis00000000000000############################################################################## # # Copyright (c) 2002 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Keyword catalog index """ import zope.index.keyword import zope.interface import zope.container.contained import zope.catalog.attribute import zope.catalog.interfaces class IKeywordIndex(zope.catalog.interfaces.IAttributeIndex, zope.catalog.interfaces.ICatalogIndex): """Interface-based catalog keyword index""" class KeywordIndex(zope.catalog.attribute.AttributeIndex, zope.index.keyword.KeywordIndex, zope.container.contained.Contained): zope.interface.implements(IKeywordIndex) class CaseInsensitiveKeywordIndex(zope.catalog.attribute.AttributeIndex, zope.index.keyword.CaseInsensitiveKeywordIndex, zope.container.contained.Contained): zope.interface.implements(IKeywordIndex) zope.catalog-3.8.2/src/zope/catalog/__init__.py0000664000177100020040000000004011665177362022537 0ustar menesismenesis00000000000000# make this directory a package zope.catalog-3.8.2/src/zope/catalog/event.txt0000664000177100020040000000551211665177362022321 0ustar menesismenesis00000000000000============================== Automatic indexing with events ============================== In order to automatically keep the catalog up-to-date any objects that are added to a intid utility are indexed automatically. Also when an object gets modified it is reindexed by listening to IObjectModified events. Let us create a fake catalog to demonstrate this behaviour. We only need to implement the index_doc method for this test. >>> from zope.catalog.interfaces import ICatalog >>> from zope import interface, component >>> class FakeCatalog(object): ... indexed = [] ... interface.implements(ICatalog) ... def index_doc(self, docid, obj): ... self.indexed.append((docid, obj)) >>> cat = FakeCatalog() >>> component.provideUtility(cat) We also need an intid util and a keyreference adapter. >>> from zope.intid import IntIds >>> from zope.intid.interfaces import IIntIds >>> intids = IntIds() >>> component.provideUtility(intids, IIntIds) >>> from zope.keyreference.testing import SimpleKeyReference >>> component.provideAdapter(SimpleKeyReference) >>> from zope.container.contained import Contained >>> class Dummy(Contained): ... def __init__(self, name): ... self.__name__ = name ... def __repr__(self): ... return '' % self.__name__ We have a subscriber to IIntidAddedEvent. >>> from zope.catalog import catalog >>> from zope.intid.interfaces import IntIdAddedEvent >>> d1 = Dummy(u'one') >>> id1 = intids.register(d1) >>> catalog.indexDocSubscriber(IntIdAddedEvent(d1, None)) Now we have indexed the object. >>> cat.indexed.pop() (..., ) When an object is modified an objectmodified event should be fired by the application. Here is the handler for such an event. >>> from zope.lifecycleevent import ObjectModifiedEvent >>> catalog.reindexDocSubscriber(ObjectModifiedEvent(d1)) >>> len(cat.indexed) 1 >>> cat.indexed.pop() (..., ) Preventing automatic indexing ============================= Sometimes it is not accurate to automatically index an object. For example when a lot of indexes are in the catalog and only specific indexes needs to be updated. There are marker interfaces to achieve this. >>> from zope.catalog.interfaces import INoAutoIndex If an object provides this interface it is not automatically indexed. >>> interface.alsoProvides(d1, INoAutoIndex) >>> catalog.indexDocSubscriber(IntIdAddedEvent(d1, None)) >>> len(cat.indexed) 0 >>> from zope.catalog.interfaces import INoAutoReindex If an object provides this interface it is not automatically reindexed. >>> interface.alsoProvides(d1, INoAutoReindex) >>> catalog.reindexDocSubscriber(ObjectModifiedEvent(d1)) >>> len(cat.indexed) 0 zope.catalog-3.8.2/src/zope/catalog/attribute.py0000664000177100020040000001076611665177362023023 0ustar menesismenesis00000000000000############################################################################# # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Index interface-defined attributes """ __docformat__ = 'restructuredtext' import zope.interface from zope.catalog.interfaces import IAttributeIndex class AttributeIndex(object): """Index interface-defined attributes Mixin for indexing a particular attribute of an object after first adapting the object to be indexed to an interface. The class is meant to be mixed with a base class that defines an index_doc method: >>> class BaseIndex(object): ... def __init__(self): ... self.data = [] ... def index_doc(self, id, value): ... self.data.append((id, value)) The class does two things. The first is to get a named field from an object: >>> class Data: ... def __init__(self, v): ... self.x = v >>> class Index(AttributeIndex, BaseIndex): ... pass >>> index = Index('x') >>> index.index_doc(11, Data(1)) >>> index.index_doc(22, Data(2)) >>> index.data [(11, 1), (22, 2)] A method can be indexed: >>> Data.z = lambda self: self.x + 20 >>> index = Index('z', field_callable=True) >>> index.index_doc(11, Data(1)) >>> index.index_doc(22, Data(2)) >>> index.data [(11, 21), (22, 22)] But you'll get an error if you try to index a method without setting field_callable: >>> index = Index('z') >>> index.index_doc(11, Data(1)) The class can also adapt an object to an interface: >>> from zope.interface import Interface >>> class I(Interface): ... pass >>> class Data: ... def __init__(self, v): ... self.x = v ... def __conform__(self, iface): ... if iface is I: ... return Data2(self.x) >>> class Data2: ... def __init__(self, v): ... self.y = v*v >>> index = Index('y', I) >>> index.index_doc(11, Data(3)) >>> index.index_doc(22, Data(2)) >>> index.data [(11, 9), (22, 4)] When you define an index class, you can define a default interface and/or a default interface: >>> class Index(AttributeIndex, BaseIndex): ... default_interface = I ... default_field_name = 'y' >>> index = Index() >>> index.index_doc(11, Data(3)) >>> index.index_doc(22, Data(2)) >>> index.data [(11, 9), (22, 4)] """ zope.interface.implements(IAttributeIndex) default_field_name = None default_interface = None def __init__(self, field_name=None, interface=None, field_callable=False, *args, **kwargs): super(AttributeIndex, self).__init__(*args, **kwargs) if field_name is None and self.default_field_name is None: raise ValueError("Must pass a field_name") if field_name is None: self.field_name = self.default_field_name else: self.field_name = field_name if interface is None: self.interface = self.default_interface else: self.interface = interface self.field_callable = field_callable def index_doc(self, docid, object): if self.interface is not None: object = self.interface(object, None) if object is None: return None value = getattr(object, self.field_name, None) if value is not None and self.field_callable: #do not eat the exception raised below value = value() if value is None: #unindex the previous value! super(AttributeIndex, self).unindex_doc(docid) return None return super(AttributeIndex, self).index_doc(docid, value) zope.catalog-3.8.2/src/zope/catalog/interfaces.py0000664000177100020040000000616011665177362023134 0ustar menesismenesis00000000000000############################################################################## # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Catalog Interfaces """ import zope.index.interfaces import zope.interface import zope.schema import zope.container.interfaces import zope.container.constraints from zope.i18nmessageid import ZopeMessageFactory as _ class ICatalogQuery(zope.interface.Interface): """Provides Catalog Queries.""" def searchResults(**kw): """Search on the given indexes. Keyword arguments dictionary keys are index names and values are queries for these indexes. Keyword arguments has some special names, used by the catalog itself: * _sort_index - The name of index to sort results with. This index must implement zope.index.interfaces.IIndexSort. * _limit - Limit result set by this number, useful when used with sorting. * _reverse - Reverse result set, also useful with sorting. """ class ICatalogEdit(zope.index.interfaces.IInjection): """Allows one to manipulate the Catalog information.""" def updateIndexes(): """Reindex all objects.""" class ICatalogIndex(zope.index.interfaces.IInjection, zope.index.interfaces.IIndexSearch, ): """An index to be used in a catalog """ __parent__ = zope.schema.Field() zope.container.constraints.containers('.ICatalog') class ICatalog(ICatalogQuery, ICatalogEdit, zope.container.interfaces.IContainer): """Marker to describe a catalog in content space.""" zope.container.constraints.contains(ICatalogIndex) class IAttributeIndex(zope.interface.Interface): """I index objects by first adapting them to an interface, then retrieving a field on the adapted object. """ interface = zope.schema.Choice( title=_(u"Interface"), description=_(u"Objects will be adapted to this interface"), vocabulary="Interfaces", required=False, ) field_name = zope.schema.BytesLine( title=_(u"Field Name"), description=_(u"Name of the field to index"), ) field_callable = zope.schema.Bool( title=_(u"Field Callable"), description=_(u"If true, then the field should be called to get the " u"value to be indexed"), ) class INoAutoIndex(zope.interface.Interface): """Marker for objects that should not be automatically indexed""" class INoAutoReindex(zope.interface.Interface): """Marker for objects that should not be automatically reindexed""" zope.catalog-3.8.2/src/zope/catalog/classes.zcml0000664000177100020040000000260211665177362022760 0ustar menesismenesis00000000000000 zope.catalog-3.8.2/src/zope/catalog/subscribers.zcml0000664000177100020040000000042711665177362023654 0ustar menesismenesis00000000000000 zope.catalog-3.8.2/src/zope/__init__.py0000664000177100020040000000007011665177362021130 0ustar menesismenesis00000000000000__import__('pkg_resources').declare_namespace(__name__) zope.catalog-3.8.2/buildout.cfg0000664000177100020040000000014511665177362017566 0ustar menesismenesis00000000000000[buildout] develop = . parts = test [test] recipe = zc.recipe.testrunner eggs = zope.catalog [test] zope.catalog-3.8.2/README.txt0000664000177100020040000000013511665177362016753 0ustar menesismenesis00000000000000Catalogs provide management of collections of related indexes with a basic search algorithm. zope.catalog-3.8.2/setup.cfg0000664000177100020040000000007311665177633017100 0ustar menesismenesis00000000000000[egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 zope.catalog-3.8.2/LICENSE.txt0000664000177100020040000000402611665177362017103 0ustar menesismenesis00000000000000Zope Public License (ZPL) Version 2.1 A copyright notice accompanies this license document that identifies the copyright holders. This license has been certified as open source. It has also been designated as GPL compatible by the Free Software Foundation (FSF). Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions in source code must retain the accompanying copyright notice, this list of conditions, and the following disclaimer. 2. Redistributions in binary form must reproduce the accompanying copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Names of the copyright holders must not be used to endorse or promote products derived from this software without prior written permission from the copyright holders. 4. The right to distribute this software or to use it for any purpose does not give you the right to use Servicemarks (sm) or Trademarks (tm) of the copyright holders. Use of them is covered by separate agreement with the copyright holders. 5. If any files are modified, you must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. Disclaimer THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.