zope.testrunner-4.4.9/0000775000000000000000000000000012552256734013430 5ustar rootrootzope.testrunner-4.4.9/MANIFEST.in0000664000000000000000000000012012527371226015153 0ustar rootrootinclude *.rst *.py buildout.cfg tox.ini recursive-include src *.py *.txt *.test zope.testrunner-4.4.9/README.rst0000664000000000000000000001126312527371226015116 0ustar rootroot*************** zope.testrunner *************** |buildstatus|_ |winbotstatus|_ .. contents:: This package provides a flexible test runner with layer support. You can find more `detailed documentation`_ on PyPI or in the ``src/`` directory. Getting started *************** Buildout-based projects ======================= zope.testrunner is often used for projects that use buildout_:: [buildout] develop = . parts = ... test ... [test] recipe = zc.recipe.testrunner eggs = mypackage The usual buildout process :: python bootstrap.py bin/buildout creates a ``bin/test`` script that will run the tests for *mypackage*. .. tip:: zc.recipe.testrunner_ takes care to specify the right ``--test-path`` option in the generated script. You can add other options (such as ``--tests-pattern``) too; check zc.recipe.testrunner_'s documentation for details. Virtualenv-based projects ========================= ``pip install zope.testrunner`` and you'll get a ``zope-testrunner`` script. Run your tests with :: zope-testrunner --test-path=path/to/your/source/tree Your source code needs to be available for the testrunner to import, so you need to run ``python setup.py install`` or ``pip install -e .`` into the same virtualenv_. Some useful command-line options to get you started =================================================== -p show a percentage indicator -v increase verbosity -c colorize the output -t test specify test names (one or more regexes) -m module specify test modules (one or more regexes) -s package specify test packages (one or more regexes) --list-tests show names of tests instead of running them -x stop on first error or failure -D, --pdb enable post-mortem debugging of test failures --help show *all* command-line options (there are many more!) For example :: bin/test -pvc -m test_foo -t TestBar runs all TestBar tests from a module called test_foo.py. Writing tests ============= ``zope.testrunner`` expects to find your tests inside your package directory, in a subpackage or module named ``tests``. Test modules in a test subpackage should be named ``test*.py``. .. tip:: You can change these assumptions with ``--tests-pattern`` and ``--test-file-pattern`` test runner options. Tests themselves should be classes inheriting from ``unittest.TestCase``, and if you wish to use doctests, please tell the test runner where to find them and what options to use for them in by supplying a function named ``test_suite``. Example:: import unittest import doctest class TestArithmetic(unittest.TestCase): def test_two_plus_two(self): self.assertEqual(2 + 2, 4) def doctest_string_formatting(): """Test Python string formatting >>> print('{} + {}'.format(2, 2)) 2 + 2 """ def test_suite(): return unittest.TestSuite([ unittest.makeSuite(TestArithmetic), doctest.DocTestSuite(), doctest.DocFileSuite('../README.txt', optionflags=doctest.ELLIPSIS), ]) Test grouping ============= In addition to per-package and per-module filtering, zope.testrunner has other mechanisms for grouping tests: * **layers** allow you to have shared setup/teardown code to be used by a group of tests, that is executed only once, and not for each test. Layers are orthogonal to the usual package/module structure and are specified by setting the ``layer`` attribute on test suites. * **levels** allow you to group slow-running tests and not run them by default. They're specified by setting the ``level`` attribute on test suites to an int. For more details please see the `detailed documentation`_. Other features ============== zope.testrunner can profile your tests, measure test coverage, check for memory leaks, integrate with subunit_, shuffle the test execution order, and run multiple tests in parallel. For more details please see the `detailed documentation`_. .. _buildout: http://www.buildout.org/ .. _virtualenv: http://www.virtualenv.org/ .. _zc.recipe.testrunner: http://pypi.python.org/pypi/zc.recipe.testrunner .. _subunit: http://pypi.python.org/pypi/subunit .. _detailed documentation: http://docs.zope.org/zope.testrunner/ .. |buildstatus| image:: https://api.travis-ci.org/zopefoundation/zope.testrunner.png?branch=master .. _buildstatus: https://travis-ci.org/zopefoundation/zope.testrunner .. |winbotstatus| image:: http://winbot.zope.org/buildstatusimage?builder=zope.testrunner_py_265_32&number=-1 .. _winbotstatus: http://winbot.zope.org/builders/zope.testrunner_py_265_32/builds/-1 zope.testrunner-4.4.9/PKG-INFO0000664000000000000000000072423412527371244014535 0ustar rootrootMetadata-Version: 1.1 Name: zope.testrunner Version: 4.4.9 Summary: Zope testrunner script. Home-page: http://pypi.python.org/pypi/zope.testrunner Author: Zope Foundation and Contributors Author-email: zope-dev@zope.org License: ZPL 2.1 Description: *************** zope.testrunner *************** |buildstatus|_ |winbotstatus|_ .. contents:: This package provides a flexible test runner with layer support. You can find more `detailed documentation`_ on PyPI or in the ``src/`` directory. Getting started *************** Buildout-based projects ======================= zope.testrunner is often used for projects that use buildout_:: [buildout] develop = . parts = ... test ... [test] recipe = zc.recipe.testrunner eggs = mypackage The usual buildout process :: python bootstrap.py bin/buildout creates a ``bin/test`` script that will run the tests for *mypackage*. .. tip:: zc.recipe.testrunner_ takes care to specify the right ``--test-path`` option in the generated script. You can add other options (such as ``--tests-pattern``) too; check zc.recipe.testrunner_'s documentation for details. Virtualenv-based projects ========================= ``pip install zope.testrunner`` and you'll get a ``zope-testrunner`` script. Run your tests with :: zope-testrunner --test-path=path/to/your/source/tree Your source code needs to be available for the testrunner to import, so you need to run ``python setup.py install`` or ``pip install -e .`` into the same virtualenv_. Some useful command-line options to get you started =================================================== -p show a percentage indicator -v increase verbosity -c colorize the output -t test specify test names (one or more regexes) -m module specify test modules (one or more regexes) -s package specify test packages (one or more regexes) --list-tests show names of tests instead of running them -x stop on first error or failure -D, --pdb enable post-mortem debugging of test failures --help show *all* command-line options (there are many more!) For example :: bin/test -pvc -m test_foo -t TestBar runs all TestBar tests from a module called test_foo.py. Writing tests ============= ``zope.testrunner`` expects to find your tests inside your package directory, in a subpackage or module named ``tests``. Test modules in a test subpackage should be named ``test*.py``. .. tip:: You can change these assumptions with ``--tests-pattern`` and ``--test-file-pattern`` test runner options. Tests themselves should be classes inheriting from ``unittest.TestCase``, and if you wish to use doctests, please tell the test runner where to find them and what options to use for them in by supplying a function named ``test_suite``. Example:: import unittest import doctest class TestArithmetic(unittest.TestCase): def test_two_plus_two(self): self.assertEqual(2 + 2, 4) def doctest_string_formatting(): """Test Python string formatting >>> print('{} + {}'.format(2, 2)) 2 + 2 """ def test_suite(): return unittest.TestSuite([ unittest.makeSuite(TestArithmetic), doctest.DocTestSuite(), doctest.DocFileSuite('../README.txt', optionflags=doctest.ELLIPSIS), ]) Test grouping ============= In addition to per-package and per-module filtering, zope.testrunner has other mechanisms for grouping tests: * **layers** allow you to have shared setup/teardown code to be used by a group of tests, that is executed only once, and not for each test. Layers are orthogonal to the usual package/module structure and are specified by setting the ``layer`` attribute on test suites. * **levels** allow you to group slow-running tests and not run them by default. They're specified by setting the ``level`` attribute on test suites to an int. For more details please see the `detailed documentation`_. Other features ============== zope.testrunner can profile your tests, measure test coverage, check for memory leaks, integrate with subunit_, shuffle the test execution order, and run multiple tests in parallel. For more details please see the `detailed documentation`_. .. _buildout: http://www.buildout.org/ .. _virtualenv: http://www.virtualenv.org/ .. _zc.recipe.testrunner: http://pypi.python.org/pypi/zc.recipe.testrunner .. _subunit: http://pypi.python.org/pypi/subunit .. _detailed documentation: http://docs.zope.org/zope.testrunner/ .. |buildstatus| image:: https://api.travis-ci.org/zopefoundation/zope.testrunner.png?branch=master .. _buildstatus: https://travis-ci.org/zopefoundation/zope.testrunner .. |winbotstatus| image:: http://winbot.zope.org/buildstatusimage?builder=zope.testrunner_py_265_32&number=-1 .. _winbotstatus: http://winbot.zope.org/builders/zope.testrunner_py_265_32/builds/-1 zope.testrunner Changelog ************************* 4.4.9 (2015-05-21) ================== - When using ``-j``, parallelize all the tests, including the first test layer (`#28 `_). 4.4.8 (2015-05-01) ================== - Support skipped tests in subunit output (`#25 `_). - More efficient test filtering (`#26 `_). 4.4.7 (2015-04-02) ================== - Work around a bug in PyPy3's curses module (`#24 `_). 4.4.6 (2015-01-21) ================== - Restore support for instance-based test layers that regressed in 4.4.5 (`#20 `_). 4.4.5 (2015-01-06) ================== - Sort related layers close to each other to reduce the number of unnecessary teardowns (fixes `#14 `_). - Run the unit test layer first (fixes `LP #497871 `__). 4.4.4 (2014-12-27) ================== - When looking for the right location of test code, start with longest location paths first. This fixes problems with nested code locations. 4.4.3 (2014-03-19) ================== - Added support for Python 3.4. 4.4.2 (2014-02-22) ================== - Drop support for Python 3.1. - Fix post-mortem debugging when a non-printable exception happens (https://github.com/zopefoundation/zope.testrunner/issues/8). 4.4.1 (2013-07-10) ================== - Updated ``boostrap.py`` to version 2.2. - Fix nondeterministic test failures on Python 3.3 - Tear down layers after ``post_mortem`` debugging is finished. - Fix tests that write to source directory, it might be read-only. 4.4.0 (2013-06-06) ================== - Fix tests selection when the negative "!" pattern is used several times (LP #1160965) - Moved tests into a 'tests' subpackage. - Made ``python -m zope.testrunner`` work again. - Support 'skip' feature of unittest2 (which became the new unittest in Python 2.7). - Better diagnostics when communication with subprocess fails (https://github.com/zopefoundation/zope.testrunner/issues/5). - Do not break subprocess execution when the test suite changes the working directory (https://github.com/zopefoundation/zope.testrunner/issues/6). - Count test module import errors as errors (LP #1026576). 4.3.3 (2013-03-03) ================== - Running layers in sub-processes did not use to work when run via ``python setup.py ftest`` since it tried to run setup.py with all the command line options. It now detects ``setup.py`` runs and we run the test runner directly. 4.3.2 (2013-03-03) ================== - Fix ``SkipLayers`` class in cases where the distribution specifies a ``test_suite`` value. 4.3.1 (2013-03-02) ================== - Fixed a bug in the `ftest` command and added a test. - Fixed a trivial test failure with Python 3 of the previous release. 4.3.0 (2013-03-02) ================== - Expose `ftest` distutils command via an entry point. - Added tests for ``zope.testrunner.eggsupport``. 4.2.0 (2013-02-12) ================== - Dropped use of 2to3, rewrote source code to be compatible with all Python versions. Introduced a dependency on `six`_. 4.1.1 (2013-02-08) ================== - Dropped use of zope.fixers (LP: #1118877). - Fixed tox test error reporting; fixed tests on Pythons 2.6, 3.1, 3.2, 3.3 and PyPy 1.9. - Fix --shuffle ordering on Python 3.2 to be the same as it was on older Python versions. - Fix --shuffle nondeterminism when multiple test layers are present. Note: this will likely change the order of tests for the same --shuffle-seed. - New option: --profile-directory. Use it in the test suite so that tests executed by detox in parallel don't conflict. - Use a temporary coverage directory in the test suite so that tests executed by detox in parallel don't conflict. - Fix --post-mortem (aka -D, --pdb) when a test module cannot be imported or is invalid (LP #1119363). 4.1.0 (2013-02-07) ================== - Replaced deprecated ``zope.interface.implements`` usage with equivalent ``zope.interface.implementer`` decorator. - Dropped support for Python 2.4 and 2.5. - Made StartUpFailure compatible with unittest.TextTestRunner() (LP #1118344). 4.0.4 (2011-10-25) ================== - Work around sporadic timing-related issues in the subprocess buffering tests. Thanks to Jonathan Ballet for the patch! 4.0.3 (2011-03-17) ================== - Added back support for Python <= 2.6 which was broken in 4.0.2. 4.0.2 (2011-03-16) ================== - Added back Python 3 support which was broken in 4.0.1. - Fixed `Unexpected success`_ support by implementing the whole concept. - Added support for the new __pycache__ directories in Python 3.2. 4.0.1 (2011-02-21) ================== - LP #719369: An `Unexpected success`_ (concept introduced in Python 2.7) is no longer handled as success but as failure. This is a workaround. The whole unexpected success concept might be implemented later. .. _`Unexpected success`: http://www.voidspace.org.uk/python/articles/unittest2.shtml#more-skipping 4.0.0 (2010-10-19) ================== - Show more information about layers whose setup fails (LP #638153). 4.0.0b5 (2010-07-20) ==================== - Update fix for LP #221151 to a spelling compatible with Python 2.4. - Timestamps are now always included in subunit output (r114849). - LP #591309: fix a crash when subunit reports test failures containing UTF8-encoded data. 4.0.0b4 (2010-06-23) ==================== - Package as a zipfile to work around Python 2.4 distutils bug (no feature changes or bugfixes in ``zope.testrunner`` itself). 4.0.0b3 (2010-06-16) ==================== - LP #221151: keep ``unittest.TestCase.shortDescription`` happy by supplying a ``_testMethodDoc`` attribute. - LP #595052: keep the distribution installable under Python 2.4: its distutils appears to munge the empty ``__init__.py`` file in the ``foo.bar`` egg used for testing into a directory. - LP #580083: fix the ``bin/test`` script to run only tests from ``zope.testrunner``. - LP #579019: When layers were run in parallel, their tearDown was not called. Additionally, the first layer which was run in the main thread did not have its tearDown called either. 4.0.0b2 (2010-05-03) ==================== - Having 'sampletests' in the MANIFEST.in gave warnings, but doesn't actually seem to include any more files, so I removed it. - Moved zope.testing.exceptions to zope.testrunner.exceptions. Now zope.testrunner no longer requires zope.testing except for when running its own tests. 4.0.0b1 (2010-04-29) ==================== - Initial release of the testrunner from zope.testrunner as its own module. (Previously it was part of zope.testing.) .. _six: http://pypi.python.org/pypi/six Detailed Documentation ********************** Test Runner =========== The testrunner module is used to run automated tests defined using the unittest framework. Its primary feature is that it *finds* tests by searching directory trees. It doesn't require the manual concatenation of specific test suites. It is highly customizable and should be usable with any project. In addition to finding and running tests, it provides the following additional features: - Test filtering using specifications of: o test packages within a larger tree o regular expression patterns for test modules o regular expression patterns for individual tests - Organization of tests into levels and layers Sometimes, tests take so long to run that you don't want to run them on every run of the test runner. Tests can be defined at different levels. The test runner can be configured to only run tests at a specific level or below by default. Command-line options can be used to specify a minimum level to use for a specific run, or to run all tests. Individual tests or test suites can specify their level via a 'level' attribute. where levels are integers increasing from 1. Most tests are unit tests. They don't depend on other facilities, or set up whatever dependencies they have. For larger applications, it's useful to specify common facilities that a large number of tests share. Making each test set up and and tear down these facilities is both ineffecient and inconvenient. For this reason, we've introduced the concept of layers, based on the idea of layered application architectures. Software build for a layer should be able to depend on the facilities of lower layers already being set up. For example, Zope defines a component architecture. Much Zope software depends on that architecture. We should be able to treat the component architecture as a layer that we set up once and reuse. Similarly, Zope application software should be able to depend on the Zope application server without having to set it up in each test. The test runner introduces test layers, which are objects that can set up environments for tests within the layers to use. A layer is set up before running the tests in it. Individual tests or test suites can define a layer by defining a `layer` attribute, which is a test layer. - Reporting - progress meter - summaries of tests run - Analysis of test execution - post-mortem debugging of test failures - memory leaks - code coverage - source analysis using pychecker - memory errors - execution times - profiling Simple Usage ============ The test runner consists of an importable module. The test runner is used by providing scripts that import and invoke the `run` method from the module. The `testrunner` module is controlled via command-line options. Test scripts supply base and default options by supplying a list of default command-line options that are processed before the user-supplied command-line options are provided. Typically, a test script does 2 things: - Adds the directory containing the zope package to the Python path. - Calls the test runner with default arguments and arguments supplied to the script. Normally, it just passes default/setup arguments. The test runner uses `sys.argv` to get the user's input. This testrunner_ex subdirectory contains a number of sample packages with tests. Let's run the tests found here. First though, we'll set up our default options: >>> import os.path >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] The default options are used by a script to customize the test runner for a particular application. In this case, we use two options: path Set the path where the test runner should look for tests. This path is also added to the Python path. tests-pattern Tell the test runner how to recognize modules or packages containing tests. Now, if we run the tests, without any other options: >>> from zope import testrunner >>> import sys >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False we see the normal testrunner output, which summarizes the tests run for each layer. For each layer, we see what layers had to be torn down or set up to run the layer and we see the number of tests run, with results. The test runner returns a boolean indicating whether there were errors. In this example, there were no errors, so it returned False. (Of course, the times shown in these examples are just examples. Times will vary depending on system speed.) Layers ====== A Layer is an object providing setup and teardown methods used to setup and teardown the environment provided by the layer. It may also provide setup and teardown methods used to reset the environment provided by the layer between each test. Layers are generally implemented as classes using class methods. >>> class BaseLayer: ... def setUp(cls): ... log('BaseLayer.setUp') ... setUp = classmethod(setUp) ... ... def tearDown(cls): ... log('BaseLayer.tearDown') ... tearDown = classmethod(tearDown) ... ... def testSetUp(cls): ... log('BaseLayer.testSetUp') ... testSetUp = classmethod(testSetUp) ... ... def testTearDown(cls): ... log('BaseLayer.testTearDown') ... testTearDown = classmethod(testTearDown) ... Layers can extend other layers. Note that they do not explicitly invoke the setup and teardown methods of other layers - the test runner does this for us in order to minimize the number of invocations. >>> class TopLayer(BaseLayer): ... def setUp(cls): ... log('TopLayer.setUp') ... setUp = classmethod(setUp) ... ... def tearDown(cls): ... log('TopLayer.tearDown') ... tearDown = classmethod(tearDown) ... ... def testSetUp(cls): ... log('TopLayer.testSetUp') ... testSetUp = classmethod(testSetUp) ... ... def testTearDown(cls): ... log('TopLayer.testTearDown') ... testTearDown = classmethod(testTearDown) ... Tests or test suites specify what layer they need by storing a reference in the 'layer' attribute. >>> import unittest >>> class TestSpecifyingBaseLayer(unittest.TestCase): ... 'This TestCase explicitly specifies its layer' ... layer = BaseLayer ... name = 'TestSpecifyingBaseLayer' # For testing only ... ... def setUp(self): ... log('TestSpecifyingBaseLayer.setUp') ... ... def tearDown(self): ... log('TestSpecifyingBaseLayer.tearDown') ... ... def test1(self): ... log('TestSpecifyingBaseLayer.test1') ... ... def test2(self): ... log('TestSpecifyingBaseLayer.test2') ... >>> class TestSpecifyingNoLayer(unittest.TestCase): ... 'This TestCase specifies no layer' ... name = 'TestSpecifyingNoLayer' # For testing only ... def setUp(self): ... log('TestSpecifyingNoLayer.setUp') ... ... def tearDown(self): ... log('TestSpecifyingNoLayer.tearDown') ... ... def test1(self): ... log('TestSpecifyingNoLayer.test') ... ... def test2(self): ... log('TestSpecifyingNoLayer.test') ... Create a TestSuite containing two test suites, one for each of TestSpecifyingBaseLayer and TestSpecifyingNoLayer. >>> umbrella_suite = unittest.TestSuite() >>> umbrella_suite.addTest(unittest.makeSuite(TestSpecifyingBaseLayer)) >>> no_layer_suite = unittest.makeSuite(TestSpecifyingNoLayer) >>> umbrella_suite.addTest(no_layer_suite) Before we can run the tests, we need to setup some helpers. >>> from zope.testrunner import options >>> from zope.testing.loggingsupport import InstalledHandler >>> import logging >>> log_handler = InstalledHandler('zope.testrunner.tests') >>> def log(msg): ... logging.getLogger('zope.testrunner.tests').info(msg) >>> def fresh_options(): ... opts = options.get_options(['--test-filter', '.*']) ... opts.resume_layer = None ... opts.resume_number = 0 ... return opts Now we run the tests. Note that the BaseLayer was not setup when the TestSpecifyingNoLayer was run and setup/torn down around the TestSpecifyingBaseLayer tests. >>> from zope.testrunner.runner import Runner >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running ...BaseLayer tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up ...BaseLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...BaseLayer in N.NNN seconds. Total: 4 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. Now lets specify a layer in the suite containing TestSpecifyingNoLayer and run the tests again. This demonstrates the other method of specifying a layer. This is generally how you specify what layer doctests need. >>> no_layer_suite.layer = BaseLayer >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running ...BaseLayer tests: Set up ...BaseLayer in N.NNN seconds. Ran 4 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...BaseLayer in N.NNN seconds. Clear our logged output, as we want to inspect it shortly. >>> log_handler.clear() Now lets also specify a layer in the TestSpecifyingNoLayer class and rerun the tests. This demonstrates that the most specific layer is used. It also shows the behavior of nested layers - because TopLayer extends BaseLayer, both the BaseLayer and TopLayer environments are setup when the TestSpecifyingNoLayer tests are run. >>> TestSpecifyingNoLayer.layer = TopLayer >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running ...BaseLayer tests: Set up ...BaseLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running ...TopLayer tests: Set up ...TopLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...TopLayer in N.NNN seconds. Tear down ...BaseLayer in N.NNN seconds. Total: 4 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. If we inspect our trace of what methods got called in what order, we can see that the layer setup and teardown methods only got called once. We can also see that the layer's test setup and teardown methods got called for each test using that layer in the right order. >>> def report(): ... print("Report:") ... for record in log_handler.records: ... print(record.getMessage()) >>> report() Report: BaseLayer.setUp BaseLayer.testSetUp TestSpecifyingBaseLayer.setUp TestSpecifyingBaseLayer.test1 TestSpecifyingBaseLayer.tearDown BaseLayer.testTearDown BaseLayer.testSetUp TestSpecifyingBaseLayer.setUp TestSpecifyingBaseLayer.test2 TestSpecifyingBaseLayer.tearDown BaseLayer.testTearDown TopLayer.setUp BaseLayer.testSetUp TopLayer.testSetUp TestSpecifyingNoLayer.setUp TestSpecifyingNoLayer.test TestSpecifyingNoLayer.tearDown TopLayer.testTearDown BaseLayer.testTearDown BaseLayer.testSetUp TopLayer.testSetUp TestSpecifyingNoLayer.setUp TestSpecifyingNoLayer.test TestSpecifyingNoLayer.tearDown TopLayer.testTearDown BaseLayer.testTearDown TopLayer.tearDown BaseLayer.tearDown Now lets stack a few more layers to ensure that our setUp and tearDown methods are called in the correct order. >>> from zope.testrunner.find import name_from_layer >>> class A(object): ... def setUp(cls): ... log('%s.setUp' % name_from_layer(cls)) ... setUp = classmethod(setUp) ... ... def tearDown(cls): ... log('%s.tearDown' % name_from_layer(cls)) ... tearDown = classmethod(tearDown) ... ... def testSetUp(cls): ... log('%s.testSetUp' % name_from_layer(cls)) ... testSetUp = classmethod(testSetUp) ... ... def testTearDown(cls): ... log('%s.testTearDown' % name_from_layer(cls)) ... testTearDown = classmethod(testTearDown) ... >>> class B(A): pass >>> class C(B): pass >>> class D(A): pass >>> class E(D): pass >>> class F(C,E): pass >>> class DeepTest(unittest.TestCase): ... layer = F ... def test(self): ... pass >>> suite = unittest.makeSuite(DeepTest) >>> log_handler.clear() >>> runner = Runner(options=fresh_options(), args=[], found_suites=[suite]) >>> succeeded = runner.run() #doctest: +ELLIPSIS Running ...F tests: Set up ...A in N.NNN seconds. Set up ...B in N.NNN seconds. Set up ...C in N.NNN seconds. Set up ...D in N.NNN seconds. Set up ...E in N.NNN seconds. Set up ...F in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...F in N.NNN seconds. Tear down ...E in N.NNN seconds. Tear down ...D in N.NNN seconds. Tear down ...C in N.NNN seconds. Tear down ...B in N.NNN seconds. Tear down ...A in N.NNN seconds. >>> report() #doctest: +ELLIPSIS Report: ...A.setUp ...B.setUp ...C.setUp ...D.setUp ...E.setUp ...F.setUp ...A.testSetUp ...B.testSetUp ...C.testSetUp ...D.testSetUp ...E.testSetUp ...F.testSetUp ...F.testTearDown ...E.testTearDown ...D.testTearDown ...C.testTearDown ...B.testTearDown ...A.testTearDown ...F.tearDown ...E.tearDown ...D.tearDown ...C.tearDown ...B.tearDown ...A.tearDown Layer Selection =============== We can select which layers to run using the --layer option: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 112 --layer Unit'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 182 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False We can also specify that we want to run only the unit tests: >>> sys.argv = 'test -u'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Or that we want to run all of the tests except for the unit tests: >>> sys.argv = 'test -f'.split() >>> testrunner.run_internal(defaults) Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 165 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Or we can explicitly say that we want both unit and non-unit tests. >>> sys.argv = 'test -uf'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False It is possible to force the layers to run in subprocesses and parallelize them. ``EmptyLayer`` will be inserted as first to start spreading out subprocesses ASAP. >>> sys.argv = [testrunner_script, '-j2'] >>> testrunner.run_internal(defaults) Running .EmptyLayer tests: Set up .EmptyLayer in N.NNN seconds. Ran 0 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running zope.testrunner.layer.UnitTests tests: Running in a subprocess. Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Running samplelayers.Layer1 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer11 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer111 tests: Running in a subprocess. Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer111 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer112 tests: Running in a subprocess. Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer12 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer121 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer121 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer122 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Tearing down left over layers: Tear down .EmptyLayer in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Passing arguments explicitly ============================ In most of the examples here, we set up `sys.argv`. In normal usage, the testrunner just uses `sys.argv`. It is possible to pass arguments explicitly. >>> import os.path >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> testrunner.run_internal(defaults, 'test --layer 111'.split()) Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False If options already have default values, then passing a different default will override. For example, --list-tests defaults to being turned off, but if we pass in a different default, that one takes effect. >>> defaults = [ ... '--list-tests', ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> testrunner.run_internal(defaults, 'test --layer 111'.split()) Listing samplelayers.Layer111 tests: test_x1 (sample1.sampletests.test111.TestA) test_y0 (sample1.sampletests.test111.TestA) test_z0 (sample1.sampletests.test111.TestA) test_x0 (sample1.sampletests.test111.TestB) test_y1 (sample1.sampletests.test111.TestB) test_z0 (sample1.sampletests.test111.TestB) test_1 (sample1.sampletests.test111.TestNotMuch) test_2 (sample1.sampletests.test111.TestNotMuch) test_3 (sample1.sampletests.test111.TestNotMuch) test_x0 (sample1.sampletests.test111) test_y0 (sample1.sampletests.test111) test_z1 (sample1.sampletests.test111) /home/benji/workspace/zope.testrunner/1/src/zope/testing/testrunner/testrunner-ex/sample1/sampletests/../../sampletestsl.txt test_x1 (sampletests.test111.TestA) test_y0 (sampletests.test111.TestA) test_z0 (sampletests.test111.TestA) test_x0 (sampletests.test111.TestB) test_y1 (sampletests.test111.TestB) test_z0 (sampletests.test111.TestB) test_1 (sampletests.test111.TestNotMuch) test_2 (sampletests.test111.TestNotMuch) test_3 (sampletests.test111.TestNotMuch) test_x0 (sampletests.test111) test_y0 (sampletests.test111) test_z1 (sampletests.test111) /home/benji/workspace/zope.testrunner/1/src/zope/testing/testrunner/testrunner-ex/sampletests/../sampletestsl.txt False Verbose Output ============== Normally, we just get a summary. We can use the -v option to get increasingly more information. If we use a single --verbose (-v) option, we get a dot printed for each test: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 122 -v'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: .................................. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.007 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False If there are more than 50 tests, the dots are printed in groups of 50: >>> sys.argv = 'test -uv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: ................................................................................................................................................................................................ Ran 156 tests with 0 failures, 0 errors and 0 skipped in 0.035 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False If the --verbose (-v) option is used twice, then the name and location of each test is printed as it is run: >>> sys.argv = 'test --layer 122 -vv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122.TestA) test_z0 (sample1.sampletests.test122.TestA) test_x0 (sample1.sampletests.test122.TestB) test_y1 (sample1.sampletests.test122.TestB) test_z0 (sample1.sampletests.test122.TestB) test_1 (sample1.sampletests.test122.TestNotMuch) test_2 (sample1.sampletests.test122.TestNotMuch) test_3 (sample1.sampletests.test122.TestNotMuch) test_x0 (sample1.sampletests.test122) test_y0 (sample1.sampletests.test122) test_z1 (sample1.sampletests.test122) testrunner-ex/sample1/sampletests/../../sampletestsl.txt test_x1 (sampletests.test122.TestA) test_y0 (sampletests.test122.TestA) test_z0 (sampletests.test122.TestA) test_x0 (sampletests.test122.TestB) test_y1 (sampletests.test122.TestB) test_z0 (sampletests.test122.TestB) test_1 (sampletests.test122.TestNotMuch) test_2 (sampletests.test122.TestNotMuch) test_3 (sampletests.test122.TestNotMuch) test_x0 (sampletests.test122) test_y0 (sampletests.test122) test_z1 (sampletests.test122) testrunner-ex/sampletests/../sampletestsl.txt Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.009 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False if the --verbose (-v) option is used three times, then individual test-execution times are printed: >>> sys.argv = 'test --layer 122 -vvv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) (0.000 s) test_y0 (sample1.sampletests.test122.TestA) (0.000 s) test_z0 (sample1.sampletests.test122.TestA) (0.000 s) test_x0 (sample1.sampletests.test122.TestB) (0.000 s) test_y1 (sample1.sampletests.test122.TestB) (0.000 s) test_z0 (sample1.sampletests.test122.TestB) (0.000 s) test_1 (sample1.sampletests.test122.TestNotMuch) (0.000 s) test_2 (sample1.sampletests.test122.TestNotMuch) (0.000 s) test_3 (sample1.sampletests.test122.TestNotMuch) (0.000 s) test_x0 (sample1.sampletests.test122) (0.001 s) test_y0 (sample1.sampletests.test122) (0.001 s) test_z1 (sample1.sampletests.test122) (0.001 s) testrunner-ex/sample1/sampletests/../../sampletestsl.txt (0.001 s) test_x1 (sampletests.test122.TestA) (0.000 s) test_y0 (sampletests.test122.TestA) (0.000 s) test_z0 (sampletests.test122.TestA) (0.000 s) test_x0 (sampletests.test122.TestB) (0.000 s) test_y1 (sampletests.test122.TestB) (0.000 s) test_z0 (sampletests.test122.TestB) (0.000 s) test_1 (sampletests.test122.TestNotMuch) (0.000 s) test_2 (sampletests.test122.TestNotMuch) (0.000 s) test_3 (sampletests.test122.TestNotMuch) (0.000 s) test_x0 (sampletests.test122) (0.001 s) test_y0 (sampletests.test122) (0.001 s) test_z1 (sampletests.test122) (0.001 s) testrunner-ex/sampletests/../sampletestsl.txt (0.001 s) Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.009 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False Quiet output ------------ The --quiet (-q) option cancels all verbose options. It's useful when the default verbosity is non-zero: >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... '-v' ... ] >>> sys.argv = 'test -q -u'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in 0.034 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Test Selection ============== We've already seen that we can select tests by layer. There are three other ways we can select tests. We can select tests by package: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 122 -ssample1 -vv'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122.TestA) test_z0 (sample1.sampletests.test122.TestA) test_x0 (sample1.sampletests.test122.TestB) test_y1 (sample1.sampletests.test122.TestB) test_z0 (sample1.sampletests.test122.TestB) test_1 (sample1.sampletests.test122.TestNotMuch) test_2 (sample1.sampletests.test122.TestNotMuch) test_3 (sample1.sampletests.test122.TestNotMuch) test_x0 (sample1.sampletests.test122) test_y0 (sample1.sampletests.test122) test_z1 (sample1.sampletests.test122) testrunner-ex/sample1/sampletests/../../sampletestsl.txt Ran 13 tests with 0 failures, 0 errors and 0 skipped in 0.005 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False You can specify multiple packages: >>> sys.argv = 'test -u -vv -ssample1 -ssample2'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf.TestA) test_z0 (sample1.sampletestsf.TestA) test_x0 (sample1.sampletestsf.TestB) test_y1 (sample1.sampletestsf.TestB) test_z0 (sample1.sampletestsf.TestB) test_1 (sample1.sampletestsf.TestNotMuch) test_2 (sample1.sampletestsf.TestNotMuch) test_3 (sample1.sampletestsf.TestNotMuch) test_x0 (sample1.sampletestsf) test_y0 (sample1.sampletestsf) test_z1 (sample1.sampletestsf) testrunner-ex/sample1/../sampletests.txt test_x1 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sample11.sampletests.TestA) test_z0 (sample1.sample11.sampletests.TestA) test_x0 (sample1.sample11.sampletests.TestB) test_y1 (sample1.sample11.sampletests.TestB) test_z0 (sample1.sample11.sampletests.TestB) test_1 (sample1.sample11.sampletests.TestNotMuch) test_2 (sample1.sample11.sampletests.TestNotMuch) test_3 (sample1.sample11.sampletests.TestNotMuch) test_x0 (sample1.sample11.sampletests) test_y0 (sample1.sample11.sampletests) test_z1 (sample1.sample11.sampletests) testrunner-ex/sample1/sample11/../../sampletests.txt test_x1 (sample1.sample13.sampletests.TestA) test_y0 (sample1.sample13.sampletests.TestA) test_z0 (sample1.sample13.sampletests.TestA) test_x0 (sample1.sample13.sampletests.TestB) test_y1 (sample1.sample13.sampletests.TestB) test_z0 (sample1.sample13.sampletests.TestB) test_1 (sample1.sample13.sampletests.TestNotMuch) test_2 (sample1.sample13.sampletests.TestNotMuch) test_3 (sample1.sample13.sampletests.TestNotMuch) test_x0 (sample1.sample13.sampletests) test_y0 (sample1.sample13.sampletests) test_z1 (sample1.sample13.sampletests) testrunner-ex/sample1/sample13/../../sampletests.txt test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample2.sample21.sampletests.TestA) test_y0 (sample2.sample21.sampletests.TestA) test_z0 (sample2.sample21.sampletests.TestA) test_x0 (sample2.sample21.sampletests.TestB) test_y1 (sample2.sample21.sampletests.TestB) test_z0 (sample2.sample21.sampletests.TestB) test_1 (sample2.sample21.sampletests.TestNotMuch) test_2 (sample2.sample21.sampletests.TestNotMuch) test_3 (sample2.sample21.sampletests.TestNotMuch) test_x0 (sample2.sample21.sampletests) test_y0 (sample2.sample21.sampletests) test_z1 (sample2.sample21.sampletests) testrunner-ex/sample2/sample21/../../sampletests.txt test_x1 (sample2.sampletests.test_1.TestA) test_y0 (sample2.sampletests.test_1.TestA) test_z0 (sample2.sampletests.test_1.TestA) test_x0 (sample2.sampletests.test_1.TestB) test_y1 (sample2.sampletests.test_1.TestB) test_z0 (sample2.sampletests.test_1.TestB) test_1 (sample2.sampletests.test_1.TestNotMuch) test_2 (sample2.sampletests.test_1.TestNotMuch) test_3 (sample2.sampletests.test_1.TestNotMuch) test_x0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.test_1) test_z1 (sample2.sampletests.test_1) testrunner-ex/sample2/sampletests/../../sampletests.txt test_x1 (sample2.sampletests.testone.TestA) test_y0 (sample2.sampletests.testone.TestA) test_z0 (sample2.sampletests.testone.TestA) test_x0 (sample2.sampletests.testone.TestB) test_y1 (sample2.sampletests.testone.TestB) test_z0 (sample2.sampletests.testone.TestB) test_1 (sample2.sampletests.testone.TestNotMuch) test_2 (sample2.sampletests.testone.TestNotMuch) test_3 (sample2.sampletests.testone.TestNotMuch) test_x0 (sample2.sampletests.testone) test_y0 (sample2.sampletests.testone) test_z1 (sample2.sampletests.testone) testrunner-ex/sample2/sampletests/../../sampletests.txt Ran 104 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False You can specify directory names instead of packages (useful for tab-completion): >>> subdir = os.path.join(directory_with_tests, 'sample1') >>> sys.argv = ['test', '--layer', '122', '-s', subdir, '-vv'] >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122.TestA) test_z0 (sample1.sampletests.test122.TestA) test_x0 (sample1.sampletests.test122.TestB) test_y1 (sample1.sampletests.test122.TestB) test_z0 (sample1.sampletests.test122.TestB) test_1 (sample1.sampletests.test122.TestNotMuch) test_2 (sample1.sampletests.test122.TestNotMuch) test_3 (sample1.sampletests.test122.TestNotMuch) test_x0 (sample1.sampletests.test122) test_y0 (sample1.sampletests.test122) test_z1 (sample1.sampletests.test122) testrunner-ex/sample1/sampletests/../../sampletestsl.txt Ran 13 tests with 0 failures, 0 errors and 0 skipped in 0.005 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False We can select by test module name using the --module (-m) option: >>> sys.argv = 'test -u -vv -ssample1 -m_one -mtest1'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False and by test within the module using the --test (-t) option: >>> sys.argv = 'test -u -vv -ssample1 -m_one -mtest1 -tx0 -ty0'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) Ran 8 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False >>> sys.argv = 'test -u -vv -ssample1 -ttxt'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: testrunner-ex/sample1/../sampletests.txt testrunner-ex/sample1/sample11/../../sampletests.txt testrunner-ex/sample1/sample13/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 5 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False The --module and --test options take regular expressions. If the regular expressions specified begin with '!', then tests that don't match the regular expression are selected: >>> sys.argv = 'test -u -vv -ssample1 -m!sample1[.]sample1'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf.TestA) test_z0 (sample1.sampletestsf.TestA) test_x0 (sample1.sampletestsf.TestB) test_y1 (sample1.sampletestsf.TestB) test_z0 (sample1.sampletestsf.TestB) test_1 (sample1.sampletestsf.TestNotMuch) test_2 (sample1.sampletestsf.TestNotMuch) test_3 (sample1.sampletestsf.TestNotMuch) test_x0 (sample1.sampletestsf) test_y0 (sample1.sampletestsf) test_z1 (sample1.sampletestsf) testrunner-ex/sample1/../sampletests.txt test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 39 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Module and test filters can also be given as positional arguments: >>> sys.argv = 'test -u -vv -ssample1 !sample1[.]sample1'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf.TestA) test_z0 (sample1.sampletestsf.TestA) test_x0 (sample1.sampletestsf.TestB) test_y1 (sample1.sampletestsf.TestB) test_z0 (sample1.sampletestsf.TestB) test_1 (sample1.sampletestsf.TestNotMuch) test_2 (sample1.sampletestsf.TestNotMuch) test_3 (sample1.sampletestsf.TestNotMuch) test_x0 (sample1.sampletestsf) test_y0 (sample1.sampletestsf) test_z1 (sample1.sampletestsf) testrunner-ex/sample1/../sampletests.txt test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 39 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False >>> sys.argv = 'test -u -vv -ssample1 . txt'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: testrunner-ex/sample1/../sampletests.txt testrunner-ex/sample1/sample11/../../sampletests.txt testrunner-ex/sample1/sample13/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 5 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Sometimes, There are tests that you don't want to run by default. For example, you might have tests that take a long time. Tests can have a level attribute. If no level is specified, a level of 1 is assumed and, by default, only tests at level one are run. to run tests at a higher level, use the --at-level (-a) option to specify a higher level. For example, with the following options: >>> sys.argv = 'test -u -vv -t test_y1 -t test_y0'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sampletestsf.TestA) test_y1 (sampletestsf.TestB) test_y0 (sampletestsf) test_y0 (sample1.sampletestsf.TestA) test_y1 (sample1.sampletestsf.TestB) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y1 (sample1.sample11.sampletests.TestB) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y1 (sample1.sample13.sampletests.TestB) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y1 (sample1.sampletests.test1.TestB) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y1 (sample1.sampletests.test_one.TestB) test_y0 (sample1.sampletests.test_one) test_y0 (sample2.sample21.sampletests.TestA) test_y1 (sample2.sample21.sampletests.TestB) test_y0 (sample2.sample21.sampletests) test_y0 (sample2.sampletests.test_1.TestA) test_y1 (sample2.sampletests.test_1.TestB) test_y0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.testone.TestA) test_y1 (sample2.sampletests.testone.TestB) test_y0 (sample2.sampletests.testone) test_y0 (sample3.sampletests.TestA) test_y1 (sample3.sampletests.TestB) test_y0 (sample3.sampletests) test_y0 (sampletests.test1.TestA) test_y1 (sampletests.test1.TestB) test_y0 (sampletests.test1) test_y0 (sampletests.test_one.TestA) test_y1 (sampletests.test_one.TestB) test_y0 (sampletests.test_one) Ran 36 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False We get run 36 tests. If we specify a level of 2, we get some additional tests: >>> sys.argv = 'test -u -vv -a 2 -t test_y1 -t test_y0'.split() >>> testrunner.run_internal(defaults) Running tests at level 2 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sampletestsf.TestA) test_y0 (sampletestsf.TestA2) test_y1 (sampletestsf.TestB) test_y0 (sampletestsf) test_y0 (sample1.sampletestsf.TestA) test_y1 (sample1.sampletestsf.TestB) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y1 (sample1.sample11.sampletests.TestB) test_y1 (sample1.sample11.sampletests.TestB2) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y1 (sample1.sample13.sampletests.TestB) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y1 (sample1.sampletests.test1.TestB) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y1 (sample1.sampletests.test_one.TestB) test_y0 (sample1.sampletests.test_one) test_y0 (sample2.sample21.sampletests.TestA) test_y1 (sample2.sample21.sampletests.TestB) test_y0 (sample2.sample21.sampletests) test_y0 (sample2.sampletests.test_1.TestA) test_y1 (sample2.sampletests.test_1.TestB) test_y0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.testone.TestA) test_y1 (sample2.sampletests.testone.TestB) test_y0 (sample2.sampletests.testone) test_y0 (sample3.sampletests.TestA) test_y1 (sample3.sampletests.TestB) test_y0 (sample3.sampletests) test_y0 (sampletests.test1.TestA) test_y1 (sampletests.test1.TestB) test_y0 (sampletests.test1) test_y0 (sampletests.test_one.TestA) test_y1 (sampletests.test_one.TestB) test_y0 (sampletests.test_one) Ran 38 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False We can use the --all option to run tests at all levels: >>> sys.argv = 'test -u -vv --all -t test_y1 -t test_y0'.split() >>> testrunner.run_internal(defaults) Running tests at all levels Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sampletestsf.TestA) test_y0 (sampletestsf.TestA2) test_y1 (sampletestsf.TestB) test_y0 (sampletestsf) test_y0 (sample1.sampletestsf.TestA) test_y1 (sample1.sampletestsf.TestB) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sample11.sampletests.TestA3) test_y1 (sample1.sample11.sampletests.TestB) test_y1 (sample1.sample11.sampletests.TestB2) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y1 (sample1.sample13.sampletests.TestB) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y1 (sample1.sampletests.test1.TestB) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y1 (sample1.sampletests.test_one.TestB) test_y0 (sample1.sampletests.test_one) test_y0 (sample2.sample21.sampletests.TestA) test_y1 (sample2.sample21.sampletests.TestB) test_y0 (sample2.sample21.sampletests) test_y0 (sample2.sampletests.test_1.TestA) test_y1 (sample2.sampletests.test_1.TestB) test_y0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.testone.TestA) test_y1 (sample2.sampletests.testone.TestB) test_y0 (sample2.sampletests.testone) test_y0 (sample3.sampletests.TestA) test_y1 (sample3.sampletests.TestB) test_y0 (sample3.sampletests) test_y0 (sampletests.test1.TestA) test_y1 (sampletests.test1.TestB) test_y0 (sampletests.test1) test_y0 (sampletests.test_one.TestA) test_y1 (sampletests.test_one.TestB) test_y0 (sampletests.test_one) Ran 39 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Listing Selected Tests ---------------------- When you're trying to figure out why the test you want is not matched by the pattern you specified, it is convenient to see which tests match your specifications. >>> sys.argv = 'test --all -m sample1 -t test_y0 --list-tests'.split() >>> testrunner.run_internal(defaults) Listing zope.testrunner.layer.UnitTests tests: test_y0 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sample11.sampletests.TestA3) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one) Listing samplelayers.Layer11 tests: test_y0 (sample1.sampletests.test11.TestA) test_y0 (sample1.sampletests.test11) Listing samplelayers.Layer111 tests: test_y0 (sample1.sampletests.test111.TestA) test_y0 (sample1.sampletests.test111) Listing samplelayers.Layer112 tests: test_y0 (sample1.sampletests.test112.TestA) test_y0 (sample1.sampletests.test112) Listing samplelayers.Layer12 tests: test_y0 (sample1.sampletests.test12.TestA) test_y0 (sample1.sampletests.test12) Listing samplelayers.Layer121 tests: test_y0 (sample1.sampletests.test121.TestA) test_y0 (sample1.sampletests.test121) Listing samplelayers.Layer122 tests: test_y0 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122) False Test Progress ============= If the --progress (-p) option is used, progress information is printed and a carriage return (rather than a new-line) is printed between detail lines. Let's look at the effect of --progress (-p) at different levels of verbosity. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 122 -p'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Running: 1/26 (3.8%)##r## ##r## 2/26 (7.7%)##r## ##r## 3/26 (11.5%)##r## ##r## 4/26 (15.4%)##r## ##r## 5/26 (19.2%)##r## ##r## 6/26 (23.1%)##r## ##r## 7/26 (26.9%)##r## ##r## 8/26 (30.8%)##r## ##r## 9/26 (34.6%)##r## ##r## 10/26 (38.5%)##r## ##r## 11/26 (42.3%)##r## ##r## 12/26 (46.2%)##r## ##r## 13/26 (50.0%)##r## ##r## 14/26 (53.8%)##r## ##r## 15/26 (57.7%)##r## ##r## 16/26 (61.5%)##r## ##r## 17/26 (65.4%)##r## ##r## 18/26 (69.2%)##r## ##r## 19/26 (73.1%)##r## ##r## 20/26 (76.9%)##r## ##r## 21/26 (80.8%)##r## ##r## 22/26 (84.6%)##r## ##r## 23/26 (88.5%)##r## ##r## 24/26 (92.3%)##r## ##r## 25/26 (96.2%)##r## ##r## 26/26 (100.0%)##r## ##r## Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False (Note that, in the examples above and below, we show "##r##" followed by new lines where carriage returns would appear in actual output.) Using a single level of verbosity causes test descriptions to be output, but only if they fit in the terminal width. The default width, when the terminal width can't be determined, is 80: >>> sys.argv = 'test --layer 122 -pv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Running: 1/26 (3.8%) test_x1 (sample1.sampletests.test122.TestA)##r## ##r## 2/26 (7.7%) test_y0 (sample1.sampletests.test122.TestA)##r## ##r## 3/26 (11.5%) test_z0 (sample1.sampletests.test122.TestA)##r## ##r## 4/26 (15.4%) test_x0 (sample1.sampletests.test122.TestB)##r## ##r## 5/26 (19.2%) test_y1 (sample1.sampletests.test122.TestB)##r## ##r## 6/26 (23.1%) test_z0 (sample1.sampletests.test122.TestB)##r## ##r## 7/26 (26.9%) test_1 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 8/26 (30.8%) test_2 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 9/26 (34.6%) test_3 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 10/26 (38.5%) test_x0 (sample1.sampletests.test122)##r## ##r## 11/26 (42.3%) test_y0 (sample1.sampletests.test122)##r## ##r## 12/26 (46.2%) test_z1 (sample1.sampletests.test122)##r## ##r## testrunner-ex/sample1/sampletests/../../sampletestsl.txt##r## ##r## 14/26 (53.8%) test_x1 (sampletests.test122.TestA)##r## ##r## 15/26 (57.7%) test_y0 (sampletests.test122.TestA)##r## ##r## 16/26 (61.5%) test_z0 (sampletests.test122.TestA)##r## ##r## 17/26 (65.4%) test_x0 (sampletests.test122.TestB)##r## ##r## 18/26 (69.2%) test_y1 (sampletests.test122.TestB)##r## ##r## 19/26 (73.1%) test_z0 (sampletests.test122.TestB)##r## ##r## 20/26 (76.9%) test_1 (sampletests.test122.TestNotMuch)##r## ##r## 21/26 (80.8%) test_2 (sampletests.test122.TestNotMuch)##r## ##r## 22/26 (84.6%) test_3 (sampletests.test122.TestNotMuch)##r## ##r## 23/26 (88.5%) test_x0 (sampletests.test122)##r## ##r## 24/26 (92.3%) test_y0 (sampletests.test122)##r## ##r## 25/26 (96.2%) test_z1 (sampletests.test122)##r## ##r## testrunner-ex/sampletests/../sampletestsl.txt##r## ##r## Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False The terminal width is determined using the curses module. To see that, we'll provide a fake curses module: >>> class FakeCurses: ... class error(Exception): ... pass ... def setupterm(self): ... pass ... def tigetnum(self, ignored): ... return 60 >>> old_curses = sys.modules.get('curses') >>> sys.modules['curses'] = FakeCurses() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Running: 1/26 (3.8%) test_x1 (sample1.sampletests.test122.TestA)##r## ##r## 2/26 (7.7%) test_y0 (sample1.sampletests.test122.TestA)##r## ##r## 3/26 (11.5%) test_z0 (...le1.sampletests.test122.TestA)##r## ##r## 4/26 (15.4%) test_x0 (...le1.sampletests.test122.TestB)##r## ##r## 5/26 (19.2%) test_y1 (...le1.sampletests.test122.TestB)##r## ##r## 6/26 (23.1%) test_z0 (...le1.sampletests.test122.TestB)##r## ##r## 7/26 (26.9%) test_1 (...ampletests.test122.TestNotMuch)##r## ##r## 8/26 (30.8%) test_2 (...ampletests.test122.TestNotMuch)##r## ##r## 9/26 (34.6%) test_3 (...ampletests.test122.TestNotMuch)##r## ##r## 10/26 (38.5%) test_x0 (sample1.sampletests.test122)##r## ##r## 11/26 (42.3%) test_y0 (sample1.sampletests.test122)##r## ##r## 12/26 (46.2%) test_z1 (sample1.sampletests.test122)##r## ##r## 13/26 (50.0%) ... e1/sampletests/../../sampletestsl.txt##r## ##r## 14/26 (53.8%) test_x1 (sampletests.test122.TestA)##r## ##r## 15/26 (57.7%) test_y0 (sampletests.test122.TestA)##r## ##r## 16/26 (61.5%) test_z0 (sampletests.test122.TestA)##r## ##r## 17/26 (65.4%) test_x0 (sampletests.test122.TestB)##r## ##r## 18/26 (69.2%) test_y1 (sampletests.test122.TestB)##r## ##r## 19/26 (73.1%) test_z0 (sampletests.test122.TestB)##r## ##r## 20/26 (76.9%) test_1 (sampletests.test122.TestNotMuch)##r## ##r## 21/26 (80.8%) test_2 (sampletests.test122.TestNotMuch)##r## ##r## 22/26 (84.6%) test_3 (sampletests.test122.TestNotMuch)##r## ##r## 23/26 (88.5%) test_x0 (sampletests.test122)##r## ##r## 24/26 (92.3%) test_y0 (sampletests.test122)##r## ##r## 25/26 (96.2%) test_z1 (sampletests.test122)##r## ##r## 26/26 (100.0%) ... r-ex/sampletests/../sampletestsl.txt##r## ##r## Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False >>> sys.modules['curses'] = old_curses If a second or third level of verbosity are added, we get additional information. >>> sys.argv = 'test --layer 122 -pvv -t !txt'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: 1/24 (4.2%) test_x1 (sample1.sampletests.test122.TestA)##r## ##r## 2/24 (8.3%) test_y0 (sample1.sampletests.test122.TestA)##r## ##r## 3/24 (12.5%) test_z0 (sample1.sampletests.test122.TestA)##r## ##r## 4/24 (16.7%) test_x0 (sample1.sampletests.test122.TestB)##r## ##r## 5/24 (20.8%) test_y1 (sample1.sampletests.test122.TestB)##r## ##r## 6/24 (25.0%) test_z0 (sample1.sampletests.test122.TestB)##r## ##r## 7/24 (29.2%) test_1 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 8/24 (33.3%) test_2 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 9/24 (37.5%) test_3 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 10/24 (41.7%) test_x0 (sample1.sampletests.test122)##r## ##r## 11/24 (45.8%) test_y0 (sample1.sampletests.test122)##r## ##r## 12/24 (50.0%) test_z1 (sample1.sampletests.test122)##r## ##r## 13/24 (54.2%) test_x1 (sampletests.test122.TestA)##r## ##r## 14/24 (58.3%) test_y0 (sampletests.test122.TestA)##r## ##r## 15/24 (62.5%) test_z0 (sampletests.test122.TestA)##r## ##r## 16/24 (66.7%) test_x0 (sampletests.test122.TestB)##r## ##r## 17/24 (70.8%) test_y1 (sampletests.test122.TestB)##r## ##r## 18/24 (75.0%) test_z0 (sampletests.test122.TestB)##r## ##r## 19/24 (79.2%) test_1 (sampletests.test122.TestNotMuch)##r## ##r## 20/24 (83.3%) test_2 (sampletests.test122.TestNotMuch)##r## ##r## 21/24 (87.5%) test_3 (sampletests.test122.TestNotMuch)##r## ##r## 22/24 (91.7%) test_x0 (sampletests.test122)##r## ##r## 23/24 (95.8%) test_y0 (sampletests.test122)##r## ##r## 24/24 (100.0%) test_z1 (sampletests.test122)##r## ##r## Ran 24 tests with 0 failures, 0 errors and 0 skipped in 0.006 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False Note that, in this example, we used a test-selection pattern starting with '!' to exclude tests containing the string "txt". >>> sys.argv = 'test --layer 122 -pvvv -t!(txt|NotMuch)'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: 1/18 (5.6%) test_x1 (sample1.sampletests.test122.TestA) (0.000 s)##r## ##r## 2/18 (11.1%) test_y0 (sample1.sampletests.test122.TestA) (0.000 s)##r## ##r## 3/18 (16.7%) test_z0 (sample1.sampletests.test122.TestA) (0.000 s)##r## ##r## 4/18 (22.2%) test_x0 (sample1.sampletests.test122.TestB) (0.000 s)##r## ##r## 5/18 (27.8%) test_y1 (sample1.sampletests.test122.TestB) (0.000 s)##r## ##r## 6/18 (33.3%) test_z0 (sample1.sampletests.test122.TestB) (0.000 s)##r## ##r## 7/18 (38.9%) test_x0 (sample1.sampletests.test122) (0.001 s)##r## ##r## 8/18 (44.4%) test_y0 (sample1.sampletests.test122) (0.001 s)##r## ##r## 9/18 (50.0%) test_z1 (sample1.sampletests.test122) (0.001 s)##r## ##r## 10/18 (55.6%) test_x1 (sampletests.test122.TestA) (0.000 s)##r## ##r## 11/18 (61.1%) test_y0 (sampletests.test122.TestA) (0.000 s)##r## ##r## 12/18 (66.7%) test_z0 (sampletests.test122.TestA) (0.000 s)##r## ##r## 13/18 (72.2%) test_x0 (sampletests.test122.TestB) (0.000 s)##r## ##r## 14/18 (77.8%) test_y1 (sampletests.test122.TestB) (0.000 s)##r## ##r## 15/18 (83.3%) test_z0 (sampletests.test122.TestB) (0.000 s)##r## ##r## 16/18 (88.9%) test_x0 (sampletests.test122) (0.001 s)##r## ##r## 17/18 (94.4%) test_y0 (sampletests.test122) (0.001 s)##r## ##r## 18/18 (100.0%) test_z1 (sampletests.test122) (0.001 s)##r## ##r## Ran 18 tests with 0 failures, 0 errors and 0 skipped in 0.006 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False In this example, we also excluded tests with "NotMuch" in their names. Unfortunately, the time data above doesn't buy us much because, in practice, the line is cleared before there is time to see the times. :/ Autodetecting progress ---------------------- The --auto-progress option will determine if stdout is a terminal, and only enable progress output if so. Let's pretend we have a terminal >>> class Terminal(object): ... def __init__(self, stream): ... self._stream = stream ... def __getattr__(self, attr): ... return getattr(self._stream, attr) ... def isatty(self): ... return True >>> real_stdout = sys.stdout >>> sys.stdout = Terminal(sys.stdout) >>> sys.argv = 'test -u -t test_one.TestNotMuch --auto-progress'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: 1/6 (16.7%)##r## ##r## 2/6 (33.3%)##r## ##r## 3/6 (50.0%)##r## ##r## 4/6 (66.7%)##r## ##r## 5/6 (83.3%)##r## ##r## 6/6 (100.0%)##r## ##r## Ran 6 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Let's stop pretending >>> sys.stdout = real_stdout >>> sys.argv = 'test -u -t test_one.TestNotMuch --auto-progress'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 6 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Disabling progress indication ----------------------------- If -p or --progress have been previously provided on the command line (perhaps by a wrapper script) but you do not desire progress indication, you can switch it off with --no-progress: >>> sys.argv = 'test -u -t test_one.TestNotMuch -p --no-progress'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 6 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Debugging ========= The testrunner module supports post-mortem debugging and debugging using `pdb.set_trace`. Let's look first at using `pdb.set_trace`. To demonstrate this, we'll provide input via helper Input objects: >>> class Input: ... def __init__(self, src): ... self.lines = src.split('\n') ... def readline(self): ... line = self.lines.pop(0) ... print(line) ... return line+'\n' If a test or code called by a test calls pdb.set_trace, then the runner will enter pdb at that point: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> real_stdin = sys.stdin >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace1').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... > testrunner-ex/sample3/sampletests_d.py(27)test_set_trace1() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.001 seconds. ... False Note that, prior to Python 2.4, calling pdb.set_trace caused pdb to break in the pdb.set_trace function. It was necessary to use 'next' or 'up' to get to the application code that called pdb.set_trace. In Python 2.4, pdb.set_trace causes pdb to stop right after the call to pdb.set_trace. You can also do post-mortem debugging, using the --post-mortem (-D) option: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem1 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Error in test test_post_mortem1 (sample3.sampletests_d.TestSomething) Traceback (most recent call last): File "testrunner-ex/sample3/sampletests_d.py", line 34, in test_post_mortem1 raise ValueError ValueError ...ValueError > testrunner-ex/sample3/sampletests_d.py(34)test_post_mortem1() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Note that the test runner exits after post-mortem debugging. In the example above, we debugged an error. Failures are actually converted to errors and can be debugged the same way: >>> sys.stdin = Input('p x\np y\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure1 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Error in test test_post_mortem_failure1 (sample3.sampletests_d.TestSomething) Traceback (most recent call last): File ".../unittest.py", line 252, in debug getattr(self, self.__testMethodName)() File "testrunner-ex/sample3/sampletests_d.py", line 42, in test_post_mortem_failure1 assert x == y AssertionError ...AssertionError > testrunner-ex/sample3/sampletests_d.py(42)test_post_mortem_failure1() -> assert x == y (Pdb) p x 1 (Pdb) p y 2 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Layers that can't be torn down ============================== A layer can have a tearDown method that raises NotImplementedError. If this is the case and there are no remaining tests to run, the test runner will just note that the tear down couldn't be done: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test -ssample2 --tests-pattern sampletests_ntd$'.split() >>> testrunner.run_internal(defaults) Running sample2.sampletests_ntd.Layer tests: Set up sample2.sampletests_ntd.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Tearing down left over layers: Tear down sample2.sampletests_ntd.Layer ... not supported False If the tearDown method raises NotImplementedError and there are remaining layers to run, the test runner will restart itself as a new process, resuming tests where it left off: >>> sys.argv = [testrunner_script, '--tests-pattern', 'sampletests_ntd$'] >>> testrunner.run_internal(defaults) Running sample1.sampletests_ntd.Layer tests: Set up sample1.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running sample2.sampletests_ntd.Layer tests: Tear down sample1.sampletests_ntd.Layer ... not supported Running in a subprocess. Set up sample2.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down sample2.sampletests_ntd.Layer ... not supported Running sample3.sampletests_ntd.Layer tests: Running in a subprocess. Set up sample3.sampletests_ntd.Layer in N.NNN seconds. Error in test test_error1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error1 raise TypeError("Can we see errors") TypeError: Can we see errors Error in test test_error2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error2 raise TypeError("I hope so") TypeError: I hope so Failure in test test_fail1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail1 self.assertEqual(1, 2) AssertionError: 1 != 2 Failure in test test_fail2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail2 self.assertEqual(1, 3) AssertionError: 1 != 3 Ran 6 tests with 2 failures, 2 errors and 0 skipped in N.NNN seconds. Tear down sample3.sampletests_ntd.Layer ... not supported Total: 8 tests, 2 failures, 2 errors and 0 skipped in N.NNN seconds. True in the example above, some of the tests run as a subprocess had errors and failures. They were displayed as usual and the failure and error statistice were updated as usual. Note that debugging doesn't work when running tests in a subprocess: >>> sys.argv = [testrunner_script, '--tests-pattern', 'sampletests_ntd$', ... '-D', ] >>> testrunner.run_internal(defaults) Running sample1.sampletests_ntd.Layer tests: Set up sample1.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running sample2.sampletests_ntd.Layer tests: Tear down sample1.sampletests_ntd.Layer ... not supported Running in a subprocess. Set up sample2.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down sample2.sampletests_ntd.Layer ... not supported Running sample3.sampletests_ntd.Layer tests: Running in a subprocess. Set up sample3.sampletests_ntd.Layer in N.NNN seconds. Error in test test_error1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error1 raise TypeError("Can we see errors") TypeError: Can we see errors ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Error in test test_error2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error2 raise TypeError("I hope so") TypeError: I hope so ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Error in test test_fail1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail1 self.assertEqual(1, 2) AssertionError: 1 != 2 ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Error in test test_fail2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail2 self.assertEqual(1, 3) AssertionError: 1 != 3 ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Ran 6 tests with 0 failures, 4 errors and 0 skipped in N.NNN seconds. Tear down sample3.sampletests_ntd.Layer ... not supported Total: 8 tests, 0 failures, 4 errors and 0 skipped in N.NNN seconds. True Similarly, pdb.set_trace doesn't work when running tests in a layer that is run as a subprocess: >>> sys.argv = [testrunner_script, '--tests-pattern', 'sampletests_ntds'] >>> testrunner.run_internal(defaults) Running sample1.sampletests_ntds.Layer tests: Set up sample1.sampletests_ntds.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Running sample2.sampletests_ntds.Layer tests: Tear down sample1.sampletests_ntds.Layer ... not supported Running in a subprocess. Set up sample2.sampletests_ntds.Layer in 0.000 seconds. --Return-- > testrunner-ex/sample2/sampletests_ntds.py(37)test_something()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(40)test_something2()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(43)test_something3()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(46)test_something4()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(52)f()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > doctest.py(351)set_trace()->None -> Pdb().set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > doctest.py(351)set_trace()->None -> Pdb().set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** Ran 7 tests with 0 failures, 0 errors and 0 skipped in 0.008 seconds. Tear down sample2.sampletests_ntds.Layer ... not supported Total: 8 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False If you want to use pdb from a test in a layer that is run as a subprocess, then rerun the test runner selecting *just* that layer so that it's not run as a subprocess. If a test is run in a subprocess and it generates output on stderr (as stderrtest does), the output is ignored (but it doesn't cause a SubprocessError like it once did). >>> from six import StringIO >>> real_stderr = sys.stderr >>> sys.stderr = StringIO() >>> sys.argv = [testrunner_script, '-s', 'sample2', '--tests-pattern', ... '(sampletests_ntd$|stderrtest)'] >>> testrunner.run_internal(defaults) Running sample2.sampletests_ntd.Layer tests: Set up sample2.sampletests_ntd.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Running sample2.stderrtest.Layer tests: Tear down sample2.sampletests_ntd.Layer ... not supported Running in a subprocess. Set up sample2.stderrtest.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.002 seconds. Tear down sample2.stderrtest.Layer in 0.000 seconds. Total: 2 tests, 0 failures, 0 errors and 0 skipped in 0.197 seconds. False >>> print((sys.stderr.getvalue())) A message on stderr. Please ignore (expected in test output). >>> sys.stderr = real_stderr Egg Support =========== The `ftest` Setup Command ------------------------- The `ftest` class is a proper `distutils` command and `zope.testrunner` exposes it as such via an entry point. >>> import pkg_resources >>> ws = pkg_resources.WorkingSet() >>> eps = dict([(ep.name, ep) ... for ep in ws.iter_entry_points('distutils.commands')]) >>> 'ftest' in eps True >>> eps['ftest'] EntryPoint.parse('ftest = zope.testrunner.eggsupport:ftest') Let's now run this command: >>> import zope.testrunner >>> org_run = zope.testrunner.run >>> def run(args): ... print(' '.join(args)) >>> zope.testrunner.run = run >>> import os, tempfile, shutil >>> tmpdir = tempfile.mkdtemp() >>> srcdir = os.path.join(tmpdir, 'src') >>> os.mkdir(srcdir) >>> import setuptools.dist >>> dist = setuptools.dist.Distribution( ... {'package_dir': {'': srcdir}, ... 'script_name': __file__}) >>> from zope.testrunner.eggsupport import ftest >>> ftest(dist).run() IGNORE_ME --test-path .../src Cleanup: >>> zope.testrunner.run = org_run >>> shutil.rmtree(tmpdir) Skipping Tests with Layers -------------------------- The ``SkipLayers`` scanning test loader can replace the standard test loader, so that any tests that require layers are skipped. This is necessary, since the standard setuptools testing facility does not handle layers. It can be used as follows:: setup( ... setup_requires=['eggtestinfo' # captures testing metadata in EGG-INFO ], tests_require=['zope.testrunner', ], ... test_loader='zope.testrunner.eggsupport:SkipLayers', ... ) Let's now crete some test suites to make sure that all tests with layers are properly skipped. >>> import doctest >>> import unittest >>> all = unittest.TestSuite() >>> class T1(unittest.TestCase): ... def test_t1(self): ... pass >>> class T2(unittest.TestCase): ... layer = 'layer' ... def test_t2(self): ... pass >>> T3 = doctest.DocTestSuite('zope.testrunner.find') >>> T4 = doctest.DocTestSuite('zope.testrunner.options') >>> T4.layer = 'layer' >>> T5 = doctest.DocFileSuite('testrunner.txt', package='zope.testrunner.tests') >>> T6 = doctest.DocFileSuite('testrunner-gc.txt', package='zope.testrunner.tests') >>> T6.layer = 'layer' >>> all = unittest.TestSuite(( ... unittest.makeSuite(T1), unittest.makeSuite(T2), T3, T4, T5, T6, ... )) Let's return those tests from the scan: >>> from setuptools.command.test import ScanningLoader >>> orig_loadTestsFromModule = ScanningLoader.loadTestsFromModule >>> ScanningLoader.loadTestsFromModule = lambda *args: all Now we can retrieve the modules from the layer skipping loader: >>> from zope.testrunner.eggsupport import SkipLayers >>> filtered = SkipLayers().loadTestsFromModule('zope.testrunner') >>> len(filtered._tests) 3 >>> from pprint import pprint >>> pprint(filtered._tests) [<...T1 testMethod=test_t1>, StartUpFailure (zope.testrunner.find), .../zope/testrunner/tests/testrunner.txt] Cleanup: >>> ScanningLoader.loadTestsFromModule = orig_loadTestsFromModule When the distribution specified a ``test_suite``, another method is used to load the tests. >>> orig_loadTestsFromNames = ScanningLoader.loadTestsFromNames >>> ScanningLoader.loadTestsFromNames = lambda *args: all Now we can retrieve the modules from the layer skipping loader: >>> from zope.testrunner.eggsupport import SkipLayers >>> filtered = SkipLayers().loadTestsFromNames( ... 'zope.testrunner.tests.test_suite', 'zope.testrunner') >>> len(filtered._tests) 3 >>> from pprint import pprint >>> pprint(filtered._tests) [<...T1 testMethod=test_t1>, StartUpFailure (zope.testrunner.find), .../zope/testrunner/tests/testrunner.txt] Cleanup: >>> ScanningLoader.loadTestsFromNames = orig_loadTestsFromNames Code Coverage ============= If the --coverage option is used, test coverage reports will be generated. The directory name given as the parameter will be used to hold the reports. >>> import os.path, sys, tempfile >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> tempdir = tempfile.mkdtemp(prefix='zope.testrunner-') >>> coverage_dir = os.path.join(tempdir, 'coverage_dir') >>> sys.argv = ['test', '--coverage', coverage_dir] >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in 0.000 seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in 0.687 seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in 0.000 seconds. Set up samplelayers.Layer112 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.140 seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. lines cov% module (path) ... 48 100% sampletests.test1 (testrunner-ex/sampletests/test1.py) 74 100% sampletests.test11 (testrunner-ex/sampletests/test11.py) 74 100% sampletests.test111 (testrunner-ex/sampletests/test111.py) 76 100% sampletests.test112 (testrunner-ex/sampletests/test112.py) 74 100% sampletests.test12 (testrunner-ex/sampletests/test12.py) 74 100% sampletests.test121 (testrunner-ex/sampletests/test121.py) 74 100% sampletests.test122 (testrunner-ex/sampletests/test122.py) 48 100% sampletests.test_one (testrunner-ex/sampletests/test_one.py) 112 95% sampletestsf (testrunner-ex/sampletestsf.py) Total: 321 tests, 0 failures, 0 errors and 0 skipped in 0.630 seconds. False The directory specified with the --coverage option will have been created and will hold the coverage reports. >>> os.path.exists(coverage_dir) True >>> os.listdir(coverage_dir) [...] (We should clean up after ourselves.) >>> import shutil >>> shutil.rmtree(tempdir) Ignoring Tests -------------- The ``trace`` module supports ignoring directories and modules based the test selection. Only directories selected for testing should report coverage. The test runner provides a custom implementation of the relevant API. The ``TestIgnore`` class, the class managing the ignoring, is initialized by passing the command line options. It uses the options to determine the directories that should be covered. >>> class FauxOptions(object): ... package = None ... test_path = [('/myproject/src/blah/foo', ''), ... ('/myproject/src/blah/bar', '')] >>> from zope.testrunner import coverage >>> from zope.testrunner.find import test_dirs >>> ignore = coverage.TestIgnore(test_dirs(FauxOptions(), {})) >>> ignore._test_dirs ['/myproject/src/blah/foo/', '/myproject/src/blah/bar/'] We can now ask whether a particular module should be ignored: >>> ignore.names('/myproject/src/blah/foo/baz.py', 'baz') False >>> ignore.names('/myproject/src/blah/bar/mine.py', 'mine') False >>> ignore.names('/myproject/src/blah/foo/__init__.py', 'foo') False >>> ignore.names('/myproject/src/blah/hello.py', 'hello') True When running the test runner, modules are sometimes created from text strings. Those should *always* be ignored: >>> ignore.names('/myproject/src/blah/hello.txt', '') True To make this check fast, the class implements a cache. In an early implementation, the result was cached by the module name, which was a problem, since a lot of modules carry the same name (not the Python dotted name here!). So just because a module has the same name in an ignored and tested directory, does not mean it is always ignored: >>> ignore.names('/myproject/src/blah/module.py', 'module') True >>> ignore.names('/myproject/src/blah/foo/module.py', 'module') False Profiling ========= The testrunner supports hotshot and cProfile profilers. Hotshot profiler support does not work with python2.6 >>> import os.path, sys, tempfile >>> profiler = '--profile=hotshot' >>> if sys.hexversion >= 0x02060000: ... profiler = '--profile=cProfile' The testrunner includes the ability to profile the test execution with hotshot via the --profile option, if it a python <= 2.6 >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> sys.path.append(directory_with_tests) >>> tempdir = tempfile.mkdtemp(prefix='zope.testrunner-') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... '--profile-directory', tempdir, ... ] >>> sys.argv = [testrunner_script, profiler] When the tests are run, we get profiling output. >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: ... Running samplelayers.Layer1 tests: ... Running samplelayers.Layer11 tests: ... ncalls tottime percall cumtime percall filename:lineno(function) ... Total: ... tests, 0 failures, 0 errors and 0 skipped in ... seconds. False Profiling also works across layers. >>> sys.argv = [testrunner_script, '-ssample2', profiler, ... '--tests-pattern', 'sampletests_ntd'] >>> testrunner.run_internal(defaults) Running... Tear down ... not supported... ncalls tottime percall cumtime percall filename:lineno(function)... The testrunner creates temnporary files containing hotshot profiler data: >>> os.listdir(tempdir) ['tests_profile.cZj2jt.prof', 'tests_profile.yHD-so.prof'] It deletes these when rerun. We'll delete these ourselves: >>> import shutil >>> shutil.rmtree(tempdir) Running Without Source Code =========================== The ``--usecompiled`` option allows running tests in a tree without .py source code, provided compiled .pyc or .pyo files exist (without ``--usecompiled``, .py files are necessary). We have a very simple directory tree, under ``usecompiled/``, to test this. Because we're going to delete its .py files, we want to work in a copy of that: >>> import os.path, shutil, sys, tempfile >>> directory_with_tests = tempfile.mkdtemp() >>> NEWNAME = "unlikely_package_name" >>> src = os.path.join(this_directory, 'testrunner-ex', 'usecompiled') >>> os.path.isdir(src) True >>> dst = os.path.join(directory_with_tests, NEWNAME) >>> os.path.isdir(dst) False Have to use our own copying code, to avoid copying read-only SVN files that can't be deleted later. >>> n = len(src) + 1 >>> for root, dirs, files in os.walk(src): ... dirs[:] = [d for d in dirs if d == "package"] # prune cruft ... os.mkdir(os.path.join(dst, root[n:])) ... for f in files: ... _ = shutil.copy(os.path.join(root, f), ... os.path.join(dst, root[n:], f)) Now run the tests in the copy: >>> from zope import testrunner >>> mydefaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^compiletest$', ... '--package', NEWNAME, ... '-vv', ... ] >>> sys.argv = ['test'] >>> testrunner.run_internal(mydefaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test1 (unlikely_package_name.compiletest.Test) test2 (unlikely_package_name.compiletest.Test) test1 (unlikely_package_name.package.compiletest.Test) test2 (unlikely_package_name.package.compiletest.Test) Ran 4 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False If we delete the source files, it's normally a disaster: the test runner doesn't believe any test files, or even packages, exist. Note that we pass ``--keepbytecode`` this time, because otherwise the test runner would delete the compiled Python files too: >>> for root, dirs, files in os.walk(dst): ... for f in files: ... if f.endswith(".py"): ... os.remove(os.path.join(root, f)) >>> testrunner.run_internal(mydefaults, ["test", "--keepbytecode"]) Running tests at level 1 Total: 0 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Finally, passing ``--usecompiled`` asks the test runner to treat .pyc and .pyo files as adequate replacements for .py files. Note that the output is the same as when running with .py source above. The absence of "removing stale bytecode ..." messages shows that ``--usecompiled`` also implies ``--keepbytecode``: >>> if sys.version_info >= (3, 2): ... # PEP-3147: pyc files in __pycache__ directories cannot be ... # imported; legacy source-less imports need to use the legacy ... # layout ... for root, dirs, files in os.walk(dst): ... for f in files: ... if f.endswith((".pyc", ".pyo")): ... # "root/f" is "dirname/__pycache__/name.magic.ext" ... dirname = os.path.dirname(os.path.abspath(root)) ... namewmagic, ext = os.path.splitext(os.path.basename(f)) ... newname = os.path.splitext(namewmagic)[0] + ext ... os.rename(os.path.join(root, f), ... os.path.join(dirname, newname)) >>> testrunner.run_internal(mydefaults, ["test", "--usecompiled"]) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test1 (unlikely_package_name.compiletest.Test) test2 (unlikely_package_name.compiletest.Test) test1 (unlikely_package_name.package.compiletest.Test) test2 (unlikely_package_name.package.compiletest.Test) Ran 4 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Remove the temporary directory: >>> shutil.rmtree(directory_with_tests) Repeating Tests =============== The --repeat option can be used to repeat tests some number of times. Repeating tests is useful to help make sure that tests clean up after themselves. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 112 --layer UnitTests --repeat 3'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Iteration 1 Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Iteration 2 Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Iteration 3 Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer112 in 0.000 seconds. Iteration 1 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.010 seconds. Iteration 2 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.010 seconds. Iteration 3 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.010 seconds. Tearing down left over layers: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 182 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False The tests are repeated by layer. Layers are set up and torn down only once. Garbage Collection Control ========================== When having problems that seem to be caused my memory-management errors, it can be helpful to adjust Python's cyclic garbage collector or to get garbage colection statistics. The --gc option can be used for this purpose. If you think you are getting a test failure due to a garbage collection problem, you can try disabling garbage collection by using the --gc option with a value of zero. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = ['--path', directory_with_tests] >>> from zope import testrunner >>> sys.argv = 'test --tests-pattern ^gc0$ --gc 0 -vv'.split() >>> _ = testrunner.run_internal(defaults) Cyclic garbage collection is disabled. Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: make_sure_gc_is_disabled (gc0) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Alternatively, if you think you are having a garbage collection related problem, you can cause garbage collection to happen more often by providing a low threshold: >>> sys.argv = 'test --tests-pattern ^gc1$ --gc 1 -vv'.split() >>> _ = testrunner.run_internal(defaults) Cyclic garbage collection threshold set to: (1,) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: make_sure_gc_threshold_is_one (gc1) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. You can specify up to 3 --gc options to set each of the 3 gc threshold values: >>> sys.argv = ('test --tests-pattern ^gcset$ --gc 701 --gc 11 --gc 9 -vv' ... .split()) >>> _ = testrunner.run_internal(defaults) Cyclic garbage collection threshold set to: (701, 11, 9) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: make_sure_gc_threshold_is_701_11_9 (gcset) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Specifying more than 3 --gc options is not allowed: >>> from six import StringIO >>> out = StringIO() >>> stdout = sys.stdout >>> sys.stdout = out >>> sys.argv = ('test --tests-pattern ^gcset$ --gc 701 --gc 42 --gc 11 --gc 9 -vv' ... .split()) >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 >>> sys.stdout = stdout >>> print(out.getvalue()) Too many --gc options Garbage Collection Statistics ----------------------------- You can enable gc debugging statistics using the --gc-options (-G) option. You should provide names of one or more of the flags described in the library documentation for the gc module. The output statistics are written to standard error. >>> from six import StringIO >>> err = StringIO() >>> stderr = sys.stderr >>> sys.stderr = err >>> sys.argv = ('test --tests-pattern ^gcstats$ -G DEBUG_STATS' ... ' -G DEBUG_COLLECTABLE -vv' ... .split()) >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: generate_some_gc_statistics (gcstats) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. >>> sys.stderr = stderr >>> print(err.getvalue()) # doctest: +ELLIPSIS gc: collecting generation ... Debugging Memory Leaks ====================== The --report-refcounts (-r) option can be used with the --repeat (-N) option to detect and diagnose memory leaks. To use this option, you must configure Python with the --with-pydebug option. (On Unix, pass this option to configure and then build Python.) >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> sys.argv = 'test --layer Layer11$ --layer Layer12$ -N4 -r'.split() >>> _ = testrunner.run_internal(defaults) Running samplelayers.Layer11 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Iteration 1 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.013 seconds. Iteration 2 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100401 change=0 Iteration 3 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100401 change=0 Iteration 4 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.013 seconds. sys refcount=100401 change=0 Running samplelayers.Layer12 tests: Tear down samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Iteration 1 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.013 seconds. Iteration 2 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100411 change=0 Iteration 3 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100411 change=0 Iteration 4 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100411 change=0 Tearing down left over layers: Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. Total: 68 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. Each layer is repeated the requested number of times. For each iteration after the first, the system refcount and change in system refcount is shown. The system refcount is the total of all refcount in the system. When a refcount on any object is changed, the system refcount is changed by the same amount. Tests that don't leak show zero changes in systen refcount. Let's look at an example test that leaks: >>> sys.argv = 'test --tests-pattern leak -N4 -r'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests:... Iteration 1 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Iteration 2 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sys refcount=92506 change=12 Iteration 3 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sys refcount=92513 change=12 Iteration 4 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sys refcount=92520 change=12 Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Here we see that the system refcount is increating. If we specify a verbosity greater than one, we can get details broken out by object type (or class): >>> sys.argv = 'test --tests-pattern leak -N5 -r -v'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests:... Iteration 1 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Iteration 2 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95832 sys refcount=105668 change=16 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 int 2 2 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 3 ------------------------------------------------------- ----- ---- total 8 16 Iteration 3 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95844 sys refcount=105680 change=12 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 int -1 0 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 1 ------------------------------------------------------- ----- ---- total 5 12 Iteration 4 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95856 sys refcount=105692 change=12 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 1 ------------------------------------------------------- ----- ---- total 6 12 Iteration 5 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95868 sys refcount=105704 change=12 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 1 ------------------------------------------------------- ----- ---- total 6 12 Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. It is instructive to analyze the results in some detail. The test being run was designed to intentionally leak: class ClassicLeakable: def __init__(self): self.x = 'x' class Leakable(object): def __init__(self): self.x = 'x' leaked = [] class TestSomething(unittest.TestCase): def testleak(self): leaked.append((ClassicLeakable(), Leakable(), time.time())) Let's go through this by type. float, leak.ClassicLeakable, leak.Leakable, and tuple We leak one of these every time. This is to be expected because we are adding one of these to the list every time. str We don't leak any instances, but we leak 4 references. These are due to the instance attributes avd values. dict We leak 2 of these, one for each ClassicLeakable and Leakable instance. classobj We increase the number of classobj instance references by one each time because each ClassicLeakable instance has a reference to its class. This instances increases the references in it's class, which increases the total number of references to classic classes (clasobj instances). type For most interations, we increase the number of type references by one for the same reason we increase the number of clasobj references by one. The increase of the number of type references by 3 in the second iteration is puzzling, but illustrates that this sort of data is often puzzling. int The change in the number of int instances and references in this example is a side effect of the statistics being gathered. Lots of integers are created to keep the memory statistics used here. The summary statistics include the sum of the detail refcounts. (Note that this sum is less than the system refcount. This is because the detailed analysis doesn't inspect every object. Not all objects in the system are returned by sys.getobjects.) Knitting in extra package directories ===================================== Python packages have __path__ variables that can be manipulated to add extra directories cntaining software used in the packages. The testrunner needs to be given extra information about this sort of situation. Let's look at an example. The testrunner-ex-knit-lib directory is a directory that we want to add to the Python path, but that we don't want to search for tests. It has a sample4 package and a products subpackage. The products subpackage adds the testrunner-ex-knit-products to it's __path__. We want to run tests from the testrunner-ex-knit-products directory. When we import these tests, we need to import them from the sample4.products package. We can't use the --path option to name testrunner-ex-knit-products. It isn't enough to add the containing directory to the test path because then we wouldn't be able to determine the package name properly. We might be able to use the --package option to run the tests from the sample4/products package, but we want to run tests in testrunner-ex that aren't in this package. We can use the --package-path option in this case. The --package-path option is like the --test-path option in that it defines a path to be searched for tests without affecting the python path. It differs in that it supplied a package name that is added a profex when importing any modules found. The --package-path option takes *two* arguments, a package name and file path. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> sys.path.append(os.path.join(this_directory, 'testrunner-ex-pp-lib')) >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... '--package-path', ... os.path.join(this_directory, 'testrunner-ex-pp-products'), ... 'sample4.products', ... ] >>> from zope import testrunner >>> sys.argv = 'test --layer Layer111 -vv'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test111.TestA) test_y0 (sample1.sampletests.test111.TestA) ... test_y0 (sampletests.test111) test_z1 (sampletests.test111) testrunner-ex/sampletests/../sampletestsl.txt test_extra_test_in_products (sample4.products.sampletests.Test) test_another_test_in_products (sample4.products.more.sampletests.Test) Ran 28 tests with 0 failures, 0 errors and 0 skipped in 0.008 seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. In the example, the last test, test_extra_test_in_products, came from the products directory. As usual, we can select the knit-in packages or individual packages within knit-in packages: >>> sys.argv = 'test --package sample4.products -vv'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Running: test_extra_test_in_products (sample4.products.sampletests.Test) test_another_test_in_products (sample4.products.more.sampletests.Test) Ran 2 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. >>> sys.argv = 'test --package sample4.products.more -vv'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Running: test_another_test_in_products (sample4.products.more.sampletests.Test) Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. testrunner Edge Cases ===================== This document has some edge-case examples to test various aspects of the test runner. Separating Python path and test directories ------------------------------------------- The --path option defines a directory to be searched for tests *and* a directory to be added to Python's search path. The --test-path option can be used when you want to set a test search path without also affecting the Python path: >>> import os, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--test-path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) ... # doctest: +ELLIPSIS Test-module import failures: Module: sampletestsf Traceback (most recent call last): ImportError: No module named sampletestsf ... >>> sys.path.append(directory_with_tests) >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layer1 in 0.000 seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. ... Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Bug #251759: The test runner's protection against descending into non-package directories was ineffective, e.g. picking up tests from eggs that were stored close by: >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex-251759') >>> defaults = [ ... '--test-path', directory_with_tests, ... ] >>> testrunner.run_internal(defaults) Total: 0 tests, 0 failures, 0 errors and 0 skipped in 0.000 seconds. False Debugging Edge Cases -------------------- >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--test-path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> class Input: ... def __init__(self, src): ... self.lines = src.split('\n') ... def readline(self): ... line = self.lines.pop(0) ... print(line) ... return line+'\n' >>> real_stdin = sys.stdin Using pdb.set_trace in a function called by an ordinary test: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace2').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests:... > testrunner-ex/sample3/sampletests_d.py(47)f() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.001 seconds. ... False Using pdb.set_trace in a function called by a doctest in a doc string: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace4').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > testrunner-ex/sample3/sampletests_d.py(NNN)f() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Using pdb in a docstring-based doctest >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace3').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > (3)?() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Using pdb.set_trace in a doc file: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace5').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > (3)?() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Using pdb.set_trace in a function called by a doctest in a doc file: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace6').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > testrunner-ex/sample3/sampletests_d.py(NNN)f() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging function called from ordinary test: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem2 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Error in test test_post_mortem2 (sample3.sampletests_d.TestSomething) Traceback (most recent call last): File "testrunner-ex/sample3/sampletests_d.py", line 37, in test_post_mortem2 g() File "testrunner-ex/sample3/sampletests_d.py", line 46, in g raise ValueError ValueError ...ValueError > testrunner-ex/sample3/sampletests_d.py(46)g() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging docstring-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem3 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error in test post_mortem3 (sample3.sampletests_d) Traceback (most recent call last): ...UnexpectedException: testrunner-ex/sample3/sampletests_d.py:NNN (2 examples)> ...ValueError > (1)?() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging function called from docstring-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem4 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error in test post_mortem4 (sample3.sampletests_d) Traceback (most recent call last): ...UnexpectedException: testrunner-ex/sample3/sampletests_d.py:NNN (1 example)> ...ValueError > testrunner-ex/sample3/sampletests_d.py(NNN)g() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging file-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem5 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error testrunner-ex/sample3/post_mortem5.txt Traceback (most recent call last): ...UnexpectedException: testrunner-ex/sample3/post_mortem5.txt:0 (2 examples)> ...ValueError > (1)?() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging function called from file-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem6 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error testrunner-ex/sample3/post_mortem6.txt Traceback (most recent call last): File ".../zope/testing/doctest/__init__.py", Line NNN, in debug runner.run(self._dt_test, clear_globs=False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run r = DocTestRunner.run(self, test, compileflags, out, False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run return self.__run(test, compileflags, out) File ".../zope/testing/doctest/__init__.py", Line NNN, in __run exc_info) File ".../zope/testing/doctest/__init__.py", Line NNN, in report_unexpected_exception raise UnexpectedException(test, example, exc_info) ...UnexpectedException: testrunner-ex/sample3/post_mortem6.txt:0 (2 examples)> ...ValueError > testrunner-ex/sample3/sampletests_d.py(NNN)g() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging of a docstring doctest failure: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure2 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Error in test post_mortem_failure2 (sample3.sampletests_d) File "testrunner-ex/sample3/sampletests_d.py", line 81, in sample3.sampletests_d.post_mortem_failure2 x Want: 2 Got: 1 > testrunner-ex/sample3/sampletests_d.py(81)_() ...ValueError: Expected and actual output are different > (1)...() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging of a docfile doctest failure: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure.txt -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Error in test /home/jim/z3/zope.testrunner/src/zope/testing/testrunner-ex/sample3/post_mortem_failure.txt File "testrunner-ex/sample3/post_mortem_failure.txt", line 2, in post_mortem_failure.txt x Want: 2 Got: 1 > testrunner-ex/sample3/post_mortem_failure.txt(2)_() ...ValueError: Expected and actual output are different > (1)...() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging with triple verbosity >>> sys.argv = 'test --layer samplelayers.Layer1$ -vvv -D'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in 0.000 seconds. Running: test_x1 (sampletestsf.TestA1) (0.000 s) test_y0 (sampletestsf.TestA1) (0.000 s) test_z0 (sampletestsf.TestA1) (0.000 s) test_x0 (sampletestsf.TestB1) (0.000 s) test_y1 (sampletestsf.TestB1) (0.000 s) test_z0 (sampletestsf.TestB1) (0.000 s) test_1 (sampletestsf.TestNotMuch1) (0.000 s) test_2 (sampletestsf.TestNotMuch1) (0.000 s) test_3 (sampletestsf.TestNotMuch1) (0.000 s) Ran 9 tests with 0 failures, 0 errors and 0 skipped in 0.001 seconds. Tearing down left over layers: Tear down samplelayers.Layer1 in 0.000 seconds. False Test Suites with None for suites or tests ----------------------------------------- >>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_none_suite$', ... ] >>> testrunner.run_internal(defaults) Test-module import failures: Module: sample1.sampletests_none_suite Traceback (most recent call last): TypeError: Invalid test_suite, None, in sample1.sampletests_none_suite Test-modules with import problems: sample1.sampletests_none_suite Total: 0 tests, 0 failures, 1 errors and 0 skipped in N.NNN seconds. True >>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_none_test$', ... ] >>> testrunner.run_internal(defaults) Test-module import failures: Module: sample1.sampletests_none_test Traceback (most recent call last): TypeError: ... Test-modules with import problems: sample1.sampletests_none_test Total: 0 tests, 0 failures, 1 errors and 0 skipped in N.NNN seconds. True You must use --repeat with --report-refcounts --------------------------------------------- It is an error to specify --report-refcounts (-r) without specifying a repeat count greater than 1 >>> sys.argv = 'test -r'.split() >>> testrunner.run_internal(defaults) You must use the --repeat (-N) option to specify a repeat count greater than 1 when using the --report_refcounts (-r) option. True >>> sys.argv = 'test -r -N1'.split() >>> testrunner.run_internal(defaults) You must use the --repeat (-N) option to specify a repeat count greater than 1 when using the --report_refcounts (-r) option. True Selection --------- Several tests can be excluded using the '!' notation: >>> sys.argv = 'test -u -vv -ssample1.sample13 -t!test_x -t!test_y'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_z0 (sample1.sample13.sampletests.TestA) test_z0 (sample1.sample13.sampletests.TestB) test_1 (sample1.sample13.sampletests.TestNotMuch) test_2 (sample1.sample13.sampletests.TestNotMuch) test_3 (sample1.sample13.sampletests.TestNotMuch) test_z1 (sample1.sample13.sampletests) testrunner-ex/sample1/sample13/../../sampletests.txt Ran 7 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Errors and Failures =================== Let's look at tests that have errors and failures, first we need to make a temporary copy of the entire testing directory (except .svn files which may be read only): >>> import os.path, sys, tempfile, shutil >>> tmpdir = tempfile.mkdtemp() >>> directory_with_tests = os.path.join(tmpdir, 'testrunner-ex') >>> source = os.path.join(this_directory, 'testrunner-ex') >>> n = len(source) + 1 >>> for root, dirs, files in os.walk(source): ... dirs[:] = [d for d in dirs if d != ".svn"] # prune cruft ... os.mkdir(os.path.join(directory_with_tests, root[n:])) ... for f in files: ... _ = shutil.copy(os.path.join(root, f), ... os.path.join(directory_with_tests, root[n:], f)) >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --tests-pattern ^sampletests(f|_e|_f)?$ '.split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Failure in test eek (sample2.sampletests_e) Failed doctest test for sample2.sampletests_e.eek File "testrunner-ex/sample2/sampletests_e.py", line 28, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_e.py", line 30, in sample2.sampletests_e.eek Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined Error in test test3 (sample2.sampletests_e.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_e.py", line 43, in test3 f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined Failure in test testrunner-ex/sample2/e.txt Failed doctest test for e.txt File "testrunner-ex/sample2/e.txt", line 0 ---------------------------------------------------------------------- File "testrunner-ex/sample2/e.txt", line 4, in e.txt Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "", line 2, in f return x NameError: global name 'x' is not defined Failure in test test (sample2.sampletests_f.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_f.py", line 21, in test self.assertEqual(1,0) File "/usr/local/python/2.3/lib/python2.3/unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: 1 != 0 Ran 164 tests with 3 failures, 1 errors and 0 skipped in N.NNN seconds. ... Total: 329 tests, 3 failures, 1 errors and 0 skipped in N.NNN seconds. True We see that we get an error report and a traceback for the failing test. In addition, the test runner returned True, indicating that there was an error. If we ask for verbosity, the dotted output will be interrupted, and there'll be a summary of the errors at the end of the test: >>> sys.argv = 'test --tests-pattern ^sampletests(f|_e|_f)?$ -uv'.split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: ... Running: ................................................................................................. Failure in test eek (sample2.sampletests_e) Failed doctest test for sample2.sampletests_e.eek File "testrunner-ex/sample2/sampletests_e.py", line 28, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_e.py", line 30, in sample2.sampletests_e.eek Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined ... Error in test test3 (sample2.sampletests_e.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_e.py", line 43, in test3 f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined ... Failure in test testrunner-ex/sample2/e.txt Failed doctest test for e.txt File "testrunner-ex/sample2/e.txt", line 0 ---------------------------------------------------------------------- File "testrunner-ex/sample2/e.txt", line 4, in e.txt Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "", line 2, in f return x NameError: global name 'x' is not defined . Failure in test test (sample2.sampletests_f.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_f.py", line 21, in test self.assertEqual(1,0) File ".../unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: 1 != 0 ................................................................................................ Ran 164 tests with 3 failures, 1 errors and 0 skipped in 0.040 seconds. ... Tests with errors: test3 (sample2.sampletests_e.Test) Tests with failures: eek (sample2.sampletests_e) testrunner-ex/sample2/e.txt test (sample2.sampletests_f.Test) True Similarly for progress output, the progress ticker will be interrupted: >>> sys.argv = ('test --tests-pattern ^sampletests(f|_e|_f)?$ -u -ssample2' ... ' -p').split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: 1/47 (2.1%) Failure in test eek (sample2.sampletests_e) Failed doctest test for sample2.sampletests_e.eek File "testrunner-ex/sample2/sampletests_e.py", line 28, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_e.py", line 30, in sample2.sampletests_e.eek Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined 2/47 (4.3%)\r \r 3/47 (6.4%)\r \r 4/47 (8.5%) Error in test test3 (sample2.sampletests_e.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_e.py", line 43, in test3 f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined 5/47 (10.6%)\r \r 6/47 (12.8%)\r \r 7/47 (14.9%) Failure in test testrunner-ex/sample2/e.txt Failed doctest test for e.txt File "testrunner-ex/sample2/e.txt", line 0 ---------------------------------------------------------------------- File "testrunner-ex/sample2/e.txt", line 4, in e.txt Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "", line 2, in f return x NameError: global name 'x' is not defined 8/47 (17.0%) Failure in test test (sample2.sampletests_f.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_f.py", line 21, in test self.assertEqual(1,0) File ".../unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: 1 != 0 9/47 (19.1%)\r \r 10/47 (21.3%)\r \r 11/47 (23.4%)\r \r 12/47 (25.5%)\r \r 13/47 (27.7%)\r \r 14/47 (29.8%)\r \r 15/47 (31.9%)\r \r 16/47 (34.0%)\r \r 17/47 (36.2%)\r \r 18/47 (38.3%)\r \r 19/47 (40.4%)\r \r 20/47 (42.6%)\r \r 21/47 (44.7%)\r \r 22/47 (46.8%)\r \r 23/47 (48.9%)\r \r 24/47 (51.1%)\r \r 25/47 (53.2%)\r \r 26/47 (55.3%)\r \r 27/47 (57.4%)\r \r 28/47 (59.6%)\r \r 29/47 (61.7%)\r \r 30/47 (63.8%)\r \r 31/47 (66.0%)\r \r 32/47 (68.1%)\r \r 33/47 (70.2%)\r \r 34/47 (72.3%)\r \r 35/47 (74.5%)\r \r 36/47 (76.6%)\r \r 37/47 (78.7%)\r \r 38/47 (80.9%)\r \r 39/47 (83.0%)\r \r 40/47 (85.1%)\r \r 41/47 (87.2%)\r \r 42/47 (89.4%)\r \r 43/47 (91.5%)\r \r 44/47 (93.6%)\r \r 45/47 (95.7%)\r \r 46/47 (97.9%)\r \r 47/47 (100.0%)\r \r Ran 47 tests with 3 failures, 1 errors and 0 skipped in 0.054 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True If you also want a summary of errors at the end, ask for verbosity as well as progress output. Suppressing multiple doctest errors ----------------------------------- Often, when a doctest example fails, the failure will cause later examples in the same test to fail. Each failure is reported: >>> sys.argv = 'test --tests-pattern ^sampletests_1$'.split() >>> testrunner.run_internal(defaults) # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 21, in sample2.sampletests_1.eek Failed example: x Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x NameError: name 'x' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 24, in sample2.sampletests_1.eek Failed example: z = x + 1 Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? z = x + 1 NameError: name 'x' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True This can be a bit confusing, especially when there are enough tests that they scroll off a screen. Often you just want to see the first failure. This can be accomplished with the -1 option (for "just show me the first failed example in a doctest" :) >>> sys.argv = 'test --tests-pattern ^sampletests_1$ -1'.split() >>> testrunner.run_internal(defaults) # doctest: Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.001 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True The --hide-secondary-failures option is an alias for -1: >>> sys.argv = ( ... 'test --tests-pattern ^sampletests_1$' ... ' --hide-secondary-failures' ... ).split() >>> testrunner.run_internal(defaults) # doctest: Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.001 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True The --show-secondary-failures option counters -1 (or it's alias), causing the second and subsequent errors to be shown. This is useful if -1 is provided by a test script by inserting it ahead of command-line options in sys.argv. >>> sys.argv = ( ... 'test --tests-pattern ^sampletests_1$' ... ' --hide-secondary-failures --show-secondary-failures' ... ).split() >>> testrunner.run_internal(defaults) # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 21, in sample2.sampletests_1.eek Failed example: x Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x NameError: name 'x' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 24, in sample2.sampletests_1.eek Failed example: z = x + 1 Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? z = x + 1 NameError: name 'x' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True Getting diff output for doctest failures ---------------------------------------- If a doctest has large expected and actual output, it can be hard to see differences when expected and actual output differ. The --ndiff, --udiff, and --cdiff options can be used to get diff output of various kinds. >>> sys.argv = 'test --tests-pattern ^pledge$'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Expected: I give my pledge, as an earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. It's soils, minerals, forests, waters, and wildlife. Got: I give my pledge, as and earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. It's soils, minerals, forests, waters, and wildlife. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Here, the actual output uses the word "and" rather than the word "an", but it's a bit hard to pick this out. We can use the various diff outputs to see this better. We could modify the test to ask for diff output, but it's easier to use one of the diff options. The --ndiff option requests a diff using Python's ndiff utility. This is the only method that marks differences within lines as well as across lines. For example, if a line of expected output contains digit 1 where actual output contains letter l, a line is inserted with a caret marking the mismatching column positions. >>> sys.argv = 'test --tests-pattern ^pledge$ --ndiff'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Differences (ndiff with -expected +actual): - I give my pledge, as an earthling, + I give my pledge, as and earthling, ? + to save, and faithfully, to defend from waste, the natural resources of my planet. It's soils, minerals, forests, waters, and wildlife. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.003 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. The -udiff option requests a standard "unified" diff: >>> sys.argv = 'test --tests-pattern ^pledge$ --udiff'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Differences (unified diff with -expected +actual): @@ -1,3 +1,3 @@ -I give my pledge, as an earthling, +I give my pledge, as and earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. The -cdiff option requests a standard "context" diff: >>> sys.argv = 'test --tests-pattern ^pledge$ --cdiff'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Differences (context diff with expected followed by actual): *************** *** 1,3 **** ! I give my pledge, as an earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. --- 1,3 ---- ! I give my pledge, as and earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Specifying more than one diff option at once causes an error: >>> sys.argv = 'test --tests-pattern ^pledge$ --cdiff --udiff'.split() >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 >>> sys.argv = 'test --tests-pattern ^pledge$ --cdiff --ndiff'.split() >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 >>> sys.argv = 'test --tests-pattern ^pledge$ --udiff --ndiff'.split() >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 Testing-Module Import Errors ---------------------------- If there are errors when importing a test module, these errors are reported. In order to illustrate a module with a syntax error, we create one now: this module used to be checked in to the project, but then it was included in distributions of projects using zope.testrunner too, and distutils complained about the syntax error when it compiled Python files during installation of such projects. So first we create a module with bad syntax: >>> badsyntax_path = os.path.join(directory_with_tests, ... "sample2", "sampletests_i.py") >>> f = open(badsyntax_path, "w") >>> print("importx unittest", file=f) # syntax error >>> f.close() Then run the tests: >>> sys.argv = ('test --tests-pattern ^sampletests(f|_i)?$ --layer 1 ' ... ).split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE Test-module import failures: Module: sample2.sampletests_i Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_i.py", line 1 importx unittest ^ SyntaxError: invalid syntax Module: sample2.sample21.sampletests_i Traceback (most recent call last): File "testrunner-ex/sample2/sample21/sampletests_i.py", line 15, in ? import zope.testrunner.huh ImportError: No module named huh Module: sample2.sample23.sampletests_i Traceback (most recent call last): File "testrunner-ex/sample2/sample23/sampletests_i.py", line 18, in ? class Test(unittest.TestCase): File "testrunner-ex/sample2/sample23/sampletests_i.py", line 23, in Test raise TypeError('eek') TypeError: eek Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in 0.000 seconds. Ran 9 tests with 0 failures, 3 errors and 0 skipped in 0.000 seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in 0.000 seconds. Set up samplelayers.Layer112 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.006 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. Test-modules with import problems: sample2.sampletests_i sample2.sample21.sampletests_i sample2.sample23.sampletests_i Total: 165 tests, 0 failures, 3 errors and 0 skipped in N.NNN seconds. True Reporting Errors to Calling Processes ------------------------------------- The testrunner returns the error status, indicating that the tests failed. This can be useful for an invoking process that wants to monitor the result of a test run. This is applied when invoking the testrunner using the ``run()`` function instead of ``run_internal()``: >>> sys.argv = ( ... 'test --tests-pattern ^sampletests_1$'.split()) >>> try: ... testrunner.run(defaults) ... except SystemExit as e: ... print('exited with code', e.code) ... else: ... print('sys.exit was not called') ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. ... Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. exited with code 1 Passing tests exit with code 0 according to UNIX practices: >>> sys.argv = ( ... 'test --tests-pattern ^sampletests$'.split()) >>> try: ... testrunner.run(defaults) ... except SystemExit as e2: ... print('exited with code', e2.code) ... else: ... print('sys.exit was not called') ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Total: 286 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. exited with code 0 And remove the temporary directory: >>> shutil.rmtree(tmpdir) Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Console Classifier: Framework :: Zope3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Zope Public License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Topic :: Software Development :: Testing zope.testrunner-4.4.9/bootstrap.py0000664000000000000000000001454512527371226016024 0ustar rootroot############################################################################## # # Copyright (c) 2006 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Bootstrap a buildout-based project Simply run this script in a directory containing a buildout.cfg. The script accepts buildout command-line options, so you can use the -c option to specify an alternate configuration file. """ import os import shutil import sys import tempfile from optparse import OptionParser tmpeggs = tempfile.mkdtemp() usage = '''\ [DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options] Bootstraps a buildout-based project. Simply run this script in a directory containing a buildout.cfg, using the Python that you want bin/buildout to use. Note that by using --find-links to point to local resources, you can keep this script from going over the network. ''' parser = OptionParser(usage=usage) parser.add_option("-v", "--version", help="use a specific zc.buildout version") parser.add_option("-t", "--accept-buildout-test-releases", dest='accept_buildout_test_releases', action="store_true", default=False, help=("Normally, if you do not specify a --version, the " "bootstrap script and buildout gets the newest " "*final* versions of zc.buildout and its recipes and " "extensions for you. If you use this flag, " "bootstrap and buildout will get the newest releases " "even if they are alphas or betas.")) parser.add_option("-c", "--config-file", help=("Specify the path to the buildout configuration " "file to be used.")) parser.add_option("-f", "--find-links", help=("Specify a URL to search for buildout releases")) parser.add_option("--allow-site-packages", action="store_true", default=False, help=("Let bootstrap.py use existing site packages")) parser.add_option("--setuptools-version", help="use a specific setuptools version") options, args = parser.parse_args() ###################################################################### # load/install setuptools try: if options.allow_site_packages: import setuptools import pkg_resources from urllib.request import urlopen except ImportError: from urllib2 import urlopen ez = {} exec(urlopen('https://bootstrap.pypa.io/ez_setup.py').read(), ez) if not options.allow_site_packages: # ez_setup imports site, which adds site packages # this will remove them from the path to ensure that incompatible versions # of setuptools are not in the path import site # inside a virtualenv, there is no 'getsitepackages'. # We can't remove these reliably if hasattr(site, 'getsitepackages'): for sitepackage_path in site.getsitepackages(): sys.path[:] = [x for x in sys.path if sitepackage_path not in x] setup_args = dict(to_dir=tmpeggs, download_delay=0) if options.setuptools_version is not None: setup_args['version'] = options.setuptools_version ez['use_setuptools'](**setup_args) import setuptools import pkg_resources # This does not (always?) update the default working set. We will # do it. for path in sys.path: if path not in pkg_resources.working_set.entries: pkg_resources.working_set.add_entry(path) ###################################################################### # Install buildout ws = pkg_resources.working_set cmd = [sys.executable, '-c', 'from setuptools.command.easy_install import main; main()', '-mZqNxd', tmpeggs] find_links = os.environ.get( 'bootstrap-testing-find-links', options.find_links or ('http://downloads.buildout.org/' if options.accept_buildout_test_releases else None) ) if find_links: cmd.extend(['-f', find_links]) setuptools_path = ws.find( pkg_resources.Requirement.parse('setuptools')).location requirement = 'zc.buildout' version = options.version if version is None and not options.accept_buildout_test_releases: # Figure out the most recent final version of zc.buildout. import setuptools.package_index _final_parts = '*final-', '*final' def _final_version(parsed_version): try: return not parsed_version.is_prerelease except AttributeError: # Older setuptools for part in parsed_version: if (part[:1] == '*') and (part not in _final_parts): return False return True index = setuptools.package_index.PackageIndex( search_path=[setuptools_path]) if find_links: index.add_find_links((find_links,)) req = pkg_resources.Requirement.parse(requirement) if index.obtain(req) is not None: best = [] bestv = None for dist in index[req.project_name]: distv = dist.parsed_version if _final_version(distv): if bestv is None or distv > bestv: best = [dist] bestv = distv elif distv == bestv: best.append(dist) if best: best.sort() version = best[-1].version if version: requirement = '=='.join((requirement, version)) cmd.append(requirement) import subprocess if subprocess.call(cmd, env=dict(os.environ, PYTHONPATH=setuptools_path)) != 0: raise Exception( "Failed to execute command:\n%s" % repr(cmd)[1:-1]) ###################################################################### # Import and run buildout ws.add_entry(tmpeggs) ws.require(requirement) import zc.buildout.buildout if not [a for a in args if '=' not in a]: args.append('bootstrap') # if -c was provided, we push it back into args for buildout' main function if options.config_file is not None: args[0:0] = ['-c', options.config_file] zc.buildout.buildout.main(args) shutil.rmtree(tmpeggs) zope.testrunner-4.4.9/src/0000775000000000000000000000000012552256734014217 5ustar rootrootzope.testrunner-4.4.9/src/zope.testrunner.egg-info/0000775000000000000000000000000012552256734021076 5ustar rootrootzope.testrunner-4.4.9/src/zope.testrunner.egg-info/entry_points.txt0000664000000000000000000000017012527371232024363 0ustar rootroot[console_scripts] zope-testrunner = zope.testrunner:run [distutils.commands] ftest = zope.testrunner.eggsupport:ftest zope.testrunner-4.4.9/src/zope.testrunner.egg-info/PKG-INFO0000664000000000000000000072423412527371232022200 0ustar rootrootMetadata-Version: 1.1 Name: zope.testrunner Version: 4.4.9 Summary: Zope testrunner script. Home-page: http://pypi.python.org/pypi/zope.testrunner Author: Zope Foundation and Contributors Author-email: zope-dev@zope.org License: ZPL 2.1 Description: *************** zope.testrunner *************** |buildstatus|_ |winbotstatus|_ .. contents:: This package provides a flexible test runner with layer support. You can find more `detailed documentation`_ on PyPI or in the ``src/`` directory. Getting started *************** Buildout-based projects ======================= zope.testrunner is often used for projects that use buildout_:: [buildout] develop = . parts = ... test ... [test] recipe = zc.recipe.testrunner eggs = mypackage The usual buildout process :: python bootstrap.py bin/buildout creates a ``bin/test`` script that will run the tests for *mypackage*. .. tip:: zc.recipe.testrunner_ takes care to specify the right ``--test-path`` option in the generated script. You can add other options (such as ``--tests-pattern``) too; check zc.recipe.testrunner_'s documentation for details. Virtualenv-based projects ========================= ``pip install zope.testrunner`` and you'll get a ``zope-testrunner`` script. Run your tests with :: zope-testrunner --test-path=path/to/your/source/tree Your source code needs to be available for the testrunner to import, so you need to run ``python setup.py install`` or ``pip install -e .`` into the same virtualenv_. Some useful command-line options to get you started =================================================== -p show a percentage indicator -v increase verbosity -c colorize the output -t test specify test names (one or more regexes) -m module specify test modules (one or more regexes) -s package specify test packages (one or more regexes) --list-tests show names of tests instead of running them -x stop on first error or failure -D, --pdb enable post-mortem debugging of test failures --help show *all* command-line options (there are many more!) For example :: bin/test -pvc -m test_foo -t TestBar runs all TestBar tests from a module called test_foo.py. Writing tests ============= ``zope.testrunner`` expects to find your tests inside your package directory, in a subpackage or module named ``tests``. Test modules in a test subpackage should be named ``test*.py``. .. tip:: You can change these assumptions with ``--tests-pattern`` and ``--test-file-pattern`` test runner options. Tests themselves should be classes inheriting from ``unittest.TestCase``, and if you wish to use doctests, please tell the test runner where to find them and what options to use for them in by supplying a function named ``test_suite``. Example:: import unittest import doctest class TestArithmetic(unittest.TestCase): def test_two_plus_two(self): self.assertEqual(2 + 2, 4) def doctest_string_formatting(): """Test Python string formatting >>> print('{} + {}'.format(2, 2)) 2 + 2 """ def test_suite(): return unittest.TestSuite([ unittest.makeSuite(TestArithmetic), doctest.DocTestSuite(), doctest.DocFileSuite('../README.txt', optionflags=doctest.ELLIPSIS), ]) Test grouping ============= In addition to per-package and per-module filtering, zope.testrunner has other mechanisms for grouping tests: * **layers** allow you to have shared setup/teardown code to be used by a group of tests, that is executed only once, and not for each test. Layers are orthogonal to the usual package/module structure and are specified by setting the ``layer`` attribute on test suites. * **levels** allow you to group slow-running tests and not run them by default. They're specified by setting the ``level`` attribute on test suites to an int. For more details please see the `detailed documentation`_. Other features ============== zope.testrunner can profile your tests, measure test coverage, check for memory leaks, integrate with subunit_, shuffle the test execution order, and run multiple tests in parallel. For more details please see the `detailed documentation`_. .. _buildout: http://www.buildout.org/ .. _virtualenv: http://www.virtualenv.org/ .. _zc.recipe.testrunner: http://pypi.python.org/pypi/zc.recipe.testrunner .. _subunit: http://pypi.python.org/pypi/subunit .. _detailed documentation: http://docs.zope.org/zope.testrunner/ .. |buildstatus| image:: https://api.travis-ci.org/zopefoundation/zope.testrunner.png?branch=master .. _buildstatus: https://travis-ci.org/zopefoundation/zope.testrunner .. |winbotstatus| image:: http://winbot.zope.org/buildstatusimage?builder=zope.testrunner_py_265_32&number=-1 .. _winbotstatus: http://winbot.zope.org/builders/zope.testrunner_py_265_32/builds/-1 zope.testrunner Changelog ************************* 4.4.9 (2015-05-21) ================== - When using ``-j``, parallelize all the tests, including the first test layer (`#28 `_). 4.4.8 (2015-05-01) ================== - Support skipped tests in subunit output (`#25 `_). - More efficient test filtering (`#26 `_). 4.4.7 (2015-04-02) ================== - Work around a bug in PyPy3's curses module (`#24 `_). 4.4.6 (2015-01-21) ================== - Restore support for instance-based test layers that regressed in 4.4.5 (`#20 `_). 4.4.5 (2015-01-06) ================== - Sort related layers close to each other to reduce the number of unnecessary teardowns (fixes `#14 `_). - Run the unit test layer first (fixes `LP #497871 `__). 4.4.4 (2014-12-27) ================== - When looking for the right location of test code, start with longest location paths first. This fixes problems with nested code locations. 4.4.3 (2014-03-19) ================== - Added support for Python 3.4. 4.4.2 (2014-02-22) ================== - Drop support for Python 3.1. - Fix post-mortem debugging when a non-printable exception happens (https://github.com/zopefoundation/zope.testrunner/issues/8). 4.4.1 (2013-07-10) ================== - Updated ``boostrap.py`` to version 2.2. - Fix nondeterministic test failures on Python 3.3 - Tear down layers after ``post_mortem`` debugging is finished. - Fix tests that write to source directory, it might be read-only. 4.4.0 (2013-06-06) ================== - Fix tests selection when the negative "!" pattern is used several times (LP #1160965) - Moved tests into a 'tests' subpackage. - Made ``python -m zope.testrunner`` work again. - Support 'skip' feature of unittest2 (which became the new unittest in Python 2.7). - Better diagnostics when communication with subprocess fails (https://github.com/zopefoundation/zope.testrunner/issues/5). - Do not break subprocess execution when the test suite changes the working directory (https://github.com/zopefoundation/zope.testrunner/issues/6). - Count test module import errors as errors (LP #1026576). 4.3.3 (2013-03-03) ================== - Running layers in sub-processes did not use to work when run via ``python setup.py ftest`` since it tried to run setup.py with all the command line options. It now detects ``setup.py`` runs and we run the test runner directly. 4.3.2 (2013-03-03) ================== - Fix ``SkipLayers`` class in cases where the distribution specifies a ``test_suite`` value. 4.3.1 (2013-03-02) ================== - Fixed a bug in the `ftest` command and added a test. - Fixed a trivial test failure with Python 3 of the previous release. 4.3.0 (2013-03-02) ================== - Expose `ftest` distutils command via an entry point. - Added tests for ``zope.testrunner.eggsupport``. 4.2.0 (2013-02-12) ================== - Dropped use of 2to3, rewrote source code to be compatible with all Python versions. Introduced a dependency on `six`_. 4.1.1 (2013-02-08) ================== - Dropped use of zope.fixers (LP: #1118877). - Fixed tox test error reporting; fixed tests on Pythons 2.6, 3.1, 3.2, 3.3 and PyPy 1.9. - Fix --shuffle ordering on Python 3.2 to be the same as it was on older Python versions. - Fix --shuffle nondeterminism when multiple test layers are present. Note: this will likely change the order of tests for the same --shuffle-seed. - New option: --profile-directory. Use it in the test suite so that tests executed by detox in parallel don't conflict. - Use a temporary coverage directory in the test suite so that tests executed by detox in parallel don't conflict. - Fix --post-mortem (aka -D, --pdb) when a test module cannot be imported or is invalid (LP #1119363). 4.1.0 (2013-02-07) ================== - Replaced deprecated ``zope.interface.implements`` usage with equivalent ``zope.interface.implementer`` decorator. - Dropped support for Python 2.4 and 2.5. - Made StartUpFailure compatible with unittest.TextTestRunner() (LP #1118344). 4.0.4 (2011-10-25) ================== - Work around sporadic timing-related issues in the subprocess buffering tests. Thanks to Jonathan Ballet for the patch! 4.0.3 (2011-03-17) ================== - Added back support for Python <= 2.6 which was broken in 4.0.2. 4.0.2 (2011-03-16) ================== - Added back Python 3 support which was broken in 4.0.1. - Fixed `Unexpected success`_ support by implementing the whole concept. - Added support for the new __pycache__ directories in Python 3.2. 4.0.1 (2011-02-21) ================== - LP #719369: An `Unexpected success`_ (concept introduced in Python 2.7) is no longer handled as success but as failure. This is a workaround. The whole unexpected success concept might be implemented later. .. _`Unexpected success`: http://www.voidspace.org.uk/python/articles/unittest2.shtml#more-skipping 4.0.0 (2010-10-19) ================== - Show more information about layers whose setup fails (LP #638153). 4.0.0b5 (2010-07-20) ==================== - Update fix for LP #221151 to a spelling compatible with Python 2.4. - Timestamps are now always included in subunit output (r114849). - LP #591309: fix a crash when subunit reports test failures containing UTF8-encoded data. 4.0.0b4 (2010-06-23) ==================== - Package as a zipfile to work around Python 2.4 distutils bug (no feature changes or bugfixes in ``zope.testrunner`` itself). 4.0.0b3 (2010-06-16) ==================== - LP #221151: keep ``unittest.TestCase.shortDescription`` happy by supplying a ``_testMethodDoc`` attribute. - LP #595052: keep the distribution installable under Python 2.4: its distutils appears to munge the empty ``__init__.py`` file in the ``foo.bar`` egg used for testing into a directory. - LP #580083: fix the ``bin/test`` script to run only tests from ``zope.testrunner``. - LP #579019: When layers were run in parallel, their tearDown was not called. Additionally, the first layer which was run in the main thread did not have its tearDown called either. 4.0.0b2 (2010-05-03) ==================== - Having 'sampletests' in the MANIFEST.in gave warnings, but doesn't actually seem to include any more files, so I removed it. - Moved zope.testing.exceptions to zope.testrunner.exceptions. Now zope.testrunner no longer requires zope.testing except for when running its own tests. 4.0.0b1 (2010-04-29) ==================== - Initial release of the testrunner from zope.testrunner as its own module. (Previously it was part of zope.testing.) .. _six: http://pypi.python.org/pypi/six Detailed Documentation ********************** Test Runner =========== The testrunner module is used to run automated tests defined using the unittest framework. Its primary feature is that it *finds* tests by searching directory trees. It doesn't require the manual concatenation of specific test suites. It is highly customizable and should be usable with any project. In addition to finding and running tests, it provides the following additional features: - Test filtering using specifications of: o test packages within a larger tree o regular expression patterns for test modules o regular expression patterns for individual tests - Organization of tests into levels and layers Sometimes, tests take so long to run that you don't want to run them on every run of the test runner. Tests can be defined at different levels. The test runner can be configured to only run tests at a specific level or below by default. Command-line options can be used to specify a minimum level to use for a specific run, or to run all tests. Individual tests or test suites can specify their level via a 'level' attribute. where levels are integers increasing from 1. Most tests are unit tests. They don't depend on other facilities, or set up whatever dependencies they have. For larger applications, it's useful to specify common facilities that a large number of tests share. Making each test set up and and tear down these facilities is both ineffecient and inconvenient. For this reason, we've introduced the concept of layers, based on the idea of layered application architectures. Software build for a layer should be able to depend on the facilities of lower layers already being set up. For example, Zope defines a component architecture. Much Zope software depends on that architecture. We should be able to treat the component architecture as a layer that we set up once and reuse. Similarly, Zope application software should be able to depend on the Zope application server without having to set it up in each test. The test runner introduces test layers, which are objects that can set up environments for tests within the layers to use. A layer is set up before running the tests in it. Individual tests or test suites can define a layer by defining a `layer` attribute, which is a test layer. - Reporting - progress meter - summaries of tests run - Analysis of test execution - post-mortem debugging of test failures - memory leaks - code coverage - source analysis using pychecker - memory errors - execution times - profiling Simple Usage ============ The test runner consists of an importable module. The test runner is used by providing scripts that import and invoke the `run` method from the module. The `testrunner` module is controlled via command-line options. Test scripts supply base and default options by supplying a list of default command-line options that are processed before the user-supplied command-line options are provided. Typically, a test script does 2 things: - Adds the directory containing the zope package to the Python path. - Calls the test runner with default arguments and arguments supplied to the script. Normally, it just passes default/setup arguments. The test runner uses `sys.argv` to get the user's input. This testrunner_ex subdirectory contains a number of sample packages with tests. Let's run the tests found here. First though, we'll set up our default options: >>> import os.path >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] The default options are used by a script to customize the test runner for a particular application. In this case, we use two options: path Set the path where the test runner should look for tests. This path is also added to the Python path. tests-pattern Tell the test runner how to recognize modules or packages containing tests. Now, if we run the tests, without any other options: >>> from zope import testrunner >>> import sys >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False we see the normal testrunner output, which summarizes the tests run for each layer. For each layer, we see what layers had to be torn down or set up to run the layer and we see the number of tests run, with results. The test runner returns a boolean indicating whether there were errors. In this example, there were no errors, so it returned False. (Of course, the times shown in these examples are just examples. Times will vary depending on system speed.) Layers ====== A Layer is an object providing setup and teardown methods used to setup and teardown the environment provided by the layer. It may also provide setup and teardown methods used to reset the environment provided by the layer between each test. Layers are generally implemented as classes using class methods. >>> class BaseLayer: ... def setUp(cls): ... log('BaseLayer.setUp') ... setUp = classmethod(setUp) ... ... def tearDown(cls): ... log('BaseLayer.tearDown') ... tearDown = classmethod(tearDown) ... ... def testSetUp(cls): ... log('BaseLayer.testSetUp') ... testSetUp = classmethod(testSetUp) ... ... def testTearDown(cls): ... log('BaseLayer.testTearDown') ... testTearDown = classmethod(testTearDown) ... Layers can extend other layers. Note that they do not explicitly invoke the setup and teardown methods of other layers - the test runner does this for us in order to minimize the number of invocations. >>> class TopLayer(BaseLayer): ... def setUp(cls): ... log('TopLayer.setUp') ... setUp = classmethod(setUp) ... ... def tearDown(cls): ... log('TopLayer.tearDown') ... tearDown = classmethod(tearDown) ... ... def testSetUp(cls): ... log('TopLayer.testSetUp') ... testSetUp = classmethod(testSetUp) ... ... def testTearDown(cls): ... log('TopLayer.testTearDown') ... testTearDown = classmethod(testTearDown) ... Tests or test suites specify what layer they need by storing a reference in the 'layer' attribute. >>> import unittest >>> class TestSpecifyingBaseLayer(unittest.TestCase): ... 'This TestCase explicitly specifies its layer' ... layer = BaseLayer ... name = 'TestSpecifyingBaseLayer' # For testing only ... ... def setUp(self): ... log('TestSpecifyingBaseLayer.setUp') ... ... def tearDown(self): ... log('TestSpecifyingBaseLayer.tearDown') ... ... def test1(self): ... log('TestSpecifyingBaseLayer.test1') ... ... def test2(self): ... log('TestSpecifyingBaseLayer.test2') ... >>> class TestSpecifyingNoLayer(unittest.TestCase): ... 'This TestCase specifies no layer' ... name = 'TestSpecifyingNoLayer' # For testing only ... def setUp(self): ... log('TestSpecifyingNoLayer.setUp') ... ... def tearDown(self): ... log('TestSpecifyingNoLayer.tearDown') ... ... def test1(self): ... log('TestSpecifyingNoLayer.test') ... ... def test2(self): ... log('TestSpecifyingNoLayer.test') ... Create a TestSuite containing two test suites, one for each of TestSpecifyingBaseLayer and TestSpecifyingNoLayer. >>> umbrella_suite = unittest.TestSuite() >>> umbrella_suite.addTest(unittest.makeSuite(TestSpecifyingBaseLayer)) >>> no_layer_suite = unittest.makeSuite(TestSpecifyingNoLayer) >>> umbrella_suite.addTest(no_layer_suite) Before we can run the tests, we need to setup some helpers. >>> from zope.testrunner import options >>> from zope.testing.loggingsupport import InstalledHandler >>> import logging >>> log_handler = InstalledHandler('zope.testrunner.tests') >>> def log(msg): ... logging.getLogger('zope.testrunner.tests').info(msg) >>> def fresh_options(): ... opts = options.get_options(['--test-filter', '.*']) ... opts.resume_layer = None ... opts.resume_number = 0 ... return opts Now we run the tests. Note that the BaseLayer was not setup when the TestSpecifyingNoLayer was run and setup/torn down around the TestSpecifyingBaseLayer tests. >>> from zope.testrunner.runner import Runner >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running ...BaseLayer tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up ...BaseLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...BaseLayer in N.NNN seconds. Total: 4 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. Now lets specify a layer in the suite containing TestSpecifyingNoLayer and run the tests again. This demonstrates the other method of specifying a layer. This is generally how you specify what layer doctests need. >>> no_layer_suite.layer = BaseLayer >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running ...BaseLayer tests: Set up ...BaseLayer in N.NNN seconds. Ran 4 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...BaseLayer in N.NNN seconds. Clear our logged output, as we want to inspect it shortly. >>> log_handler.clear() Now lets also specify a layer in the TestSpecifyingNoLayer class and rerun the tests. This demonstrates that the most specific layer is used. It also shows the behavior of nested layers - because TopLayer extends BaseLayer, both the BaseLayer and TopLayer environments are setup when the TestSpecifyingNoLayer tests are run. >>> TestSpecifyingNoLayer.layer = TopLayer >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running ...BaseLayer tests: Set up ...BaseLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running ...TopLayer tests: Set up ...TopLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...TopLayer in N.NNN seconds. Tear down ...BaseLayer in N.NNN seconds. Total: 4 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. If we inspect our trace of what methods got called in what order, we can see that the layer setup and teardown methods only got called once. We can also see that the layer's test setup and teardown methods got called for each test using that layer in the right order. >>> def report(): ... print("Report:") ... for record in log_handler.records: ... print(record.getMessage()) >>> report() Report: BaseLayer.setUp BaseLayer.testSetUp TestSpecifyingBaseLayer.setUp TestSpecifyingBaseLayer.test1 TestSpecifyingBaseLayer.tearDown BaseLayer.testTearDown BaseLayer.testSetUp TestSpecifyingBaseLayer.setUp TestSpecifyingBaseLayer.test2 TestSpecifyingBaseLayer.tearDown BaseLayer.testTearDown TopLayer.setUp BaseLayer.testSetUp TopLayer.testSetUp TestSpecifyingNoLayer.setUp TestSpecifyingNoLayer.test TestSpecifyingNoLayer.tearDown TopLayer.testTearDown BaseLayer.testTearDown BaseLayer.testSetUp TopLayer.testSetUp TestSpecifyingNoLayer.setUp TestSpecifyingNoLayer.test TestSpecifyingNoLayer.tearDown TopLayer.testTearDown BaseLayer.testTearDown TopLayer.tearDown BaseLayer.tearDown Now lets stack a few more layers to ensure that our setUp and tearDown methods are called in the correct order. >>> from zope.testrunner.find import name_from_layer >>> class A(object): ... def setUp(cls): ... log('%s.setUp' % name_from_layer(cls)) ... setUp = classmethod(setUp) ... ... def tearDown(cls): ... log('%s.tearDown' % name_from_layer(cls)) ... tearDown = classmethod(tearDown) ... ... def testSetUp(cls): ... log('%s.testSetUp' % name_from_layer(cls)) ... testSetUp = classmethod(testSetUp) ... ... def testTearDown(cls): ... log('%s.testTearDown' % name_from_layer(cls)) ... testTearDown = classmethod(testTearDown) ... >>> class B(A): pass >>> class C(B): pass >>> class D(A): pass >>> class E(D): pass >>> class F(C,E): pass >>> class DeepTest(unittest.TestCase): ... layer = F ... def test(self): ... pass >>> suite = unittest.makeSuite(DeepTest) >>> log_handler.clear() >>> runner = Runner(options=fresh_options(), args=[], found_suites=[suite]) >>> succeeded = runner.run() #doctest: +ELLIPSIS Running ...F tests: Set up ...A in N.NNN seconds. Set up ...B in N.NNN seconds. Set up ...C in N.NNN seconds. Set up ...D in N.NNN seconds. Set up ...E in N.NNN seconds. Set up ...F in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...F in N.NNN seconds. Tear down ...E in N.NNN seconds. Tear down ...D in N.NNN seconds. Tear down ...C in N.NNN seconds. Tear down ...B in N.NNN seconds. Tear down ...A in N.NNN seconds. >>> report() #doctest: +ELLIPSIS Report: ...A.setUp ...B.setUp ...C.setUp ...D.setUp ...E.setUp ...F.setUp ...A.testSetUp ...B.testSetUp ...C.testSetUp ...D.testSetUp ...E.testSetUp ...F.testSetUp ...F.testTearDown ...E.testTearDown ...D.testTearDown ...C.testTearDown ...B.testTearDown ...A.testTearDown ...F.tearDown ...E.tearDown ...D.tearDown ...C.tearDown ...B.tearDown ...A.tearDown Layer Selection =============== We can select which layers to run using the --layer option: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 112 --layer Unit'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 182 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False We can also specify that we want to run only the unit tests: >>> sys.argv = 'test -u'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Or that we want to run all of the tests except for the unit tests: >>> sys.argv = 'test -f'.split() >>> testrunner.run_internal(defaults) Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 165 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Or we can explicitly say that we want both unit and non-unit tests. >>> sys.argv = 'test -uf'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False It is possible to force the layers to run in subprocesses and parallelize them. ``EmptyLayer`` will be inserted as first to start spreading out subprocesses ASAP. >>> sys.argv = [testrunner_script, '-j2'] >>> testrunner.run_internal(defaults) Running .EmptyLayer tests: Set up .EmptyLayer in N.NNN seconds. Ran 0 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running zope.testrunner.layer.UnitTests tests: Running in a subprocess. Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Running samplelayers.Layer1 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer11 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer111 tests: Running in a subprocess. Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer111 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer112 tests: Running in a subprocess. Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer12 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer121 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer121 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer122 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Tearing down left over layers: Tear down .EmptyLayer in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Passing arguments explicitly ============================ In most of the examples here, we set up `sys.argv`. In normal usage, the testrunner just uses `sys.argv`. It is possible to pass arguments explicitly. >>> import os.path >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> testrunner.run_internal(defaults, 'test --layer 111'.split()) Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False If options already have default values, then passing a different default will override. For example, --list-tests defaults to being turned off, but if we pass in a different default, that one takes effect. >>> defaults = [ ... '--list-tests', ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> testrunner.run_internal(defaults, 'test --layer 111'.split()) Listing samplelayers.Layer111 tests: test_x1 (sample1.sampletests.test111.TestA) test_y0 (sample1.sampletests.test111.TestA) test_z0 (sample1.sampletests.test111.TestA) test_x0 (sample1.sampletests.test111.TestB) test_y1 (sample1.sampletests.test111.TestB) test_z0 (sample1.sampletests.test111.TestB) test_1 (sample1.sampletests.test111.TestNotMuch) test_2 (sample1.sampletests.test111.TestNotMuch) test_3 (sample1.sampletests.test111.TestNotMuch) test_x0 (sample1.sampletests.test111) test_y0 (sample1.sampletests.test111) test_z1 (sample1.sampletests.test111) /home/benji/workspace/zope.testrunner/1/src/zope/testing/testrunner/testrunner-ex/sample1/sampletests/../../sampletestsl.txt test_x1 (sampletests.test111.TestA) test_y0 (sampletests.test111.TestA) test_z0 (sampletests.test111.TestA) test_x0 (sampletests.test111.TestB) test_y1 (sampletests.test111.TestB) test_z0 (sampletests.test111.TestB) test_1 (sampletests.test111.TestNotMuch) test_2 (sampletests.test111.TestNotMuch) test_3 (sampletests.test111.TestNotMuch) test_x0 (sampletests.test111) test_y0 (sampletests.test111) test_z1 (sampletests.test111) /home/benji/workspace/zope.testrunner/1/src/zope/testing/testrunner/testrunner-ex/sampletests/../sampletestsl.txt False Verbose Output ============== Normally, we just get a summary. We can use the -v option to get increasingly more information. If we use a single --verbose (-v) option, we get a dot printed for each test: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 122 -v'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: .................................. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.007 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False If there are more than 50 tests, the dots are printed in groups of 50: >>> sys.argv = 'test -uv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: ................................................................................................................................................................................................ Ran 156 tests with 0 failures, 0 errors and 0 skipped in 0.035 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False If the --verbose (-v) option is used twice, then the name and location of each test is printed as it is run: >>> sys.argv = 'test --layer 122 -vv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122.TestA) test_z0 (sample1.sampletests.test122.TestA) test_x0 (sample1.sampletests.test122.TestB) test_y1 (sample1.sampletests.test122.TestB) test_z0 (sample1.sampletests.test122.TestB) test_1 (sample1.sampletests.test122.TestNotMuch) test_2 (sample1.sampletests.test122.TestNotMuch) test_3 (sample1.sampletests.test122.TestNotMuch) test_x0 (sample1.sampletests.test122) test_y0 (sample1.sampletests.test122) test_z1 (sample1.sampletests.test122) testrunner-ex/sample1/sampletests/../../sampletestsl.txt test_x1 (sampletests.test122.TestA) test_y0 (sampletests.test122.TestA) test_z0 (sampletests.test122.TestA) test_x0 (sampletests.test122.TestB) test_y1 (sampletests.test122.TestB) test_z0 (sampletests.test122.TestB) test_1 (sampletests.test122.TestNotMuch) test_2 (sampletests.test122.TestNotMuch) test_3 (sampletests.test122.TestNotMuch) test_x0 (sampletests.test122) test_y0 (sampletests.test122) test_z1 (sampletests.test122) testrunner-ex/sampletests/../sampletestsl.txt Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.009 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False if the --verbose (-v) option is used three times, then individual test-execution times are printed: >>> sys.argv = 'test --layer 122 -vvv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) (0.000 s) test_y0 (sample1.sampletests.test122.TestA) (0.000 s) test_z0 (sample1.sampletests.test122.TestA) (0.000 s) test_x0 (sample1.sampletests.test122.TestB) (0.000 s) test_y1 (sample1.sampletests.test122.TestB) (0.000 s) test_z0 (sample1.sampletests.test122.TestB) (0.000 s) test_1 (sample1.sampletests.test122.TestNotMuch) (0.000 s) test_2 (sample1.sampletests.test122.TestNotMuch) (0.000 s) test_3 (sample1.sampletests.test122.TestNotMuch) (0.000 s) test_x0 (sample1.sampletests.test122) (0.001 s) test_y0 (sample1.sampletests.test122) (0.001 s) test_z1 (sample1.sampletests.test122) (0.001 s) testrunner-ex/sample1/sampletests/../../sampletestsl.txt (0.001 s) test_x1 (sampletests.test122.TestA) (0.000 s) test_y0 (sampletests.test122.TestA) (0.000 s) test_z0 (sampletests.test122.TestA) (0.000 s) test_x0 (sampletests.test122.TestB) (0.000 s) test_y1 (sampletests.test122.TestB) (0.000 s) test_z0 (sampletests.test122.TestB) (0.000 s) test_1 (sampletests.test122.TestNotMuch) (0.000 s) test_2 (sampletests.test122.TestNotMuch) (0.000 s) test_3 (sampletests.test122.TestNotMuch) (0.000 s) test_x0 (sampletests.test122) (0.001 s) test_y0 (sampletests.test122) (0.001 s) test_z1 (sampletests.test122) (0.001 s) testrunner-ex/sampletests/../sampletestsl.txt (0.001 s) Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.009 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False Quiet output ------------ The --quiet (-q) option cancels all verbose options. It's useful when the default verbosity is non-zero: >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... '-v' ... ] >>> sys.argv = 'test -q -u'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in 0.034 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Test Selection ============== We've already seen that we can select tests by layer. There are three other ways we can select tests. We can select tests by package: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 122 -ssample1 -vv'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122.TestA) test_z0 (sample1.sampletests.test122.TestA) test_x0 (sample1.sampletests.test122.TestB) test_y1 (sample1.sampletests.test122.TestB) test_z0 (sample1.sampletests.test122.TestB) test_1 (sample1.sampletests.test122.TestNotMuch) test_2 (sample1.sampletests.test122.TestNotMuch) test_3 (sample1.sampletests.test122.TestNotMuch) test_x0 (sample1.sampletests.test122) test_y0 (sample1.sampletests.test122) test_z1 (sample1.sampletests.test122) testrunner-ex/sample1/sampletests/../../sampletestsl.txt Ran 13 tests with 0 failures, 0 errors and 0 skipped in 0.005 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False You can specify multiple packages: >>> sys.argv = 'test -u -vv -ssample1 -ssample2'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf.TestA) test_z0 (sample1.sampletestsf.TestA) test_x0 (sample1.sampletestsf.TestB) test_y1 (sample1.sampletestsf.TestB) test_z0 (sample1.sampletestsf.TestB) test_1 (sample1.sampletestsf.TestNotMuch) test_2 (sample1.sampletestsf.TestNotMuch) test_3 (sample1.sampletestsf.TestNotMuch) test_x0 (sample1.sampletestsf) test_y0 (sample1.sampletestsf) test_z1 (sample1.sampletestsf) testrunner-ex/sample1/../sampletests.txt test_x1 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sample11.sampletests.TestA) test_z0 (sample1.sample11.sampletests.TestA) test_x0 (sample1.sample11.sampletests.TestB) test_y1 (sample1.sample11.sampletests.TestB) test_z0 (sample1.sample11.sampletests.TestB) test_1 (sample1.sample11.sampletests.TestNotMuch) test_2 (sample1.sample11.sampletests.TestNotMuch) test_3 (sample1.sample11.sampletests.TestNotMuch) test_x0 (sample1.sample11.sampletests) test_y0 (sample1.sample11.sampletests) test_z1 (sample1.sample11.sampletests) testrunner-ex/sample1/sample11/../../sampletests.txt test_x1 (sample1.sample13.sampletests.TestA) test_y0 (sample1.sample13.sampletests.TestA) test_z0 (sample1.sample13.sampletests.TestA) test_x0 (sample1.sample13.sampletests.TestB) test_y1 (sample1.sample13.sampletests.TestB) test_z0 (sample1.sample13.sampletests.TestB) test_1 (sample1.sample13.sampletests.TestNotMuch) test_2 (sample1.sample13.sampletests.TestNotMuch) test_3 (sample1.sample13.sampletests.TestNotMuch) test_x0 (sample1.sample13.sampletests) test_y0 (sample1.sample13.sampletests) test_z1 (sample1.sample13.sampletests) testrunner-ex/sample1/sample13/../../sampletests.txt test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample2.sample21.sampletests.TestA) test_y0 (sample2.sample21.sampletests.TestA) test_z0 (sample2.sample21.sampletests.TestA) test_x0 (sample2.sample21.sampletests.TestB) test_y1 (sample2.sample21.sampletests.TestB) test_z0 (sample2.sample21.sampletests.TestB) test_1 (sample2.sample21.sampletests.TestNotMuch) test_2 (sample2.sample21.sampletests.TestNotMuch) test_3 (sample2.sample21.sampletests.TestNotMuch) test_x0 (sample2.sample21.sampletests) test_y0 (sample2.sample21.sampletests) test_z1 (sample2.sample21.sampletests) testrunner-ex/sample2/sample21/../../sampletests.txt test_x1 (sample2.sampletests.test_1.TestA) test_y0 (sample2.sampletests.test_1.TestA) test_z0 (sample2.sampletests.test_1.TestA) test_x0 (sample2.sampletests.test_1.TestB) test_y1 (sample2.sampletests.test_1.TestB) test_z0 (sample2.sampletests.test_1.TestB) test_1 (sample2.sampletests.test_1.TestNotMuch) test_2 (sample2.sampletests.test_1.TestNotMuch) test_3 (sample2.sampletests.test_1.TestNotMuch) test_x0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.test_1) test_z1 (sample2.sampletests.test_1) testrunner-ex/sample2/sampletests/../../sampletests.txt test_x1 (sample2.sampletests.testone.TestA) test_y0 (sample2.sampletests.testone.TestA) test_z0 (sample2.sampletests.testone.TestA) test_x0 (sample2.sampletests.testone.TestB) test_y1 (sample2.sampletests.testone.TestB) test_z0 (sample2.sampletests.testone.TestB) test_1 (sample2.sampletests.testone.TestNotMuch) test_2 (sample2.sampletests.testone.TestNotMuch) test_3 (sample2.sampletests.testone.TestNotMuch) test_x0 (sample2.sampletests.testone) test_y0 (sample2.sampletests.testone) test_z1 (sample2.sampletests.testone) testrunner-ex/sample2/sampletests/../../sampletests.txt Ran 104 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False You can specify directory names instead of packages (useful for tab-completion): >>> subdir = os.path.join(directory_with_tests, 'sample1') >>> sys.argv = ['test', '--layer', '122', '-s', subdir, '-vv'] >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122.TestA) test_z0 (sample1.sampletests.test122.TestA) test_x0 (sample1.sampletests.test122.TestB) test_y1 (sample1.sampletests.test122.TestB) test_z0 (sample1.sampletests.test122.TestB) test_1 (sample1.sampletests.test122.TestNotMuch) test_2 (sample1.sampletests.test122.TestNotMuch) test_3 (sample1.sampletests.test122.TestNotMuch) test_x0 (sample1.sampletests.test122) test_y0 (sample1.sampletests.test122) test_z1 (sample1.sampletests.test122) testrunner-ex/sample1/sampletests/../../sampletestsl.txt Ran 13 tests with 0 failures, 0 errors and 0 skipped in 0.005 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False We can select by test module name using the --module (-m) option: >>> sys.argv = 'test -u -vv -ssample1 -m_one -mtest1'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False and by test within the module using the --test (-t) option: >>> sys.argv = 'test -u -vv -ssample1 -m_one -mtest1 -tx0 -ty0'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) Ran 8 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False >>> sys.argv = 'test -u -vv -ssample1 -ttxt'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: testrunner-ex/sample1/../sampletests.txt testrunner-ex/sample1/sample11/../../sampletests.txt testrunner-ex/sample1/sample13/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 5 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False The --module and --test options take regular expressions. If the regular expressions specified begin with '!', then tests that don't match the regular expression are selected: >>> sys.argv = 'test -u -vv -ssample1 -m!sample1[.]sample1'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf.TestA) test_z0 (sample1.sampletestsf.TestA) test_x0 (sample1.sampletestsf.TestB) test_y1 (sample1.sampletestsf.TestB) test_z0 (sample1.sampletestsf.TestB) test_1 (sample1.sampletestsf.TestNotMuch) test_2 (sample1.sampletestsf.TestNotMuch) test_3 (sample1.sampletestsf.TestNotMuch) test_x0 (sample1.sampletestsf) test_y0 (sample1.sampletestsf) test_z1 (sample1.sampletestsf) testrunner-ex/sample1/../sampletests.txt test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 39 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Module and test filters can also be given as positional arguments: >>> sys.argv = 'test -u -vv -ssample1 !sample1[.]sample1'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf.TestA) test_z0 (sample1.sampletestsf.TestA) test_x0 (sample1.sampletestsf.TestB) test_y1 (sample1.sampletestsf.TestB) test_z0 (sample1.sampletestsf.TestB) test_1 (sample1.sampletestsf.TestNotMuch) test_2 (sample1.sampletestsf.TestNotMuch) test_3 (sample1.sampletestsf.TestNotMuch) test_x0 (sample1.sampletestsf) test_y0 (sample1.sampletestsf) test_z1 (sample1.sampletestsf) testrunner-ex/sample1/../sampletests.txt test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 39 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False >>> sys.argv = 'test -u -vv -ssample1 . txt'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: testrunner-ex/sample1/../sampletests.txt testrunner-ex/sample1/sample11/../../sampletests.txt testrunner-ex/sample1/sample13/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 5 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Sometimes, There are tests that you don't want to run by default. For example, you might have tests that take a long time. Tests can have a level attribute. If no level is specified, a level of 1 is assumed and, by default, only tests at level one are run. to run tests at a higher level, use the --at-level (-a) option to specify a higher level. For example, with the following options: >>> sys.argv = 'test -u -vv -t test_y1 -t test_y0'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sampletestsf.TestA) test_y1 (sampletestsf.TestB) test_y0 (sampletestsf) test_y0 (sample1.sampletestsf.TestA) test_y1 (sample1.sampletestsf.TestB) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y1 (sample1.sample11.sampletests.TestB) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y1 (sample1.sample13.sampletests.TestB) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y1 (sample1.sampletests.test1.TestB) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y1 (sample1.sampletests.test_one.TestB) test_y0 (sample1.sampletests.test_one) test_y0 (sample2.sample21.sampletests.TestA) test_y1 (sample2.sample21.sampletests.TestB) test_y0 (sample2.sample21.sampletests) test_y0 (sample2.sampletests.test_1.TestA) test_y1 (sample2.sampletests.test_1.TestB) test_y0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.testone.TestA) test_y1 (sample2.sampletests.testone.TestB) test_y0 (sample2.sampletests.testone) test_y0 (sample3.sampletests.TestA) test_y1 (sample3.sampletests.TestB) test_y0 (sample3.sampletests) test_y0 (sampletests.test1.TestA) test_y1 (sampletests.test1.TestB) test_y0 (sampletests.test1) test_y0 (sampletests.test_one.TestA) test_y1 (sampletests.test_one.TestB) test_y0 (sampletests.test_one) Ran 36 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False We get run 36 tests. If we specify a level of 2, we get some additional tests: >>> sys.argv = 'test -u -vv -a 2 -t test_y1 -t test_y0'.split() >>> testrunner.run_internal(defaults) Running tests at level 2 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sampletestsf.TestA) test_y0 (sampletestsf.TestA2) test_y1 (sampletestsf.TestB) test_y0 (sampletestsf) test_y0 (sample1.sampletestsf.TestA) test_y1 (sample1.sampletestsf.TestB) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y1 (sample1.sample11.sampletests.TestB) test_y1 (sample1.sample11.sampletests.TestB2) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y1 (sample1.sample13.sampletests.TestB) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y1 (sample1.sampletests.test1.TestB) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y1 (sample1.sampletests.test_one.TestB) test_y0 (sample1.sampletests.test_one) test_y0 (sample2.sample21.sampletests.TestA) test_y1 (sample2.sample21.sampletests.TestB) test_y0 (sample2.sample21.sampletests) test_y0 (sample2.sampletests.test_1.TestA) test_y1 (sample2.sampletests.test_1.TestB) test_y0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.testone.TestA) test_y1 (sample2.sampletests.testone.TestB) test_y0 (sample2.sampletests.testone) test_y0 (sample3.sampletests.TestA) test_y1 (sample3.sampletests.TestB) test_y0 (sample3.sampletests) test_y0 (sampletests.test1.TestA) test_y1 (sampletests.test1.TestB) test_y0 (sampletests.test1) test_y0 (sampletests.test_one.TestA) test_y1 (sampletests.test_one.TestB) test_y0 (sampletests.test_one) Ran 38 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False We can use the --all option to run tests at all levels: >>> sys.argv = 'test -u -vv --all -t test_y1 -t test_y0'.split() >>> testrunner.run_internal(defaults) Running tests at all levels Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sampletestsf.TestA) test_y0 (sampletestsf.TestA2) test_y1 (sampletestsf.TestB) test_y0 (sampletestsf) test_y0 (sample1.sampletestsf.TestA) test_y1 (sample1.sampletestsf.TestB) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sample11.sampletests.TestA3) test_y1 (sample1.sample11.sampletests.TestB) test_y1 (sample1.sample11.sampletests.TestB2) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y1 (sample1.sample13.sampletests.TestB) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y1 (sample1.sampletests.test1.TestB) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y1 (sample1.sampletests.test_one.TestB) test_y0 (sample1.sampletests.test_one) test_y0 (sample2.sample21.sampletests.TestA) test_y1 (sample2.sample21.sampletests.TestB) test_y0 (sample2.sample21.sampletests) test_y0 (sample2.sampletests.test_1.TestA) test_y1 (sample2.sampletests.test_1.TestB) test_y0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.testone.TestA) test_y1 (sample2.sampletests.testone.TestB) test_y0 (sample2.sampletests.testone) test_y0 (sample3.sampletests.TestA) test_y1 (sample3.sampletests.TestB) test_y0 (sample3.sampletests) test_y0 (sampletests.test1.TestA) test_y1 (sampletests.test1.TestB) test_y0 (sampletests.test1) test_y0 (sampletests.test_one.TestA) test_y1 (sampletests.test_one.TestB) test_y0 (sampletests.test_one) Ran 39 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Listing Selected Tests ---------------------- When you're trying to figure out why the test you want is not matched by the pattern you specified, it is convenient to see which tests match your specifications. >>> sys.argv = 'test --all -m sample1 -t test_y0 --list-tests'.split() >>> testrunner.run_internal(defaults) Listing zope.testrunner.layer.UnitTests tests: test_y0 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sample11.sampletests.TestA3) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one) Listing samplelayers.Layer11 tests: test_y0 (sample1.sampletests.test11.TestA) test_y0 (sample1.sampletests.test11) Listing samplelayers.Layer111 tests: test_y0 (sample1.sampletests.test111.TestA) test_y0 (sample1.sampletests.test111) Listing samplelayers.Layer112 tests: test_y0 (sample1.sampletests.test112.TestA) test_y0 (sample1.sampletests.test112) Listing samplelayers.Layer12 tests: test_y0 (sample1.sampletests.test12.TestA) test_y0 (sample1.sampletests.test12) Listing samplelayers.Layer121 tests: test_y0 (sample1.sampletests.test121.TestA) test_y0 (sample1.sampletests.test121) Listing samplelayers.Layer122 tests: test_y0 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122) False Test Progress ============= If the --progress (-p) option is used, progress information is printed and a carriage return (rather than a new-line) is printed between detail lines. Let's look at the effect of --progress (-p) at different levels of verbosity. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 122 -p'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Running: 1/26 (3.8%)##r## ##r## 2/26 (7.7%)##r## ##r## 3/26 (11.5%)##r## ##r## 4/26 (15.4%)##r## ##r## 5/26 (19.2%)##r## ##r## 6/26 (23.1%)##r## ##r## 7/26 (26.9%)##r## ##r## 8/26 (30.8%)##r## ##r## 9/26 (34.6%)##r## ##r## 10/26 (38.5%)##r## ##r## 11/26 (42.3%)##r## ##r## 12/26 (46.2%)##r## ##r## 13/26 (50.0%)##r## ##r## 14/26 (53.8%)##r## ##r## 15/26 (57.7%)##r## ##r## 16/26 (61.5%)##r## ##r## 17/26 (65.4%)##r## ##r## 18/26 (69.2%)##r## ##r## 19/26 (73.1%)##r## ##r## 20/26 (76.9%)##r## ##r## 21/26 (80.8%)##r## ##r## 22/26 (84.6%)##r## ##r## 23/26 (88.5%)##r## ##r## 24/26 (92.3%)##r## ##r## 25/26 (96.2%)##r## ##r## 26/26 (100.0%)##r## ##r## Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False (Note that, in the examples above and below, we show "##r##" followed by new lines where carriage returns would appear in actual output.) Using a single level of verbosity causes test descriptions to be output, but only if they fit in the terminal width. The default width, when the terminal width can't be determined, is 80: >>> sys.argv = 'test --layer 122 -pv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Running: 1/26 (3.8%) test_x1 (sample1.sampletests.test122.TestA)##r## ##r## 2/26 (7.7%) test_y0 (sample1.sampletests.test122.TestA)##r## ##r## 3/26 (11.5%) test_z0 (sample1.sampletests.test122.TestA)##r## ##r## 4/26 (15.4%) test_x0 (sample1.sampletests.test122.TestB)##r## ##r## 5/26 (19.2%) test_y1 (sample1.sampletests.test122.TestB)##r## ##r## 6/26 (23.1%) test_z0 (sample1.sampletests.test122.TestB)##r## ##r## 7/26 (26.9%) test_1 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 8/26 (30.8%) test_2 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 9/26 (34.6%) test_3 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 10/26 (38.5%) test_x0 (sample1.sampletests.test122)##r## ##r## 11/26 (42.3%) test_y0 (sample1.sampletests.test122)##r## ##r## 12/26 (46.2%) test_z1 (sample1.sampletests.test122)##r## ##r## testrunner-ex/sample1/sampletests/../../sampletestsl.txt##r## ##r## 14/26 (53.8%) test_x1 (sampletests.test122.TestA)##r## ##r## 15/26 (57.7%) test_y0 (sampletests.test122.TestA)##r## ##r## 16/26 (61.5%) test_z0 (sampletests.test122.TestA)##r## ##r## 17/26 (65.4%) test_x0 (sampletests.test122.TestB)##r## ##r## 18/26 (69.2%) test_y1 (sampletests.test122.TestB)##r## ##r## 19/26 (73.1%) test_z0 (sampletests.test122.TestB)##r## ##r## 20/26 (76.9%) test_1 (sampletests.test122.TestNotMuch)##r## ##r## 21/26 (80.8%) test_2 (sampletests.test122.TestNotMuch)##r## ##r## 22/26 (84.6%) test_3 (sampletests.test122.TestNotMuch)##r## ##r## 23/26 (88.5%) test_x0 (sampletests.test122)##r## ##r## 24/26 (92.3%) test_y0 (sampletests.test122)##r## ##r## 25/26 (96.2%) test_z1 (sampletests.test122)##r## ##r## testrunner-ex/sampletests/../sampletestsl.txt##r## ##r## Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False The terminal width is determined using the curses module. To see that, we'll provide a fake curses module: >>> class FakeCurses: ... class error(Exception): ... pass ... def setupterm(self): ... pass ... def tigetnum(self, ignored): ... return 60 >>> old_curses = sys.modules.get('curses') >>> sys.modules['curses'] = FakeCurses() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Running: 1/26 (3.8%) test_x1 (sample1.sampletests.test122.TestA)##r## ##r## 2/26 (7.7%) test_y0 (sample1.sampletests.test122.TestA)##r## ##r## 3/26 (11.5%) test_z0 (...le1.sampletests.test122.TestA)##r## ##r## 4/26 (15.4%) test_x0 (...le1.sampletests.test122.TestB)##r## ##r## 5/26 (19.2%) test_y1 (...le1.sampletests.test122.TestB)##r## ##r## 6/26 (23.1%) test_z0 (...le1.sampletests.test122.TestB)##r## ##r## 7/26 (26.9%) test_1 (...ampletests.test122.TestNotMuch)##r## ##r## 8/26 (30.8%) test_2 (...ampletests.test122.TestNotMuch)##r## ##r## 9/26 (34.6%) test_3 (...ampletests.test122.TestNotMuch)##r## ##r## 10/26 (38.5%) test_x0 (sample1.sampletests.test122)##r## ##r## 11/26 (42.3%) test_y0 (sample1.sampletests.test122)##r## ##r## 12/26 (46.2%) test_z1 (sample1.sampletests.test122)##r## ##r## 13/26 (50.0%) ... e1/sampletests/../../sampletestsl.txt##r## ##r## 14/26 (53.8%) test_x1 (sampletests.test122.TestA)##r## ##r## 15/26 (57.7%) test_y0 (sampletests.test122.TestA)##r## ##r## 16/26 (61.5%) test_z0 (sampletests.test122.TestA)##r## ##r## 17/26 (65.4%) test_x0 (sampletests.test122.TestB)##r## ##r## 18/26 (69.2%) test_y1 (sampletests.test122.TestB)##r## ##r## 19/26 (73.1%) test_z0 (sampletests.test122.TestB)##r## ##r## 20/26 (76.9%) test_1 (sampletests.test122.TestNotMuch)##r## ##r## 21/26 (80.8%) test_2 (sampletests.test122.TestNotMuch)##r## ##r## 22/26 (84.6%) test_3 (sampletests.test122.TestNotMuch)##r## ##r## 23/26 (88.5%) test_x0 (sampletests.test122)##r## ##r## 24/26 (92.3%) test_y0 (sampletests.test122)##r## ##r## 25/26 (96.2%) test_z1 (sampletests.test122)##r## ##r## 26/26 (100.0%) ... r-ex/sampletests/../sampletestsl.txt##r## ##r## Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False >>> sys.modules['curses'] = old_curses If a second or third level of verbosity are added, we get additional information. >>> sys.argv = 'test --layer 122 -pvv -t !txt'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: 1/24 (4.2%) test_x1 (sample1.sampletests.test122.TestA)##r## ##r## 2/24 (8.3%) test_y0 (sample1.sampletests.test122.TestA)##r## ##r## 3/24 (12.5%) test_z0 (sample1.sampletests.test122.TestA)##r## ##r## 4/24 (16.7%) test_x0 (sample1.sampletests.test122.TestB)##r## ##r## 5/24 (20.8%) test_y1 (sample1.sampletests.test122.TestB)##r## ##r## 6/24 (25.0%) test_z0 (sample1.sampletests.test122.TestB)##r## ##r## 7/24 (29.2%) test_1 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 8/24 (33.3%) test_2 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 9/24 (37.5%) test_3 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 10/24 (41.7%) test_x0 (sample1.sampletests.test122)##r## ##r## 11/24 (45.8%) test_y0 (sample1.sampletests.test122)##r## ##r## 12/24 (50.0%) test_z1 (sample1.sampletests.test122)##r## ##r## 13/24 (54.2%) test_x1 (sampletests.test122.TestA)##r## ##r## 14/24 (58.3%) test_y0 (sampletests.test122.TestA)##r## ##r## 15/24 (62.5%) test_z0 (sampletests.test122.TestA)##r## ##r## 16/24 (66.7%) test_x0 (sampletests.test122.TestB)##r## ##r## 17/24 (70.8%) test_y1 (sampletests.test122.TestB)##r## ##r## 18/24 (75.0%) test_z0 (sampletests.test122.TestB)##r## ##r## 19/24 (79.2%) test_1 (sampletests.test122.TestNotMuch)##r## ##r## 20/24 (83.3%) test_2 (sampletests.test122.TestNotMuch)##r## ##r## 21/24 (87.5%) test_3 (sampletests.test122.TestNotMuch)##r## ##r## 22/24 (91.7%) test_x0 (sampletests.test122)##r## ##r## 23/24 (95.8%) test_y0 (sampletests.test122)##r## ##r## 24/24 (100.0%) test_z1 (sampletests.test122)##r## ##r## Ran 24 tests with 0 failures, 0 errors and 0 skipped in 0.006 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False Note that, in this example, we used a test-selection pattern starting with '!' to exclude tests containing the string "txt". >>> sys.argv = 'test --layer 122 -pvvv -t!(txt|NotMuch)'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: 1/18 (5.6%) test_x1 (sample1.sampletests.test122.TestA) (0.000 s)##r## ##r## 2/18 (11.1%) test_y0 (sample1.sampletests.test122.TestA) (0.000 s)##r## ##r## 3/18 (16.7%) test_z0 (sample1.sampletests.test122.TestA) (0.000 s)##r## ##r## 4/18 (22.2%) test_x0 (sample1.sampletests.test122.TestB) (0.000 s)##r## ##r## 5/18 (27.8%) test_y1 (sample1.sampletests.test122.TestB) (0.000 s)##r## ##r## 6/18 (33.3%) test_z0 (sample1.sampletests.test122.TestB) (0.000 s)##r## ##r## 7/18 (38.9%) test_x0 (sample1.sampletests.test122) (0.001 s)##r## ##r## 8/18 (44.4%) test_y0 (sample1.sampletests.test122) (0.001 s)##r## ##r## 9/18 (50.0%) test_z1 (sample1.sampletests.test122) (0.001 s)##r## ##r## 10/18 (55.6%) test_x1 (sampletests.test122.TestA) (0.000 s)##r## ##r## 11/18 (61.1%) test_y0 (sampletests.test122.TestA) (0.000 s)##r## ##r## 12/18 (66.7%) test_z0 (sampletests.test122.TestA) (0.000 s)##r## ##r## 13/18 (72.2%) test_x0 (sampletests.test122.TestB) (0.000 s)##r## ##r## 14/18 (77.8%) test_y1 (sampletests.test122.TestB) (0.000 s)##r## ##r## 15/18 (83.3%) test_z0 (sampletests.test122.TestB) (0.000 s)##r## ##r## 16/18 (88.9%) test_x0 (sampletests.test122) (0.001 s)##r## ##r## 17/18 (94.4%) test_y0 (sampletests.test122) (0.001 s)##r## ##r## 18/18 (100.0%) test_z1 (sampletests.test122) (0.001 s)##r## ##r## Ran 18 tests with 0 failures, 0 errors and 0 skipped in 0.006 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False In this example, we also excluded tests with "NotMuch" in their names. Unfortunately, the time data above doesn't buy us much because, in practice, the line is cleared before there is time to see the times. :/ Autodetecting progress ---------------------- The --auto-progress option will determine if stdout is a terminal, and only enable progress output if so. Let's pretend we have a terminal >>> class Terminal(object): ... def __init__(self, stream): ... self._stream = stream ... def __getattr__(self, attr): ... return getattr(self._stream, attr) ... def isatty(self): ... return True >>> real_stdout = sys.stdout >>> sys.stdout = Terminal(sys.stdout) >>> sys.argv = 'test -u -t test_one.TestNotMuch --auto-progress'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: 1/6 (16.7%)##r## ##r## 2/6 (33.3%)##r## ##r## 3/6 (50.0%)##r## ##r## 4/6 (66.7%)##r## ##r## 5/6 (83.3%)##r## ##r## 6/6 (100.0%)##r## ##r## Ran 6 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Let's stop pretending >>> sys.stdout = real_stdout >>> sys.argv = 'test -u -t test_one.TestNotMuch --auto-progress'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 6 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Disabling progress indication ----------------------------- If -p or --progress have been previously provided on the command line (perhaps by a wrapper script) but you do not desire progress indication, you can switch it off with --no-progress: >>> sys.argv = 'test -u -t test_one.TestNotMuch -p --no-progress'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 6 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Debugging ========= The testrunner module supports post-mortem debugging and debugging using `pdb.set_trace`. Let's look first at using `pdb.set_trace`. To demonstrate this, we'll provide input via helper Input objects: >>> class Input: ... def __init__(self, src): ... self.lines = src.split('\n') ... def readline(self): ... line = self.lines.pop(0) ... print(line) ... return line+'\n' If a test or code called by a test calls pdb.set_trace, then the runner will enter pdb at that point: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> real_stdin = sys.stdin >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace1').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... > testrunner-ex/sample3/sampletests_d.py(27)test_set_trace1() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.001 seconds. ... False Note that, prior to Python 2.4, calling pdb.set_trace caused pdb to break in the pdb.set_trace function. It was necessary to use 'next' or 'up' to get to the application code that called pdb.set_trace. In Python 2.4, pdb.set_trace causes pdb to stop right after the call to pdb.set_trace. You can also do post-mortem debugging, using the --post-mortem (-D) option: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem1 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Error in test test_post_mortem1 (sample3.sampletests_d.TestSomething) Traceback (most recent call last): File "testrunner-ex/sample3/sampletests_d.py", line 34, in test_post_mortem1 raise ValueError ValueError ...ValueError > testrunner-ex/sample3/sampletests_d.py(34)test_post_mortem1() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Note that the test runner exits after post-mortem debugging. In the example above, we debugged an error. Failures are actually converted to errors and can be debugged the same way: >>> sys.stdin = Input('p x\np y\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure1 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Error in test test_post_mortem_failure1 (sample3.sampletests_d.TestSomething) Traceback (most recent call last): File ".../unittest.py", line 252, in debug getattr(self, self.__testMethodName)() File "testrunner-ex/sample3/sampletests_d.py", line 42, in test_post_mortem_failure1 assert x == y AssertionError ...AssertionError > testrunner-ex/sample3/sampletests_d.py(42)test_post_mortem_failure1() -> assert x == y (Pdb) p x 1 (Pdb) p y 2 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Layers that can't be torn down ============================== A layer can have a tearDown method that raises NotImplementedError. If this is the case and there are no remaining tests to run, the test runner will just note that the tear down couldn't be done: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test -ssample2 --tests-pattern sampletests_ntd$'.split() >>> testrunner.run_internal(defaults) Running sample2.sampletests_ntd.Layer tests: Set up sample2.sampletests_ntd.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Tearing down left over layers: Tear down sample2.sampletests_ntd.Layer ... not supported False If the tearDown method raises NotImplementedError and there are remaining layers to run, the test runner will restart itself as a new process, resuming tests where it left off: >>> sys.argv = [testrunner_script, '--tests-pattern', 'sampletests_ntd$'] >>> testrunner.run_internal(defaults) Running sample1.sampletests_ntd.Layer tests: Set up sample1.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running sample2.sampletests_ntd.Layer tests: Tear down sample1.sampletests_ntd.Layer ... not supported Running in a subprocess. Set up sample2.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down sample2.sampletests_ntd.Layer ... not supported Running sample3.sampletests_ntd.Layer tests: Running in a subprocess. Set up sample3.sampletests_ntd.Layer in N.NNN seconds. Error in test test_error1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error1 raise TypeError("Can we see errors") TypeError: Can we see errors Error in test test_error2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error2 raise TypeError("I hope so") TypeError: I hope so Failure in test test_fail1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail1 self.assertEqual(1, 2) AssertionError: 1 != 2 Failure in test test_fail2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail2 self.assertEqual(1, 3) AssertionError: 1 != 3 Ran 6 tests with 2 failures, 2 errors and 0 skipped in N.NNN seconds. Tear down sample3.sampletests_ntd.Layer ... not supported Total: 8 tests, 2 failures, 2 errors and 0 skipped in N.NNN seconds. True in the example above, some of the tests run as a subprocess had errors and failures. They were displayed as usual and the failure and error statistice were updated as usual. Note that debugging doesn't work when running tests in a subprocess: >>> sys.argv = [testrunner_script, '--tests-pattern', 'sampletests_ntd$', ... '-D', ] >>> testrunner.run_internal(defaults) Running sample1.sampletests_ntd.Layer tests: Set up sample1.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running sample2.sampletests_ntd.Layer tests: Tear down sample1.sampletests_ntd.Layer ... not supported Running in a subprocess. Set up sample2.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down sample2.sampletests_ntd.Layer ... not supported Running sample3.sampletests_ntd.Layer tests: Running in a subprocess. Set up sample3.sampletests_ntd.Layer in N.NNN seconds. Error in test test_error1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error1 raise TypeError("Can we see errors") TypeError: Can we see errors ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Error in test test_error2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error2 raise TypeError("I hope so") TypeError: I hope so ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Error in test test_fail1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail1 self.assertEqual(1, 2) AssertionError: 1 != 2 ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Error in test test_fail2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail2 self.assertEqual(1, 3) AssertionError: 1 != 3 ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Ran 6 tests with 0 failures, 4 errors and 0 skipped in N.NNN seconds. Tear down sample3.sampletests_ntd.Layer ... not supported Total: 8 tests, 0 failures, 4 errors and 0 skipped in N.NNN seconds. True Similarly, pdb.set_trace doesn't work when running tests in a layer that is run as a subprocess: >>> sys.argv = [testrunner_script, '--tests-pattern', 'sampletests_ntds'] >>> testrunner.run_internal(defaults) Running sample1.sampletests_ntds.Layer tests: Set up sample1.sampletests_ntds.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Running sample2.sampletests_ntds.Layer tests: Tear down sample1.sampletests_ntds.Layer ... not supported Running in a subprocess. Set up sample2.sampletests_ntds.Layer in 0.000 seconds. --Return-- > testrunner-ex/sample2/sampletests_ntds.py(37)test_something()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(40)test_something2()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(43)test_something3()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(46)test_something4()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(52)f()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > doctest.py(351)set_trace()->None -> Pdb().set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > doctest.py(351)set_trace()->None -> Pdb().set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** Ran 7 tests with 0 failures, 0 errors and 0 skipped in 0.008 seconds. Tear down sample2.sampletests_ntds.Layer ... not supported Total: 8 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False If you want to use pdb from a test in a layer that is run as a subprocess, then rerun the test runner selecting *just* that layer so that it's not run as a subprocess. If a test is run in a subprocess and it generates output on stderr (as stderrtest does), the output is ignored (but it doesn't cause a SubprocessError like it once did). >>> from six import StringIO >>> real_stderr = sys.stderr >>> sys.stderr = StringIO() >>> sys.argv = [testrunner_script, '-s', 'sample2', '--tests-pattern', ... '(sampletests_ntd$|stderrtest)'] >>> testrunner.run_internal(defaults) Running sample2.sampletests_ntd.Layer tests: Set up sample2.sampletests_ntd.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Running sample2.stderrtest.Layer tests: Tear down sample2.sampletests_ntd.Layer ... not supported Running in a subprocess. Set up sample2.stderrtest.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.002 seconds. Tear down sample2.stderrtest.Layer in 0.000 seconds. Total: 2 tests, 0 failures, 0 errors and 0 skipped in 0.197 seconds. False >>> print((sys.stderr.getvalue())) A message on stderr. Please ignore (expected in test output). >>> sys.stderr = real_stderr Egg Support =========== The `ftest` Setup Command ------------------------- The `ftest` class is a proper `distutils` command and `zope.testrunner` exposes it as such via an entry point. >>> import pkg_resources >>> ws = pkg_resources.WorkingSet() >>> eps = dict([(ep.name, ep) ... for ep in ws.iter_entry_points('distutils.commands')]) >>> 'ftest' in eps True >>> eps['ftest'] EntryPoint.parse('ftest = zope.testrunner.eggsupport:ftest') Let's now run this command: >>> import zope.testrunner >>> org_run = zope.testrunner.run >>> def run(args): ... print(' '.join(args)) >>> zope.testrunner.run = run >>> import os, tempfile, shutil >>> tmpdir = tempfile.mkdtemp() >>> srcdir = os.path.join(tmpdir, 'src') >>> os.mkdir(srcdir) >>> import setuptools.dist >>> dist = setuptools.dist.Distribution( ... {'package_dir': {'': srcdir}, ... 'script_name': __file__}) >>> from zope.testrunner.eggsupport import ftest >>> ftest(dist).run() IGNORE_ME --test-path .../src Cleanup: >>> zope.testrunner.run = org_run >>> shutil.rmtree(tmpdir) Skipping Tests with Layers -------------------------- The ``SkipLayers`` scanning test loader can replace the standard test loader, so that any tests that require layers are skipped. This is necessary, since the standard setuptools testing facility does not handle layers. It can be used as follows:: setup( ... setup_requires=['eggtestinfo' # captures testing metadata in EGG-INFO ], tests_require=['zope.testrunner', ], ... test_loader='zope.testrunner.eggsupport:SkipLayers', ... ) Let's now crete some test suites to make sure that all tests with layers are properly skipped. >>> import doctest >>> import unittest >>> all = unittest.TestSuite() >>> class T1(unittest.TestCase): ... def test_t1(self): ... pass >>> class T2(unittest.TestCase): ... layer = 'layer' ... def test_t2(self): ... pass >>> T3 = doctest.DocTestSuite('zope.testrunner.find') >>> T4 = doctest.DocTestSuite('zope.testrunner.options') >>> T4.layer = 'layer' >>> T5 = doctest.DocFileSuite('testrunner.txt', package='zope.testrunner.tests') >>> T6 = doctest.DocFileSuite('testrunner-gc.txt', package='zope.testrunner.tests') >>> T6.layer = 'layer' >>> all = unittest.TestSuite(( ... unittest.makeSuite(T1), unittest.makeSuite(T2), T3, T4, T5, T6, ... )) Let's return those tests from the scan: >>> from setuptools.command.test import ScanningLoader >>> orig_loadTestsFromModule = ScanningLoader.loadTestsFromModule >>> ScanningLoader.loadTestsFromModule = lambda *args: all Now we can retrieve the modules from the layer skipping loader: >>> from zope.testrunner.eggsupport import SkipLayers >>> filtered = SkipLayers().loadTestsFromModule('zope.testrunner') >>> len(filtered._tests) 3 >>> from pprint import pprint >>> pprint(filtered._tests) [<...T1 testMethod=test_t1>, StartUpFailure (zope.testrunner.find), .../zope/testrunner/tests/testrunner.txt] Cleanup: >>> ScanningLoader.loadTestsFromModule = orig_loadTestsFromModule When the distribution specified a ``test_suite``, another method is used to load the tests. >>> orig_loadTestsFromNames = ScanningLoader.loadTestsFromNames >>> ScanningLoader.loadTestsFromNames = lambda *args: all Now we can retrieve the modules from the layer skipping loader: >>> from zope.testrunner.eggsupport import SkipLayers >>> filtered = SkipLayers().loadTestsFromNames( ... 'zope.testrunner.tests.test_suite', 'zope.testrunner') >>> len(filtered._tests) 3 >>> from pprint import pprint >>> pprint(filtered._tests) [<...T1 testMethod=test_t1>, StartUpFailure (zope.testrunner.find), .../zope/testrunner/tests/testrunner.txt] Cleanup: >>> ScanningLoader.loadTestsFromNames = orig_loadTestsFromNames Code Coverage ============= If the --coverage option is used, test coverage reports will be generated. The directory name given as the parameter will be used to hold the reports. >>> import os.path, sys, tempfile >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> tempdir = tempfile.mkdtemp(prefix='zope.testrunner-') >>> coverage_dir = os.path.join(tempdir, 'coverage_dir') >>> sys.argv = ['test', '--coverage', coverage_dir] >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in 0.000 seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in 0.687 seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in 0.000 seconds. Set up samplelayers.Layer112 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.140 seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. lines cov% module (path) ... 48 100% sampletests.test1 (testrunner-ex/sampletests/test1.py) 74 100% sampletests.test11 (testrunner-ex/sampletests/test11.py) 74 100% sampletests.test111 (testrunner-ex/sampletests/test111.py) 76 100% sampletests.test112 (testrunner-ex/sampletests/test112.py) 74 100% sampletests.test12 (testrunner-ex/sampletests/test12.py) 74 100% sampletests.test121 (testrunner-ex/sampletests/test121.py) 74 100% sampletests.test122 (testrunner-ex/sampletests/test122.py) 48 100% sampletests.test_one (testrunner-ex/sampletests/test_one.py) 112 95% sampletestsf (testrunner-ex/sampletestsf.py) Total: 321 tests, 0 failures, 0 errors and 0 skipped in 0.630 seconds. False The directory specified with the --coverage option will have been created and will hold the coverage reports. >>> os.path.exists(coverage_dir) True >>> os.listdir(coverage_dir) [...] (We should clean up after ourselves.) >>> import shutil >>> shutil.rmtree(tempdir) Ignoring Tests -------------- The ``trace`` module supports ignoring directories and modules based the test selection. Only directories selected for testing should report coverage. The test runner provides a custom implementation of the relevant API. The ``TestIgnore`` class, the class managing the ignoring, is initialized by passing the command line options. It uses the options to determine the directories that should be covered. >>> class FauxOptions(object): ... package = None ... test_path = [('/myproject/src/blah/foo', ''), ... ('/myproject/src/blah/bar', '')] >>> from zope.testrunner import coverage >>> from zope.testrunner.find import test_dirs >>> ignore = coverage.TestIgnore(test_dirs(FauxOptions(), {})) >>> ignore._test_dirs ['/myproject/src/blah/foo/', '/myproject/src/blah/bar/'] We can now ask whether a particular module should be ignored: >>> ignore.names('/myproject/src/blah/foo/baz.py', 'baz') False >>> ignore.names('/myproject/src/blah/bar/mine.py', 'mine') False >>> ignore.names('/myproject/src/blah/foo/__init__.py', 'foo') False >>> ignore.names('/myproject/src/blah/hello.py', 'hello') True When running the test runner, modules are sometimes created from text strings. Those should *always* be ignored: >>> ignore.names('/myproject/src/blah/hello.txt', '') True To make this check fast, the class implements a cache. In an early implementation, the result was cached by the module name, which was a problem, since a lot of modules carry the same name (not the Python dotted name here!). So just because a module has the same name in an ignored and tested directory, does not mean it is always ignored: >>> ignore.names('/myproject/src/blah/module.py', 'module') True >>> ignore.names('/myproject/src/blah/foo/module.py', 'module') False Profiling ========= The testrunner supports hotshot and cProfile profilers. Hotshot profiler support does not work with python2.6 >>> import os.path, sys, tempfile >>> profiler = '--profile=hotshot' >>> if sys.hexversion >= 0x02060000: ... profiler = '--profile=cProfile' The testrunner includes the ability to profile the test execution with hotshot via the --profile option, if it a python <= 2.6 >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> sys.path.append(directory_with_tests) >>> tempdir = tempfile.mkdtemp(prefix='zope.testrunner-') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... '--profile-directory', tempdir, ... ] >>> sys.argv = [testrunner_script, profiler] When the tests are run, we get profiling output. >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: ... Running samplelayers.Layer1 tests: ... Running samplelayers.Layer11 tests: ... ncalls tottime percall cumtime percall filename:lineno(function) ... Total: ... tests, 0 failures, 0 errors and 0 skipped in ... seconds. False Profiling also works across layers. >>> sys.argv = [testrunner_script, '-ssample2', profiler, ... '--tests-pattern', 'sampletests_ntd'] >>> testrunner.run_internal(defaults) Running... Tear down ... not supported... ncalls tottime percall cumtime percall filename:lineno(function)... The testrunner creates temnporary files containing hotshot profiler data: >>> os.listdir(tempdir) ['tests_profile.cZj2jt.prof', 'tests_profile.yHD-so.prof'] It deletes these when rerun. We'll delete these ourselves: >>> import shutil >>> shutil.rmtree(tempdir) Running Without Source Code =========================== The ``--usecompiled`` option allows running tests in a tree without .py source code, provided compiled .pyc or .pyo files exist (without ``--usecompiled``, .py files are necessary). We have a very simple directory tree, under ``usecompiled/``, to test this. Because we're going to delete its .py files, we want to work in a copy of that: >>> import os.path, shutil, sys, tempfile >>> directory_with_tests = tempfile.mkdtemp() >>> NEWNAME = "unlikely_package_name" >>> src = os.path.join(this_directory, 'testrunner-ex', 'usecompiled') >>> os.path.isdir(src) True >>> dst = os.path.join(directory_with_tests, NEWNAME) >>> os.path.isdir(dst) False Have to use our own copying code, to avoid copying read-only SVN files that can't be deleted later. >>> n = len(src) + 1 >>> for root, dirs, files in os.walk(src): ... dirs[:] = [d for d in dirs if d == "package"] # prune cruft ... os.mkdir(os.path.join(dst, root[n:])) ... for f in files: ... _ = shutil.copy(os.path.join(root, f), ... os.path.join(dst, root[n:], f)) Now run the tests in the copy: >>> from zope import testrunner >>> mydefaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^compiletest$', ... '--package', NEWNAME, ... '-vv', ... ] >>> sys.argv = ['test'] >>> testrunner.run_internal(mydefaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test1 (unlikely_package_name.compiletest.Test) test2 (unlikely_package_name.compiletest.Test) test1 (unlikely_package_name.package.compiletest.Test) test2 (unlikely_package_name.package.compiletest.Test) Ran 4 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False If we delete the source files, it's normally a disaster: the test runner doesn't believe any test files, or even packages, exist. Note that we pass ``--keepbytecode`` this time, because otherwise the test runner would delete the compiled Python files too: >>> for root, dirs, files in os.walk(dst): ... for f in files: ... if f.endswith(".py"): ... os.remove(os.path.join(root, f)) >>> testrunner.run_internal(mydefaults, ["test", "--keepbytecode"]) Running tests at level 1 Total: 0 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Finally, passing ``--usecompiled`` asks the test runner to treat .pyc and .pyo files as adequate replacements for .py files. Note that the output is the same as when running with .py source above. The absence of "removing stale bytecode ..." messages shows that ``--usecompiled`` also implies ``--keepbytecode``: >>> if sys.version_info >= (3, 2): ... # PEP-3147: pyc files in __pycache__ directories cannot be ... # imported; legacy source-less imports need to use the legacy ... # layout ... for root, dirs, files in os.walk(dst): ... for f in files: ... if f.endswith((".pyc", ".pyo")): ... # "root/f" is "dirname/__pycache__/name.magic.ext" ... dirname = os.path.dirname(os.path.abspath(root)) ... namewmagic, ext = os.path.splitext(os.path.basename(f)) ... newname = os.path.splitext(namewmagic)[0] + ext ... os.rename(os.path.join(root, f), ... os.path.join(dirname, newname)) >>> testrunner.run_internal(mydefaults, ["test", "--usecompiled"]) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test1 (unlikely_package_name.compiletest.Test) test2 (unlikely_package_name.compiletest.Test) test1 (unlikely_package_name.package.compiletest.Test) test2 (unlikely_package_name.package.compiletest.Test) Ran 4 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Remove the temporary directory: >>> shutil.rmtree(directory_with_tests) Repeating Tests =============== The --repeat option can be used to repeat tests some number of times. Repeating tests is useful to help make sure that tests clean up after themselves. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 112 --layer UnitTests --repeat 3'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Iteration 1 Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Iteration 2 Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Iteration 3 Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer112 in 0.000 seconds. Iteration 1 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.010 seconds. Iteration 2 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.010 seconds. Iteration 3 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.010 seconds. Tearing down left over layers: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 182 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False The tests are repeated by layer. Layers are set up and torn down only once. Garbage Collection Control ========================== When having problems that seem to be caused my memory-management errors, it can be helpful to adjust Python's cyclic garbage collector or to get garbage colection statistics. The --gc option can be used for this purpose. If you think you are getting a test failure due to a garbage collection problem, you can try disabling garbage collection by using the --gc option with a value of zero. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = ['--path', directory_with_tests] >>> from zope import testrunner >>> sys.argv = 'test --tests-pattern ^gc0$ --gc 0 -vv'.split() >>> _ = testrunner.run_internal(defaults) Cyclic garbage collection is disabled. Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: make_sure_gc_is_disabled (gc0) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Alternatively, if you think you are having a garbage collection related problem, you can cause garbage collection to happen more often by providing a low threshold: >>> sys.argv = 'test --tests-pattern ^gc1$ --gc 1 -vv'.split() >>> _ = testrunner.run_internal(defaults) Cyclic garbage collection threshold set to: (1,) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: make_sure_gc_threshold_is_one (gc1) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. You can specify up to 3 --gc options to set each of the 3 gc threshold values: >>> sys.argv = ('test --tests-pattern ^gcset$ --gc 701 --gc 11 --gc 9 -vv' ... .split()) >>> _ = testrunner.run_internal(defaults) Cyclic garbage collection threshold set to: (701, 11, 9) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: make_sure_gc_threshold_is_701_11_9 (gcset) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Specifying more than 3 --gc options is not allowed: >>> from six import StringIO >>> out = StringIO() >>> stdout = sys.stdout >>> sys.stdout = out >>> sys.argv = ('test --tests-pattern ^gcset$ --gc 701 --gc 42 --gc 11 --gc 9 -vv' ... .split()) >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 >>> sys.stdout = stdout >>> print(out.getvalue()) Too many --gc options Garbage Collection Statistics ----------------------------- You can enable gc debugging statistics using the --gc-options (-G) option. You should provide names of one or more of the flags described in the library documentation for the gc module. The output statistics are written to standard error. >>> from six import StringIO >>> err = StringIO() >>> stderr = sys.stderr >>> sys.stderr = err >>> sys.argv = ('test --tests-pattern ^gcstats$ -G DEBUG_STATS' ... ' -G DEBUG_COLLECTABLE -vv' ... .split()) >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: generate_some_gc_statistics (gcstats) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. >>> sys.stderr = stderr >>> print(err.getvalue()) # doctest: +ELLIPSIS gc: collecting generation ... Debugging Memory Leaks ====================== The --report-refcounts (-r) option can be used with the --repeat (-N) option to detect and diagnose memory leaks. To use this option, you must configure Python with the --with-pydebug option. (On Unix, pass this option to configure and then build Python.) >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> sys.argv = 'test --layer Layer11$ --layer Layer12$ -N4 -r'.split() >>> _ = testrunner.run_internal(defaults) Running samplelayers.Layer11 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Iteration 1 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.013 seconds. Iteration 2 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100401 change=0 Iteration 3 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100401 change=0 Iteration 4 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.013 seconds. sys refcount=100401 change=0 Running samplelayers.Layer12 tests: Tear down samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Iteration 1 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.013 seconds. Iteration 2 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100411 change=0 Iteration 3 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100411 change=0 Iteration 4 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100411 change=0 Tearing down left over layers: Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. Total: 68 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. Each layer is repeated the requested number of times. For each iteration after the first, the system refcount and change in system refcount is shown. The system refcount is the total of all refcount in the system. When a refcount on any object is changed, the system refcount is changed by the same amount. Tests that don't leak show zero changes in systen refcount. Let's look at an example test that leaks: >>> sys.argv = 'test --tests-pattern leak -N4 -r'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests:... Iteration 1 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Iteration 2 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sys refcount=92506 change=12 Iteration 3 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sys refcount=92513 change=12 Iteration 4 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sys refcount=92520 change=12 Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Here we see that the system refcount is increating. If we specify a verbosity greater than one, we can get details broken out by object type (or class): >>> sys.argv = 'test --tests-pattern leak -N5 -r -v'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests:... Iteration 1 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Iteration 2 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95832 sys refcount=105668 change=16 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 int 2 2 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 3 ------------------------------------------------------- ----- ---- total 8 16 Iteration 3 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95844 sys refcount=105680 change=12 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 int -1 0 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 1 ------------------------------------------------------- ----- ---- total 5 12 Iteration 4 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95856 sys refcount=105692 change=12 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 1 ------------------------------------------------------- ----- ---- total 6 12 Iteration 5 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95868 sys refcount=105704 change=12 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 1 ------------------------------------------------------- ----- ---- total 6 12 Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. It is instructive to analyze the results in some detail. The test being run was designed to intentionally leak: class ClassicLeakable: def __init__(self): self.x = 'x' class Leakable(object): def __init__(self): self.x = 'x' leaked = [] class TestSomething(unittest.TestCase): def testleak(self): leaked.append((ClassicLeakable(), Leakable(), time.time())) Let's go through this by type. float, leak.ClassicLeakable, leak.Leakable, and tuple We leak one of these every time. This is to be expected because we are adding one of these to the list every time. str We don't leak any instances, but we leak 4 references. These are due to the instance attributes avd values. dict We leak 2 of these, one for each ClassicLeakable and Leakable instance. classobj We increase the number of classobj instance references by one each time because each ClassicLeakable instance has a reference to its class. This instances increases the references in it's class, which increases the total number of references to classic classes (clasobj instances). type For most interations, we increase the number of type references by one for the same reason we increase the number of clasobj references by one. The increase of the number of type references by 3 in the second iteration is puzzling, but illustrates that this sort of data is often puzzling. int The change in the number of int instances and references in this example is a side effect of the statistics being gathered. Lots of integers are created to keep the memory statistics used here. The summary statistics include the sum of the detail refcounts. (Note that this sum is less than the system refcount. This is because the detailed analysis doesn't inspect every object. Not all objects in the system are returned by sys.getobjects.) Knitting in extra package directories ===================================== Python packages have __path__ variables that can be manipulated to add extra directories cntaining software used in the packages. The testrunner needs to be given extra information about this sort of situation. Let's look at an example. The testrunner-ex-knit-lib directory is a directory that we want to add to the Python path, but that we don't want to search for tests. It has a sample4 package and a products subpackage. The products subpackage adds the testrunner-ex-knit-products to it's __path__. We want to run tests from the testrunner-ex-knit-products directory. When we import these tests, we need to import them from the sample4.products package. We can't use the --path option to name testrunner-ex-knit-products. It isn't enough to add the containing directory to the test path because then we wouldn't be able to determine the package name properly. We might be able to use the --package option to run the tests from the sample4/products package, but we want to run tests in testrunner-ex that aren't in this package. We can use the --package-path option in this case. The --package-path option is like the --test-path option in that it defines a path to be searched for tests without affecting the python path. It differs in that it supplied a package name that is added a profex when importing any modules found. The --package-path option takes *two* arguments, a package name and file path. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> sys.path.append(os.path.join(this_directory, 'testrunner-ex-pp-lib')) >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... '--package-path', ... os.path.join(this_directory, 'testrunner-ex-pp-products'), ... 'sample4.products', ... ] >>> from zope import testrunner >>> sys.argv = 'test --layer Layer111 -vv'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test111.TestA) test_y0 (sample1.sampletests.test111.TestA) ... test_y0 (sampletests.test111) test_z1 (sampletests.test111) testrunner-ex/sampletests/../sampletestsl.txt test_extra_test_in_products (sample4.products.sampletests.Test) test_another_test_in_products (sample4.products.more.sampletests.Test) Ran 28 tests with 0 failures, 0 errors and 0 skipped in 0.008 seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. In the example, the last test, test_extra_test_in_products, came from the products directory. As usual, we can select the knit-in packages or individual packages within knit-in packages: >>> sys.argv = 'test --package sample4.products -vv'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Running: test_extra_test_in_products (sample4.products.sampletests.Test) test_another_test_in_products (sample4.products.more.sampletests.Test) Ran 2 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. >>> sys.argv = 'test --package sample4.products.more -vv'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Running: test_another_test_in_products (sample4.products.more.sampletests.Test) Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. testrunner Edge Cases ===================== This document has some edge-case examples to test various aspects of the test runner. Separating Python path and test directories ------------------------------------------- The --path option defines a directory to be searched for tests *and* a directory to be added to Python's search path. The --test-path option can be used when you want to set a test search path without also affecting the Python path: >>> import os, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--test-path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) ... # doctest: +ELLIPSIS Test-module import failures: Module: sampletestsf Traceback (most recent call last): ImportError: No module named sampletestsf ... >>> sys.path.append(directory_with_tests) >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layer1 in 0.000 seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. ... Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Bug #251759: The test runner's protection against descending into non-package directories was ineffective, e.g. picking up tests from eggs that were stored close by: >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex-251759') >>> defaults = [ ... '--test-path', directory_with_tests, ... ] >>> testrunner.run_internal(defaults) Total: 0 tests, 0 failures, 0 errors and 0 skipped in 0.000 seconds. False Debugging Edge Cases -------------------- >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--test-path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> class Input: ... def __init__(self, src): ... self.lines = src.split('\n') ... def readline(self): ... line = self.lines.pop(0) ... print(line) ... return line+'\n' >>> real_stdin = sys.stdin Using pdb.set_trace in a function called by an ordinary test: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace2').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests:... > testrunner-ex/sample3/sampletests_d.py(47)f() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.001 seconds. ... False Using pdb.set_trace in a function called by a doctest in a doc string: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace4').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > testrunner-ex/sample3/sampletests_d.py(NNN)f() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Using pdb in a docstring-based doctest >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace3').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > (3)?() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Using pdb.set_trace in a doc file: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace5').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > (3)?() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Using pdb.set_trace in a function called by a doctest in a doc file: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace6').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > testrunner-ex/sample3/sampletests_d.py(NNN)f() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging function called from ordinary test: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem2 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Error in test test_post_mortem2 (sample3.sampletests_d.TestSomething) Traceback (most recent call last): File "testrunner-ex/sample3/sampletests_d.py", line 37, in test_post_mortem2 g() File "testrunner-ex/sample3/sampletests_d.py", line 46, in g raise ValueError ValueError ...ValueError > testrunner-ex/sample3/sampletests_d.py(46)g() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging docstring-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem3 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error in test post_mortem3 (sample3.sampletests_d) Traceback (most recent call last): ...UnexpectedException: testrunner-ex/sample3/sampletests_d.py:NNN (2 examples)> ...ValueError > (1)?() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging function called from docstring-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem4 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error in test post_mortem4 (sample3.sampletests_d) Traceback (most recent call last): ...UnexpectedException: testrunner-ex/sample3/sampletests_d.py:NNN (1 example)> ...ValueError > testrunner-ex/sample3/sampletests_d.py(NNN)g() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging file-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem5 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error testrunner-ex/sample3/post_mortem5.txt Traceback (most recent call last): ...UnexpectedException: testrunner-ex/sample3/post_mortem5.txt:0 (2 examples)> ...ValueError > (1)?() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging function called from file-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem6 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error testrunner-ex/sample3/post_mortem6.txt Traceback (most recent call last): File ".../zope/testing/doctest/__init__.py", Line NNN, in debug runner.run(self._dt_test, clear_globs=False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run r = DocTestRunner.run(self, test, compileflags, out, False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run return self.__run(test, compileflags, out) File ".../zope/testing/doctest/__init__.py", Line NNN, in __run exc_info) File ".../zope/testing/doctest/__init__.py", Line NNN, in report_unexpected_exception raise UnexpectedException(test, example, exc_info) ...UnexpectedException: testrunner-ex/sample3/post_mortem6.txt:0 (2 examples)> ...ValueError > testrunner-ex/sample3/sampletests_d.py(NNN)g() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging of a docstring doctest failure: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure2 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Error in test post_mortem_failure2 (sample3.sampletests_d) File "testrunner-ex/sample3/sampletests_d.py", line 81, in sample3.sampletests_d.post_mortem_failure2 x Want: 2 Got: 1 > testrunner-ex/sample3/sampletests_d.py(81)_() ...ValueError: Expected and actual output are different > (1)...() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging of a docfile doctest failure: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure.txt -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Error in test /home/jim/z3/zope.testrunner/src/zope/testing/testrunner-ex/sample3/post_mortem_failure.txt File "testrunner-ex/sample3/post_mortem_failure.txt", line 2, in post_mortem_failure.txt x Want: 2 Got: 1 > testrunner-ex/sample3/post_mortem_failure.txt(2)_() ...ValueError: Expected and actual output are different > (1)...() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging with triple verbosity >>> sys.argv = 'test --layer samplelayers.Layer1$ -vvv -D'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in 0.000 seconds. Running: test_x1 (sampletestsf.TestA1) (0.000 s) test_y0 (sampletestsf.TestA1) (0.000 s) test_z0 (sampletestsf.TestA1) (0.000 s) test_x0 (sampletestsf.TestB1) (0.000 s) test_y1 (sampletestsf.TestB1) (0.000 s) test_z0 (sampletestsf.TestB1) (0.000 s) test_1 (sampletestsf.TestNotMuch1) (0.000 s) test_2 (sampletestsf.TestNotMuch1) (0.000 s) test_3 (sampletestsf.TestNotMuch1) (0.000 s) Ran 9 tests with 0 failures, 0 errors and 0 skipped in 0.001 seconds. Tearing down left over layers: Tear down samplelayers.Layer1 in 0.000 seconds. False Test Suites with None for suites or tests ----------------------------------------- >>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_none_suite$', ... ] >>> testrunner.run_internal(defaults) Test-module import failures: Module: sample1.sampletests_none_suite Traceback (most recent call last): TypeError: Invalid test_suite, None, in sample1.sampletests_none_suite Test-modules with import problems: sample1.sampletests_none_suite Total: 0 tests, 0 failures, 1 errors and 0 skipped in N.NNN seconds. True >>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_none_test$', ... ] >>> testrunner.run_internal(defaults) Test-module import failures: Module: sample1.sampletests_none_test Traceback (most recent call last): TypeError: ... Test-modules with import problems: sample1.sampletests_none_test Total: 0 tests, 0 failures, 1 errors and 0 skipped in N.NNN seconds. True You must use --repeat with --report-refcounts --------------------------------------------- It is an error to specify --report-refcounts (-r) without specifying a repeat count greater than 1 >>> sys.argv = 'test -r'.split() >>> testrunner.run_internal(defaults) You must use the --repeat (-N) option to specify a repeat count greater than 1 when using the --report_refcounts (-r) option. True >>> sys.argv = 'test -r -N1'.split() >>> testrunner.run_internal(defaults) You must use the --repeat (-N) option to specify a repeat count greater than 1 when using the --report_refcounts (-r) option. True Selection --------- Several tests can be excluded using the '!' notation: >>> sys.argv = 'test -u -vv -ssample1.sample13 -t!test_x -t!test_y'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_z0 (sample1.sample13.sampletests.TestA) test_z0 (sample1.sample13.sampletests.TestB) test_1 (sample1.sample13.sampletests.TestNotMuch) test_2 (sample1.sample13.sampletests.TestNotMuch) test_3 (sample1.sample13.sampletests.TestNotMuch) test_z1 (sample1.sample13.sampletests) testrunner-ex/sample1/sample13/../../sampletests.txt Ran 7 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Errors and Failures =================== Let's look at tests that have errors and failures, first we need to make a temporary copy of the entire testing directory (except .svn files which may be read only): >>> import os.path, sys, tempfile, shutil >>> tmpdir = tempfile.mkdtemp() >>> directory_with_tests = os.path.join(tmpdir, 'testrunner-ex') >>> source = os.path.join(this_directory, 'testrunner-ex') >>> n = len(source) + 1 >>> for root, dirs, files in os.walk(source): ... dirs[:] = [d for d in dirs if d != ".svn"] # prune cruft ... os.mkdir(os.path.join(directory_with_tests, root[n:])) ... for f in files: ... _ = shutil.copy(os.path.join(root, f), ... os.path.join(directory_with_tests, root[n:], f)) >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --tests-pattern ^sampletests(f|_e|_f)?$ '.split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Failure in test eek (sample2.sampletests_e) Failed doctest test for sample2.sampletests_e.eek File "testrunner-ex/sample2/sampletests_e.py", line 28, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_e.py", line 30, in sample2.sampletests_e.eek Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined Error in test test3 (sample2.sampletests_e.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_e.py", line 43, in test3 f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined Failure in test testrunner-ex/sample2/e.txt Failed doctest test for e.txt File "testrunner-ex/sample2/e.txt", line 0 ---------------------------------------------------------------------- File "testrunner-ex/sample2/e.txt", line 4, in e.txt Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "", line 2, in f return x NameError: global name 'x' is not defined Failure in test test (sample2.sampletests_f.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_f.py", line 21, in test self.assertEqual(1,0) File "/usr/local/python/2.3/lib/python2.3/unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: 1 != 0 Ran 164 tests with 3 failures, 1 errors and 0 skipped in N.NNN seconds. ... Total: 329 tests, 3 failures, 1 errors and 0 skipped in N.NNN seconds. True We see that we get an error report and a traceback for the failing test. In addition, the test runner returned True, indicating that there was an error. If we ask for verbosity, the dotted output will be interrupted, and there'll be a summary of the errors at the end of the test: >>> sys.argv = 'test --tests-pattern ^sampletests(f|_e|_f)?$ -uv'.split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: ... Running: ................................................................................................. Failure in test eek (sample2.sampletests_e) Failed doctest test for sample2.sampletests_e.eek File "testrunner-ex/sample2/sampletests_e.py", line 28, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_e.py", line 30, in sample2.sampletests_e.eek Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined ... Error in test test3 (sample2.sampletests_e.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_e.py", line 43, in test3 f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined ... Failure in test testrunner-ex/sample2/e.txt Failed doctest test for e.txt File "testrunner-ex/sample2/e.txt", line 0 ---------------------------------------------------------------------- File "testrunner-ex/sample2/e.txt", line 4, in e.txt Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "", line 2, in f return x NameError: global name 'x' is not defined . Failure in test test (sample2.sampletests_f.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_f.py", line 21, in test self.assertEqual(1,0) File ".../unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: 1 != 0 ................................................................................................ Ran 164 tests with 3 failures, 1 errors and 0 skipped in 0.040 seconds. ... Tests with errors: test3 (sample2.sampletests_e.Test) Tests with failures: eek (sample2.sampletests_e) testrunner-ex/sample2/e.txt test (sample2.sampletests_f.Test) True Similarly for progress output, the progress ticker will be interrupted: >>> sys.argv = ('test --tests-pattern ^sampletests(f|_e|_f)?$ -u -ssample2' ... ' -p').split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: 1/47 (2.1%) Failure in test eek (sample2.sampletests_e) Failed doctest test for sample2.sampletests_e.eek File "testrunner-ex/sample2/sampletests_e.py", line 28, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_e.py", line 30, in sample2.sampletests_e.eek Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined 2/47 (4.3%)\r \r 3/47 (6.4%)\r \r 4/47 (8.5%) Error in test test3 (sample2.sampletests_e.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_e.py", line 43, in test3 f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined 5/47 (10.6%)\r \r 6/47 (12.8%)\r \r 7/47 (14.9%) Failure in test testrunner-ex/sample2/e.txt Failed doctest test for e.txt File "testrunner-ex/sample2/e.txt", line 0 ---------------------------------------------------------------------- File "testrunner-ex/sample2/e.txt", line 4, in e.txt Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "", line 2, in f return x NameError: global name 'x' is not defined 8/47 (17.0%) Failure in test test (sample2.sampletests_f.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_f.py", line 21, in test self.assertEqual(1,0) File ".../unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: 1 != 0 9/47 (19.1%)\r \r 10/47 (21.3%)\r \r 11/47 (23.4%)\r \r 12/47 (25.5%)\r \r 13/47 (27.7%)\r \r 14/47 (29.8%)\r \r 15/47 (31.9%)\r \r 16/47 (34.0%)\r \r 17/47 (36.2%)\r \r 18/47 (38.3%)\r \r 19/47 (40.4%)\r \r 20/47 (42.6%)\r \r 21/47 (44.7%)\r \r 22/47 (46.8%)\r \r 23/47 (48.9%)\r \r 24/47 (51.1%)\r \r 25/47 (53.2%)\r \r 26/47 (55.3%)\r \r 27/47 (57.4%)\r \r 28/47 (59.6%)\r \r 29/47 (61.7%)\r \r 30/47 (63.8%)\r \r 31/47 (66.0%)\r \r 32/47 (68.1%)\r \r 33/47 (70.2%)\r \r 34/47 (72.3%)\r \r 35/47 (74.5%)\r \r 36/47 (76.6%)\r \r 37/47 (78.7%)\r \r 38/47 (80.9%)\r \r 39/47 (83.0%)\r \r 40/47 (85.1%)\r \r 41/47 (87.2%)\r \r 42/47 (89.4%)\r \r 43/47 (91.5%)\r \r 44/47 (93.6%)\r \r 45/47 (95.7%)\r \r 46/47 (97.9%)\r \r 47/47 (100.0%)\r \r Ran 47 tests with 3 failures, 1 errors and 0 skipped in 0.054 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True If you also want a summary of errors at the end, ask for verbosity as well as progress output. Suppressing multiple doctest errors ----------------------------------- Often, when a doctest example fails, the failure will cause later examples in the same test to fail. Each failure is reported: >>> sys.argv = 'test --tests-pattern ^sampletests_1$'.split() >>> testrunner.run_internal(defaults) # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 21, in sample2.sampletests_1.eek Failed example: x Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x NameError: name 'x' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 24, in sample2.sampletests_1.eek Failed example: z = x + 1 Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? z = x + 1 NameError: name 'x' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True This can be a bit confusing, especially when there are enough tests that they scroll off a screen. Often you just want to see the first failure. This can be accomplished with the -1 option (for "just show me the first failed example in a doctest" :) >>> sys.argv = 'test --tests-pattern ^sampletests_1$ -1'.split() >>> testrunner.run_internal(defaults) # doctest: Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.001 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True The --hide-secondary-failures option is an alias for -1: >>> sys.argv = ( ... 'test --tests-pattern ^sampletests_1$' ... ' --hide-secondary-failures' ... ).split() >>> testrunner.run_internal(defaults) # doctest: Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.001 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True The --show-secondary-failures option counters -1 (or it's alias), causing the second and subsequent errors to be shown. This is useful if -1 is provided by a test script by inserting it ahead of command-line options in sys.argv. >>> sys.argv = ( ... 'test --tests-pattern ^sampletests_1$' ... ' --hide-secondary-failures --show-secondary-failures' ... ).split() >>> testrunner.run_internal(defaults) # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 21, in sample2.sampletests_1.eek Failed example: x Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x NameError: name 'x' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 24, in sample2.sampletests_1.eek Failed example: z = x + 1 Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? z = x + 1 NameError: name 'x' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True Getting diff output for doctest failures ---------------------------------------- If a doctest has large expected and actual output, it can be hard to see differences when expected and actual output differ. The --ndiff, --udiff, and --cdiff options can be used to get diff output of various kinds. >>> sys.argv = 'test --tests-pattern ^pledge$'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Expected: I give my pledge, as an earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. It's soils, minerals, forests, waters, and wildlife. Got: I give my pledge, as and earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. It's soils, minerals, forests, waters, and wildlife. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Here, the actual output uses the word "and" rather than the word "an", but it's a bit hard to pick this out. We can use the various diff outputs to see this better. We could modify the test to ask for diff output, but it's easier to use one of the diff options. The --ndiff option requests a diff using Python's ndiff utility. This is the only method that marks differences within lines as well as across lines. For example, if a line of expected output contains digit 1 where actual output contains letter l, a line is inserted with a caret marking the mismatching column positions. >>> sys.argv = 'test --tests-pattern ^pledge$ --ndiff'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Differences (ndiff with -expected +actual): - I give my pledge, as an earthling, + I give my pledge, as and earthling, ? + to save, and faithfully, to defend from waste, the natural resources of my planet. It's soils, minerals, forests, waters, and wildlife. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.003 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. The -udiff option requests a standard "unified" diff: >>> sys.argv = 'test --tests-pattern ^pledge$ --udiff'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Differences (unified diff with -expected +actual): @@ -1,3 +1,3 @@ -I give my pledge, as an earthling, +I give my pledge, as and earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. The -cdiff option requests a standard "context" diff: >>> sys.argv = 'test --tests-pattern ^pledge$ --cdiff'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Differences (context diff with expected followed by actual): *************** *** 1,3 **** ! I give my pledge, as an earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. --- 1,3 ---- ! I give my pledge, as and earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Specifying more than one diff option at once causes an error: >>> sys.argv = 'test --tests-pattern ^pledge$ --cdiff --udiff'.split() >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 >>> sys.argv = 'test --tests-pattern ^pledge$ --cdiff --ndiff'.split() >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 >>> sys.argv = 'test --tests-pattern ^pledge$ --udiff --ndiff'.split() >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 Testing-Module Import Errors ---------------------------- If there are errors when importing a test module, these errors are reported. In order to illustrate a module with a syntax error, we create one now: this module used to be checked in to the project, but then it was included in distributions of projects using zope.testrunner too, and distutils complained about the syntax error when it compiled Python files during installation of such projects. So first we create a module with bad syntax: >>> badsyntax_path = os.path.join(directory_with_tests, ... "sample2", "sampletests_i.py") >>> f = open(badsyntax_path, "w") >>> print("importx unittest", file=f) # syntax error >>> f.close() Then run the tests: >>> sys.argv = ('test --tests-pattern ^sampletests(f|_i)?$ --layer 1 ' ... ).split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE Test-module import failures: Module: sample2.sampletests_i Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_i.py", line 1 importx unittest ^ SyntaxError: invalid syntax Module: sample2.sample21.sampletests_i Traceback (most recent call last): File "testrunner-ex/sample2/sample21/sampletests_i.py", line 15, in ? import zope.testrunner.huh ImportError: No module named huh Module: sample2.sample23.sampletests_i Traceback (most recent call last): File "testrunner-ex/sample2/sample23/sampletests_i.py", line 18, in ? class Test(unittest.TestCase): File "testrunner-ex/sample2/sample23/sampletests_i.py", line 23, in Test raise TypeError('eek') TypeError: eek Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in 0.000 seconds. Ran 9 tests with 0 failures, 3 errors and 0 skipped in 0.000 seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in 0.000 seconds. Set up samplelayers.Layer112 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.006 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. Test-modules with import problems: sample2.sampletests_i sample2.sample21.sampletests_i sample2.sample23.sampletests_i Total: 165 tests, 0 failures, 3 errors and 0 skipped in N.NNN seconds. True Reporting Errors to Calling Processes ------------------------------------- The testrunner returns the error status, indicating that the tests failed. This can be useful for an invoking process that wants to monitor the result of a test run. This is applied when invoking the testrunner using the ``run()`` function instead of ``run_internal()``: >>> sys.argv = ( ... 'test --tests-pattern ^sampletests_1$'.split()) >>> try: ... testrunner.run(defaults) ... except SystemExit as e: ... print('exited with code', e.code) ... else: ... print('sys.exit was not called') ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. ... Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. exited with code 1 Passing tests exit with code 0 according to UNIX practices: >>> sys.argv = ( ... 'test --tests-pattern ^sampletests$'.split()) >>> try: ... testrunner.run(defaults) ... except SystemExit as e2: ... print('exited with code', e2.code) ... else: ... print('sys.exit was not called') ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Total: 286 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. exited with code 0 And remove the temporary directory: >>> shutil.rmtree(tmpdir) Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Console Classifier: Framework :: Zope3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Zope Public License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Topic :: Software Development :: Testing zope.testrunner-4.4.9/src/zope.testrunner.egg-info/SOURCES.txt0000664000000000000000000002345112527371232022760 0ustar rootrootCHANGES.rst COPYRIGHT.rst DEVELOPING.rst LICENSE.rst MANIFEST.in README.rst bootstrap.py buildout.cfg setup.cfg setup.py tox.ini src/zope/__init__.py src/zope.testrunner.egg-info/PKG-INFO src/zope.testrunner.egg-info/SOURCES.txt src/zope.testrunner.egg-info/dependency_links.txt src/zope.testrunner.egg-info/entry_points.txt src/zope.testrunner.egg-info/namespace_packages.txt src/zope.testrunner.egg-info/not-zip-safe src/zope.testrunner.egg-info/pbr.json src/zope.testrunner.egg-info/requires.txt src/zope.testrunner.egg-info/top_level.txt src/zope/testrunner/__init__.py src/zope/testrunner/__main__.py src/zope/testrunner/_doctest.py src/zope/testrunner/coverage.py src/zope/testrunner/debug.py src/zope/testrunner/eggsupport.py src/zope/testrunner/exceptions.py src/zope/testrunner/feature.py src/zope/testrunner/filter.py src/zope/testrunner/find.py src/zope/testrunner/formatter.py src/zope/testrunner/garbagecollection.py src/zope/testrunner/interfaces.py src/zope/testrunner/layer.py src/zope/testrunner/listing.py src/zope/testrunner/logsupport.py src/zope/testrunner/options.py src/zope/testrunner/process.py src/zope/testrunner/profiling.py src/zope/testrunner/refcount.py src/zope/testrunner/runner.py src/zope/testrunner/selftest.py src/zope/testrunner/shuffle.py src/zope/testrunner/statistics.py src/zope/testrunner/tb_format.py src/zope/testrunner/tests/__init__.py src/zope/testrunner/tests/test_doctest.py src/zope/testrunner/tests/test_filter.py src/zope/testrunner/tests/test_runner.py src/zope/testrunner/tests/test_subunit.py src/zope/testrunner/tests/testrunner-arguments.txt src/zope/testrunner/tests/testrunner-colors.txt src/zope/testrunner/tests/testrunner-coverage-win32.txt src/zope/testrunner/tests/testrunner-coverage.txt src/zope/testrunner/tests/testrunner-debugging-import-failure.test src/zope/testrunner/tests/testrunner-debugging-layer-setup.test src/zope/testrunner/tests/testrunner-debugging-nonprintable-exc.test src/zope/testrunner/tests/testrunner-debugging.txt src/zope/testrunner/tests/testrunner-discovery.txt src/zope/testrunner/tests/testrunner-edge-cases.txt src/zope/testrunner/tests/testrunner-eggsupport.txt src/zope/testrunner/tests/testrunner-errors.txt src/zope/testrunner/tests/testrunner-gc.txt src/zope/testrunner/tests/testrunner-knit.txt src/zope/testrunner/tests/testrunner-layers-api.txt src/zope/testrunner/tests/testrunner-layers-buff.txt src/zope/testrunner/tests/testrunner-layers-cantfind.txt src/zope/testrunner/tests/testrunner-layers-cwd.txt src/zope/testrunner/tests/testrunner-layers-instances.txt src/zope/testrunner/tests/testrunner-layers-ntd.txt src/zope/testrunner/tests/testrunner-layers-topological-sort.txt src/zope/testrunner/tests/testrunner-layers.txt src/zope/testrunner/tests/testrunner-leaks-err.txt src/zope/testrunner/tests/testrunner-leaks.txt src/zope/testrunner/tests/testrunner-nestedcode.txt src/zope/testrunner/tests/testrunner-profiling-cprofiler.txt src/zope/testrunner/tests/testrunner-profiling.txt src/zope/testrunner/tests/testrunner-progress.txt src/zope/testrunner/tests/testrunner-repeat.txt src/zope/testrunner/tests/testrunner-report-skipped.txt src/zope/testrunner/tests/testrunner-shuffle.txt src/zope/testrunner/tests/testrunner-simple.txt src/zope/testrunner/tests/testrunner-subprocess-errors.txt src/zope/testrunner/tests/testrunner-subunit-err.txt src/zope/testrunner/tests/testrunner-subunit-leaks.txt src/zope/testrunner/tests/testrunner-subunit.txt src/zope/testrunner/tests/testrunner-tb-format.txt src/zope/testrunner/tests/testrunner-test-selection.txt src/zope/testrunner/tests/testrunner-unexpected-success.txt src/zope/testrunner/tests/testrunner-verbose.txt src/zope/testrunner/tests/testrunner-wo-source.txt src/zope/testrunner/tests/testrunner.txt src/zope/testrunner/tests/testrunner-ex/README.txt src/zope/testrunner/tests/testrunner-ex/gc0.py src/zope/testrunner/tests/testrunner-ex/gc1.py src/zope/testrunner/tests/testrunner-ex/gcset.py src/zope/testrunner/tests/testrunner-ex/gcstats.py src/zope/testrunner/tests/testrunner-ex/leak.py src/zope/testrunner/tests/testrunner-ex/pledge.py src/zope/testrunner/tests/testrunner-ex/samplelayers.py src/zope/testrunner/tests/testrunner-ex/sampletests.txt src/zope/testrunner/tests/testrunner-ex/sampletests_buffering.py src/zope/testrunner/tests/testrunner-ex/sampletestsf.py src/zope/testrunner/tests/testrunner-ex/sampletestsl.txt src/zope/testrunner/tests/testrunner-ex/unicode.py src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/test.py src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/__init__.py src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/bar/__init__.py src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/bar/tests.py src/zope/testrunner/tests/testrunner-ex-6/cwdtests.py src/zope/testrunner/tests/testrunner-ex-719369/sampletestsf.py src/zope/testrunner/tests/testrunner-ex-pp-lib/sample4/__init__.py src/zope/testrunner/tests/testrunner-ex-pp-lib/sample4/products/__init__.py src/zope/testrunner/tests/testrunner-ex-pp-products/__init__.py src/zope/testrunner/tests/testrunner-ex-pp-products/sampletests.py src/zope/testrunner/tests/testrunner-ex-pp-products/more/__init__.py src/zope/testrunner/tests/testrunner-ex-pp-products/more/sampletests.py src/zope/testrunner/tests/testrunner-ex-skip/sample_skipped_tests.py src/zope/testrunner/tests/testrunner-ex/sample1/__init__.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_discover.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_discover_notests.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_none_suite.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_none_test.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_ntd.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_ntds.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletestsf.py src/zope/testrunner/tests/testrunner-ex/sample1/sample11/__init__.py src/zope/testrunner/tests/testrunner-ex/sample1/sample11/sampletests.py src/zope/testrunner/tests/testrunner-ex/sample1/sample12/__init__.py src/zope/testrunner/tests/testrunner-ex/sample1/sample13/__init__.py src/zope/testrunner/tests/testrunner-ex/sample1/sample13/sampletests.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/__init__.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test1.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test11.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test111.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test112.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test12.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test121.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test122.py src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py src/zope/testrunner/tests/testrunner-ex/sample2/__init__.py src/zope/testrunner/tests/testrunner-ex/sample2/e.txt src/zope/testrunner/tests/testrunner-ex/sample2/sampletests_1.py src/zope/testrunner/tests/testrunner-ex/sample2/sampletests_e.py src/zope/testrunner/tests/testrunner-ex/sample2/sampletests_f.py src/zope/testrunner/tests/testrunner-ex/sample2/sampletests_ntd.py src/zope/testrunner/tests/testrunner-ex/sample2/sampletests_ntds.py src/zope/testrunner/tests/testrunner-ex/sample2/stderrtest.py src/zope/testrunner/tests/testrunner-ex/sample2/do-not-enter/sampletests.py src/zope/testrunner/tests/testrunner-ex/sample2/donotenter/sampletests.py src/zope/testrunner/tests/testrunner-ex/sample2/sample21/__init__.py src/zope/testrunner/tests/testrunner-ex/sample2/sample21/sampletests.py src/zope/testrunner/tests/testrunner-ex/sample2/sample21/sampletests_i.py src/zope/testrunner/tests/testrunner-ex/sample2/sample22/__init__.py src/zope/testrunner/tests/testrunner-ex/sample2/sample22/sampletests_i.py src/zope/testrunner/tests/testrunner-ex/sample2/sample23/__init__.py src/zope/testrunner/tests/testrunner-ex/sample2/sample23/sampletests_i.py src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/__init__.py src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/testone.py src/zope/testrunner/tests/testrunner-ex/sample3/__init__.py src/zope/testrunner/tests/testrunner-ex/sample3/post_mortem5.txt src/zope/testrunner/tests/testrunner-ex/sample3/post_mortem6.txt src/zope/testrunner/tests/testrunner-ex/sample3/post_mortem_failure.txt src/zope/testrunner/tests/testrunner-ex/sample3/sampletests.py src/zope/testrunner/tests/testrunner-ex/sample3/sampletests_d.py src/zope/testrunner/tests/testrunner-ex/sample3/sampletests_ntd.py src/zope/testrunner/tests/testrunner-ex/sample3/set_trace5.txt src/zope/testrunner/tests/testrunner-ex/sample3/set_trace6.txt src/zope/testrunner/tests/testrunner-ex/sample3/sample31/__init__.py src/zope/testrunner/tests/testrunner-ex/sample3/sample32/__init__.py src/zope/testrunner/tests/testrunner-ex/sample3/sample33/__init__.py src/zope/testrunner/tests/testrunner-ex/sampletests/__init__.py src/zope/testrunner/tests/testrunner-ex/sampletests/test1.py src/zope/testrunner/tests/testrunner-ex/sampletests/test11.py src/zope/testrunner/tests/testrunner-ex/sampletests/test111.py src/zope/testrunner/tests/testrunner-ex/sampletests/test112.py src/zope/testrunner/tests/testrunner-ex/sampletests/test12.py src/zope/testrunner/tests/testrunner-ex/sampletests/test121.py src/zope/testrunner/tests/testrunner-ex/sampletests/test122.py src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py src/zope/testrunner/tests/testrunner-ex/usecompiled/README.txt src/zope/testrunner/tests/testrunner-ex/usecompiled/__init__.py src/zope/testrunner/tests/testrunner-ex/usecompiled/compiletest.py src/zope/testrunner/tests/testrunner-ex/usecompiled/package/__init__.py src/zope/testrunner/tests/testrunner-ex/usecompiled/package/compiletest.pyzope.testrunner-4.4.9/src/zope.testrunner.egg-info/namespace_packages.txt0000664000000000000000000000000512527371232025415 0ustar rootrootzope zope.testrunner-4.4.9/src/zope.testrunner.egg-info/dependency_links.txt0000664000000000000000000000000112527371232025135 0ustar rootroot zope.testrunner-4.4.9/src/zope.testrunner.egg-info/not-zip-safe0000664000000000000000000000000112527371226023320 0ustar rootroot zope.testrunner-4.4.9/src/zope.testrunner.egg-info/top_level.txt0000664000000000000000000000000512527371232023614 0ustar rootrootzope zope.testrunner-4.4.9/src/zope.testrunner.egg-info/requires.txt0000664000000000000000000000012212527371232023462 0ustar rootrootsetuptools six zope.exceptions zope.interface [test] zope.testing python-subunit zope.testrunner-4.4.9/src/zope.testrunner.egg-info/pbr.json0000664000000000000000000000005612527371232022546 0ustar rootroot{"is_release": true, "git_version": "0bf6a09"}zope.testrunner-4.4.9/src/zope/0000775000000000000000000000000012552256734015174 5ustar rootrootzope.testrunner-4.4.9/src/zope/__init__.py0000664000000000000000000000007012527371226017276 0ustar rootroot__import__('pkg_resources').declare_namespace(__name__) zope.testrunner-4.4.9/src/zope/testrunner/0000775000000000000000000000000012552256734017405 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/_doctest.py0000664000000000000000000000372312527371226021564 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Doc test support for the test runner. """ import sys import doctest import zope.testrunner.feature class DocTest(zope.testrunner.feature.Feature): active = True def global_setup(self): options = self.runner.options output = options.output self.old_reporting_flags = doctest.set_unittest_reportflags(0) reporting_flags = 0 if options.ndiff: reporting_flags = doctest.REPORT_NDIFF if options.udiff: if reporting_flags: output.error("Can only give one of --ndiff, --udiff, or --cdiff") sys.exit(1) reporting_flags = doctest.REPORT_UDIFF if options.cdiff: if reporting_flags: output.error("Can only give one of --ndiff, --udiff, or --cdiff") sys.exit(1) reporting_flags = doctest.REPORT_CDIFF if options.report_only_first_failure: reporting_flags |= doctest.REPORT_ONLY_FIRST_FAILURE if reporting_flags: doctest.set_unittest_reportflags(reporting_flags) def global_shutdown(self): doctest.set_unittest_reportflags(self.old_reporting_flags) # Use a special exception for the test runner. from zope.testrunner.exceptions import DocTestFailureException doctest.DocTestCase.failureException = DocTestFailureException zope.testrunner-4.4.9/src/zope/testrunner/feature.py0000664000000000000000000000374612527371226021420 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Generic features for the test runner. """ import zope.interface import zope.testrunner.interfaces @zope.interface.implementer(zope.testrunner.interfaces.IFeature) class Feature(object): """A base class implementing no-op methods for the IFeature interface.""" active = False def __init__(self, runner): self.runner = runner def global_setup(self): """Executed once when the test runner is being set up.""" pass def late_setup(self): """Executed once right before the actual tests get executed and after all global setups have happened. """ pass def layer_setup(self, layer): """Executed once after a layer was set up.""" pass def layer_teardown(self, layer): """Executed once after a layer was run.""" pass def test_setup(self): """Executed once before each test.""" pass def test_teardown(self): """Executed once after each test.""" pass def early_teardown(self): """Executed once directly after all tests.""" pass def global_teardown(self): """Executed once after all tests where run and early teardowns have happened.""" pass def report(self): """Executed once after all tests have been run and all setup was torn down.""" pass zope.testrunner-4.4.9/src/zope/testrunner/garbagecollection.py0000664000000000000000000000432012527371226023416 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Garbage collection support. """ import gc import sys import zope.testrunner.feature class Threshold(zope.testrunner.feature.Feature): def __init__(self, runner): super(Threshold, self).__init__(runner) self.threshold = self.runner.options.gc self.active = bool(self.threshold) if not self.active: return if len(self.threshold) > 3: self.runner.options.output.error("Too many --gc options") sys.exit(1) def global_setup(self): self.old_threshold = gc.get_threshold() if self.threshold[0]: self.runner.options.output.info( "Cyclic garbage collection threshold set to: %s" % repr(tuple(self.threshold))) else: self.runner.options.output.info( "Cyclic garbage collection is disabled.") gc.set_threshold(*self.threshold) def global_teardown(self): gc.set_threshold(*self.old_threshold) class Debug(zope.testrunner.feature.Feature): """Manages garbage collection debug flags.""" def __init__(self, runner): super(Debug, self).__init__(runner) self.flags = self.runner.options.gc_option self.active = bool(self.flags) if not self.active: return def global_setup(self): # Set garbage collection debug flags self.old_flags = gc.get_debug() new_flags = 0 for op in self.flags: new_flags |= getattr(gc, op) gc.set_debug(new_flags) def global_teardown(self): gc.set_debug(self.old_flags) zope.testrunner-4.4.9/src/zope/testrunner/process.py0000664000000000000000000000413712527371226021436 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Subprocess support. """ from __future__ import print_function import sys import zope.testrunner.feature class SubProcess(zope.testrunner.feature.Feature): """Lists all tests in the report instead of running the tests.""" def __init__(self, runner): super(SubProcess, self).__init__(runner) self.active = bool(runner.options.resume_layer) def global_setup(self): self.original_stderr = sys.stderr sys.stderr = sys.stdout if self.runner.options.processes > 1: # If we only have one subprocess, there's absolutely # no reason to squelch. We will let the messages through in a # timely manner, if they have been requested. On the other hand, if # there are multiple processes, we do squelch to 0. self.runner.options.verbose = 0 self.progress = False def report(self): sys.stdout.close() # Communicate with the parent. The protocol is obvious: print(self.runner.ran, len(self.runner.failures), len(self.runner.errors), file=self.original_stderr) for test, exc_info in self.runner.failures: print(' '.join(str(test).strip().split('\n')), file=self.original_stderr) for test, exc_info in self.runner.errors: print(' '.join(str(test).strip().split('\n')), file=self.original_stderr) # You need to flush in Python 3, and it doesn't hurt in Python 2: self.original_stderr.flush() zope.testrunner-4.4.9/src/zope/testrunner/filter.py0000664000000000000000000001012612527371226021240 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Filter which tests to run. """ import re import zope.testrunner.feature UNITTEST_LAYER = 'zope.testrunner.layer.UnitTests' class Filter(zope.testrunner.feature.Feature): """Filters and orders all tests registered until now.""" active = True def global_setup(self): layers = self.runner.tests_by_layer_name options = self.runner.options if UNITTEST_LAYER in layers: # We start out assuming unit tests should run and look for reasons # why they shouldn't be run. should_run = True if (not options.non_unit): if options.layer: accept = build_filtering_func(options.layer) should_run = accept(UNITTEST_LAYER) else: should_run = True else: should_run = False if not should_run: layers.pop(UNITTEST_LAYER) if self.runner.options.resume_layer is not None: for name in list(layers): if name != self.runner.options.resume_layer: layers.pop(name) if not layers: self.runner.options.output.error_with_banner( "Cannot find layer %s" % self.runner.options.resume_layer) self.runner.errors.append( ("subprocess failed for %s" % self.runner.options.resume_layer, None)) elif self.runner.options.layer: accept = build_filtering_func(self.runner.options.layer) for name in list(layers): if not accept(name): # No pattern matched this name so we remove it layers.pop(name) if (self.runner.options.verbose and not self.runner.options.resume_layer): if self.runner.options.all: msg = "Running tests at all levels" else: msg = "Running tests at level %d" % self.runner.options.at_level self.runner.options.output.info(msg) def report(self): if not self.runner.do_run_tests: return if self.runner.options.resume_layer: return if self.runner.options.verbose: self.runner.options.output.tests_with_errors(self.runner.errors) self.runner.options.output.tests_with_failures(self.runner.failures) def build_filtering_func(patterns): """Build a filtering function from a set of patterns Patterns are understood as regular expressions, with the additional feature that, prefixed by "!", they create a "don't match" rule. This returns a function which returns True if a string matches the set of patterns, or False if it doesn't match. """ selected = [] unselected = [] for pattern in patterns: if pattern.startswith('!'): store = unselected.append pattern = pattern[1:] else: store = selected.append store(re.compile(pattern).search) if not selected and unselected: # If there's no selection patterns but some un-selection patterns, # suppose we want everything (that is, everything that matches '.'), # minus the un-selection ones. selected.append(re.compile('.').search) def accept(value): return (any(search(value) for search in selected) and not any(search(value) for search in unselected)) return accept zope.testrunner-4.4.9/src/zope/testrunner/tb_format.py0000664000000000000000000000350012527371226021726 0ustar rootroot############################################################################## # # Copyright (c) 2001, 2002 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Set up testing environment """ import sys import traceback import zope.exceptions.exceptionformatter import zope.testrunner.feature def format_exception(t, v, tb, limit=None, chain=None): if chain: values = traceback._iter_chain(v, tb) else: values = [(v, tb)] fmt = zope.exceptions.exceptionformatter.TextExceptionFormatter( limit=None, with_filenames=True) for v, tb in values: return fmt.formatException(t, v, tb) def print_exception(t, v, tb, limit=None, file=None, chain=None): if chain: values = traceback._iter_chain(v, tb) else: values = [(v, tb)] if file is None: file = sys.stdout for v, tb in values: file.writelines(format_exception(t, v, tb, limit)) class Traceback(zope.testrunner.feature.Feature): active = True def global_setup(self): self.old_format = traceback.format_exception traceback.format_exception = format_exception self.old_print = traceback.print_exception traceback.print_exception = print_exception def global_teardown(self): traceback.format_exception = self.old_format traceback.print_exception = self.old_print zope.testrunner-4.4.9/src/zope/testrunner/coverage.py0000664000000000000000000001144012527371226021546 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Code coverage analysis """ import trace import sys import os.path import threading import zope.testrunner.feature from zope.testrunner.find import test_dirs # For some reason, the doctest module resets the trace callable randomly, thus # disabling the coverage. Simply disallow the code from doing this. A real # trace can be set, so that debugging still works. osettrace = sys.settrace def settrace(trace): if trace is None: return osettrace(trace) class TestTrace(trace.Trace): """Simple tracer. >>> tracer = TestTrace([], count=False, trace=False) Simple rules for use: you can't stop the tracer if it not started and you can't start the tracer if it already started: >>> tracer.stop() Traceback (most recent call last): File 'testrunner.py' AssertionError: can't stop if not started >>> tracer.start() >>> tracer.start() Traceback (most recent call last): File 'testrunner.py' AssertionError: can't start if already started >>> tracer.stop() >>> tracer.stop() Traceback (most recent call last): File 'testrunner.py' AssertionError: can't stop if not started """ def __init__(self, directories, **kw): trace.Trace.__init__(self, **kw) self.ignore = TestIgnore(directories) self.started = False def start(self): assert not self.started, "can't start if already started" if not self.donothing: sys.settrace = settrace sys.settrace(self.globaltrace) threading.settrace(self.globaltrace) self.started = True def stop(self): assert self.started, "can't stop if not started" if not self.donothing: sys.settrace = osettrace sys.settrace(None) threading.settrace(None) self.started = False class TestIgnore: def __init__(self, directories): self._test_dirs = [self._filenameFormat(d[0]) + os.path.sep for d in directories] self._ignore = {} self._ignored = self._ignore.get def names(self, filename, modulename): # Special case: Modules generated from text files; i.e. doctests if modulename == '': return True filename = self._filenameFormat(filename) ignore = self._ignored(filename) if ignore is None: ignore = True if filename is not None: for d in self._test_dirs: if filename.startswith(d): ignore = False break self._ignore[filename] = ignore return ignore def _filenameFormat(self, filename): return os.path.abspath(filename) if sys.platform == 'win32': #on win32 drive name can be passed with different case to `names` #that lets e.g. the coverage profiler skip complete files #_filenameFormat will make sure that all drive and filenames get lowercased #albeit trace coverage has still problems with lowercase drive letters #when determining the dotted module name OldTestIgnore = TestIgnore class TestIgnore(OldTestIgnore): def _filenameFormat(self, filename): return os.path.normcase(os.path.abspath(filename)) class Coverage(zope.testrunner.feature.Feature): tracer = None directory = None def __init__(self, runner): super(Coverage, self).__init__(runner) self.active = bool(runner.options.coverage) def global_setup(self): """Executed once when the test runner is being set up.""" self.directory = os.path.join(os.getcwd(), self.runner.options.coverage) # FIXME: This shouldn't rely on the find feature directly. self.tracer = TestTrace(test_dirs(self.runner.options, {}), trace=False, count=True) self.tracer.start() def early_teardown(self): """Executed once directly after all tests.""" self.tracer.stop() def report(self): """Executed once after all tests have been run and all setup was torn down.""" r = self.tracer.results() r.write_results(summary=True, coverdir=self.directory) zope.testrunner-4.4.9/src/zope/testrunner/interfaces.py0000664000000000000000000001063712527371226022105 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test runner interfaces XXX Note: These interfaces are still being sketched out. Please do not rely on them, yet. """ import zope.interface class EndRun(Exception): """Indicate that the existing run call should stop Used to prevent additional test output after post-mortem debugging. """ class IFeature(zope.interface.Interface): """Features extend the test runners functionality in a pipe-lined order. """ active = zope.interface.Attribute( "Flag whether this feature is activated. If it is not activated than " "its methods won't be called by the runner.") def global_setup(): """Executed once when the test runner is being set up.""" def late_setup(): """Executed once right before the actual tests get executed and after all global setups have happened. Should do as little work as possible to avoid timing interferences with other features. It is guaranteed that the calling stack frame is not left until early_teardown was called. """ def layer_setup(layer): """Executed once after a layer was set up.""" def layer_teardown(layer): """Executed once after a layer was run.""" def test_setup(test): """Executed once before each test.""" def test_teardown(test): """Executed once after each test.""" def early_teardown(): """Executed once directly after all tests. This method should do as little as possible to avoid timing issues. It is guaranteed to be called directly from the same stack frame that called `late_setup`. """ def global_teardown(): """Executed once after all tests where run and early teardowns have happened. """ def report(): """Executed once after all tests have been run and all setup was torn down. This is the only method that should produce output. """ class ITestRunner(zope.interface.Interface): """The test runner manages test layers and their execution. The functionality of a test runner can be extended by configuring features. """ options = zope.interface.Attribute( "Provides access to configuration options.") class IMinimalTestLayer(zope.interface.Interface): """A test layer. This is the bare minimum that a ``layer`` attribute on a test suite should provide. """ __bases__ = zope.interface.Attribute( "A tuple of base layers.") __name__ = zope.interface.Attribute( "Name of the layer") __module__ = zope.interface.Attribute( "Dotted name of the module that defines this layer") def __hash__(): """A test layer must be hashable. The default identity-based __eq__ and __hash__ that Python provides usually suffice. """ def __eq__(): """A test layer must be hashable. The default identity-based __eq__ and __hash__ that Python provides usually suffice. """ class IFullTestLayer(IMinimalTestLayer): """A test layer. This is the full list of optional methods that a test layer can specify. """ def setUp(): """Shared layer setup. Called once before any of the tests in this layer (or sublayers) are run. """ def tearDown(): """Shared layer teardown. Called once after all the tests in this layer (and sublayers) are run. May raise NotImplementedError. """ def testSetUp(): """Additional test setup. Called once before every of test in this layer (or sublayers) is run. """ def testTearDown(): """Additional test teardown. Called once after every of test in this layer (or sublayers) is run. """ zope.testrunner-4.4.9/src/zope/testrunner/__init__.py0000664000000000000000000000453312527371226021517 0ustar rootroot############################################################################## # # Copyright (c) 2004-2013 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test runner """ import os import sys def run(defaults=None, args=None, script_parts=None, cwd=None): """Main runner function which can be and is being used from main programs. Will execute the tests and exit the process according to the test result. """ failed = run_internal(defaults, args, script_parts=script_parts, cwd=cwd) sys.exit(int(failed)) def run_internal(defaults=None, args=None, script_parts=None, cwd=None): """Execute tests. Returns whether errors or failures occured during testing. """ if script_parts is None: script_parts = _script_parts(args) if cwd is None: cwd = os.getcwd() # XXX Bah. Lazy import to avoid circular/early import problems from zope.testrunner.runner import Runner runner = Runner(defaults, args, script_parts=script_parts, cwd=cwd) runner.run() return runner.failed def _script_parts(args=None): script_parts = (args or sys.argv)[0:1] # If we are running via setup.py, then we'll have to run the # sub-process differently. if script_parts[0] == 'setup.py': script_parts = ['-c', 'from zope.testrunner import run; run()'] else: # make sure we remember the absolute path early -- the tests might # do an os.chdir() script_parts[0] = os.path.abspath(script_parts[0]) return script_parts if __name__ == '__main__': # this used to allow people to try out the test runner with # python -m zope.testrunner --test-path . # on Python 2.5. This broke on 2.6, and 2.7 and newer use __main__.py # for that. But there are some users out there who actually use # python -e zope.testrunner.__init__ --test-path . # so let's keep this for BBB run() zope.testrunner-4.4.9/src/zope/testrunner/tests/0000775000000000000000000000000012552256734020547 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-coverage-win32.txt0000664000000000000000000000171712527371226026254 0ustar rootrootCode Coverage ============= On Windows drive names can be upper and lower case, these can be randomly passed to TestIgnore.names. Watch out for the case of the R drive! >>> from zope.testrunner.coverage import TestIgnore >>> ignore = TestIgnore((('r:\\winproject\\src\\blah\\foo', ''), ... ('R:\\winproject\\src\\blah\\bar', ''))) >>> ignore._test_dirs ['r:\\winproject\\src\\blah\\foo\\', 'R:\\winproject\\src\\blah\\bar\\'] We can now ask whether a particular module should be ignored: >>> ignore.names('r:\\winproject\\src\\blah\\foo\\baz.py', 'baz') False >>> ignore.names('R:\\winproject\\src\\blah\\foo\\baz.py', 'baz') False >>> ignore.names('r:\\winproject\\src\\blah\\bar\\zab.py', 'zab') False >>> ignore.names('R:\\winproject\\src\\blah\\bar\\zab.py', 'zab') False >>> ignore.names('r:\\winproject\\src\\blah\\hello.py', 'hello') True >>> ignore.names('R:\\winproject\\src\\blah\\hello.py', 'hello') True zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-layers-instances.txt0000664000000000000000000001421012527371226026775 0ustar rootrootLayers implemented via object instances ======================================= Layers are generally implemented as classes using class methods, but regular objects can be used as well. They need to provide __module__, __name__, and __bases__ attributes. >>> class TestLayer: ... def __init__(self, name, *bases): ... self.__name__ = name ... self.__bases__ = bases ... ... def setUp(self): ... log('%s.setUp' % self.__name__) ... ... def tearDown(self): ... log('%s.tearDown' % self.__name__) ... ... def testSetUp(self): ... log('%s.testSetUp' % self.__name__) ... ... def testTearDown(self): ... log('%s.testTearDown' % self.__name__) >>> BaseLayer = TestLayer('BaseLayer') >>> TopLayer = TestLayer('TopLayer', BaseLayer) Tests or test suites specify what layer they need by storing a reference in the 'layer' attribute. >>> import unittest >>> class TestSpecifyingBaseLayer(unittest.TestCase): ... 'This TestCase explicitly specifies its layer' ... layer = BaseLayer ... name = 'TestSpecifyingBaseLayer' # For testing only ... ... def setUp(self): ... log('TestSpecifyingBaseLayer.setUp') ... ... def tearDown(self): ... log('TestSpecifyingBaseLayer.tearDown') ... ... def test1(self): ... log('TestSpecifyingBaseLayer.test1') ... ... def test2(self): ... log('TestSpecifyingBaseLayer.test2') ... >>> class TestSpecifyingNoLayer(unittest.TestCase): ... 'This TestCase specifies no layer' ... name = 'TestSpecifyingNoLayer' # For testing only ... def setUp(self): ... log('TestSpecifyingNoLayer.setUp') ... ... def tearDown(self): ... log('TestSpecifyingNoLayer.tearDown') ... ... def test1(self): ... log('TestSpecifyingNoLayer.test') ... ... def test2(self): ... log('TestSpecifyingNoLayer.test') ... Create a TestSuite containing two test suites, one for each of TestSpecifyingBaseLayer and TestSpecifyingNoLayer. >>> umbrella_suite = unittest.TestSuite() >>> umbrella_suite.addTest(unittest.makeSuite(TestSpecifyingBaseLayer)) >>> no_layer_suite = unittest.makeSuite(TestSpecifyingNoLayer) >>> umbrella_suite.addTest(no_layer_suite) Before we can run the tests, we need to setup some helpers. >>> from zope.testrunner import options >>> from zope.testing.loggingsupport import InstalledHandler >>> import logging >>> log_handler = InstalledHandler('zope.testrunner.tests') >>> def log(msg): ... logging.getLogger('zope.testrunner.tests').info(msg) >>> def fresh_options(): ... opts = options.get_options(['--test-filter', '.*']) ... opts.resume_layer = None ... opts.resume_number = 0 ... return opts Now we run the tests. Note that the BaseLayer was not setup when the TestSpecifyingNoLayer was run and setup/torn down around the TestSpecifyingBaseLayer tests. >>> from zope.testrunner.runner import Runner >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running ...BaseLayer tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up ...BaseLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...BaseLayer in N.NNN seconds. Total: 4 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. Now lets specify a layer in the suite containing TestSpecifyingNoLayer and run the tests again. This demonstrates the other method of specifying a layer. This is generally how you specify what layer doctests need. >>> no_layer_suite.layer = BaseLayer >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running ...BaseLayer tests: Set up ...BaseLayer in N.NNN seconds. Ran 4 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...BaseLayer in N.NNN seconds. Clear our logged output, as we want to inspect it shortly. >>> log_handler.clear() Now lets also specify a layer in the TestSpecifyingNoLayer class and rerun the tests. This demonstrates that the most specific layer is used. It also shows the behavior of nested layers - because TopLayer extends BaseLayer, both the BaseLayer and TopLayer environments are setup when the TestSpecifyingNoLayer tests are run. >>> TestSpecifyingNoLayer.layer = TopLayer >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running ...BaseLayer tests: Set up ...BaseLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running ...TopLayer tests: Set up ...TopLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...TopLayer in N.NNN seconds. Tear down ...BaseLayer in N.NNN seconds. Total: 4 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. If we inspect our trace of what methods got called in what order, we can see that the layer setup and teardown methods only got called once. We can also see that the layer's test setup and teardown methods got called for each test using that layer in the right order. >>> def report(): ... print("Report:") ... for record in log_handler.records: ... print(record.getMessage()) >>> report() Report: BaseLayer.setUp BaseLayer.testSetUp TestSpecifyingBaseLayer.setUp TestSpecifyingBaseLayer.test1 TestSpecifyingBaseLayer.tearDown BaseLayer.testTearDown BaseLayer.testSetUp TestSpecifyingBaseLayer.setUp TestSpecifyingBaseLayer.test2 TestSpecifyingBaseLayer.tearDown BaseLayer.testTearDown TopLayer.setUp BaseLayer.testSetUp TopLayer.testSetUp TestSpecifyingNoLayer.setUp TestSpecifyingNoLayer.test TestSpecifyingNoLayer.tearDown TopLayer.testTearDown BaseLayer.testTearDown BaseLayer.testSetUp TopLayer.testSetUp TestSpecifyingNoLayer.setUp TestSpecifyingNoLayer.test TestSpecifyingNoLayer.tearDown TopLayer.testTearDown BaseLayer.testTearDown TopLayer.tearDown BaseLayer.tearDown zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-subprocess-errors.txt0000664000000000000000000001333312527371226027220 0ustar rootrootThis is a test for error handling when the test runner runs a test layer in a subprocess. Fake an IOError reading the output of the subprocess to exercise the reporting of that error: >>> class FakeStdout(object): ... raised = False ... def __init__(self, msg): ... self.msg = msg ... def readline(self): ... if not self.raised: ... self.raised = True ... raise IOError(self.msg) >>> class FakeStderr(object): ... def __init__(self, msg): ... self.msg = msg ... def read(self): ... return self.msg >>> class FakeProcess(object): ... def __init__(self, out, err): ... self.stdout = FakeStdout(out) ... self.stderr = FakeStderr(err) >>> class FakePopen(object): ... def __init__(self, out, err): ... self.out = out ... self.err = err ... def __call__(self, *args, **kw): ... return FakeProcess(self.out, self.err) >>> import subprocess >>> Popen = subprocess.Popen >>> subprocess.Popen = FakePopen( ... "Failure triggered to verify error reporting", ... "0 0 0") >>> import os, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... ] >>> argv = [sys.argv[0], ... '-vv', '--tests-pattern', '^sampletests_buffering.*'] >>> _ = testrunner.run_internal(defaults, argv) Running tests at level 1 Running sampletests_buffering.Layer1 tests: Set up sampletests_buffering.Layer1 in N.NNN seconds. Running: test_something (sampletests_buffering.TestSomething1) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running sampletests_buffering.Layer2 tests: Tear down sampletests_buffering.Layer1 ... not supported Error reading subprocess output for sampletests_buffering.Layer2 Failure triggered to verify error reporting Total: 1 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. Now fake some unexpected stderr to test reporting a failure when communicating with the subprocess: >>> subprocess.Popen = FakePopen( ... "Failure triggered to verify error reporting", ... b"segmentation fault (core dumped muahahaha)") >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... ] >>> argv = [sys.argv[0], ... '-vv', '--tests-pattern', '^sampletests_buffering.*'] >>> _ = testrunner.run_internal(defaults, argv) # doctest: +ELLIPSIS Running tests at level 1 Running sampletests_buffering.Layer1 tests: Set up sampletests_buffering.Layer1 in N.NNN seconds. Running: test_something (sampletests_buffering.TestSomething1) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running sampletests_buffering.Layer2 tests: Tear down sampletests_buffering.Layer1 ... not supported Error reading subprocess output for sampletests_buffering.Layer2 Failure triggered to verify error reporting ********************************************************************** Could not communicate with subprocess! Child command line: ['...', '--resume-layer', 'sampletests_buffering.Layer2', '0', '--default', '--path', '--default', 'testrunner-ex', '-vv', '--tests-pattern', '^sampletests_buffering.*'] Child stderr was: segmentation fault (core dumped muahahaha) ********************************************************************** Tests with errors: subprocess for sampletests_buffering.Layer2 Total: 1 tests, 0 failures, 1 errors and 0 skipped in N.NNN seconds. Very large subprocess output is trimmed, unless you ask for extra verbosity >>> subprocess.Popen = FakePopen( ... "Failure triggered to verify error reporting", ... "\n".join(str(n) for n in range(1, 101)).encode()) >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... ] >>> argv = [sys.argv[0], ... '-v', '--tests-pattern', '^sampletests_buffering.*'] >>> _ = testrunner.run_internal(defaults, argv) # doctest: +ELLIPSIS Running tests at level 1 Running sampletests_buffering.Layer1 tests: Set up sampletests_buffering.Layer1 in N.NNN seconds. Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running sampletests_buffering.Layer2 tests: Tear down sampletests_buffering.Layer1 ... not supported Error reading subprocess output for sampletests_buffering.Layer2 Failure triggered to verify error reporting ********************************************************************** Could not communicate with subprocess! Child command line: ['...', '--resume-layer', 'sampletests_buffering.Layer2', '0', '--default', '--path', '--default', 'testrunner-ex', '-v', '--tests-pattern', '^sampletests_buffering.*'] Child stderr was: 1 2 3 4 5 6 7 8 9 10 ... 91 92 93 94 95 96 97 98 99 100 ********************************************************************** Tests with errors: subprocess for sampletests_buffering.Layer2 Total: 1 tests, 0 failures, 1 errors and 0 skipped in N.NNN seconds. >>> subprocess.Popen = Popen zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-layers-cantfind.txt0000664000000000000000000000332312527371226026577 0ustar rootrootFailure to find layers should not pass silently =============================================== This is a regression test for the following bug: you try to run several test layers using subprocesses (e.g. because you used bin/test -j99), and the child process somehow is unable to find the layer it was supposed to be running. This is a serious problem that should not pass silently. Instead of setting up the conditions for this problem to actually occur in practice we'll simulate the subprocess invocation using --resume-layer. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] The test runner does some funky stuff in this case, specifically, it closes sys.stdout, which makes doctest unhappy, so we stub the close method out. >>> sys.stdout.close = lambda: None >>> from six import StringIO >>> orig_stderr = sys.stderr >>> sys.stderr = fake_stderr = StringIO() >>> sys.argv = 'test --resume-layer NoSuchLayer 0'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) ********************************************************************** Cannot find layer NoSuchLayer ********************************************************************** Total: 0 tests, 0 failures, 1 errors and 0 skipped in 0.000 seconds. True It also prints to stderr to communicate with the parent process >>> print(fake_stderr.getvalue(), end='') 0 0 1 subprocess failed for NoSuchLayer Cleanup >>> del sys.stdout.close >>> sys.stderr = orig_stderr zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-errors.txt0000664000000000000000000007660312527371226025043 0ustar rootrootErrors and Failures =================== Let's look at tests that have errors and failures, first we need to make a temporary copy of the entire testing directory (except .svn files which may be read only): >>> import os.path, sys, tempfile, shutil >>> tmpdir = tempfile.mkdtemp() >>> directory_with_tests = os.path.join(tmpdir, 'testrunner-ex') >>> source = os.path.join(this_directory, 'testrunner-ex') >>> n = len(source) + 1 >>> for root, dirs, files in os.walk(source): ... dirs[:] = [d for d in dirs if d != ".svn"] # prune cruft ... os.mkdir(os.path.join(directory_with_tests, root[n:])) ... for f in files: ... _ = shutil.copy(os.path.join(root, f), ... os.path.join(directory_with_tests, root[n:], f)) >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --tests-pattern ^sampletests(f|_e|_f)?$ '.split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Failure in test eek (sample2.sampletests_e) Failed doctest test for sample2.sampletests_e.eek File "testrunner-ex/sample2/sampletests_e.py", line 28, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_e.py", line 30, in sample2.sampletests_e.eek Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined Error in test test3 (sample2.sampletests_e.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_e.py", line 43, in test3 f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined Failure in test testrunner-ex/sample2/e.txt Failed doctest test for e.txt File "testrunner-ex/sample2/e.txt", line 0 ---------------------------------------------------------------------- File "testrunner-ex/sample2/e.txt", line 4, in e.txt Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "", line 2, in f return x NameError: global name 'x' is not defined Failure in test test (sample2.sampletests_f.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_f.py", line 21, in test self.assertEqual(1,0) File "/usr/local/python/2.3/lib/python2.3/unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: 1 != 0 Ran 164 tests with 3 failures, 1 errors and 0 skipped in N.NNN seconds. ... Total: 329 tests, 3 failures, 1 errors and 0 skipped in N.NNN seconds. True We see that we get an error report and a traceback for the failing test. In addition, the test runner returned True, indicating that there was an error. If we ask for verbosity, the dotted output will be interrupted, and there'll be a summary of the errors at the end of the test: >>> sys.argv = 'test --tests-pattern ^sampletests(f|_e|_f)?$ -uv'.split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: ... Running: ................................................................................................. Failure in test eek (sample2.sampletests_e) Failed doctest test for sample2.sampletests_e.eek File "testrunner-ex/sample2/sampletests_e.py", line 28, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_e.py", line 30, in sample2.sampletests_e.eek Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined ... Error in test test3 (sample2.sampletests_e.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_e.py", line 43, in test3 f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined ... Failure in test testrunner-ex/sample2/e.txt Failed doctest test for e.txt File "testrunner-ex/sample2/e.txt", line 0 ---------------------------------------------------------------------- File "testrunner-ex/sample2/e.txt", line 4, in e.txt Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "", line 2, in f return x NameError: global name 'x' is not defined . Failure in test test (sample2.sampletests_f.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_f.py", line 21, in test self.assertEqual(1,0) File ".../unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: 1 != 0 ................................................................................................ Ran 164 tests with 3 failures, 1 errors and 0 skipped in 0.040 seconds. ... Tests with errors: test3 (sample2.sampletests_e.Test) Tests with failures: eek (sample2.sampletests_e) testrunner-ex/sample2/e.txt test (sample2.sampletests_f.Test) True Similarly for progress output, the progress ticker will be interrupted: >>> sys.argv = ('test --tests-pattern ^sampletests(f|_e|_f)?$ -u -ssample2' ... ' -p').split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: 1/47 (2.1%) Failure in test eek (sample2.sampletests_e) Failed doctest test for sample2.sampletests_e.eek File "testrunner-ex/sample2/sampletests_e.py", line 28, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_e.py", line 30, in sample2.sampletests_e.eek Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined 2/47 (4.3%)\r \r 3/47 (6.4%)\r \r 4/47 (8.5%) Error in test test3 (sample2.sampletests_e.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_e.py", line 43, in test3 f() File "testrunner-ex/sample2/sampletests_e.py", line 19, in f g() File "testrunner-ex/sample2/sampletests_e.py", line 24, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined 5/47 (10.6%)\r \r 6/47 (12.8%)\r \r 7/47 (14.9%) Failure in test testrunner-ex/sample2/e.txt Failed doctest test for e.txt File "testrunner-ex/sample2/e.txt", line 0 ---------------------------------------------------------------------- File "testrunner-ex/sample2/e.txt", line 4, in e.txt Failed example: f() Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? f() File "", line 2, in f return x NameError: global name 'x' is not defined 8/47 (17.0%) Failure in test test (sample2.sampletests_f.Test) Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_f.py", line 21, in test self.assertEqual(1,0) File ".../unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: 1 != 0 9/47 (19.1%)\r \r 10/47 (21.3%)\r \r 11/47 (23.4%)\r \r 12/47 (25.5%)\r \r 13/47 (27.7%)\r \r 14/47 (29.8%)\r \r 15/47 (31.9%)\r \r 16/47 (34.0%)\r \r 17/47 (36.2%)\r \r 18/47 (38.3%)\r \r 19/47 (40.4%)\r \r 20/47 (42.6%)\r \r 21/47 (44.7%)\r \r 22/47 (46.8%)\r \r 23/47 (48.9%)\r \r 24/47 (51.1%)\r \r 25/47 (53.2%)\r \r 26/47 (55.3%)\r \r 27/47 (57.4%)\r \r 28/47 (59.6%)\r \r 29/47 (61.7%)\r \r 30/47 (63.8%)\r \r 31/47 (66.0%)\r \r 32/47 (68.1%)\r \r 33/47 (70.2%)\r \r 34/47 (72.3%)\r \r 35/47 (74.5%)\r \r 36/47 (76.6%)\r \r 37/47 (78.7%)\r \r 38/47 (80.9%)\r \r 39/47 (83.0%)\r \r 40/47 (85.1%)\r \r 41/47 (87.2%)\r \r 42/47 (89.4%)\r \r 43/47 (91.5%)\r \r 44/47 (93.6%)\r \r 45/47 (95.7%)\r \r 46/47 (97.9%)\r \r 47/47 (100.0%)\r \r Ran 47 tests with 3 failures, 1 errors and 0 skipped in 0.054 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True If you also want a summary of errors at the end, ask for verbosity as well as progress output. Suppressing multiple doctest errors ----------------------------------- Often, when a doctest example fails, the failure will cause later examples in the same test to fail. Each failure is reported: >>> sys.argv = 'test --tests-pattern ^sampletests_1$'.split() >>> testrunner.run_internal(defaults) # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 21, in sample2.sampletests_1.eek Failed example: x Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x NameError: name 'x' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 24, in sample2.sampletests_1.eek Failed example: z = x + 1 Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? z = x + 1 NameError: name 'x' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True This can be a bit confusing, especially when there are enough tests that they scroll off a screen. Often you just want to see the first failure. This can be accomplished with the -1 option (for "just show me the first failed example in a doctest" :) >>> sys.argv = 'test --tests-pattern ^sampletests_1$ -1'.split() >>> testrunner.run_internal(defaults) # doctest: Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.001 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True The --hide-secondary-failures option is an alias for -1: >>> sys.argv = ( ... 'test --tests-pattern ^sampletests_1$' ... ' --hide-secondary-failures' ... ).split() >>> testrunner.run_internal(defaults) # doctest: Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.001 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True The --show-secondary-failures option counters -1 (or it's alias), causing the second and subsequent errors to be shown. This is useful if -1 is provided by a test script by inserting it ahead of command-line options in sys.argv. >>> sys.argv = ( ... 'test --tests-pattern ^sampletests_1$' ... ' --hide-secondary-failures --show-secondary-failures' ... ).split() >>> testrunner.run_internal(defaults) # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test eek (sample2.sampletests_1) Failed doctest test for sample2.sampletests_1.eek File "testrunner-ex/sample2/sampletests_1.py", line 17, in eek ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 19, in sample2.sampletests_1.eek Failed example: x = y Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x = y NameError: name 'y' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 21, in sample2.sampletests_1.eek Failed example: x Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? x NameError: name 'x' is not defined ---------------------------------------------------------------------- File "testrunner-ex/sample2/sampletests_1.py", line 24, in sample2.sampletests_1.eek Failed example: z = x + 1 Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1256, in __run compileflags, 1) in test.globs File "", line 1, in ? z = x + 1 NameError: name 'x' is not defined Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True Getting diff output for doctest failures ---------------------------------------- If a doctest has large expected and actual output, it can be hard to see differences when expected and actual output differ. The --ndiff, --udiff, and --cdiff options can be used to get diff output of various kinds. >>> sys.argv = 'test --tests-pattern ^pledge$'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Expected: I give my pledge, as an earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. It's soils, minerals, forests, waters, and wildlife. Got: I give my pledge, as and earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. It's soils, minerals, forests, waters, and wildlife. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Here, the actual output uses the word "and" rather than the word "an", but it's a bit hard to pick this out. We can use the various diff outputs to see this better. We could modify the test to ask for diff output, but it's easier to use one of the diff options. The --ndiff option requests a diff using Python's ndiff utility. This is the only method that marks differences within lines as well as across lines. For example, if a line of expected output contains digit 1 where actual output contains letter l, a line is inserted with a caret marking the mismatching column positions. >>> sys.argv = 'test --tests-pattern ^pledge$ --ndiff'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Differences (ndiff with -expected +actual): - I give my pledge, as an earthling, + I give my pledge, as and earthling, ? + to save, and faithfully, to defend from waste, the natural resources of my planet. It's soils, minerals, forests, waters, and wildlife. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.003 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. The -udiff option requests a standard "unified" diff: >>> sys.argv = 'test --tests-pattern ^pledge$ --udiff'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Differences (unified diff with -expected +actual): @@ -1,3 +1,3 @@ -I give my pledge, as an earthling, +I give my pledge, as and earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. The -cdiff option requests a standard "context" diff: >>> sys.argv = 'test --tests-pattern ^pledge$ --cdiff'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Failure in test pledge (pledge) Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- File "testrunner-ex/pledge.py", line 26, in pledge.pledge Failed example: print_pledge() Differences (context diff with expected followed by actual): *************** *** 1,3 **** ! I give my pledge, as an earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. --- 1,3 ---- ! I give my pledge, as and earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Specifying more than one diff option at once causes an error: >>> sys.argv = 'test --tests-pattern ^pledge$ --cdiff --udiff'.split() >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 >>> sys.argv = 'test --tests-pattern ^pledge$ --cdiff --ndiff'.split() >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 >>> sys.argv = 'test --tests-pattern ^pledge$ --udiff --ndiff'.split() >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 Testing-Module Import Errors ---------------------------- If there are errors when importing a test module, these errors are reported. In order to illustrate a module with a syntax error, we create one now: this module used to be checked in to the project, but then it was included in distributions of projects using zope.testrunner too, and distutils complained about the syntax error when it compiled Python files during installation of such projects. So first we create a module with bad syntax: >>> badsyntax_path = os.path.join(directory_with_tests, ... "sample2", "sampletests_i.py") >>> f = open(badsyntax_path, "w") >>> print("importx unittest", file=f) # syntax error >>> f.close() Then run the tests: >>> sys.argv = ('test --tests-pattern ^sampletests(f|_i)?$ --layer 1 ' ... ).split() >>> testrunner.run_internal(defaults) ... # doctest: +NORMALIZE_WHITESPACE Test-module import failures: Module: sample2.sampletests_i Traceback (most recent call last): File "testrunner-ex/sample2/sampletests_i.py", line 1 importx unittest ^ SyntaxError: invalid syntax Module: sample2.sample21.sampletests_i Traceback (most recent call last): File "testrunner-ex/sample2/sample21/sampletests_i.py", line 15, in ? import zope.testrunner.huh ImportError: No module named huh Module: sample2.sample23.sampletests_i Traceback (most recent call last): File "testrunner-ex/sample2/sample23/sampletests_i.py", line 18, in ? class Test(unittest.TestCase): File "testrunner-ex/sample2/sample23/sampletests_i.py", line 23, in Test raise TypeError('eek') TypeError: eek Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in 0.000 seconds. Ran 9 tests with 0 failures, 3 errors and 0 skipped in 0.000 seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in 0.000 seconds. Set up samplelayers.Layer112 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.007 seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Ran 26 tests with 0 failures, 3 errors and 0 skipped in 0.006 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. Test-modules with import problems: sample2.sampletests_i sample2.sample21.sampletests_i sample2.sample23.sampletests_i Total: 165 tests, 0 failures, 3 errors and 0 skipped in N.NNN seconds. True Reporting Errors to Calling Processes ------------------------------------- The testrunner returns the error status, indicating that the tests failed. This can be useful for an invoking process that wants to monitor the result of a test run. This is applied when invoking the testrunner using the ``run()`` function instead of ``run_internal()``: >>> sys.argv = ( ... 'test --tests-pattern ^sampletests_1$'.split()) >>> try: ... testrunner.run(defaults) ... except SystemExit as e: ... print('exited with code', e.code) ... else: ... print('sys.exit was not called') ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. ... Ran 1 tests with 1 failures, 0 errors and 0 skipped in 0.002 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. exited with code 1 Passing tests exit with code 0 according to UNIX practices: >>> sys.argv = ( ... 'test --tests-pattern ^sampletests$'.split()) >>> try: ... testrunner.run(defaults) ... except SystemExit as e2: ... print('exited with code', e2.code) ... else: ... print('sys.exit was not called') ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Total: 286 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. exited with code 0 And remove the temporary directory: >>> shutil.rmtree(tmpdir) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-719369/0000775000000000000000000000000012552256734024152 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-719369/sampletestsf.py0000664000000000000000000000140212527371226027227 0ustar rootroot############################################################################## # # Copyright (c) 2011 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest class TestUnexpectedSuccess(unittest.TestCase): @unittest.expectedFailure def test_ef(self): pass zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-profiling.txt0000664000000000000000000000354712527371226025515 0ustar rootrootProfiling ========= The testrunner supports hotshot and cProfile profilers. Hotshot profiler support does not work with python2.6 >>> import os.path, sys, tempfile >>> profiler = '--profile=hotshot' >>> if sys.hexversion >= 0x02060000: ... profiler = '--profile=cProfile' The testrunner includes the ability to profile the test execution with hotshot via the --profile option, if it a python <= 2.6 >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> sys.path.append(directory_with_tests) >>> tempdir = tempfile.mkdtemp(prefix='zope.testrunner-') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... '--profile-directory', tempdir, ... ] >>> sys.argv = [testrunner_script, profiler] When the tests are run, we get profiling output. >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: ... Running samplelayers.Layer1 tests: ... Running samplelayers.Layer11 tests: ... ncalls tottime percall cumtime percall filename:lineno(function) ... Total: ... tests, 0 failures, 0 errors and 0 skipped in ... seconds. False Profiling also works across layers. >>> sys.argv = [testrunner_script, '-ssample2', profiler, ... '--tests-pattern', 'sampletests_ntd'] >>> testrunner.run_internal(defaults) Running... Tear down ... not supported... ncalls tottime percall cumtime percall filename:lineno(function)... The testrunner creates temnporary files containing hotshot profiler data: >>> os.listdir(tempdir) ['tests_profile.cZj2jt.prof', 'tests_profile.yHD-so.prof'] It deletes these when rerun. We'll delete these ourselves: >>> import shutil >>> shutil.rmtree(tempdir) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-subunit-err.txt0000664000000000000000000000126512527371226025776 0ustar rootrootUsing subunit output without subunit installed ============================================== To use the --subunit reporting option, you must have subunit installed. If you do not, you will get an error message: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> sys.argv = 'test --subunit'.split() >>> _ = testrunner.run_internal(defaults) Subunit is not installed. Please install Subunit to generate subunit output. zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-debugging.txt0000664000000000000000000000750412527371226025454 0ustar rootrootDebugging ========= The testrunner module supports post-mortem debugging and debugging using `pdb.set_trace`. Let's look first at using `pdb.set_trace`. To demonstrate this, we'll provide input via helper Input objects: >>> class Input: ... def __init__(self, src): ... self.lines = src.split('\n') ... def readline(self): ... line = self.lines.pop(0) ... print(line) ... return line+'\n' If a test or code called by a test calls pdb.set_trace, then the runner will enter pdb at that point: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> real_stdin = sys.stdin >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace1').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... > testrunner-ex/sample3/sampletests_d.py(27)test_set_trace1() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.001 seconds. ... False Note that, prior to Python 2.4, calling pdb.set_trace caused pdb to break in the pdb.set_trace function. It was necessary to use 'next' or 'up' to get to the application code that called pdb.set_trace. In Python 2.4, pdb.set_trace causes pdb to stop right after the call to pdb.set_trace. You can also do post-mortem debugging, using the --post-mortem (-D) option: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem1 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Error in test test_post_mortem1 (sample3.sampletests_d.TestSomething) Traceback (most recent call last): File "testrunner-ex/sample3/sampletests_d.py", line 34, in test_post_mortem1 raise ValueError ValueError ...ValueError > testrunner-ex/sample3/sampletests_d.py(34)test_post_mortem1() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Note that the test runner exits after post-mortem debugging. In the example above, we debugged an error. Failures are actually converted to errors and can be debugged the same way: >>> sys.stdin = Input('p x\np y\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure1 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: ... Error in test test_post_mortem_failure1 (sample3.sampletests_d.TestSomething) Traceback (most recent call last): File ".../unittest.py", line 252, in debug getattr(self, self.__testMethodName)() File "testrunner-ex/sample3/sampletests_d.py", line 42, in test_post_mortem_failure1 assert x == y AssertionError ...AssertionError > testrunner-ex/sample3/sampletests_d.py(42)test_post_mortem_failure1() -> assert x == y (Pdb) p x 1 (Pdb) p y 2 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-subunit.txt0000664000000000000000000006411212527371226025210 0ustar rootrootSubunit Output ============== Subunit is a streaming protocol for interchanging test results. More information can be found at https://launchpad.net/subunit. First we need to make a temporary copy of the entire testing directory: >>> import os.path, sys, tempfile, shutil >>> tmpdir = tempfile.mkdtemp() >>> directory_with_tests = os.path.join(tmpdir, 'testrunner-ex') >>> source = os.path.join(this_directory, 'testrunner-ex') >>> n = len(source) + 1 >>> for root, dirs, files in os.walk(source): ... dirs[:] = [d for d in dirs if d != ".svn"] # prune cruft ... os.mkdir(os.path.join(directory_with_tests, root[n:])) ... for f in files: ... _ = shutil.copy(os.path.join(root, f), ... os.path.join(directory_with_tests, root[n:], f)) >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner Basic output ------------ Subunit output is line-based, with a 'test:' line for the start of each test and a 'successful:' line for each successful test. Zope layer set up and tear down events are represented as tests tagged with 'zope:layer'. This allows them to be distinguished from actual tests, provides a place for the layer timing information in the subunit stream and allows us to include error information if necessary. Once the layer is set up, all future tests are tagged with 'zope:layer:LAYER_NAME'. >>> sys.argv = 'test --layer 122 --subunit -t TestNotMuch'.split() >>> testrunner.run_internal(defaults) time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: samplelayers.Layer1:setUp tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: samplelayers.Layer1:setUp tags: zope:layer:samplelayers.Layer1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: samplelayers.Layer12:setUp tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: samplelayers.Layer12:setUp tags: zope:layer:samplelayers.Layer12 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: samplelayers.Layer122:setUp tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: samplelayers.Layer122:setUp tags: zope:layer:samplelayers.Layer122 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample1.sampletests.test122.TestNotMuch.test_1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample1.sampletests.test122.TestNotMuch.test_1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample1.sampletests.test122.TestNotMuch.test_2 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample1.sampletests.test122.TestNotMuch.test_2 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample1.sampletests.test122.TestNotMuch.test_3 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample1.sampletests.test122.TestNotMuch.test_3 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sampletests.test122.TestNotMuch.test_1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sampletests.test122.TestNotMuch.test_1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sampletests.test122.TestNotMuch.test_2 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sampletests.test122.TestNotMuch.test_2 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sampletests.test122.TestNotMuch.test_3 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sampletests.test122.TestNotMuch.test_3 tags: -zope:layer:samplelayers.Layer122 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: samplelayers.Layer122:tearDown tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: samplelayers.Layer122:tearDown tags: -zope:layer:samplelayers.Layer12 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: samplelayers.Layer12:tearDown tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: samplelayers.Layer12:tearDown tags: -zope:layer:samplelayers.Layer1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: samplelayers.Layer1:tearDown tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: samplelayers.Layer1:tearDown False Listing tests ------------- A subunit stream is a stream of test results, more or less, so the most natural way of listing tests in subunit is to simply emit successful test results without actually running the tests. Note that in this stream, we don't emit fake tests for the layer set up and tear down, because it simply doesn't happen. We also don't include the dependent layers in the stream (in this case Layer1 and Layer12), since they are not provided to the reporter. >>> sys.argv = ( ... 'test --layer 122 --list-tests --subunit -t TestNotMuch').split() >>> testrunner.run_internal(defaults) tags: zope:layer:samplelayers.Layer122 test: sample1.sampletests.test122.TestNotMuch.test_1 successful: sample1.sampletests.test122.TestNotMuch.test_1 test: sample1.sampletests.test122.TestNotMuch.test_2 successful: sample1.sampletests.test122.TestNotMuch.test_2 test: sample1.sampletests.test122.TestNotMuch.test_3 successful: sample1.sampletests.test122.TestNotMuch.test_3 test: sampletests.test122.TestNotMuch.test_1 successful: sampletests.test122.TestNotMuch.test_1 test: sampletests.test122.TestNotMuch.test_2 successful: sampletests.test122.TestNotMuch.test_2 test: sampletests.test122.TestNotMuch.test_3 successful: sampletests.test122.TestNotMuch.test_3 tags: -zope:layer:samplelayers.Layer122 False Profiling tests --------------- Test suites often cover a lot of code, and the performance of test suites themselves is often a critical part of the development process. Thus, it's good to be able to profile a test run. >>> import tempfile >>> tempdir = tempfile.mkdtemp(prefix='zope.testrunner-') >>> sys.argv = [ ... 'test', '--layer=122', '--profile=cProfile', '--subunit', ... '--profile-directory', tempdir, ... '-t', 'TestNotMuch'] >>> testrunner.run_internal(defaults) time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: samplelayers.Layer1:setUp ... time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: samplelayers.Layer1:tearDown test: zope:profiler_stats tags: zope:profiler_stats successful: zope:profiler_stats [ multipart Content-Type: application/x-binary-profile profiler-stats ...\r ... ] False >>> import shutil >>> shutil.rmtree(tempdir) Errors ------ Errors are recorded in the subunit stream as MIME-encoded chunks of text. >>> sys.argv = [ ... 'test', '--subunit' , '--tests-pattern', '^sampletests_e$', ... ] >>> testrunner.run_internal(defaults) time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: zope.testrunner.layer.UnitTests:setUp tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: zope.testrunner.layer.UnitTests:setUp tags: zope:layer:zope.testrunner.layer.UnitTests time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_e.eek time: YYYY-MM-DD HH:MM:SS.mmmmmmZ failure: sample2.sampletests_e.eek [ multipart Content-Type: text/x-traceback... traceback NNN\r Failed doctest test for sample2.sampletests_e.eek testrunner-ex/sample2/sampletests_e.py", Line NNN, in eek ---------------------------------------------------------------------- File testrunner-ex/sample2/sampletests_e.py", Line NNN, in sample2.sampletests_e.eek Failed example: f() Exception raised: Traceback (most recent call last): File "", Line NNN, in ? f() testrunner-ex/sample2/sampletests_e.py", Line NNN, in f g() testrunner-ex/sample2/sampletests_e.py", Line NNN, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined 0\r ] time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_e.Test.test1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample2.sampletests_e.Test.test1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_e.Test.test2 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample2.sampletests_e.Test.test2 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_e.Test.test3 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ error: sample2.sampletests_e.Test.test3 [ multipart Content-Type: text/x-traceback... traceback NNN\r Traceback (most recent call last): testrunner-ex/sample2/sampletests_e.py", Line NNN, in test3 f() testrunner-ex/sample2/sampletests_e.py", Line NNN, in f g() testrunner-ex/sample2/sampletests_e.py", Line NNN, in g x = y + 1 - __traceback_info__: I don't know what Y should be. NameError: global name 'y' is not defined 0\r ] time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_e.Test.test4 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample2.sampletests_e.Test.test4 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_e.Test.test5 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample2.sampletests_e.Test.test5 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: e_txt time: YYYY-MM-DD HH:MM:SS.mmmmmmZ failure: e_txt [ multipart Content-Type: text/x-traceback... traceback NNN\r Failed doctest test for e.txt testrunner-ex/sample2/e.txt", line 0 ---------------------------------------------------------------------- File testrunner-ex/sample2/e.txt", Line NNN, in e.txt Failed example: f() Exception raised: Traceback (most recent call last): File "", Line NNN, in ? f() File "", Line NNN, in f return x NameError: global name 'x' is not defined 0\r ] tags: -zope:layer:zope.testrunner.layer.UnitTests time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: zope.testrunner.layer.UnitTests:tearDown tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: zope.testrunner.layer.UnitTests:tearDown True Layers that can't be torn down ------------------------------ A layer can have a tearDown method that raises NotImplementedError. If this is the case and there are no remaining tests to run, the subunit stream will say that the layer skipped its tearDown. >>> defaults = [ ... '--subunit', ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test -ssample2 --tests-pattern sampletests_ntd$'.split() >>> testrunner.run_internal(defaults) time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_ntd.Layer:setUp tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample2.sampletests_ntd.Layer:setUp tags: zope:layer:sample2.sampletests_ntd.Layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_ntd.TestSomething.test_something time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample2.sampletests_ntd.TestSomething.test_something tags: -zope:layer:sample2.sampletests_ntd.Layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_ntd.Layer:tearDown tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ skip: sample2.sampletests_ntd.Layer:tearDown [ tearDown not supported ] False Module import errors -------------------- We report module import errors too. They get encoded as tests with errors. The name of the test is the module that could not be imported, the test's result is an error containing the traceback. These "tests" are tagged with zope:import_error. Let's create a module with some bad syntax: >>> badsyntax_path = os.path.join(directory_with_tests, ... "sample2", "sampletests_i.py") >>> f = open(badsyntax_path, "w") >>> print("importx unittest", file=f) # syntax error >>> f.close() And then run the tests: >>> sys.argv = ( ... 'test --subunit --tests-pattern ^sampletests(f|_i)?$ --layer 1 ' ... ).split() >>> testrunner.run_internal(defaults) test: sample2.sampletests_i tags: zope:import_error error: sample2.sampletests_i [ Traceback (most recent call last): File "/home/benji/workspace/all-the-trunks/zope.testrunner/src/zope/testrunner/testrunner-ex/sample2/sampletests_i.py", line 1 importx unittest ^ SyntaxError: invalid syntax ] test: sample2.sample21.sampletests_i tags: zope:import_error error: sample2.sample21.sampletests_i [ Traceback (most recent call last): File "/home/benji/workspace/all-the-trunks/zope.testrunner/src/zope/testrunner/testrunner-ex/sample2/sample21/sampletests_i.py", line 16, in import zope.testrunner.huh ImportError: No module named huh ] test: sample2.sample23.sampletests_i tags: zope:import_error error: sample2.sample23.sampletests_i [ Traceback (most recent call last): File "/home/benji/workspace/all-the-trunks/zope.testrunner/src/zope/testrunner/testrunner-ex/sample2/sample23/sampletests_i.py", line 17, in class Test(unittest.TestCase): File "/home/benji/workspace/all-the-trunks/zope.testrunner/src/zope/testrunner/testrunner-ex/sample2/sample23/sampletests_i.py", line 22, in Test raise TypeError('eek') TypeError: eek ] time: 2010-07-19 21:27:16.708260Z test: samplelayers.Layer1:setUp tags: zope:layer ... True Of course, because we care deeply about test isolation, we're going to have to delete the module with bad syntax now, lest it contaminate other tests or even future test runs. >>> os.unlink(badsyntax_path) Tests in subprocesses --------------------- If the tearDown method raises NotImplementedError and there are remaining layers to run, the test runner will restart itself as a new process, resuming tests where it left off: >>> sys.argv = [testrunner_script, '--tests-pattern', 'sampletests_ntd$'] >>> testrunner.run_internal(defaults) time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample1.sampletests_ntd.Layer:setUp tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample1.sampletests_ntd.Layer:setUp tags: zope:layer:sample1.sampletests_ntd.Layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample1.sampletests_ntd.TestSomething.test_something time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample1.sampletests_ntd.TestSomething.test_something tags: -zope:layer:sample1.sampletests_ntd.Layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample1.sampletests_ntd.Layer:tearDown tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ skip: sample1.sampletests_ntd.Layer:tearDown [ tearDown not supported ] test: Running in a subprocess. tags: zope:info_suboptimal successful: Running in a subprocess. time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_ntd.Layer:setUp tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample2.sampletests_ntd.Layer:setUp tags: zope:layer:sample2.sampletests_ntd.Layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_ntd.TestSomething.test_something time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample2.sampletests_ntd.TestSomething.test_something tags: -zope:layer:sample2.sampletests_ntd.Layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_ntd.Layer:tearDown tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ skip: sample2.sampletests_ntd.Layer:tearDown [ tearDown not supported ] test: Running in a subprocess. tags: zope:info_suboptimal successful: Running in a subprocess. time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.Layer:setUp tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample3.sampletests_ntd.Layer:setUp tags: zope:layer:sample3.sampletests_ntd.Layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_error1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ error: sample3.sampletests_ntd.TestSomething.test_error1 [ multipart Content-Type: text/x-traceback... traceback 14F\r Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error1 raise TypeError("Can we see errors") TypeError: Can we see errors 0\r ] time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_error2 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ error: sample3.sampletests_ntd.TestSomething.test_error2 [ multipart Content-Type: text/x-traceback... traceback 13F\r Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error2 raise TypeError("I hope so") TypeError: I hope so 0\r ] time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_fail1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ failure: sample3.sampletests_ntd.TestSomething.test_fail1 [ multipart Content-Type: text/x-traceback... traceback 1AA\r Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail1 self.assertEqual(1, 2) AssertionError: 1 != 2 0\r ] time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_fail2 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ failure: sample3.sampletests_ntd.TestSomething.test_fail2 [ multipart Content-Type: text/x-traceback... traceback 1AA\r Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail2 self.assertEqual(1, 3) AssertionError: 1 != 3 0\r ] time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_something time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample3.sampletests_ntd.TestSomething.test_something time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_something_else time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample3.sampletests_ntd.TestSomething.test_something_else tags: -zope:layer:sample3.sampletests_ntd.Layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.Layer:tearDown tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ skip: sample3.sampletests_ntd.Layer:tearDown [ tearDown not supported ] True Note that debugging doesn't work when running tests in a subprocess: >>> sys.argv = [testrunner_script, '--tests-pattern', 'sampletests_ntd$', ... '-D', ] >>> testrunner.run_internal(defaults) time: 2010-02-10 22:41:25.279692Z test: sample1.sampletests_ntd.Layer:setUp tags: zope:layer time: 2010-02-10 22:41:25.279695Z successful: sample1.sampletests_ntd.Layer:setUp tags: zope:layer:sample1.sampletests_ntd.Layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample1.sampletests_ntd.TestSomething.test_something time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample1.sampletests_ntd.TestSomething.test_something tags: -zope:layer:sample1.sampletests_ntd.Layer time: 2010-02-10 22:41:25.310078Z test: sample1.sampletests_ntd.Layer:tearDown tags: zope:layer time: 2010-02-10 22:41:25.310171Z skip: sample1.sampletests_ntd.Layer:tearDown [ tearDown not supported ] test: Running in a subprocess. tags: zope:info_suboptimal successful: Running in a subprocess. time: 2010-02-10 22:41:25.753076Z test: sample2.sampletests_ntd.Layer:setUp tags: zope:layer time: 2010-02-10 22:41:25.753079Z successful: sample2.sampletests_ntd.Layer:setUp tags: zope:layer:sample2.sampletests_ntd.Layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample2.sampletests_ntd.TestSomething.test_something time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample2.sampletests_ntd.TestSomething.test_something tags: -zope:layer:sample2.sampletests_ntd.Layer time: 2010-02-10 22:41:25.779256Z test: sample2.sampletests_ntd.Layer:tearDown tags: zope:layer time: 2010-02-10 22:41:25.779326Z skip: sample2.sampletests_ntd.Layer:tearDown [ tearDown not supported ] test: Running in a subprocess. tags: zope:info_suboptimal successful: Running in a subprocess. time: 2010-02-10 22:41:26.310296Z test: sample3.sampletests_ntd.Layer:setUp tags: zope:layer time: 2010-02-10 22:41:26.310299Z successful: sample3.sampletests_ntd.Layer:setUp tags: zope:layer:sample3.sampletests_ntd.Layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_error1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ error: sample3.sampletests_ntd.TestSomething.test_error1 [ multipart Content-Type: text/x-traceback... traceback 16A\r Traceback (most recent call last): File "/usr/lib/python2.6/unittest.py", line 305, in debug getattr(self, self._testMethodName)() File "/home/jml/src/zope.testrunner/subunit-output-formatter/src/zope/testing/testrunner/testrunner-ex/sample3/sampletests_ntd.py", line 42, in test_error1 raise TypeError("Can we see errors") TypeError: Can we see errors 0\r ] test: Can't post-mortem debug when running a layer as a subprocess! tags: zope:error_with_banner successful: Can't post-mortem debug when running a layer as a subprocess! time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_error2 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ error: sample3.sampletests_ntd.TestSomething.test_error2 [ multipart Content-Type: text/x-traceback... traceback 15A\r Traceback (most recent call last): File "/usr/lib/python2.6/unittest.py", line 305, in debug getattr(self, self._testMethodName)() File "/home/jml/src/zope.testrunner/subunit-output-formatter/src/zope/testing/testrunner/testrunner-ex/sample3/sampletests_ntd.py", line 45, in test_error2 raise TypeError("I hope so") TypeError: I hope so 0\r ] test: Can't post-mortem debug when running a layer as a subprocess! tags: zope:error_with_banner successful: Can't post-mortem debug when running a layer as a subprocess! time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_fail1 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ error: sample3.sampletests_ntd.TestSomething.test_fail1 [ multipart Content-Type: text/x-traceback... traceback 1C5\r Traceback (most recent call last): File "/usr/lib/python2.6/unittest.py", line 305, in debug getattr(self, self._testMethodName)() File "/home/jml/src/zope.testrunner/subunit-output-formatter/src/zope/testing/testrunner/testrunner-ex/sample3/sampletests_ntd.py", line 48, in test_fail1 self.assertEqual(1, 2) File "/usr/lib/python2.6/unittest.py", line 350, in failUnlessEqual (msg or '%r != %r' % (first, second)) AssertionError: 1 != 2 0\r ] test: Can't post-mortem debug when running a layer as a subprocess! tags: zope:error_with_banner successful: Can't post-mortem debug when running a layer as a subprocess! time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_fail2 time: YYYY-MM-DD HH:MM:SS.mmmmmmZ error: sample3.sampletests_ntd.TestSomething.test_fail2 [ multipart Content-Type: text/x-traceback... traceback 1C5\r Traceback (most recent call last): File "/usr/lib/python2.6/unittest.py", line 305, in debug getattr(self, self._testMethodName)() File "/home/jml/src/zope.testrunner/subunit-output-formatter/src/zope/testing/testrunner/testrunner-ex/sample3/sampletests_ntd.py", line 51, in test_fail2 self.assertEqual(1, 3) File "/usr/lib/python2.6/unittest.py", line 350, in failUnlessEqual (msg or '%r != %r' % (first, second)) AssertionError: 1 != 3 0\r ] test: Can't post-mortem debug when running a layer as a subprocess! tags: zope:error_with_banner successful: Can't post-mortem debug when running a layer as a subprocess! time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_something time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample3.sampletests_ntd.TestSomething.test_something time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: sample3.sampletests_ntd.TestSomething.test_something_else time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: sample3.sampletests_ntd.TestSomething.test_something_else tags: -zope:layer:sample3.sampletests_ntd.Layer time: 2010-02-10 22:41:26.340878Z test: sample3.sampletests_ntd.Layer:tearDown tags: zope:layer time: 2010-02-10 22:41:26.340945Z skip: sample3.sampletests_ntd.Layer:tearDown [ tearDown not supported ] True Support skipped tests --------------------- >>> directory_with_skipped_tests = os.path.join(this_directory, ... 'testrunner-ex-skip') >>> skip_defaults = [ ... '--path', directory_with_skipped_tests, ... '--tests-pattern', '^sample_skipped_tests$', ... ] >>> sys.argv = ['test'] >>> testrunner.run_internal( ... skip_defaults + ["--subunit", "-t", "TestSkipppedNoLayer"]) time: ... test: zope.testrunner.layer.UnitTests:setUp tags: zope:layer time: ... successful: zope.testrunner.layer.UnitTests:setUp tags: zope:layer:zope.testrunner.layer.UnitTests time: ... test: sample_skipped_tests.TestSkipppedNoLayer.test_skipped skip: sample_skipped_tests.TestSkipppedNoLayer.test_skipped [ I'm a skipped test! ] tags: -zope:layer:zope.testrunner.layer.UnitTests time: ... test: zope.testrunner.layer.UnitTests:tearDown tags: zope:layer time: ... successful: zope.testrunner.layer.UnitTests:tearDown False And remove the temporary directory: >>> shutil.rmtree(tmpdir) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-layers-ntd.txt0000664000000000000000000002642312527371226025604 0ustar rootrootLayers that can't be torn down ============================== A layer can have a tearDown method that raises NotImplementedError. If this is the case and there are no remaining tests to run, the test runner will just note that the tear down couldn't be done: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test -ssample2 --tests-pattern sampletests_ntd$'.split() >>> testrunner.run_internal(defaults) Running sample2.sampletests_ntd.Layer tests: Set up sample2.sampletests_ntd.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Tearing down left over layers: Tear down sample2.sampletests_ntd.Layer ... not supported False If the tearDown method raises NotImplementedError and there are remaining layers to run, the test runner will restart itself as a new process, resuming tests where it left off: >>> sys.argv = [testrunner_script, '--tests-pattern', 'sampletests_ntd$'] >>> testrunner.run_internal(defaults) Running sample1.sampletests_ntd.Layer tests: Set up sample1.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running sample2.sampletests_ntd.Layer tests: Tear down sample1.sampletests_ntd.Layer ... not supported Running in a subprocess. Set up sample2.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down sample2.sampletests_ntd.Layer ... not supported Running sample3.sampletests_ntd.Layer tests: Running in a subprocess. Set up sample3.sampletests_ntd.Layer in N.NNN seconds. Error in test test_error1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error1 raise TypeError("Can we see errors") TypeError: Can we see errors Error in test test_error2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error2 raise TypeError("I hope so") TypeError: I hope so Failure in test test_fail1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail1 self.assertEqual(1, 2) AssertionError: 1 != 2 Failure in test test_fail2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail2 self.assertEqual(1, 3) AssertionError: 1 != 3 Ran 6 tests with 2 failures, 2 errors and 0 skipped in N.NNN seconds. Tear down sample3.sampletests_ntd.Layer ... not supported Total: 8 tests, 2 failures, 2 errors and 0 skipped in N.NNN seconds. True in the example above, some of the tests run as a subprocess had errors and failures. They were displayed as usual and the failure and error statistice were updated as usual. Note that debugging doesn't work when running tests in a subprocess: >>> sys.argv = [testrunner_script, '--tests-pattern', 'sampletests_ntd$', ... '-D', ] >>> testrunner.run_internal(defaults) Running sample1.sampletests_ntd.Layer tests: Set up sample1.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running sample2.sampletests_ntd.Layer tests: Tear down sample1.sampletests_ntd.Layer ... not supported Running in a subprocess. Set up sample2.sampletests_ntd.Layer in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down sample2.sampletests_ntd.Layer ... not supported Running sample3.sampletests_ntd.Layer tests: Running in a subprocess. Set up sample3.sampletests_ntd.Layer in N.NNN seconds. Error in test test_error1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error1 raise TypeError("Can we see errors") TypeError: Can we see errors ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Error in test test_error2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_error2 raise TypeError("I hope so") TypeError: I hope so ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Error in test test_fail1 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail1 self.assertEqual(1, 2) AssertionError: 1 != 2 ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Error in test test_fail2 (sample3.sampletests_ntd.TestSomething) Traceback (most recent call last): testrunner-ex/sample3/sampletests_ntd.py", Line NNN, in test_fail2 self.assertEqual(1, 3) AssertionError: 1 != 3 ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! ********************************************************************** Ran 6 tests with 0 failures, 4 errors and 0 skipped in N.NNN seconds. Tear down sample3.sampletests_ntd.Layer ... not supported Total: 8 tests, 0 failures, 4 errors and 0 skipped in N.NNN seconds. True Similarly, pdb.set_trace doesn't work when running tests in a layer that is run as a subprocess: >>> sys.argv = [testrunner_script, '--tests-pattern', 'sampletests_ntds'] >>> testrunner.run_internal(defaults) Running sample1.sampletests_ntds.Layer tests: Set up sample1.sampletests_ntds.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Running sample2.sampletests_ntds.Layer tests: Tear down sample1.sampletests_ntds.Layer ... not supported Running in a subprocess. Set up sample2.sampletests_ntds.Layer in 0.000 seconds. --Return-- > testrunner-ex/sample2/sampletests_ntds.py(37)test_something()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(40)test_something2()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(43)test_something3()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(46)test_something4()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > testrunner-ex/sample2/sampletests_ntds.py(52)f()->None -> import pdb; pdb.set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > doctest.py(351)set_trace()->None -> Pdb().set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** --Return-- > doctest.py(351)set_trace()->None -> Pdb().set_trace() (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** Ran 7 tests with 0 failures, 0 errors and 0 skipped in 0.008 seconds. Tear down sample2.sampletests_ntds.Layer ... not supported Total: 8 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False If you want to use pdb from a test in a layer that is run as a subprocess, then rerun the test runner selecting *just* that layer so that it's not run as a subprocess. If a test is run in a subprocess and it generates output on stderr (as stderrtest does), the output is ignored (but it doesn't cause a SubprocessError like it once did). >>> from six import StringIO >>> real_stderr = sys.stderr >>> sys.stderr = StringIO() >>> sys.argv = [testrunner_script, '-s', 'sample2', '--tests-pattern', ... '(sampletests_ntd$|stderrtest)'] >>> testrunner.run_internal(defaults) Running sample2.sampletests_ntd.Layer tests: Set up sample2.sampletests_ntd.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Running sample2.stderrtest.Layer tests: Tear down sample2.sampletests_ntd.Layer ... not supported Running in a subprocess. Set up sample2.stderrtest.Layer in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.002 seconds. Tear down sample2.stderrtest.Layer in 0.000 seconds. Total: 2 tests, 0 failures, 0 errors and 0 skipped in 0.197 seconds. False >>> print((sys.stderr.getvalue())) A message on stderr. Please ignore (expected in test output). >>> sys.stderr = real_stderr zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-layers-api.txt0000664000000000000000000002142012527371226025560 0ustar rootrootLayers ====== A Layer is an object providing setup and teardown methods used to setup and teardown the environment provided by the layer. It may also provide setup and teardown methods used to reset the environment provided by the layer between each test. Layers are generally implemented as classes using class methods. >>> class BaseLayer: ... def setUp(cls): ... log('BaseLayer.setUp') ... setUp = classmethod(setUp) ... ... def tearDown(cls): ... log('BaseLayer.tearDown') ... tearDown = classmethod(tearDown) ... ... def testSetUp(cls): ... log('BaseLayer.testSetUp') ... testSetUp = classmethod(testSetUp) ... ... def testTearDown(cls): ... log('BaseLayer.testTearDown') ... testTearDown = classmethod(testTearDown) ... Layers can extend other layers. Note that they do not explicitly invoke the setup and teardown methods of other layers - the test runner does this for us in order to minimize the number of invocations. >>> class TopLayer(BaseLayer): ... def setUp(cls): ... log('TopLayer.setUp') ... setUp = classmethod(setUp) ... ... def tearDown(cls): ... log('TopLayer.tearDown') ... tearDown = classmethod(tearDown) ... ... def testSetUp(cls): ... log('TopLayer.testSetUp') ... testSetUp = classmethod(testSetUp) ... ... def testTearDown(cls): ... log('TopLayer.testTearDown') ... testTearDown = classmethod(testTearDown) ... Tests or test suites specify what layer they need by storing a reference in the 'layer' attribute. >>> import unittest >>> class TestSpecifyingBaseLayer(unittest.TestCase): ... 'This TestCase explicitly specifies its layer' ... layer = BaseLayer ... name = 'TestSpecifyingBaseLayer' # For testing only ... ... def setUp(self): ... log('TestSpecifyingBaseLayer.setUp') ... ... def tearDown(self): ... log('TestSpecifyingBaseLayer.tearDown') ... ... def test1(self): ... log('TestSpecifyingBaseLayer.test1') ... ... def test2(self): ... log('TestSpecifyingBaseLayer.test2') ... >>> class TestSpecifyingNoLayer(unittest.TestCase): ... 'This TestCase specifies no layer' ... name = 'TestSpecifyingNoLayer' # For testing only ... def setUp(self): ... log('TestSpecifyingNoLayer.setUp') ... ... def tearDown(self): ... log('TestSpecifyingNoLayer.tearDown') ... ... def test1(self): ... log('TestSpecifyingNoLayer.test') ... ... def test2(self): ... log('TestSpecifyingNoLayer.test') ... Create a TestSuite containing two test suites, one for each of TestSpecifyingBaseLayer and TestSpecifyingNoLayer. >>> umbrella_suite = unittest.TestSuite() >>> umbrella_suite.addTest(unittest.makeSuite(TestSpecifyingBaseLayer)) >>> no_layer_suite = unittest.makeSuite(TestSpecifyingNoLayer) >>> umbrella_suite.addTest(no_layer_suite) Before we can run the tests, we need to setup some helpers. >>> from zope.testrunner import options >>> from zope.testing.loggingsupport import InstalledHandler >>> import logging >>> log_handler = InstalledHandler('zope.testrunner.tests') >>> def log(msg): ... logging.getLogger('zope.testrunner.tests').info(msg) >>> def fresh_options(): ... opts = options.get_options(['--test-filter', '.*']) ... opts.resume_layer = None ... opts.resume_number = 0 ... return opts Now we run the tests. Note that the BaseLayer was not setup when the TestSpecifyingNoLayer was run and setup/torn down around the TestSpecifyingBaseLayer tests. >>> from zope.testrunner.runner import Runner >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running ...BaseLayer tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up ...BaseLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...BaseLayer in N.NNN seconds. Total: 4 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. Now lets specify a layer in the suite containing TestSpecifyingNoLayer and run the tests again. This demonstrates the other method of specifying a layer. This is generally how you specify what layer doctests need. >>> no_layer_suite.layer = BaseLayer >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running ...BaseLayer tests: Set up ...BaseLayer in N.NNN seconds. Ran 4 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...BaseLayer in N.NNN seconds. Clear our logged output, as we want to inspect it shortly. >>> log_handler.clear() Now lets also specify a layer in the TestSpecifyingNoLayer class and rerun the tests. This demonstrates that the most specific layer is used. It also shows the behavior of nested layers - because TopLayer extends BaseLayer, both the BaseLayer and TopLayer environments are setup when the TestSpecifyingNoLayer tests are run. >>> TestSpecifyingNoLayer.layer = TopLayer >>> runner = Runner(options=fresh_options(), args=[], found_suites=[umbrella_suite]) >>> succeeded = runner.run() Running ...BaseLayer tests: Set up ...BaseLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running ...TopLayer tests: Set up ...TopLayer in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...TopLayer in N.NNN seconds. Tear down ...BaseLayer in N.NNN seconds. Total: 4 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. If we inspect our trace of what methods got called in what order, we can see that the layer setup and teardown methods only got called once. We can also see that the layer's test setup and teardown methods got called for each test using that layer in the right order. >>> def report(): ... print("Report:") ... for record in log_handler.records: ... print(record.getMessage()) >>> report() Report: BaseLayer.setUp BaseLayer.testSetUp TestSpecifyingBaseLayer.setUp TestSpecifyingBaseLayer.test1 TestSpecifyingBaseLayer.tearDown BaseLayer.testTearDown BaseLayer.testSetUp TestSpecifyingBaseLayer.setUp TestSpecifyingBaseLayer.test2 TestSpecifyingBaseLayer.tearDown BaseLayer.testTearDown TopLayer.setUp BaseLayer.testSetUp TopLayer.testSetUp TestSpecifyingNoLayer.setUp TestSpecifyingNoLayer.test TestSpecifyingNoLayer.tearDown TopLayer.testTearDown BaseLayer.testTearDown BaseLayer.testSetUp TopLayer.testSetUp TestSpecifyingNoLayer.setUp TestSpecifyingNoLayer.test TestSpecifyingNoLayer.tearDown TopLayer.testTearDown BaseLayer.testTearDown TopLayer.tearDown BaseLayer.tearDown Now lets stack a few more layers to ensure that our setUp and tearDown methods are called in the correct order. >>> from zope.testrunner.find import name_from_layer >>> class A(object): ... def setUp(cls): ... log('%s.setUp' % name_from_layer(cls)) ... setUp = classmethod(setUp) ... ... def tearDown(cls): ... log('%s.tearDown' % name_from_layer(cls)) ... tearDown = classmethod(tearDown) ... ... def testSetUp(cls): ... log('%s.testSetUp' % name_from_layer(cls)) ... testSetUp = classmethod(testSetUp) ... ... def testTearDown(cls): ... log('%s.testTearDown' % name_from_layer(cls)) ... testTearDown = classmethod(testTearDown) ... >>> class B(A): pass >>> class C(B): pass >>> class D(A): pass >>> class E(D): pass >>> class F(C,E): pass >>> class DeepTest(unittest.TestCase): ... layer = F ... def test(self): ... pass >>> suite = unittest.makeSuite(DeepTest) >>> log_handler.clear() >>> runner = Runner(options=fresh_options(), args=[], found_suites=[suite]) >>> succeeded = runner.run() #doctest: +ELLIPSIS Running ...F tests: Set up ...A in N.NNN seconds. Set up ...B in N.NNN seconds. Set up ...C in N.NNN seconds. Set up ...D in N.NNN seconds. Set up ...E in N.NNN seconds. Set up ...F in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...F in N.NNN seconds. Tear down ...E in N.NNN seconds. Tear down ...D in N.NNN seconds. Tear down ...C in N.NNN seconds. Tear down ...B in N.NNN seconds. Tear down ...A in N.NNN seconds. >>> report() #doctest: +ELLIPSIS Report: ...A.setUp ...B.setUp ...C.setUp ...D.setUp ...E.setUp ...F.setUp ...A.testSetUp ...B.testSetUp ...C.testSetUp ...D.testSetUp ...E.testSetUp ...F.testSetUp ...F.testTearDown ...E.testTearDown ...D.testTearDown ...C.testTearDown ...B.testTearDown ...A.testTearDown ...F.tearDown ...E.tearDown ...D.tearDown ...C.tearDown ...B.tearDown ...A.tearDown zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-eggsupport.txt0000664000000000000000000000760712527371226025724 0ustar rootrootEgg Support =========== The `ftest` Setup Command ------------------------- The `ftest` class is a proper `distutils` command and `zope.testrunner` exposes it as such via an entry point. >>> import pkg_resources >>> ws = pkg_resources.WorkingSet() >>> eps = dict([(ep.name, ep) ... for ep in ws.iter_entry_points('distutils.commands')]) >>> 'ftest' in eps True >>> eps['ftest'] EntryPoint.parse('ftest = zope.testrunner.eggsupport:ftest') Let's now run this command: >>> import zope.testrunner >>> org_run = zope.testrunner.run >>> def run(args): ... print(' '.join(args)) >>> zope.testrunner.run = run >>> import os, tempfile, shutil >>> tmpdir = tempfile.mkdtemp() >>> srcdir = os.path.join(tmpdir, 'src') >>> os.mkdir(srcdir) >>> import setuptools.dist >>> dist = setuptools.dist.Distribution( ... {'package_dir': {'': srcdir}, ... 'script_name': __file__}) >>> from zope.testrunner.eggsupport import ftest >>> ftest(dist).run() IGNORE_ME --test-path .../src Cleanup: >>> zope.testrunner.run = org_run >>> shutil.rmtree(tmpdir) Skipping Tests with Layers -------------------------- The ``SkipLayers`` scanning test loader can replace the standard test loader, so that any tests that require layers are skipped. This is necessary, since the standard setuptools testing facility does not handle layers. It can be used as follows:: setup( ... setup_requires=['eggtestinfo' # captures testing metadata in EGG-INFO ], tests_require=['zope.testrunner', ], ... test_loader='zope.testrunner.eggsupport:SkipLayers', ... ) Let's now crete some test suites to make sure that all tests with layers are properly skipped. >>> import doctest >>> import unittest >>> all = unittest.TestSuite() >>> class T1(unittest.TestCase): ... def test_t1(self): ... pass >>> class T2(unittest.TestCase): ... layer = 'layer' ... def test_t2(self): ... pass >>> T3 = doctest.DocTestSuite('zope.testrunner.find') >>> T4 = doctest.DocTestSuite('zope.testrunner.options') >>> T4.layer = 'layer' >>> T5 = doctest.DocFileSuite('testrunner.txt', package='zope.testrunner.tests') >>> T6 = doctest.DocFileSuite('testrunner-gc.txt', package='zope.testrunner.tests') >>> T6.layer = 'layer' >>> all = unittest.TestSuite(( ... unittest.makeSuite(T1), unittest.makeSuite(T2), T3, T4, T5, T6, ... )) Let's return those tests from the scan: >>> from setuptools.command.test import ScanningLoader >>> orig_loadTestsFromModule = ScanningLoader.loadTestsFromModule >>> ScanningLoader.loadTestsFromModule = lambda *args: all Now we can retrieve the modules from the layer skipping loader: >>> from zope.testrunner.eggsupport import SkipLayers >>> filtered = SkipLayers().loadTestsFromModule('zope.testrunner') >>> len(filtered._tests) 3 >>> from pprint import pprint >>> pprint(filtered._tests) [<...T1 testMethod=test_t1>, StartUpFailure (zope.testrunner.find), .../zope/testrunner/tests/testrunner.txt] Cleanup: >>> ScanningLoader.loadTestsFromModule = orig_loadTestsFromModule When the distribution specified a ``test_suite``, another method is used to load the tests. >>> orig_loadTestsFromNames = ScanningLoader.loadTestsFromNames >>> ScanningLoader.loadTestsFromNames = lambda *args: all Now we can retrieve the modules from the layer skipping loader: >>> from zope.testrunner.eggsupport import SkipLayers >>> filtered = SkipLayers().loadTestsFromNames( ... 'zope.testrunner.tests.test_suite', 'zope.testrunner') >>> len(filtered._tests) 3 >>> from pprint import pprint >>> pprint(filtered._tests) [<...T1 testMethod=test_t1>, StartUpFailure (zope.testrunner.find), .../zope/testrunner/tests/testrunner.txt] Cleanup: >>> ScanningLoader.loadTestsFromNames = orig_loadTestsFromNames zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner.txt0000664000000000000000000000504412527371226023520 0ustar rootrootTest Runner =========== The testrunner module is used to run automated tests defined using the unittest framework. Its primary feature is that it *finds* tests by searching directory trees. It doesn't require the manual concatenation of specific test suites. It is highly customizable and should be usable with any project. In addition to finding and running tests, it provides the following additional features: - Test filtering using specifications of: o test packages within a larger tree o regular expression patterns for test modules o regular expression patterns for individual tests - Organization of tests into levels and layers Sometimes, tests take so long to run that you don't want to run them on every run of the test runner. Tests can be defined at different levels. The test runner can be configured to only run tests at a specific level or below by default. Command-line options can be used to specify a minimum level to use for a specific run, or to run all tests. Individual tests or test suites can specify their level via a 'level' attribute. where levels are integers increasing from 1. Most tests are unit tests. They don't depend on other facilities, or set up whatever dependencies they have. For larger applications, it's useful to specify common facilities that a large number of tests share. Making each test set up and and tear down these facilities is both ineffecient and inconvenient. For this reason, we've introduced the concept of layers, based on the idea of layered application architectures. Software build for a layer should be able to depend on the facilities of lower layers already being set up. For example, Zope defines a component architecture. Much Zope software depends on that architecture. We should be able to treat the component architecture as a layer that we set up once and reuse. Similarly, Zope application software should be able to depend on the Zope application server without having to set it up in each test. The test runner introduces test layers, which are objects that can set up environments for tests within the layers to use. A layer is set up before running the tests in it. Individual tests or test suites can define a layer by defining a `layer` attribute, which is a test layer. - Reporting - progress meter - summaries of tests run - Analysis of test execution - post-mortem debugging of test failures - memory leaks - code coverage - source analysis using pychecker - memory errors - execution times - profiling zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-6/0000775000000000000000000000000012552256734023535 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-6/cwdtests.py0000664000000000000000000000044312527371226025744 0ustar rootrootimport unittest import os class Layer1: pass class Layer2: pass class Test1(unittest.TestCase): layer = Layer1 def test_that_chdirs(self): os.chdir(os.path.dirname(__file__)) class Test2(unittest.TestCase): layer = Layer2 def test(self): pass zope.testrunner-4.4.9/src/zope/testrunner/tests/test_filter.py0000664000000000000000000000545712527371226023454 0ustar rootroot############################################################################## # # Copyright (c) 2010 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Unit tests for the testrunner's filtering functions """ import unittest from zope.testrunner import filter class TestFilterMakerFunction(unittest.TestCase): def test_no_filters(self): accept = filter.build_filtering_func([]) self.assertFalse(accept('test_xx')) self.assertFalse(accept('test_yy')) self.assertFalse(accept('test_zz')) def test_match_all_filter(self): accept = filter.build_filtering_func(['.']) self.assertTrue(accept('test_xx')) self.assertTrue(accept('test_yy')) self.assertTrue(accept('test_zz')) def test_select_specific_pattern(self): accept = filter.build_filtering_func(['xx']) self.assertTrue(accept('test_xx')) self.assertFalse(accept('test_yy')) self.assertFalse(accept('test_zz')) def test_select_several_patterns(self): accept = filter.build_filtering_func(['xx', 'yy']) self.assertTrue(accept('test_xx')) self.assertTrue(accept('test_yy')) self.assertFalse(accept('test_zz')) def test_reject_one_pattern(self): accept = filter.build_filtering_func(['!xx']) self.assertFalse(accept('test_xx')) self.assertTrue(accept('test_yy')) self.assertTrue(accept('test_zz')) def test_reject_several_patterns(self): accept = filter.build_filtering_func(['!xx', '!zz']) self.assertFalse(accept('test_xx')) self.assertTrue(accept('test_yy')) self.assertFalse(accept('test_zz')) def test_accept_and_reject(self): accept = filter.build_filtering_func(['!xx', 'yy']) self.assertFalse(accept('test_xx')) self.assertTrue(accept('test_yy')) self.assertFalse(accept('test_zz')) def test_accept_and_reject_overlap(self): accept = filter.build_filtering_func(['!test_', 'yy']) self.assertFalse(accept('test_xx')) self.assertFalse(accept('test_yy')) self.assertFalse(accept('test_zz')) def test_accept_and_reject_same(self): accept = filter.build_filtering_func(['yy', '!yy']) self.assertFalse(accept('test_xx')) self.assertFalse(accept('test_yy')) self.assertFalse(accept('test_zz')) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-profiling-cprofiler.txt0000664000000000000000000000267112527371226027475 0ustar rootrootProfiling ========= The testrunner includes the ability to profile the test execution with cProfile via the `--profile=cProfile` option:: >>> import os, sys, tempfile >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> sys.path.append(directory_with_tests) >>> tempdir = tempfile.mkdtemp(prefix='zope.testrunner-') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... '--profile-directory', tempdir, ... ] >>> sys.argv = [testrunner_script, '--profile=cProfile'] When the tests are run, we get profiling output:: >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running... ... ncalls tottime percall cumtime percall filename:lineno(function)... ... Profiling also works across layers:: >>> sys.argv = [testrunner_script, '-ssample2', '--profile=cProfile', ... '--tests-pattern', 'sampletests_ntd'] >>> testrunner.run_internal(defaults) Running... Tear down ... not supported... ncalls tottime percall cumtime percall filename:lineno(function)... The testrunner creates temnporary files containing cProfiler profiler data:: >>> os.listdir(tempdir) ['tests_profile.cZj2jt.prof', 'tests_profile.yHD-so.prof'] It deletes these when rerun. We'll delete these ourselves:: >>> import shutil >>> shutil.rmtree(tempdir) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-simple.txt0000664000000000000000000001011712527371226025004 0ustar rootrootSimple Usage ============ The test runner consists of an importable module. The test runner is used by providing scripts that import and invoke the `run` method from the module. The `testrunner` module is controlled via command-line options. Test scripts supply base and default options by supplying a list of default command-line options that are processed before the user-supplied command-line options are provided. Typically, a test script does 2 things: - Adds the directory containing the zope package to the Python path. - Calls the test runner with default arguments and arguments supplied to the script. Normally, it just passes default/setup arguments. The test runner uses `sys.argv` to get the user's input. This testrunner_ex subdirectory contains a number of sample packages with tests. Let's run the tests found here. First though, we'll set up our default options: >>> import os.path >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] The default options are used by a script to customize the test runner for a particular application. In this case, we use two options: path Set the path where the test runner should look for tests. This path is also added to the Python path. tests-pattern Tell the test runner how to recognize modules or packages containing tests. Now, if we run the tests, without any other options: >>> from zope import testrunner >>> import sys >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False we see the normal testrunner output, which summarizes the tests run for each layer. For each layer, we see what layers had to be torn down or set up to run the layer and we see the number of tests run, with results. The test runner returns a boolean indicating whether there were errors. In this example, there were no errors, so it returned False. (Of course, the times shown in these examples are just examples. Times will vary depending on system speed.) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-test-selection.txt0000664000000000000000000006327212527371226026467 0ustar rootrootTest Selection ============== We've already seen that we can select tests by layer. There are three other ways we can select tests. We can select tests by package: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 122 -ssample1 -vv'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122.TestA) test_z0 (sample1.sampletests.test122.TestA) test_x0 (sample1.sampletests.test122.TestB) test_y1 (sample1.sampletests.test122.TestB) test_z0 (sample1.sampletests.test122.TestB) test_1 (sample1.sampletests.test122.TestNotMuch) test_2 (sample1.sampletests.test122.TestNotMuch) test_3 (sample1.sampletests.test122.TestNotMuch) test_x0 (sample1.sampletests.test122) test_y0 (sample1.sampletests.test122) test_z1 (sample1.sampletests.test122) testrunner-ex/sample1/sampletests/../../sampletestsl.txt Ran 13 tests with 0 failures, 0 errors and 0 skipped in 0.005 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False You can specify multiple packages: >>> sys.argv = 'test -u -vv -ssample1 -ssample2'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf.TestA) test_z0 (sample1.sampletestsf.TestA) test_x0 (sample1.sampletestsf.TestB) test_y1 (sample1.sampletestsf.TestB) test_z0 (sample1.sampletestsf.TestB) test_1 (sample1.sampletestsf.TestNotMuch) test_2 (sample1.sampletestsf.TestNotMuch) test_3 (sample1.sampletestsf.TestNotMuch) test_x0 (sample1.sampletestsf) test_y0 (sample1.sampletestsf) test_z1 (sample1.sampletestsf) testrunner-ex/sample1/../sampletests.txt test_x1 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sample11.sampletests.TestA) test_z0 (sample1.sample11.sampletests.TestA) test_x0 (sample1.sample11.sampletests.TestB) test_y1 (sample1.sample11.sampletests.TestB) test_z0 (sample1.sample11.sampletests.TestB) test_1 (sample1.sample11.sampletests.TestNotMuch) test_2 (sample1.sample11.sampletests.TestNotMuch) test_3 (sample1.sample11.sampletests.TestNotMuch) test_x0 (sample1.sample11.sampletests) test_y0 (sample1.sample11.sampletests) test_z1 (sample1.sample11.sampletests) testrunner-ex/sample1/sample11/../../sampletests.txt test_x1 (sample1.sample13.sampletests.TestA) test_y0 (sample1.sample13.sampletests.TestA) test_z0 (sample1.sample13.sampletests.TestA) test_x0 (sample1.sample13.sampletests.TestB) test_y1 (sample1.sample13.sampletests.TestB) test_z0 (sample1.sample13.sampletests.TestB) test_1 (sample1.sample13.sampletests.TestNotMuch) test_2 (sample1.sample13.sampletests.TestNotMuch) test_3 (sample1.sample13.sampletests.TestNotMuch) test_x0 (sample1.sample13.sampletests) test_y0 (sample1.sample13.sampletests) test_z1 (sample1.sample13.sampletests) testrunner-ex/sample1/sample13/../../sampletests.txt test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample2.sample21.sampletests.TestA) test_y0 (sample2.sample21.sampletests.TestA) test_z0 (sample2.sample21.sampletests.TestA) test_x0 (sample2.sample21.sampletests.TestB) test_y1 (sample2.sample21.sampletests.TestB) test_z0 (sample2.sample21.sampletests.TestB) test_1 (sample2.sample21.sampletests.TestNotMuch) test_2 (sample2.sample21.sampletests.TestNotMuch) test_3 (sample2.sample21.sampletests.TestNotMuch) test_x0 (sample2.sample21.sampletests) test_y0 (sample2.sample21.sampletests) test_z1 (sample2.sample21.sampletests) testrunner-ex/sample2/sample21/../../sampletests.txt test_x1 (sample2.sampletests.test_1.TestA) test_y0 (sample2.sampletests.test_1.TestA) test_z0 (sample2.sampletests.test_1.TestA) test_x0 (sample2.sampletests.test_1.TestB) test_y1 (sample2.sampletests.test_1.TestB) test_z0 (sample2.sampletests.test_1.TestB) test_1 (sample2.sampletests.test_1.TestNotMuch) test_2 (sample2.sampletests.test_1.TestNotMuch) test_3 (sample2.sampletests.test_1.TestNotMuch) test_x0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.test_1) test_z1 (sample2.sampletests.test_1) testrunner-ex/sample2/sampletests/../../sampletests.txt test_x1 (sample2.sampletests.testone.TestA) test_y0 (sample2.sampletests.testone.TestA) test_z0 (sample2.sampletests.testone.TestA) test_x0 (sample2.sampletests.testone.TestB) test_y1 (sample2.sampletests.testone.TestB) test_z0 (sample2.sampletests.testone.TestB) test_1 (sample2.sampletests.testone.TestNotMuch) test_2 (sample2.sampletests.testone.TestNotMuch) test_3 (sample2.sampletests.testone.TestNotMuch) test_x0 (sample2.sampletests.testone) test_y0 (sample2.sampletests.testone) test_z1 (sample2.sampletests.testone) testrunner-ex/sample2/sampletests/../../sampletests.txt Ran 104 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False You can specify directory names instead of packages (useful for tab-completion): >>> subdir = os.path.join(directory_with_tests, 'sample1') >>> sys.argv = ['test', '--layer', '122', '-s', subdir, '-vv'] >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122.TestA) test_z0 (sample1.sampletests.test122.TestA) test_x0 (sample1.sampletests.test122.TestB) test_y1 (sample1.sampletests.test122.TestB) test_z0 (sample1.sampletests.test122.TestB) test_1 (sample1.sampletests.test122.TestNotMuch) test_2 (sample1.sampletests.test122.TestNotMuch) test_3 (sample1.sampletests.test122.TestNotMuch) test_x0 (sample1.sampletests.test122) test_y0 (sample1.sampletests.test122) test_z1 (sample1.sampletests.test122) testrunner-ex/sample1/sampletests/../../sampletestsl.txt Ran 13 tests with 0 failures, 0 errors and 0 skipped in 0.005 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False We can select by test module name using the --module (-m) option: >>> sys.argv = 'test -u -vv -ssample1 -m_one -mtest1'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False and by test within the module using the --test (-t) option: >>> sys.argv = 'test -u -vv -ssample1 -m_one -mtest1 -tx0 -ty0'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) Ran 8 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False >>> sys.argv = 'test -u -vv -ssample1 -ttxt'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: testrunner-ex/sample1/../sampletests.txt testrunner-ex/sample1/sample11/../../sampletests.txt testrunner-ex/sample1/sample13/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 5 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False The --module and --test options take regular expressions. If the regular expressions specified begin with '!', then tests that don't match the regular expression are selected: >>> sys.argv = 'test -u -vv -ssample1 -m!sample1[.]sample1'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf.TestA) test_z0 (sample1.sampletestsf.TestA) test_x0 (sample1.sampletestsf.TestB) test_y1 (sample1.sampletestsf.TestB) test_z0 (sample1.sampletestsf.TestB) test_1 (sample1.sampletestsf.TestNotMuch) test_2 (sample1.sampletestsf.TestNotMuch) test_3 (sample1.sampletestsf.TestNotMuch) test_x0 (sample1.sampletestsf) test_y0 (sample1.sampletestsf) test_z1 (sample1.sampletestsf) testrunner-ex/sample1/../sampletests.txt test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 39 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Module and test filters can also be given as positional arguments: >>> sys.argv = 'test -u -vv -ssample1 !sample1[.]sample1'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_x1 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf.TestA) test_z0 (sample1.sampletestsf.TestA) test_x0 (sample1.sampletestsf.TestB) test_y1 (sample1.sampletestsf.TestB) test_z0 (sample1.sampletestsf.TestB) test_1 (sample1.sampletestsf.TestNotMuch) test_2 (sample1.sampletestsf.TestNotMuch) test_3 (sample1.sampletestsf.TestNotMuch) test_x0 (sample1.sampletestsf) test_y0 (sample1.sampletestsf) test_z1 (sample1.sampletestsf) testrunner-ex/sample1/../sampletests.txt test_x1 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1.TestA) test_z0 (sample1.sampletests.test1.TestA) test_x0 (sample1.sampletests.test1.TestB) test_y1 (sample1.sampletests.test1.TestB) test_z0 (sample1.sampletests.test1.TestB) test_1 (sample1.sampletests.test1.TestNotMuch) test_2 (sample1.sampletests.test1.TestNotMuch) test_3 (sample1.sampletests.test1.TestNotMuch) test_x0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1) test_z1 (sample1.sampletests.test1) testrunner-ex/sample1/sampletests/../../sampletests.txt test_x1 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one.TestA) test_z0 (sample1.sampletests.test_one.TestA) test_x0 (sample1.sampletests.test_one.TestB) test_y1 (sample1.sampletests.test_one.TestB) test_z0 (sample1.sampletests.test_one.TestB) test_1 (sample1.sampletests.test_one.TestNotMuch) test_2 (sample1.sampletests.test_one.TestNotMuch) test_3 (sample1.sampletests.test_one.TestNotMuch) test_x0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test_one) test_z1 (sample1.sampletests.test_one) testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 39 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False >>> sys.argv = 'test -u -vv -ssample1 . txt'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: testrunner-ex/sample1/../sampletests.txt testrunner-ex/sample1/sample11/../../sampletests.txt testrunner-ex/sample1/sample13/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt testrunner-ex/sample1/sampletests/../../sampletests.txt Ran 5 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Sometimes, There are tests that you don't want to run by default. For example, you might have tests that take a long time. Tests can have a level attribute. If no level is specified, a level of 1 is assumed and, by default, only tests at level one are run. to run tests at a higher level, use the --at-level (-a) option to specify a higher level. For example, with the following options: >>> sys.argv = 'test -u -vv -t test_y1 -t test_y0'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sampletestsf.TestA) test_y1 (sampletestsf.TestB) test_y0 (sampletestsf) test_y0 (sample1.sampletestsf.TestA) test_y1 (sample1.sampletestsf.TestB) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y1 (sample1.sample11.sampletests.TestB) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y1 (sample1.sample13.sampletests.TestB) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y1 (sample1.sampletests.test1.TestB) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y1 (sample1.sampletests.test_one.TestB) test_y0 (sample1.sampletests.test_one) test_y0 (sample2.sample21.sampletests.TestA) test_y1 (sample2.sample21.sampletests.TestB) test_y0 (sample2.sample21.sampletests) test_y0 (sample2.sampletests.test_1.TestA) test_y1 (sample2.sampletests.test_1.TestB) test_y0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.testone.TestA) test_y1 (sample2.sampletests.testone.TestB) test_y0 (sample2.sampletests.testone) test_y0 (sample3.sampletests.TestA) test_y1 (sample3.sampletests.TestB) test_y0 (sample3.sampletests) test_y0 (sampletests.test1.TestA) test_y1 (sampletests.test1.TestB) test_y0 (sampletests.test1) test_y0 (sampletests.test_one.TestA) test_y1 (sampletests.test_one.TestB) test_y0 (sampletests.test_one) Ran 36 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False We get run 36 tests. If we specify a level of 2, we get some additional tests: >>> sys.argv = 'test -u -vv -a 2 -t test_y1 -t test_y0'.split() >>> testrunner.run_internal(defaults) Running tests at level 2 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sampletestsf.TestA) test_y0 (sampletestsf.TestA2) test_y1 (sampletestsf.TestB) test_y0 (sampletestsf) test_y0 (sample1.sampletestsf.TestA) test_y1 (sample1.sampletestsf.TestB) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y1 (sample1.sample11.sampletests.TestB) test_y1 (sample1.sample11.sampletests.TestB2) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y1 (sample1.sample13.sampletests.TestB) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y1 (sample1.sampletests.test1.TestB) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y1 (sample1.sampletests.test_one.TestB) test_y0 (sample1.sampletests.test_one) test_y0 (sample2.sample21.sampletests.TestA) test_y1 (sample2.sample21.sampletests.TestB) test_y0 (sample2.sample21.sampletests) test_y0 (sample2.sampletests.test_1.TestA) test_y1 (sample2.sampletests.test_1.TestB) test_y0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.testone.TestA) test_y1 (sample2.sampletests.testone.TestB) test_y0 (sample2.sampletests.testone) test_y0 (sample3.sampletests.TestA) test_y1 (sample3.sampletests.TestB) test_y0 (sample3.sampletests) test_y0 (sampletests.test1.TestA) test_y1 (sampletests.test1.TestB) test_y0 (sampletests.test1) test_y0 (sampletests.test_one.TestA) test_y1 (sampletests.test_one.TestB) test_y0 (sampletests.test_one) Ran 38 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False We can use the --all option to run tests at all levels: >>> sys.argv = 'test -u -vv --all -t test_y1 -t test_y0'.split() >>> testrunner.run_internal(defaults) Running tests at all levels Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_y0 (sampletestsf.TestA) test_y0 (sampletestsf.TestA2) test_y1 (sampletestsf.TestB) test_y0 (sampletestsf) test_y0 (sample1.sampletestsf.TestA) test_y1 (sample1.sampletestsf.TestB) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sample11.sampletests.TestA3) test_y1 (sample1.sample11.sampletests.TestB) test_y1 (sample1.sample11.sampletests.TestB2) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y1 (sample1.sample13.sampletests.TestB) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y1 (sample1.sampletests.test1.TestB) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y1 (sample1.sampletests.test_one.TestB) test_y0 (sample1.sampletests.test_one) test_y0 (sample2.sample21.sampletests.TestA) test_y1 (sample2.sample21.sampletests.TestB) test_y0 (sample2.sample21.sampletests) test_y0 (sample2.sampletests.test_1.TestA) test_y1 (sample2.sampletests.test_1.TestB) test_y0 (sample2.sampletests.test_1) test_y0 (sample2.sampletests.testone.TestA) test_y1 (sample2.sampletests.testone.TestB) test_y0 (sample2.sampletests.testone) test_y0 (sample3.sampletests.TestA) test_y1 (sample3.sampletests.TestB) test_y0 (sample3.sampletests) test_y0 (sampletests.test1.TestA) test_y1 (sampletests.test1.TestB) test_y0 (sampletests.test1) test_y0 (sampletests.test_one.TestA) test_y1 (sampletests.test_one.TestB) test_y0 (sampletests.test_one) Ran 39 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Listing Selected Tests ---------------------- When you're trying to figure out why the test you want is not matched by the pattern you specified, it is convenient to see which tests match your specifications. >>> sys.argv = 'test --all -m sample1 -t test_y0 --list-tests'.split() >>> testrunner.run_internal(defaults) Listing zope.testrunner.layer.UnitTests tests: test_y0 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sample11.sampletests.TestA3) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one) Listing samplelayers.Layer11 tests: test_y0 (sample1.sampletests.test11.TestA) test_y0 (sample1.sampletests.test11) Listing samplelayers.Layer111 tests: test_y0 (sample1.sampletests.test111.TestA) test_y0 (sample1.sampletests.test111) Listing samplelayers.Layer112 tests: test_y0 (sample1.sampletests.test112.TestA) test_y0 (sample1.sampletests.test112) Listing samplelayers.Layer12 tests: test_y0 (sample1.sampletests.test12.TestA) test_y0 (sample1.sampletests.test12) Listing samplelayers.Layer121 tests: test_y0 (sample1.sampletests.test121.TestA) test_y0 (sample1.sampletests.test121) Listing samplelayers.Layer122 tests: test_y0 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122) False zope.testrunner-4.4.9/src/zope/testrunner/tests/test_doctest.py0000664000000000000000000003525212527371226023630 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test harness for the test runner itself. """ from __future__ import print_function import re import gc import os import sys import unittest import doctest from zope.testing import renormalizing #separated checkers for the different platform, #because it s...s to maintain just one if sys.platform == 'win32': checker = renormalizing.RENormalizing([ # 2.5 changed the way pdb reports exceptions (re.compile(r":"), r'exceptions.\1Error:'), #rewrite pdb prompt to ... the current location #windows, py2.4 pdb seems not to put the '>' on doctest locations #therefore we cut it here (re.compile('^> doctest[^\n]+->None$', re.M), '...->None'), #rewrite pdb prompt to ... the current location (re.compile('^> [^\n]+->None$', re.M), '> ...->None'), (re.compile(r""),(r'?')), (re.compile(r":"), r'exceptions.\1Error:'), # testtools content formatter is used to mime-encode # tracebacks when the SubunitOutputFormatter is used, and the # resulting text includes a size which can vary depending on # the path included in the traceback. (re.compile(r'traceback\n[A-F\d]+', re.MULTILINE), r'traceback\nNNN'), (re.compile("'[A-Za-z]:\\\\"), "'"), # hopefully, we'll make Windows happy # replaces drives with nothing (re.compile(r'\\\\'), '/'), # more Windows happiness # double backslashes in coverage??? (re.compile(r'\\'), '/'), # even more Windows happiness # replaces backslashes in paths (re.compile(r'/r$', re.MULTILINE), '\\r'), # undo some of that #this is a magic to put linefeeds into the doctest (re.compile('##r##\n'), '\r'), (re.compile(r'\d+[.]\d\d\d seconds'), 'N.NNN seconds'), (re.compile(r'\d+[.]\d\d\d s'), 'N.NNN s'), (re.compile(r'\d+[.]\d\d\d{'), 'N.NNN{'), (re.compile(r'\d{4}-\d\d-\d\d \d\d:\d\d:\d\d\.\d+'), 'YYYY-MM-DD HH:MM:SS.mmmmmm'), (re.compile('( |")[^\n]+testrunner-ex'), r'\1testrunner-ex'), (re.compile('( |")[^\n]+testrunner.py'), r'\1testrunner.py'), (re.compile(r'> [^\n]*(doc|unit)test[.]py\(\d+\)'), r'\1test.py(NNN)'), (re.compile(r'[.]py\(\d+\)'), r'.py(NNN)'), (re.compile(r'[.]py:\d+'), r'.py:NNN'), (re.compile(r' line \d+,', re.IGNORECASE), r' Line NNN,'), (re.compile(r' line {([a-z]+)}\d+{', re.IGNORECASE), r' Line {\1}NNN{'), # omit traceback entries for unittest.py or doctest.py (and # their package variants) from output: (re.compile(r'^ +File "[^\n]*(doctest|unittest|case)(/__init__)?.py", [^\n]+\n[^\n]+\n', re.MULTILINE), r''), (re.compile(r'^{\w+} +File "{\w+}[^\n]*(doctest|unittest|case)(/__init__)?.py{\w+}", [^\n]+\n[^\n]+\n', re.MULTILINE), r''), #(re.compile('^> [^\n]+->None$', re.M), '> ...->None'), (re.compile('import pdb; pdb'), 'Pdb()'), # Py 2.3 # Python 3 exceptions are from the builtins module (re.compile(r'builtins\.(SyntaxError|TypeError)'), r'exceptions.\1'), # Python 3.3 has better exception messages (re.compile("ImportError: No module named '(?:[^']*[.])?([^'.]*)'"), r'ImportError: No module named \1'), # PyPy has different exception messages too (re.compile("ImportError: No module named (?:[a-zA-Z_0-9.]*[.])?([a-zA-Z_0-9]*)"), r'ImportError: No module named \1'), (re.compile("NameError: global name '([^']*)' is not defined"), r"NameError: name '\1' is not defined"), ]) else: #*nix checker = renormalizing.RENormalizing([ # 2.5 changed the way pdb reports exceptions (re.compile(r":"), r'exceptions.\1Error:'), #rewrite pdb prompt to ... the current location (re.compile('^> [^\n]+->None$', re.M), '> ...->None'), (re.compile(r""),(r'?')), (re.compile(r":"), r'exceptions.\1Error:'), #this is a magic to put linefeeds into the doctest #on win it takes one step, linux is crazy about the same... (re.compile('##r##'), r'\r'), (re.compile(r'\r'), '\\\\r\n'), (re.compile(r'\d+[.]\d\d\d seconds'), 'N.NNN seconds'), (re.compile(r'\d+[.]\d\d\d s'), 'N.NNN s'), (re.compile(r'\d+[.]\d\d\d{'), 'N.NNN{'), (re.compile(r'\d{4}-\d\d-\d\d \d\d:\d\d:\d\d\.\d+'), 'YYYY-MM-DD HH:MM:SS.mmmmmm'), (re.compile('( |"|\')[^\'\n]+testrunner-ex'), r'\1testrunner-ex'), (re.compile('( |"|\')[^\'\n]+testrunner.py'), r'\1testrunner.py'), (re.compile(r'> [^\n]*(doc|unit)test[.]py\(\d+\)'), r'\1test.py(NNN)'), (re.compile(r'[.]py\(\d+\)'), r'.py(NNN)'), (re.compile(r'[.]py:\d+'), r'.py:NNN'), (re.compile(r' line \d+,', re.IGNORECASE), r' Line NNN,'), (re.compile(r' line {([a-z]+)}\d+{', re.IGNORECASE), r' Line {\1}NNN{'), # testtools content formatter is used to mime-encode # tracebacks when the SubunitOutputFormatter is used, and the # resulting text includes a size which can vary depending on # the path included in the traceback. (re.compile(r'traceback\n[A-F\d]+', re.MULTILINE), r'traceback\nNNN'), # omit traceback entries for unittest.py or doctest.py (and # their package variants) from output: (re.compile(r'^ +File "[^\n]*(doctest|unittest|case)(/__init__)?.py", [^\n]+\n[^\n]+\n', re.MULTILINE), r''), (re.compile(r'^{\w+} +File "{\w+}[^\n]*(doctest|unittest|case)(/__init__)?.py{\w+}", [^\n]+\n[^\n]+\n', re.MULTILINE), r''), (re.compile('import pdb; pdb'), 'Pdb()'), # Py 2.3 # Python 3 exceptions are from the builtins module (re.compile(r'builtins\.(SyntaxError|TypeError)'), r'exceptions.\1'), # Python 3.3 has better exception messages (re.compile("ImportError: No module named '(?:[^']*[.])?([^'.]*)'"), r'ImportError: No module named \1'), # PyPy has different exception messages too (re.compile("ImportError: No module named (?:[a-zA-Z_0-9.]*[.])?([a-zA-Z_0-9]*)"), r'ImportError: No module named \1'), (re.compile("NameError: global name '([^']*)' is not defined"), r"NameError: name '\1' is not defined"), ]) def setUp(test): test.globs['print_function'] = print_function test.globs['saved-sys-info'] = ( sys.path[:], sys.argv[:], sys.modules.copy(), ) if hasattr(gc, 'get_threshold'): test.globs['saved-gc-threshold'] = gc.get_threshold() test.globs['this_directory'] = os.path.split(__file__)[0] test.globs['testrunner_script'] = sys.argv[0] def tearDown(test): sys.path[:], sys.argv[:] = test.globs['saved-sys-info'][:2] if hasattr(gc, 'get_threshold'): gc.set_threshold(*test.globs['saved-gc-threshold']) sys.modules.clear() sys.modules.update(test.globs['saved-sys-info'][2]) def test_suite(): suites = [ doctest.DocFileSuite( 'testrunner-arguments.txt', 'testrunner-coverage.txt', 'testrunner-debugging-layer-setup.test', 'testrunner-debugging-import-failure.test', 'testrunner-debugging-nonprintable-exc.test', 'testrunner-debugging.txt', 'testrunner-edge-cases.txt', 'testrunner-errors.txt', 'testrunner-layers-api.txt', 'testrunner-layers-instances.txt', 'testrunner-layers-buff.txt', 'testrunner-subprocess-errors.txt', 'testrunner-layers-cantfind.txt', 'testrunner-layers-cwd.txt', 'testrunner-layers-ntd.txt', 'testrunner-layers-topological-sort.txt', 'testrunner-layers.txt', 'testrunner-progress.txt', 'testrunner-colors.txt', 'testrunner-simple.txt', 'testrunner-nestedcode.txt', 'testrunner-test-selection.txt', 'testrunner-verbose.txt', 'testrunner-repeat.txt', 'testrunner-knit.txt', 'testrunner-shuffle.txt', 'testrunner-eggsupport.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS+doctest.NORMALIZE_WHITESPACE, checker=checker), doctest.DocTestSuite('zope.testrunner'), doctest.DocTestSuite('zope.testrunner.coverage', optionflags=doctest.ELLIPSIS+doctest.NORMALIZE_WHITESPACE), doctest.DocTestSuite('zope.testrunner.options'), doctest.DocTestSuite('zope.testrunner.find'), ] # PyPy uses a different garbage collector if hasattr(gc, 'get_threshold'): suites.append( doctest.DocFileSuite( 'testrunner-gc.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS+doctest.NORMALIZE_WHITESPACE, checker=checker)) # PyPy does not support sourceless imports, apparently (tried version 1.9) if 'PyPy' not in sys.version: suites.append( doctest.DocFileSuite( 'testrunner-wo-source.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS+doctest.NORMALIZE_WHITESPACE, checker=checker)) if sys.platform == 'win32': suites.append( doctest.DocFileSuite( 'testrunner-coverage-win32.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS+doctest.NORMALIZE_WHITESPACE, checker=checker)) # Python <= 2.4.1 had a bug that prevented hotshot from running in # non-optimize mode if sys.version_info[:3] > (2,4,1) or not __debug__: # some Linux distributions don't include the profiling module (which # hotshot.stats depends on) try: import hotshot.stats except ImportError: pass else: suites.append( doctest.DocFileSuite( 'testrunner-profiling.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS+doctest.NORMALIZE_WHITESPACE, checker = renormalizing.RENormalizing([ (re.compile(r'tests_profile[.]\S*[.]prof'), 'tests_profile.*.prof'), ]), ) ) try: import cProfile import pstats except ImportError: pass else: suites.append( doctest.DocFileSuite( 'testrunner-profiling-cprofiler.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS+doctest.NORMALIZE_WHITESPACE, checker = renormalizing.RENormalizing([ (re.compile(r'tests_profile[.]\S*[.]prof'), 'tests_profile.*.prof'), ]), ) ) skip_feature = True if sys.version_info < (2, 7, 0): try: import unittest2 except ImportError: skip_feature = False if skip_feature: suites.append( doctest.DocFileSuite( 'testrunner-report-skipped.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS+doctest.NORMALIZE_WHITESPACE, checker=checker) ) if hasattr(sys, 'gettotalrefcount'): suites.append( doctest.DocFileSuite( 'testrunner-leaks.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS+doctest.NORMALIZE_WHITESPACE, checker = renormalizing.RENormalizing([ (re.compile(r'\d+[.]\d\d\d seconds'), 'N.NNN seconds'), (re.compile(r'sys refcount=\d+ +change=\d+'), 'sys refcount=NNNNNN change=NN'), (re.compile(r'sum detail refcount=\d+ +'), 'sum detail refcount=NNNNNN '), (re.compile(r'total +\d+ +\d+'), 'total NNNN NNNN'), (re.compile(r"^ +(int|type) +-?\d+ +-?\d+ *\n", re.M), ''), ]), ) ) else: suites.append( doctest.DocFileSuite( 'testrunner-leaks-err.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS+doctest.NORMALIZE_WHITESPACE, checker=checker, ) ) try: import subunit except ImportError: suites.append( doctest.DocFileSuite( 'testrunner-subunit-err.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS + doctest.NORMALIZE_WHITESPACE, checker=checker)) else: suites.append( doctest.DocFileSuite( 'testrunner-subunit.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS + doctest.NORMALIZE_WHITESPACE + doctest.REPORT_NDIFF, checker=checker)) if hasattr(sys, 'gettotalrefcount'): suites.append( doctest.DocFileSuite( 'testrunner-subunit-leaks.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS + doctest.NORMALIZE_WHITESPACE, checker=checker)) if sys.version_info[:3] >= (2,7,0): # Python 2.7 adds support for unittest.expectedFailure suites.append(doctest.DocFileSuite( 'testrunner-unexpected-success.txt', setUp=setUp, tearDown=tearDown, optionflags=doctest.ELLIPSIS+doctest.NORMALIZE_WHITESPACE, checker=checker)) return unittest.TestSuite(suites) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/0000775000000000000000000000000012552256734023372 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests_buffering.py0000664000000000000000000000306412527371226030336 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample tests with sleep and layers that can't be torn down """ import unittest, time class Layer1: def setUp(self): pass setUp = classmethod(setUp) def tearDown(self): raise NotImplementedError tearDown = classmethod(tearDown) class Layer2: def setUp(self): pass setUp = classmethod(setUp) def tearDown(self): raise NotImplementedError tearDown = classmethod(tearDown) class TestSomething1(unittest.TestCase): layer = Layer1 def test_something(self): pass class TestSomething2(unittest.TestCase): layer = Layer2 def test_something(self): time.sleep(0.5) def test_something2(self): time.sleep(0.5) def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestSomething1)) suite.addTest(unittest.makeSuite(TestSomething2)) return suite if __name__ == '__main__': unittest.main() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/unicode.py0000664000000000000000000000131612527371226025367 0ustar rootroot############################################################################## # # Copyright (c) 2007 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import doctest def test_suite(): return doctest.DocFileSuite('unicode.txt') zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/0000775000000000000000000000000012552256734024735 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/do-not-enter/0000775000000000000000000000000012552256734027250 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/do-not-enter/sampletests.py0000664000000000000000000000227112527371226032164 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest def f(): g() def g(): x = 1 x = x + 1 x = y + 1 x = x + 1 def eek(self): """ >>> f() 1 """ class Test(unittest.TestCase): def test1(self): pass def test2(self): pass def test3(self): f() def test4(self): pass def test5(self): pass def test_suite(): suite = unittest.TestSuite() suite.addTest(doctest.DocTestSuite()) suite.addTest(unittest.makeSuite(Test)) suite.addTest(doctest.DocFileSuite('e.txt')) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sample23/0000775000000000000000000000000012552256734026363 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sample23/__init__.py0000664000000000000000000000000212527371226030460 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sample23/sampletests_i.py0000664000000000000000000000146312527371226031611 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest class Test(unittest.TestCase): def test(self): self.assertEqual(1,0) raise TypeError('eek') def test_suite(): return unittest.makeSuite(Test) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests_f.py0000664000000000000000000000142712527371226030160 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest class Test(unittest.TestCase): def test(self): self.assertEqual(1,0) def test_suite(): return unittest.makeSuite(Test) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/stderrtest.py0000664000000000000000000000247612527371226027517 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample tests with a layer that produces output on stderr """ import unittest import doctest import sys class Layer: def setUp(self): pass setUp = classmethod(setUp) def tearDown(self): pass tearDown = classmethod(tearDown) def test_something(): """ >>> 1 + 1 2 """ def test_suite(): # Generate some text on stderr to be sure the test runner can handle it. sys.stderr.write('A message on stderr.' ' Please ignore (expected in test output).\n') suite = unittest.TestSuite() d = doctest.DocTestSuite() d.layer = Layer suite.addTest(d) return suite if __name__ == '__main__': unittest.main() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/donotenter/0000775000000000000000000000000012552256734027116 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/donotenter/sampletests.py0000664000000000000000000000227112527371226032032 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest def f(): g() def g(): x = 1 x = x + 1 x = y + 1 x = x + 1 def eek(self): """ >>> f() 1 """ class Test(unittest.TestCase): def test1(self): pass def test2(self): pass def test3(self): f() def test4(self): pass def test5(self): pass def test_suite(): suite = unittest.TestSuite() suite.addTest(doctest.DocTestSuite()) suite.addTest(unittest.makeSuite(Test)) suite.addTest(doctest.DocFileSuite('e.txt')) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests_1.py0000664000000000000000000000142712527371226030073 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import doctest def eek(self): """ >>> x = y >>> x >>> z = x + 1 """ def test_suite(): return doctest.DocTestSuite() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/0000775000000000000000000000000012552256734027301 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/test_1.py0000664000000000000000000000376412527371226031060 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('../../sampletests.txt', setUp=setUp)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/testone.py0000664000000000000000000000376412527371226031342 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('../../sampletests.txt', setUp=setUp)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests/__init__.py0000664000000000000000000000000212527371226031376 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/__init__.py0000664000000000000000000000000212527371226027032 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests_ntd.py0000664000000000000000000000221212527371226030511 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample tests with a layer that can't be torn down """ import unittest class Layer: def setUp(self): pass setUp = classmethod(setUp) def tearDown(self): raise NotImplementedError tearDown = classmethod(tearDown) class TestSomething(unittest.TestCase): layer = Layer def test_something(self): pass def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestSomething)) return suite if __name__ == '__main__': unittest.main() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sample22/0000775000000000000000000000000012552256734026362 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sample22/__init__.py0000664000000000000000000000000212527371226030457 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sample22/sampletests_i.py0000664000000000000000000000144012527371226031603 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest class Test(unittest.TestCase): def test(self): self.assertEqual(1,0) def test_suitex(): return unittest.makeSuite(Test) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sample21/0000775000000000000000000000000012552256734026361 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sample21/__init__.py0000664000000000000000000000000212527371226030456 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sample21/sampletests.py0000664000000000000000000000376412527371226031305 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('../../sampletests.txt', setUp=setUp)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sample21/sampletests_i.py0000664000000000000000000000147212527371226031607 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import zope.testrunner.huh class Test(unittest.TestCase): def test(self): self.assertEqual(1,0) def test_suite(): return unittest.makeSuite(Test) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests_ntds.py0000664000000000000000000000330112527371226030674 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample tests with a layer that can't be torn down """ import unittest import doctest class Layer: def setUp(self): pass setUp = classmethod(setUp) def tearDown(self): raise NotImplementedError tearDown = classmethod(tearDown) class TestSomething(unittest.TestCase): layer = Layer def test_something(self): import pdb; pdb.set_trace() def test_something2(self): import pdb; pdb.set_trace() def test_something3(self): import pdb; pdb.set_trace() def test_something4(self): import pdb; pdb.set_trace() def test_something5(self): f() def f(): import pdb; pdb.set_trace() def test_set_trace(): """ >>> if 1: ... x = 1 ... import pdb; pdb.set_trace() """ def test_set_trace2(): """ >>> f() """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestSomething)) d = doctest.DocTestSuite() d.layer = Layer suite.addTest(d) return suite if __name__ == '__main__': unittest.main() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/e.txt0000664000000000000000000000006612527371226025720 0ustar rootroot >>> def f(): ... return x >>> f() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample2/sampletests_e.py0000664000000000000000000000236312527371226030157 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest def f(): g() def g(): x = 1 x = x + 1 __traceback_info__ = "I don't know what Y should be." x = y + 1 x = x + 1 def eek(self): """ >>> f() 1 """ class Test(unittest.TestCase): def test1(self): pass def test2(self): pass def test3(self): f() def test4(self): pass def test5(self): pass def test_suite(): suite = unittest.TestSuite() suite.addTest(doctest.DocTestSuite()) suite.addTest(unittest.makeSuite(Test)) suite.addTest(doctest.DocFileSuite('e.txt')) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/samplelayers.py0000664000000000000000000001033512527371226026443 0ustar rootroot############################################################################## # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample test layers """ layer = '0' # Internal to samples. Not part of layer API layerx = '0' class Layer1: # Internal to samples. Not part of layer API: layer = '1' base = '0' layerx = '0' def setUp(self): global layer if layer != self.base: raise ValueError("Bad layer, %s, for %s." % (layer, self)) layer = self.layer setUp = classmethod(setUp) def tearDown(self): global layer if layer != self.layer: raise ValueError("Bad layer, %s, for %s." % (layer, self)) layer = self.base tearDown = classmethod(tearDown) class Layerx: layerx = '1' # Internal to samples. Not part of layer API basex = '0' def setUp(self): global layerx if layerx != self.basex: raise ValueError("Bad layerx, %s, for %s." % (layerx, self)) layerx = self.layerx setUp = classmethod(setUp) def tearDown(self): global layerx if layerx != self.layerx: raise ValueError("Bad layerx, %s, for %s." % (layerx, self)) layerx = self.basex tearDown = classmethod(tearDown) class Layer11(Layer1): layer = '11' # Internal to samples. Not part of layer API base = '1' # Internal to samples. Not part of layer API class Layer12(Layer1): layer = '12' # Internal to samples. Not part of layer API base = '1' # Internal to samples. Not part of layer API class Layer111(Layerx, Layer11): layer = '111' # Internal to samples. Not part of layer API base = '11' # Internal to samples. Not part of layer API layerx = '2' # Internal to samples. Not part of layer API basex = '1' def setUp(self): global layer if layer != self.base: raise ValueError("Bad layer, %s, for %s." % (layer, self)) layer = self.layer global layerx if layerx != self.basex: raise ValueError("Bad layerx, %s, for %s." % (layerx, self)) layerx = self.layerx setUp = classmethod(setUp) def tearDown(self): global layer if layer != self.layer: raise ValueError("Bad layer, %s, for %s." % (layer, self)) layer = self.base global layerx if layerx != self.layerx: raise ValueError("Bad layerx, %s, for %s." % (layerx, self)) layerx = self.basex tearDown = classmethod(tearDown) class Layer121(Layer12): layer = '121' # Internal to samples. Not part of layer API base = '12' # Internal to samples. Not part of layer API class Layer112(Layerx, Layer11): layer = '112' # Internal to samples. Not part of layer API base = '11' # Internal to samples. Not part of layer API layerx = '2' # Internal to samples. Not part of layer API basex = '1' def setUp(self): global layer if layer != self.base: raise ValueError("Bad layer, %s, for %s." % (layer, self)) layer = self.layer global layerx if layerx != self.basex: raise ValueError("Bad layerx, %s, for %s." % (layerx, self)) layerx = self.layerx setUp = classmethod(setUp) def tearDown(self): global layer if layer != self.layer: raise ValueError("Bad layer, %s, for %s." % (layer, self)) layer = self.base global layerx if layerx != self.layerx: raise ValueError("Bad layerx, %s, for %s." % (layerx, self)) layerx = self.basex tearDown = classmethod(tearDown) class Layer122(Layer12): layer = '122' # Internal to samples. Not part of layer API base = '12' # Internal to samples. Not part of layer API zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests.txt0000664000000000000000000000023112527371226026467 0ustar rootrootThis is a sample doctest >>> x=1 >>> x 1 Blah blah blah >>> x 1 Blah blah blah >>> x 1 Blah blah blah >>> x 1 zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/leak.py0000664000000000000000000000213412527371226024654 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest, time class ClassicLeakable: def __init__(self): self.x = 'x' class Leakable(object): def __init__(self): self.x = 'x' leaked = [] class TestSomething(unittest.TestCase): def testleak(self): leaked.append((ClassicLeakable(), Leakable(), time.time())) def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestSomething)) return suite if __name__ == '__main__': unittest.main() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/0000775000000000000000000000000012552256734024734 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sample12/0000775000000000000000000000000012552256734026360 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sample12/__init__.py0000664000000000000000000000000212527371226030455 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_none_test.py0000664000000000000000000000144312527371226031726 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample tests with a layer that can't be torn down """ import unittest def test_suite(): suite = unittest.TestSuite() suite.addTest(None) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/0000775000000000000000000000000012552256734027300 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test121.py0000664000000000000000000000705412527371226031057 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer121' layer = samplelayers.Layer121 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test122.py0000664000000000000000000000705412527371226031060 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer122' layer = samplelayers.Layer122 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/__init__.py0000664000000000000000000000000212527371226031375 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test112.py0000664000000000000000000000705512527371226031060 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer112' layer = samplelayers.Layer112 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test111.py0000664000000000000000000000705412527371226031056 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer111' layer = samplelayers.Layer111 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test12.py0000664000000000000000000000705212527371226030774 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer12' layer = samplelayers.Layer12 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test11.py0000664000000000000000000000705212527371226030773 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer11' layer = samplelayers.Layer11 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test_one.py0000664000000000000000000000376412527371226031500 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('../../sampletests.txt', setUp=setUp)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests/test1.py0000664000000000000000000000376412527371226030720 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('../../sampletests.txt', setUp=setUp)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/__init__.py0000664000000000000000000000000212527371226027031 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_ntd.py0000664000000000000000000000221212527371226030510 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample tests with a layer that can't be torn down """ import unittest class Layer: def setUp(self): pass setUp = classmethod(setUp) def tearDown(self): raise NotImplementedError tearDown = classmethod(tearDown) class TestSomething(unittest.TestCase): layer = Layer def test_something(self): pass def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestSomething)) return suite if __name__ == '__main__': unittest.main() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_none_suite.py0000664000000000000000000000132112527371226032073 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample tests with a layer that can't be torn down """ def test_suite(): pass zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sample11/0000775000000000000000000000000012552256734026357 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sample11/__init__.py0000664000000000000000000000000212527371226030454 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sample11/sampletests.py0000664000000000000000000000531512527371226031275 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestA3(unittest.TestCase): level = 3 def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestB2(unittest.TestCase): level = 2 def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestA3)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestB2)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('../../sampletests.txt', setUp=setUp)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_ntds.py0000664000000000000000000000221212527371226030673 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample tests with a layer that can't be torn down """ import unittest class Layer: def setUp(self): pass setUp = classmethod(setUp) def tearDown(self): raise NotImplementedError tearDown = classmethod(tearDown) class TestSomething(unittest.TestCase): layer = Layer def test_something(self): pass def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestSomething)) return suite if __name__ == '__main__': unittest.main() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_discover.py0000664000000000000000000000014612527371226031545 0ustar rootrootimport unittest class TestA(unittest.TestCase): def test_truth(self): self.assert_(True) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sample13/0000775000000000000000000000000012552256734026361 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sample13/__init__.py0000664000000000000000000000000212527371226030456 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sample13/sampletests.py0000664000000000000000000000376412527371226031305 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('../../sampletests.txt', setUp=setUp)) return suite ././@LongLink0000644000000000000000000000014600000000000011604 Lustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_discover_notests.pyzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletests_discover_notests.p0000664000000000000000000000010412527371226033125 0ustar rootrootdef test_function_that_would_never_be_run(): self.assert_(True) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample1/sampletestsf.py0000664000000000000000000000376112527371226030023 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('../sampletests.txt', setUp=setUp)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/gcstats.py0000664000000000000000000000150312527371226025407 0ustar rootroot############################################################################## # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import doctest def generate_some_gc_statistics(): """ >>> import gc >>> l = []; l.append(l); del l >>> _ = gc.collect() """ def test_suite(): return doctest.DocTestSuite() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletestsl.txt0000664000000000000000000000045712527371226026655 0ustar rootrootThis is a sample doctest >>> x=1 >>> x 1 Blah blah blah >>> x 1 Blah blah blah >>> x 1 Blah blah blah >>> x 1 are we in the right laters? >>> import samplelayers >>> layer == samplelayers.layer True >>> layerx == samplelayers.layerx True zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/pledge.py0000664000000000000000000000233112527371226025177 0ustar rootroot############################################################################## # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import doctest pledge_template = """\ I give my pledge, as %s, to save, and faithfully, to defend from waste, the natural resources of my %s. It's soils, minerals, forests, waters, and wildlife. """ def pledge(): """ >>> def print_pledge(): ... print(pledge_template % ('and earthling', 'planet')) >>> print_pledge() I give my pledge, as an earthling, to save, and faithfully, to defend from waste, the natural resources of my planet. It's soils, minerals, forests, waters, and wildlife. """ def test_suite(): return doctest.DocTestSuite() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests/0000775000000000000000000000000012552256734025736 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests/test121.py0000664000000000000000000000705112527371226027512 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer121' layer = samplelayers.Layer121 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests/test122.py0000664000000000000000000000705112527371226027513 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer122' layer = samplelayers.Layer122 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests/__init__.py0000664000000000000000000000000212527371226030033 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests/test112.py0000664000000000000000000000741412527371226027515 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer112' layer = samplelayers.Layer112 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 self.clean = getattr(self, 'clean', 0) + 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) # This is a test that the test runner clears attributes # that are set in setUp but not cleared in tearDown. self.assertEqual(self.clean, 1) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests/test111.py0000664000000000000000000000705112527371226027511 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer111' layer = samplelayers.Layer111 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests/test12.py0000664000000000000000000000704712527371226027436 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer12' layer = samplelayers.Layer12 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests/test11.py0000664000000000000000000000704712527371226027435 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest import samplelayers layername = 'samplelayers.Layer11' layer = samplelayers.Layer11 x=0 y=0 z=0 class TestA(unittest.TestCase): layer = layername def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestB(unittest.TestCase): layer = layername def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) class TestNotMuch(unittest.TestCase): layer = layername def test_1(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_2(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def test_3(self): self.assertEqual(samplelayers.layer, layer.layer) self.assertEqual(samplelayers.layerx, layer.layerx) def setUp(test): test.globs['z'] = 1 test.globs['layer'] = layer.layer test.globs['layerx'] = layer.layerx test.globs['samplelayers'] = samplelayers def test_y0(self): """ >>> y = 0 >>> y 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_x0(self): """ >>> x = 0 >>> x 0 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_z1(self): """ >>> z 1 >>> (layer == samplelayers.layer), (layerx == samplelayers.layerx) (True, True) """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) s = doctest.DocTestSuite(setUp=setUp) s.layer = layer suite.addTest(s) s = doctest.DocFileSuite('../sampletestsl.txt', setUp=setUp) s.layer = layer suite.addTest(s) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests/test_one.py0000664000000000000000000000376112527371226030133 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('../sampletests.txt', setUp=setUp)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletests/test1.py0000664000000000000000000000376112527371226027353 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('../sampletests.txt', setUp=setUp)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/README.txt0000664000000000000000000000015012527371226025060 0ustar rootrootThis directory and its subdirectories contain example tests for testing the test runner, testrunner.py. zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/gcset.py0000664000000000000000000000147212527371226025051 0ustar rootroot############################################################################## # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import doctest def make_sure_gc_threshold_is_701_11_9(): """ >>> import gc >>> gc.get_threshold() (701, 11, 9) """ def test_suite(): return doctest.DocTestSuite() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/gc0.py0000664000000000000000000000145012527371226024411 0ustar rootroot############################################################################## # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import doctest def make_sure_gc_is_disabled(): """ >>> import gc >>> gc.get_threshold()[0] 0 """ def test_suite(): return doctest.DocTestSuite() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/usecompiled/0000775000000000000000000000000012552256734025703 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/usecompiled/package/0000775000000000000000000000000012552256734027276 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/usecompiled/package/__init__.py0000664000000000000000000000003012527371226031374 0ustar rootroot# Makes this a package. zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/usecompiled/package/compiletest.py0000664000000000000000000000160712527371226032200 0ustar rootroot############################################################################## # # Copyright (c) 2005 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest class Test(unittest.TestCase): def test1(self): self.assertEqual(1, 1) def test2(self): self.assertEqual(1, 1) def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(Test)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/usecompiled/__init__.py0000664000000000000000000000003012527371226030001 0ustar rootroot# Makes this a package. zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/usecompiled/README.txt0000664000000000000000000000014012527371226027370 0ustar rootrootThe tests in this subtree are trivial, and used only to test testrunner's --usecompiled option. zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/usecompiled/compiletest.py0000664000000000000000000000160712527371226030605 0ustar rootroot############################################################################## # # Copyright (c) 2005 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest class Test(unittest.TestCase): def test1(self): self.assertEqual(1, 1) def test2(self): self.assertEqual(1, 1) def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(Test)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/gc1.py0000664000000000000000000000145512527371226024417 0ustar rootroot############################################################################## # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import doctest def make_sure_gc_threshold_is_one(): """ >>> import gc >>> gc.get_threshold()[0] 1 """ def test_suite(): return doctest.DocTestSuite() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/0000775000000000000000000000000012552256734024736 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/set_trace5.txt0000664000000000000000000000012612527371226027530 0ustar rootroot >>> x = 1 >>> if 1: ... import pdb; pdb.set_trace() ... y = x zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/sample32/0000775000000000000000000000000012552256734026364 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/sample32/__init__.py0000664000000000000000000000000212527371226030461 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/post_mortem6.txt0000664000000000000000000000007012527371226030126 0ustar rootroot >>> from sample3.sampletests_d import g >>> g() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/sample31/0000775000000000000000000000000012552256734026363 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/sample31/__init__.py0000664000000000000000000000000212527371226030460 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/post_mortem_failure.txt0000664000000000000000000000003612527371226031551 0ustar rootroot >>> x = 1 >>> x 2 zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/__init__.py0000664000000000000000000000000212527371226027033 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/post_mortem5.txt0000664000000000000000000000004712527371226030131 0ustar rootroot >>> x = 1 >>> raise ValueError zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/sampletests_ntd.py0000664000000000000000000000267112527371226030523 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample tests with a layer that can't be torn down """ import unittest class Layer: def setUp(self): pass setUp = classmethod(setUp) def tearDown(self): raise NotImplementedError tearDown = classmethod(tearDown) class TestSomething(unittest.TestCase): layer = Layer def test_something(self): pass def test_something_else(self): pass def test_error1(self): raise TypeError("Can we see errors") def test_error2(self): raise TypeError("I hope so") def test_fail1(self): self.assertEqual(1, 2) def test_fail2(self): self.assertEqual(1, 3) def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestSomething)) return suite if __name__ == '__main__': unittest.main() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/sampletests.py0000664000000000000000000000376112527371226027657 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('../sampletests.txt', setUp=setUp)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/set_trace6.txt0000664000000000000000000000007012527371226027527 0ustar rootroot >>> from sample3.sampletests_d import f >>> f() zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/sample33/0000775000000000000000000000000012552256734026365 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/sample33/__init__.py0000664000000000000000000000000212527371226030462 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sample3/sampletests_d.py0000664000000000000000000000373312527371226030161 0ustar rootroot############################################################################## # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest class TestSomething(unittest.TestCase): def test_set_trace1(self): x = 1 import pdb; pdb.set_trace() y = x def test_set_trace2(self): f() def test_post_mortem1(self): x = 1 raise ValueError def test_post_mortem2(self): g() def test_post_mortem_failure1(self): x = 1 y = 2 assert x == y def f(): x = 1 import pdb; pdb.set_trace() y = x def g(): x = 1 raise ValueError def set_trace3(self): """ >>> x = 1 >>> if 1: ... import pdb; pdb.set_trace() ... y = x """ def set_trace4(self): """ >>> f() """ def post_mortem3(self): """ >>> x = 1 >>> raise ValueError """ def post_mortem4(self): """ >>> g() """ def post_mortem_failure2(): """ >>> x = 1 >>> x 2 """ def test_suite(): return unittest.TestSuite(( doctest.DocTestSuite(), unittest.makeSuite(TestSomething), doctest.DocFileSuite('set_trace5.txt'), doctest.DocFileSuite('set_trace6.txt'), doctest.DocFileSuite('post_mortem5.txt'), doctest.DocFileSuite('post_mortem6.txt'), doctest.DocFileSuite('post_mortem_failure.txt'), )) if __name__ == '__main__': unittest.main(defaultTest='test_suite') zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex/sampletestsf.py0000664000000000000000000001046312527371226026456 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import doctest x=0 y=0 z=0 class TestA(unittest.TestCase): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestA2(unittest.TestCase): level = 2 def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) def test_y0(self): self.assertEqual(y, 0) def test_z0(self): self.assertEqual(z, 0) class TestB(unittest.TestCase): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) def test_x0(self): self.assertEqual(x, 0) def test_z0(self): self.assertEqual(z, 0) class TestNotMuch(unittest.TestCase): def test_1(self): pass def test_2(self): pass def test_3(self): pass def setUp(test): test.globs['z'] = 1 def test_y0(self): """ >>> y = 0 >>> y 0 """ def test_x0(self): """ >>> x = 0 >>> x 0 """ def test_z1(self): """ >>> z 1 """ import samplelayers class Layered: layer = 'samplelayers.Layer1' layerv = '1' layerx = '0' class TestA1(unittest.TestCase, Layered): def setUp(self): global x x = 1 def tearDown(self): global x x = 0 def test_x1(self): self.assertEqual(x, 1) self.assertEqual(samplelayers.layer, self.layerv) self.assertEqual(samplelayers.layerx, self.layerx) def test_y0(self): self.assertEqual(y, 0) self.assertEqual(samplelayers.layer, self.layerv) self.assertEqual(samplelayers.layerx, self.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, self.layerv) self.assertEqual(samplelayers.layerx, self.layerx) class TestB1(unittest.TestCase, Layered): def setUp(self): global y y = 1 def tearDown(self): global y y = 0 def test_y1(self): self.assertEqual(y, 1) self.assertEqual(samplelayers.layer, self.layerv) self.assertEqual(samplelayers.layerx, self.layerx) def test_x0(self): self.assertEqual(x, 0) self.assertEqual(samplelayers.layer, self.layerv) self.assertEqual(samplelayers.layerx, self.layerx) def test_z0(self): self.assertEqual(z, 0) self.assertEqual(samplelayers.layer, self.layerv) self.assertEqual(samplelayers.layerx, self.layerx) class TestNotMuch1(unittest.TestCase, Layered): def test_1(self): self.assertEqual(samplelayers.layer, self.layerv) self.assertEqual(samplelayers.layerx, self.layerx) def test_2(self): self.assertEqual(samplelayers.layer, self.layerv) self.assertEqual(samplelayers.layerx, self.layerx) def test_3(self): self.assertEqual(samplelayers.layer, self.layerv) self.assertEqual(samplelayers.layerx, self.layerx) def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestA)) suite.addTest(unittest.makeSuite(TestA2)) suite.addTest(unittest.makeSuite(TestB)) suite.addTest(unittest.makeSuite(TestNotMuch)) suite.addTest(doctest.DocTestSuite(setUp=setUp)) suite.addTest(doctest.DocFileSuite('sampletests.txt', setUp=setUp)) suite.addTest(unittest.makeSuite(TestA1)) suite.addTest(unittest.makeSuite(TestB1)) suite.addTest(unittest.makeSuite(TestNotMuch1)) return suite zope.testrunner-4.4.9/src/zope/testrunner/tests/__init__.py0000664000000000000000000000002212527371226022646 0ustar rootroot# Make a package. zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-verbose.txt0000664000000000000000000001513312527371226025163 0ustar rootrootVerbose Output ============== Normally, we just get a summary. We can use the -v option to get increasingly more information. If we use a single --verbose (-v) option, we get a dot printed for each test: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 122 -v'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: .................................. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.007 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False If there are more than 50 tests, the dots are printed in groups of 50: >>> sys.argv = 'test -uv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: ................................................................................................................................................................................................ Ran 156 tests with 0 failures, 0 errors and 0 skipped in 0.035 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False If the --verbose (-v) option is used twice, then the name and location of each test is printed as it is run: >>> sys.argv = 'test --layer 122 -vv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) test_y0 (sample1.sampletests.test122.TestA) test_z0 (sample1.sampletests.test122.TestA) test_x0 (sample1.sampletests.test122.TestB) test_y1 (sample1.sampletests.test122.TestB) test_z0 (sample1.sampletests.test122.TestB) test_1 (sample1.sampletests.test122.TestNotMuch) test_2 (sample1.sampletests.test122.TestNotMuch) test_3 (sample1.sampletests.test122.TestNotMuch) test_x0 (sample1.sampletests.test122) test_y0 (sample1.sampletests.test122) test_z1 (sample1.sampletests.test122) testrunner-ex/sample1/sampletests/../../sampletestsl.txt test_x1 (sampletests.test122.TestA) test_y0 (sampletests.test122.TestA) test_z0 (sampletests.test122.TestA) test_x0 (sampletests.test122.TestB) test_y1 (sampletests.test122.TestB) test_z0 (sampletests.test122.TestB) test_1 (sampletests.test122.TestNotMuch) test_2 (sampletests.test122.TestNotMuch) test_3 (sampletests.test122.TestNotMuch) test_x0 (sampletests.test122) test_y0 (sampletests.test122) test_z1 (sampletests.test122) testrunner-ex/sampletests/../sampletestsl.txt Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.009 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False if the --verbose (-v) option is used three times, then individual test-execution times are printed: >>> sys.argv = 'test --layer 122 -vvv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test122.TestA) (0.000 s) test_y0 (sample1.sampletests.test122.TestA) (0.000 s) test_z0 (sample1.sampletests.test122.TestA) (0.000 s) test_x0 (sample1.sampletests.test122.TestB) (0.000 s) test_y1 (sample1.sampletests.test122.TestB) (0.000 s) test_z0 (sample1.sampletests.test122.TestB) (0.000 s) test_1 (sample1.sampletests.test122.TestNotMuch) (0.000 s) test_2 (sample1.sampletests.test122.TestNotMuch) (0.000 s) test_3 (sample1.sampletests.test122.TestNotMuch) (0.000 s) test_x0 (sample1.sampletests.test122) (0.001 s) test_y0 (sample1.sampletests.test122) (0.001 s) test_z1 (sample1.sampletests.test122) (0.001 s) testrunner-ex/sample1/sampletests/../../sampletestsl.txt (0.001 s) test_x1 (sampletests.test122.TestA) (0.000 s) test_y0 (sampletests.test122.TestA) (0.000 s) test_z0 (sampletests.test122.TestA) (0.000 s) test_x0 (sampletests.test122.TestB) (0.000 s) test_y1 (sampletests.test122.TestB) (0.000 s) test_z0 (sampletests.test122.TestB) (0.000 s) test_1 (sampletests.test122.TestNotMuch) (0.000 s) test_2 (sampletests.test122.TestNotMuch) (0.000 s) test_3 (sampletests.test122.TestNotMuch) (0.000 s) test_x0 (sampletests.test122) (0.001 s) test_y0 (sampletests.test122) (0.001 s) test_z1 (sampletests.test122) (0.001 s) testrunner-ex/sampletests/../sampletestsl.txt (0.001 s) Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.009 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False Quiet output ------------ The --quiet (-q) option cancels all verbose options. It's useful when the default verbosity is non-zero: >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... '-v' ... ] >>> sys.argv = 'test -q -u'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in 0.034 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-colors.txt0000664000000000000000000004504212527371226025021 0ustar rootrootColorful output =============== If you're on a Unix-like system, you can ask for colorized output. The test runner emits terminal control sequences to highlight important pieces of information (such as the names of failing tests) in different colors. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner Since it wouldn't be a good idea to have terminal control characters in a test file, let's wrap sys.stdout in a simple terminal interpreter >>> import re >>> class Terminal(object): ... _color_regexp = re.compile('\033[[]([0-9;]*)m') ... _colors = {'0': 'normal', '1': 'bold', '30': 'black', '31': 'red', ... '32': 'green', '33': 'yellow', '34': 'blue', ... '35': 'magenta', '36': 'cyan', '37': 'grey'} ... def __init__(self, stream): ... self._stream = stream ... def __getattr__(self, attr): ... return getattr(self._stream, attr) ... def isatty(self): ... return True ... def write(self, text): ... if '\033[' in text: ... text = self._color_regexp.sub(self._color, text) ... self._stream.write(text) ... def writelines(self, lines): ... for line in lines: ... self.write(line) ... def _color(self, match): ... colorstring = '{' ... for number in match.group(1).split(';'): ... colorstring += self._colors.get(number, '?') ... return colorstring + '}' >>> real_stdout = sys.stdout >>> sys.stdout = Terminal(sys.stdout) Successful test --------------- A successful test run soothes the developer with warm green colors: >>> sys.argv = 'test --layer 122 -c'.split() >>> testrunner.run_internal(defaults) {normal}Running samplelayers.Layer122 tests:{normal} Set up samplelayers.Layer1 in {green}0.000{normal} seconds. Set up samplelayers.Layer12 in {green}0.000{normal} seconds. Set up samplelayers.Layer122 in {green}0.000{normal} seconds. {normal} Ran {green}26{normal} tests with {green}0{normal} failures, {green}0{normal} errors, {green}0{normal} skipped in {green}0.007{normal} seconds.{normal} {normal}Tearing down left over layers:{normal} Tear down samplelayers.Layer122 in {green}0.000{normal} seconds. Tear down samplelayers.Layer12 in {green}0.000{normal} seconds. Tear down samplelayers.Layer1 in {green}0.000{normal} seconds. False Failed test ----------- A failed test run highlights the failures in red: >>> sys.argv = 'test -c --tests-pattern ^sampletests(f|_e|_f)?$ '.split() >>> testrunner.run_internal(defaults) {normal}Running zope.testrunner.layer.UnitTests tests:{normal} Set up zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds. {boldred}Failure in test eek (sample2.sampletests_e){normal} Failed doctest test for sample2.sampletests_e.eek File "testrunner-ex/sample2/sampletests_e.py", line 28, in eek ---------------------------------------------------------------------- {normal}File "{boldblue}testrunner-ex/sample2/sampletests_e.py{normal}", line {boldred}30{normal}, in {boldcyan}sample2.sampletests_e.eek{normal} Failed example: {cyan} f(){normal} Exception raised: {red} Traceback (most recent call last):{normal} {red} File ".../doctest.py", line 1356, in __run{normal} {red} compileflags, 1)...{normal} {red} File "", line 1, in ?{normal} {red} f(){normal} {red} File "testrunner-ex/sample2/sampletests_e.py", line 19, in f{normal} {red} g(){normal} {red} File "testrunner-ex/sample2/sampletests_e.py", line 24, in g{normal} {red} x = y + 1{normal} {red} - __traceback_info__: I don't know what Y should be.{normal} {red} NameError: global name 'y' is not defined{normal} {boldred}Error in test test3 (sample2.sampletests_e.Test){normal} Traceback (most recent call last): {normal} File "{boldblue}unittest.py{normal}", line {boldred}260{normal}, in {boldcyan}run{normal} {cyan} testMethod(){normal} {normal} File "{boldblue}testrunner-ex/sample2/sampletests_e.py{normal}", line {boldred}43{normal}, in {boldcyan}test3{normal} {cyan} f(){normal} {normal} File "{boldblue}testrunner-ex/sample2/sampletests_e.py{normal}", line {boldred}19{normal}, in {boldcyan}f{normal} {cyan} g(){normal} {normal} File "{boldblue}testrunner-ex/sample2/sampletests_e.py{normal}", line {boldred}24{normal}, in {boldcyan}g{normal} {cyan} x = y + 1{normal} {red} - __traceback_info__: I don't know what Y should be.{normal} {red}NameError: global name 'y' is not defined{normal} {boldred}Failure in test testrunner-ex/sample2/e.txt{normal} Failed doctest test for e.txt File "testrunner-ex/sample2/e.txt", line 0 ---------------------------------------------------------------------- {normal}File "{boldblue}testrunner-ex/sample2/e.txt{normal}", line {boldred}4{normal}, in {boldcyan}e.txt{normal} Failed example: {cyan} f(){normal} Exception raised: {red} Traceback (most recent call last):{normal} {red} File ".../doctest.py", line 1356, in __run{normal} {red} compileflags, 1)...{normal} {red} File "", line 1, in ?{normal} {red} f(){normal} {red} File "", line 2, in f{normal} {red} return x{normal} {red} NameError: global name 'x' is not defined{normal} {boldred}Failure in test test (sample2.sampletests_f.Test){normal} Traceback (most recent call last): {normal} File "{boldblue}unittest.py{normal}", line {boldred}260{normal}, in {boldcyan}run{normal} {cyan} testMethod(){normal} {normal} File "{boldblue}testrunner-ex/sample2/sampletests_f.py{normal}", line {boldred}21{normal}, in {boldcyan}test{normal} {cyan} self.assertEqual(1,0){normal} {normal} File "{boldblue}unittest.py{normal}", line {boldred}333{normal}, in {boldcyan}failUnlessEqual{normal} {cyan} raise self.failureException, \{normal} {red}AssertionError: 1 != 0{normal} {normal} Ran {green}164{normal} tests with {boldred}3{normal} failures, {boldred}1{normal} errors, {green}0{normal} skipped in {green}0.045{normal} seconds.{normal} ... {normal}Total: {green}329{normal} tests, {boldred}3{normal} failures, {boldred}1{normal} errors, {green}0{normal} skipped in {green}N.NNN{normal} seconds.{normal} True Skipped tests ------------- Tests which are skipped are colorized:: >>> defaults_skip = [ ... '--path', os.path.join(this_directory, 'testrunner-ex-skip'), ... '--tests-pattern', '^sample_skipped_tests$', ... ] >>> sys.argv = 'test --tests-pattern ^sample_skipped_tests$ -t NoLayer -c -vvvvvv'.split() >>> _ = testrunner.run_internal(defaults_skip) {normal}Running tests at level 1{normal} {normal}Running zope.testrunner.layer.UnitTests tests:{normal} Set up zope.testrunner.layer.UnitTests in {green}0.000{normal} seconds. {normal} Running:{normal} test_skipped (sample_skipped_tests.TestSkipppedNoLayer) ({boldyellow}skipped: I'm a skipped test!{normal}) {normal} Ran {green}1{normal} tests with {green}0{normal} failures, {green}0{normal} errors, {boldyellow}1{normal} skipped in {green}N.NNN{normal} seconds.{normal} {normal}Tearing down left over layers:{normal} Tear down zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds. Doctest failures ---------------- The expected and actual outputs of failed doctests are shown in different colors: >>> sys.argv = 'test --tests-pattern ^pledge$ -c'.split() >>> _ = testrunner.run_internal(defaults) {normal}Running zope.testrunner.layer.UnitTests tests:{normal} Set up zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds. {boldred}Failure in test pledge (pledge){normal} Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- {normal}File testrunner-ex/pledge.py{normal}", line {boldred}26{normal}, in {boldcyan}pledge.pledge{normal} Failed example: {cyan} print_pledge(){normal} Expected: {green} I give my pledge, as an earthling,{normal} {green} to save, and faithfully, to defend from waste,{normal} {green} the natural resources of my planet.{normal} {green} It's soils, minerals, forests, waters, and wildlife.{normal} {green} {normal} Got: {red} I give my pledge, as and earthling,{normal} {red} to save, and faithfully, to defend from waste,{normal} {red} the natural resources of my planet.{normal} {red} It's soils, minerals, forests, waters, and wildlife.{normal} {red} {normal} {normal} Ran {green}1{normal} tests with {boldred}1{normal} failures, {green}0{normal} errors, {green}0{normal} skipped in {green}0.002{normal} seconds.{normal} {normal}Tearing down left over layers:{normal} Tear down zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds. Diffs are highlighted so you can easily tell the context and the mismatches apart: >>> sys.argv = 'test --tests-pattern ^pledge$ --ndiff -c'.split() >>> _ = testrunner.run_internal(defaults) {normal}Running zope.testrunner.layer.UnitTests tests:{normal} Set up zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds. {boldred}Failure in test pledge (pledge){normal} Failed doctest test for pledge.pledge File "testrunner-ex/pledge.py", line 24, in pledge ---------------------------------------------------------------------- {normal}File testrunner-ex/pledge.py{normal}", line {boldred}26{normal}, in {boldcyan}pledge.pledge{normal} Failed example: {cyan} print_pledge(){normal} Differences (ndiff with -expected +actual): {green} - I give my pledge, as an earthling,{normal} {red} + I give my pledge, as and earthling,{normal} {magenta} ? +{normal} {normal} to save, and faithfully, to defend from waste,{normal} {normal} the natural resources of my planet.{normal} {normal} It's soils, minerals, forests, waters, and wildlife.{normal} {normal} {normal} {normal} Ran {green}1{normal} tests with {boldred}1{normal} failures, {green}0{normal} errors, {green}0{normal} skipped in {green}0.003{normal} seconds.{normal} {normal}Tearing down left over layers:{normal} Tear down zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds. Even test failures that have actual blank lines (as opposed to ) in them are highlighted correctly. >>> import zope.testrunner.formatter >>> formatter = zope.testrunner.formatter.ColorfulOutputFormatter(None) >>> formatter.print_doctest_failure("""\ ... File "sometest.txt", line 221, in sometest.txt ... Failed example: ... foo() ... Expected: ... Output that contains ... ... blank lines. ... Got: ... Output that still contains ... ... blank lines.""") {normal} File "sometest.txt", line 221, in sometest.txt{normal} Failed example: {cyan} foo(){normal} Expected: {green} Output that contains{normal} {green} blank lines.{normal} Got: {red} Output that still contains{normal} {red} blank lines.{normal} Timing individual tests ----------------------- At very high verbosity levels you can see the time taken by each test >>> sys.argv = 'test -u -t test_one.TestNotMuch -c -vvv'.split() >>> testrunner.run_internal(defaults) {normal}Running tests at level 1{normal} {normal}Running zope.testrunner.layer.UnitTests tests:{normal} Set up zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds. {normal} Running:{normal} test_1 (sample1.sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal}) test_2 (sample1.sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal}) test_3 (sample1.sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal}) test_1 (sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal}) test_2 (sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal}) test_3 (sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal}) {normal} Ran {green}6{normal} tests with {green}0{normal} failures, {green}0{normal} errors, {green}0{normal} skipped in {green}N.NNN{normal} seconds.{normal} {normal}Tearing down left over layers:{normal} Tear down zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds. False If we had very slow tests we would see their times highlighted in a different color. Instead of creating a test that waits 10 seconds, let's lower the slow test threshold in the test runner to 0 seconds to make all of the tests seem slow. >>> sys.argv = 'test -u -t test_one.TestNotMuch -c -vvv --slow-test 0'.split() >>> testrunner.run_internal(defaults) {normal}Running tests at level 1{normal} {normal}Running zope.testrunner.layer.UnitTests tests:{normal} Set up zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds. {normal} Running:{normal} test_1 (sample1.sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal}) test_2 (sample1.sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal}) test_3 (sample1.sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal}) test_1 (sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal}) test_2 (sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal}) test_3 (sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal}) {normal} Ran {green}6{normal} tests with {green}0{normal} failures, {green}0{normal} errors, {green}0{normal} skipped in {green}N.NNN{normal} seconds.{normal} {normal}Tearing down left over layers:{normal} Tear down zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds. False Disabling colors ---------------- If -c or --color have been previously provided on the command line (perhaps by a test runner wrapper script), but no colorized output is desired, the -C or --no-color options will disable colorized output: >>> sys.argv = 'test --layer 122 -c -C'.split() >>> testrunner.run_internal(defaults) Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.007 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False >>> sys.argv = 'test --layer 122 -c --no-color'.split() >>> testrunner.run_internal(defaults) Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.007 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False Autodetecting colors -------------------- The --auto-color option will determine if stdout is a terminal that supports colors, and only enable colorized output if so. Our ``Terminal`` wrapper pretends it is a terminal, but the curses module will realize it isn't: >>> sys.argv = 'test --layer 122 --auto-color'.split() >>> testrunner.run_internal(defaults) Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.007 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False We can fake it >>> class FakeCurses(object): ... class error(Exception): ... pass ... def setupterm(self): ... pass ... def tigetnum(self, attr): ... return dict(colors=8).get(attr, -2) >>> sys.modules['curses'] = FakeCurses() >>> sys.argv = 'test --layer 122 --auto-color'.split() >>> testrunner.run_internal(defaults) {normal}Running samplelayers.Layer122 tests:{normal} Set up samplelayers.Layer1 in {green}0.000{normal} seconds. Set up samplelayers.Layer12 in {green}0.000{normal} seconds. Set up samplelayers.Layer122 in {green}0.000{normal} seconds. {normal} Ran {green}26{normal} tests with {green}0{normal} failures, {green}0{normal} errors, {green}0{normal} skipped in {green}0.007{normal} seconds.{normal} {normal}Tearing down left over layers:{normal} Tear down samplelayers.Layer122 in {green}0.000{normal} seconds. Tear down samplelayers.Layer12 in {green}0.000{normal} seconds. Tear down samplelayers.Layer1 in {green}0.000{normal} seconds. False >>> del sys.modules['curses'] The real stdout is not a terminal in a doctest: >>> sys.stdout = real_stdout >>> sys.argv = 'test --layer 122 --auto-color'.split() >>> testrunner.run_internal(defaults) Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.007 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-layers-topological-sort.txt0000664000000000000000000000614312527371226030315 0ustar rootrootNow lets stack a few more layers to ensure that our setUp and tearDown methods are called in the correct order. >>> from zope.testing.loggingsupport import InstalledHandler >>> from zope.testrunner.find import name_from_layer >>> from zope.testrunner import options >>> from zope.testrunner.runner import Runner >>> import logging >>> import unittest >>> def fresh_options(): ... opts = options.get_options(['--test-filter', '.*']) ... opts.resume_layer = None ... opts.resume_number = 0 ... return opts >>> def log(msg): ... logging.getLogger('zope.testrunner.tests').info(msg) >>> log_handler = InstalledHandler('zope.testrunner.tests') >>> class A(object): ... def setUp(cls): ... log('%s.setUp' % name_from_layer(cls)) ... setUp = classmethod(setUp) ... ... def tearDown(cls): ... log('%s.tearDown' % name_from_layer(cls)) ... tearDown = classmethod(tearDown) ... ... def testSetUp(cls): ... log('%s.testSetUp' % name_from_layer(cls)) ... testSetUp = classmethod(testSetUp) ... ... def testTearDown(cls): ... log('%s.testTearDown' % name_from_layer(cls)) ... testTearDown = classmethod(testTearDown) ... >>> class AB(A): pass >>> class AC(A): pass >>> class AAAABD(AB): pass >>> class ZZZABE(AB): pass >>> class MMMACF(AC): pass >>> class DeepTest1(unittest.TestCase): ... layer = AAAABD ... def test(self): ... pass >>> class DeepTest2(unittest.TestCase): ... layer = MMMACF ... def test(self): ... pass >>> class DeepTest3(unittest.TestCase): ... layer = ZZZABE ... def test(self): ... pass >>> class QuickTests(unittest.TestCase): ... def test(self): ... pass >>> suite = unittest.TestSuite() >>> suite.addTest(unittest.makeSuite(DeepTest1)) >>> suite.addTest(unittest.makeSuite(DeepTest2)) >>> suite.addTest(unittest.makeSuite(DeepTest3)) >>> suite.addTest(unittest.makeSuite(QuickTests)) >>> log_handler.clear() >>> runner = Runner(options=fresh_options(), args=[], found_suites=[suite]) >>> succeeded = runner.run() #doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running ...AAAABD tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up ...A in N.NNN seconds. Set up ...AB in N.NNN seconds. Set up ...AAAABD in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running ...ZZZABE tests: Tear down ...AAAABD in N.NNN seconds. Set up ...ZZZABE in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running ...MMMACF tests: Tear down ...ZZZABE in N.NNN seconds. Tear down ...AB in N.NNN seconds. Set up ...AC in N.NNN seconds. Set up ...MMMACF in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down ...MMMACF in N.NNN seconds. Tear down ...AC in N.NNN seconds. Tear down ...A in N.NNN seconds. Total: 4 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-layers.txt0000664000000000000000000002327012527371226025016 0ustar rootrootLayer Selection =============== We can select which layers to run using the --layer option: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 112 --layer Unit'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 182 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False We can also specify that we want to run only the unit tests: >>> sys.argv = 'test -u'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Or that we want to run all of the tests except for the unit tests: >>> sys.argv = 'test -f'.split() >>> testrunner.run_internal(defaults) Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 165 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Or we can explicitly say that we want both unit and non-unit tests. >>> sys.argv = 'test -uf'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False It is possible to force the layers to run in subprocesses and parallelize them. ``EmptyLayer`` will be inserted as first to start spreading out subprocesses ASAP. >>> sys.argv = [testrunner_script, '-j2'] >>> testrunner.run_internal(defaults) Running .EmptyLayer tests: Set up .EmptyLayer in N.NNN seconds. Ran 0 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running zope.testrunner.layer.UnitTests tests: Running in a subprocess. Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Running samplelayers.Layer1 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer11 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer111 tests: Running in a subprocess. Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer111 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer112 tests: Running in a subprocess. Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer12 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer121 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer121 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Running samplelayers.Layer122 tests: Running in a subprocess. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Tearing down left over layers: Tear down .EmptyLayer in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-discovery.txt0000664000000000000000000000345712527371226025533 0ustar rootrootAutomatically discovering tests =============================== You can explicitly specify which tests to run by providing a function that returns a unittest.TestSuite in the test modules (the name of the function can be configured with the --suite-name parameter, it defaults to 'test_suite'). If no such function is present, testrunner will use all classes in the module that inherit from unittest.TestCase as tests: >>> import os, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_discover$', ... ] >>> testrunner.run(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False If the module neither provides a TestSuite nor has discoverable tests, testrunner will exit with an error to prevent acidentally missing test cases: >>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_discover_notests$', ... ] >>> testrunner.run(defaults) Test-module import failures: Module: sample1.sampletests_discover_notests TypeError: Module sample1.sampletests_discover_notests does not define any tests Test-modules with import problems: sample1.sampletests_discover_notests Total: 0 tests, 0 failures, 0 errors and 0 skipped in 0.000 seconds. True zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-skip/0000775000000000000000000000000012552256734024336 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-skip/sample_skipped_tests.py0000664000000000000000000000230612527371226031127 0ustar rootroot############################################################################## # # Copyright (c) 2013 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## try: import unittest2 as unittest except ImportError: import unittest class TestSkipppedWithLayer(unittest.TestCase): layer = "sample_skipped_tests.Layer" @unittest.skip('Hop, skipped') def test_layer_skipped(self): pass def test_layer_pass(self): pass class Layer: pass class TestSkipppedNoLayer(unittest.TestCase): # Only one test, so we don't have to bother about ordering when the runner # displays the tests' list while in verbose mode. @unittest.skip("I'm a skipped test!") def test_skipped(self): pass zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-nestedcode.txt0000664000000000000000000000523712527371226025637 0ustar rootrootThis testrunner_ex subdirectory contains a number of sample packages with tests. Let's run the tests found here. We simulate here, that you use nested locations for your different source packages and that within your source, you checked out other source directories >>> import os.path >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', this_directory, ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> import sys >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in N.NNN seconds. Set up samplelayers.Layer112 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-leaks-err.txt0000664000000000000000000000145312527371226025403 0ustar rootrootDebugging Memory Leaks without a debug build of Python ====================================================== To use the --report-refcounts (-r) to detect or debug memory leaks, you must have a debug build of Python. Without a debug build, you will get an error message: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> sys.argv = 'test -r -N6'.split() >>> _ = testrunner.run_internal(defaults) The Python you are running was not configured with --with-pydebug. This is required to use the --report-refcounts option. zope.testrunner-4.4.9/src/zope/testrunner/tests/test_subunit.py0000664000000000000000000000552512527371226023654 0ustar rootroot############################################################################## # # Copyright (c) 2010 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Unit tests for the testrunner's subunit integration. """ import sys import unittest from six import StringIO from zope.testrunner import formatter try: unichr except NameError: unichr = chr # Python 3 try: import subunit except ImportError: def test_suite(): return unittest.TestSuite() else: class TestSubunitTracebackPrinting(unittest.TestCase): def makeByteStringFailure(self, text, encoding): try: # Note that this deliberately throws a string of bytes instead # of a unicode object. This simulates errors thrown by # utf8-encoded doctests. bytestr = text.encode(encoding) self.fail(bytestr) except self.failureException: return sys.exc_info() def setUp(self): class FormatterOptions: verbose=False options = FormatterOptions() self.output = StringIO() self.subunit_formatter = formatter.SubunitOutputFormatter( options, stream=self.output) def test_print_failure_containing_utf8_bytestrings(self): exc_info = self.makeByteStringFailure(unichr(6514), 'utf8') self.subunit_formatter.test_failure(self, 0, exc_info) assert "AssertionError: \xe1\xa5\xb2" in self.output.getvalue() # '\xe1\xa5\xb2'.decode('utf-8') == unichr(6514) def test_print_error_containing_utf8_bytestrings(self): exc_info = self.makeByteStringFailure(unichr(6514), 'utf8') self.subunit_formatter.test_error(self, 0, exc_info) assert "AssertionError: \xe1\xa5\xb2" in self.output.getvalue() # '\xe1\xa5\xb2'.decode('utf-8') == unichr(6514) def test_print_failure_containing_latin1_bytestrings(self): exc_info = self.makeByteStringFailure(unichr(241), 'latin1') self.subunit_formatter.test_failure(self, 0, exc_info) assert "AssertionError: \xef\xbf\xbd" in self.output.getvalue() # '\xef\xbf\xbd'.decode('utf-8') = unichr(0xFFFD) def test_suite(): return unittest.TestSuite(( unittest.makeSuite(TestSubunitTracebackPrinting), )) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-subunit-leaks.txt0000664000000000000000000000672612527371226026314 0ustar rootrootDebugging Memory Leaks with subunit output ========================================== The --report-refcounts (-r) option can be used with the --repeat (-N) option to detect and diagnose memory leaks. To use this option, you must configure Python with the --with-pydebug option. (On Unix, pass this option to configure and then build Python.) For more detailed documentation, see testrunner-leaks.txt. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner Each layer is repeated the requested number of times. For each iteration after the first, the system refcount and change in system refcount is shown. The system refcount is the total of all refcount in the system. When a refcount on any object is changed, the system refcount is changed by the same amount. Tests that don't leak show zero changes in system refcount. Let's look at an example test that leaks: >>> sys.argv = 'test --subunit --tests-pattern leak -N2 -r'.split() >>> _ = testrunner.run_internal(defaults) time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: zope.testrunner.layer.UnitTests:setUp tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: zope.testrunner.layer.UnitTests:setUp tags: zope:layer:zope.testrunner.layer.UnitTests test: leak.TestSomething.testleak successful: leak.TestSomething.testleak test: leak.TestSomething.testleak successful: leak.TestSomething.testleak test: zope:refcounts tags: zope:refcounts successful: zope:refcounts [ multipart Content-Type: text/plain;charset=utf8 ... ...\r ...\r Content-Type: text/plain;charset=utf8 ... ...\r ...\r ] tags: -zope:layer:zope.testrunner.layer.UnitTests time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: zope.testrunner.layer.UnitTests:tearDown tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: zope.testrunner.layer.UnitTests:tearDown Here we see that the system refcount is increasing. If we specify a verbosity greater than one, we can get details broken out by object type (or class): >>> sys.argv = 'test --subunit --tests-pattern leak -N2 -v -r'.split() >>> _ = testrunner.run_internal(defaults) time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: zope.testrunner.layer.UnitTests:setUp tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: zope.testrunner.layer.UnitTests:setUp tags: zope:layer:zope.testrunner.layer.UnitTests test: leak.TestSomething.testleak successful: leak.TestSomething.testleak test: leak.TestSomething.testleak successful: leak.TestSomething.testleak test: zope:refcounts tags: zope:refcounts successful: zope:refcounts [ multipart Content-Type: text/plain;charset=utf8 ... ...\r ...\r Content-Type: text/plain;charset=utf8 ... ...\r ...\r Content-Type: text/plain;charset=utf8 ... ...\r ... ] tags: -zope:layer:zope.testrunner.layer.UnitTests time: YYYY-MM-DD HH:MM:SS.mmmmmmZ test: zope.testrunner.layer.UnitTests:tearDown tags: zope:layer time: YYYY-MM-DD HH:MM:SS.mmmmmmZ successful: zope.testrunner.layer.UnitTests:tearDown zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-wo-source.txt0000664000000000000000000001061012527371226025434 0ustar rootrootRunning Without Source Code =========================== The ``--usecompiled`` option allows running tests in a tree without .py source code, provided compiled .pyc or .pyo files exist (without ``--usecompiled``, .py files are necessary). We have a very simple directory tree, under ``usecompiled/``, to test this. Because we're going to delete its .py files, we want to work in a copy of that: >>> import os.path, shutil, sys, tempfile >>> directory_with_tests = tempfile.mkdtemp() >>> NEWNAME = "unlikely_package_name" >>> src = os.path.join(this_directory, 'testrunner-ex', 'usecompiled') >>> os.path.isdir(src) True >>> dst = os.path.join(directory_with_tests, NEWNAME) >>> os.path.isdir(dst) False Have to use our own copying code, to avoid copying read-only SVN files that can't be deleted later. >>> n = len(src) + 1 >>> for root, dirs, files in os.walk(src): ... dirs[:] = [d for d in dirs if d == "package"] # prune cruft ... os.mkdir(os.path.join(dst, root[n:])) ... for f in files: ... _ = shutil.copy(os.path.join(root, f), ... os.path.join(dst, root[n:], f)) Now run the tests in the copy: >>> from zope import testrunner >>> mydefaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^compiletest$', ... '--package', NEWNAME, ... '-vv', ... ] >>> sys.argv = ['test'] >>> testrunner.run_internal(mydefaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test1 (unlikely_package_name.compiletest.Test) test2 (unlikely_package_name.compiletest.Test) test1 (unlikely_package_name.package.compiletest.Test) test2 (unlikely_package_name.package.compiletest.Test) Ran 4 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False If we delete the source files, it's normally a disaster: the test runner doesn't believe any test files, or even packages, exist. Note that we pass ``--keepbytecode`` this time, because otherwise the test runner would delete the compiled Python files too: >>> for root, dirs, files in os.walk(dst): ... for f in files: ... if f.endswith(".py"): ... os.remove(os.path.join(root, f)) >>> testrunner.run_internal(mydefaults, ["test", "--keepbytecode"]) Running tests at level 1 Total: 0 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Finally, passing ``--usecompiled`` asks the test runner to treat .pyc and .pyo files as adequate replacements for .py files. Note that the output is the same as when running with .py source above. The absence of "removing stale bytecode ..." messages shows that ``--usecompiled`` also implies ``--keepbytecode``: >>> if sys.version_info >= (3, 2): ... # PEP-3147: pyc files in __pycache__ directories cannot be ... # imported; legacy source-less imports need to use the legacy ... # layout ... for root, dirs, files in os.walk(dst): ... for f in files: ... if f.endswith((".pyc", ".pyo")): ... # "root/f" is "dirname/__pycache__/name.magic.ext" ... dirname = os.path.dirname(os.path.abspath(root)) ... namewmagic, ext = os.path.splitext(os.path.basename(f)) ... newname = os.path.splitext(namewmagic)[0] + ext ... os.rename(os.path.join(root, f), ... os.path.join(dirname, newname)) >>> testrunner.run_internal(mydefaults, ["test", "--usecompiled"]) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test1 (unlikely_package_name.compiletest.Test) test2 (unlikely_package_name.compiletest.Test) test1 (unlikely_package_name.package.compiletest.Test) test2 (unlikely_package_name.package.compiletest.Test) Ran 4 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Remove the temporary directory: >>> shutil.rmtree(directory_with_tests) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-report-skipped.txt0000664000000000000000000000555212527371226026472 0ustar rootroottestrunner handling of skipped tests ==================================== `unittest 2`_, which has been merged into Python 2.7 and Python 3.1, provides a way to "skip" tests: http://docs.python.org/2/library/unittest.html#skipping-tests-and-expected-failures This feature is reported by the test runner. :: >>> import os, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex-skip') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sample_skipped_tests$', ... ] >>> sys.argv = ['test'] Show how many skipped tests has been run ---------------------------------------- By default, the runner displays how many tests has been skipped, as part of the result line:: >>> testrunner.run_internal(defaults + ["-t", "TestSkipppedNoLayer"]) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 1 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False If tests has been skipped in a layer, they are reported in the layer's status line, and in the total status line:: >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 1 skipped in 0.000 seconds. Running sample_skipped_tests.Layer tests: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds. Set up sample_skipped_tests.Layer in 0.000 seconds. Ran 2 tests with 0 failures, 0 errors and 1 skipped in 0.000 seconds. Tearing down left over layers: Tear down sample_skipped_tests.Layer in 0.000 seconds. Total: 3 tests, 0 failures, 0 errors and 2 skipped in 0.034 seconds. False Show the reason why tests have been skipped ------------------------------------------- By changing the verbose level of the runner itself, you can get increasing number of information:: >>> testrunner.run_internal(defaults + ["-t", "TestSkipppedNoLayer"]) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 1 tests with 0 failures, 0 errors and 1 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False >>> testrunner.run_internal(defaults + ["-t", "TestSkipppedNoLayer", "-vv"]) Running tests at level 1 ... test_skipped (sample_skipped_tests.TestSkipppedNoLayer) (skipped) ... >>> testrunner.run_internal(defaults + ["-t", "TestSkipppedNoLayer", "-vvv"]) Running tests at level 1 ... test_skipped (sample_skipped_tests.TestSkipppedNoLayer) (skipped: I'm a skipped test!) ... zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/0000775000000000000000000000000012552256734024144 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/0000775000000000000000000000000012552256734025071 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/0000775000000000000000000000000012552256734030331 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/0000775000000000000000000000000012552256734031114 5ustar rootroot././@LongLink0000644000000000000000000000015100000000000011600 Lustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/bar/zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/0000775000000000000000000000000012552256734031114 5ustar rootroot././@LongLink0000644000000000000000000000016400000000000011604 Lustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/bar/__init__.pyzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/0000664000000000000000000000001112527371226031102 0ustar rootroot#package ././@LongLink0000644000000000000000000000016100000000000011601 Lustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/bar/tests.pyzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/0000664000000000000000000000000712527371226031107 0ustar rootroot#tests ././@LongLink0000644000000000000000000000016000000000000011600 Lustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/__init__.pyzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/foo/0000664000000000000000000000000012527371226031100 0ustar rootroot././@LongLink0000644000000000000000000000015000000000000011577 Lustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/test.pyzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/test0000664000000000000000000000000012527371226031215 0ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-debugging-nonprintable-exc.test0000664000000000000000000000326712527371226031064 0ustar rootrootRegression tests for https://github.com/zopefoundation/zope.testrunner/issues/8 Post-mortem debugging also works when the exception cannot be printed >>> import os, shutil, sys, tempfile >>> tdir = tempfile.mkdtemp() >>> dir = os.path.join(tdir, 'TESTS-DIR') >>> os.mkdir(dir) >>> def write_file(filename, body): ... with open(os.path.join(dir, filename), 'w') as f: ... f.write(body) ... try: ... # Need to do this on Python 3.3 after creating new modules ... import importlib; importlib.invalidate_caches() ... except (ImportError, AttributeError): ... pass >>> write_file('tests.py', ... r''' ... import unittest ... import six ... ... class MyTest(unittest.TestCase): ... def test(self): ... self.assertEqual(six.u('a'), six.b('\xc4\x85').decode('UTF-8')) ... ''') >>> class Input: ... def __init__(self, src): ... self.lines = src.split('\n') ... def readline(self): ... line = self.lines.pop(0) ... print(line) ... return line+'\n' >>> real_stdin = sys.stdin >>> sys.stdin = Input('c') >>> sys.argv = [testrunner_script] >>> import zope.testrunner >>> try: ... zope.testrunner.run_internal(['--path', dir, '-D']) ... except zope.testrunner.interfaces.EndRun: print('EndRun raised') ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS +REPORT_NDIFF Running zope.testrunner.layer.UnitTests tests: ... AssertionError: ...'a' != ...'...' ... (Pdb) c Tearing down left over layers: ... False >>> shutil.rmtree(tdir) zope.testrunner-4.4.9/src/zope/testrunner/tests/test_runner.py0000664000000000000000000001417112527371226023471 0ustar rootroot############################################################################## # # Copyright (c) 2015 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Unit tests for the testrunner's runner logic """ import unittest from zope.testrunner import runner from zope.testrunner.layer import UnitTests class TestLayerOrdering(unittest.TestCase): def sort_key(self, layer): return ', '.join(qn.rpartition('.')[-1] for qn in runner.layer_sort_key(layer)) def order(self, *layers): return ', '.join(l.__name__ for l in runner.order_by_bases(layers)) def test_order_by_bases(self): # A B # /|\ / # / | \ / # A1 A2 AB class A(object): pass class A1(A): pass class A2(A): pass class B(object): pass class AB(A, B): pass self.assertEqual(self.order(B, A1, A2, A1, AB, UnitTests), 'UnitTests, A1, A2, B, AB') self.assertEqual(self.sort_key(UnitTests), '') self.assertEqual(self.sort_key(A1), 'A, A1') self.assertEqual(self.sort_key(A2), 'A, A2') self.assertEqual(self.sort_key(B), 'B') self.assertEqual(self.sort_key(AB), 'B, A, AB') def test_order_by_bases_alphabetical_order(self): class X(object): pass class Y(object): pass class Z(object): pass class A(Y): pass class B(X): pass class C(Z): pass # It'd be nice to get A, B, C here, but that's not trivial. self.assertEqual(self.order(A, B, C), 'B, A, C') self.assertEqual(self.sort_key(B), 'X, B') self.assertEqual(self.sort_key(A), 'Y, A') self.assertEqual(self.sort_key(C), 'Z, C') def test_order_by_bases_diamond_hierarchy(self): # A # / \ # B D # | | # C E # \ / # F class A(object): pass class B(A): pass class C(B): pass class D(A): pass class E(D): pass class F(C, E): pass self.assertEqual(self.order(A, B, C, D, E, F), 'A, B, C, D, E, F') self.assertEqual(self.sort_key(A), 'A') self.assertEqual(self.sort_key(B), 'A, B') self.assertEqual(self.sort_key(C), 'A, B, C') self.assertEqual(self.sort_key(D), 'A, D') self.assertEqual(self.sort_key(E), 'A, D, E') self.assertEqual(self.sort_key(F), 'A, D, E, B, C, F') # only those layers that were passed in are returned self.assertEqual(self.order(B, E), 'B, E') def test_order_by_bases_shared_setup_trumps_alphabetical_order(self): # A # / \ # AB AC # / \ \ # AAAABD \ MMMACF # ZZZABE class A(object): pass class AB(A): pass class AC(A): pass class AAAABD(AB): pass class ZZZABE(AB): pass class MMMACF(AC): pass self.assertEqual(self.order(AAAABD, MMMACF, ZZZABE), 'AAAABD, ZZZABE, MMMACF') self.assertEqual(self.sort_key(AAAABD), 'A, AB, AAAABD') self.assertEqual(self.sort_key(ZZZABE), 'A, AB, ZZZABE') self.assertEqual(self.sort_key(MMMACF), 'A, AC, MMMACF') def test_order_by_bases_multiple_inheritance(self): # Layerx Layer1 # | \ / \ # | \ Layer11 Layer12 # | \ | | / \ # | Layer111 | Layer121 Layer122 # \ / # \ / # Layer112 # class Layer1: pass class Layerx: pass class Layer11(Layer1): pass class Layer12(Layer1): pass class Layer111(Layerx, Layer11): pass class Layer121(Layer12): pass class Layer112(Layerx, Layer11): pass class Layer122(Layer12): pass self.assertEqual( self.order(Layer1, Layer11, Layer12, Layer111, Layer112, Layer121, Layer122), 'Layer1, Layer11, Layer111, Layer112, Layer12, Layer121, Layer122') self.assertEqual(self.order(Layer111, Layer12), 'Layer111, Layer12') self.assertEqual(self.order(Layerx, Layer1, Layer11, Layer112), 'Layer1, Layer11, Layerx, Layer112') self.assertEqual(self.sort_key(Layer111), 'Layer1, Layer11, Layerx, Layer111') self.assertEqual(self.sort_key(Layer12), 'Layer1, Layer12') def test_order_by_bases_reverse_tree(self): # E D F # \ / \ / # B C # \ / # A class F(object): pass class E(object): pass class D(object): pass class C(D, F): pass class B(D, E): pass class A(B, C): pass self.assertEqual(self.order(A, B, C), 'B, C, A') self.assertEqual(self.order(A, B, C, D, E, F), 'D, E, B, F, C, A') def test_order_by_bases_mro_is_complicated(self): # A C B E D # / \ | / \ | / \ # \ K1 K2 / # `---. _--' # \ \ / | # \ K3 / # \ |/ # ZZ class A(object): pass class B(object): pass class C(object): pass class D(object): pass class E(object): pass class K1(A, B, C): pass class K2(D, B, E): pass class K3(D, A): pass class ZZ(K1, K2, K3): pass self.assertEqual(self.order(K1, K2, K3, ZZ), 'K3, K2, K1, ZZ') # Sorting by reverse MRO, as computed by Python's MRO algorithm, # would put the layers in a different order: K3, K1, K2, ZZ. # Does that matter? The class diagram is symmetric, so I think not. zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-layers-buff.txt0000664000000000000000000001542212527371226025736 0ustar rootrootThis is a test for a fix in buffering of output from a layer in a subprocess. Prior to the change that this tests, output from within a test layer in a subprocess would be buffered. This could wreak havoc on supervising processes (or human) that would kill a test run if no output was seen for some period of time. First, we wrap stdout with an object that instruments it. It notes the time at which a given line was written. >>> import os, sys, datetime >>> class RecordingStreamWrapper: ... def __init__(self, wrapped): ... self.record = [] ... self.wrapped = wrapped ... def write(self, out): ... self.record.append((out, datetime.datetime.now())) ... self.wrapped.write(out) ... def writelines(self, lines): ... for line in lines: ... self.write(line) ... def flush(self): ... self.wrapped.flush() ... >>> sys.stdout = RecordingStreamWrapper(sys.stdout) Now we actually call our test. If you open the file to which we are referring here (zope/testrunner/tests/testrunner-ex/sampletests_buffering.py) you will see two test suites, each with its own layer that does not know how to tear down. This forces the second suite to be run in a subprocess. That second suite has two tests. Both sleep for half a second each. >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... ] >>> argv = [sys.argv[0], ... '-vv', '--tests-pattern', '^sampletests_buffering.*'] >>> try: ... testrunner.run_internal(defaults, argv) ... record = sys.stdout.record ... finally: ... sys.stdout = sys.stdout.wrapped ... Running tests at level 1 Running sampletests_buffering.Layer1 tests: Set up sampletests_buffering.Layer1 in N.NNN seconds. Running: test_something (sampletests_buffering.TestSomething1) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running sampletests_buffering.Layer2 tests: Tear down sampletests_buffering.Layer1 ... not supported Running in a subprocess. Set up sampletests_buffering.Layer2 in N.NNN seconds. Running: test_something (sampletests_buffering.TestSomething2) test_something2 (sampletests_buffering.TestSomething2) Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down sampletests_buffering.Layer2 ... not supported Total: 3 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Now we actually check the results we care about. We should see that there are two pauses of about half a second, one around the first test and one around the second. Before the change that this test verifies, there was a single pause of more than a second after the second suite ran. >>> def assert_progressive_output(): ... pause = datetime.timedelta(seconds=0.3) ... last_line, last_time = record.pop(0) ... print('---') ... for line, time in record: ... if time-last_time >= pause: ... # We paused! ... print('PAUSE FOUND BETWEEN THESE LINES:') ... print(''.join([last_line, line, '-'*70])) ... last_line, last_time = line, time >>> assert_progressive_output() # doctest: +ELLIPSIS ---... PAUSE FOUND BETWEEN THESE LINES:... Running: test_something (sampletests_buffering.TestSomething2) ---------------------------------------------------------------------- PAUSE FOUND BETWEEN THESE LINES: test_something (sampletests_buffering.TestSomething2) test_something2 (sampletests_buffering.TestSomething2) ---... Because this is a test based on timing, it may be somewhat fragile. However, on a relatively slow machine, this timing works out fine; I'm hopeful that this test will not be a source of spurious errors. If it is, we will have to readdress. Now let's do the same thing, but with multiple processes at once. We'll get a different result that has similar characteristics. Note that we don't have to use a special layer that doesn't support teardown to force the layer we're interested in to run in a subprocess: the test runner now does that when you ask for parallel execution. The other layer now just makes the test output non-deterministic, so we'll skip it. >>> sys.stdout = RecordingStreamWrapper(sys.stdout) >>> argv.extend(['-j', '2', '--layer=sampletests_buffering.Layer2']) >>> try: ... testrunner.run_internal(defaults, argv) ... record = sys.stdout.record ... finally: ... sys.stdout = sys.stdout.wrapped ... Running tests at level 1 Running .EmptyLayer tests: Set up .EmptyLayer in N.NNN seconds. Running: Ran 0 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. [Parallel tests running in sampletests_buffering.Layer2: .. LAYER FINISHED] Running sampletests_buffering.Layer2 tests: Running in a subprocess. Set up sampletests_buffering.Layer2 in N.NNN seconds. Ran 2 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tear down sampletests_buffering.Layer2 ... not supported Tearing down left over layers: Tear down .EmptyLayer in N.NNN seconds. Total: 2 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Notice that, with a -vv (or greater) verbosity, the parallel test run includes a progress report to keep track of tests run in the various layers. Because the actual results are saved to be displayed assembled in the original test order, the progress report shows up before we are given the news that the testrunner is starting Layer2. This is counterintuitive, but lets us keep the primary reporting information for the given layer in one location, while also giving us progress reports that can be used for keepalive analysis by a human or automated agent. In particular for the second point, notice below that, as before, the progress output is not buffered. >>> def assert_progressive_output(): ... pause = datetime.timedelta(seconds=0.3) ... last_line, last_time = record.pop(0) ... print('---') ... for line, time in record: ... if time-last_time >= pause: ... # We paused! ... print('PAUSE FOUND BETWEEN THIS OUTPUT:') ... print('\n'.join([last_line, line, '-'*70])) ... last_line, last_time = line, time >>> assert_progressive_output() # doctest: +ELLIPSIS ---... PAUSE FOUND BETWEEN THIS OUTPUT:... . . ---------------------------------------------------------------------- PAUSE FOUND BETWEEN THIS OUTPUT: . LAYER FINISHED ---... zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-unexpected-success.txt0000664000000000000000000000312612527371226027327 0ustar rootroottestrunner handling of unexpected success ========================================= Python 2.7 introduced the concept of expectedFailures to unittest. See http://www.voidspace.org.uk/python/articles/unittest2.shtml#more-skipping Although testrunner is currently not able to hande unexpected successes correctly at least it does not report them as successes. This document has some edge-case examples to test various aspects of the test runner. Separating Python path and test directories ------------------------------------------- The --path option defines a directory to be searched for tests *and* a directory to be added to Python's search path. The --test-path option can be used when you want to set a test search path without also affecting the Python path: >>> import os, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex-719369') >>> from zope import testrunner >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error in test test_ef (sampletestsf.TestUnexpectedSuccess) Traceback (most recent call last): ..._UnexpectedSuccess Ran 1 tests with 1 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. True zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-debugging-import-failure.test0000664000000000000000000000425612527371226030552 0ustar rootrootRegression tests for https://bugs.launchpad.net/zope.testrunner/+bug/1119363 Post-mortem debugging also works when there is an import failure. >>> import os, shutil, sys, tempfile >>> tdir = tempfile.mkdtemp() >>> dir = os.path.join(tdir, 'TESTS-DIR') >>> os.mkdir(dir) >>> def write_file(filename, body): ... with open(os.path.join(dir, filename), 'w') as f: ... f.write(body) ... try: ... # Need to do this on Python 3.3 after creating new modules ... import importlib; importlib.invalidate_caches() ... except (ImportError, AttributeError): ... pass >>> write_file('tests.py', ... ''' ... impot doctest ... ''') >>> class Input: ... def __init__(self, src): ... self.lines = src.split('\n') ... def readline(self): ... line = self.lines.pop(0) ... print(line) ... return line+'\n' >>> real_stdin = sys.stdin >>> sys.stdin = Input('c') >>> sys.argv = [testrunner_script] >>> import zope.testrunner >>> try: ... zope.testrunner.run_internal(['--path', dir, '-D']) ... except zope.testrunner.interfaces.EndRun: print('EndRun raised') ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS +REPORT_NDIFF File ".../TESTS-DIR/tests.py", line 2 impot doctest ^ SyntaxError: invalid syntax > ...find.py(399)import_name() -> __import__(name) (Pdb) c EndRun raised Post-mortem debugging also works when the test suite is invalid: >>> sys.stdin = Input('c') >>> write_file('tests2.py', ... ''' ... def test_suite(): ... return None ... ''') >>> import sys >>> try: ... zope.testrunner.run_internal( ... ['--path', dir, '-Dvv', '--tests-pattern', 'tests2']) ... except zope.testrunner.interfaces.EndRun: print('EndRun raised') ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS +REPORT_NDIFF TypeError: Invalid test_suite, None, in tests2 > ...find.py(217)find_suites() -> % (suite, module_name) (Pdb) c EndRun raised >>> shutil.rmtree(tdir) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-tb-format.txt0000664000000000000000000000000012527371226025374 0ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-coverage.txt0000664000000000000000000001301012527371226025301 0ustar rootrootCode Coverage ============= If the --coverage option is used, test coverage reports will be generated. The directory name given as the parameter will be used to hold the reports. >>> import os.path, sys, tempfile >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> tempdir = tempfile.mkdtemp(prefix='zope.testrunner-') >>> coverage_dir = os.path.join(tempdir, 'coverage_dir') >>> sys.argv = ['test', '--coverage', coverage_dir] >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in 0.000 seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in 0.687 seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Running samplelayers.Layer11 tests: Set up samplelayers.Layer11 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer112 tests: Tear down samplelayers.Layer111 in 0.000 seconds. Set up samplelayers.Layer112 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer12 tests: Tear down samplelayers.Layer112 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.140 seconds. Running samplelayers.Layer121 tests: Set up samplelayers.Layer121 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Running samplelayers.Layer122 tests: Tear down samplelayers.Layer121 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. lines cov% module (path) ... 48 100% sampletests.test1 (testrunner-ex/sampletests/test1.py) 74 100% sampletests.test11 (testrunner-ex/sampletests/test11.py) 74 100% sampletests.test111 (testrunner-ex/sampletests/test111.py) 76 100% sampletests.test112 (testrunner-ex/sampletests/test112.py) 74 100% sampletests.test12 (testrunner-ex/sampletests/test12.py) 74 100% sampletests.test121 (testrunner-ex/sampletests/test121.py) 74 100% sampletests.test122 (testrunner-ex/sampletests/test122.py) 48 100% sampletests.test_one (testrunner-ex/sampletests/test_one.py) 112 95% sampletestsf (testrunner-ex/sampletestsf.py) Total: 321 tests, 0 failures, 0 errors and 0 skipped in 0.630 seconds. False The directory specified with the --coverage option will have been created and will hold the coverage reports. >>> os.path.exists(coverage_dir) True >>> os.listdir(coverage_dir) [...] (We should clean up after ourselves.) >>> import shutil >>> shutil.rmtree(tempdir) Ignoring Tests -------------- The ``trace`` module supports ignoring directories and modules based the test selection. Only directories selected for testing should report coverage. The test runner provides a custom implementation of the relevant API. The ``TestIgnore`` class, the class managing the ignoring, is initialized by passing the command line options. It uses the options to determine the directories that should be covered. >>> class FauxOptions(object): ... package = None ... test_path = [('/myproject/src/blah/foo', ''), ... ('/myproject/src/blah/bar', '')] >>> from zope.testrunner import coverage >>> from zope.testrunner.find import test_dirs >>> ignore = coverage.TestIgnore(test_dirs(FauxOptions(), {})) >>> ignore._test_dirs ['/myproject/src/blah/foo/', '/myproject/src/blah/bar/'] We can now ask whether a particular module should be ignored: >>> ignore.names('/myproject/src/blah/foo/baz.py', 'baz') False >>> ignore.names('/myproject/src/blah/bar/mine.py', 'mine') False >>> ignore.names('/myproject/src/blah/foo/__init__.py', 'foo') False >>> ignore.names('/myproject/src/blah/hello.py', 'hello') True When running the test runner, modules are sometimes created from text strings. Those should *always* be ignored: >>> ignore.names('/myproject/src/blah/hello.txt', '') True To make this check fast, the class implements a cache. In an early implementation, the result was cached by the module name, which was a problem, since a lot of modules carry the same name (not the Python dotted name here!). So just because a module has the same name in an ignored and tested directory, does not mean it is always ignored: >>> ignore.names('/myproject/src/blah/module.py', 'module') True >>> ignore.names('/myproject/src/blah/foo/module.py', 'module') False zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-debugging-layer-setup.test0000664000000000000000000000755612527371226030073 0ustar rootrootPost-mortem debugging also works when there is a failure in layer setup. >>> import os, shutil, sys, tempfile >>> tdir = tempfile.mkdtemp() >>> dir = os.path.join(tdir, 'TESTS-DIR') >>> os.mkdir(dir) >>> def write_file(filename, body): ... with open(os.path.join(dir, filename), 'w') as f: ... f.write(body) ... try: ... # Need to do this on Python 3.3 after creating new modules ... import importlib; importlib.invalidate_caches() ... except (ImportError, AttributeError): ... pass >>> write_file('tests.py', ... ''' ... import doctest ... ... class Layer: ... @classmethod ... def setUp(self): ... x = 1 ... raise ValueError ... ... def a_test(): ... """ ... >>> None ... """ ... def test_suite(): ... suite = doctest.DocTestSuite() ... suite.layer = Layer ... return suite ... ... ''') >>> class Input: ... def __init__(self, src): ... self.lines = src.split('\n') ... def readline(self): ... line = self.lines.pop(0) ... print(line) ... return line+'\n' >>> real_stdin = sys.stdin >>> sys.stdin = Input('p x\nc') >>> sys.argv = [testrunner_script] >>> import zope.testrunner >>> try: ... zope.testrunner.run_internal(['--path', dir, '-D']) ... except zope.testrunner.interfaces.EndRun: print("unexpected EndRun!") ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS Running tests.Layer tests: Set up tests.Layer ...ValueError > ...tests.py(8)setUp() -> raise ValueError (Pdb) p x 1 (Pdb) c False Note that post-mortem debugging doesn't work when the layer is run in a subprocess: >>> sys.stdin = Input('p x\nc') >>> write_file('tests2.py', ... ''' ... import doctest, unittest ... ... class Layer1: ... @classmethod ... def setUp(self): ... pass ... ... @classmethod ... def tearDown(self): ... raise NotImplementedError ... ... class Layer2: ... @classmethod ... def setUp(self): ... x = 1 ... raise ValueError ... ... def a_test(): ... """ ... >>> None ... """ ... def test_suite(): ... suite1 = doctest.DocTestSuite() ... suite1.layer = Layer1 ... suite2 = doctest.DocTestSuite() ... suite2.layer = Layer2 ... return unittest.TestSuite((suite1, suite2)) ... ... ''') >>> import sys >>> try: ... zope.testrunner.run_internal( ... ['--path', dir, '-Dvv', '--tests-pattern', 'tests2']) ... except zope.testrunner.interfaces.EndRun: print("unexpected EndRun!") ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS +REPORT_NDIFF Running tests at level 1 Running tests2.Layer1 tests: Set up tests2.Layer1 in 0.000 seconds. Running: a_test (tests2) Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.001 seconds. Running tests2.Layer2 tests: Tear down tests2.Layer1 ... not supported Running in a subprocess. Set up tests2.Layer2 ********************************************************************** Can't post-mortem debug when running a layer as a subprocess! Try running layer 'tests2.Layer2' by itself. ********************************************************************** Traceback (most recent call last): ... raise ValueError ValueError Tests with errors: Layer: tests2.Layer2 Total: 1 tests, 0 failures, 1 errors and 0 skipped in 0.210 seconds. True >>> shutil.rmtree(tdir) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-knit.txt0000664000000000000000000001161312527371226024462 0ustar rootrootKnitting in extra package directories ===================================== Python packages have __path__ variables that can be manipulated to add extra directories cntaining software used in the packages. The testrunner needs to be given extra information about this sort of situation. Let's look at an example. The testrunner-ex-knit-lib directory is a directory that we want to add to the Python path, but that we don't want to search for tests. It has a sample4 package and a products subpackage. The products subpackage adds the testrunner-ex-knit-products to it's __path__. We want to run tests from the testrunner-ex-knit-products directory. When we import these tests, we need to import them from the sample4.products package. We can't use the --path option to name testrunner-ex-knit-products. It isn't enough to add the containing directory to the test path because then we wouldn't be able to determine the package name properly. We might be able to use the --package option to run the tests from the sample4/products package, but we want to run tests in testrunner-ex that aren't in this package. We can use the --package-path option in this case. The --package-path option is like the --test-path option in that it defines a path to be searched for tests without affecting the python path. It differs in that it supplied a package name that is added a profex when importing any modules found. The --package-path option takes *two* arguments, a package name and file path. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> sys.path.append(os.path.join(this_directory, 'testrunner-ex-pp-lib')) >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... '--package-path', ... os.path.join(this_directory, 'testrunner-ex-pp-products'), ... 'sample4.products', ... ] >>> from zope import testrunner >>> sys.argv = 'test --layer Layer111 -vv'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Running: test_x1 (sample1.sampletests.test111.TestA) test_y0 (sample1.sampletests.test111.TestA) ... test_y0 (sampletests.test111) test_z1 (sampletests.test111) testrunner-ex/sampletests/../sampletestsl.txt test_extra_test_in_products (sample4.products.sampletests.Test) test_another_test_in_products (sample4.products.more.sampletests.Test) Ran 28 tests with 0 failures, 0 errors and 0 skipped in 0.008 seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. In the example, the last test, test_extra_test_in_products, came from the products directory. As usual, we can select the knit-in packages or individual packages within knit-in packages: >>> sys.argv = 'test --package sample4.products -vv'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Running: test_extra_test_in_products (sample4.products.sampletests.Test) test_another_test_in_products (sample4.products.more.sampletests.Test) Ran 2 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. >>> sys.argv = 'test --package sample4.products.more -vv'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer111 in 0.000 seconds. Running: test_another_test_in_products (sample4.products.more.sampletests.Test) Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in 0.000 seconds. Tear down samplelayers.Layerx in 0.000 seconds. Tear down samplelayers.Layer11 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-edge-cases.txt0000664000000000000000000004234312527371226025521 0ustar rootroottestrunner Edge Cases ===================== This document has some edge-case examples to test various aspects of the test runner. Separating Python path and test directories ------------------------------------------- The --path option defines a directory to be searched for tests *and* a directory to be added to Python's search path. The --test-path option can be used when you want to set a test search path without also affecting the Python path: >>> import os, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope import testrunner >>> defaults = [ ... '--test-path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) ... # doctest: +ELLIPSIS Test-module import failures: Module: sampletestsf Traceback (most recent call last): ImportError: No module named sampletestsf ... >>> sys.path.append(directory_with_tests) >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer1 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layer1 in 0.000 seconds. Ran 9 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. ... Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 321 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False Bug #251759: The test runner's protection against descending into non-package directories was ineffective, e.g. picking up tests from eggs that were stored close by: >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex-251759') >>> defaults = [ ... '--test-path', directory_with_tests, ... ] >>> testrunner.run_internal(defaults) Total: 0 tests, 0 failures, 0 errors and 0 skipped in 0.000 seconds. False Debugging Edge Cases -------------------- >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--test-path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> class Input: ... def __init__(self, src): ... self.lines = src.split('\n') ... def readline(self): ... line = self.lines.pop(0) ... print(line) ... return line+'\n' >>> real_stdin = sys.stdin Using pdb.set_trace in a function called by an ordinary test: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace2').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS Running zope.testrunner.layer.UnitTests tests:... > testrunner-ex/sample3/sampletests_d.py(47)f() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.001 seconds. ... False Using pdb.set_trace in a function called by a doctest in a doc string: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace4').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > testrunner-ex/sample3/sampletests_d.py(NNN)f() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Using pdb in a docstring-based doctest >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace3').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > (3)?() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Using pdb.set_trace in a doc file: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace5').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > (3)?() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Using pdb.set_trace in a function called by a doctest in a doc file: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace6').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. > testrunner-ex/sample3/sampletests_d.py(NNN)f() -> y = x (Pdb) n --Return-- > ...->None -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging function called from ordinary test: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem2 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Error in test test_post_mortem2 (sample3.sampletests_d.TestSomething) Traceback (most recent call last): File "testrunner-ex/sample3/sampletests_d.py", line 37, in test_post_mortem2 g() File "testrunner-ex/sample3/sampletests_d.py", line 46, in g raise ValueError ValueError ...ValueError > testrunner-ex/sample3/sampletests_d.py(46)g() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging docstring-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem3 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error in test post_mortem3 (sample3.sampletests_d) Traceback (most recent call last): ...UnexpectedException: testrunner-ex/sample3/sampletests_d.py:NNN (2 examples)> ...ValueError > (1)?() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging function called from docstring-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem4 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error in test post_mortem4 (sample3.sampletests_d) Traceback (most recent call last): ...UnexpectedException: testrunner-ex/sample3/sampletests_d.py:NNN (1 example)> ...ValueError > testrunner-ex/sample3/sampletests_d.py(NNN)g() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging file-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem5 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error testrunner-ex/sample3/post_mortem5.txt Traceback (most recent call last): ...UnexpectedException: testrunner-ex/sample3/post_mortem5.txt:0 (2 examples)> ...ValueError > (1)?() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging function called from file-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem6 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Error testrunner-ex/sample3/post_mortem6.txt Traceback (most recent call last): File ".../zope/testing/doctest/__init__.py", Line NNN, in debug runner.run(self._dt_test, clear_globs=False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run r = DocTestRunner.run(self, test, compileflags, out, False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run return self.__run(test, compileflags, out) File ".../zope/testing/doctest/__init__.py", Line NNN, in __run exc_info) File ".../zope/testing/doctest/__init__.py", Line NNN, in report_unexpected_exception raise UnexpectedException(test, example, exc_info) ...UnexpectedException: testrunner-ex/sample3/post_mortem6.txt:0 (2 examples)> ...ValueError > testrunner-ex/sample3/sampletests_d.py(NNN)g() -> raise ValueError (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging of a docstring doctest failure: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure2 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Error in test post_mortem_failure2 (sample3.sampletests_d) File "testrunner-ex/sample3/sampletests_d.py", line 81, in sample3.sampletests_d.post_mortem_failure2 x Want: 2 Got: 1 > testrunner-ex/sample3/sampletests_d.py(81)_() ...ValueError: Expected and actual output are different > (1)...() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging of a docfile doctest failure: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure.txt -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testrunner.layer.UnitTests tests:... Error in test /home/jim/z3/zope.testrunner/src/zope/testing/testrunner-ex/sample3/post_mortem_failure.txt File "testrunner-ex/sample3/post_mortem_failure.txt", line 2, in post_mortem_failure.txt x Want: 2 Got: 1 > testrunner-ex/sample3/post_mortem_failure.txt(2)_() ...ValueError: Expected and actual output are different > (1)...() (Pdb) p x 1 (Pdb) c Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Post-mortem debugging with triple verbosity >>> sys.argv = 'test --layer samplelayers.Layer1$ -vvv -D'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in 0.000 seconds. Running: test_x1 (sampletestsf.TestA1) (0.000 s) test_y0 (sampletestsf.TestA1) (0.000 s) test_z0 (sampletestsf.TestA1) (0.000 s) test_x0 (sampletestsf.TestB1) (0.000 s) test_y1 (sampletestsf.TestB1) (0.000 s) test_z0 (sampletestsf.TestB1) (0.000 s) test_1 (sampletestsf.TestNotMuch1) (0.000 s) test_2 (sampletestsf.TestNotMuch1) (0.000 s) test_3 (sampletestsf.TestNotMuch1) (0.000 s) Ran 9 tests with 0 failures, 0 errors and 0 skipped in 0.001 seconds. Tearing down left over layers: Tear down samplelayers.Layer1 in 0.000 seconds. False Test Suites with None for suites or tests ----------------------------------------- >>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_none_suite$', ... ] >>> testrunner.run_internal(defaults) Test-module import failures: Module: sample1.sampletests_none_suite Traceback (most recent call last): TypeError: Invalid test_suite, None, in sample1.sampletests_none_suite Test-modules with import problems: sample1.sampletests_none_suite Total: 0 tests, 0 failures, 1 errors and 0 skipped in N.NNN seconds. True >>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_none_test$', ... ] >>> testrunner.run_internal(defaults) Test-module import failures: Module: sample1.sampletests_none_test Traceback (most recent call last): TypeError: ... Test-modules with import problems: sample1.sampletests_none_test Total: 0 tests, 0 failures, 1 errors and 0 skipped in N.NNN seconds. True You must use --repeat with --report-refcounts --------------------------------------------- It is an error to specify --report-refcounts (-r) without specifying a repeat count greater than 1 >>> sys.argv = 'test -r'.split() >>> testrunner.run_internal(defaults) You must use the --repeat (-N) option to specify a repeat count greater than 1 when using the --report_refcounts (-r) option. True >>> sys.argv = 'test -r -N1'.split() >>> testrunner.run_internal(defaults) You must use the --repeat (-N) option to specify a repeat count greater than 1 when using the --report_refcounts (-r) option. True Selection --------- Several tests can be excluded using the '!' notation: >>> sys.argv = 'test -u -vv -ssample1.sample13 -t!test_x -t!test_y'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: test_z0 (sample1.sample13.sampletests.TestA) test_z0 (sample1.sample13.sampletests.TestB) test_1 (sample1.sample13.sampletests.TestNotMuch) test_2 (sample1.sample13.sampletests.TestNotMuch) test_3 (sample1.sample13.sampletests.TestNotMuch) test_z1 (sample1.sample13.sampletests) testrunner-ex/sample1/sample13/../../sampletests.txt Ran 7 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-gc.txt0000664000000000000000000001007612527371226024110 0ustar rootrootGarbage Collection Control ========================== When having problems that seem to be caused my memory-management errors, it can be helpful to adjust Python's cyclic garbage collector or to get garbage colection statistics. The --gc option can be used for this purpose. If you think you are getting a test failure due to a garbage collection problem, you can try disabling garbage collection by using the --gc option with a value of zero. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = ['--path', directory_with_tests] >>> from zope import testrunner >>> sys.argv = 'test --tests-pattern ^gc0$ --gc 0 -vv'.split() >>> _ = testrunner.run_internal(defaults) Cyclic garbage collection is disabled. Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: make_sure_gc_is_disabled (gc0) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Alternatively, if you think you are having a garbage collection related problem, you can cause garbage collection to happen more often by providing a low threshold: >>> sys.argv = 'test --tests-pattern ^gc1$ --gc 1 -vv'.split() >>> _ = testrunner.run_internal(defaults) Cyclic garbage collection threshold set to: (1,) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: make_sure_gc_threshold_is_one (gc1) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. You can specify up to 3 --gc options to set each of the 3 gc threshold values: >>> sys.argv = ('test --tests-pattern ^gcset$ --gc 701 --gc 11 --gc 9 -vv' ... .split()) >>> _ = testrunner.run_internal(defaults) Cyclic garbage collection threshold set to: (701, 11, 9) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: make_sure_gc_threshold_is_701_11_9 (gcset) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Specifying more than 3 --gc options is not allowed: >>> from six import StringIO >>> out = StringIO() >>> stdout = sys.stdout >>> sys.stdout = out >>> sys.argv = ('test --tests-pattern ^gcset$ --gc 701 --gc 42 --gc 11 --gc 9 -vv' ... .split()) >>> _ = testrunner.run_internal(defaults) Traceback (most recent call last): ... SystemExit: 1 >>> sys.stdout = stdout >>> print(out.getvalue()) Too many --gc options Garbage Collection Statistics ----------------------------- You can enable gc debugging statistics using the --gc-options (-G) option. You should provide names of one or more of the flags described in the library documentation for the gc module. The output statistics are written to standard error. >>> from six import StringIO >>> err = StringIO() >>> stderr = sys.stderr >>> sys.stderr = err >>> sys.argv = ('test --tests-pattern ^gcstats$ -G DEBUG_STATS' ... ' -G DEBUG_COLLECTABLE -vv' ... .split()) >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: generate_some_gc_statistics (gcstats) Ran 1 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. >>> sys.stderr = stderr >>> print(err.getvalue()) # doctest: +ELLIPSIS gc: collecting generation ... zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-leaks.txt0000664000000000000000000002511312527371226024614 0ustar rootrootDebugging Memory Leaks ====================== The --report-refcounts (-r) option can be used with the --repeat (-N) option to detect and diagnose memory leaks. To use this option, you must configure Python with the --with-pydebug option. (On Unix, pass this option to configure and then build Python.) >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> sys.argv = 'test --layer Layer11$ --layer Layer12$ -N4 -r'.split() >>> _ = testrunner.run_internal(defaults) Running samplelayers.Layer11 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Iteration 1 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.013 seconds. Iteration 2 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100401 change=0 Iteration 3 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100401 change=0 Iteration 4 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.013 seconds. sys refcount=100401 change=0 Running samplelayers.Layer12 tests: Tear down samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Iteration 1 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.013 seconds. Iteration 2 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100411 change=0 Iteration 3 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100411 change=0 Iteration 4 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.012 seconds. sys refcount=100411 change=0 Tearing down left over layers: Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. Total: 68 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. Each layer is repeated the requested number of times. For each iteration after the first, the system refcount and change in system refcount is shown. The system refcount is the total of all refcount in the system. When a refcount on any object is changed, the system refcount is changed by the same amount. Tests that don't leak show zero changes in systen refcount. Let's look at an example test that leaks: >>> sys.argv = 'test --tests-pattern leak -N4 -r'.split() >>> _ = testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests:... Iteration 1 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Iteration 2 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sys refcount=92506 change=12 Iteration 3 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sys refcount=92513 change=12 Iteration 4 Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sys refcount=92520 change=12 Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Here we see that the system refcount is increating. If we specify a verbosity greater than one, we can get details broken out by object type (or class): >>> sys.argv = 'test --tests-pattern leak -N5 -r -v'.split() >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running zope.testrunner.layer.UnitTests tests:... Iteration 1 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Iteration 2 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95832 sys refcount=105668 change=16 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 int 2 2 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 3 ------------------------------------------------------- ----- ---- total 8 16 Iteration 3 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95844 sys refcount=105680 change=12 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 int -1 0 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 1 ------------------------------------------------------- ----- ---- total 5 12 Iteration 4 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95856 sys refcount=105692 change=12 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 1 ------------------------------------------------------- ----- ---- total 6 12 Iteration 5 Running: . Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. sum detail refcount=95868 sys refcount=105704 change=12 Leak details, changes in instances and refcounts by type/class: type/class insts refs ------------------------------------------------------- ----- ---- classobj 0 1 dict 2 2 float 1 1 leak.ClassicLeakable 1 1 leak.Leakable 1 1 str 0 4 tuple 1 1 type 0 1 ------------------------------------------------------- ----- ---- total 6 12 Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. It is instructive to analyze the results in some detail. The test being run was designed to intentionally leak: class ClassicLeakable: def __init__(self): self.x = 'x' class Leakable(object): def __init__(self): self.x = 'x' leaked = [] class TestSomething(unittest.TestCase): def testleak(self): leaked.append((ClassicLeakable(), Leakable(), time.time())) Let's go through this by type. float, leak.ClassicLeakable, leak.Leakable, and tuple We leak one of these every time. This is to be expected because we are adding one of these to the list every time. str We don't leak any instances, but we leak 4 references. These are due to the instance attributes avd values. dict We leak 2 of these, one for each ClassicLeakable and Leakable instance. classobj We increase the number of classobj instance references by one each time because each ClassicLeakable instance has a reference to its class. This instances increases the references in it's class, which increases the total number of references to classic classes (clasobj instances). type For most interations, we increase the number of type references by one for the same reason we increase the number of clasobj references by one. The increase of the number of type references by 3 in the second iteration is puzzling, but illustrates that this sort of data is often puzzling. int The change in the number of int instances and references in this example is a side effect of the statistics being gathered. Lots of integers are created to keep the memory statistics used here. The summary statistics include the sum of the detail refcounts. (Note that this sum is less than the system refcount. This is because the detailed analysis doesn't inspect every object. Not all objects in the system are returned by sys.getobjects.) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-repeat.txt0000664000000000000000000000367412527371226025005 0ustar rootrootRepeating Tests =============== The --repeat option can be used to repeat tests some number of times. Repeating tests is useful to help make sure that tests clean up after themselves. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 112 --layer UnitTests --repeat 3'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Iteration 1 Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Iteration 2 Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Iteration 3 Ran 156 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running samplelayers.Layer112 tests: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. Set up samplelayers.Layerx in 0.000 seconds. Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. Set up samplelayers.Layer112 in 0.000 seconds. Iteration 1 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.010 seconds. Iteration 2 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.010 seconds. Iteration 3 Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.010 seconds. Tearing down left over layers: Tear down samplelayers.Layer112 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Total: 182 tests, 0 failures, 0 errors and 0 skipped in N.NNN seconds. False The tests are repeated by layer. Layers are set up and torn down only once. zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-progress.txt0000664000000000000000000005153112527371226025364 0ustar rootrootTest Progress ============= If the --progress (-p) option is used, progress information is printed and a carriage return (rather than a new-line) is printed between detail lines. Let's look at the effect of --progress (-p) at different levels of verbosity. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = 'test --layer 122 -p'.split() >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Running: 1/26 (3.8%)##r## ##r## 2/26 (7.7%)##r## ##r## 3/26 (11.5%)##r## ##r## 4/26 (15.4%)##r## ##r## 5/26 (19.2%)##r## ##r## 6/26 (23.1%)##r## ##r## 7/26 (26.9%)##r## ##r## 8/26 (30.8%)##r## ##r## 9/26 (34.6%)##r## ##r## 10/26 (38.5%)##r## ##r## 11/26 (42.3%)##r## ##r## 12/26 (46.2%)##r## ##r## 13/26 (50.0%)##r## ##r## 14/26 (53.8%)##r## ##r## 15/26 (57.7%)##r## ##r## 16/26 (61.5%)##r## ##r## 17/26 (65.4%)##r## ##r## 18/26 (69.2%)##r## ##r## 19/26 (73.1%)##r## ##r## 20/26 (76.9%)##r## ##r## 21/26 (80.8%)##r## ##r## 22/26 (84.6%)##r## ##r## 23/26 (88.5%)##r## ##r## 24/26 (92.3%)##r## ##r## 25/26 (96.2%)##r## ##r## 26/26 (100.0%)##r## ##r## Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False (Note that, in the examples above and below, we show "##r##" followed by new lines where carriage returns would appear in actual output.) Using a single level of verbosity causes test descriptions to be output, but only if they fit in the terminal width. The default width, when the terminal width can't be determined, is 80: >>> sys.argv = 'test --layer 122 -pv'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Running: 1/26 (3.8%) test_x1 (sample1.sampletests.test122.TestA)##r## ##r## 2/26 (7.7%) test_y0 (sample1.sampletests.test122.TestA)##r## ##r## 3/26 (11.5%) test_z0 (sample1.sampletests.test122.TestA)##r## ##r## 4/26 (15.4%) test_x0 (sample1.sampletests.test122.TestB)##r## ##r## 5/26 (19.2%) test_y1 (sample1.sampletests.test122.TestB)##r## ##r## 6/26 (23.1%) test_z0 (sample1.sampletests.test122.TestB)##r## ##r## 7/26 (26.9%) test_1 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 8/26 (30.8%) test_2 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 9/26 (34.6%) test_3 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 10/26 (38.5%) test_x0 (sample1.sampletests.test122)##r## ##r## 11/26 (42.3%) test_y0 (sample1.sampletests.test122)##r## ##r## 12/26 (46.2%) test_z1 (sample1.sampletests.test122)##r## ##r## testrunner-ex/sample1/sampletests/../../sampletestsl.txt##r## ##r## 14/26 (53.8%) test_x1 (sampletests.test122.TestA)##r## ##r## 15/26 (57.7%) test_y0 (sampletests.test122.TestA)##r## ##r## 16/26 (61.5%) test_z0 (sampletests.test122.TestA)##r## ##r## 17/26 (65.4%) test_x0 (sampletests.test122.TestB)##r## ##r## 18/26 (69.2%) test_y1 (sampletests.test122.TestB)##r## ##r## 19/26 (73.1%) test_z0 (sampletests.test122.TestB)##r## ##r## 20/26 (76.9%) test_1 (sampletests.test122.TestNotMuch)##r## ##r## 21/26 (80.8%) test_2 (sampletests.test122.TestNotMuch)##r## ##r## 22/26 (84.6%) test_3 (sampletests.test122.TestNotMuch)##r## ##r## 23/26 (88.5%) test_x0 (sampletests.test122)##r## ##r## 24/26 (92.3%) test_y0 (sampletests.test122)##r## ##r## 25/26 (96.2%) test_z1 (sampletests.test122)##r## ##r## testrunner-ex/sampletests/../sampletestsl.txt##r## ##r## Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False The terminal width is determined using the curses module. To see that, we'll provide a fake curses module: >>> class FakeCurses: ... class error(Exception): ... pass ... def setupterm(self): ... pass ... def tigetnum(self, ignored): ... return 60 >>> old_curses = sys.modules.get('curses') >>> sys.modules['curses'] = FakeCurses() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer12 in N.NNN seconds. Set up samplelayers.Layer122 in N.NNN seconds. Running: 1/26 (3.8%) test_x1 (sample1.sampletests.test122.TestA)##r## ##r## 2/26 (7.7%) test_y0 (sample1.sampletests.test122.TestA)##r## ##r## 3/26 (11.5%) test_z0 (...le1.sampletests.test122.TestA)##r## ##r## 4/26 (15.4%) test_x0 (...le1.sampletests.test122.TestB)##r## ##r## 5/26 (19.2%) test_y1 (...le1.sampletests.test122.TestB)##r## ##r## 6/26 (23.1%) test_z0 (...le1.sampletests.test122.TestB)##r## ##r## 7/26 (26.9%) test_1 (...ampletests.test122.TestNotMuch)##r## ##r## 8/26 (30.8%) test_2 (...ampletests.test122.TestNotMuch)##r## ##r## 9/26 (34.6%) test_3 (...ampletests.test122.TestNotMuch)##r## ##r## 10/26 (38.5%) test_x0 (sample1.sampletests.test122)##r## ##r## 11/26 (42.3%) test_y0 (sample1.sampletests.test122)##r## ##r## 12/26 (46.2%) test_z1 (sample1.sampletests.test122)##r## ##r## 13/26 (50.0%) ... e1/sampletests/../../sampletestsl.txt##r## ##r## 14/26 (53.8%) test_x1 (sampletests.test122.TestA)##r## ##r## 15/26 (57.7%) test_y0 (sampletests.test122.TestA)##r## ##r## 16/26 (61.5%) test_z0 (sampletests.test122.TestA)##r## ##r## 17/26 (65.4%) test_x0 (sampletests.test122.TestB)##r## ##r## 18/26 (69.2%) test_y1 (sampletests.test122.TestB)##r## ##r## 19/26 (73.1%) test_z0 (sampletests.test122.TestB)##r## ##r## 20/26 (76.9%) test_1 (sampletests.test122.TestNotMuch)##r## ##r## 21/26 (80.8%) test_2 (sampletests.test122.TestNotMuch)##r## ##r## 22/26 (84.6%) test_3 (sampletests.test122.TestNotMuch)##r## ##r## 23/26 (88.5%) test_x0 (sampletests.test122)##r## ##r## 24/26 (92.3%) test_y0 (sampletests.test122)##r## ##r## 25/26 (96.2%) test_z1 (sampletests.test122)##r## ##r## 26/26 (100.0%) ... r-ex/sampletests/../sampletestsl.txt##r## ##r## Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False >>> sys.modules['curses'] = old_curses If a second or third level of verbosity are added, we get additional information. >>> sys.argv = 'test --layer 122 -pvv -t !txt'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: 1/24 (4.2%) test_x1 (sample1.sampletests.test122.TestA)##r## ##r## 2/24 (8.3%) test_y0 (sample1.sampletests.test122.TestA)##r## ##r## 3/24 (12.5%) test_z0 (sample1.sampletests.test122.TestA)##r## ##r## 4/24 (16.7%) test_x0 (sample1.sampletests.test122.TestB)##r## ##r## 5/24 (20.8%) test_y1 (sample1.sampletests.test122.TestB)##r## ##r## 6/24 (25.0%) test_z0 (sample1.sampletests.test122.TestB)##r## ##r## 7/24 (29.2%) test_1 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 8/24 (33.3%) test_2 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 9/24 (37.5%) test_3 (sample1.sampletests.test122.TestNotMuch)##r## ##r## 10/24 (41.7%) test_x0 (sample1.sampletests.test122)##r## ##r## 11/24 (45.8%) test_y0 (sample1.sampletests.test122)##r## ##r## 12/24 (50.0%) test_z1 (sample1.sampletests.test122)##r## ##r## 13/24 (54.2%) test_x1 (sampletests.test122.TestA)##r## ##r## 14/24 (58.3%) test_y0 (sampletests.test122.TestA)##r## ##r## 15/24 (62.5%) test_z0 (sampletests.test122.TestA)##r## ##r## 16/24 (66.7%) test_x0 (sampletests.test122.TestB)##r## ##r## 17/24 (70.8%) test_y1 (sampletests.test122.TestB)##r## ##r## 18/24 (75.0%) test_z0 (sampletests.test122.TestB)##r## ##r## 19/24 (79.2%) test_1 (sampletests.test122.TestNotMuch)##r## ##r## 20/24 (83.3%) test_2 (sampletests.test122.TestNotMuch)##r## ##r## 21/24 (87.5%) test_3 (sampletests.test122.TestNotMuch)##r## ##r## 22/24 (91.7%) test_x0 (sampletests.test122)##r## ##r## 23/24 (95.8%) test_y0 (sampletests.test122)##r## ##r## 24/24 (100.0%) test_z1 (sampletests.test122)##r## ##r## Ran 24 tests with 0 failures, 0 errors and 0 skipped in 0.006 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False Note that, in this example, we used a test-selection pattern starting with '!' to exclude tests containing the string "txt". >>> sys.argv = 'test --layer 122 -pvvv -t!(txt|NotMuch)'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer122 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer12 in 0.000 seconds. Set up samplelayers.Layer122 in 0.000 seconds. Running: 1/18 (5.6%) test_x1 (sample1.sampletests.test122.TestA) (0.000 s)##r## ##r## 2/18 (11.1%) test_y0 (sample1.sampletests.test122.TestA) (0.000 s)##r## ##r## 3/18 (16.7%) test_z0 (sample1.sampletests.test122.TestA) (0.000 s)##r## ##r## 4/18 (22.2%) test_x0 (sample1.sampletests.test122.TestB) (0.000 s)##r## ##r## 5/18 (27.8%) test_y1 (sample1.sampletests.test122.TestB) (0.000 s)##r## ##r## 6/18 (33.3%) test_z0 (sample1.sampletests.test122.TestB) (0.000 s)##r## ##r## 7/18 (38.9%) test_x0 (sample1.sampletests.test122) (0.001 s)##r## ##r## 8/18 (44.4%) test_y0 (sample1.sampletests.test122) (0.001 s)##r## ##r## 9/18 (50.0%) test_z1 (sample1.sampletests.test122) (0.001 s)##r## ##r## 10/18 (55.6%) test_x1 (sampletests.test122.TestA) (0.000 s)##r## ##r## 11/18 (61.1%) test_y0 (sampletests.test122.TestA) (0.000 s)##r## ##r## 12/18 (66.7%) test_z0 (sampletests.test122.TestA) (0.000 s)##r## ##r## 13/18 (72.2%) test_x0 (sampletests.test122.TestB) (0.000 s)##r## ##r## 14/18 (77.8%) test_y1 (sampletests.test122.TestB) (0.000 s)##r## ##r## 15/18 (83.3%) test_z0 (sampletests.test122.TestB) (0.000 s)##r## ##r## 16/18 (88.9%) test_x0 (sampletests.test122) (0.001 s)##r## ##r## 17/18 (94.4%) test_y0 (sampletests.test122) (0.001 s)##r## ##r## 18/18 (100.0%) test_z1 (sampletests.test122) (0.001 s)##r## ##r## Ran 18 tests with 0 failures, 0 errors and 0 skipped in 0.006 seconds. Tearing down left over layers: Tear down samplelayers.Layer122 in 0.000 seconds. Tear down samplelayers.Layer12 in 0.000 seconds. Tear down samplelayers.Layer1 in 0.000 seconds. False In this example, we also excluded tests with "NotMuch" in their names. Unfortunately, the time data above doesn't buy us much because, in practice, the line is cleared before there is time to see the times. :/ Autodetecting progress ---------------------- The --auto-progress option will determine if stdout is a terminal, and only enable progress output if so. Let's pretend we have a terminal >>> class Terminal(object): ... def __init__(self, stream): ... self._stream = stream ... def __getattr__(self, attr): ... return getattr(self._stream, attr) ... def isatty(self): ... return True >>> real_stdout = sys.stdout >>> sys.stdout = Terminal(sys.stdout) >>> sys.argv = 'test -u -t test_one.TestNotMuch --auto-progress'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Running: 1/6 (16.7%)##r## ##r## 2/6 (33.3%)##r## ##r## 3/6 (50.0%)##r## ##r## 4/6 (66.7%)##r## ##r## 5/6 (83.3%)##r## ##r## 6/6 (100.0%)##r## ##r## Ran 6 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Let's stop pretending >>> sys.stdout = real_stdout >>> sys.argv = 'test -u -t test_one.TestNotMuch --auto-progress'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 6 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False Disabling progress indication ----------------------------- If -p or --progress have been previously provided on the command line (perhaps by a wrapper script) but you do not desire progress indication, you can switch it off with --no-progress: >>> sys.argv = 'test -u -t test_one.TestNotMuch -p --no-progress'.split() >>> testrunner.run_internal(defaults) Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in N.NNN seconds. Ran 6 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in N.NNN seconds. False zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-shuffle.txt0000664000000000000000000001337612527371226025161 0ustar rootrootShuffling tests =============== By default, every time you launch the testrunner it will run the tests in a specific order. However, if you want to ensure that your tests are well isolated then running them in a varying order can be helpful. (Proper isolation meaning your tests don't depend on each others outcomes or side effects and thus the setup and tear down methods are written properly.) The ``--shuffle`` option tells the test runner to shuffle the list of tests before running them: >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> default_argv = 'test -u -m sample1 -t test_y0 --list-tests ' Running shuffled tests ---------------------- When specifying the ``--shuffle`` option tests are ordered differently each time you run the tests: >>> argv = (default_argv + '--shuffle').split() >>> testrunner.run_internal(defaults, argv) Tests were shuffled using seed number ... Listing zope.testrunner.layer.UnitTests tests: ... False Note: The runner prints out a new piece of information which is the seed number used to generate the shuffled list of tests. This seed number can later be used to re-run the tests in exactly the same order to support debugging. Specifying a seed number to control tests shuffling --------------------------------------------------- Along with the ``--shuffle`` option comes the ``--shuffle-seed`` option which takes a seed number as an argument. If you want to reproduce test failures that happened during a randomly shuffled test run then simply write down the seed number that was printed out and run it again using the ``--shuffle-seed`` option. The order is guaranteed to be the same. There was an issue with the guaranteed order in zope.testrunner before 4.0.2: the order could change depending on dictionary iteration order, if you had multiple test layers. This bug was fixed in 4.0.2, but a side effect of the fix means that the new guaranteed order is likely to be different from what you used to get with an older zope.testrunner. For example, using the seed number 0 will give us the following, stable, list of tests: >>> argv = (default_argv + '--shuffle --shuffle-seed 0').split() >>> testrunner.run_internal(defaults, argv) Tests were shuffled using seed number 0. Listing zope.testrunner.layer.UnitTests tests: test_y0 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletestsf) test_y0 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sampletests.test1) False >>> testrunner.run_internal(defaults, argv) Tests were shuffled using seed number 0. Listing zope.testrunner.layer.UnitTests tests: test_y0 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletests.test_one) test_y0 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletestsf) test_y0 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sampletests.test1) False Whereas using the seed number 42 will give us the following, different but stable, list of tests: >>> argv = (default_argv + '--shuffle --shuffle-seed 42').split() >>> testrunner.run_internal(defaults, argv) Tests were shuffled using seed number 42. Listing zope.testrunner.layer.UnitTests tests: test_y0 (sample1.sample13.sampletests.TestA) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1.TestA) test_y0 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sampletestsf) test_y0 (sample1.sampletests.test_one) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletests.test_one.TestA) False >>> testrunner.run_internal(defaults, argv) Tests were shuffled using seed number 42. Listing zope.testrunner.layer.UnitTests tests: test_y0 (sample1.sample13.sampletests.TestA) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test1.TestA) test_y0 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sampletestsf) test_y0 (sample1.sampletests.test_one) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletests.test_one.TestA) False Selecting a seed number without ``--shuffle`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Note that the ``--shuffle-seed`` option must be used along with ``--shuffle`` option or tests will not be re-ordered: >>> argv = (default_argv + '--shuffle-seed 42').split() >>> testrunner.run_internal(defaults, argv) Listing zope.testrunner.layer.UnitTests tests: test_y0 (sample1.sampletestsf.TestA) test_y0 (sample1.sampletestsf) test_y0 (sample1.sample11.sampletests.TestA) test_y0 (sample1.sample11.sampletests) test_y0 (sample1.sample13.sampletests.TestA) test_y0 (sample1.sample13.sampletests) test_y0 (sample1.sampletests.test1.TestA) test_y0 (sample1.sampletests.test1) test_y0 (sample1.sampletests.test_one.TestA) test_y0 (sample1.sampletests.test_one) False zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-layers-cwd.txt0000664000000000000000000000261112527371226025565 0ustar rootrootRegression test for https://github.com/zopefoundation/zope.testrunner/issues/6: when the test suite changes the current working directory, subprocess invocation might fail. >>> import os.path, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex-6') >>> defaults = [ ... '--path', os.path.relpath(directory_with_tests), ... '--tests-pattern', '^cwdtests?$', ... ] >>> orig_cwd = os.getcwd() >>> sys.argv = [os.path.relpath(testrunner_script), '-j2'] >>> from zope import testrunner >>> testrunner.run_internal(defaults) Running .EmptyLayer tests: Set up .EmptyLayer in N.NNN seconds. Ran 0 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Running cwdtests.Layer1 tests: Running in a subprocess. Set up cwdtests.Layer1 in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Tear down cwdtests.Layer1 in 0.000 seconds. Running cwdtests.Layer2 tests: Running in a subprocess. Set up cwdtests.Layer2 in 0.000 seconds. Ran 1 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds. Tear down cwdtests.Layer2 in 0.000 seconds. Tearing down left over layers: Tear down .EmptyLayer in N.NNN seconds. Total: 2 tests, 0 failures, 0 errors and 0 skipped in 0.162 seconds. False >>> os.chdir(orig_cwd) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-pp-products/0000775000000000000000000000000012552256734025650 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-pp-products/__init__.py0000664000000000000000000000000212527371226027745 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-pp-products/sampletests.py0000664000000000000000000000151212527371226030561 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest class Test(unittest.TestCase): layer = 'samplelayers.Layer111' def test_extra_test_in_products(self): pass def test_suite(): return unittest.makeSuite(Test) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-pp-products/more/0000775000000000000000000000000012552256734026612 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-pp-products/more/__init__.py0000664000000000000000000000000212527371226030707 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-pp-products/more/sampletests.py0000664000000000000000000000151412527371226031525 0ustar rootroot############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest class Test(unittest.TestCase): layer = 'samplelayers.Layer111' def test_another_test_in_products(self): pass def test_suite(): return unittest.makeSuite(Test) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-pp-lib/0000775000000000000000000000000012552256734024553 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-pp-lib/sample4/0000775000000000000000000000000012552256734026120 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-pp-lib/sample4/__init__.py0000664000000000000000000000000212527371226030215 0ustar rootroot# zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-pp-lib/sample4/products/0000775000000000000000000000000012552256734027763 5ustar rootrootzope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-ex-pp-lib/sample4/products/__init__.py0000664000000000000000000000201412527371226032065 0ustar rootroot############################################################################## # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample package that knits in extra directories. """ import os __path__.append( os.path.join( os.path.dirname( # testing os.path.dirname( # testrunner-ex-knit-lib os.path.dirname( # sample4 os.path.dirname(__file__) # products ) ) ) , "testrunner-ex-pp-products" ) ) zope.testrunner-4.4.9/src/zope/testrunner/tests/testrunner-arguments.txt0000664000000000000000000000570112527371226025523 0ustar rootrootPassing arguments explicitly ============================ In most of the examples here, we set up `sys.argv`. In normal usage, the testrunner just uses `sys.argv`. It is possible to pass arguments explicitly. >>> import os.path >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> testrunner.run_internal(defaults, 'test --layer 111'.split()) Running samplelayers.Layer111 tests: Set up samplelayers.Layerx in N.NNN seconds. Set up samplelayers.Layer1 in N.NNN seconds. Set up samplelayers.Layer11 in N.NNN seconds. Set up samplelayers.Layer111 in N.NNN seconds. Ran 26 tests with 0 failures, 0 errors and 0 skipped in N.NNN seconds. Tearing down left over layers: Tear down samplelayers.Layer111 in N.NNN seconds. Tear down samplelayers.Layerx in N.NNN seconds. Tear down samplelayers.Layer11 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. False If options already have default values, then passing a different default will override. For example, --list-tests defaults to being turned off, but if we pass in a different default, that one takes effect. >>> defaults = [ ... '--list-tests', ... '--path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> from zope import testrunner >>> testrunner.run_internal(defaults, 'test --layer 111'.split()) Listing samplelayers.Layer111 tests: test_x1 (sample1.sampletests.test111.TestA) test_y0 (sample1.sampletests.test111.TestA) test_z0 (sample1.sampletests.test111.TestA) test_x0 (sample1.sampletests.test111.TestB) test_y1 (sample1.sampletests.test111.TestB) test_z0 (sample1.sampletests.test111.TestB) test_1 (sample1.sampletests.test111.TestNotMuch) test_2 (sample1.sampletests.test111.TestNotMuch) test_3 (sample1.sampletests.test111.TestNotMuch) test_x0 (sample1.sampletests.test111) test_y0 (sample1.sampletests.test111) test_z1 (sample1.sampletests.test111) /home/benji/workspace/zope.testrunner/1/src/zope/testing/testrunner/testrunner-ex/sample1/sampletests/../../sampletestsl.txt test_x1 (sampletests.test111.TestA) test_y0 (sampletests.test111.TestA) test_z0 (sampletests.test111.TestA) test_x0 (sampletests.test111.TestB) test_y1 (sampletests.test111.TestB) test_z0 (sampletests.test111.TestB) test_1 (sampletests.test111.TestNotMuch) test_2 (sampletests.test111.TestNotMuch) test_3 (sampletests.test111.TestNotMuch) test_x0 (sampletests.test111) test_y0 (sampletests.test111) test_z1 (sampletests.test111) /home/benji/workspace/zope.testrunner/1/src/zope/testing/testrunner/testrunner-ex/sampletests/../sampletestsl.txt False zope.testrunner-4.4.9/src/zope/testrunner/__main__.py0000664000000000000000000000144712527371226021501 0ustar rootroot############################################################################## # # Copyright (c) 2013 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## if __name__ == '__main__': # allow people to try out the test runner with # python -m zope.testrunner --test-path . from zope.testrunner import run run() zope.testrunner-4.4.9/src/zope/testrunner/options.py0000664000000000000000000005736512527371226021466 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Command-line option parsing """ from __future__ import print_function import optparse import re import os import sys import pkg_resources from zope.testrunner.profiling import available_profilers from zope.testrunner.formatter import ( OutputFormatter, ColorfulOutputFormatter, SubunitOutputFormatter, ) from zope.testrunner.formatter import terminal_has_colors parser = optparse.OptionParser("Usage: %prog [options] [MODULE] [TEST]") ###################################################################### # Searching and filtering searching = optparse.OptionGroup(parser, "Searching and filtering", """\ Options in this group are used to define which tests to run. """) searching.add_option( '--package', '--dir', '-s', action="append", dest='package', help="""\ Search the given package's directories for tests. This can be specified more than once to run tests in multiple parts of the source tree. For example, if refactoring interfaces, you don't want to see the way you have broken setups for tests in other packages. You *just* want to run the interface tests. Packages are supplied as dotted names. For compatibility with the old test runner, forward and backward slashed in package names are converted to dots. (In the special case of packages spread over multiple directories, only directories within the test search path are searched. See the --path option.) """) searching.add_option( '--module', '-m', action="append", dest='module', help="""\ Specify a test-module filter as a regular expression. This is a case-sensitive regular expression, used in search (not match) mode, to limit which test modules are searched for tests. The regular expressions are checked against dotted module names. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The option can be specified multiple test-module filters. Test modules matching any of the test filters are searched. If no test-module filter is specified, then all test modules are used. """) searching.add_option( '--test', '-t', action="append", dest='test', help="""\ Specify a test filter as a regular expression. This is a case-sensitive regular expression, used in search (not match) mode, to limit which tests are run. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The option can be specified multiple test filters. Tests matching any of the test filters are included. If no test filter is specified, then all tests are run. """) searching.add_option( '--unit', '-u', action="store_true", dest='unit', help="""\ Run only unit tests, ignoring any layer options. """) searching.add_option( '--non-unit', '-f', action="store_true", dest='non_unit', help="""\ Run tests other than unit tests. """) searching.add_option( '--layer', action="append", dest='layer', help="""\ Specify a test layer to run. The option can be given multiple times to specify more than one layer. If not specified, all layers are run. It is common for the running script to provide default values for this option. Layers are specified regular expressions, used in search mode, for dotted names of objects that define a layer. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The layer named 'zope.testrunner.layer.UnitTests' is reserved for unit tests, however, take note of the --unit and non-unit options. """) searching.add_option( '-a', '--at-level', type='int', dest='at_level', help="""\ Run the tests at the given level. Any test at a level at or below this is run, any test at a level above this is not run. Level 0 runs all tests. """) searching.add_option( '--all', action="store_true", dest='all', help="Run tests at all levels.") searching.add_option( '--list-tests', action="store_true", dest='list_tests', help="List all tests that matched your filters. Do not run any tests.") parser.add_option_group(searching) ###################################################################### # Reporting reporting = optparse.OptionGroup(parser, "Reporting", """\ Reporting options control basic aspects of test-runner output """) reporting.add_option( '--verbose', '-v', action="count", dest='verbose', help="""\ Make output more verbose. Increment the verbosity level. """) reporting.add_option( '--quiet', '-q', action="store_true", dest='quiet', help="""\ Make the output minimal, overriding any verbosity options. """) reporting.add_option( '--progress', '-p', action="store_true", dest='progress', help="""\ Output progress status """) reporting.add_option( '--no-progress',action="store_false", dest='progress', help="""\ Do not output progress status. This is the default, but can be used to counter a previous use of --progress or -p. """) # We use a noop callback because the actual processing will be done in the # get_options function, but we want optparse to generate appropriate help info # for us, so we add an option anyway. reporting.add_option( '--auto-progress', action="callback", callback=lambda *args: None, help="""\ Output progress status, but only when stdout is a terminal. """) reporting.add_option( '--color', '-c', action="store_true", dest='color', help="""\ Colorize the output. """) reporting.add_option( '--no-color', '-C', action="store_false", dest='color', help="""\ Do not colorize the output. This is the default, but can be used to counter a previous use of --color or -c. """) # We use a noop callback because the actual processing will be done in the # get_options function, but we want optparse to generate appropriate help info # for us, so we add an option anyway. reporting.add_option( '--auto-color', action="callback", callback=lambda *args: None, help="""\ Colorize the output, but only when stdout is a terminal. """) reporting.add_option( '--subunit', action="store_true", dest='subunit', help="""\ Use subunit output. Will not be colorized. """) reporting.add_option( '--slow-test', type='float', dest='slow_test_threshold', metavar='N', help="""\ With -c and -vvv, highlight tests that take longer than N seconds (default: %default). """) reporting.add_option( '-1', '--hide-secondary-failures', action="store_true", dest='report_only_first_failure', help="""\ Report only the first failure in a doctest. (Examples after the failure are still executed, in case they do any cleanup.) """) reporting.add_option( '--show-secondary-failures', action="store_false", dest='report_only_first_failure', help="""\ Report all failures in a doctest. This is the default, but can be used to counter a default use of -1 or --hide-secondary-failures. """) reporting.add_option( '--ndiff', action="store_true", dest="ndiff", help="""\ When there is a doctest failure, show it as a diff using the ndiff.py utility. """) reporting.add_option( '--udiff', action="store_true", dest="udiff", help="""\ When there is a doctest failure, show it as a unified diff. """) reporting.add_option( '--cdiff', action="store_true", dest="cdiff", help="""\ When there is a doctest failure, show it as a context diff. """) parser.add_option_group(reporting) ###################################################################### # Analysis analysis = optparse.OptionGroup(parser, "Analysis", """\ Analysis options provide tools for analysing test output. """) analysis.add_option( '--stop-on-error', '--stop', '-x', action="store_true", dest='stop_on_error', help="Stop running tests after first test failure or error." ) analysis.add_option( '--post-mortem', '--pdb', '-D', action="store_true", dest='post_mortem', help="Enable post-mortem debugging of test failures" ) analysis.add_option( '--gc', '-g', action="append", dest='gc', type="int", help="""\ Set the garbage collector generation threshold. This can be used to stress memory and gc correctness. Some crashes are only reproducible when the threshold is set to 1 (aggressive garbage collection). Do "--gc 0" to disable garbage collection altogether. The --gc option can be used up to 3 times to specify up to 3 of the 3 Python gc_threshold settings. """) analysis.add_option( '--gc-option', '-G', action="append", dest='gc_option', type="choice", choices=['DEBUG_STATS', 'DEBUG_COLLECTABLE', 'DEBUG_UNCOLLECTABLE', 'DEBUG_INSTANCES', 'DEBUG_OBJECTS', 'DEBUG_SAVEALL', 'DEBUG_LEAK'], help="""\ Set a Python gc-module debug flag. This option can be used more than once to set multiple flags. """) analysis.add_option( '--repeat', '-N', action="store", type="int", dest='repeat', help="""\ Repeat the tests the given number of times. This option is used to make sure that tests leave their environment in the state they found it and, with the --report-refcounts option to look for memory leaks. """) analysis.add_option( '--report-refcounts', '-r', action="store_true", dest='report_refcounts', help="""\ After each run of the tests, output a report summarizing changes in refcounts by object type. This option that requires that Python was built with the --with-pydebug option to configure. """) analysis.add_option( '--coverage', action="store", type='string', dest='coverage', help="""\ Perform code-coverage analysis, saving trace data to the directory with the given name. A code coverage summary is printed to standard out. """) analysis.add_option( '--profile', action="store", dest='profile', type="choice", choices=list(available_profilers), help="""\ Run the tests under cProfiler or hotshot and display the top 50 stats, sorted by cumulative time and number of calls. """) analysis.add_option( '--profile-directory', action="store", dest='prof_dir', default='.', help="""\ Directory for temporary profiler files. All files named tests_profile.*.prof in this directory will be removed. If you intend to run multiple instances of the test runner in parallel, be sure to tell them to use different directories, so they won't step on each other's toes. """) def do_pychecker(*args): if not os.environ.get("PYCHECKER"): os.environ["PYCHECKER"] = "-q" import pychecker.checker analysis.add_option( '--pychecker', action="callback", callback=do_pychecker, help="""\ Run the tests under pychecker """) parser.add_option_group(analysis) ###################################################################### # Setup setup = optparse.OptionGroup(parser, "Setup", """\ Setup options are normally supplied by the testrunner script, although they can be overridden by users. """) setup.add_option( '--path', action="append", dest='path', help="""\ Specify a path to be added to Python's search path. This option can be used multiple times to specify multiple search paths. The path is usually specified by the test-runner script itself, rather than by users of the script, although it can be overridden by users. Only tests found in the path will be run. This option also specifies directories to be searched for tests. See the search_directory. """) setup.add_option( '--test-path', action="append", dest='test_path', help="""\ Specify a path to be searched for tests, but not added to the Python search path. This option can be used multiple times to specify multiple search paths. The path is usually specified by the test-runner script itself, rather than by users of the script, although it can be overridden by users. Only tests found in the path will be run. """) setup.add_option( '--package-path', action="append", dest='package_path', nargs=2, help="""\ Specify a path to be searched for tests, but not added to the Python search path. Also specify a package for files found in this path. This is used to deal with directories that are stitched into packages that are not otherwise searched for tests. This option takes 2 arguments. The first is a path name. The second is the package name. This option can be used multiple times to specify multiple search paths. The path is usually specified by the test-runner script itself, rather than by users of the script, although it can be overridden by users. Only tests found in the path will be run. """) setup.add_option( '--tests-pattern', action="store", dest='tests_pattern', help="""\ The test runner looks for modules containing tests. It uses this pattern to identify these modules. The modules may be either packages or python files. If a test module is a package, it uses the value given by the test-file-pattern to identify python files within the package containing tests. """) setup.add_option( '--suite-name', action="store", dest='suite_name', help="""\ Specify the name of the object in each test_module that contains the module's test suite. """) setup.add_option( '--test-file-pattern', action="store", dest='test_file_pattern', help="""\ Specify a pattern for identifying python files within a tests package. See the documentation for the --tests-pattern option. """) setup.add_option( '--ignore_dir', action="append", dest='ignore_dir', help="""\ Specifies the name of a directory to ignore when looking for tests. """) setup.add_option( '--shuffle', action="store_true", dest='shuffle', help="""\ Shuffles the order in which tests are ran. """) setup.add_option( '--shuffle-seed', action="store", dest='shuffle_seed', type="int", help="""\ Value used to initialize the tests shuffler. Specify a value to create repeatable random ordered tests. """) parser.add_option_group(setup) ###################################################################### # Other other = optparse.OptionGroup(parser, "Other", "Other options") other.add_option( '--version', action="store_true", dest='showversion', help="Print the version of the testrunner, and exit.") other.add_option( '-j', action="store", type="int", dest='processes', help="""\ Use up to given number of parallel processes to execute tests. May decrease test run time substantially. Defaults to %default. """) other.add_option( '--keepbytecode', '-k', action="store_true", dest='keepbytecode', help="""\ Normally, the test runner scans the test paths and the test directories looking for and deleting pyc or pyo files without corresponding py files. This is to prevent spurious test failures due to finding compiled modules where source modules have been deleted. This scan can be time consuming. Using this option disables this scan. If you know you haven't removed any modules since last running the tests, can make the test run go much faster. """) other.add_option( '--usecompiled', action="store_true", dest='usecompiled', help="""\ Normally, a package must contain an __init__.py file, and only .py files can contain test code. When this option is specified, compiled Python files (.pyc and .pyo) can be used instead: a directory containing __init__.pyc or __init__.pyo is also considered to be a package, and if file XYZ.py contains tests but is absent while XYZ.pyc or XYZ.pyo exists then the compiled files will be used. This is necessary when running tests against a tree where the .py files have been removed after compilation to .pyc/.pyo. Use of this option implies --keepbytecode. """) other.add_option( '--exit-with-status', action="store_true", dest='exitwithstatus', help="""DEPRECATED: The test runner will always exit with a status.\ """) parser.add_option_group(other) ###################################################################### # Default values parser.set_defaults( ignore_dir=['.git', '.svn', 'CVS', '{arch}', '.arch-ids', '_darcs'], tests_pattern='^tests$', at_level=1, test_file_pattern='^test', suite_name='test_suite', list_tests=False, slow_test_threshold=10, processes=1, verbose=0, repeat=1, ) ###################################################################### # Command-line processing def merge_options(options, defaults): odict = options.__dict__ for name, value in defaults.__dict__.items(): if (value is not None) and (odict[name] is None): odict[name] = value def get_options(args=None, defaults=None): # Because we want to inspect stdout and decide to colorize or not, we # replace the --auto-color option with the appropriate --color or # --no-color option. That way the subprocess doesn't have to decide (which # it would do incorrectly anyway because stdout would be a pipe). def apply_auto_color(args): if args and '--auto-color' in args: if sys.stdout.isatty() and terminal_has_colors(): colorization = '--color' else: colorization = '--no-color' args[:] = [arg.replace('--auto-color', colorization) for arg in args] # The comment of apply_auto_color applies here as well def apply_auto_progress(args): if args and '--auto-progress' in args: if sys.stdout.isatty(): progress = '--progress' else: progress = '--no-progress' args[:] = [arg.replace('--auto-progress', progress) for arg in args] apply_auto_color(args) apply_auto_color(defaults) apply_auto_progress(args) apply_auto_progress(defaults) if defaults: defaults, _ = parser.parse_args(defaults) assert not _ else: defaults = None if args is None: args = sys.argv options, positional = parser.parse_args(args[1:], defaults) options.original_testrunner_args = args if options.showversion: dist = pkg_resources.require('zope.testrunner')[0] print('zope.testrunner version %s' % dist.version) options.fail = True return options if options.subunit: try: import subunit except ImportError: print("""\ Subunit is not installed. Please install Subunit to generate subunit output. """) options.fail = True return options else: options.output = SubunitOutputFormatter(options) elif options.color: options.output = ColorfulOutputFormatter(options) options.output.slow_test_threshold = options.slow_test_threshold else: options.output = OutputFormatter(options) options.fail = False if positional: module_filter = positional.pop(0) if module_filter != '.': if options.module: options.module.append(module_filter) else: options.module = [module_filter] if positional: test_filter = positional.pop(0) if options.test: options.test.append(test_filter) else: options.test = [test_filter] if positional: parser.error("Too many positional arguments") options.ignore_dir = dict([(d,1) for d in options.ignore_dir]) options.test_file_pattern = re.compile(options.test_file_pattern).search options.tests_pattern = re.compile(options.tests_pattern).search options.test = [t for t in options.test or ('.')] options.module = [m for m in options.module or ('.')] options.path = [os.path.abspath(p) for p in options.path or ()] options.test_path = [os.path.abspath(p) for p in options.test_path or ()] options.test_path += options.path options.test_path = ([(path, '') for path in options.test_path] + [(os.path.abspath(path), package) for (path, package) in options.package_path or () ]) if options.package: pkgmap = dict(options.test_path) options.package = [normalize_package(p, pkgmap) for p in options.package] options.prefix = [(path + os.path.sep, package) for (path, package) in options.test_path] # Sort prefixes so that longest prefixes come first. # That is because only first match is evaluated which # can be a problem with nested source packages. options.prefix.sort(key=lambda p: len(p[0]), reverse=True) if options.all: options.at_level = sys.maxsize if options.unit and options.non_unit: # The test runner interprets this as "run only those tests that are # both unit and non-unit at the same time". The user, however, wants # to run both unit and non-unit tests. Disable the filtering so that # the user will get what she wants: options.unit = options.non_unit = False if options.unit: # XXX Argh. options.layer = ['zope.testrunner.layer.UnitTests'] options.layer = options.layer and dict([(l, 1) for l in options.layer]) if options.usecompiled: options.keepbytecode = options.usecompiled if options.quiet: options.verbose = 0 if options.report_refcounts and options.repeat < 2: print("""\ You must use the --repeat (-N) option to specify a repeat count greater than 1 when using the --report_refcounts (-r) option. """) options.fail = True return options if options.report_refcounts and not hasattr(sys, "gettotalrefcount"): print("""\ The Python you are running was not configured with --with-pydebug. This is required to use the --report-refcounts option. """) options.fail = True return options return options def normalize_package(package, package_map={}): r"""Normalize package name passed to the --package option. >>> normalize_package('zope.testrunner') 'zope.testrunner' Converts path names into package names for compatibility with the old test runner. >>> normalize_package('zope/testrunner') 'zope.testrunner' >>> normalize_package('zope/testrunner/') 'zope.testrunner' >>> normalize_package('zope\\testrunner') 'zope.testrunner' Can use a map of absolute pathnames to package names >>> a = os.path.abspath >>> normalize_package('src/zope/testrunner/', ... {a('src'): ''}) 'zope.testrunner' >>> normalize_package('src/zope_testrunner/', ... {a('src/zope_testrunner'): 'zope.testrunner'}) 'zope.testrunner' >>> normalize_package('src/zope_something/tests', ... {a('src/zope_something'): 'zope.something', ... a('src'): ''}) 'zope.something.tests' """ package = package.replace('\\', '/') if package.endswith('/'): package = package[:-1] bits = package.split('/') for n in range(len(bits), 0, -1): pkg = package_map.get(os.path.abspath('/'.join(bits[:n]))) if pkg is not None: bits = bits[n:] if pkg: bits = [pkg] + bits return '.'.join(bits) return package.replace('/', '.') zope.testrunner-4.4.9/src/zope/testrunner/find.py0000664000000000000000000004267012527371226020704 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test discovery """ import re import os import unittest import sys import six import zope.testrunner.feature import zope.testrunner.layer import zope.testrunner.debug from zope.testrunner.filter import build_filtering_func identifier = re.compile(r'[_a-zA-Z]\w*$').match class StartUpFailure(unittest.TestCase): """Empty test case added to the test suite to indicate import failures. >>> class Options(object): ... post_mortem = False >>> options = Options() Normally the StartUpFailure just acts as an empty test suite to satisfy the test runner and statistics: >>> s = StartUpFailure(options, 'fauxmodule', None) >>> s >>> isinstance(s,unittest.TestCase) True >>> s.shortDescription() 'StartUpFailure: import errors in fauxmodule.' Sometimes test suited collected by zope.testrunner end up being run by a regular unittest TestRunner, and it's not hard to make sure StartUpFailure does something sensible in that case >>> r = unittest.TestResult() >>> _ = s.run(r) >>> print(r.failures[0][1].rstrip()) # doctest: +ELLIPSIS Traceback (most recent call last): ... AssertionError: could not import fauxmodule If you'd like more details, be sure to pass the original exc_info:: >>> import sys >>> try: ... raise Exception('something bad happened during import') ... except: ... exc_info = sys.exc_info() >>> s = StartUpFailure(options, 'fauxmodule', exc_info) >>> r = unittest.TestResult() >>> _ = s.run(r) >>> print(r.errors[0][0].shortDescription()) StartUpFailure: import errors in fauxmodule. >>> print(r.errors[0][1].rstrip()) # doctest: +ELLIPSIS Traceback (most recent call last): ... Exception: something bad happened during import However, if the post mortem option is enabled: >>> options.post_mortem = True ...then the the StartUpFailure will start the debugger and stop the test run after the debugger quits. To simulate the user pressing 'c' and hitting return in the debugger, we use a FakeInputContinueGenerator: >>> from zope.testrunner.runner import FakeInputContinueGenerator >>> old_stdin = sys.stdin >>> sys.stdin = FakeInputContinueGenerator() Now we can see the EndRun exception that is raised by the postmortem debugger to indicate that debugging is finished and the test run should be terminated: >>> from zope.testrunner.interfaces import EndRun >>> try: #doctest: +ELLIPSIS ... try: # try...except...finally doesn't work in Python 2.4 ... print("Result:") # Needed to prevent the result from starting with '...' ... StartUpFailure(options, None, exc_info) ... except EndRun: ... print("EndRun raised") ... finally: ... sys.stdin = old_stdin Result: Exception: something bad happened during import ... (Pdb) c ********************************************************************** Can't use pdb.set_trace when running a layer as a subprocess! ********************************************************************** EndRun raised Annoyingly, sometimes StartUpFailures occur when postmortem debugging is enabled but no exc_info is passed. In this case, we raise a sensible exception rather than letting the debugger barf with an AttributeError: >>> options.post_mortem = True >>> StartUpFailure(options, None, exc_info[:2]+(None,)) Traceback (most recent call last): ... TypeError: If post_mortem is specified, full exc_info must be passed! """ def __init__(self, options, module, exc_info): if options.post_mortem: for item in exc_info: if item is None: raise TypeError('If post_mortem is specified, ' 'full exc_info must be passed!') zope.testrunner.debug.post_mortem(exc_info) self.module = module self.exc_info = exc_info super(StartUpFailure, self).__init__() def shortDescription(self): return 'StartUpFailure: import errors in %s.' % self.module def __repr__(self): return '' % self.module def runTest(self): if self.exc_info is None or any(x is None for x in self.exc_info): self.fail("could not import %s" % self.module) else: six.reraise(*self.exc_info) def find_tests(options, found_suites=None): """Creates a dictionary mapping layer name to a suite of tests to be run in that layer. Passing a list of suites using the found_suites parameter will cause that list of suites to be used instead of attempting to load them from the filesystem. This is useful for unit testing the test runner. """ remove_stale_bytecode(options) suites = {} test_accept = build_filtering_func(options.test) module_accept = build_filtering_func(options.module) if found_suites is None: found_suites = find_suites(options, accept=module_accept) for suite in found_suites: for test, layer_name in tests_from_suite(suite, options, accept=test_accept): suite = suites.get(layer_name) if not suite: suite = suites[layer_name] = unittest.TestSuite() suite.addTest(test) return suites def find_suites(options, accept=None): for fpath, package in find_test_files(options): for (prefix, prefix_package) in options.prefix: if fpath.startswith(prefix) and package == prefix_package: # strip prefix, strip .py suffix and convert separator to dots noprefix = fpath[len(prefix):] noext = strip_py_ext(options, noprefix) assert noext is not None module_name = noext.replace(os.path.sep, '.') if package: module_name = package + '.' + module_name if accept is not None and not accept(module_name): continue try: module = import_name(module_name) except KeyboardInterrupt: raise except: exc_info = sys.exc_info() if not options.post_mortem: # Skip a couple of frames exc_info = exc_info[:2] + (exc_info[2].tb_next.tb_next,) suite = StartUpFailure( options, module_name, exc_info) else: try: if hasattr(module, options.suite_name): suite = getattr(module, options.suite_name)() else: suite = unittest.defaultTestLoader.loadTestsFromModule(module) if suite.countTestCases() == 0: raise TypeError( "Module %s does not define any tests" % module_name) if isinstance(suite, unittest.TestSuite): check_suite(suite, module_name) else: raise TypeError( "Invalid test_suite, %r, in %s" % (suite, module_name) ) except KeyboardInterrupt: raise except: exc_info = sys.exc_info() if not options.post_mortem: # Suppress traceback exc_info = exc_info[:2] + (None,) suite = StartUpFailure( options, module_name, exc_info) yield suite break def find_test_files(options): found = {} for f, package in find_test_files_(options): if f not in found: found[f] = 1 yield f, package def find_test_files_(options): tests_pattern = options.tests_pattern test_file_pattern = options.test_file_pattern # If options.usecompiled, we can accept .pyc or .pyo files instead # of .py files. We'd rather use a .py file if one exists. `root2ext` # maps a test file path, sans extension, to the path with the best # extension found (.py if it exists, else .pyc or .pyo). # Note that "py" < "pyc" < "pyo", so if more than one extension is # found, the lexicographically smaller one is best. # Found a new test file, in directory `dirname`. `noext` is the # file name without an extension, and `withext` is the file name # with its extension. def update_root2ext(dirname, noext, withext): key = os.path.join(dirname, noext) new = os.path.join(dirname, withext) if key in root2ext: root2ext[key] = min(root2ext[key], new) else: root2ext[key] = new for (p, package) in test_dirs(options, {}): for dirname, dirs, files in walk_with_symlinks(options, p): if dirname != p and not contains_init_py(options, files): # This is not a plausible test directory. Avoid descending # further. del dirs[:] continue root2ext = {} dirs[:] = [d for d in dirs if identifier(d)] d = os.path.split(dirname)[1] if tests_pattern(d) and contains_init_py(options, files): # tests directory for file in files: noext = strip_py_ext(options, file) if noext and test_file_pattern(noext): update_root2ext(dirname, noext, file) for file in files: noext = strip_py_ext(options, file) if noext and tests_pattern(noext): update_root2ext(dirname, noext, file) winners = sorted(root2ext.values()) for file in winners: yield file, package def strip_py_ext(options, path): """Return path without its .py (or .pyc or .pyo) extension, or None. If options.usecompiled is false: If path ends with ".py", the path without the extension is returned. Else None is returned. If options.usecompiled is true: If Python is running with -O, a .pyo extension is also accepted. If Python is running without -O, a .pyc extension is also accepted. """ if path.endswith(".py"): return path[:-3] if options.usecompiled: if __debug__: # Python is running without -O. ext = ".pyc" else: # Python is running with -O. ext = ".pyo" if path.endswith(ext): return path[:-len(ext)] return None def test_dirs(options, seen): if options.package: for p in options.package: p = import_name(p) for p in p.__path__: p = os.path.abspath(p) if p in seen: continue for (prefix, package) in options.prefix: if p.startswith(prefix) or p == prefix[:-1]: seen[p] = 1 yield p, package break else: for dpath in options.test_path: yield dpath def walk_with_symlinks(options, dir): # TODO -- really should have test of this that uses symlinks # this is hard on a number of levels ... for dirpath, dirs, files in os.walk(dir): dirs.sort() files.sort() dirs[:] = [d for d in dirs if d not in options.ignore_dir] yield (dirpath, dirs, files) for d in dirs: p = os.path.join(dirpath, d) if os.path.islink(p): for sdirpath, sdirs, sfiles in walk_with_symlinks(options, p): yield (sdirpath, sdirs, sfiles) compiled_suffixes = '.pyc', '.pyo' def remove_stale_bytecode(options): if options.keepbytecode: return for (p, _) in options.test_path: for dirname, dirs, files in walk_with_symlinks(options, p): if '__pycache__' in dirs: # Do not recurse in there: we would end up removing all pyc # files because the main loop checks for py files in the same # directory. Besides, stale pyc files in __pycache__ are # harmless, see PEP-3147 for details (sourceless imports # work only when pyc lives in the source dir directly). dirs.remove('__pycache__') for file in files: if file[-4:] in compiled_suffixes and file[:-1] not in files: fullname = os.path.join(dirname, file) options.output.info("Removing stale bytecode file %s" % fullname) os.unlink(fullname) def contains_init_py(options, fnamelist): """Return true iff fnamelist contains a suitable spelling of __init__.py. If options.usecompiled is false, this is so iff "__init__.py" is in the list. If options.usecompiled is true, then "__init__.pyo" is also acceptable if Python is running with -O, and "__init__.pyc" is also acceptable if Python is running without -O. """ if "__init__.py" in fnamelist: return True if options.usecompiled: if __debug__: # Python is running without -O. return "__init__.pyc" in fnamelist else: # Python is running with -O. return "__init__.pyo" in fnamelist return False def import_name(name): __import__(name) return sys.modules[name] def tests_from_suite(suite, options, dlevel=1, dlayer=zope.testrunner.layer.UnitTests, accept=None): """Returns a sequence of (test, layer_name) The tree of suites is recursively visited, with the most specific layer taking precedence. So if a TestCase with a layer of 'foo' is contained in a TestSuite with a layer of 'bar', the test case would be returned with 'foo' as the layer. Tests are also filtered out based on the test level and accept predicate. accept is a function, returning boolean for given test name (see also build_filtering_func()). """ level = getattr(suite, 'level', dlevel) layer = getattr(suite, 'layer', dlayer) if not isinstance(layer, six.string_types): layer = name_from_layer(layer) if isinstance(suite, unittest.TestSuite): for possible_suite in suite: for r in tests_from_suite(possible_suite, options, level, layer, accept=accept): yield r elif isinstance(suite, StartUpFailure): yield (suite, None) else: if level <= options.at_level: if accept is None or accept(str(suite)): yield (suite, layer) def check_suite(suite, module_name): """Check for bad tests in a test suite. "Bad tests" are those that do not inherit from unittest.TestCase. Note that this function is pointless on Python 2.5, because unittest itself checks for this in TestSuite.addTest. It is, however, useful on earlier Pythons. """ for x in suite: if isinstance(x, unittest.TestSuite): check_suite(x, module_name) elif not isinstance(x, unittest.TestCase): raise TypeError( "Invalid test, %r,\nin test_suite from %s" % (x, module_name) ) _layer_name_cache = {} def name_from_layer(layer): """Determine a name for the Layer using the namespace to avoid conflicts. We also cache a name -> layer mapping to enable layer_from_name to work in cases where the layer cannot be imported (such as layers defined in doctests) """ if layer.__module__ == '__builtin__': name = layer.__name__ else: name = layer.__module__ + '.' + layer.__name__ _layer_name_cache[name] = layer return name class Find(zope.testrunner.feature.Feature): """Finds tests and registers them with the test runner.""" active = True def global_setup(self): # Add directories to the path for path in self.runner.options.path: if path not in sys.path: sys.path.append(path) tests = find_tests(self.runner.options, self.runner.found_suites) self.import_errors = tests.pop(None, None) self.runner.register_tests(tests) # XXX move to reporting ??? self.runner.options.output.import_errors(self.import_errors) self.runner.import_errors = list(self.import_errors or []) def report(self): self.runner.options.output.modules_with_import_problems( self.import_errors) zope.testrunner-4.4.9/src/zope/testrunner/refcount.py0000664000000000000000000000624312527371226021605 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Support for tracking reference counts. """ from __future__ import print_function import gc import sys import types class TrackRefs(object): """Object to track reference counts across test runs.""" def __init__(self): self.type2count = {} self.type2all = {} self.delta = None self.n = 0 self.update() self.delta = None def update(self): gc.collect() obs = sys.getobjects(0) type2count = {} type2all = {} n = 0 for o in obs: if type(o) is str and o == '': # avoid dictionary madness continue all = sys.getrefcount(o) - 3 n += all t = type(o) if t is types.InstanceType: t = o.__class__ if t in type2count: type2count[t] += 1 type2all[t] += all else: type2count[t] = 1 type2all[t] = all ct = [( type_or_class_title(t), type2count[t] - self.type2count.get(t, 0), type2all[t] - self.type2all.get(t, 0), ) for t in type2count] ct += [( type_or_class_title(t), - self.type2count[t], - self.type2all[t], ) for t in self.type2count if t not in type2count] ct.sort() self.delta = ct self.type2count = type2count self.type2all = type2all self.n = n def output(self): printed = False s1 = s2 = 0 for t, delta1, delta2 in self.delta: if delta1 or delta2: if not printed: print ( ' Leak details, changes in instances and refcounts' ' by type/class:') print(" %-55s %6s %6s" % ('type/class', 'insts', 'refs')) print(" %-55s %6s %6s" % ('-' * 55, '-----', '----')) printed = True print(" %-55s %6d %6d" % (t, delta1, delta2)) s1 += delta1 s2 += delta2 if printed: print(" %-55s %6s %6s" % ('-' * 55, '-----', '----')) print(" %-55s %6s %6s" % ('total', s1, s2)) self.delta = None def type_or_class_title(t): module = getattr(t, '__module__', '__builtin__') if module == '__builtin__': return t.__name__ return "%s.%s" % (module, t.__name__) zope.testrunner-4.4.9/src/zope/testrunner/runner.py0000664000000000000000000010114212527371226021263 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test execution """ from __future__ import print_function import collections import subprocess import errno import gc import inspect import re import sys import threading import time import traceback try: import unittest2 as unittest except ImportError: import unittest from six import StringIO from zope.testrunner.find import import_name from zope.testrunner.find import name_from_layer, _layer_name_cache from zope.testrunner.layer import UnitTests from zope.testrunner.layer import EmptyLayer from zope.testrunner.layer import EmptySuite from zope.testrunner.options import get_options from zope.testrunner.refcount import TrackRefs import zope.testrunner import zope.testrunner.coverage import zope.testrunner._doctest import zope.testrunner.logsupport import zope.testrunner.selftest import zope.testrunner.profiling import zope.testrunner.filter import zope.testrunner.garbagecollection import zope.testrunner.listing import zope.testrunner.statistics import zope.testrunner.process import zope.testrunner.interfaces import zope.testrunner.debug import zope.testrunner.tb_format import zope.testrunner.shuffle try: import Queue # Python 2 except ImportError: # Python 3 import queue as Queue # Python 3 try: from unittest import _UnexpectedSuccess # Python 3.1 except ImportError: try: from unittest.case import _UnexpectedSuccess # Python 2.7 and 3.2 except ImportError: class _UnexpectedSuccess(Exception): pass PYREFCOUNT_PATTERN = re.compile('\[[0-9]+ refs\]') is_jython = sys.platform.startswith('java') class SubprocessError(Exception): """An error occurred when running a subprocess """ def __init__(self, reason, stderr): self.reason = reason self.stderr = stderr def __str__(self): return '%s: %s' % (self.reason, self.stderr) class CanNotTearDown(Exception): "Couldn't tear down a test" class Runner(object): """The test runner. It is the central point of this package and responsible for finding and executing tests as well as configuring itself from the (command-line) options passed into it. """ def __init__(self, defaults=None, args=None, found_suites=None, options=None, script_parts=None, cwd=None): if defaults is None: self.defaults = [] else: self.defaults = defaults self.args = args self.found_suites = found_suites self.options = options self.script_parts = script_parts self.cwd = cwd self.failed = True self.ran = 0 self.skipped = [] self.failures = [] self.errors = [] self.import_errors = [] self.show_report = True self.do_run_tests = True self.features = [] self.tests_by_layer_name = {} def ordered_layers(self): if (self.options.processes > 1 and not self.options.resume_layer): # if we want multiple processes, we need a fake layer as first # to start spreading out layers/tests to subprocesses # but only if this is not in the subprocess yield (name_from_layer(EmptyLayer), EmptyLayer, EmptySuite()) layer_names = dict([(layer_from_name(layer_name), layer_name) for layer_name in self.tests_by_layer_name]) for layer in order_by_bases(layer_names): layer_name = layer_names[layer] yield layer_name, layer, self.tests_by_layer_name[layer_name] def register_tests(self, tests): """Registers tests.""" # XXX To support multiple features that find tests this shouldn't be # an update but merge the various layers individually. self.tests_by_layer_name.update(tests) def run(self): self.configure() if self.options.fail: return True # XXX Hacky to support existing code. self.layer_name_cache = _layer_name_cache self.layer_name_cache.clear() # Global setup for feature in self.features: feature.global_setup() # Late setup # # Some system tools like profilers are really bad with stack frames. # E.g. hotshot doesn't like it when we leave the stack frame that we # called start() from. for feature in self.features: feature.late_setup() try: if self.do_run_tests: self.run_tests() finally: # Early teardown for feature in reversed(self.features): feature.early_teardown() # Global teardown for feature in reversed(self.features): feature.global_teardown() if self.show_report: for feature in self.features: feature.report() def configure(self): if self.args is None: self.args = sys.argv[:] # Check to see if we are being run as a subprocess. If we are, # then use the resume-layer and defaults passed in. if len(self.args) > 1 and self.args[1] == '--resume-layer': self.args.pop(1) resume_layer = self.args.pop(1) resume_number = int(self.args.pop(1)) self.defaults = [] while len(self.args) > 1 and self.args[1] == '--default': self.args.pop(1) self.defaults.append(self.args.pop(1)) sys.stdin = FakeInputContinueGenerator() else: resume_layer = resume_number = None options = get_options(self.args, self.defaults) options.testrunner_defaults = self.defaults options.resume_layer = resume_layer options.resume_number = resume_number self.options = options self.features.append(zope.testrunner.selftest.SelfTest(self)) self.features.append(zope.testrunner.logsupport.Logging(self)) self.features.append(zope.testrunner.coverage.Coverage(self)) self.features.append(zope.testrunner._doctest.DocTest(self)) self.features.append(zope.testrunner.profiling.Profiling(self)) if is_jython: # Jython GC support is not yet implemented pass else: self.features.append( zope.testrunner.garbagecollection.Threshold(self)) self.features.append( zope.testrunner.garbagecollection.Debug(self)) self.features.append(zope.testrunner.find.Find(self)) self.features.append(zope.testrunner.shuffle.Shuffle(self)) self.features.append(zope.testrunner.process.SubProcess(self)) self.features.append(zope.testrunner.filter.Filter(self)) self.features.append(zope.testrunner.listing.Listing(self)) self.features.append( zope.testrunner.statistics.Statistics(self)) self.features.append(zope.testrunner.tb_format.Traceback(self)) # Remove all features that aren't activated self.features = [f for f in self.features if f.active] def run_tests(self): """Run all tests that were registered. Returns True if there where failures or False if all tests passed. """ setup_layers = {} layers_to_run = list(self.ordered_layers()) should_resume = False while layers_to_run: layer_name, layer, tests = layers_to_run[0] for feature in self.features: feature.layer_setup(layer) try: self.ran += run_layer(self.options, layer_name, layer, tests, setup_layers, self.failures, self.errors, self.skipped, self.import_errors) except zope.testrunner.interfaces.EndRun: self.failed = True break except CanNotTearDown: if not self.options.resume_layer: should_resume = True break layers_to_run.pop(0) if self.options.processes > 1: should_resume = True break if should_resume: if layers_to_run: self.ran += resume_tests( self.script_parts, self.options, self.features, layers_to_run, self.failures, self.errors, self.skipped, self.cwd) if setup_layers: if self.options.resume_layer is None: self.options.output.info("Tearing down left over layers:") tear_down_unneeded(self.options, (), setup_layers, True) self.failed = bool(self.import_errors or self.failures or self.errors) def run_tests(options, tests, name, failures, errors, skipped, import_errors): repeat = options.repeat or 1 repeat_range = iter(range(repeat)) ran = 0 output = options.output if is_jython: # Jython has no GC suppport - set count to 0 lgarbage = 0 else: gc.collect() lgarbage = len(gc.garbage) if options.report_refcounts: if options.verbose: # XXX This code path is untested track = TrackRefs() rc = sys.gettotalrefcount() for iteration in repeat_range: if repeat > 1: output.info("Iteration %d" % (iteration + 1)) if options.verbose > 0 or options.progress: output.info(' Running:') result = TestResult(options, tests, layer_name=name) t = time.time() if options.post_mortem: # post-mortem debugging for test in tests: if result.shouldStop: break result.startTest(test) state = test.__dict__.copy() try: try: test.debug() except KeyboardInterrupt: raise except: result.addError( test, sys.exc_info()[:2] + (sys.exc_info()[2].tb_next, ), ) else: result.addSuccess(test) finally: result.stopTest(test) test.__dict__.clear() test.__dict__.update(state) else: # normal for test in tests: if result.shouldStop: break state = test.__dict__.copy() test(result) test.__dict__.clear() test.__dict__.update(state) t = time.time() - t output.stop_tests() failures.extend(result.failures) if hasattr(result, 'unexpectedSuccesses'): # Python versions prior to 2.7 do not have the concept of # unexpectedSuccesses. failures.extend(result.unexpectedSuccesses) if not hasattr(result, 'skipped'): # Only in Python >= 2.7, Python >= 3.1, and when using unittest2 result.skipped = [] skipped.extend(result.skipped) errors.extend(result.errors) output.summary(n_tests=result.testsRun, n_failures=len(failures), n_errors=len(result.errors) + len(import_errors), n_seconds=t, n_skipped=len(result.skipped)) ran = result.testsRun if is_jython: lgarbage = 0 else: gc.collect() if len(gc.garbage) > lgarbage: output.garbage(gc.garbage[lgarbage:]) lgarbage = len(gc.garbage) if options.report_refcounts: # If we are being tested, we don't want stdout itself to # foul up the numbers. :) try: sys.stdout.getvalue() except AttributeError: pass prev = rc rc = sys.gettotalrefcount() if options.verbose: track.update() if iteration > 0: output.detailed_refcounts(track, rc, prev) else: track.delta = None elif iteration > 0: output.refcounts(rc, prev) return ran def run_layer(options, layer_name, layer, tests, setup_layers, failures, errors, skipped, import_errors): output = options.output gathered = [] gather_layers(layer, gathered) needed = dict([(l, 1) for l in gathered]) if options.resume_number != 0: output.info("Running %s tests:" % layer_name) tear_down_unneeded(options, needed, setup_layers) if options.resume_layer is not None: output.info_suboptimal(" Running in a subprocess.") try: setup_layer(options, layer, setup_layers) except zope.testrunner.interfaces.EndRun: raise except Exception: f = StringIO() traceback.print_exc(file=f) output.error(f.getvalue()) errors.append((SetUpLayerFailure(layer), sys.exc_info())) return 0 else: return run_tests(options, tests, layer_name, failures, errors, skipped, import_errors) class SetUpLayerFailure(unittest.TestCase): def __init__(self, layer): super(SetUpLayerFailure, self).__init__() self.layer = layer def runTest(self): pass def __str__(self): return "Layer: %s" % (name_from_layer(self.layer)) def spawn_layer_in_subprocess(result, script_parts, options, features, layer_name, layer, failures, errors, skipped, resume_number, cwd=None): output = options.output try: # BBB if script_parts is None: script_parts = zope.testrunner._script_parts() args = [sys.executable] args.extend(script_parts) args.extend(['--resume-layer', layer_name, str(resume_number)]) for d in options.testrunner_defaults: args.extend(['--default', d]) args.extend(options.original_testrunner_args[1:]) # this is because of a bug in Python (http://www.python.org/sf/900092) if (options.profile == 'hotshot' and sys.version_info[:3] <= (2, 4, 1)): args.insert(1, '-O') debugargs = args # save them before messing up for windows if sys.platform.startswith('win'): args = args[0] + ' ' + ' '.join([ ('"' + a.replace('\\', '\\\\').replace('"', '\\"') + '"') for a in args[1:]]) for feature in features: feature.layer_setup(layer) child = subprocess.Popen(args, shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd, close_fds=not sys.platform.startswith('win')) while True: try: while True: # We use readline() instead of iterating over stdout # because it appears that iterating over stdout causes a # lot more buffering to take place (probably so it can # return its lines as a batch). We don't want too much # buffering because this foils automatic and human monitors # trying to verify that the subprocess is still alive. l = child.stdout.readline() if not l: break result.write(l) except IOError as e: if e.errno == errno.EINTR: # If the subprocess dies before we finish reading its # output, a SIGCHLD signal can interrupt the reading. # The correct thing to to in that case is to retry. continue output.error( "Error reading subprocess output for %s" % layer_name) output.info(str(e)) else: break # Now stderr should be ready to read the whole thing. errlines = child.stderr.read().splitlines() erriter = iter(errlines) nfail = nerr = 0 for line in erriter: try: result.num_ran, nfail, nerr = map(int, line.strip().split()) except ValueError: continue else: break else: errmsg = "Could not communicate with subprocess!" errors.append(("subprocess for %s" % layer_name, None)) if options.verbose >= 1: errmsg += "\nChild command line: %s" % debugargs if (options.verbose >= 2 or (options.verbose == 1 and len(errlines) < 20)): errmsg += ("\nChild stderr was:\n" + "\n".join(" " + line.decode('utf-8', 'replace') for line in errlines)) elif options.verbose >= 1: errmsg += ("\nChild stderr was:\n" + "\n".join(" " + line.decode('utf-8', 'replace') for line in errlines[:10]) + "\n...\n" + "\n".join(" " + line.decode('utf-8', 'replace') for line in errlines[-10:])) output.error_with_banner(errmsg) while nfail > 0: nfail -= 1 # Doing erriter.next().strip() confuses the 2to3 fixer, so # we need to do it on a separate line. Also, in python 3 this # returns bytes, so we decode it. next_fail = next(erriter) failures.append((next_fail.strip().decode(), None)) while nerr > 0: nerr -= 1 # Doing erriter.next().strip() confuses the 2to3 fixer, so # we need to do it on a separate line. Also, in python 3 this # returns bytes, so we decode it. next_err = next(erriter) errors.append((next_err.strip().decode(), None)) finally: result.done = True class AbstractSubprocessResult(object): """A result of a subprocess layer run.""" num_ran = 0 done = False def __init__(self, layer_name, queue): self.layer_name = layer_name self.queue = queue self.stdout = [] def write(self, out): """Receive a line of the subprocess out.""" class DeferredSubprocessResult(AbstractSubprocessResult): """Keeps stdout around for later processing,""" def write(self, out): if not isinstance(out, str): # It's binary, which means it's Python 3 out = out.decode() if not _is_dots(out): self.stdout.append(out) class ImmediateSubprocessResult(AbstractSubprocessResult): """Sends complete output to queue.""" def write(self, out): if not isinstance(out, str): # In Python 3, a Popen process stdout uses bytes, # While normal stdout uses strings. So we need to convert # from bytes to strings here. out = out.decode() sys.stdout.write(out) # Help keep-alive monitors (human or automated) keep up-to-date. sys.stdout.flush() _is_dots = re.compile(r'\.+(\r\n?|\n)').match # Windows sneaks in a \r\n. class KeepaliveSubprocessResult(AbstractSubprocessResult): "Keeps stdout for later processing; sends marks to queue to show activity." _done = False def _set_done(self, value): self._done = value assert value, 'Internal error: unexpectedly setting done to False' self.queue.put((self.layer_name, ' LAYER FINISHED')) done = property(lambda self: self._done, _set_done) def write(self, out): if not isinstance(out, str): # It's binary, which means it's Python 3 out = out.decode() if _is_dots(out): self.queue.put((self.layer_name, out.strip())) else: self.stdout.append(out) def resume_tests(script_parts, options, features, layers, failures, errors, skipped, cwd=None): results = [] stdout_queue = None if options.processes == 1: result_factory = ImmediateSubprocessResult elif options.verbose > 1: result_factory = KeepaliveSubprocessResult stdout_queue = Queue.Queue() else: result_factory = DeferredSubprocessResult resume_number = int(options.processes > 1) ready_threads = [] for layer_name, layer, tests in layers: result = result_factory(layer_name, stdout_queue) results.append(result) ready_threads.append(threading.Thread( target=spawn_layer_in_subprocess, args=(result, script_parts, options, features, layer_name, layer, failures, errors, skipped, resume_number, cwd))) resume_number += 1 # Now start a few threads at a time. running_threads = [] results_iter = iter(results) current_result = next(results_iter) last_layer_intermediate_output = None output = None while ready_threads or running_threads: while len(running_threads) < options.processes and ready_threads: thread = ready_threads.pop(0) thread.start() running_threads.append(thread) for index, thread in reversed(list(enumerate(running_threads))): if not thread.isAlive(): del running_threads[index] # Clear out any messages in queue while stdout_queue is not None: previous_output = output try: layer_name, output = stdout_queue.get(False) except Queue.Empty: break if layer_name != last_layer_intermediate_output: # Clarify what layer is reporting activity. if previous_output is not None: sys.stdout.write(']\n') sys.stdout.write( '[Parallel tests running in %s:\n ' % (layer_name,)) last_layer_intermediate_output = layer_name sys.stdout.write(output) # Display results in the order they would have been displayed, had the # work not been done in parallel. while current_result and current_result.done: if output is not None: sys.stdout.write(']\n') output = None sys.stdout.writelines(current_result.stdout) try: current_result = next(results_iter) except StopIteration: current_result = None # Help keep-alive monitors (human or automated) keep up-to-date. sys.stdout.flush() time.sleep(0.01) # Keep the loop from being too tight. # Return the total number of tests run. return sum(r.num_ran for r in results) def tear_down_unneeded(options, needed, setup_layers, optional=False): # Tear down any layers not needed for these tests. The unneeded layers # might interfere. unneeded = [l for l in setup_layers if l not in needed] unneeded = order_by_bases(unneeded) unneeded.reverse() output = options.output for l in unneeded: output.start_tear_down(name_from_layer(l)) t = time.time() try: try: if hasattr(l, 'tearDown'): l.tearDown() except NotImplementedError: output.tear_down_not_supported() if not optional: raise CanNotTearDown(l) else: output.stop_tear_down(time.time() - t) finally: del setup_layers[l] cant_pm_in_subprocess_message = """ Can't post-mortem debug when running a layer as a subprocess! Try running layer %r by itself. """ def setup_layer(options, layer, setup_layers): assert layer is not object output = options.output if layer not in setup_layers: for base in layer.__bases__: if base is not object: setup_layer(options, base, setup_layers) output.start_set_up(name_from_layer(layer)) t = time.time() if hasattr(layer, 'setUp'): try: layer.setUp() except Exception: if options.post_mortem: if options.resume_layer: options.output.error_with_banner( cant_pm_in_subprocess_message % options.resume_layer) raise else: zope.testrunner.debug.post_mortem( sys.exc_info()) else: raise output.stop_set_up(time.time() - t) setup_layers[layer] = 1 class TestResult(unittest.TestResult): def __init__(self, options, tests, layer_name=None): unittest.TestResult.__init__(self) self.options = options # Calculate our list of relevant layers we need to call testSetUp # and testTearDown on. layers = [] gather_layers(layer_from_name(layer_name), layers) self.layers = order_by_bases(layers) count = 0 for test in tests: count += test.countTestCases() self.count = count def testSetUp(self): """A layer may define a setup method to be called before each individual test. """ for layer in self.layers: if hasattr(layer, 'testSetUp'): layer.testSetUp() def testTearDown(self): """A layer may define a teardown method to be called after each individual test. This is useful for clearing the state of global resources or resetting external systems such as relational databases or daemons. """ for layer in self.layers[-1::-1]: if hasattr(layer, 'testTearDown'): layer.testTearDown() def startTest(self, test): self.testSetUp() unittest.TestResult.startTest(self, test) testsRun = self.testsRun - 1 # subtract the one the base class added count = test.countTestCases() self.testsRun = testsRun + count self.options.output.start_test(test, self.testsRun, self.count) self._threads = threading.enumerate() self._start_time = time.time() def addSuccess(self, test): t = max(time.time() - self._start_time, 0.0) self.options.output.test_success(test, t) def addSkip(self, test, reason): unittest.TestResult.addSkip(self, test, reason) self.options.output.test_skipped(test, reason) def addError(self, test, exc_info): self.options.output.test_error(test, time.time() - self._start_time, exc_info) unittest.TestResult.addError(self, test, exc_info) if self.options.post_mortem: if self.options.resume_layer: self.options.output.error_with_banner("Can't post-mortem debug" " when running a layer" " as a subprocess!") else: zope.testrunner.debug.post_mortem(exc_info) elif self.options.stop_on_error: self.stop() def addFailure(self, test, exc_info): self.options.output.test_failure(test, time.time() - self._start_time, exc_info) unittest.TestResult.addFailure(self, test, exc_info) if self.options.post_mortem: # XXX: mgedmin: why isn't there a resume_layer check here like # in addError? zope.testrunner.debug.post_mortem(exc_info) elif self.options.stop_on_error: self.stop() def addExpectedFailure(self, test, exc_info): t = max(time.time() - self._start_time, 0.0) self.options.output.test_success(test, t) unittest.TestResult.addExpectedFailure(self, test, exc_info) def addUnexpectedSuccess(self, test): self.options.output.test_error(test, time.time() - self._start_time, (_UnexpectedSuccess, None, None)) unittest.TestResult.addUnexpectedSuccess(self, test) if self.options.post_mortem: if self.options.resume_layer: self.options.output.error_with_banner("Can't post-mortem debug" " when running a layer" " as a subprocess!") else: # XXX: what exc_info? there's no exc_info! zope.testrunner.debug.post_mortem(exc_info) elif self.options.stop_on_error: self.stop() def stopTest(self, test): self.testTearDown() self.options.output.stop_test(test) if is_jython: pass else: if gc.garbage: self.options.output.test_garbage(test, gc.garbage) # TODO: Perhaps eat the garbage here, so that the garbage isn't # printed for every subsequent test. # Did the test leave any new threads behind? new_threads = [t for t in threading.enumerate() if (t.isAlive() and t not in self._threads)] if new_threads: self.options.output.test_threads(test, new_threads) def layer_from_name(layer_name): """Return the layer for the corresponding layer_name by discovering and importing the necessary module if necessary. Note that a name -> layer cache is maintained by name_from_layer to allow locating layers in cases where it would otherwise be impossible. """ if layer_name in _layer_name_cache: return _layer_name_cache[layer_name] layer_names = layer_name.split('.') layer_module, module_layer_name = layer_names[:-1], layer_names[-1] module_name = '.'.join(layer_module) module = import_name(module_name) try: return getattr(module, module_layer_name) except AttributeError: # the default error is very uninformative: # AttributeError: 'module' object has no attribute 'DemoLayer' # it doesn't say *which* module raise AttributeError('module %r has no attribute %r' % (module_name, module_layer_name)) def layer_sort_key(layer): """Compute sort key for layers. Based on the reverse MRO ordering in order to put layers with shared base layers next to each other. """ seen = set([]) key = [] # Note: we cannot reuse gather_layers() here because it uses a # different traversal order. def _gather(layer): seen.add(layer) # We make the simplifying assumption that the order of initialization # of base layers does not matter. Given that, traversing the bases # in reverse order here keeps the ordering of layers in # testrunner-layers.txt the same as it was in older versions of # zope.testrunner, so let's use that. for base in layer.__bases__[::-1]: if base is not object and base not in seen: _gather(base) key.append(layer) _gather(layer) return tuple(name_from_layer(l) for l in key if l != UnitTests) def order_by_bases(layers): """Order the layers from least to most specific (bottom to top). Puts unit tests first. Groups layers with common base layers together. Sorts the rest alphabetically. Removes duplicates. """ layers = sorted(layers, key=layer_sort_key, reverse=True) gathered = [] for layer in layers: gather_layers(layer, gathered) gathered.reverse() seen = {} result = [] for layer in gathered: if layer not in seen: seen[layer] = 1 if layer in layers: result.append(layer) return result def gather_layers(layer, result): if layer is not object: result.append(layer) for b in layer.__bases__: gather_layers(b, result) class FakeInputContinueGenerator: def readline(self): print('c\n') print('*'*70) print ("Can't use pdb.set_trace when running a layer" " as a subprocess!") print('*'*70) print() return 'c\n' zope.testrunner-4.4.9/src/zope/testrunner/listing.py0000664000000000000000000000237712527371226021435 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Filter which tests to run. """ import zope.testrunner.feature class Listing(zope.testrunner.feature.Feature): """Lists all tests in the report instead of running the tests.""" def __init__(self, runner): super(Listing, self).__init__(runner) self.active = bool(runner.options.list_tests) def global_setup(self): self.runner.do_run_tests = False self.runner.failed = False def report(self): layers = self.runner.tests_by_layer_name for layer_name, layer, tests in self.runner.ordered_layers(): self.runner.options.output.list_of_tests(tests, layer_name) zope.testrunner-4.4.9/src/zope/testrunner/statistics.py0000664000000000000000000000300512527371226022143 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test runner statistics """ import time import zope.testrunner.feature class Statistics(zope.testrunner.feature.Feature): active = True layers_run = 0 tests_run = 0 def late_setup(self): self.start_time = time.time() def early_teardown(self): self.end_time = time.time() def global_teardown(self): self.total_time = self.end_time - self.start_time def layer_setup(self, layer): self.layers_run += 1 def report(self): if not self.runner.do_run_tests: return if self.layers_run == 1: return self.runner.options.output.totals( n_tests=self.runner.ran, n_failures=len(self.runner.failures), n_errors=len(self.runner.errors) + len(self.runner.import_errors), n_seconds=self.total_time, n_skipped=len(self.runner.skipped)) zope.testrunner-4.4.9/src/zope/testrunner/shuffle.py0000664000000000000000000000544712527371226021421 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Shuffle tests discovered before executing them. """ import sys import time import random import zope.testrunner.feature class Shuffle(zope.testrunner.feature.Feature): """Take the tests found so far and shuffle them.""" def __init__(self, runner): super(Shuffle, self).__init__(runner) self.active = runner.options.shuffle self.seed = runner.options.shuffle_seed if self.seed is None: # We can't rely on the random modules seed initialization because # we can't introspect the seed later for reporting. This is a # simple emulation of what random.Random.seed does anyway. self.seed = int(time.time() * 256) # use fractional seconds def global_setup(self): rng = random.Random(self.seed) if sys.version_info >= (3, 2): # in case somebody tries to use a string as the seed rng.seed(self.seed, version=1) # Be careful to shuffle the layers in a deterministic order! for layer, suite in sorted(self.runner.tests_by_layer_name.items()): # Test suites cannot be modified through a public API. We thus # take a mutable copy of the list of tests of that suite, shuffle # that and replace the test suite instance with a new one of the # same class. tests = list(suite) # Black magic: if we don't pass random=rng.random to rng.shuffle, # we get different results on Python 2.7--3.1 and 3.2. The # standard library guarantees rng.random() will return the same # floats if given the same seed. It makes no guarantees for # rng.randrange, or rng.choice, or rng.shuffle. Experiments # show that Python happily breaks backwards compatibility for # these functions. By passing the random function we trick # shuffle() into using the old algorithm. rng.shuffle(tests, random=rng.random) self.runner.tests_by_layer_name[layer] = suite.__class__(tests) def report(self): msg = "Tests were shuffled using seed number %d." % self.seed self.runner.options.output.info(msg) zope.testrunner-4.4.9/src/zope/testrunner/formatter.py0000664000000000000000000012023212527371226021756 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Output formatting. """ from __future__ import print_function import doctest import os import sys import re import tempfile import traceback from datetime import datetime, timedelta from zope.testrunner.exceptions import DocTestFailureException try: unicode except NameError: unicode = str doctest_template = """ File "%s", line %s, in %s %s Want: %s Got: %s """ class OutputFormatter(object): """Test runner output formatter.""" # Implementation note: be careful about printing stuff to sys.stderr. # It is used for interprocess communication between the parent and the # child test runner, when you run some test layers in a subprocess. # resume_layer() reasigns sys.stderr for this reason, but be careful # and don't store the original one in __init__ or something. max_width = 80 def __init__(self, options): self.options = options self.last_width = 0 self.compute_max_width() progress = property(lambda self: self.options.progress) verbose = property(lambda self: self.options.verbose) in_subprocess = property( lambda self: self.options.resume_layer is not None and self.options.processes > 1) def compute_max_width(self): """Try to determine the terminal width.""" # Note that doing this every time is more test friendly. self.max_width = tigetnum('cols', self.max_width) def getShortDescription(self, test, room): """Return a description of a test that fits in ``room`` characters.""" room -= 1 s = str(test) if len(s) > room: pos = s.find(" (") if pos >= 0: w = room - (pos + 5) if w < 1: # first portion (test method name) is too long s = s[:room-3] + "..." else: pre = s[:pos+2] post = s[-w:] s = "%s...%s" % (pre, post) else: w = room - 4 s = '... ' + s[-w:] return ' ' + s[:room] def info(self, message): """Print an informative message.""" print(message) def info_suboptimal(self, message): """Print an informative message about losing some of the features. For example, when you run some tests in a subprocess, you lose the ability to use the debugger. """ print(message) def error(self, message): """Report an error.""" print(message) def error_with_banner(self, message): """Report an error with a big ASCII banner.""" print() print('*'*70) self.error(message) print('*'*70) print() def profiler_stats(self, stats): """Report profiler stats.""" stats.print_stats(50) def import_errors(self, import_errors): """Report test-module import errors (if any).""" if import_errors: print("Test-module import failures:") for error in import_errors: self.print_traceback("Module: %s\n" % error.module, error.exc_info), print() def tests_with_errors(self, errors): """Report names of tests with errors (if any).""" if errors: print() print("Tests with errors:") for test, exc_info in errors: print(" ", test) def tests_with_failures(self, failures): """Report names of tests with failures (if any).""" if failures: print() print("Tests with failures:") for test, exc_info in failures: print(" ", test) def modules_with_import_problems(self, import_errors): """Report names of modules with import problems (if any).""" if import_errors: print() print("Test-modules with import problems:") for test in import_errors: print(" " + test.module) def format_seconds(self, n_seconds): """Format a time in seconds.""" if n_seconds >= 60: n_minutes, n_seconds = divmod(n_seconds, 60) return "%d minutes %.3f seconds" % (n_minutes, n_seconds) else: return "%.3f seconds" % n_seconds def format_seconds_short(self, n_seconds): """Format a time in seconds (short version).""" return "%.3f s" % n_seconds def summary(self, n_tests, n_failures, n_errors, n_seconds, n_skipped=0): """Summarize the results of a single test layer.""" print (" Ran %s tests with %s failures, %s errors and " "%s skipped in %s." % (n_tests, n_failures, n_errors, n_skipped, self.format_seconds(n_seconds))) def totals(self, n_tests, n_failures, n_errors, n_seconds, n_skipped=0): """Summarize the results of all layers.""" print("Total: %s tests, %s failures, %s errors and %s skipped in %s." % (n_tests, n_failures, n_errors, n_skipped, self.format_seconds(n_seconds))) def list_of_tests(self, tests, layer_name): """Report a list of test names.""" print("Listing %s tests:" % layer_name) for test in tests: print(' ', test) def garbage(self, garbage): """Report garbage generated by tests.""" if garbage: print("Tests generated new (%d) garbage:" % len(garbage)) print(garbage) def test_garbage(self, test, garbage): """Report garbage generated by a test.""" if garbage: print("The following test left garbage:") print(test) print(garbage) def test_threads(self, test, new_threads): """Report threads left behind by a test.""" if new_threads: print("The following test left new threads behind:") print(test) print("New thread(s):", new_threads) def refcounts(self, rc, prev): """Report a change in reference counts.""" print(" sys refcount=%-8d change=%-6d" % (rc, rc - prev)) def detailed_refcounts(self, track, rc, prev): """Report a change in reference counts, with extra detail.""" print((" sum detail refcount=%-8d" " sys refcount=%-8d" " change=%-6d" % (track.n, rc, rc - prev))) track.output() def start_set_up(self, layer_name): """Report that we're setting up a layer. The next output operation should be stop_set_up(). """ print(" Set up %s" % layer_name, end=' ') sys.stdout.flush() def stop_set_up(self, seconds): """Report that we've set up a layer. Should be called right after start_set_up(). """ print("in %s." % self.format_seconds(seconds)) def start_tear_down(self, layer_name): """Report that we're tearing down a layer. The next output operation should be stop_tear_down() or tear_down_not_supported(). """ print(" Tear down %s" % layer_name, end=' ') sys.stdout.flush() def stop_tear_down(self, seconds): """Report that we've tore down a layer. Should be called right after start_tear_down(). """ print("in %s." % self.format_seconds(seconds)) def tear_down_not_supported(self): """Report that we could not tear down a layer. Should be called right after start_tear_down(). """ print("... not supported") def start_test(self, test, tests_run, total_tests): """Report that we're about to run a test. The next output operation should be test_success(), test_error(), or test_failure(). """ self.test_width = 0 if self.progress: if self.last_width: sys.stdout.write('\r' + (' ' * self.last_width) + '\r') s = " %d/%d (%.1f%%)" % (tests_run, total_tests, tests_run * 100.0 / total_tests) sys.stdout.write(s) self.test_width += len(s) if self.verbose == 1: room = self.max_width - self.test_width - 1 s = self.getShortDescription(test, room) sys.stdout.write(s) self.test_width += len(s) elif self.verbose == 1: sys.stdout.write('.' * test.countTestCases()) elif self.in_subprocess: sys.stdout.write('.' * test.countTestCases()) # Give the parent process a new line so it sees the progress # in a timely manner. sys.stdout.write('\n') if self.verbose > 1: s = str(test) sys.stdout.write(' ') sys.stdout.write(s) self.test_width += len(s) + 1 sys.stdout.flush() def test_success(self, test, seconds): """Report that a test was successful. Should be called right after start_test(). The next output operation should be stop_test(). """ if self.verbose > 2: s = " (%s)" % self.format_seconds_short(seconds) sys.stdout.write(s) self.test_width += len(s) + 1 def test_skipped(self, test, reason): """Report that a test was skipped. Should be called right after start_test(). The next output operation should be stop_test(). """ if self.verbose > 2: s = " (skipped: %s)" % reason elif self.verbose > 1: s = " (skipped)" else: return sys.stdout.write(s) self.test_width += len(s) + 1 def test_error(self, test, seconds, exc_info): """Report that an error occurred while running a test. Should be called right after start_test(). The next output operation should be stop_test(). """ if self.verbose > 2: print(" (%s)" % self.format_seconds_short(seconds)) print() self.print_traceback("Error in test %s" % test, exc_info) self.test_width = self.last_width = 0 def test_failure(self, test, seconds, exc_info): """Report that a test failed. Should be called right after start_test(). The next output operation should be stop_test(). """ if self.verbose > 2: print(" (%s)" % self.format_seconds_short(seconds)) print() self.print_traceback("Failure in test %s" % test, exc_info) self.test_width = self.last_width = 0 def print_traceback(self, msg, exc_info): """Report an error with a traceback.""" print() print(msg) print(self.format_traceback(exc_info)) def format_traceback(self, exc_info): """Format the traceback.""" v = exc_info[1] if isinstance(v, DocTestFailureException): tb = v.args[0] elif isinstance(v, doctest.DocTestFailure): tb = doctest_template % ( v.test.filename, v.test.lineno + v.example.lineno + 1, v.test.name, v.example.source, v.example.want, v.got, ) else: tb = "".join(traceback.format_exception(*exc_info)) return tb def stop_test(self, test): """Clean up the output state after a test.""" if self.progress: self.last_width = self.test_width elif self.verbose > 1: print() sys.stdout.flush() def stop_tests(self): """Clean up the output state after a collection of tests.""" if self.progress and self.last_width: sys.stdout.write('\r' + (' ' * self.last_width) + '\r') if self.verbose == 1 or self.progress: print() def tigetnum(attr, default=None): """Return a value from the terminfo database. Terminfo is used on Unix-like systems to report various terminal attributes (such as width, height or the number of supported colors). Returns ``default`` when the ``curses`` module is not available, or when sys.stdout is not a terminal. """ try: import curses except ImportError: # avoid reimporting a broken module in python 2.3 sys.modules['curses'] = None else: # If sys.stdout is not a real file object (e.g. in unit tests that # use various wrappers), you get an error, different depending on # Python version: expected_exceptions = (curses.error, TypeError, AttributeError) if sys.version_info >= (3,): import io expected_exceptions += (io.UnsupportedOperation, ) try: curses.setupterm() except expected_exceptions: # You get curses.error when $TERM is set to an unknown name pass else: try: return curses.tigetnum(attr) except expected_exceptions: # You get TypeError on PyPy3 due to a bug: # https://bitbucket.org/pypy/pypy/issue/2016/pypy3-cursestigetnum-raises-ctype pass return default def terminal_has_colors(): """Determine whether the terminal supports colors. Some terminals (e.g. the emacs built-in one) don't. """ return tigetnum('colors', -1) >= 8 class ColorfulOutputFormatter(OutputFormatter): """Output formatter that uses ANSI color codes. Like syntax highlighting in your text editor, colorizing test failures helps the developer. """ # These colors are carefully chosen to have enough contrast # on terminals with both black and white background. colorscheme = {'normal': 'normal', 'default': 'default', 'info': 'normal', 'suboptimal-behaviour': 'magenta', 'error': 'brightred', 'number': 'green', 'slow-test': 'brightmagenta', 'ok-number': 'green', 'error-number': 'brightred', 'filename': 'lightblue', 'lineno': 'lightred', 'testname': 'lightcyan', 'failed-example': 'cyan', 'expected-output': 'green', 'actual-output': 'red', 'character-diffs': 'magenta', 'diff-chunk': 'magenta', 'exception': 'red', 'skipped': 'brightyellow', } # Map prefix character to color in diff output. This handles ndiff and # udiff correctly, but not cdiff. In cdiff we ought to highlight '!' as # expected-output until we see a '-', then highlight '!' as actual-output, # until we see a '*', then switch back to highlighting '!' as # expected-output. Nevertheless, coloried cdiffs are reasonably readable, # so I'm not going to fix this. # -- mgedmin diff_color = {'-': 'expected-output', '+': 'actual-output', '?': 'character-diffs', '@': 'diff-chunk', '*': 'diff-chunk', '!': 'actual-output',} prefixes = [('dark', '0;'), ('light', '1;'), ('bright', '1;'), ('bold', '1;'),] colorcodes = {'default': 0, 'normal': 0, 'black': 30, 'red': 31, 'green': 32, 'brown': 33, 'yellow': 33, 'blue': 34, 'magenta': 35, 'cyan': 36, 'grey': 37, 'gray': 37, 'white': 37} slow_test_threshold = 10.0 # seconds def color_code(self, color): """Convert a color description (e.g. 'lightgray') to a terminal code.""" prefix_code = '' for prefix, code in self.prefixes: if color.startswith(prefix): color = color[len(prefix):] prefix_code = code break color_code = self.colorcodes[color] return '\033[%s%sm' % (prefix_code, color_code) def color(self, what): """Pick a named color from the color scheme""" return self.color_code(self.colorscheme[what]) def colorize(self, what, message, normal='normal'): """Wrap message in color.""" return self.color(what) + message + self.color(normal) def error_count_color(self, n): """Choose a color for the number of errors.""" if n: return self.color('error-number') else: return self.color('ok-number') def skip_count_color(self, n): """Choose a color for the number of skipped tests.""" if n: return self.color('skipped') else: return self.color('ok-number') def test_skipped(self, test, reason): """Report that a test was skipped. Should be called right after start_test(). The next output operation should be stop_test(). """ if self.verbose > 2: s = " (%sskipped: %s%s)" % ( self.color('skipped'), reason, self.color('info')) elif self.verbose > 1: s = " (%sskipped%s)" % ( self.color('skipped'), self.color('info')) else: return sys.stdout.write(s) self.test_width += len(s) + 1 def info(self, message): """Print an informative message.""" print(self.colorize('info', message)) def info_suboptimal(self, message): """Print an informative message about losing some of the features. For example, when you run some tests in a subprocess, you lose the ability to use the debugger. """ print(self.colorize('suboptimal-behaviour', message)) def error(self, message): """Report an error.""" print(self.colorize('error', message)) def error_with_banner(self, message): """Report an error with a big ASCII banner.""" print() print(self.colorize('error', '*'*70)) self.error(message) print(self.colorize('error', '*'*70)) print() def tear_down_not_supported(self): """Report that we could not tear down a layer. Should be called right after start_tear_down(). """ print("...", self.colorize('suboptimal-behaviour', "not supported")) def format_seconds(self, n_seconds, normal='normal'): """Format a time in seconds.""" if n_seconds >= 60: n_minutes, n_seconds = divmod(n_seconds, 60) return "%s minutes %s seconds" % ( self.colorize('number', '%d' % n_minutes, normal), self.colorize('number', '%.3f' % n_seconds, normal)) else: return "%s seconds" % ( self.colorize('number', '%.3f' % n_seconds, normal)) def format_seconds_short(self, n_seconds): """Format a time in seconds (short version).""" if n_seconds >= self.slow_test_threshold: color = 'slow-test' else: color = 'number' return self.colorize(color, "%.3f s" % n_seconds) def summary(self, n_tests, n_failures, n_errors, n_seconds, n_skipped=0): """Summarize the results.""" sys.stdout.writelines([ self.color('info'), ' Ran ', self.color('number'), str(n_tests), self.color('info'), ' tests with ', self.error_count_color(n_failures), str(n_failures), self.color('info'), ' failures, ', self.error_count_color(n_errors), str(n_errors), self.color('info'), ' errors, ', self.skip_count_color(n_skipped), str(n_skipped), self.color('info'), ' skipped in ', self.format_seconds(n_seconds, 'info'), '.', self.color('normal'), '\n', ]) def totals(self, n_tests, n_failures, n_errors, n_seconds, n_skipped=0): """Report totals (number of tests, failures, and errors).""" sys.stdout.writelines([ self.color('info'), 'Total: ', self.color('number'), str(n_tests), self.color('info'), ' tests, ', self.error_count_color(n_failures), str(n_failures), self.color('info'), ' failures, ', self.error_count_color(n_errors), str(n_errors), self.color('info'), ' errors, ', self.skip_count_color(n_skipped), str(n_skipped), self.color('info'), ' skipped in ', self.format_seconds(n_seconds, 'info'), '.', self.color('normal'), '\n']) def print_traceback(self, msg, exc_info): """Report an error with a traceback.""" print() print(self.colorize('error', msg)) v = exc_info[1] if isinstance(v, DocTestFailureException): self.print_doctest_failure(v.args[0]) elif isinstance(v, doctest.DocTestFailure): # I don't think these are ever used... -- mgedmin tb = self.format_traceback(exc_info) print(tb) else: tb = self.format_traceback(exc_info) self.print_colorized_traceback(tb) def print_doctest_failure(self, formatted_failure): """Report a doctest failure. ``formatted_failure`` is a string -- that's what DocTestSuite/DocFileSuite gives us. """ color_of_indented_text = 'normal' colorize_diff = False for line in formatted_failure.splitlines(): if line.startswith('File '): m = re.match(r'File "(.*)", line (\d*), in (.*)$', line) if m: filename, lineno, test = m.groups() sys.stdout.writelines([ self.color('normal'), 'File "', self.color('filename'), filename, self.color('normal'), '", line ', self.color('lineno'), lineno, self.color('normal'), ', in ', self.color('testname'), test, self.color('normal'), '\n']) else: print(line) elif line.startswith(' ') or line.strip() == '': if colorize_diff and len(line) > 4: color = self.diff_color.get(line[4], color_of_indented_text) print(self.colorize(color, line)) else: if line.strip() != '': print(self.colorize(color_of_indented_text, line)) else: print(line) else: colorize_diff = False if line.startswith('Failed example'): color_of_indented_text = 'failed-example' elif line.startswith('Expected:'): color_of_indented_text = 'expected-output' elif line.startswith('Got:'): color_of_indented_text = 'actual-output' elif line.startswith('Exception raised:'): color_of_indented_text = 'exception' elif line.startswith('Differences '): color_of_indented_text = 'normal' colorize_diff = True else: color_of_indented_text = 'normal' print(line) print() def print_colorized_traceback(self, formatted_traceback): """Report a test failure. ``formatted_traceback`` is a string. """ for line in formatted_traceback.splitlines(): if line.startswith(' File'): m = re.match(r' File "(.*)", line (\d*), in (.*)$', line) if m: filename, lineno, test = m.groups() sys.stdout.writelines([ self.color('normal'), ' File "', self.color('filename'), filename, self.color('normal'), '", line ', self.color('lineno'), lineno, self.color('normal'), ', in ', self.color('testname'), test, self.color('normal'), '\n']) else: print(line) elif line.startswith(' '): print(self.colorize('failed-example', line)) elif line.startswith('Traceback (most recent call last)'): print(line) else: print(self.colorize('exception', line)) print() class FakeTest(object): """A fake test object that only has an id.""" failureException = None def __init__(self, test_id): self._id = test_id def id(self): return self._id # Conditional imports, we don't want zope.testrunner to have a hard dependency on # subunit. try: import subunit from subunit.iso8601 import Utc except ImportError: subunit = None # testtools is a hard dependency of subunit itself, guarding separately for # richer error messages. try: from testtools import content except ImportError: content = None class SubunitOutputFormatter(object): """A subunit output formatter. This output formatter generates subunit compatible output (see https://launchpad.net/subunit). Subunit output is essentially a stream of results of unit tests. In this formatter, non-test events (such as layer set up) are encoded as specially tagged tests and summary-generating methods (such as modules_with_import_problems) deliberately do nothing. In particular, for a layer 'foo', the fake tests related to layer set up and tear down are tagged with 'zope:layer' and are called 'foo:setUp' and 'foo:tearDown'. Any tests within layer 'foo' are tagged with 'zope:layer:foo'. Note that all tags specific to this formatter begin with 'zope:' """ # subunit output is designed for computers, so displaying a progress bar # isn't helpful. progress = False verbose = property(lambda self: self.options.verbose) TAG_INFO_SUBOPTIMAL = 'zope:info_suboptimal' TAG_ERROR_WITH_BANNER = 'zope:error_with_banner' TAG_LAYER = 'zope:layer' TAG_IMPORT_ERROR = 'zope:import_error' TAG_PROFILER_STATS = 'zope:profiler_stats' TAG_GARBAGE = 'zope:garbage' TAG_THREADS = 'zope:threads' TAG_REFCOUNTS = 'zope:refcounts' def __init__(self, options, stream=None): if subunit is None: raise Exception("Requires subunit 0.0.5 or better") if content is None: raise Exception("Requires testtools 0.9.2 or better") self.options = options if stream is None: stream = sys.stdout self._stream = stream self._subunit = subunit.TestProtocolClient(self._stream) # Used to track the last layer that was set up or torn down. Either # None or (layer_name, last_touched_time). self._last_layer = None self.UTC = Utc() # Content types used in the output. self.TRACEBACK_CONTENT_TYPE = content.ContentType( 'text', 'x-traceback', dict(language='python', charset='utf8')) self.PROFILE_CONTENT_TYPE = content.ContentType( 'application', 'x-binary-profile') self.PLAIN_TEXT = content.ContentType( 'text', 'plain', {'charset': 'utf8'}) def _emit_timestamp(self, now=None): """Emit a timestamp to the subunit stream. If 'now' is not specified, use the current time on the system clock. """ if now is None: now = datetime.now(self.UTC) self._subunit.time(now) return now def _emit_tag(self, tag): self._stream.write('tags: %s\n' % (tag,)) def _stop_tag(self, tag): self._stream.write('tags: -%s\n' % (tag,)) def _emit_fake_test(self, message, tag, details=None): """Emit a successful fake test to the subunit stream. Use this to print tagged informative messages. """ test = FakeTest(message) self._subunit.startTest(test) self._emit_tag(tag) self._subunit.addSuccess(test, details=details) def _emit_error(self, error_id, tag, exc_info): """Emit an error to the subunit stream. Use this to pass on information about errors that occur outside of tests. """ test = FakeTest(error_id) self._subunit.startTest(test) self._emit_tag(tag) self._subunit.addError(test, exc_info) def info(self, message): """Print an informative message, but only if verbose.""" # info() output is not relevant to actual test results. It only says # things like "Running tests" or "Tearing down left over layers", # things that are communicated already by the subunit stream. Just # suppress the info() output. pass def info_suboptimal(self, message): """Print an informative message about losing some of the features. For example, when you run some tests in a subprocess, you lose the ability to use the debugger. """ # Used _only_ to indicate running in a subprocess. self._emit_fake_test(message.strip(), self.TAG_INFO_SUBOPTIMAL) def error(self, message): """Report an error.""" # XXX: Mostly used for user errors, sometimes used for errors in the # test framework, sometimes used to record layer setUp failure (!!!). self._stream.write('%s\n' % (message,)) def error_with_banner(self, message): """Report an error with a big ASCII banner.""" # Either "Could not communicate with subprocess" # Or "Can't post-mortem debug when running a layer as a subprocess!" self._emit_fake_test(message, self.TAG_ERROR_WITH_BANNER) def _enter_layer(self, layer_name): """Signal in the subunit stream that we are entering a layer.""" self._emit_tag('zope:layer:%s' % (layer_name,)) def _exit_layer(self, layer_name): """Signal in the subunit stream that we are leaving a layer.""" self._stop_tag('zope:layer:%s' % (layer_name,)) def start_set_up(self, layer_name): """Report that we're setting up a layer. We do this by emitting a tag of the form 'layer:$LAYER_NAME'. """ now = self._emit_timestamp() self._subunit.startTest(FakeTest('%s:setUp' % (layer_name,))) self._emit_tag(self.TAG_LAYER) self._last_layer = (layer_name, now) def stop_set_up(self, seconds): layer_name, start_time = self._last_layer self._last_layer = None self._emit_timestamp(start_time + timedelta(seconds=seconds)) self._subunit.addSuccess(FakeTest('%s:setUp' % (layer_name,))) self._enter_layer(layer_name) def start_tear_down(self, layer_name): """Report that we're tearing down a layer. We do this by removing a tag of the form 'layer:$LAYER_NAME'. """ self._exit_layer(layer_name) now = self._emit_timestamp() self._subunit.startTest(FakeTest('%s:tearDown' % (layer_name,))) self._emit_tag(self.TAG_LAYER) self._last_layer = (layer_name, now) def stop_tear_down(self, seconds): layer_name, start_time = self._last_layer self._last_layer = None self._emit_timestamp(start_time + timedelta(seconds=seconds)) self._subunit.addSuccess(FakeTest('%s:tearDown' % (layer_name,))) def tear_down_not_supported(self): """Report that we could not tear down a layer. Should be called right after start_tear_down(). """ layer_name, start_time = self._last_layer self._last_layer = None self._emit_timestamp(datetime.now(self.UTC)) self._subunit.addSkip( FakeTest('%s:tearDown' % (layer_name,)), "tearDown not supported") def summary(self, n_tests, n_failures, n_errors, n_seconds, n_skipped=0): """Print out a summary. Since subunit is a stream protocol format, it has no need for a summary. When the stream is finished other tools can generate a summary if so desired. """ pass def start_test(self, test, tests_run, total_tests): """Report that we're about to run a test. The next output operation should be test_success(), test_error(), or test_failure(). """ self._emit_timestamp() # Note that this always emits newlines, so it will function as well as # other start_test implementations if we are running in a subprocess. self._subunit.startTest(test) def stop_test(self, test): """Clean up the output state after a test.""" self._subunit.stopTest(test) self._stream.flush() def stop_tests(self): """Clean up the output state after a collection of tests.""" self._stream.flush() def test_success(self, test, seconds): self._emit_timestamp() self._subunit.addSuccess(test) def test_skipped(self, test, reason): self._subunit.addSkip(test, reason) def import_errors(self, import_errors): """Report test-module import errors (if any).""" if not import_errors: return for error in import_errors: self._emit_error( error.module, self.TAG_IMPORT_ERROR, error.exc_info) def modules_with_import_problems(self, import_errors): """Report names of modules with import problems (if any).""" # This is simply a summary method, and subunit output doesn't benefit # from summaries. pass def _exc_info_to_details(self, exc_info): """Translate 'exc_info' into a details dictionary usable with subunit. """ # In an ideal world, we'd use the pre-bundled 'TracebackContent' class # from testtools. However, 'OutputFormatter' contains special logic to # handle errors from doctests, so we have to use that and manually # create an object equivalent to an instance of 'TracebackContent'. formatter = OutputFormatter(None) traceback = formatter.format_traceback(exc_info) # We have no idea if the traceback is a unicode object or a bytestring # with non-ASCII characters. We had best be careful when handling it. if isinstance(traceback, unicode): unicode_tb = traceback else: # Assume the traceback was utf-8 encoded, but still be careful. unicode_tb = traceback.decode('utf8', 'replace') return { 'traceback': content.Content( self.TRACEBACK_CONTENT_TYPE, lambda: [unicode_tb.encode('utf8')])} def test_error(self, test, seconds, exc_info): """Report that an error occurred while running a test. Should be called right after start_test(). The next output operation should be stop_test(). """ self._emit_timestamp() details = self._exc_info_to_details(exc_info) self._subunit.addError(test, details=details) def test_failure(self, test, seconds, exc_info): """Report that a test failed. Should be called right after start_test(). The next output operation should be stop_test(). """ self._emit_timestamp() details = self._exc_info_to_details(exc_info) self._subunit.addFailure(test, details=details) def profiler_stats(self, stats): """Report profiler stats.""" fd, filename = tempfile.mkstemp() os.close(fd) try: stats.dump_stats(filename) stats_dump = open(filename) try: profile_content = content.Content( self.PROFILE_CONTENT_TYPE, stats_dump.read) details = {'profiler-stats': profile_content} # Name the test 'zope:profiler_stats' just like its tag. self._emit_fake_test( self.TAG_PROFILER_STATS, self.TAG_PROFILER_STATS, details) finally: stats_dump.close() finally: os.unlink(filename) def tests_with_errors(self, errors): """Report tests with errors. Simply not supported by the subunit formatter. Fancy summary output doesn't make sense. """ pass def tests_with_failures(self, failures): """Report tests with failures. Simply not supported by the subunit formatter. Fancy summary output doesn't make sense. """ pass def totals(self, n_tests, n_failures, n_errors, n_seconds, n_skipped=0): """Summarize the results of all layers. Simply not supported by the subunit formatter. Fancy summary output doesn't make sense. """ pass def list_of_tests(self, tests, layer_name): """Report a list of test names.""" self._enter_layer(layer_name) for test in tests: self._subunit.startTest(test) self._subunit.addSuccess(test) self._exit_layer(layer_name) def _get_text_details(self, name, text): """Get a details dictionary that just has some plain text.""" return { name: content.Content( self.PLAIN_TEXT, lambda: [text.encode('utf8')])} def garbage(self, garbage): """Report garbage generated by tests.""" # XXX: Really, 'garbage', 'profiler_stats' and the 'refcounts' twins # ought to add extra details to a fake test that represents the # summary information for the whole suite. However, there's no event # on output formatters for "everything is really finished, honest". -- # jml, 2010-02-14 details = self._get_text_details('garbage', unicode(garbage)) self._emit_fake_test( self.TAG_GARBAGE, self.TAG_GARBAGE, details) def test_garbage(self, test, garbage): """Report garbage generated by a test. Encoded in the subunit stream as a test error. Clients can filter out these tests based on the tag if they don't think garbage should fail the test run. """ # XXX: Perhaps 'test_garbage' and 'test_threads' ought to be within # the output for the actual test, appended as details to whatever # result the test gets. Not an option with the present API, as there's # no event for "no more output for this test". -- jml, 2010-02-14 self._subunit.startTest(test) self._emit_tag(self.TAG_GARBAGE) self._subunit.addError( test, details=self._get_text_details('garbage', unicode(garbage))) def test_threads(self, test, new_threads): """Report threads left behind by a test. Encoded in the subunit stream as a test error. Clients can filter out these tests based on the tag if they don't think left-over threads should fail the test run. """ self._subunit.startTest(test) self._emit_tag(self.TAG_THREADS) self._subunit.addError( test, details=self._get_text_details('garbage', unicode(new_threads))) def refcounts(self, rc, prev): """Report a change in reference counts.""" details = self._get_text_details('sys-refcounts', str(rc)) details.update( self._get_text_details('changes', str(rc - prev))) # XXX: Emit the details dict as JSON? self._emit_fake_test( self.TAG_REFCOUNTS, self.TAG_REFCOUNTS, details) def detailed_refcounts(self, track, rc, prev): """Report a change in reference counts, with extra detail.""" details = self._get_text_details('sys-refcounts', str(rc)) details.update( self._get_text_details('changes', str(rc - prev))) details.update( self._get_text_details('track', str(track.delta))) self._emit_fake_test( self.TAG_REFCOUNTS, self.TAG_REFCOUNTS, details) zope.testrunner-4.4.9/src/zope/testrunner/logsupport.py0000664000000000000000000000373512527371226022201 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Logging support. This code is pretty much untested and was only mechanically refactored. The module name is not 'logging' because of a name collision with Python's logging module. """ import logging import logging.config import os.path import zope.testrunner.feature class Logging(zope.testrunner.feature.Feature): active = True def global_setup(self): # Get the log.ini file from the current directory instead of # possibly buried in the build directory. TODO: This isn't # perfect because if log.ini specifies a log file, it'll be # relative to the build directory. Hmm... logini = # os.path.abspath("log.ini") logini = os.path.abspath("log.ini") if os.path.exists(logini): logging.config.fileConfig(logini) else: # If there's no log.ini, cause the logging package to be # silent during testing. root = logging.getLogger() root.addHandler(NullHandler()) logging.basicConfig() if "LOGGING" in os.environ: level = int(os.environ["LOGGING"]) logging.getLogger().setLevel(level) class NullHandler(logging.Handler): """Logging handler that drops everything on the floor. We require silence in the test environment. Hush. """ def emit(self, record): pass zope.testrunner-4.4.9/src/zope/testrunner/profiling.py0000664000000000000000000001145012527371226021745 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Profiler support for the test runner """ import os import glob import sys import tempfile import zope.testrunner.feature available_profilers = {} try: import cProfile import pstats except ImportError: pass else: class CProfiler(object): """cProfiler""" def __init__(self, filepath): self.filepath = filepath self.profiler = cProfile.Profile() self.enable = self.profiler.enable self.disable = self.profiler.disable def finish(self): self.profiler.dump_stats(self.filepath) def loadStats(self, prof_glob): stats = None for file_name in glob.glob(prof_glob): if stats is None: stats = pstats.Stats(file_name) else: stats.add(file_name) return stats available_profilers['cProfile'] = CProfiler # some Linux distributions don't include the profiler, which hotshot uses if not sys.hexversion >= 0x02060000: # Hotshot is not maintained any longer in 2.6. It does not support # merging to hotshot files. Thus we won't use it in python2.6 and # onwards try: import hotshot import hotshot.stats except ImportError: pass else: class HotshotProfiler(object): """hotshot interface""" def __init__(self, filepath): self.profiler = hotshot.Profile(filepath) self.enable = self.profiler.start self.disable = self.profiler.stop def finish(self): self.profiler.close() def loadStats(self, prof_glob): stats = None for file_name in glob.glob(prof_glob): loaded = hotshot.stats.load(file_name) if stats is None: stats = loaded else: stats.add(loaded) return stats available_profilers['hotshot'] = HotshotProfiler class Profiling(zope.testrunner.feature.Feature): def __init__(self, runner): super(Profiling, self).__init__(runner) if (self.runner.options.profile and sys.version_info[:3] <= (2,4,1) and __debug__): self.runner.options.output.error( 'Because of a bug in Python < 2.4.1, profiling ' 'during tests requires the -O option be passed to ' 'Python (not the test runner).') sys.exit() self.active = bool(self.runner.options.profile) self.profiler = self.runner.options.profile def global_setup(self): self.prof_prefix = 'tests_profile.' self.prof_suffix = '.prof' self.prof_glob = os.path.join(self.runner.options.prof_dir, self.prof_prefix + '*' + self.prof_suffix) # if we are going to be profiling, and this isn't a subprocess, # clean up any stale results files if not self.runner.options.resume_layer: for file_name in glob.glob(self.prof_glob): os.unlink(file_name) # set up the output file self.oshandle, self.file_path = tempfile.mkstemp( self.prof_suffix, self.prof_prefix, self.runner.options.prof_dir) self.profiler = available_profilers[self.runner.options.profile](self.file_path) # Need to do this rebinding to support the stack-frame annoyance with # hotshot. self.late_setup = self.profiler.enable self.early_teardown = self.profiler.disable def global_teardown(self): self.profiler.finish() # We must explicitly close the handle mkstemp returned, else on # Windows this dies the next time around just above due to an # attempt to unlink a still-open file. os.close(self.oshandle) if not self.runner.options.resume_layer: self.profiler_stats = self.profiler.loadStats(self.prof_glob) self.profiler_stats.sort_stats('cumulative', 'calls') def report(self): if not self.runner.options.resume_layer: self.runner.options.output.profiler_stats(self.profiler_stats) zope.testrunner-4.4.9/src/zope/testrunner/selftest.py0000664000000000000000000000176312527371226021613 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Self-test support. Provides setup routines that enable the test runner to test itself. """ import pdb import zope.testrunner.feature real_pdb_set_trace = pdb.set_trace class SelfTest(zope.testrunner.feature.Feature): active = True def global_setup(self): # Make sure we start with real pdb.set_trace. pdb.set_trace = real_pdb_set_trace zope.testrunner-4.4.9/src/zope/testrunner/debug.py0000664000000000000000000000376212527371226021051 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Debug functions """ from __future__ import print_function import doctest import sys import pdb import traceback import zope.testrunner.interfaces def post_mortem(exc_info): err = exc_info[1] if isinstance(err, (doctest.UnexpectedException, doctest.DocTestFailure)): if isinstance(err, doctest.UnexpectedException): exc_info = err.exc_info # Print out location info if the error was in a doctest if exc_info[2].tb_frame.f_code.co_filename == '': print_doctest_location(err) else: print_doctest_location(err) # Hm, we have a DocTestFailure exception. We need to # generate our own traceback try: exec(('raise ValueError' '("Expected and actual output are different")' ), err.test.globs) except: exc_info = sys.exc_info() print(''.join(traceback.format_exception_only(exc_info[0], exc_info[1]))) pdb.post_mortem(exc_info[2]) raise zope.testrunner.interfaces.EndRun() def print_doctest_location(err): # This mimics pdb's output, which gives way cool results in emacs :) filename = err.test.filename if filename.endswith('.pyc'): filename = filename[:-1] print("> %s(%s)_()" % (filename, err.test.lineno+err.example.lineno+1)) zope.testrunner-4.4.9/src/zope/testrunner/exceptions.py0000664000000000000000000000142012527371226022131 0ustar rootroot############################################################################## # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Exceptions for zope.testrunner """ class DocTestFailureException(AssertionError): """Use custom exception for doctest unit test failures""" zope.testrunner-4.4.9/src/zope/testrunner/layer.py0000664000000000000000000000173412527371226021074 0ustar rootroot############################################################################## # # Copyright (c) 2004-2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Layer definitions """ import unittest class UnitTests(object): """A layer for gathering all unit tests.""" class EmptyLayer(object): """An empty layer to start spreading out subprocesses.""" __bases__ = () __module__ = '' def EmptySuite(): suite = unittest.TestSuite() suite.layer = EmptyLayer() return suite zope.testrunner-4.4.9/src/zope/testrunner/eggsupport.py0000664000000000000000000000751512527371226022162 0ustar rootroot""" Add unit and functional testing support to setuptools-driven eggs. """ from __future__ import print_function from setuptools.command.test import ScanningLoader from setuptools.command.test import test as BaseCommand def skipLayers(suite, _result=None): """ Walk the suite returned by setuptools' testloader. o Skip any tests which have a 'layer' defined. """ from unittest import TestSuite if _result is None: _result = TestSuite() # Sometimes test suites do not have tests, like DocFileSuite. if len(suite._tests) == 0: _result.addTest(suite) return _result for test in suite._tests: layer = getattr(test, 'layer', None) if layer is not None: continue if isinstance(test, TestSuite): skipLayers(test, _result) else: _result.addTest(test) return _result class SkipLayers(ScanningLoader): """ Load only unit tests (those which have no layer associated with them). o Running the tests using 'setup.py test' cannot, by default, drive the full testrunner, with its support for layers (in functional tests). This loader allows the command to work, by running only those tests which don't need the layer support. o To run layer-dependent tests, use 'setup.py ftest' (see below for adding the command to your setup.py). o To use this loader your package add the following your 'setup()' call:: setup( ... setup_requires=['eggtestinfo' # captures testing metadata in EGG-INFO ], tests_require=['zope.testrunner', ], ... test_loader='zope.testrunner.eggsupport:SkipLayers', ... ) """ def loadTestsFromModule(self, module): return skipLayers( ScanningLoader.loadTestsFromModule(self, module)) def loadTestsFromNames(self, testNames, module): return skipLayers( ScanningLoader.loadTestsFromNames(self, testNames, module)) def print_usage(): print('python setup.py ftest') print() print(ftest.__doc__) class ftest(BaseCommand): """ Run unit and functional tests after an in-place build. o Note that this command runs *all* tests (unit *and* functional). o This command does not provide any of the configuration options which the usual testrunner provided by 'zope.testrunner' offers: it is intended to allow easy verification that a package has been installed correctly via setuptools, but is not likely to be useful for developers working on the package. o Developers working on the package will likely prefer to work with the stock testrunner, e.g., by using buildout with a recipe which configures the testrunner as a standalone script. o To use this in your pacakge add the following to the 'entry_points' section:: setup( ... setup_requires=['zope.testrunner', 'eggtestinfo' # captures testing metadata in EGG-INFO ], ... ) """ description = "Run all functional and unit tests after in-place build" user_options = [] help_options = [('usage', '?', 'Show usage', print_usage)] def initialize_options(self): pass # suppress normal handling def finalize_options(self): pass # suppress normal handling def run(self): from zope.testrunner import run args = ['IGNORE_ME'] dist = self.distribution for src_loc in (dist.package_dir.values() or ['.']): args += ['--test-path', src_loc] if dist.install_requires: dist.fetch_build_eggs(dist.install_requires) if dist.tests_require: dist.fetch_build_eggs(dist.tests_require) def _run(): run(args=args) self.with_project_on_sys_path(_run) zope.testrunner-4.4.9/DEVELOPING.rst0000664000000000000000000000572412527371226015622 0ustar rootroot************************** developing zope.testrunner ************************** Zope testrunner needs itself to run its own tests. There are two ways to do that. Using zc.buildout ----------------- The standard way to set up a testrunner to test zope.testrunner with, is to use buildout:: $ python bootstrap.py $ bin/buildout You can now run the tests:: $ bin/test -pvc Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in 0.000 seconds. Ran 28 tests with 0 failures and 0 errors in 17.384 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds. Using setup.py -------------- You may run the tests without buildout as well, as the setup.py has a custom test command that will run the tests:: $ python setup.py test running test running egg_info writing requirements to src/zope.testrunner.egg-info/requires.txt writing src/zope.testrunner.egg-info/PKG-INFO writing namespace_packages to src/zope.testrunner.egg-info/namespace_packages.txt writing top-level names to src/zope.testrunner.egg-info/top_level.txt writing dependency_links to src/zope.testrunner.egg-info/dependency_links.txt writing entry points to src/zope.testrunner.egg-info/entry_points.txt reading manifest template 'MANIFEST.in' warning: no files found matching 'sampletests' under directory 'src' writing manifest file 'src/zope.testrunner.egg-info/SOURCES.txt' running build_ext Running zope.testrunner.layer.UnitTests tests: Set up zope.testrunner.layer.UnitTests in 0.000 seconds. Ran 27 tests with 0 failures and 0 errors in 17.600 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds. Using tox/detox --------------- This is a convenient way to run the test suite for all supported Python versions, either sequentially (`tox`_) or in parallel (`detox`_):: $ detox GLOB sdist-make: /home/mg/src/zope.testrunner/setup.py py33 sdist-reinst: .../.tox/dist/zope.testrunner-4.1.2.dev0.zip py27 sdist-reinst: .../.tox/dist/zope.testrunner-4.1.2.dev0.zip pypy sdist-reinst: .../.tox/dist/zope.testrunner-4.1.2.dev0.zip py31 sdist-reinst: .../.tox/dist/zope.testrunner-4.1.2.dev0.zip py32 sdist-reinst: .../.tox/dist/zope.testrunner-4.1.2.dev0.zip py26 sdist-reinst: .../.tox/dist/zope.testrunner-4.1.2.dev0.zip py27 runtests: commands[0] py26 runtests: commands[0] pypy runtests: commands[0] py32 runtests: commands[0] py33 runtests: commands[0] py31 runtests: commands[0] ______________________________ summary _______________________________ py26: commands succeeded py27: commands succeeded py31: commands succeeded py32: commands succeeded py33: commands succeeded pypy: commands succeeded congratulations :) .. _tox: http://pypi.python.org/pypi/tox .. _detox: http://pypi.python.org/pypi/detox zope.testrunner-4.4.9/COPYRIGHT.rst0000664000000000000000000000004012527371226015520 0ustar rootrootZope Foundation and Contributorszope.testrunner-4.4.9/setup.py0000664000000000000000000001364012527371226015142 0ustar rootroot############################################################################## # # Copyright (c) 2004, 2013 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## # This package is developed by the Zope Toolkit project, documented here: # http://docs.zope.org/zopetoolkit # When developing and releasing this package, please follow the documented # Zope Toolkit policies as described by this documentation. ############################################################################## """Setup for zope.testrunner package """ import os import sys from setuptools import setup from setuptools.command.test import test if sys.version_info < (2,6) or sys.version_info[:2] == (3,0): raise ValueError("zope.testrunner requires Python 2.6 or higher, " "or 3.1 or higher.") if sys.version_info >= (3,): tests_require = ['zope.testing'] extra = dict(# XXX: python-subunit is not yet ported to Python3. extras_require = {'test': ['zope.testing']}, ) else: tests_require = ['zope.testing', 'python-subunit'] if sys.version_info[0:2] == (2, 6): tests_require.append('unittest2') extra = dict(tests_require = tests_require, extras_require = {'test': tests_require}) class custom_test(test): # The zope.testrunner tests MUST be run using its own testrunner. This is # because its subprocess testing will call the script it was run with. We # therefore create a script to run the testrunner, and call that. def run(self): if self.distribution.install_requires: self.distribution.fetch_build_eggs(self.distribution.install_requires) if self.distribution.tests_require: self.distribution.fetch_build_eggs(self.distribution.tests_require) self.with_project_on_sys_path(self.run_tests) def run_tests(self): template = """ import sys sys.path = %s import os os.chdir('%s') import zope.testrunner if __name__ == '__main__': zope.testrunner.run([ '--test-path', '%s', '-c' ]) """ import tempfile fd, filename = tempfile.mkstemp(prefix='temprunner', text=True) scriptfile = open(filename, 'w') script = template % (sys.path, os.path.abspath(os.curdir), os.path.abspath('src')) scriptfile.write(script) scriptfile.close() import subprocess process = subprocess.Popen([sys.executable, filename]) rc = process.wait() os.unlink(filename) sys.exit(rc) chapters = '\n'.join([ open(os.path.join('src', 'zope', 'testrunner', 'tests', name)).read() for name in ( 'testrunner.txt', 'testrunner-simple.txt', 'testrunner-layers-api.txt', 'testrunner-layers.txt', 'testrunner-arguments.txt', 'testrunner-verbose.txt', 'testrunner-test-selection.txt', 'testrunner-progress.txt', 'testrunner-debugging.txt', 'testrunner-layers-ntd.txt', 'testrunner-eggsupport.txt', 'testrunner-coverage.txt', 'testrunner-profiling.txt', 'testrunner-wo-source.txt', 'testrunner-repeat.txt', 'testrunner-gc.txt', 'testrunner-leaks.txt', 'testrunner-knit.txt', 'testrunner-edge-cases.txt', # The following seems to cause weird unicode in the output: :( 'testrunner-errors.txt', )]) long_description=( open('README.rst').read() + '\n' + open('CHANGES.rst').read() + '\n' + 'Detailed Documentation\n' '**********************\n' + '\n' + chapters ) setup( name='zope.testrunner', version='4.4.9', url='http://pypi.python.org/pypi/zope.testrunner', license='ZPL 2.1', description='Zope testrunner script.', long_description=long_description, author='Zope Foundation and Contributors', author_email='zope-dev@zope.org', packages=["zope", "zope.testrunner", "zope.testrunner.tests.testrunner-ex"], package_dir = {'': 'src'}, classifiers=[ "Development Status :: 5 - Production/Stable", "Environment :: Console", "Framework :: Zope3", "Intended Audience :: Developers", "License :: OSI Approved :: Zope Public License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.2", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: 3.4", 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python :: Implementation :: PyPy', "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: Software Development :: Testing", ], namespace_packages=['zope',], tests_require = tests_require, extras_require = {'test': tests_require}, install_requires = ['setuptools', 'six', 'zope.exceptions', 'zope.interface', ], entry_points = { 'console_scripts': ['zope-testrunner = zope.testrunner:run',], 'distutils.commands': [ 'ftest = zope.testrunner.eggsupport:ftest',], }, include_package_data = True, zip_safe = False, cmdclass = {'test': custom_test}, ) zope.testrunner-4.4.9/tox.ini0000664000000000000000000000051212527371226014735 0ustar rootroot[tox] envlist = py26,py27,pypy,py32,py33,py34,pypy3 [testenv] deps = zope.testing commands = python setup.py test -q [testenv:py26] deps = {[testenv]deps} python-subunit unittest2 [testenv:py27] deps = {[testenv]deps} python-subunit [testenv:pypy] deps = {[testenv]deps} python-subunit zope.testrunner-4.4.9/CHANGES.rst0000664000000000000000000001566312527371226015241 0ustar rootrootzope.testrunner Changelog ************************* 4.4.9 (2015-05-21) ================== - When using ``-j``, parallelize all the tests, including the first test layer (`#28 `_). 4.4.8 (2015-05-01) ================== - Support skipped tests in subunit output (`#25 `_). - More efficient test filtering (`#26 `_). 4.4.7 (2015-04-02) ================== - Work around a bug in PyPy3's curses module (`#24 `_). 4.4.6 (2015-01-21) ================== - Restore support for instance-based test layers that regressed in 4.4.5 (`#20 `_). 4.4.5 (2015-01-06) ================== - Sort related layers close to each other to reduce the number of unnecessary teardowns (fixes `#14 `_). - Run the unit test layer first (fixes `LP #497871 `__). 4.4.4 (2014-12-27) ================== - When looking for the right location of test code, start with longest location paths first. This fixes problems with nested code locations. 4.4.3 (2014-03-19) ================== - Added support for Python 3.4. 4.4.2 (2014-02-22) ================== - Drop support for Python 3.1. - Fix post-mortem debugging when a non-printable exception happens (https://github.com/zopefoundation/zope.testrunner/issues/8). 4.4.1 (2013-07-10) ================== - Updated ``boostrap.py`` to version 2.2. - Fix nondeterministic test failures on Python 3.3 - Tear down layers after ``post_mortem`` debugging is finished. - Fix tests that write to source directory, it might be read-only. 4.4.0 (2013-06-06) ================== - Fix tests selection when the negative "!" pattern is used several times (LP #1160965) - Moved tests into a 'tests' subpackage. - Made ``python -m zope.testrunner`` work again. - Support 'skip' feature of unittest2 (which became the new unittest in Python 2.7). - Better diagnostics when communication with subprocess fails (https://github.com/zopefoundation/zope.testrunner/issues/5). - Do not break subprocess execution when the test suite changes the working directory (https://github.com/zopefoundation/zope.testrunner/issues/6). - Count test module import errors as errors (LP #1026576). 4.3.3 (2013-03-03) ================== - Running layers in sub-processes did not use to work when run via ``python setup.py ftest`` since it tried to run setup.py with all the command line options. It now detects ``setup.py`` runs and we run the test runner directly. 4.3.2 (2013-03-03) ================== - Fix ``SkipLayers`` class in cases where the distribution specifies a ``test_suite`` value. 4.3.1 (2013-03-02) ================== - Fixed a bug in the `ftest` command and added a test. - Fixed a trivial test failure with Python 3 of the previous release. 4.3.0 (2013-03-02) ================== - Expose `ftest` distutils command via an entry point. - Added tests for ``zope.testrunner.eggsupport``. 4.2.0 (2013-02-12) ================== - Dropped use of 2to3, rewrote source code to be compatible with all Python versions. Introduced a dependency on `six`_. 4.1.1 (2013-02-08) ================== - Dropped use of zope.fixers (LP: #1118877). - Fixed tox test error reporting; fixed tests on Pythons 2.6, 3.1, 3.2, 3.3 and PyPy 1.9. - Fix --shuffle ordering on Python 3.2 to be the same as it was on older Python versions. - Fix --shuffle nondeterminism when multiple test layers are present. Note: this will likely change the order of tests for the same --shuffle-seed. - New option: --profile-directory. Use it in the test suite so that tests executed by detox in parallel don't conflict. - Use a temporary coverage directory in the test suite so that tests executed by detox in parallel don't conflict. - Fix --post-mortem (aka -D, --pdb) when a test module cannot be imported or is invalid (LP #1119363). 4.1.0 (2013-02-07) ================== - Replaced deprecated ``zope.interface.implements`` usage with equivalent ``zope.interface.implementer`` decorator. - Dropped support for Python 2.4 and 2.5. - Made StartUpFailure compatible with unittest.TextTestRunner() (LP #1118344). 4.0.4 (2011-10-25) ================== - Work around sporadic timing-related issues in the subprocess buffering tests. Thanks to Jonathan Ballet for the patch! 4.0.3 (2011-03-17) ================== - Added back support for Python <= 2.6 which was broken in 4.0.2. 4.0.2 (2011-03-16) ================== - Added back Python 3 support which was broken in 4.0.1. - Fixed `Unexpected success`_ support by implementing the whole concept. - Added support for the new __pycache__ directories in Python 3.2. 4.0.1 (2011-02-21) ================== - LP #719369: An `Unexpected success`_ (concept introduced in Python 2.7) is no longer handled as success but as failure. This is a workaround. The whole unexpected success concept might be implemented later. .. _`Unexpected success`: http://www.voidspace.org.uk/python/articles/unittest2.shtml#more-skipping 4.0.0 (2010-10-19) ================== - Show more information about layers whose setup fails (LP #638153). 4.0.0b5 (2010-07-20) ==================== - Update fix for LP #221151 to a spelling compatible with Python 2.4. - Timestamps are now always included in subunit output (r114849). - LP #591309: fix a crash when subunit reports test failures containing UTF8-encoded data. 4.0.0b4 (2010-06-23) ==================== - Package as a zipfile to work around Python 2.4 distutils bug (no feature changes or bugfixes in ``zope.testrunner`` itself). 4.0.0b3 (2010-06-16) ==================== - LP #221151: keep ``unittest.TestCase.shortDescription`` happy by supplying a ``_testMethodDoc`` attribute. - LP #595052: keep the distribution installable under Python 2.4: its distutils appears to munge the empty ``__init__.py`` file in the ``foo.bar`` egg used for testing into a directory. - LP #580083: fix the ``bin/test`` script to run only tests from ``zope.testrunner``. - LP #579019: When layers were run in parallel, their tearDown was not called. Additionally, the first layer which was run in the main thread did not have its tearDown called either. 4.0.0b2 (2010-05-03) ==================== - Having 'sampletests' in the MANIFEST.in gave warnings, but doesn't actually seem to include any more files, so I removed it. - Moved zope.testing.exceptions to zope.testrunner.exceptions. Now zope.testrunner no longer requires zope.testing except for when running its own tests. 4.0.0b1 (2010-04-29) ==================== - Initial release of the testrunner from zope.testrunner as its own module. (Previously it was part of zope.testing.) .. _six: http://pypi.python.org/pypi/six zope.testrunner-4.4.9/LICENSE.rst0000664000000000000000000000402612527371226015242 0ustar rootrootZope Public License (ZPL) Version 2.1 A copyright notice accompanies this license document that identifies the copyright holders. This license has been certified as open source. It has also been designated as GPL compatible by the Free Software Foundation (FSF). Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions in source code must retain the accompanying copyright notice, this list of conditions, and the following disclaimer. 2. Redistributions in binary form must reproduce the accompanying copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Names of the copyright holders must not be used to endorse or promote products derived from this software without prior written permission from the copyright holders. 4. The right to distribute this software or to use it for any purpose does not give you the right to use Servicemarks (sm) or Trademarks (tm) of the copyright holders. Use of them is covered by separate agreement with the copyright holders. 5. If any files are modified, you must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. Disclaimer THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. zope.testrunner-4.4.9/buildout.cfg0000664000000000000000000000031312527371226015731 0ustar rootroot[buildout] develop = . parts = test python [test] recipe = zc.recipe.testrunner eggs = zope.testrunner[test] [python] recipe = zc.recipe.egg eggs = ${test:eggs} scripts = nothanks interpreter = python zope.testrunner-4.4.9/setup.cfg0000664000000000000000000000012212527371244015240 0ustar rootroot[sdist] formats = zip [egg_info] tag_build = tag_date = 0 tag_svn_revision = 0